Uvicorn >=0.30 dies on Kubernetes only

[Epse]

I have a FastAPI app that runs via uvicorn, in a Docker container on Kubernetes. Recently I attempted to upgrade uvicorn to the latest version, but when upgrading past 0.29 (into 0.30 and above), the webserver never comes alive. All worker processes seem to just die (log at the bottom). The odd part is that this does not happen at all locally, even when using the exact same Docker image.

I found some references to a similar issue, which notes it should be fixed in 0.30.2, but in my experience 0.29 is the most recent version that works.

Does anyone have a similar experience? If not, I might try to build a repro example, if the issue is not too specific to just our cluster...

Log (repeats at infinitum untill killed by kubernetes):

INFO:     Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
INFO:     Started parent process [1]
INFO:     Waiting for child process [7]
INFO:     Child process [7] died
INFO:     Waiting for child process [15]
INFO:     Child process [15] died
INFO:     Waiting for child process [8]
INFO:     Child process [8] died
INFO:     Waiting for child process [10]
INFO:     Child process [10] died

[thaikoh]

Similar problem here, processes works fine for a limited amount of time, then dies and restarts with no given reason. These restarts lead to front nginx throws 502 Bad Gateway. Log of uvicorn follows:

INFO:     Waiting for child process [11]
INFO:     Child process [11] died
INFO:     Started server process [32]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     172.24.0.1:39740 - "GET ... HTTP/1.0" 200 OK
INFO:     172.24.0.1:39728 - "GET ... HTTP/1.0" 200 OK
INFO:     Waiting for child process [10]
INFO:     Child process [10] died
INFO:     Started server process [39]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     172.24.0.1:47692 - "GET ... HTTP/1.0" 200 OK
INFO:     Waiting for child process [9]
INFO:     Child process [9] died
INFO:     172.24.0.1:47702 - "GET ... HTTP/1.0" 200 OK
INFO:     Started server process [46]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     172.24.0.1:46558 - "GET ... HTTP/1.0" 200 OK
INFO:     172.24.0.1:46548 - "GET ... HTTP/1.0" 200 OK
INFO:     172.24.0.1:47996 - "GET ... HTTP/1.0" 200 OK
INFO:     172.24.0.1:48000 - "GET ... HTTP/1.0" 200 OK
INFO:     Waiting for child process [12]
INFO:     Child process [12] died
INFO:     172.24.0.1:33276 - "GET ... HTTP/1.0" 200 OK
INFO:     Started server process [53]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Even --log-level debug doesn't give more details.
Version I use uvicorn==0.30.6.

[Epse]

Someone suggested adding tini as a wrapper / main process; I can confirm this does not make any difference (on my setup)

[thaikoh]

Another 50 cents: low memory could cause that, but I can't confirm it.

[bitoffdev]

For what it's worth, in my case the issue was not enough CPU. I was running within a kubernetes pod with requests and limits of 500m and spotted the issue via kubernetes metrics. Increasing the CPU requests/limits to 1 fixed the issue.

Specifically, there is a timeout of 5 seconds for worker startup, and due to CPU throttling, the workers are failing to start up before hitting the timeout. This is confirmed by the fact that (for me) the child died messages are emitted every 5 seconds. If you also see child died messages every 5 seconds, your issue is probably CPU-throttling.

[linchpinlin]

One of the interfaces of my fastapi is designed to handle PPT files and uses a process pool. I have observed that when it processes very large PPT files (~30Mb), the child processes exit and then restart for no apparent reason:

INFO:     Waiting for child process [105]
INFO:     Child process [105] died

It doesn't show any errors, and I can't see errors using trace level logging.
I can confirm that it's not OOM.
When I switched from uvicorn=0.30.6 to version 0.29.0, the problem disappeared.
I'm very sorry I can't provide more information to assist in resolving this issue, just wanted to provide a temporary solution.

[linchpinlin]

docker image: python:3.10.14-bookworm
uvicorn=0.30.6

[Alex37882388]

same here, it's worked for me. THX

[mqfs]

Worked for me. THX +1

[josumoreno-BP]

Hey, I've been checking this for some hours. I'm able to reproduce it in low/limited resource environments. For example, limiting a docker CPU to 0.2 with 2 workers shows some "Child process [x] died" messages. It happens since version 0.30.0.

Reproducible example

cat requiremets.txt

uvicorn==0.30.6
fastapi==0.114.2

cat main.py

from fastapi import FastAPI
# import time

def app():
    api = FastAPI()
    return api    

if __name__ == "__main__":
    import uvicorn
    uvicorn.run("main:app", host="0.0.0.0", port=8000, factory=True, workers=2)

cat Dockerfile

FROM python:3.11-slim-bullseye

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

CMD ["python", "main.py"]

Build and run with low resources:

docker build -t app . && docker run -p 8000:8000 --cpus=".2" app

Expected output (I get 1 "Child process [x] died"):

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started parent process [1]
INFO:     Waiting for child process [8]
INFO:     Child process [8] died
INFO:     Started server process [9]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [10]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

If we test this example with uvicorn==0.29.0 it starts up normally:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)     
INFO:     Started parent process [1]
INFO:     Started server process [8]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [9]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Getting more Child process died messages
Seems the more we import, the more it will fail. Uncomment the # import time and build and run again. I get:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started parent process [1]
INFO:     Waiting for child process [8]
INFO:     Child process [8] died
INFO:     Waiting for child process [9]
INFO:     Child process [9] died
INFO:     Waiting for child process [10]
INFO:     Child process [10] died
INFO:     Waiting for child process [11]
INFO:     Child process [11] died
INFO:     Started server process [12]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [13]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

If we test this example with uvicorn==0.29.0 it starts up normally:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)     
INFO:     Started parent process [1]
INFO:     Started server process [8]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [9]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Not getting the error with slightly different code, example 1

Move the import inside app function like so:

def app():
    from fastapi import FastAPI
    api = FastAPI()
    return api    

if __name__ == "__main__":
    import uvicorn
    uvicorn.run("main:app", host="0.0.0.0", port=8000, factory=True, workers=2)

I get a clean startup:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started parent process [1]
INFO:     Started server process [8]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [9]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Not getting the error with slightly different code, example 2

Changle Dockerfile to run uvicorn directly:

FROM python:3.11-slim-bullseye

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

CMD ["uvicorn", "--host=0.0.0.0", "--workers=2", "--factory","main:app"]

Use the minimum FastAPI factory with the import outside:

from fastapi import FastAPI

def app():
    api = FastAPI()
    return api

I get a clean startup:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started parent process [1]
INFO:     Started server process [9]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [8]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Would you mind @Kludex or @abersheeran opening an issue based on these evidences? Or if this is something expected with the new multiprocess manager implementation released in 0.30.0 at least clarify it? Thanks a lot!

[hunter2009pf]

downgrade uvicorn to version0.29.0. It works for me. Thanks!

[jonross]

The diffs from 0.29.0 to 0.30.0 show this in deployment.md

Unlike gunicorn, uvicorn does not use pre-fork, but uses spawn, which allows uvicorn's multiprocess manager to still work well on Windows.

If this is the difference, it could explain the behavior in 0.30.0. Process forking uses copy-on-write for process memory so the initial hit to overall container memory is small. Spawning creates a completely new process.

If you cat /sys/fs/cgroup/memory/memory.oom_control the oom_kill field will show how many times processes in the container have been terminated for exceeding memory limits.

[Epse]

I am personally running unicorn directly and still getting the issue, but it is a complex app with many imports Odd that perhaps it's gotten more resource intensive, I'll see if I can get different behaviour with different CPU limits, thank you for the research

[Epse]

As also mentioned below, I have attempted removing all resource limits, to no avail. I have noticed that when running the docker image locally it appears to be much more CPU intensive when running under uvicorn >= 0.30 than 0.29 and all our k8s worker nodes are heavily CPU overprovisioned, so this may be the root cause.

[Epse]

Dropping the worker count drastically does allow it to run again normally

[abersheeran]

#2397 Could you please try out this PR? It might solve the problem. If it is confirmed to be a solution, we will expedite the merging of this PR.

[josumoreno-BP]

I have tried now your PR with a higher value than the default with the example that I sent and it worked.

What I don't fully understand is why my example 2 was working compared to the first one, because at the end it is loading the same amount of imports right?

[Epse]

For me the PR doesn't change anything, even setting the timeout up to 120 seconds. It will eventually have enough living workers to start accepting requests, but the logs are still 90% "Child process died". As I'll mention above really quick, I have attempted removing all resource limits, to no avail. I have noticed that when running the docker image locally it appears to be much more CPU intensive when running under uvicorn >= 0.30 than 0.29 and all our k8s worker nodes are heavily CPU overprovisioned, so this may be the root cause.

[VicenteMerino]

is there any workaround for this? im facing a similar issue using google cloud run

[abersheeran]

My guess is that the program you are running requires more memory than the CloudRun configuration allows. We also have Uvicorn running on the CloudRun platform, and for such an auto-scaling platform, there is no need to specify multiple workers, which would increase memory usage in a single container.

[Yilimmilk]

same here.

I encountered issues running my FastAPI project using different versions of Uvicorn within a Docker container. Details are as follows:

1. Environment

   • FastAPI version: 0.115.6
   • Python version: 3.12-alpine
   • Running in Docker

2. Issue with uvicorn <= 0.29.0
   When I use Uvicorn versions <= 0.29.0, the process gets permanently stuck at the following log message with no more output:

2024/12/16 11:44:36.354     INFO: [uvicorn.error] Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)  
2024/12/16 11:44:36.354     INFO: [uvicorn.error] Started parent process [1]  

3. Issue with uvicorn >= 0.30.0 (including the latest version 0.34.0)
   With newer versions of Uvicorn, the process starts, but the child processes keep dying repeatedly:

2024/12/16 12:20:06.593     INFO: [uvicorn.error] Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024/12/16 12:20:06.594     INFO: [uvicorn.error] Started parent process [1]
2024/12/16 12:20:07.622     INFO: [uvicorn.error] Waiting for child process [18]
2024/12/16 12:20:07.623     INFO: [uvicorn.error] Child process [18] died
2024/12/16 12:20:07.623     INFO: [uvicorn.error] Waiting for child process [19]
2024/12/16 12:20:07.624     INFO: [uvicorn.error] Child process [19] died
2024/12/16 12:20:08.127     INFO: [uvicorn.error] Waiting for child process [19]
2024/12/16 12:20:08.129     INFO: [uvicorn.error] Child process [19] died
2024/12/16 12:20:13.136     INFO: [uvicorn.error] Waiting for child process [23]
2024/12/16 12:20:13.137     INFO: [uvicorn.error] Child process [23] died
... (repeats indefinitely) ...  

4. Docker Setup
   It's my Dockerfile:

FROM python:3.12-alpine  

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories \  
    && apk add --update caddy gcc musl-dev libffi-dev  

WORKDIR /app  
COPY Backend/requirements.txt /tmp/requirements.txt  
RUN pip install --no-cache-dir -r /tmp/requirements.txt -i https://mirrors.aliyun.com/pypi/simple  

COPY Backend ./backend  

EXPOSE 8000  

COPY startup.sh /app/startup.sh  
RUN chmod +x /app/startup.sh  
CMD ["/app/startup.sh"]  

My startup.sh script:

#!/bin/sh  
set -e  

cd /app/backend  

exec uvicorn main:get_app \  
    --host 0.0.0.0 \  
    --port 8000 \  
    --workers 2 \  
    --proxy-headers \  
    --forwarded-allow-ips '*' \  
    --factory \  
    --log-config logging_config.yaml  

5. Local Machine Behavior
   When I run the same command directly on my Windows11 PC, The program can run normally:

uvicorn main:get_app --host 0.0.0.0 --port 8000 --workers 2 --proxy-headers --factory --log-config logging_config.yaml  

[Yilimmilk]

New Findings:
After this, I found that the issue with "Child process died" occurs consistently when using the Docker image python:3.12-alpine3.21. However, when I switch to python:3.12-alpine3.20, everything works as expected, and the errors disappear.

[abersheeran]

🤔 Python 3.12.21 what‘s change?

[pblaas]

Thanks a lot for this. Just upgraded to 3.12 which ran fine in the pyenv, but not as containerised app with python:3.12-alpine. The change to python:3.12-alpine3.20 fixed it.

[gmmeyer]

wow after reading this thread and reverting to 0.29 it completely fixed a weird, somewhat transient long running problem I've had where sometimes but not always the app would take 2-4 minutes to start up

[vjeranc]

It's the GIL lock. threading.Thread is used to always pong to the ping of the parent. If you don't pong for 5 seconds (and it's very possible if the GIL is not released -- if your app takes a while to import all of the code dependencies) uvicorn will terminate the child process.

Not sure why this child management was necessary. Especially given that uvicorn uses spawn instead of fork, obliterating all of the copy-on-write savings from fork.

Instead of uvicorn watching your workers with their ping-pong protocol, just have 1 worker per container and watch your containers with proven mechanisms instead. You don't really lose anything, because fork is not used for workers=2+ anyways.

[abersheeran]

Because we need a way to test whether the process has been suspended.

[vjeranc]

Yes, but there are cases where it is not suspended, it's just always_pong being GIL locked.

Aren't OS primitives enough? os or psutil could be used to figure out if child process is actually suspended.

[abersheeran]

Sometimes a process enters a suspended state, but appears alive from the outside.

[vjeranc]

I don't get it, /proc/PID/stat is lying?

[vjeranc]

I've tested this, and it's not GIL. It's just that main.py can be a heavy process (with a lot of imports), so spawn has to "copy" all of that for the child, but main thinks child has started, yet it has not, so the 5s counter starts. thread.start() for the always_pong runs much later.

[racinmat]

It seems #2500 would be able to fix it, and increasing the timeout would fix the issue.

[Petr-Siegl]

Wrote down my findings here FYI:
#2506 (comment)

[vjeranc]

This is what I wrote 3 months ago.

[Petr-Siegl]

Yes, from the technical side, my comment is for anybody looking for a quick solution for the issue right now. Honestly, noticed this discussion after writing it.

At the end the end of the day, you can use other manager or write your own, when not looking for a quick fix. :)