Gunicorn integration gunicorn.workers likely to gauge workers blocked on network I/O as idle #20171
Description
The Gunicorn integration offers metrics that are not provided by Gunicorn's own statsd reporting capabilities.
Perhaps most notably, it provides gunicorn.workers which gauges the worker processes and attempts to to tag them as busy if their CPU times (measured with psutil) change over a 0.1 second interval; otherwise it tags them as idle.
If a worker is blocking on non-block device I/O (say, using requests to query another API; querying a database; etc.) before and after the check interval, the worker will be tagged as idle but I believe a user would expect this to be tagged as busy.
This was trivial to induce on a container running python:3.9-slim and using two flask apps: one which made requests to the other, which would sleep for 200ms. I was able to reproduce it with sync and gthread workers.
I want to recognize that what this integration is trying to do is quite difficult given what Gunicorn provides, but I wish either of the following were provided instead:
- tag names and documentation that don't give the impression that this integration can gauge worker saturation for all workloads: it's probably only viable for only services that can consistently respond in <100ms.
- the quite substantial work required to monkeypatch Gunicorn and provide a more correct approximation. Without this, the Gunicorn integration provides no value to the projects I work on.