FPM Status: active processes greater than pm.max_children

[bcremer]

Description

Given the following FPM Pool configuration the Pool is configured using 1500 static workers and a listen.backlog of 100:

$ cat /etc/php/8.3/fpm/pool.d/backoffice.conf 
[backoffice]
listen = /run/php8.3-fpm_backoffice.sock
listen.backlog = 100

pm = static
pm.max_children = 1500
pm.status_path = /php8.3-fpm-status-backoffice.php

; The number of requests each child process should execute before respawning. This can be useful to work around memory leaks
pm.max_requests = 2000

The listen.backlog works like expected on the socket.

$ ss -lxn src ==  /run/php8.3-fpm_backoffice.sock
Netid                      State                       Recv-Q                      Send-Q                                                              Local Address:Port                                              Peer Address:Port                      Process
u_str                      LISTEN                      0                           100                                               /run/php8.3-fpm_backoffice.sock 47129                                                        * 0

Without high traffic total processes reports 1500 as expected.

$ curl -s 127.0.0.1:80/php8.3-fpm-status-backoffice.php
pool:                 backoffice
process manager:      static
start time:           13/May/2024:04:02:02 +0200
start since:          22533
accepted conn:        3057525
listen queue:         0
max listen queue:     0
listen queue len:     0
idle processes:       1460
active processes:     40
total processes:      1500
max active processes: 549
max children reached: 0
slow requests:        0

When generating high concurrent requests using a tool like siege or k6 I can observe the following output from the status page:

$ curl 127.0.0.1:80/php8.3-fpm-status-backoffice.php
pool:                 backoffice
process manager:      static
start time:           08/May/2024:04:02:01 +0200
start since:          19565
accepted conn:        10120636
listen queue:         0
max listen queue:     0
listen queue len:     0
idle processes:       1
active processes:     3347
total processes:      3348
max active processes: 3503
max children reached: 0
slow requests:        0

active processes/total processes/max active processes are way over the defined pm.max_children = 1500.

How can active processes be greater than pm.max_children?

Also max children reachedreports 0.

Relates to: #9943 (comment)

PHP Version

PHP 8.3.6

Operating System

Debian 11

[bcremer]

Maybe related:

I can also observe that the number of requests I can measure via the FPM access.log does not match the delta for accepted conn for a given timespan.

The accepted conn delta is approx double the abount of requests shown in the fpm access log.

[txuna]

@bcremer Is it still occurring?

[s2x]

I also have the same problem. We wanted to use this metric to scale pods but unfortunately it returns incorrect results.

fpm settings:

pm = static
pm.max_children = 20

version: PHP 8.4.7 (fpm-fcgi) (built: May 8 2025 21:27:16) (NTS)

can be reproduced by sending large traffic to status php-fpm endpoint

no-load status:

pool: www
process manager: static
start time: 03/Jun/2025:13:11:48 +0000
start since: 23846
accepted conn: 154295
listen queue: 0
max listen queue: 20
listen queue len: 4096
idle processes: 19
active processes: 1 --- one active for status page
total processes: 20  --- this is fine
max active processes: 1597
max children reached: 0
slow requests: 0
memory peak: 14680064

when we run wrk http://localhost/status

pool: www
process manager: static
start time: 03/Jun/2025:13:11:48 +0000
start since: 23914
accepted conn: 158110
listen queue: 0
max listen queue: 20
listen queue len: 4096
idle processes: 0
active processes: 698
total processes: 698 --- sooo big number
max active processes: 1597
max children reached: 0
slow requests: 0
memory peak: 14680064

The number of php processes, checked with the ps command, is correct, so the problem is probably with the display.

[s2x]

Hi @bukka ,

Recently I had some time to properly debug this issue. I wanted to find a minimal configuration that reproduces the problem. At first, I couldn’t reproduce the bug using the official minimal nginx and php-fpm images from Docker Hub.
After many tests, it turned out that the issue is caused by a specific combination of nginx and php-fpm configurations.

It does not happen with the default settings. However, if you enable the fastcgi_keep_conn on; directive in the nginx config, the number of active processes is counted incorrectly. Disabling this directive makes the process counting work as expected again. Below is a link to a minimal example repo that triggers the bug: repo

I also tested older versions of PHP, and it seems that this issue has been present for a long time.

Let me know if you need any more info!

[bcremer]

I can confirm that we also have fastcgi_keep_conn on; in our nginx config.

But we don't use a upstream with keepalive on but directly pass to the fpm socket instead.

fastcgi_pass /run/php8.3-fpm_backoffice.sock

So fastcgi_keep_conn on; is useless in our case anyway.

[bukka]

Apology for delay. I completely missed or forgot about the initial report.

I have been looking into it and managed to recreate the issue using my FPM testing tool (WST) using this test: https://github.com/wstool/wst-php-fpm/blob/70fcd28106ba91967a0243354537b4bf99b03533/spec/instances/status-procs-counter-keepalive.yaml

I did a bit of checking of the code and I can see the issue. The active parameter (used for total processes and some other status fields) gets incremented during the reading headers stage and decremented when in accepting stage. The problem is that with keepalive connection, there is just one accepting stage and multiple reading headers stage. I think it's not right to do incremented in reading headers if it doesn't follow accepting stage. I just need to think how to best implement it. I will try to figure something out when I'm back on this in roughly 2 or 3 weeks.