pmm3 won't start after reboot #4031
Description
Description
Built in database. After stack reboot it not runnig due * entered FATAL state, too many start retries too quickly
services:
pmm-server:
image: percona/pmm-server:3
ports:
- 8443:8443
- 8080:8080
Expected Results
start?
Actual Results
nothing.
Version
3.1.0
Steps to reproduce
No response
Relevant logs
2025-05-19 10:22:31,344 INFO Included extra file "/etc/supervisord.d/grafana.ini" during parsing
2025-05-19 10:22:31,344 INFO Included extra file "/etc/supervisord.d/pmm.ini" during parsing
2025-05-19 10:22:31,344 INFO Included extra file "/etc/supervisord.d/qan-api2.ini" during parsing
2025-05-19 10:22:31,344 INFO Included extra file "/etc/supervisord.d/supervisord.ini" during parsing
2025-05-19 10:22:31,344 INFO Included extra file "/etc/supervisord.d/victoriametrics.ini" during parsing
2025-05-19 10:22:31,344 INFO Included extra file "/etc/supervisord.d/vmalert.ini" during parsing
2025-05-19 10:22:31,344 INFO Included extra file "/etc/supervisord.d/vmproxy.ini" during parsing
2025-05-19 10:22:31,344 INFO Set uid to user 1000 succeeded
2025-05-19 10:22:31,361 INFO RPC interface 'supervisor' initialized
2025-05-19 10:22:31,362 INFO supervisord started with pid 1
2025-05-19 10:22:32,365 INFO spawned: 'pmm-init' with pid 12
2025-05-19 10:22:32,369 INFO spawned: 'postgresql' with pid 13
2025-05-19 10:22:32,372 INFO spawned: 'clickhouse' with pid 14
2025-05-19 10:22:32,376 INFO spawned: 'grafana' with pid 15
2025-05-19 10:22:32,379 INFO spawned: 'nginx' with pid 16
2025-05-19 10:22:32,385 INFO spawned: 'victoriametrics' with pid 17
2025-05-19 10:22:32,389 INFO spawned: 'vmalert' with pid 18
2025-05-19 10:22:32,392 INFO spawned: 'vmproxy' with pid 24
2025-05-19 10:22:32,395 INFO spawned: 'qan-api2' with pid 29
2025-05-19 10:22:32,399 INFO spawned: 'pmm-managed' with pid 34
2025-05-19 10:22:32,401 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:22:32,415 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:22:32,532 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:22:33,511 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-19 10:22:33,522 INFO spawned: 'postgresql' with pid 389
2025-05-19 10:22:33,523 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-19 10:22:33,523 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-19 10:22:33,523 INFO success: victoriametrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-19 10:22:33,523 INFO success: vmalert entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-19 10:22:33,523 INFO success: vmproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-19 10:22:33,526 INFO spawned: 'qan-api2' with pid 390
2025-05-19 10:22:33,527 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-05-19 10:22:33,624 INFO spawned: 'grafana' with pid 397
2025-05-19 10:22:33,626 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:22:33,632 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:22:34,236 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:22:36,217 INFO spawned: 'postgresql' with pid 428
2025-05-19 10:22:36,223 INFO spawned: 'qan-api2' with pid 429
2025-05-19 10:22:36,322 INFO spawned: 'grafana' with pid 436
2025-05-19 10:22:36,323 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:22:36,328 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:22:36,632 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:22:39,355 INFO spawned: 'postgresql' with pid 585
2025-05-19 10:22:39,359 INFO spawned: 'qan-api2' with pid 586
2025-05-19 10:22:39,450 INFO exited: pmm-init (exit status 0; expected)
2025-05-19 10:22:39,452 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:22:39,461 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:22:40,517 INFO spawned: 'grafana' with pid 594
2025-05-19 10:22:40,624 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:22:43,519 INFO spawned: 'postgresql' with pid 615
2025-05-19 10:22:43,522 INFO spawned: 'qan-api2' with pid 616
2025-05-19 10:22:43,542 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:22:43,553 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:22:45,521 INFO spawned: 'grafana' with pid 630
2025-05-19 10:22:45,628 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:22:49,523 INFO spawned: 'postgresql' with pid 648
2025-05-19 10:22:49,526 INFO spawned: 'qan-api2' with pid 649
2025-05-19 10:22:49,546 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:22:49,555 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:22:51,524 INFO spawned: 'grafana' with pid 663
2025-05-19 10:22:51,631 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:22:55,764 INFO spawned: 'postgresql' with pid 683
2025-05-19 10:22:55,767 INFO spawned: 'qan-api2' with pid 684
2025-05-19 10:22:55,795 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:22:55,801 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:22:58,211 INFO spawned: 'grafana' with pid 698
2025-05-19 10:22:58,325 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:23:02,900 INFO spawned: 'postgresql' with pid 725
2025-05-19 10:23:02,903 INFO spawned: 'qan-api2' with pid 726
2025-05-19 10:23:02,930 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:23:02,938 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:23:05,534 INFO spawned: 'grafana' with pid 734
2025-05-19 10:23:05,642 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:23:11,537 INFO spawned: 'postgresql' with pid 761
2025-05-19 10:23:11,540 INFO spawned: 'qan-api2' with pid 762
2025-05-19 10:23:11,571 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:23:11,576 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:23:14,539 INFO spawned: 'grafana' with pid 775
2025-05-19 10:23:14,642 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:23:21,356 INFO spawned: 'postgresql' with pid 807
2025-05-19 10:23:21,360 INFO spawned: 'qan-api2' with pid 808
2025-05-19 10:23:21,389 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:23:21,395 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:23:24,433 INFO spawned: 'grafana' with pid 822
2025-05-19 10:23:24,542 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:23:31,551 INFO spawned: 'postgresql' with pid 848
2025-05-19 10:23:31,553 INFO spawned: 'qan-api2' with pid 849
2025-05-19 10:23:31,575 INFO exited: postgresql (exit status 1; not expected)
2025-05-19 10:23:31,580 INFO gave up: postgresql entered FATAL state, too many start retries too quickly
2025-05-19 10:23:31,581 INFO exited: qan-api2 (exit status 1; not expected)
2025-05-19 10:23:34,554 INFO spawned: 'grafana' with pid 863
2025-05-19 10:23:34,659 INFO exited: grafana (exit status 1; not expected)
2025-05-19 10:23:35,553 INFO gave up: grafana entered FATAL state, too many start retries too quicklyCode of Conduct
- I agree to follow Percona Community Code of Conduct