PMM3 external Postgres environment variables not working, log says unknown #3954
Description
Description
When passing PMM_POSTGRES_ADDR, PMM_POSTGRES_DBNAME, etc. as documented here: https://docs.percona.com/percona-monitoring-and-management/3/reference/third-party/postgresql.html into the pmm-server docker container, they don't seem to do anything. It still tries to start the internal server, it logs out that these env vars are unknown.
The goal is to not have PMM3 run its own postgres at all, as it would be redundant.
Expected Results
I expected to be able to use the environment variables documented here: https://docs.percona.com/percona-monitoring-and-management/3/reference/third-party/postgresql.html to set up pmm-server to use a external Postgres DB.
Actual Results
After setting these variables in my docker-compose.yml environment section for pmm-server, i noticed the following behaviour:
starting PostgreSQL 14.17 - Percona Distribution on x86_64-pc-linux-gnuinpmm-serverlogs (this should not happen, i've set PMM_DISABLE_BUILTIN_POSTGRES=1)Configuration warning: unknown environment variable \"PMM_POSTGRES_USERNAME ...for every PMM_POSTGRES_* variable i've setpmm-managedexits with status 1 every ~10 seconds- web-ui is never accessible
Version
Version: pmm-server:3 tag from Docker Hub
Steps to reproduce
All that should be required to reproduce is starting this docker-compose.yml (approx. equivalent to the documented docker run command with extra env vars)
services:
pmm-server:
image: percona/pmm-server:3
restart: unless-stopped
ports:
- 8443:8443
environment:
- PMM_ENABLE_ALERTING=false
- PMM_PUBLIC_ADDRESS=pmm.example.com
- PMM_POSTGRES_ADDR=192.168.178.99:5432
- PMM_POSTGRES_DBNAME=percona_monitoring
- PMM_POSTGRES_USERNAME=percona_monitoring
- PMM_POSTGRES_DBPASSWORD=[REDACTED]
- PMM_POSTGRES_SSL_MODE=DISABLE
- PMM_DISABLE_BUILTIN_POSTGRES=1
- GF_DATABASE_TYPE=postgres
- GF_DATABASE_HOST=192.168.178.99:5432
- GF_DATABASE_NAME=percona_monitoring
- GF_DATABASE_USER=percona_monitoring
- GF_DATABASE_PASSWORD=[REDACTED]
volumes:
- ./pmm:/srv
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
Relevant logs
docker compose logs -f pmm-server
pmm-server-1 | Initializing /srv...
pmm-server-1 | Copying grafana plugins and the VERSION file...
pmm-server-1 | Generating self-signed certificates for nginx...
pmm-server-1 | Initializing Postgres...
pmm-server-1 | The files belonging to this database system will be owned by user "pmm".
pmm-server-1 | This user must also own the server process.
pmm-server-1 |
pmm-server-1 | The database cluster will be initialized with locale "en_US.utf8".
pmm-server-1 | The default database encoding has accordingly been set to "UTF8".
pmm-server-1 | The default text search configuration will be set to "english".
pmm-server-1 |
pmm-server-1 | Data page checksums are disabled.
pmm-server-1 |
pmm-server-1 | fixing permissions on existing directory /srv/postgres14 ... ok
pmm-server-1 | creating subdirectories ... ok
pmm-server-1 | selecting dynamic shared memory implementation ... posix
pmm-server-1 | selecting default max_connections ... 100
pmm-server-1 | selecting default shared_buffers ... 128MB
pmm-server-1 | selecting default time zone ... Europe/Oslo
pmm-server-1 | creating configuration files ... ok
pmm-server-1 | running bootstrap script ... ok
pmm-server-1 | performing post-bootstrap initialization ... ok
pmm-server-1 | syncing data to disk ... ok
pmm-server-1 |
pmm-server-1 | Success. You can now start the database server using:
pmm-server-1 |
pmm-server-1 | /usr/pgsql-14/bin/pg_ctl -D /srv/postgres14 -l logfile start
pmm-server-1 |
pmm-server-1 | Enabling pg_stat_statements extension for PostgreSQL...
pmm-server-1 | waiting for server to start....2025-04-28 18:55:22.375 CEST [29] LOG: starting PostgreSQL 14.17 - Percona Distribution on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.2.1 20240801 (Red Hat 14.2.1-1), 64-bit
pmm-server-1 | 2025-04-28 18:55:22.375 CEST [29] LOG: listening on IPv6 address "::1", port 5432
pmm-server-1 | 2025-04-28 18:55:22.375 CEST [29] LOG: listening on IPv4 address "127.0.0.1", port 5432
pmm-server-1 | 2025-04-28 18:55:22.387 CEST [29] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
pmm-server-1 | 2025-04-28 18:55:22.396 CEST [29] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
pmm-server-1 | 2025-04-28 18:55:22.402 CEST [30] LOG: database system was shut down at 2025-04-28 18:55:22 CEST
pmm-server-1 | 2025-04-28 18:55:22.414 CEST [29] LOG: database system is ready to accept connections
pmm-server-1 | 2025-04-28 18:55:22.420 CEST [36] LOG: Could not open file /usr/local/percona/telemetry/pg/7498408691604910103.temp for writing.
pmm-server-1 | 2025-04-28 18:55:22.420 CEST [36] LOG: percona_pg_telemetry bgworker exiting with error_code = 2
pmm-server-1 | done
pmm-server-1 | server started
pmm-server-1 | CREATE EXTENSION
pmm-server-1 | waiting for server to shut down....2025-04-28 18:55:22.507 CEST [29] LOG: received fast shutdown request
pmm-server-1 | 2025-04-28 18:55:22.509 CEST [29] LOG: aborting any active transactions
pmm-server-1 | 2025-04-28 18:55:22.511 CEST [29] LOG: background worker "logical replication launcher" (PID 37) exited with exit code 1
pmm-server-1 | 2025-04-28 18:55:22.511 CEST [31] LOG: shutting down
pmm-server-1 | 2025-04-28 18:55:22.541 CEST [29] LOG: database system is shut down
pmm-server-1 | done
pmm-server-1 | server stopped
pmm-server-1 | time="2025-04-28T18:55:22.616+02:00" level=warning msg="Configuration warning: unknown environment variable \"PMM_POSTGRES_USERNAME=percona_monitoring\"."
pmm-server-1 | time="2025-04-28T18:55:22.616+02:00" level=warning msg="Configuration warning: unknown environment variable \"PMM_POSTGRES_ADDR=192.168.3.99:5432\"."
pmm-server-1 | time="2025-04-28T18:55:22.616+02:00" level=warning msg="Configuration warning: unknown environment variable \"PMM_POSTGRES_SSL_MODE=disable\"."
pmm-server-1 | time="2025-04-28T18:55:22.616+02:00" level=warning msg="Configuration warning: unknown environment variable \"PMM_DISABLE_BUILTIN_POSTGRES=1\"."
pmm-server-1 | time="2025-04-28T18:55:22.616+02:00" level=warning msg="Configuration warning: unknown environment variable \"PMM_POSTGRES_DBNAME=percona_monitoring\"."
pmm-server-1 | time="2025-04-28T18:55:22.616+02:00" level=warning msg="Configuration warning: unknown environment variable \"PMM_POSTGRES_DBPASSWORD=[REDACTED]\"."
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Included extra file "/etc/supervisord.d/grafana.ini" during parsing
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Included extra file "/etc/supervisord.d/pmm.ini" during parsing
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Included extra file "/etc/supervisord.d/qan-api2.ini" during parsing
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Included extra file "/etc/supervisord.d/supervisord.ini" during parsing
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Included extra file "/etc/supervisord.d/victoriametrics.ini" during parsing
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Included extra file "/etc/supervisord.d/vmalert.ini" during parsing
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Included extra file "/etc/supervisord.d/vmproxy.ini" during parsing
pmm-server-1 | 2025-04-28 18:55:22,700 INFO Set uid to user 1000 succeeded
pmm-server-1 | 2025-04-28 18:55:22,704 INFO RPC interface 'supervisor' initialized
pmm-server-1 | 2025-04-28 18:55:22,704 INFO supervisord started with pid 1
pmm-server-1 | 2025-04-28 18:55:23,708 INFO spawned: 'pmm-init' with pid 46
pmm-server-1 | 2025-04-28 18:55:23,712 INFO spawned: 'clickhouse' with pid 47
pmm-server-1 | 2025-04-28 18:55:23,715 INFO spawned: 'grafana' with pid 48
pmm-server-1 | 2025-04-28 18:55:23,715 INFO spawned: 'nginx' with pid 49
pmm-server-1 | 2025-04-28 18:55:23,716 INFO spawned: 'victoriametrics' with pid 50
pmm-server-1 | 2025-04-28 18:55:23,717 INFO spawned: 'vmalert' with pid 51
pmm-server-1 | 2025-04-28 18:55:23,718 INFO spawned: 'vmproxy' with pid 52
pmm-server-1 | 2025-04-28 18:55:23,719 INFO spawned: 'qan-api2' with pid 62
pmm-server-1 | 2025-04-28 18:55:23,720 INFO spawned: 'pmm-managed' with pid 67
pmm-server-1 | 2025-04-28 18:55:23,728 INFO exited: qan-api2 (exit status 1; not expected)
pmm-server-1 | 2025-04-28 18:55:24,331 INFO waiting for grafana to stop
pmm-server-1 | 2025-04-28 18:55:24,338 INFO stopped: grafana (exit status 0)
pmm-server-1 | 2025-04-28 18:55:24,557 INFO spawned: 'grafana' with pid 458
pmm-server-1 | 2025-04-28 18:55:24,651 INFO spawned: 'pmm-agent' with pid 486
pmm-server-1 | 2025-04-28 18:55:24,713 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:24,713 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:24,723 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:24,723 INFO success: victoriametrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:24,723 INFO success: vmalert entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:24,723 INFO success: vmproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:24,723 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:24,734 INFO spawned: 'qan-api2' with pid 526
pmm-server-1 | 2025-04-28 18:55:25,568 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:25,673 INFO success: pmm-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:55:25,770 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 18:57:55,548 INFO exited: pmm-init (exit status 2; not expected)
pmm-server-1 | 2025-04-28 18:57:55,864 INFO spawned: 'pmm-init' with pid 1233
pmm-server-1 | 2025-04-28 18:57:56,870 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 19:00:27,526 INFO exited: pmm-init (exit status 2; not expected)
pmm-server-1 | 2025-04-28 19:00:27,880 INFO spawned: 'pmm-init' with pid 1850
pmm-server-1 | 2025-04-28 19:00:29,170 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 19:02:59,577 INFO exited: pmm-init (exit status 2; not expected)
pmm-server-1 | 2025-04-28 19:03:00,004 INFO spawned: 'pmm-init' with pid 2392
pmm-server-1 | 2025-04-28 19:03:01,007 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-server-1 | 2025-04-28 19:05:31,683 INFO exited: pmm-init (exit status 2; not expected)
pmm-server-1 | 2025-04-28 19:05:31,694 INFO spawned: 'pmm-init' with pid 3030
pmm-server-1 | 2025-04-28 19:05:32,871 INFO success: pmm-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
pmm-init.log
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]
TASK [initialization : Detect AMI distribution] ********************************
ok: [127.0.0.1]
TASK [initialization : Set is_ami fact] ****************************************
ok: [127.0.0.1]
TASK [initialization : Get current PMM version (aka 'update from')] ************
[WARNING]: Unable to find '/srv/grafana/PERCONA_DASHBOARDS_VERSION' in expected
paths (use -vvvvv to see paths)
ok: [127.0.0.1]
TASK [initialization : Get PMM image version for update] ***********************
ok: [127.0.0.1]
TASK [initialization : Set need_initialization fact] ***************************
ok: [127.0.0.1]
TASK [initialization : Set need_upgrade fact] **********************************
ok: [127.0.0.1]
TASK [initialization : Print PMM distribution] *********************************
ok: [127.0.0.1] => {
"msg": "PMM distribution: docker, Is AMI: False"
}
TASK [initialization : Show PMM current and target image versions] *************
ok: [127.0.0.1] => {
"msg": "Current version: 0, Image Version: 3.1.0"
}
TASK [initialization : Print need_initialization fact] *************************
ok: [127.0.0.1] => {
"msg": "Need initialization: True"
}
TASK [initialization : Print need_upgrade fact] ********************************
ok: [127.0.0.1] => {
"msg": "Need upgrade: True"
}
TASK [initialization : Enable maintenance mode before upgrade] *****************
changed: [127.0.0.1]
TASK [initialization : Wait for PostgreSQL to become responsive] ***************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "elapsed": 150, "msg": "Timeout when waiting for 127.0.0.1:5432"}
PLAY RECAP *********************************************************************
127.0.0.1 : ok=12 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]
TASK [initialization : Detect AMI distribution] ********************************
ok: [127.0.0.1]
TASK [initialization : Set is_ami fact] ****************************************
ok: [127.0.0.1]
TASK [initialization : Get current PMM version (aka 'update from')] ************
[WARNING]: Unable to find '/srv/grafana/PERCONA_DASHBOARDS_VERSION' in expected
paths (use -vvvvv to see paths)
ok: [127.0.0.1]
TASK [initialization : Get PMM image version for update] ***********************
ok: [127.0.0.1]
TASK [initialization : Set need_initialization fact] ***************************
ok: [127.0.0.1]
TASK [initialization : Set need_upgrade fact] **********************************
ok: [127.0.0.1]
TASK [initialization : Print PMM distribution] *********************************
ok: [127.0.0.1] => {
"msg": "PMM distribution: docker, Is AMI: False"
}
TASK [initialization : Show PMM current and target image versions] *************
ok: [127.0.0.1] => {
"msg": "Current version: 0, Image Version: 3.1.0"
}
TASK [initialization : Print need_initialization fact] *************************
ok: [127.0.0.1] => {
"msg": "Need initialization: True"
}
TASK [initialization : Print need_upgrade fact] ********************************
ok: [127.0.0.1] => {
"msg": "Need upgrade: True"
}
TASK [initialization : Enable maintenance mode before upgrade] *****************
ok: [127.0.0.1]
TASK [initialization : Wait for PostgreSQL to become responsive] ***************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "elapsed": 150, "msg": "Timeout when waiting for 127.0.0.1:5432"}
PLAY RECAP *********************************************************************
127.0.0.1 : ok=12 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]
TASK [initialization : Detect AMI distribution] ********************************
ok: [127.0.0.1]
TASK [initialization : Set is_ami fact] ****************************************
ok: [127.0.0.1]
TASK [initialization : Get current PMM version (aka 'update from')] ************
[WARNING]: Unable to find '/srv/grafana/PERCONA_DASHBOARDS_VERSION' in expected
paths (use -vvvvv to see paths)
ok: [127.0.0.1]
TASK [initialization : Get PMM image version for update] ***********************
ok: [127.0.0.1]
TASK [initialization : Set need_initialization fact] ***************************
ok: [127.0.0.1]
TASK [initialization : Set need_upgrade fact] **********************************
ok: [127.0.0.1]
TASK [initialization : Print PMM distribution] *********************************
ok: [127.0.0.1] => {
"msg": "PMM distribution: docker, Is AMI: False"
}
TASK [initialization : Show PMM current and target image versions] *************
ok: [127.0.0.1] => {
"msg": "Current version: 0, Image Version: 3.1.0"
}
TASK [initialization : Print need_initialization fact] *************************
ok: [127.0.0.1] => {
"msg": "Need initialization: True"
}
TASK [initialization : Print need_upgrade fact] ********************************
ok: [127.0.0.1] => {
"msg": "Need upgrade: True"
}
TASK [initialization : Enable maintenance mode before upgrade] *****************
ok: [127.0.0.1]
TASK [initialization : Wait for PostgreSQL to become responsive] ***************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "elapsed": 150, "msg": "Timeout when waiting for 127.0.0.1:5432"}
PLAY RECAP *********************************************************************
127.0.0.1 : ok=12 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]
TASK [initialization : Detect AMI distribution] ********************************
ok: [127.0.0.1]
TASK [initialization : Set is_ami fact] ****************************************
ok: [127.0.0.1]
TASK [initialization : Get current PMM version (aka 'update from')] ************
[WARNING]: Unable to find '/srv/grafana/PERCONA_DASHBOARDS_VERSION' in expected
paths (use -vvvvv to see paths)
ok: [127.0.0.1]
TASK [initialization : Get PMM image version for update] ***********************
ok: [127.0.0.1]
TASK [initialization : Set need_initialization fact] ***************************
ok: [127.0.0.1]
TASK [initialization : Set need_upgrade fact] **********************************
ok: [127.0.0.1]
TASK [initialization : Print PMM distribution] *********************************
ok: [127.0.0.1] => {
"msg": "PMM distribution: docker, Is AMI: False"
}
TASK [initialization : Show PMM current and target image versions] *************
ok: [127.0.0.1] => {
"msg": "Current version: 0, Image Version: 3.1.0"
}
TASK [initialization : Print need_initialization fact] *************************
ok: [127.0.0.1] => {
"msg": "Need initialization: True"
}
TASK [initialization : Print need_upgrade fact] ********************************
ok: [127.0.0.1] => {
"msg": "Need upgrade: True"
}
TASK [initialization : Enable maintenance mode before upgrade] *****************
ok: [127.0.0.1]
TASK [initialization : Wait for PostgreSQL to become responsive] ***************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "elapsed": 150, "msg": "Timeout when waiting for 127.0.0.1:5432"}
PLAY RECAP *********************************************************************
127.0.0.1 : ok=12 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [127.0.0.1]
TASK [initialization : Detect AMI distribution] ********************************
ok: [127.0.0.1]
TASK [initialization : Set is_ami fact] ****************************************
ok: [127.0.0.1]
TASK [initialization : Get current PMM version (aka 'update from')] ************
[WARNING]: Unable to find '/srv/grafana/PERCONA_DASHBOARDS_VERSION' in expected
paths (use -vvvvv to see paths)
ok: [127.0.0.1]
TASK [initialization : Get PMM image version for update] ***********************
ok: [127.0.0.1]
TASK [initialization : Set need_initialization fact] ***************************
ok: [127.0.0.1]
TASK [initialization : Set need_upgrade fact] **********************************
ok: [127.0.0.1]
TASK [initialization : Print PMM distribution] *********************************
ok: [127.0.0.1] => {
"msg": "PMM distribution: docker, Is AMI: False"
}
TASK [initialization : Show PMM current and target image versions] *************
ok: [127.0.0.1] => {
"msg": "Current version: 0, Image Version: 3.1.0"
}
TASK [initialization : Print need_initialization fact] *************************
ok: [127.0.0.1] => {
"msg": "Need initialization: True"
}
TASK [initialization : Print need_upgrade fact] ********************************
ok: [127.0.0.1] => {
"msg": "Need upgrade: True"
}
TASK [initialization : Enable maintenance mode before upgrade] *****************
ok: [127.0.0.1]
TASK [initialization : Wait for PostgreSQL to become responsive] ***************
pmm-managed.log
time="2025-04-28T18:55:23.731+02:00" level=info msg="Log level: info."
time="2025-04-28T18:55:23.732+02:00" level=info msg="Using default metrics config" component=config
time="2025-04-28T18:55:23.733+02:00" level=info msg="Using default Percona Platform address \"https://check.percona.com\"."
time="2025-04-28T18:55:23.733+02:00" level=info msg="Migrating database..." component=migration
time="2025-04-28T18:55:23.740+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:23.740+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:23.740+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:23.740+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:23.742+02:00" level=info msg="Using default Percona Platform address \"https://check.percona.com\"."
time="2025-04-28T18:55:23.742+02:00" level=info msg="Environment variable \"PMM_DEV_PERCONA_PLATFORM_API_TIMEOUT\" is not set, using \"30s\" as a default timeout for platform API." component="portal client"
time="2025-04-28T18:55:23.745+02:00" level=warning msg="ClickHouse DB is not reachable [tcp://127.0.0.1:9000/pmm]: dial tcp 127.0.0.1:9000: connect: connection refused" component=telemetry
time="2025-04-28T18:55:23.748+02:00" level=info msg="Downloading templates from https://check.percona.com ..." component="portal client"
time="2025-04-28T18:55:24.153+02:00" level=info msg="Updating settings..." component=setup
time="2025-04-28T18:55:24.153+02:00" level=warning msg="unknown environment variable \"PMM_DISABLE_BUILTIN_POSTGRES=1\"" component=server
time="2025-04-28T18:55:24.153+02:00" level=warning msg="unknown environment variable \"PMM_POSTGRES_ADDR=192.168.3.99:5432\"" component=server
time="2025-04-28T18:55:24.153+02:00" level=warning msg="unknown environment variable \"PMM_POSTGRES_DBNAME=percona_monitoring\"" component=server
time="2025-04-28T18:55:24.153+02:00" level=warning msg="unknown environment variable \"PMM_POSTGRES_DBPASSWORD=[REDACTED]\"" component=server
time="2025-04-28T18:55:24.153+02:00" level=warning msg="unknown environment variable \"PMM_POSTGRES_SSL_MODE=disable\"" component=server
time="2025-04-28T18:55:24.153+02:00" level=warning msg="unknown environment variable \"PMM_POSTGRES_USERNAME=percona_monitoring\"" component=server
time="2025-04-28T18:55:24.156+02:00" level=info msg="Updating supervisord configuration..." component=setup
time="2025-04-28T18:55:24.156+02:00" level=info msg="vmproxy configuration not changed, doing nothing." component=supervisord
time="2025-04-28T18:55:24.156+02:00" level=info msg="qan-api2 configuration not changed, doing nothing." component=supervisord
time="2025-04-28T18:55:24.565+02:00" level=info msg="grafana configuration reloaded." component=supervisord
time="2025-04-28T18:55:24.565+02:00" level=info msg="victoriametrics configuration not changed, doing nothing." component=supervisord
time="2025-04-28T18:55:24.565+02:00" level=info msg="vmalert configuration not changed, doing nothing." component=supervisord
time="2025-04-28T18:55:24.565+02:00" level=info msg="Checking VictoriaMetrics..." component=setup
time="2025-04-28T18:55:24.566+02:00" level=info msg="Checking VMAlert..." component=setup
time="2025-04-28T18:55:24.566+02:00" level=info msg="Setup completed." component=setup
time="2025-04-28T18:55:24.566+02:00" level=info msg="Starting services..." component=main
time="2025-04-28T18:55:24.566+02:00" level=info msg=Starting... component=vmalert
time="2025-04-28T18:55:24.566+02:00" level=info msg=Starting... component=victoriametrics
time="2025-04-28T18:55:24.566+02:00" level=info msg=Starting... service=updater
time="2025-04-28T18:55:24.566+02:00" level=info msg="Starting server on http://127.0.0.1:7772/ ..." component=JSON
time="2025-04-28T18:55:24.566+02:00" level=info msg="Starting server on http://127.0.0.1:7771/ ..." component=gRPC
time="2025-04-28T18:55:24.566+02:00" level=info msg="Starting telemetry" component=ha-services
time="2025-04-28T18:55:24.566+02:00" level=info msg="Starting checks" component=ha-services
time="2025-04-28T18:55:24.567+02:00" level=info msg=Starting... component=checks
time="2025-04-28T18:55:24.567+02:00" level=info msg="Starting pmm-agent-runner" component=ha-services
time="2025-04-28T18:55:24.567+02:00" level=info msg="Starting cleaner" component=ha-services
time="2025-04-28T18:55:24.567+02:00" level=info msg="Using default Percona Platform address \"https://check.percona.com\"."
time="2025-04-28T18:55:24.566+02:00" level=info msg="Starting versionCache" component=ha-services
time="2025-04-28T18:55:24.567+02:00" level=info msg="Starting agentsRegistry" component=ha-services
time="2025-04-28T18:55:24.567+02:00" level=info msg="Starting scheduler" component=ha-services
time="2025-04-28T18:55:24.567+02:00" level=info msg="Starting server on http://127.0.0.1:7773/debug\nRegistered handlers:\n\thttp://127.0.0.1:7773/debug/metrics\n\thttp://127.0.0.1:7773/debug/vars\n\thttp://127.0.0.1:7773/debug/requests\n\thttp://127.0.0.1:7773/debug/events\n\thttp://127.0.0.1:7773/debug/pprof" component=debug
time="2025-04-28T18:55:24.570+02:00" level=info msg="Downloading advisors from https://check.percona.com ..." component="portal client"
time="2025-04-28T18:55:24.662+02:00" level=info msg="Starting Stream /agent.v1.AgentService/Connect ..." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:55:24.665+02:00" level=info msg="Connected pmm-agent: &{ID:pmm-server Version:3.1.0 MetricsPort:0}." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:55:24.665+02:00" level=info msg="Starting runStateChangeHandler ..." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:55:25.568+02:00" level=info msg="Configuration reloaded." component=vmalert
time="2025-04-28T18:55:25.570+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.570+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.570+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.570+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.599+02:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2025-04-28T18:55:25.672+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.672+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.672+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.672+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.785+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:25.785+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:26.784+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:26.784+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:26.787+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:26.787+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:26.787+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:26.787+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:26.807+02:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2025-04-28T18:55:27.784+02:00" level=info msg="SetState response: ." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:55:27.784+02:00" level=warning msg="sendSetStateRequest took 2.118355622s." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:55:27.795+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:27.795+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:27.809+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:27.809+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:27.809+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:27.809+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:27.828+02:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2025-04-28T18:55:28.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.790+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.790+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.790+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.790+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.830+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.830+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.830+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.830+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:28.850+02:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2025-04-28T18:55:28.883+02:00" level=info msg="SetState response: ." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:55:29.889+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:29.889+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:29.889+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:29.889+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:29.981+02:00" level=info msg="SetState response: ." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:55:34.568+02:00" level=info msg=Starting... component=version-cache
time="2025-04-28T18:55:34.568+02:00" level=info msg="Updating versions..." component=version-cache
time="2025-04-28T18:55:34.571+02:00" level=info msg="Done. Next check in 24h0m0s." component=version-cache
time="2025-04-28T18:55:38.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:38.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:38.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:38.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:48.167+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:48.167+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:48.167+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:48.167+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:58.164+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:58.164+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:58.164+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:55:58.164+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:00.011+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:00.011+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:00.014+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:00.014+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.015+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.015+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.015+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.015+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.015+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.016+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.016+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.016+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:01.035+02:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2025-04-28T18:56:01.108+02:00" level=info msg="SetState response: ." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:56:02.040+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.040+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.040+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.040+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.065+02:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2025-04-28T18:56:02.115+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.115+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.115+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.115+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:02.214+02:00" level=info msg="SetState response: ." agent_id=pmm-server request=943f8149-2451-11f0-b9d3-72c8e7ad069c
time="2025-04-28T18:56:08.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:08.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:08.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:08.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:18.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:18.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:18.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:18.166+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:25.054+02:00" level=info msg="Downloading advisors from https://check.percona.com ..." component="portal client"
time="2025-04-28T18:56:25.541+02:00" level=info msg="Executing check: mysql_config_binlog_row_image with interval: standard" component=checks
time="2025-04-28T18:56:25.541+02:00" level=info msg="Executing check: mysql_automatic_expired_password with interval: standard" component=checks
time="2025-04-28T18:56:25.541+02:00" level=info msg="Executing check: mysql_configuration_log_verbosity with interval: standard" component=checks
time="2025-04-28T18:56:25.541+02:00" level=info msg="Executing check: mysql_performance_temp_ondisk_table_high with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_config_binlog_retention_period with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_password_expiry with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_test_database with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_security_user_ssl with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_config_general_log with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_timezone with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_replication_grants with interval: standard" component=checks
time="2025-04-28T18:56:25.542+02:00" level=info msg="Executing check: mysql_security_password_lifetime with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_security_root_not_local with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_tables_without_pk with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_configuration_secure_file_priv_empty with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_config_sql_mode with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_security_anonymous_user with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_security_user_without_password with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_version with interval: standard" component=checks
time="2025-04-28T18:56:25.543+02:00" level=info msg="Executing check: mysql_automatic_sp_privileges_enabled with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_configuration_max_connections_usage with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_version_eol_57 with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_security_password_policy with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_unsupported_version_check with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_config_tmp_table_size_limit with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_security_replication_grants_mixed with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_config_log_bin with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_security_open_to_world_host with interval: standard" component=checks
time="2025-04-28T18:56:25.544+02:00" level=info msg="Executing check: mysql_config_binlogs_checksummed with interval: standard" component=checks
time="2025-04-28T18:56:25.545+02:00" level=info msg="Executing check: mysql_security_user_super_not_local with interval: standard" component=checks
time="2025-04-28T18:56:25.545+02:00" level=info msg="Executing check: mysql_config_local_infile with interval: standard" component=checks
time="2025-04-28T18:56:25.545+02:00" level=info msg="Executing check: mysql_require_secure_transport with interval: standard" component=checks
time="2025-04-28T18:56:25.545+02:00" level=info msg="Executing check: postgresql_archiver_failing_1 with interval: standard" component=checks
time="2025-04-28T18:56:25.546+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:25.546+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:26.564+02:00" level=warning msg="Failed to execute check postgresql_archiver_failing_1 of type POSTGRESQL_SELECT on target pmm-server: action 9915911c-361b-47d9-8e8f-9a88feb08220 failed: pq: password authentication failed for user \"AfXNRL+HnXkSTRWJaiLSQnSZvL4bJLMf8SSV7LZ/pr58C+YdNMPHmeqVHqKMwdr7LBLy\"\ngithub.com/percona/pmm/managed/services/checks.(*Service).waitForResult\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:514\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQueryForSingleDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:889\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQuery\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:857\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck.func5\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:713\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/builder/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQueryForSingleDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:891\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQuery\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:857\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck.func5\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:713\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/builder/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\ncheck query failed\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:771\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeChecksForTargetType\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:657\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeChecks\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:634\ngithub.com/percona/pmm/managed/services/checks.(*Service).run\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:307\ngithub.com/percona/pmm/managed/services/checks.(*Service).runChecksGroup\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:276\ngithub.com/percona/pmm/managed/services/checks.(*Service).runChecksLoop\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:223\ngithub.com/percona/pmm/managed/services/checks.(*Service).Run.func1\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:214\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700" component=checks
time="2025-04-28T18:56:26.564+02:00" level=info msg="Executing check: postgresql_number_of_index_check with interval: standard" component=checks
time="2025-04-28T18:56:26.566+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:26.566+02:00" level=warning msg="aead_factory: decryption failed"
time="2025-04-28T18:56:27.580+02:00" level=warning msg="Failed to execute check postgresql_number_of_index_check of type on target pmm-server: action 9608bb3d-77e6-41f1-b825-47bb271304c4 failed: pq: password authentication failed for user \"AfXNRL+HnXkSTRWJaiLSQnSZvL4bJLMf8SSV7LZ/pr58C+YdNMPHmeqVHqKMwdr7LBLy\"\ngithub.com/percona/pmm/managed/services/checks.(*Service).waitForResult\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:514\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQueryForSingleDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:889\ngithub.com/percona/pmm/managed/services/checks.(*Service).discoverAvailablePGDatabases\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:1175\ngithub.com/percona/pmm/managed/services/checks.(*Service).splitPGTargetByDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:1208\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQuery\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:860\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck.func5\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:713\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/builder/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQueryForSingleDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:891\ngithub.com/percona/pmm/managed/services/checks.(*Service).discoverAvailablePGDatabases\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:1175\ngithub.com/percona/pmm/managed/services/checks.(*Service).splitPGTargetByDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:1208\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQuery\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:860\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck.func5\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:713\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/builder/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\nfailed to select available databases\ngithub.com/percona/pmm/managed/services/checks.(*Service).discoverAvailablePGDatabases\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:1177\ngithub.com/percona/pmm/managed/services/checks.(*Service).splitPGTargetByDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:1208\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQuery\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:860\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck.func5\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:713\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/builder/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\ngithub.com/percona/pmm/managed/services/checks.(*Service).splitPGTargetByDB\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:1210\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQuery\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:860\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck.func5\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:713\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/builder/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\nfailed to split target by db\ngithub.com/percona/pmm/managed/services/checks.(*Service).executePostgreSQLSelectQuery\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:862\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck.func5\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:713\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/builder/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\ncheck query failed\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeCheck\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:771\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeChecksForTargetType\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:657\ngithub.com/percona/pmm/managed/services/checks.(*Service).executeChecks\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:634\ngithub.com/percona/pmm/managed/services/checks.(*Service).run\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:307\ngithub.com/percona/pmm/managed/services/checks.(*Service).runChecksGroup\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:276\ngithub.com/percona/pmm/managed/services/checks.(*Service).runChecksLoop\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:223\ngithub.com/percona/pmm/managed/services/checks.(*Service).Run.func1\n\t/home/builder/rpm/BUILD/pmm-4b5821b21539ca4bd9b31fa46510ad9f69d322bf/src/github.com/percona/pmm/managed/services/checks/checks.go:214\nruntime.goexit\n\t/home/builder/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700" component=checks
time="2025-04-28T18:56:27.580+02:00" level=info msg="Executing check: postgresql_unused_index_check with interval: standard" component=checks
it keeps logging that aead_factory: decryption failed line evey 10 seconds
Code of Conduct
- I agree to follow Percona Community Code of Conduct