PMM fails to initiate MongoDB backups with X.509 authentication due to missing CA certificate file

[ijager]

Description

After updating Percona MongoDB, PMM server PMM agent and PBM agent to the latest versions I cannot seem to trigger backups from the PMM web interface anymore.

Whenever I try to initiate a backup it immediately goes into "error - no logs"

PMM fails to initiate MongoDB backups when MongoDB is configured with X.509 authentication. The backup job fails with an error indicating it cannot find a CA certificate file in a temporary directory.

The backup fails immediately with the following error in pmm-agent logs:

  Jul 31 11:06:00 pmm-agent[470]: time="2025-07-31T11:06:00.450+00:00" level=warning msg="Job terminated with error: exit status 1
  pbm config error: {\"Error\":\"connect to mongodb: create mongo connection: invalid connection string option: open
  /usr/local/percona/pmm/tmp/mongodb-backup-restore/d997b535-c31d-4aef-b2dc-5521356a3253/caFilePlaceholder: no such file or directory\"}

  github.com/percona/pmm/agent/runner/jobs.pbmConfigure
      /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/pbm_helpers.go:421
  github.com/percona/pmm/agent/runner/jobs.(*MongoDBBackupJob).Run
      /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_backup_job.go:131

Additional Information

• The CA certificate file exists and is readable
• pbm-agent is properly configured and running with the correct MongoDB URI
• Manual backups using pbm backup command work correctly
• PBM configuration in MongoDB shows proper S3 storage settings
• Monitoring of MongoDB works fine
• PBM-agent is recognized in the PBM dashboard
• via pbm cli I can trigger a backup and that works fine
• I can login to mongosh with the pmm user and manually insert a pbmCmd and that is recognized by phm-agent

Workaround

Manually initiate backups either via:

1. PBM CLI: pbm backup --type=logical
2. Direct MongoDB command insertion

Root Cause Analysis

PMM appears to be attempting to create a temporary copy of the CA certificate file but fails to do so properly. The error references a path
/usr/local/percona/pmm/tmp/mongodb-backup-restore/[job-id]/caFilePlaceholder

Expected Results

PMM should successfully initiate the backup by either:

• Using the existing pbm-agent configuration
• Properly copying/referencing the CA certificate file specified during service addition
• A proper error should be shown in the dashboard

Actual Results

The backup fails immediately with the following error in pmm-agent logs:

  Jul 31 11:06:00 pmm-agent[470]: time="2025-07-31T11:06:00.450+00:00" level=warning msg="Job terminated with error: exit status 1
  pbm config error: {\"Error\":\"connect to mongodb: create mongo connection: invalid connection string option: open
  /usr/local/percona/pmm/tmp/mongodb-backup-restore/d997b535-c31d-4aef-b2dc-5521356a3253/caFilePlaceholder: no such file or directory\"}

  github.com/percona/pmm/agent/runner/jobs.pbmConfigure
      /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/pbm_helpers.go:421
  github.com/percona/pmm/agent/runner/jobs.(*MongoDBBackupJob).Run
      /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_backup_job.go:131

Version

• PMM Server Version: 3.3.0
• PMM Client Version: 3.3.0 (pmm-admin and pmm-agent)
• PBM Version: 2.10.0
• MongoDB Version: 8.0.8-3
• Operating System: debian 12
• Authentication Method: MONGODB-X509 with custom CA certificate

Steps to reproduce

1. Configure MongoDB with X.509 authentication using a custom CA certificate
2. Add MongoDB service to PMM with TLS settings:
   pmm-admin add mongodb --enable-all-collectors --host=localhost
   --cluster="demo" --replication-set=rs0 --tls
   --tls-certificate-key-file=/path/to/pmm-user.pem
   --tls-ca-file=/path/to/custom_eddsa_CA.cert
   --authentication-database="$external"
   --authentication-mechanism="MONGODB-X509"
3. Configure pbm-agent with proper MongoDB URI including TLS settings
4. Attempt to create a backup through PMM web interface

Relevant logs

Jul 31 11:06:00 pmm-agent[470]: time="2025-07-31T11:06:00.450+00:00" level=warning msg="Job terminated with error: exit status 1
  pbm config error: {\"Error\":\"connect to mongodb: create mongo connection: invalid connection string option: open
  /usr/local/percona/pmm/tmp/mongodb-backup-restore/d997b535-c31d-4aef-b2dc-5521356a3253/caFilePlaceholder: no such file or directory\"}

  github.com/percona/pmm/agent/runner/jobs.pbmConfigure
      /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/pbm_helpers.go:421
  github.com/percona/pmm/agent/runner/jobs.(*MongoDBBackupJob).Run
      /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_backup_job.go:131

Code of Conduct

• I agree to follow Percona Community Code of Conduct