garbd fails to connect to PXC cluster during backup job

[lev-stas]

Report

Backup job fails with the next error

2025-06-27 13:22:33.566 ERROR: failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout)
        at ../../../../percona-xtradb-cluster-galera/gcomm/src/pc.cpp:connect():176
2025-06-27 13:22:33.566 ERROR: ../../../../percona-xtradb-cluster-galera/gcs/src/gcs_core.cpp:gcs_core_open():256: Failed to open backend connection: -110 (Connection timed out)
2025-06-27 13:22:34.566  INFO: gcomm: terminating thread
2025-06-27 13:22:34.566  INFO: gcomm: joining thread
2025-06-27 13:22:34.566 ERROR: ../../../../percona-xtradb-cluster-galera/gcs/src/gcs.cpp:gcs_open():1952: Failed to open channel 'mysql-cluster-pxc' at 'gcomm://mysql-cluster-pxc-4.mysql-cluster-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567': -110 (Connection timed out)
2025-06-27 13:22:34.566  INFO: Shifting CLOSED -> DESTROYED (TO: 0)
2025-06-27 13:22:34.567 FATAL: Garbd exiting with error: Failed to open connection to group
        at ../../../percona-xtradb-cluster-galera/garb/garb_gcs.cpp:Gcs():35
+ grep 'Will never receive state. Need to abort' /tmp/garbd.log
+ grep 'Donor is no longer in the cluster, interrupting script' /tmp/garbd.log
+ grep 'failed: Invalid argument' /tmp/garbd.log
+ '[' -f /tmp/backup-is-completed ']'
+ log ERROR 'Backup was finished unsuccessful'
+ exit 1

while cluster is in healthy and ready state

  kubectl get pxc -n mysql-main    
NAME            ENDPOINT         STATUS   PXC   PROXYSQL   HAPROXY   AGE
mysql-cluster   192.168.24.206   ready    5                3         3d19h
stas@SkyNet temp % 

More about the problem

I have checked cluster state and it looks healty

MySQL [(none)]> SELECT 
    ->   VARIABLE_NAME, VARIABLE_VALUE 
    -> FROM 
    ->   performance_schema.global_status 
    -> WHERE 
    ->   VARIABLE_NAME IN (
    ->     'wsrep_cluster_status', 
    ->     'wsrep_local_state_comment', 
    ->     'wsrep_ready', 
    ->     'wsrep_connected', 
    ->     'wsrep_cluster_size'
    ->   );
+---------------------------+----------------+
| VARIABLE_NAME             | VARIABLE_VALUE |
+---------------------------+----------------+
| wsrep_cluster_size        | 5              |
| wsrep_cluster_status      | Primary        |
| wsrep_connected           | ON             |
| wsrep_local_state_comment | Synced         |
| wsrep_ready               | ON             |
+---------------------------+----------------+
5 rows in set (0.003 sec)

MySQL [(none)]>

To rule out any network issues, I run a debug pod in the same namespace.
And made the next steps:
pod-name can be resolved to ip

net-debug:~# nslookup mysql-cluster-pxc-4.mysql-cluster-pxc
;; Got recursion not available from 10.43.96.3
Server:         10.43.96.3
Address:        10.43.96.3#53

Name:   mysql-cluster-pxc-4.mysql-cluster-pxc.mysql-main.svc.cluster.local
Address: 10.42.23.197
;; Got recursion not available from 10.43.96.3

net-debug:~# 

pod exposes target port

net-debug:~# nc -zv mysql-cluster-pxc-4.mysql-cluster-pxc 4567
Connection to mysql-cluster-pxc-4.mysql-cluster-pxc (10.42.23.197) 4567 port [tcp/*] succeeded!
net-debug:~# 

No any issues on pxc node side

2025-06-27T13:39:48.312124Z 31341 [Note] [MY-000000] [Galera] after_statement: success(31341,exec,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312160Z 31341 [Note] [MY-000000] [Galera] after_statement: enter(31341,exec,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312188Z 31341 [Note] [MY-000000] [Galera] after_statement_enter
    server: 56ff5a7b-5331-11f0-b9f9-4ae7d6ef3935, client: 31341, state: exec, mode: local
    trx_id: -1, seqno: -1, flags: 0
    state: aborted, bfa_state: executing, error: success, status: 0
    is_sr: 0, frags: 0, frags size: 0, unit: 0, size: 0, counter: 0, log_pos: 0, sr_rb: 0
    own: 1 thread_id: 7f7f010ee640
2025-06-27T13:39:48.312213Z 31341 [Note] [MY-000000] [Galera] cleanup_enter
    server: 56ff5a7b-5331-11f0-b9f9-4ae7d6ef3935, client: 31341, state: exec, mode: local
    trx_id: -1, seqno: -1, flags: 0
    state: aborted, bfa_state: executing, error: success, status: 0
    is_sr: 0, frags: 0, frags size: 0, unit: 0, size: 0, counter: 0, log_pos: 0, sr_rb: 0
    own: 1 thread_id: 7f7f010ee640
2025-06-27T13:39:48.312239Z 31341 [Note] [MY-000000] [Galera] cleanup_leave
    server: 56ff5a7b-5331-11f0-b9f9-4ae7d6ef3935, client: 31341, state: exec, mode: local
    trx_id: -1, seqno: -1, flags: 0
    state: aborted, bfa_state: executing, error: success, status: 0
    is_sr: 0, frags: 0, frags size: 0, unit: 0, size: 0, counter: 0, log_pos: 0, sr_rb: 0
    own: 1 thread_id: 7f7f010ee640
2025-06-27T13:39:48.312265Z 31341 [Note] [MY-000000] [Galera] after_statement_leave
    server: 56ff5a7b-5331-11f0-b9f9-4ae7d6ef3935, client: 31341, state: exec, mode: local
    trx_id: -1, seqno: -1, flags: 0
    state: aborted, bfa_state: executing, error: success, status: 0
    is_sr: 0, frags: 0, frags size: 0, unit: 0, size: 0, counter: 0, log_pos: 0, sr_rb: 0
    own: 1 thread_id: 7f7f010ee640
2025-06-27T13:39:48.312284Z 31341 [Note] [MY-000000] [Galera] after_statement: success(31341,exec,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312306Z 31341 [Note] [MY-000000] [Galera] after_command_before_result: enter(31341,exec,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312324Z 31341 [Note] [MY-000000] [Galera] after_command_before_result: leave(31341,result,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312433Z 31341 [Note] [MY-000000] [Galera] after_command_after_result_enter(31341,result,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312461Z 31341 [Note] [MY-000000] [Galera] after_command_after_result: leave(31341,idle,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312920Z 31341 [Note] [MY-000000] [Galera] before_command: enter(31341,idle,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312945Z 31341 [Note] [MY-000000] [Galera] before_command: success(31341,exec,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312968Z 31341 [Note] [MY-000000] [Galera] after_command_before_result: enter(31341,exec,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.312987Z 31341 [Note] [MY-000000] [Galera] after_command_before_result: leave(31341,result,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.313011Z 31341 [Note] [MY-000000] [Galera] after_command_after_result_enter(31341,result,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.313033Z 31341 [Note] [MY-000000] [Galera] after_command_after_result: leave(31341,idle,local,success,0,toi: -1,nbo: -1)
2025-06-27T13:39:48.313052Z 31341 [Note] [MY-000000] [Galera] close: enter(31341,idle,local,success,0,toi: -1,nbo: -1)

Steps to reproduce

1. Deploy custom resource from release cr.yaml manifest

...
backup:
#    allowParallel: true
    image: percona/percona-xtradb-cluster-operator:1.17.0-pxc8.0-backup-pxb8.0.35
    backoffLimit: 3
#    activeDeadlineSeconds: 3600
#    startingDeadlineSeconds: 300
#    suspendedDeadlineSeconds: 1200
    serviceAccountName: percona-xtradb-cluster-operator
#    imagePullSecrets:
#      - name: private-registry-credentials
   
    storages:
      minio:
        type: s3
        verifyTLS: true
        s3:
          bucket: percona-operator
          region: us-east-1
          endpointUrl: https://minio.mydomain.net
          credentialsSecret: mysql-cluster-s3-credentials
        resources:
          requests:
            memory: 1G
            cpu: 600m
...

2. deploy backup.yaml manifest

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterBackup
metadata:
  namespace: mysql-main
  finalizers:
    - percona.com/delete-backup
  name: test-backup
spec:
  pxcCluster: mysql-cluster
  storageName: minio

Versions

1. Kubernetes - v1.24.17
2. Operator - 1.17.0
3. Database - 8.0.41-32.1

Anything else?

No response

[hors]

Hi @lev-stas I can't reproduce this issue :( Could you please provide the full CR, maybe you have some custom config for PXC.

[franklouwers]

I seem to have the same issue. Ever found what it was @lev-stas ?

[franklouwers]

@hors to confirm, we have the same issue as @lev-stas . Had it on 1.16.1, we have now upgraded to 1.17.0, but the same issue persists.

Here's the full output from the backup pod:

(in all outputs below, I've applied a `sed 's/realname/MYNAMEREPLACED/g')

+ xb-on-demand-backup-7gdkg › xtrabackup
xb-on-demand-backup-7gdkg xtrabackup + LIB_PATH=/usr/lib/pxc
xb-on-demand-backup-7gdkg xtrabackup + . /usr/lib/pxc/backup.sh
xb-on-demand-backup-7gdkg xtrabackup ++ set -o errexit
xb-on-demand-backup-7gdkg xtrabackup ++ LIB_PATH=/usr/lib/pxc
xb-on-demand-backup-7gdkg xtrabackup ++ . /usr/lib/pxc/aws.sh
xb-on-demand-backup-7gdkg xtrabackup +++ set -o errexit
xb-on-demand-backup-7gdkg xtrabackup +++ export AWS_SHARED_CREDENTIALS_FILE=/tmp/aws-credfile
xb-on-demand-backup-7gdkg xtrabackup +++ AWS_SHARED_CREDENTIALS_FILE=/tmp/aws-credfile
xb-on-demand-backup-7gdkg xtrabackup +++ export AWS_ENDPOINT_URL=https://s3.amazonaws.com
xb-on-demand-backup-7gdkg xtrabackup +++ AWS_ENDPOINT_URL=https://s3.amazonaws.com
xb-on-demand-backup-7gdkg xtrabackup +++ export AWS_REGION=us-west-2
xb-on-demand-backup-7gdkg xtrabackup +++ AWS_REGION=us-west-2
xb-on-demand-backup-7gdkg xtrabackup +++ '[' -n true ']'
xb-on-demand-backup-7gdkg xtrabackup +++ [[ true == \f\a\l\s\e ]]
xb-on-demand-backup-7gdkg xtrabackup ++ SST_INFO_NAME=sst_info
xb-on-demand-backup-7gdkg xtrabackup ++ XBCLOUD_ARGS='--curl-retriable-errors=7 '
xb-on-demand-backup-7gdkg xtrabackup ++ INSECURE_ARG=
xb-on-demand-backup-7gdkg xtrabackup ++ '[' -n true ']'
xb-on-demand-backup-7gdkg xtrabackup ++ [[ true == \f\a\l\s\e ]]
xb-on-demand-backup-7gdkg xtrabackup +++ date +%F-%H-%M
xb-on-demand-backup-7gdkg xtrabackup ++ S3_BUCKET_PATH=MYNAMEREPLACED-pxc-db-pxc-2025-07-18-11-08-xtrabackup.stream
xb-on-demand-backup-7gdkg xtrabackup +++ date +%F-%H-%M
xb-on-demand-backup-7gdkg xtrabackup ++ BACKUP_PATH=MYNAMEREPLACED-pxc-db-pxc-2025-07-18-11-08-xtrabackup.stream
xb-on-demand-backup-7gdkg xtrabackup + GARBD_OPTS=
xb-on-demand-backup-7gdkg xtrabackup + check_ssl
xb-on-demand-backup-7gdkg xtrabackup + CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
xb-on-demand-backup-7gdkg xtrabackup + '[' -f /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt ']'
xb-on-demand-backup-7gdkg xtrabackup + SSL_DIR=/etc/mysql/ssl
xb-on-demand-backup-7gdkg xtrabackup + '[' -f /etc/mysql/ssl/ca.crt ']'
xb-on-demand-backup-7gdkg xtrabackup + SSL_INTERNAL_DIR=/etc/mysql/ssl-internal
xb-on-demand-backup-7gdkg xtrabackup + '[' -f /etc/mysql/ssl-internal/ca.crt ']'
xb-on-demand-backup-7gdkg xtrabackup + KEY=/etc/mysql/ssl/tls.key
xb-on-demand-backup-7gdkg xtrabackup + CERT=/etc/mysql/ssl/tls.crt
xb-on-demand-backup-7gdkg xtrabackup + '[' -f /etc/mysql/ssl-internal/tls.key -a -f /etc/mysql/ssl-internal/tls.crt ']'
xb-on-demand-backup-7gdkg xtrabackup + '[' -f /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -a -f /etc/mysql/ssl/tls.key -a -f /etc/mysql/ssl/tls.crt ']'
xb-on-demand-backup-7gdkg xtrabackup + '[' -n '' ']'
xb-on-demand-backup-7gdkg xtrabackup + '[' -n '' ']'
xb-on-demand-backup-7gdkg xtrabackup + request_streaming
xb-on-demand-backup-7gdkg xtrabackup ++ hostname -i
xb-on-demand-backup-7gdkg xtrabackup ++ sed -E 's/.*\b([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\b.*/\1/'
xb-on-demand-backup-7gdkg xtrabackup + local LOCAL_IP=10.0.34.194
xb-on-demand-backup-7gdkg xtrabackup ++ get_backup_source
xb-on-demand-backup-7gdkg xtrabackup +++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=MYNAMEREPLACED-pxc-db-pxc
xb-on-demand-backup-7gdkg xtrabackup +++ grep wsrep_cluster_size
xb-on-demand-backup-7gdkg xtrabackup +++ sort
xb-on-demand-backup-7gdkg xtrabackup +++ cut -d : -f 12
xb-on-demand-backup-7gdkg xtrabackup +++ tail -1
xb-on-demand-backup-7gdkg xtrabackup ++ CLUSTER_SIZE=3
xb-on-demand-backup-7gdkg xtrabackup ++ '[' -z 3 ']'
xb-on-demand-backup-7gdkg xtrabackup +++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=MYNAMEREPLACED-pxc-db-pxc
xb-on-demand-backup-7gdkg xtrabackup +++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
xb-on-demand-backup-7gdkg xtrabackup +++ sort -r
xb-on-demand-backup-7gdkg xtrabackup +++ cut -d . -f 1
xb-on-demand-backup-7gdkg xtrabackup +++ cut -d : -f 2
xb-on-demand-backup-7gdkg xtrabackup +++ tail -1
xb-on-demand-backup-7gdkg xtrabackup ++ FIRST_NODE=MYNAMEREPLACED-pxc-db-pxc-0
xb-on-demand-backup-7gdkg xtrabackup ++ SKIP_FIRST_POD='|'
xb-on-demand-backup-7gdkg xtrabackup ++ ((  3 > 1  ))
xb-on-demand-backup-7gdkg xtrabackup ++ SKIP_FIRST_POD=MYNAMEREPLACED-pxc-db-pxc-0
xb-on-demand-backup-7gdkg xtrabackup ++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=MYNAMEREPLACED-pxc-db-pxc
xb-on-demand-backup-7gdkg xtrabackup ++ sort
xb-on-demand-backup-7gdkg xtrabackup ++ grep -v MYNAMEREPLACED-pxc-db-pxc-0
xb-on-demand-backup-7gdkg xtrabackup ++ tail -1
xb-on-demand-backup-7gdkg xtrabackup ++ cut -d : -f 2
xb-on-demand-backup-7gdkg xtrabackup ++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary
xb-on-demand-backup-7gdkg xtrabackup ++ cut -d . -f 1
xb-on-demand-backup-7gdkg xtrabackup + local NODE_NAME=MYNAMEREPLACED-pxc-db-pxc-2
xb-on-demand-backup-7gdkg xtrabackup + '[' -z MYNAMEREPLACED-pxc-db-pxc-2 ']'
xb-on-demand-backup-7gdkg xtrabackup + set +o errexit
xb-on-demand-backup-7gdkg xtrabackup + log INFO 'Garbd was started'
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37 [INFO] Garbd was started
xb-on-demand-backup-7gdkg xtrabackup + garbd --address 'gcomm://MYNAMEREPLACED-pxc-db-pxc-2.MYNAMEREPLACED-pxc-db-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567' --donor MYNAMEREPLACED-pxc-db-pxc-2 --group MYNAMEREPLACED-pxc-db-pxc --options '' --sst xtrabackup-v2:10.0.34.194:4444/xtrabackup_sst//1 --recv-script=/usr/bin/run_backup.sh
xb-on-demand-backup-7gdkg xtrabackup + tee /tmp/garbd.log
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.670  INFO: CRC-32C: using 64-bit x86 acceleration.
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.670  INFO: Read config:
xb-on-demand-backup-7gdkg xtrabackup    daemon:      0
xb-on-demand-backup-7gdkg xtrabackup    name:        garb
xb-on-demand-backup-7gdkg xtrabackup    address:     gcomm://MYNAMEREPLACED-pxc-db-pxc-2.MYNAMEREPLACED-pxc-db-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567
xb-on-demand-backup-7gdkg xtrabackup    group:       MYNAMEREPLACED-pxc-db-pxc
xb-on-demand-backup-7gdkg xtrabackup    sst:         xtrabackup-v2:10.0.34.194:4444/xtrabackup_sst//1
xb-on-demand-backup-7gdkg xtrabackup    donor:       MYNAMEREPLACED-pxc-db-pxc-2
xb-on-demand-backup-7gdkg xtrabackup    options:     gcs.fc_limit=9999999; gcs.fc_factor=1.0; gcs.fc_single_primary=yes
xb-on-demand-backup-7gdkg xtrabackup    cfg:
xb-on-demand-backup-7gdkg xtrabackup    log:
xb-on-demand-backup-7gdkg xtrabackup    recv_script: /usr/bin/run_backup.sh
xb-on-demand-backup-7gdkg xtrabackup    workdir:
xb-on-demand-backup-7gdkg xtrabackup
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.672  INFO: protonet asio version 0
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.672  INFO: Using CRC-32C for message checksums.
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.672  INFO: backend: asio
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.672  INFO: gcomm thread scheduling priority set to other:0
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.673  INFO: Fail to access the file (./gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.673  INFO: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.673  INFO: GMCast version 0
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.675  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.675  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.675  INFO: EVS version 1
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:37.676  INFO: gcomm: connecting to group 'MYNAMEREPLACED-pxc-db-pxc', peer 'MYNAMEREPLACED-pxc-db-pxc-2.MYNAMEREPLACED-pxc-db-pxc:'
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:38.676  INFO: EVS version upgrade 0 -> 1
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:38.676  INFO: PC protocol upgrade 0 -> 1
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:38.676  INFO: No nodes coming from primary view, primary view is not possible
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:38.676  INFO: Current view of cluster as seen by this node
xb-on-demand-backup-7gdkg xtrabackup view (view_id(NON_PRIM,8dc69dc0-a6b8,1)
xb-on-demand-backup-7gdkg xtrabackup memb {
xb-on-demand-backup-7gdkg xtrabackup    8dc69dc0-a6b8,0
xb-on-demand-backup-7gdkg xtrabackup    }
xb-on-demand-backup-7gdkg xtrabackup joined {
xb-on-demand-backup-7gdkg xtrabackup    }
xb-on-demand-backup-7gdkg xtrabackup left {
xb-on-demand-backup-7gdkg xtrabackup    }
xb-on-demand-backup-7gdkg xtrabackup partitioned {
xb-on-demand-backup-7gdkg xtrabackup    }
xb-on-demand-backup-7gdkg xtrabackup )
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:39.176  WARN: last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT1.50102S), skipping check
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:40.677  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1095 rttvar: 547 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999637694 last_delivered_since: 2999637694 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:40.677  INFO: announce period timed out (pc.announce_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:45.177  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1043 rttvar: 521 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999167868 last_delivered_since: 2999167868 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:49.678  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 759 rttvar: 379 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999286290 last_delivered_since: 2999286290 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:53.678  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 765 rttvar: 382 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999376692 last_delivered_since: 2999376692 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:08:57.679  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 897 rttvar: 448 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999118025 last_delivered_since: 2999118025 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:01.679  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 783 rttvar: 391 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999262031 last_delivered_since: 2999262031 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:05.680  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 875 rttvar: 437 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999464579 last_delivered_since: 2999464579 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:09.681  INFO: (8dc69dc0-a6b8, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.0.2.135:4567 timed out, no messages seen in PT3S, socket stats: rtt: 786 rttvar: 393 rto: 200000 lost: 0 last_data_recv: 3000 cwnd: 10 last_queued_since: 2999326575 last_delivered_since: 2999326575 send_queue_length: 0 send_queue_bytes: 0 (gmcast.peer_timeout)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:10.683  INFO: PC protocol downgrade 1 -> 0
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:10.683  INFO: Current view of cluster as seen by this node
xb-on-demand-backup-7gdkg xtrabackup view ((empty))
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:10.683 ERROR: failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout)
xb-on-demand-backup-7gdkg xtrabackup     at ../../../../percona-xtradb-cluster-galera/gcomm/src/pc.cpp:connect():176
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:10.683 ERROR: ../../../../percona-xtradb-cluster-galera/gcs/src/gcs_core.cpp:gcs_core_open():256: Failed to open backend connection: -110 (Connection timed out)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:11.684  INFO: gcomm: terminating thread
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:11.684  INFO: gcomm: joining thread
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:11.684 ERROR: ../../../../percona-xtradb-cluster-galera/gcs/src/gcs.cpp:gcs_open():1952: Failed to open channel 'MYNAMEREPLACED-pxc-db-pxc' at 'gcomm://MYNAMEREPLACED-pxc-db-pxc-2.MYNAMEREPLACED-pxc-db-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567': -110 (Connection timed out)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:11.684  INFO: Shifting CLOSED -> DESTROYED (TO: 0)
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:11.684 FATAL: Garbd exiting with error: Failed to open connection to group
xb-on-demand-backup-7gdkg xtrabackup     at ../../../percona-xtradb-cluster-galera/garb/garb_gcs.cpp:Gcs():35
xb-on-demand-backup-7gdkg xtrabackup + grep 'Will never receive state. Need to abort' /tmp/garbd.log
xb-on-demand-backup-7gdkg xtrabackup + grep 'Donor is no longer in the cluster, interrupting script' /tmp/garbd.log
xb-on-demand-backup-7gdkg xtrabackup + grep 'failed: Invalid argument' /tmp/garbd.log
xb-on-demand-backup-7gdkg xtrabackup + '[' -f /tmp/backup-is-completed ']'
xb-on-demand-backup-7gdkg xtrabackup + log ERROR 'Backup was finished unsuccessful'
xb-on-demand-backup-7gdkg xtrabackup 2025-07-18 11:09:11 [ERROR] Backup was finished unsuccessful
xb-on-demand-backup-7gdkg xtrabackup + exit 1

Backup job yaml:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterBackup
metadata:
#  finalizers:
#    - percona.com/delete-backup
  name: on-demand-backup
  namespace: percona
spec:
  pxcCluster: MYNAMEREPLACED-pxc-db
  storageName: aws-bucket-backups

Database definition yaml:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  annotations:
    meta.helm.sh/release-name: MYNAMEREPLACED
    meta.helm.sh/release-namespace: percona
  creationTimestamp: "2024-03-30T12:00:56Z"
  finalizers:
  - percona.com/delete-pxc-pods-in-order
  generation: 8
  labels:
    app.kubernetes.io/instance: MYNAMEREPLACED
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: pxc-db
    app.kubernetes.io/version: 1.17.0
    helm.sh/chart: pxc-db-1.17.0
  name: MYNAMEREPLACED-pxc-db
  namespace: percona
  resourceVersion: "282269003"
  uid: d19e1e93-6caa-4ad1-b920-07c4c34266e9
spec:
  backup:
    image: percona/percona-xtradb-cluster-operator:1.17.0-pxc8.0-backup-pxb8.0.35
    pitr:
      enabled: false
    schedule: []
    storages:
      aws-bucket-backups:
        s3:
          bucket: MYNAMEREPLACED-dev-percona-backups/backups
          credentialsSecret: percona-backups-credentials
          endpointUrl: https://s3.eu-central-1.amazonaws.com
          region: eu-central-1
  crVersion: 1.17.0
  enableCRValidationWebhook: false
  enableVolumeExpansion: false
  haproxy:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    annotations: {}
    enabled: true
    envVarsSecret: haproxy-env-vars
    gracePeriod: 30
    image: percona/haproxy:2.8.14
    labels: {}
    livenessDelaySec: 300
    livenessProbes:
      failureThreshold: 4
      initialDelaySeconds: 60
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 5
    nodeSelector: {}
    podDisruptionBudget:
      maxUnavailable: 1
    readinessDelaySec: 15
    readinessProbes:
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits: {}
      requests:
        cpu: 100m
        memory: 100Mi
    sidecarPVCs: []
    sidecarResources:
      limits: {}
      requests: {}
    sidecarVolumes: []
    sidecars: []
    size: 3
    tolerations: []
    volumeSpec:
      emptyDir: {}
  logCollectorSecretName: MYNAMEREPLACED-pxc-db-log-collector
  logcollector:
    enabled: true
    image: percona/percona-xtradb-cluster-operator:1.17.0-logcollector-fluentbit4.0.0
    resources:
      limits: {}
      requests:
        cpu: 200m
        memory: 100M
  pause: false
  pmm:
    enabled: false
    image: percona/pmm-client:2.44.0
  proxysql:
    enabled: false
    image: percona/proxysql2:2.7.1
  pxc:
    affinity:
      antiAffinityTopologyKey: topology.kubernetes.io/zone
    annotations: {}
    autoRecovery: true
    gracePeriod: 600
    image: percona/percona-xtradb-cluster:8.0.41-32.1
    labels: {}
    livenessDelaySec: 300
    livenessProbes:
      failureThreshold: 3
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    nodeSelector: {}
    podDisruptionBudget:
      maxUnavailable: 1
    readinessDelaySec: 15
    readinessProbes:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 15
    resources:
      limits: {}
      requests:
        cpu: 300m
        memory: 1G
    sidecarPVCs: []
    sidecarResources:
      limits: {}
      requests: {}
    sidecarVolumes: []
    sidecars: []
    size: 3
    tolerations: []
    volumeSpec:
      persistentVolumeClaim:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 8Gi
  secretsName: MYNAMEREPLACED-pxc-db-secrets
  sslInternalSecretName: MYNAMEREPLACED-pxc-db-ssl-internal
  sslSecretName: MYNAMEREPLACED-pxc-db-ssl
  tls:
    enabled: true
  unsafeFlags:
    backupIfUnhealthy: true
  updateStrategy: SmartUpdate
  upgradeOptions:
    apply: disabled
    schedule: 0 4 * * *
    versionServiceEndpoint: https://check.percona.com
  vaultSecretName: MYNAMEREPLACED-pxc-db-vault
status:
  backup: {}
  conditions:
  - lastTransitionTime: "2025-07-16T10:24:45Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-16T10:28:10Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-16T10:28:10Z"
    status: enabled
    type: tls
  - lastTransitionTime: "2025-07-16T10:28:11Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-16T12:05:09Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-16T12:07:05Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-16T12:46:52Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-16T12:48:53Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-17T08:21:17Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-17T08:23:07Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-17T08:24:17Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-17T08:25:38Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-17T12:12:33Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-17T12:14:24Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-17T12:15:29Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-17T12:16:56Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-17T12:46:26Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-17T12:48:21Z"
    status: "True"
    type: ready
  - lastTransitionTime: "2025-07-18T11:01:13Z"
    status: "True"
    type: initializing
  - lastTransitionTime: "2025-07-18T11:05:15Z"
    status: "True"
    type: ready
  haproxy:
    labelSelectorPath: app.kubernetes.io/component=haproxy,app.kubernetes.io/instance=MYNAMEREPLACED-pxc-db,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster
    ready: 3
    size: 3
    status: ready
  host: MYNAMEREPLACED-pxc-db-haproxy.percona
  logcollector: {}
  observedGeneration: 8
  pmm: {}
  proxysql: {}
  pxc:
    image: percona/percona-xtradb-cluster:8.0.41-32.1
    labelSelectorPath: app.kubernetes.io/component=pxc,app.kubernetes.io/instance=MYNAMEREPLACED-pxc-db,app.kubernetes.io/managed-by=percona-xtradb-cluster-operator,app.kubernetes.io/name=percona-xtradb-cluster,app.kubernetes.io/part-of=percona-xtradb-cluster
    ready: 3
    size: 3
    status: ready
    version: 8.0.41-32.1
  ready: 6
  size: 6
  state: ready

Please let me know if you need more info, I can reproduce this on my cluster 100% of the time.

[hors]

@franklouwers I need to know the K8s platform that you are using.

Please let me know if you need more info, I can reproduce this on my cluster 100% of the time.

Does this mean that backups aren’t working at all for your cluster?

[franklouwers]

@hors We're on AWS EKS, Server Version: v1.31.9-eks-5d4a308

Yes correct, backups not working at the moment. Note that this is a dev cluster which has been running for some time and we only now want to enable backups.

[hors]

@franklouwers We tried to reproduce this issue on EKS but without any results :( Do you use Vaul and, as a result, encryption?

[franklouwers]

No. We don’t use vault. We do enable tls (see config), managed by certmanager

We have another cluster on EKS (slightly different params) that doesn’t show this.

Any additional debugging or test scripts I can run?

[eleo007]

Hi,

We tried once again to reproduce but without luck. Are the differences between clusters related to network config?

[franklouwers]

I’ll do a deeper dive later this week, but is there any way to get more debug output, or any way to get more useful info? (Debug scripts, manual things to try, …)

[franklouwers]

What we've been able to debug so far is that the issue is SSL related. On clusters where the backup works, we see this as garb command:

garbd --address 'gcomm://DBNAMEHERE-pxc-db-pxc-2.DBNAMEHERE-pxc-db-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567' --donor DBNAMEHERE-pxc-db-pxc-2 --group DBNAMEHERE-pxc-db-pxc --options 'socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight=0;' --sst xtrabackup-v2:10.0.41.42:4444/xtrabackup_sst//1 --recv-script=/usr/bin/run_backup.sh

On the cluster where it fails:

garbd --address 'gcomm://OTHERDBNAMEHERE-pxc-db-pxc-2.OTHERDBNAMEHERE-pxc-db-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567' --donor OTHERDBNAMEHERE-pxc-db-pxc-2 --group OTHERDBNAMEHERE-pxc-db-pxc --options '' --sst xtrabackup-v2:10.0.41.239:4444/xtrabackup_sst//1 --recv-script=/usr/bin/run_backup.sh

Note the missing ssl options. The secret containing the ssl certs isn't mounted in the pod either.

Is looks like https://github.com/percona/percona-xtradb-cluster-operator/blob/v1.17.0/pkg/pxc/backup/job.go#L146 isn't getting called (or isn't returning what it should return).

After much spelunking, we can link this to a misconfiguration which we (or anybody else on this thread btw) missed so far: in #2105 (comment) in our database spec, we missed backup.storages.aws-bucket-backups.type: s3 in the backup definition.

Why and how missing the type: s3 influences the loading of ssl secrets for the connection to the percona cluster (which has nothing to do with s3), beats me, but that seems to fix it.

[franklouwers]

(After this, I believe the name of this issue should be change to something like "backups fail when backup storage type is missing" and another one for "operator incorrectly accepts backup config to s3 without explicit type: s3")

[wonko]

Quickly checked the code (coworker of @franklouwers btw).

I guess you want to add a "default" to https://github.com/percona/percona-xtradb-cluster-operator/blob/main/pkg/controller/pxcbackup/controller.go#L298 to complain/error out if the switch doesn't match any case. Currently it just skips the whole block, which any of them would call the SSL-addition in the pod in the three sub-jobs (feels a bit wrong as well, tbh, should this not be lifted to the higher level, it's not really dependant on the storage type).

[eleo007]

Hi @franklouwers @wonko !

Thanks for coming back and sharing your findings! I have created a Jira task to improve behavior in future releases: https://perconadev.atlassian.net/browse/K8SPXC-1671 .

@lev-stas as i see you had type: s3 in your storages config. If you still face the issue, please provide earlier requested info: the full CR, maybe you have some custom config for PXC.

[lev-stas]

@eleo007 , @hors Sorry for the delay with the answer.
Here is the full text of our cr.yml file. I believe we made a few minor changes, just to make it compatible with our infrastructure.

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  namespace: mysql-main
  name: mysql-cluster
  finalizers:
    - percona.com/delete-pxc-pods-in-order
#    - percona.com/delete-ssl
    - percona.com/delete-proxysql-pvc
    - percona.com/delete-pxc-pvc
#  annotations:
#    percona.com/issue-vault-token: "true"
spec:
  crVersion: 1.17.0
#  enableVolumeExpansion: false
#  ignoreAnnotations:
#    - iam.amazonaws.com/role
#  ignoreLabels:
#    - rack
  secretsName: mysql-cluster-secrets
#  vaultSecretName: keyring-secret-vault
#  sslSecretName: cluster1-ssl
#  sslInternalSecretName: cluster1-ssl-internal
#  logCollectorSecretName: cluster1-log-collector-secrets
#  initContainer:
#    image: percona/percona-xtradb-cluster-operator:1.17.0
#    containerSecurityContext:
#      privileged: false
#      runAsUser: 1001
#      runAsGroup: 1001
#    resources:
#      requests:
#        memory: 100M
#        cpu: 100m
#      limits:
#        memory: 200M
#        cpu: 200m
  enableCRValidationWebhook: true
  tls:
    enabled: false
#    SANs:
#      - pxc-1.example.com
#      - pxc-2.example.com
#      - pxc-3.example.com
#    issuerConf:
#      name: special-selfsigned-issuer
#      kind: ClusterIssuer
#      group: cert-manager.io
  unsafeFlags:
    tls: true
#    pxcSize: false
#    proxySize: false
#    backupIfUnhealthy: false
  pause: false
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: "0 4 * * *"
  pxc:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9104"
    size: 5
    image: percona/percona-xtradb-cluster:8.0.41-32.1
    autoRecovery: true
#    expose:
#      enabled: true
#      type: LoadBalancer
#      externalTrafficPolicy: Local
#      internalTrafficPolicy: Local
#      loadBalancerSourceRanges:
#        - 10.0.0.0/8
#      loadBalancerIP: 127.0.0.1
#      annotations:
#        networking.gke.io/load-balancer-type: "Internal"
#      labels:
#        rack: rack-22
#    replicationChannels:
##    - name: pxc1_to_pxc2
##      isSource: true
#    - name: maindb_to_pxc
#      isSource: false
#      configuration:
#        sourceRetryCount: 5
#        sourceConnectRetry: 60
#        ssl: false
#      sourcesList:
#      - host: maindbmaster.sgdctroy.net
#        port: 3306
#        weight: 100
#    schedulerName: mycustom-scheduler
#    readinessDelaySec: 100
#    livenessDelaySec: 600
    configuration: |
      [mysqld]
      wsrep_debug=CLIENT
      wsrep_provider_options="gcache.size=16G; gcache.recover=yes"
      wsrep_log_conflicts=ON
      wsrep_cluster_name="mysql-cluster"
      binlog_format=ROW
      default_storage_engine=InnoDB
      innodb_autoinc_lock_mode=2
      wsrep_slave_threads=8
      default_authentication_plugin=caching_sha2_password
      max_connect_errors= 4294967295
      open-files-limit=32000
      max_allowed_packet=1024M
      table_open_cache=8192
      table_definition_cache=2560
      max_binlog_size=1024M
      max_connections=2500
      back_log=2048
      net_read_timeout=30
      net_write_timeout=60
      innodb_buffer_pool_size=16G
      innodb_buffer_pool_instances=8
      innodb_flush_method=O_DIRECT
      innodb_file_per_table=1
      innodb_log_buffer_size=4M
      innodb_flush_log_at_trx_commit=2
      innodb_io_capacity=3000
      innodb_io_capacity_max=5000
      innodb_read_io_threads=8
      innodb_write_io_threads=8
      innodb_purge_threads=8
      long_query_time=0
      slow_query_log=OFF
      connect_timeout=10
      [sst]
      streamfmt=xbstream
      xbstream-opts=--decompress
      [xtrabackup]
      compress=lz4
#      for PXC 5.7
#      [xtrabackup]
#      compress
#    imagePullSecrets:
#      - name: private-registry-credentials
#    priorityClassName: high-priority
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    labels:
#      rack: rack-22
    readinessProbes:
      initialDelaySeconds: 120
      timeoutSeconds: 15
      periodSeconds: 15
      successThreshold: 1
      failureThreshold: 5
    livenessProbes:
      initialDelaySeconds: 180
      timeoutSeconds: 15
      periodSeconds: 15
      successThreshold: 1
      failureThreshold: 5
#    containerSecurityContext:
#      privileged: false
#    podSecurityContext:
#      runAsUser: 1001
#      runAsGroup: 1001
#      supplementalGroups: [1001]
#    serviceAccountName: percona-xtradb-cluster-operator-workload
    imagePullPolicy: IfNotPresent
#    runtimeClassName: image-rc
    sidecars:
    - image: prom/mysqld-exporter:v0.17.2
      name: mysql-exporter
      ports:
      - containerPort: 9104
        name: metrics
      args:
        - "--config.my-cnf=/config/.my.cnf"
        - "--collect.engine_innodb_status"
        - "--collect.global_status"
        - "--collect.global_variables"
        - "--collect.info_schema.processlist"
        - "--collect.info_schema.innodb_metrics"
        - "--collect.info_schema.innodb_cmp"
        - "--collect.info_schema.innodb_cmpmem"
        - "--collect.info_schema.tables"
        - "--collect.info_schema.tablestats"
        - "--collect.info_schema.schemastats"
        - "--collect.info_schema.userstats"
        - "--collect.perf_schema.eventsstatements"
        - "--collect.perf_schema.eventswaits"
        - "--collect.perf_schema.indexiowaits"
        - "--collect.perf_schema.tableiowaits"
        - "--collect.perf_schema.tablelocks"
      volumeMounts:
      - mountPath: /config
        name: mysql-exporter-config
      resources:
        requests:
          memory: 100M
          cpu: 100m
        limits:
          memory: 200M
          cpu: 200m
    sidecarVolumes:
    - name: mysql-exporter-config
      configMap:
        name: mysql-exporter-config
#    envVarsSecret: my-env-var-secrets
    resources:
      requests:
        memory: 20G
        cpu: "1"
#        ephemeral-storage: 1G
      limits:
        memory: 40G
        cpu: "4"
#        ephemeral-storage: 1G
#    nodeSelector:
#      disktype: ssd
#    topologySpreadConstraints:
#    - labelSelector:
#        matchLabels:
#          app.kubernetes.io/name: percona-xtradb-cluster
#      maxSkew: 1
#      topologyKey: kubernetes.io/hostname
#      whenUnsatisfiable: DoNotSchedule
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
#      advanced:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: kubernetes.io/e2e-az-name
#                operator: In
#                values:
#                - e2e-az1
#                - e2e-az2
#    tolerations:
#    - key: "node.alpha.kubernetes.io/unreachable"
#      operator: "Exists"
#      effect: "NoExecute"
#      tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
#      minAvailable: 0
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: rook-ceph-block
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 750G
    gracePeriod: 600
#    lifecycle:
#      preStop:
#        exec:
#          command: [ "/bin/true" ]
#      postStart:
#        exec:
#          command: [ "/bin/true" ]
  haproxy:
    enabled: true
    size: 3
    image: percona/haproxy:2.8.14
    imagePullPolicy: IfNotPresent
#    schedulerName: mycustom-scheduler
#    readinessDelaySec: 15
#    livenessDelaySec: 600
#    configuration: |
#
#      global
#        maxconn 2048
#        external-check
#        insecure-fork-wanted
#        stats socket /etc/haproxy/pxc/haproxy.sock mode 600 expose-fd listeners level admin
#
#      defaults
#        default-server init-addr last,libc,none
#        log global
#        mode tcp
#        retries 10
#        timeout client 28800s
#        timeout connect 100500
#        timeout server 28800s
#
#      resolvers kubernetes
#        parse-resolv-conf
#
#      frontend galera-in
#        bind *:3309 accept-proxy
#        bind *:3306
#        mode tcp
#        option clitcpka
#        default_backend galera-nodes
#
#      frontend galera-admin-in
#        bind *:33062
#        mode tcp
#        option clitcpka
#        default_backend galera-admin-nodes
#
#      frontend galera-replica-in
#        bind *:3307
#        mode tcp
#        option clitcpka
#        default_backend galera-replica-nodes
#
#      frontend galera-mysqlx-in
#        bind *:33060
#        mode tcp
#        option clitcpka
#        default_backend galera-mysqlx-nodes
#
#      frontend stats
#        bind *:8404
#        mode http
#        http-request use-service prometheus-exporter if { path /metrics }
#    imagePullSecrets:
#      - name: private-registry-credentials
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    labels:
#      rack: rack-22
#    readinessProbes:
#      initialDelaySeconds: 15
#      timeoutSeconds: 3
#      periodSeconds: 5
#      successThreshold: 1
#      failureThreshold: 5
#    livenessProbes:
#      initialDelaySeconds: 60
#      timeoutSeconds: 5
#      periodSeconds: 5
#      successThreshold: 1
#      failureThreshold: 5
    exposePrimary:
      type: LoadBalancer
#      annotations:
#        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
#      externalTrafficPolicy: Cluster
#      internalTrafficPolicy: Cluster
#      labels:
#        rack: rack-22
#      loadBalancerSourceRanges:
#        - 10.0.0.0/8
      loadBalancerIP: 192.168.24.206
#    exposeReplicas:
#      enabled: true
#      onlyReaders: false
#      type: ClusterIP
#      annotations:
#        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
#      externalTrafficPolicy: Cluster
#      internalTrafficPolicy: Cluster
#      labels:
#        rack: rack-22
#      loadBalancerSourceRanges:
#        - 10.0.0.0/8
#      loadBalancerIP: 127.0.0.1
#    runtimeClassName: image-rc
#    sidecars:
#    - image: busybox
#      command: ["/bin/sh"]
#      args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
#      name: my-sidecar-1
#      resources:
#        requests:
#          memory: 100M
#          cpu: 100m
#        limits:
#          memory: 200M
#          cpu: 200m
#    envVarsSecret: my-env-var-secrets
    resources:
      requests:
        memory: 1G
        cpu: 600m
      limits:
        memory: 3G
        cpu: 1
#    priorityClassName: high-priority
#    nodeSelector:
#      disktype: ssd
#    sidecarResources:
#      requests:
#        memory: 1G
#        cpu: 500m
#      limits:
#        memory: 2G
#        cpu: 600m
#    containerSecurityContext:
#      privileged: false
#    podSecurityContext:
#      runAsUser: 1001
#      runAsGroup: 1001
#      supplementalGroups: [1001]
#    serviceAccountName: percona-xtradb-cluster-operator-workload
#    topologySpreadConstraints:
#    - labelSelector:
#        matchLabels:
#          app.kubernetes.io/name: percona-xtradb-cluster
#      maxSkew: 1
#      topologyKey: kubernetes.io/hostname
#      whenUnsatisfiable: DoNotSchedule
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
#      advanced:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: kubernetes.io/e2e-az-name
#                operator: In
#                values:
#                - e2e-az1
#                - e2e-az2
#    tolerations:
#    - key: "node.alpha.kubernetes.io/unreachable"
#      operator: "Exists"
#      effect: "NoExecute"
#      tolerationSeconds: 6000
    podDisruptionBudget:
      maxUnavailable: 1
#      minAvailable: 0
    gracePeriod: 30
#    lifecycle:
#      preStop:
#        exec:
#          command: [ "/bin/true" ]
#      postStart:
#        exec:
#          command: [ "/bin/true" ]
  proxysql:
    enabled: false
    size: 3
    image: percona/proxysql2:2.7.1
    imagePullPolicy: IfNotPresent
    configuration: |
      datadir="/var/lib/proxysql"

      admin_variables =
      {
        admin_credentials="proxyadmin:admin_password"
        mysql_ifaces="0.0.0.0:6032"
        refresh_interval=2000

        cluster_username="proxyadmin"
        cluster_password="admin_password"
        checksum_admin_variables=false
        checksum_ldap_variables=false
        checksum_mysql_variables=false
        cluster_check_interval_ms=200
        cluster_check_status_frequency=100
        cluster_mysql_query_rules_save_to_disk=true
        cluster_mysql_servers_save_to_disk=true
        cluster_mysql_users_save_to_disk=true
        cluster_proxysql_servers_save_to_disk=true
        cluster_mysql_query_rules_diffs_before_sync=1
        cluster_mysql_servers_diffs_before_sync=1
        cluster_mysql_users_diffs_before_sync=1
        cluster_proxysql_servers_diffs_before_sync=1
      }

      mysql_variables=
      {
        monitor_password="monitor"
        monitor_galera_healthcheck_interval=1000
        threads=2
        max_connections=2048
        default_query_delay=0
        default_query_timeout=10000
        poll_timeout=2000
        interfaces="0.0.0.0:3306"
        default_schema="information_schema"
        stacksize=1048576
        connect_timeout_server=10000
        monitor_history=60000
        monitor_connect_interval=20000
        monitor_ping_interval=10000
        ping_timeout_server=200
        commands_stats=true
        sessions_sort=true
        have_ssl=true
        ssl_p2s_ca="/etc/proxysql/ssl-internal/ca.crt"
        ssl_p2s_cert="/etc/proxysql/ssl-internal/tls.crt"
        ssl_p2s_key="/etc/proxysql/ssl-internal/tls.key"
        ssl_p2s_cipher="ECDHE-RSA-AES128-GCM-SHA256"
      }
#    readinessDelaySec: 15
#    livenessDelaySec: 600
#    schedulerName: mycustom-scheduler
#    imagePullSecrets:
#      - name: private-registry-credentials
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    labels:
#      rack: rack-22
    expose:
      enabled: true
      type: LoadBalancer
#      annotations:
#        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
#      externalTrafficPolicy: Cluster
#      internalTrafficPolicy: Cluster
#      labels:
#        rack: rack-22
#      loadBalancerSourceRanges:
#        - 10.0.0.0/8
      loadBalancerIP: 192.168.24.206
#    runtimeClassName: image-rc
#    sidecars:
#    - image: busybox
#      command: ["/bin/sh"]
#      args: ["-c", "while true; do trap 'exit 0' SIGINT SIGTERM SIGQUIT SIGKILL; done;"]
#      name: my-sidecar-1
#      resources:
#        requests:
#          memory: 100M
#          cpu: 100m
#        limits:
#          memory: 200M
#          cpu: 200m
#    envVarsSecret: my-env-var-secrets
    resources:
      requests:
        memory: 1G
        cpu: 600m
      limits:
        memory: 3G
        cpu: 1
#    priorityClassName: high-priority
#    nodeSelector:
#      disktype: ssd
#    sidecarResources:
#      requests:
#        memory: 1G
#        cpu: 500m
#      limits:
#        memory: 2G
#        cpu: 600m
#    containerSecurityContext:
#      privileged: false
#    podSecurityContext:
#      runAsUser: 1001
#      runAsGroup: 1001
#      supplementalGroups: [1001]
#    serviceAccountName: percona-xtradb-cluster-operator-workload
#    topologySpreadConstraints:
#    - labelSelector:
#        matchLabels:
#          app.kubernetes.io/name: percona-xtradb-cluster-operator
#      maxSkew: 1
#      topologyKey: kubernetes.io/hostname
#      whenUnsatisfiable: DoNotSchedule
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
#      advanced:
#        nodeAffinity:
#          requiredDuringSchedulingIgnoredDuringExecution:
#            nodeSelectorTerms:
#            - matchExpressions:
#              - key: kubernetes.io/e2e-az-name
#                operator: In
#                values:
#                - e2e-az1
#                - e2e-az2
#    tolerations:
#    - key: "node.alpha.kubernetes.io/unreachable"
#      operator: "Exists"
#      effect: "NoExecute"
#      tolerationSeconds: 6000
    volumeSpec:
#      emptyDir: {}
#      hostPath:
#        path: /data
#        type: Directory
      persistentVolumeClaim:
#        storageClassName: standard
#        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 2G
    podDisruptionBudget:
      maxUnavailable: 1
#      minAvailable: 0
    gracePeriod: 30
#    lifecycle:
#      preStop:
#        exec:
#          command: [ "/bin/true" ]
#      postStart:
#        exec:
#          command: [ "/bin/true" ]
#   loadBalancerSourceRanges:
#     - 10.0.0.0/8
  logcollector:
    enabled: false
    image: percona/percona-xtradb-cluster-operator:1.17.0-logcollector-fluentbit4.0.0
#    configuration: |
#      [OUTPUT]
#           Name  es
#           Match *
#           Host  192.168.2.3
#           Port  9200
#           Index my_index
#           Type  my_type
    resources:
      requests:
        memory: 100M
        cpu: 200m

#  users:
#  - name: my-user
#    dbs:
#    - db1
#    - db2
#    hosts:
#    - localhost
#    grants:
#    - SELECT
#    - DELETE
#    - INSERT
#    withGrantOption: true
#    passwordSecretRef:
#      name: my-user-pwd
#      key: my-user-pwd-key
#  - name: my-user-two

  pmm:
    enabled: true
    image: percona/pmm-client:2.44.0
    serverHost: pmm.sgdctroy.net:443
    serverUser: "api_key"
    pxcParams: "--disable-tablestats-limit=2000"
    proxysqlParams: "--custom-labels=CUSTOM-LABELS"
    containerSecurityContext:
      privileged: false
    readinessProbes:
      initialDelaySeconds: 15
      timeoutSeconds: 15
      periodSeconds: 30
      successThreshold: 1
      failureThreshold: 5
    livenessProbes:
      initialDelaySeconds: 300
      timeoutSeconds: 5
      periodSeconds: 10
      successThreshold: 1
      failureThreshold: 3
    resources:
      requests:
        memory: 150M
        cpu: 300m
  backup:
#    allowParallel: true
    image: percona/percona-xtradb-cluster-operator:1.17.0-pxc8.0-backup-pxb8.0.35
    backoffLimit: 3
#    activeDeadlineSeconds: 3600
#    startingDeadlineSeconds: 300
#    suspendedDeadlineSeconds: 1200
    serviceAccountName: percona-xtradb-cluster-operator
#    imagePullSecrets:
#      - name: private-registry-credentials
    pitr:
      enabled: false
      storageName: STORAGE-NAME-HERE
      timeBetweenUploads: 60
      timeoutSeconds: 60
#      resources:
#        requests:
#          memory: 0.1G
#          cpu: 100m
#        limits:
#          memory: 1G
#          cpu: 700m
    storages:
      minio:
        type: s3
        verifyTLS: true
        s3:
          bucket: percona-operator
          region: us-east-1
          endpointUrl: https://minio.sgdctroy.net
          credentialsSecret: mysql-cluster-s3-credentials
        resources:
          requests:
            memory: 1G
            cpu: 600m
#      s3-us-west:
#        type: s3
#        verifyTLS: true
#        nodeSelector:
#          storage: tape
#          backupWorker: 'True'
#        resources:
#          requests:
#            memory: 1G
#            cpu: 600m
#        topologySpreadConstraints:
#        - labelSelector:
#            matchLabels:
#              app.kubernetes.io/name: percona-xtradb-cluster
#          maxSkew: 1
#          topologyKey: kubernetes.io/hostname
#          whenUnsatisfiable: DoNotSchedule
#        affinity:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: backupWorker
#                  operator: In
#                  values:
#                  - 'True'
#        tolerations:
#          - key: "backupWorker"
#            operator: "Equal"
#            value: "True"
#            effect: "NoSchedule"
#        annotations:
#          testName: scheduled-backup
#        labels:
#          backupWorker: 'True'
#        schedulerName: 'default-scheduler'
#        priorityClassName: 'high-priority'
#        containerSecurityContext:
#          privileged: true
#        podSecurityContext:
#          fsGroup: 1001
#          supplementalGroups: [1001, 1002, 1003]
#        containerOptions:
#          env:
#          - name: VERIFY_TLS
#            value: "false"
#          args:
#            xtrabackup:
#            - "--someflag=abc"
#            xbcloud:
#            - "--someflag=abc"
#            xbstream:
#            - "--someflag=abc"
#        s3:
#          bucket: S3-BACKUP-BUCKET-NAME-HERE
#          credentialsSecret: my-cluster-name-backup-s3
#          region: us-west-2
#      azure-blob:
#        type: azure
#        azure:
#          credentialsSecret: azure-secret
#          container: test
#          endpointUrl: https://accountName.blob.core.windows.net
#          storageClass: Hot
#          blockSize: 4194304
#          concurrency: 4
#      fs-pvc:
#        type: filesystem
#        nodeSelector:
#          storage: tape
#          backupWorker: 'True'
#        resources:
#          requests:
#            memory: 1G
#            cpu: 600m
#        topologySpreadConstraints:
#        - labelSelector:
#            matchLabels:
#              app.kubernetes.io/name: percona-xtradb-cluster
#          maxSkew: 1
#          topologyKey: kubernetes.io/hostname
#          whenUnsatisfiable: DoNotSchedule
#        affinity:
#          nodeAffinity:
#            requiredDuringSchedulingIgnoredDuringExecution:
#              nodeSelectorTerms:
#              - matchExpressions:
#                - key: backupWorker
#                  operator: In
#                  values:
#                  - 'True'
#        tolerations:
#          - key: "backupWorker"
#            operator: "Equal"
#            value: "True"
#            effect: "NoSchedule"
#        annotations:
#          testName: scheduled-backup
#        labels:
#          backupWorker: 'True'
#        schedulerName: 'default-scheduler'
#        priorityClassName: 'high-priority'
#        containerSecurityContext:
#          privileged: true
#        podSecurityContext:
#          fsGroup: 1001
#          supplementalGroups: [1001, 1002, 1003]
#        volume:
#          persistentVolumeClaim:
##            storageClassName: standard
#            accessModes: [ "ReadWriteOnce" ]
#            resources:
#              requests:
#                storage: 6G
#    schedule:
#      - name: "full-weekly-backup"
#        schedule: "0 2 * * 6"
#        keep: 5
#        storageName: minio
#      - name: "daily-backup"
#        schedule: "0 0 * * *"
#        keep: 5
#        storageName: fs-pvc

Regarding the question about destination storage type -- we intended to use our self hosted s3 storage based on minio

JFYI: backups with the Percona Operator based on Percona Server were working as expected.