The patroni-version-check pod does not set appropriate values for restricted pod security policies

[sisve]

Report

The patroni-version-check pod does not set appropriate values for restricted pod security policies.

• allowPrivilegeEscalation != false (container "patroni-version-check" must set securityContext.allowPrivilegeEscalation=false)
• unrestricted capabilities (container "patroni-version-check" must set securityContext.capabilities.drop=["ALL"])

The following is from the stdout logs, slightly censored and very reformatted.

ERROR	Reconciler error	{
  "controller": "perconapgcluster",
  "controllerGroup": "pgv2.percona.com",
  "controllerKind": "PerconaPGCluster",
  "PerconaPGCluster": {
    "name": "postgres-cluster",
    "namespace": "..."
  },
  "namespace": "...",
  "name": "postgres-cluster",
  "reconcileID": "84550418-cd20-4dd8-ae5b-0430cd8d48e8",
  "error": "check patroni version: failed to create pod to check patroni version: pods \"postgres-cluster-patroni-version-check\" is forbidden: violates PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false (container \"patroni-version-check\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"patroni-version-check\" must set securityContext.capabilities.drop=[\"ALL\"])",
  "errorVerbose": "pods \"postgres-cluster-patroni-version-check\" is forbidden: violates PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false (container \"patroni-version-check\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"patroni-version-check\" must set securityContext.capabilities.drop=[\"ALL\"])\n<snip: stacktrace>"
}

Probable relevant code:

percona-postgresql-operator/percona/controller/pgcluster/controller.go

Lines 421 to 451 in 2b721be

	p = &corev1.Pod{
	ObjectMeta: meta,
	Spec: corev1.PodSpec{
	Containers: []corev1.Container{
	{
	Name: pNaming.ContainerPatroniVersionCheck,
	Image: cr.PostgresImage(),
	Command: []string{
	"bash",
	},
	Args: []string{
	"-c", "sleep 60",
	},
	Resources: cr.Spec.InstanceSets[0].Resources,
	},
	},
	SecurityContext: cr.Spec.InstanceSets[0].SecurityContext,
	TerminationGracePeriodSeconds: ptr.To(int64(5)),
	ImagePullSecrets: cr.Spec.ImagePullSecrets,
	Resources: &corev1.ResourceRequirements{
	Limits: corev1.ResourceList{
	corev1.ResourceCPU: resource.MustParse("100m"),
	corev1.ResourceMemory: resource.MustParse("64Mi"),
	},
	Requests: corev1.ResourceList{
	corev1.ResourceCPU: resource.MustParse("50m"),
	corev1.ResourceMemory: resource.MustParse("32Mi"),
	},
	},
	},
	}

More about the problem

This results in a lot of errors in the stdout logs of the operator. We were doing an operator upgrade from 2.5.0 to 2.6.0 (with the goal to upgrade to 2.7.0), and rollbacked without issues.

Steps to reproduce

Creating a PerconaPGCluster into a namespace with pod-security.kubernetes.io/enforce: privileged is probably enough. We're running into this during an operator upgrade from 2.5.0 to 2.6.0.

0. Make sure you have the PodSecurity admission plugin enabled in kubernetes.
1. Use operator 2.5.0
2. Create a namespace with the annotation pod-security.kubernetes.io/enforce: privileged
3. Deploy a PerconaPGCluster to the namespace. This requires some securityContext meddling to work.
4. Upgrade the CRDs: kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.7.0/deploy/crd.yaml
5. Upgrade the operator to 2.6.0: helm upgrade postgres-operator percona/pg-operator --version 2.6.0 --namespace postgres-operator --values values.yaml
6. The newly started operator pods will output lots of errors about the patroni-version-check pod for every pgcluster (since we run with watchAllNamespaces: true)

Our entire values.yaml:

watchAllNamespaces: true

Example cluster with the securityContext meddling:

apiVersion: pgv2.percona.com/v2
kind: PerconaPGCluster
metadata:
  name: postgres-cluster
spec:
  crVersion: 2.5.0

  users:
    - name: postgres

  image: percona/percona-postgresql-operator:2.5.0-ppg16.4-postgres
  imagePullPolicy: Always
  postgresVersion: 16

  instances:
    - name: master
      securityContext:
        runAsNonRoot: true
        fsGroup: 26
        seccompProfile:
          type: RuntimeDefault
      replicas: 2
      resources:
        limits:
          memory: 1Gi
      dataVolumeClaimSpec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

  proxy:
    pgBouncer:
      replicas: 1
      image: percona/percona-postgresql-operator:2.5.0-ppg16.4-pgbouncer1.23.1
      securityContext:
        runAsNonRoot: true
        fsGroup: 26
        seccompProfile:
          type: RuntimeDefault

  backups:
    pgbackrest:
      image: percona/percona-postgresql-operator:2.5.0-ppg16.4-pgbackrest2.53-1
      jobs:
        securityContext:
          runAsNonRoot: true
          fsGroup: 26
          seccompProfile:
            type: RuntimeDefault
      global:
        repo1-retention-full: "4"
        repo1-retention-full-type: count
      manual:
        repoName: repo1
        options:
          - --type=full
      repos:
        - name: repo1
          schedules:
            full: "0 0 * * 6"
            differential: "0 1 * * *"
          volume:
            volumeClaimSpec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 10Gi

Versions

1. Kubernetes v1.33.4
2. Operator v2.6.0
3. Database v16.4

Anything else?

No response

[sisve]

For those that end up here searching for a solution; there is a documented "workaround" at Customizing Patroni version. This will effectively tell the operator which version to use, avoiding the need to start the patroni-version-check pod.

Setting the annotation pgv2.percona.com/custom-patroni-version: "3" on the PerconaPGCluster worked for me, since I'm upgrading an operator from 2.5, which used Patroni 3, and want existing resources to keep running as-is.

The docs mention pgv2.percona.com/custom-patroni-version: "4", which seems like a better alternative if you create new clusters.

[nmarukovich]

@sisve thank you for your report. The fix will be included in the upcoming PG Operator release (v2.8.0)