Patroni is repeatedly patching the ha cluster ip service

[an-toine]

Report

Hello,

While investigating on an unrelated MetalLB issue, I've stumbled upon these logs :

{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"everest-managed-databases/XXX-ha","ts":"2025-07-03T10:07:39Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"everest-managed-databases/XXX-ha","level":"info","ts":"2025-07-03T10:07:39Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"everest-managed-databases/YYY-ha","ts":"2025-07-03T10:07:40Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"everest-managed-databases/YYY-ha","level":"info","ts":"2025-07-03T10:07:40Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"everest-managed-databases/ZZZ-ha","ts":"2025-07-03T10:07:42Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"everest-managed-databases/ZZZ-ha","level":"info","ts":"2025-07-03T10:07:42Z"}

Investigating further, I've found in K8S audit logs that Patroni was constantly patching the HA ClusterIP service every 10 seconds :

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f7e08f90-25f9-4d5b-acd2-5f03f29161a7","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/everest-managed-databases/endpoints/ZZZ-ha","verb":"patch","user":{"username":"system:serviceaccount:everest-managed-databases:ZZZ-instance","uid":"a96b430e-e1cf-4dcb-9571-6576866e88d8","groups":["system:serviceaccounts","system:serviceaccounts:everest-managed-databases","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=0bc30e9f-1a7b-45e3-b5cb-d344f0ddb2fc"],"authentication.kubernetes.io/node-name":["XXXX"],"authentication.kubernetes.io/node-uid":["5126e92f-f833-4ffc-9659-0f37ee864315"],"authentication.kubernetes.io/pod-name":["ZZZ-instance1-dql9-0"],"authentication.kubernetes.io/pod-uid":["c0f54e93-a40c-4761-9e65-a0552f6a9f67"]}},"sourceIPs":["XXXX"],"userAgent":"Patroni/4.0.5 Python/3.9.21 Linux","objectRef":{"resource":"endpoints","namespace":"everest-managed-databases","name":"ZZZ-ha","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-07-03T09:46:22.360065Z","stageTimestamp":"2025-07-03T09:46:22.363984Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ZZZ-instance/everest-managed-databases\" of Role \"ZZZ-instance\" to ServiceAccount \"ZZZ-instance/everest-managed-databases\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"6db138f2-4533-4b3e-be27-b749e1775b53","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/everest-managed-databases/endpoints/ZZZ-ha","verb":"patch","user":{"username":"system:serviceaccount:everest-managed-databases:ZZZ-instance","uid":"a96b430e-e1cf-4dcb-9571-6576866e88d8","groups":["system:serviceaccounts","system:serviceaccounts:everest-managed-databases","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=0bc30e9f-1a7b-45e3-b5cb-d344f0ddb2fc"],"authentication.kubernetes.io/node-name":["XXXX"],"authentication.kubernetes.io/node-uid":["5126e92f-f833-4ffc-9659-0f37ee864315"],"authentication.kubernetes.io/pod-name":["ZZZ-instance1-dql9-0"],"authentication.kubernetes.io/pod-uid":["c0f54e93-a40c-4761-9e65-a0552f6a9f67"]}},"sourceIPs":["XXXX"],"userAgent":"Patroni/4.0.5 Python/3.9.21 Linux","objectRef":{"resource":"endpoints","namespace":"everest-managed-databases","name":"ZZZ-ha","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2025-07-03T09:46:32.363414Z","stageTimestamp":"2025-07-03T09:46:32.367206Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ZZZ-instance/everest-managed-databases\" of Role \"ZZZ-instance\" to ServiceAccount \"ZZZ-instance/everest-managed-databases\""}}

Is this a normal behavior or is there a bug somewhere in my configuration ?

Antoine

More about the problem

The HA ClusterIP service is constantly patched by Patroni.

This is not a big deal but it generates some noise in audit logs and possibly other components monitoring services such as MetalLB or CNI.

Steps to reproduce

1. Deploy a postgresclusters.postgres-operator.crunchydata.com object in your cluster
2. Enable Kubernetes audit logs
3. Observe from audit logs that XXX-ha cluster ip is constantly patched by patroni.

Versions

1. Kubernetes : 1.31.8
2. Operator : PostgreSQL operator 2.6.0 / Everest 1.7.0
3. Database : PostgreSQL 17.4

Anything else?

No response