CHK pods restart all at once when a modification to ClickhouseKeeperInstalattion is made

[Tchirana]

operator version 0.25.3
I have this configured in the clickhousekeeperinstalattion:
pdbManaged: "True" pdbMaxUnavailable: 1 replicas: 3
The pdb for chk is created ok (the same is for the chi) but when i modify something in the clickhousekeeperinstallation , like the image when doing an upgrade, all the chk pods get restarted in the same time not one by one as it happens for the CHI pods.
I've read across docs and cannot find something related to this.
both CHK and CHI pods come from individual sts-es as the operator creates them. Am I missing something ?

[alex-zaitsev]

@Tchirana, PDB is set for Kubernetes, e.g. when Kubernetes decides to reschedule pods or something like that.
Operator has different logic for upgrade. Could you you explain in more detail what you were doing?

[Tchirana]

Hello, just the normal stuff of replacing the image in the chk installation manifest. What is even more weird that we have several clickhouses in different clusters , all of the chi and chk installations use exactly the same manifests, same operator version with same settings, but in every cluster they behave differently. On image change for upgrade in some clusters the pods restart 1 at a time but in otherd 2 at a time or even all of them in the same time. We use 3 pods chk clusters and each pod comes from it's own sts (that is how the operator provisions them)

[Tchirana]

@alex-zaitsev renamed the issue since you are correct with the PDB. That does take effect because operator sets up 1 pod per sts.

[wilkermichael]

We are seeing a similar issue when updating PVCs and the node selector on the CHK

[Tchirana]

I really think it is a race condition because it makes no sense. On some clusters those chk pods behaved, on others they restarted chaotically, all at once, 2 at a time . And all our clusters are exactly the same related to k8s settings, operators configs, chk configs ...

[Tchirana]

because of this we usually stop all that puts data in clickhouse, stop chi pods and after that do maint on chk, otherwise there is a very big risk in databases getting in readonly state and that is a very annoying thing to fix