[bug] StatefulSet restarts connected just to operator restart

[dbazhal]

Operator version: 0.25.3
Clickhouse clusters work fine, and have no changes/unapplied changes.
I restart clickhouse operator deployment, so the operator pod is re-created.
What I expect: clickhouse clusters work as they did, without any interuption.
What happens: all clickhouse clusters restart, with new statefulset version and all pods deleted and created again.

operator logs

I0820 18:49:33.117586 1 worker-service.go:98] switch service clickhouse-cluster-test-one/chi-test-one-0-0 to IP-less mode. ClusterIP=None I0820 18:49:33.257533 1 worker-service.go:182] updateService():CHI:clickhouse-cluster-test-one/test-one:Update Service success: clickhouse-cluster-test-one/chi-test-one-0-0 I0820 18:49:33.460375 1 worker-service.go:61] reconcileService():CHI:clickhouse-cluster-test-one/test-one:Service reconcile successful: clickhouse-cluster-test-one/chi-test-one-0-0 I0820 18:49:33.460399 1 worker-reconciler-chi.go:510] reconcileHostService():Host:0-0[0/0]:clickhouse-cluster-test-one/test-one:DONE Reconcile service of the host: 0-0 I0820 18:49:33.460426 1 worker-reconciler-chi.go:362] worker-reconciler-chi.go:362:reconcileHostStatefulSet():start:Host:0-0[0/0]:clickhouse-cluster-test-one/test-one:reconcile StatefulSet start I0820 18:49:33.460453 1 worker-reconciler-chi.go:365] reconcileHostStatefulSet():Host:0-0[0/0]:clickhouse-cluster-test-one/test-one:Reconcile host STS: 0-0. App version: 24.7.0[24.7/parsed from the tag: '24.7'] I0820 18:49:33.460782 1 worker.go:190] shouldForceRestartHost():Host:0-0[0/0]:clickhouse-cluster-test-one/test-one:Host force restart is not required. Host: 0-0 I0820 18:49:33.468689 1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: clickhouse-cluster-test-one/chi-test-one-test-one-0-0 I0820 18:49:33.468716 1 object-status.go:54] GetObjectStatusFromMetas():unknown:cur and new objects ARE DIFFERENT based on object version label: Update of the object is required. Object: clickhouse-cluster-test-one/chi-test-one-test-one-0-0. Cur: a2203f8f79f30dbfabe4d1a3ca3f986c52efa017 New: 14982f5f1011ac1a6b3f4993717d65f627f1b2fc I0820 18:49:33.468735 1 worker-reconciler-chi.go:376] reconcileHostStatefulSet():Host:0-0[0/0]:clickhouse-cluster-test-one/test-one:Reconcile host STS: 0-0. Reconcile StatefulSet I0820 18:49:33.474184 1 statefulset-reconciler.go:163] ReconcileStatefulSet():Host:0-0[0/0]:clickhouse-cluster-test-one/test-one:Need to reconcile MODIFIED StatefulSet: clickhouse-cluster-test-one/chi-test-one-test-one-0-0 I0820 18:49:33.475706 1 util.go:39] StatefulSet.Spec ARE DIFFERENT: added: none modified: Diff start ------------------------- modified .spec items num: 46 diff item [0]:'.Template.Spec.Volumes[5].Name' = '"serverkey"' diff item [1]:'.Template.Spec.Volumes[5].VolumeSource.Secret.Items[0].Path' = '"server.key"' diff item [2]:'.Template.Spec.Containers[0].Env[5].ValueFrom.FieldRef.APIVersion' = '""' diff item [3]:'.Template.Spec.Containers[0].VolumeMounts[5].MountPath' = '"/etc/clickhouse-server/secrets.d/server.crt/teleport-certificate"' diff item [4]:'.Template.Spec.Containers[0].VolumeMounts[6].Name' = '"serverkey"' diff item [5]:'.Template.Spec.Containers[0].VolumeMounts[6].MountPath' = '"/etc/clickhouse-server/secrets.d/server.key/teleport-certificate"' diff item [6]:'.Template.Spec.Containers[1].VolumeMounts[5].Name' = '"serverkey"' diff item [7]:'.Template.Spec.SchedulerName' = '""' diff item [8]:'.Template.Spec.Volumes[4].VolumeSource.Secret.Items[0].Path' = '"server.crt"' diff item [9]:'.Template.Spec.Containers[0].ReadinessProbe.TimeoutSeconds' = '0' diff item [10]:'.Template.Spec.Containers[0].ReadinessProbe.FailureThreshold' = '0' diff item [11]:'.Template.Spec.Containers[1].VolumeMounts[6].MountPath' = '"/etc/clickhouse-server/secrets.d/server.cas/teleport-certificate"' diff item [12]:'.Template.Spec.Volumes[6].VolumeSource.Secret.Items[0].Key' = '"server.cas"' diff item [13]:'.Template.Spec.Containers[0].Ports[3].Protocol' = '""' diff item [14]:'.Template.Spec.Containers[0].ReadinessProbe.ProbeHandler.HTTPGet.Scheme' = '""' diff item [15]:'.Template.Spec.Containers[1].VolumeMounts[6].Name' = '"servercas"' diff item [16]:'.Template.Spec.Volumes[5].VolumeSource.Secret.Items[0].Key' = '"server.key"' diff item [17]:'.Template.Spec.Volumes[6].VolumeSource.Secret.Items[0].Path' = '"server.cas"' diff item [18]:'.Template.Spec.Containers[0].Ports[4].Protocol' = '""' diff item [19]:'.Template.Spec.Containers[0].Ports[5].Protocol' = '""' diff item [20]:'.Template.Spec.Containers[0].TerminationMessagePolicy' = '""' diff item [21]:'.Template.Spec.Containers[1].Env[19].ValueFrom.FieldRef.APIVersion' = '""' diff item [22]:'.Template.Spec.Containers[1].VolumeMounts[5].MountPath' = '"/etc/clickhouse-server/secrets.d/server.key/teleport-certificate"' diff item [23]:'.PersistentVolumeClaimRetentionPolicy' = 'nil' diff item [24]:'.Template.Spec.Containers[0].VolumeMounts[5].Name' = '"servercrt"' diff item [25]:'.Template.Spec.Containers[1].Ports[0].Protocol' = '""' diff item [26]:'.Template.Spec.Containers[1].VolumeMounts[4].Name' = '"servercrt"' diff item [27]:'.Template.Spec.Containers[1].TerminationMessagePath' = '""' diff item [28]:'.Template.Spec.Containers[1].TerminationMessagePolicy' = '""' diff item [29]:'.Template.Spec.Volumes[4].Name' = '"servercrt"' diff item [30]:'.Template.Spec.Volumes[4].VolumeSource.Secret.Items[0].Key' = '"server.crt"' diff item [31]:'.Template.Spec.Containers[0].Ports[1].Protocol' = '""' diff item [32]:'.Template.Spec.Containers[0].Ports[2].Protocol' = '""' diff item [33]:'.Template.Spec.Containers[0].ReadinessProbe.SuccessThreshold' = '0' diff item [34]:'.Template.Spec.Containers[0].TerminationMessagePath' = '""' diff item [35]:'.Template.Spec.Containers[1].VolumeMounts[4].MountPath' = '"/etc/clickhouse-server/secrets.d/server.crt/teleport-certificate"' diff item [36]:'.Template.Spec.RestartPolicy' = '""' diff item [37]:'.Template.Spec.Containers[0].VolumeMounts[7].MountPath' = '"/etc/clickhouse-server/secrets.d/server.cas/teleport-certificate"' diff item [38]:'.Template.Spec.Containers[0].ImagePullPolicy' = '""' diff item [39]:'.Template.Spec.Containers[1].Env[20].ValueFrom.FieldRef.APIVersion' = '""' diff item [40]:'.Template.Spec.DeprecatedServiceAccount' = '""' diff item [41]:'.Template.Spec.Volumes[6].Name' = '"servercas"' diff item [42]:'.Template.Spec.Containers[0].Ports[0].Protocol' = '""' diff item [43]:'.Template.Spec.Containers[0].Env[6].ValueFrom.FieldRef.APIVersion' = '""' diff item [44]:'.Template.Spec.Containers[0].VolumeMounts[7].Name' = '"servercas"' diff item [45]:'.Template.Spec.DNSPolicy' = '""' Diff end -------------------------

removed:
none
I0820 18:49:33.475782 1 util.go:50] StatefulSet.Labels ARE DIFFERENT:
added:
none
modified:
Diff start -------------------------
modified .labels items num: 1
diff item [0]:'["clickhouse.altinity.com/object-version"]' = '"14982f5f1011ac1a6b3f4993717d65f627f1b2fc"'
Diff end -------------------------

removed:
none
I0820 18:49:33.475816 1 util.go:59] Host:0-0[0/0]:clickhouse-cluster-test-one/test-one:StatefulSet.Annotations ARE EQUAL