After operator restart, StatefulSet dependent resource fails to update

[thomasdraebing]

Bug Report

What did you do?

A StatefulSet is reconciled as part of a managed reconciliation workflow. If the CustomResource is updated while the operator was not restarted since the initial creation, the StatefulSet gets updated as expected. If the operator is restarted, the following upgrades fail. Other dependent resources, e.g. configmaps, in the workflow update as expected.

What did you expect to see?

The StatefulSet is updated with the new configuration.

What did you see instead? Under which circumstances?

The STatefulSet is not updated and he following exception is thrown:

16:41:47.062 [ERROR] io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher:61 [PID:57294] - Error during event processing ExecutionScope{ resource id: ResourceID{name='gerrit', namespace='gerrit'}, version: 10121738} failed.
io.javaoperatorsdk.operator.AggregatedOperatorException: Exception during workflow.
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileResult.createFinalException(WorkflowReconcileResult.java:67) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileResult.throwAggregateExceptionIfErrorsPresent(WorkflowReconcileResult.java:61) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:123) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:83) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:197) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:82) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:135) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:115) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:86) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:59) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ControllerExecution.run(EventProcessor.java:395) ~[operator-framework-core-3.1.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://example.com/apis/apps/v1/namespaces/gerrit/statefulsets. Message: statefulsets.apps "gerrit" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=apps, kind=statefulsets, name=gerrit, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=statefulsets.apps "gerrit" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:615) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:558) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:308) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[kubernetes-client-5.12.3.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[kubernetes-client-5.12.3.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.create(KubernetesDependentResource.java:128) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.CRUDKubernetesDependentResource.create(CRUDKubernetesDependentResource.java:17) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.handleCreate(AbstractDependentResource.java:83) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.handleCreate(KubernetesDependentResource.java:108) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.handleCreate(KubernetesDependentResource.java:32) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:39) ~[operator-framework-core-3.1.1.jar:?]
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileExecutor$NodeReconcileExecutor.run(WorkflowReconcileExecutor.java:165) ~[operator-framework-core-3.1.1.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	... 3 more

Environment

Kubernetes cluster type:

Development cluster; Kubernetes 1.24.3. The cluster is managed by Gardener [1].

$ Mention java-operator-sdk version from pom.xml file

3.1.1

$ java -version

openjdk version "17.0.3" 2022-04-19 LTS
OpenJDK Runtime Environment SapMachine (build 17.0.3+6-LTS-sapmachine)
OpenJDK 64-Bit Server VM SapMachine (build 17.0.3+6-LTS-sapmachine, mixed mode

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:21:56Z", GoVersion:"go1.18.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:23:26Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Additional context

Some debugging showed that the operator tries to create the StatefulSet instead of applying/updating it. The reason is that the DefaultPrimaryToSecondaryIndex for the dependent StatefulSet resource is empty. the onAddOrUpdate()-method of the index is never called, when the operator starts. For the other dependent resources in the workflow onAddOrUpdate() is called.

[1] https://gardener.cloud/

[thomasdraebing]

The code with which I am observing the issue is available here: https://gerrit.googlesource.com/k8s-gerrit/+/7498656b60d4d6c274a34a2383b17402e6fa9a8a/operator/
The Reconciler in question is https://gerrit.googlesource.com/k8s-gerrit/+/7498656b60d4d6c274a34a2383b17402e6fa9a8a/operator/src/main/java/com/google/gerrit/k8s/operator/gerrit/GerritReconciler.java

[csviri]

Hi @thomasdraebing ,

the DefaultPrimaryToSecondaryIndex is ok to be empty for this case. see: https://javaoperatorsdk.io/docs/features#managing-relation-between-primary-and-secondary-resources

Could you check if the labels are ok for the resource and the informer (if those are set correctly) ? it's very suspicious, since this is very basic functionaly for which we have bunch of integration tests.

So onAddOrUpdate is not called only if the event is filtered out, usually by the label.

[thomasdraebing]

Hi @csviri, thanks a lot for the quick response.

To me the labels look fine. They stay the same during updates and during the restart of the operator. Maybe I am looking at the wrong ones? Could you clarify where to look exactly?

The owner reference of the StatefulSet also looks ok. It is also the same reference as for the configMaps where the reconciliation works as expected.

I can reproduce it by starting the operator, installing the Gerrit-CustomResource and then restarting the operator. Already during the initial reconciliation after the operator started the issue occurs. I posted the complete debug logs of this here: https://gist.github.com/thomasdraebing/8752cff669ba00f569fb4395b7398416

I can't see anything suspicious in the logs. If I can provide some additional information I am happy to do so.

[csviri]

So I see a label selector on the StatefulSetDependentResource:
labelSelector = "app.kubernetes.io/component=gerrit-statefulset".
(added by the annotation)
Could you it after removing this?

I see that there is labels added to the resource in the desired, but those seems to be different:
see part:

public static Map<String, String> getLabels(GerritCluster gerritCluster, Gerrit gerrit) {
  return gerritCluster.getLabels(
  String.format("gerrit-statefulset-%s", gerrit.getMetadata().getName()),
  GerritReconciler.class.getSimpleName());
}

There should be also a label on the StatefulSet same as in label selector. But again I would first just try by removing label selector from the dependent resource.

[thomasdraebing]

That was indeed it.
Thanks for helping me figuring this out!

[csviri]

np,
if you could give us more info about the operator you writing (and if not a secret) would be helpful :)
even if it becomes production ready pls consider adding it to the list here: https://github.com/java-operator-sdk/java-operator-sdk#projects-using-josdk

[thomasdraebing]

Hi @csviri,
sure. The operator I am working on is part of the Gerrit Code Review open source project. Gerrit is a git server (like GitHub) with a focus on code review. It is written in Java, which is why the Java Operator SDK is a natural fit allowing to use APIs provided by Gerrit.
Gerrit is not scalable out of the box due to its statefulness (git repositories, git-based database and index have to be kept in sync). However, recent efforts in the community provided a so-called multisite setup, which would allow us to scale Gerrit. This setup involves multiple additional components and is quite complex to set up. An operator will greatly help in deploying, configuring and scaling Gerrit in a multisite setup.
We will for sure add the operator to the list of java-operator-sdk users as soon as it is ready. The project is fully open source and contributions are of course always welcome :-)

[csviri]

great! thank you @thomasdraebing !