Operator does not receive any events after startup

[steknowis]

Bug Report

What did you do?

Started my operator (locally or in cluster) with Spring Boot. Startup logs looking good and all custom resources are processed once. But after startup no events at all are processed. I have a reconciliation TimerEventSource that should be triggered every 30s, but nothing happens. I change a CR, nothing happens.

1. Start (Spring Boot) operator in debug mode, wait for complete startup
2. Change a watched CR or create a new one
3. Check if createOrUpdateResource method of your ResourceController is invoked

What did you expect to see?

Events (of TimerEventSource and create/update/delete) should trigger `createOrUpdateResource `or `deleteResource `method of the ResourceController after startup.

What did you see instead? Under which circumstances?

No events were triggered and processed. No error message, operator running as normal. Happening locally or in cluster with operator version 1.9.4; 1.9.3 works as expected.

Environment

Kubernetes cluster type:

OpenShift 4.6.36

$ Mention java-operator-sdk version from pom.xml file

1.9.4

$ java -version

openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.11+9-LTS, mixed mode)

$ kubectl version

Client Version: 4.6.23
Server Version: 4.6.36
Kubernetes Version: v1.19.0+b00ba52

Spring Boot version:
2.4.8

Additional context

My operator registration:

    @Bean(initMethod = "start", destroyMethod = "close")
    @Autowired
    public <R extends CustomResource<?, ?>> Operator operator(OpenShiftClient client,
                                                              List<ResourceController<R>> controllers, KubernetesEnvironmentService environmentService) {
        ConfigurationService configurationService = DefaultConfigurationService.instance();
        Operator operator = new Operator(client, configurationService);

        String namespace = environmentService.getCurrentNamespace();
        controllers.forEach(c -> registerController(operator, configurationService, c, namespace));

        return operator;
    }

    @Bean
    @Autowired
    public MyController myController(Service a, Service b) {
        return new MyController(a, b);
    }

    private <R extends CustomResource<?, ?>> void registerController(Operator operator,
                                                                     ConfigurationService configurationService, ResourceController<R> controller, String namespace) {
        // override configuration to watch only current namespace
        ControllerConfiguration<R> configuration = configurationService.getConfigurationFor(controller);
        configuration = ControllerConfigurationOverrider.override(configuration).settingNamespace(namespace).build();
        operator.register(controller, configuration);
    }

[metacosm]

Unrelated to your issue (at least, I don't think it is) but any particular reason you're not configuring the namespace via the @Controller annotation?

Also, any particular reason you're not using the starter: https://github.com/java-operator-sdk/operator-framework-spring-boot-starter/?

[steknowis]

In a future iteration of the operator, we need to watch multiple namespaces that can not be hardcoded in the @Controller annotation. That's why we already configure the namespace dynamically. Is there an alternative for that?

Not a particular reason why we don't use the starter. I will try it out, thanks.

[metacosm]

In a future iteration of the operator, we need to watch multiple namespaces that can not be hardcoded in the @Controller annotation. That's why we already configure the namespace dynamically. Is there an alternative for that?

Makes sense. You can specify the namespaces to watch in application.properties when you use the Quarkus extension but I'm not sure if something similar is possible with the Spring Boot starter (if it's not, it should be implemented, imo).

Not a particular reason why we don't use the starter. I will try it out, thanks.

It's unfortunately not that well-maintained at this point and in need of some love… :(

[metacosm]

Could you check if this still happens with #517, by any chance?

[metacosm]

Also, may I ask about your use case in greater details? It's always interesting to hear about how people use the SDK to solve their issues.

[steknowis]

Checked out the branch "configured-controller" and installed it via maven. My Operator then does not start:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'operator' defined in class path resource [de/xy/cp/run/myoperator/config/OperatorConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.javaoperatorsdk.operator.Operator]: Factory method 'operator' threw exception; nested exception is java.lang.ExceptionInInitializerError
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:771)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:763)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:438)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:339)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318)
	at de.xy.cp.run.myoperator.myOperatorApplication.main(MyOperatorApplication.java:34)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.javaoperatorsdk.operator.Operator]: Factory method 'operator' threw exception; nested exception is java.lang.ExceptionInInitializerError
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
	... 20 common frames omitted
Caused by: java.lang.ExceptionInInitializerError: null
	at de.xy.cp.run.myoperator.config.OperatorConfiguration.operator(OperatorConfiguration.java:45)
	at de.xy.cp.run.myoperator.config.OperatorConfiguration$$EnhancerBySpringCGLIB$$1623fe11.CGLIB$operator$0(<generated>)
	at de.xy.cp.run.myoperator.config.OperatorConfiguration$$EnhancerBySpringCGLIB$$1623fe11$$FastClassBySpringCGLIB$$2a07cc2b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
	at de.xy.cp.run.myoperator.config.OperatorConfiguration$$EnhancerBySpringCGLIB$$1623fe11.operator(<generated>)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 21 common frames omitted
Caused by: java.lang.NullPointerException: null
	at java.base/java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1470)
	at java.base/java.text.DateFormat.parse(DateFormat.java:393)
	at io.javaoperatorsdk.operator.api.config.Utils.loadFromProperties(Utils.java:43)
	at io.javaoperatorsdk.operator.config.runtime.DefaultConfigurationService.<init>(DefaultConfigurationService.java:14)
	at io.javaoperatorsdk.operator.config.runtime.DefaultConfigurationService.<clinit>(DefaultConfigurationService.java:11)
	... 32 common frames omitted

Also, may I ask about your use case in greater details? It's always interesting to hear about how people use the SDK to solve their issues.

We use the operator to create and update keycloak clients and realms. For example you create a "realm" CR with roles and realm name and the operator creates the realm for you in the keycloak. In general the java-operator-sdk is really nice, especially that one can relatively easy add a reconciliation loop. Even though it would be nicer to have a complete native approach (#369) ;)

[metacosm]

Checked out the branch "configured-controller" and installed it via maven. My Operator then does not start:

Caused by: java.lang.NullPointerException: null
	at java.base/java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1470)
	at java.base/java.text.DateFormat.parse(DateFormat.java:393)
	at io.javaoperatorsdk.operator.api.config.Utils.loadFromProperties(Utils.java:43)
	at io.javaoperatorsdk.operator.config.runtime.DefaultConfigurationService.<init>(DefaultConfigurationService.java:14)
	at io.javaoperatorsdk.operator.config.runtime.DefaultConfigurationService.<clinit>> 

That is weird because there's no reason why the date parsing associated with the version should start failing now… :(

We use the operator to create and update keycloak clients and realms. For example you create a "realm" CR with roles and realm name and the operator creates the realm for you in the keycloak. In general the java-operator-sdk is really nice, especially that one can relatively easy add a reconciliation loop. Even though it would be nicer to have a complete native approach (#369) ;)

Not sure I understand the "complete native approach" in the context of the referenced issue… 🤔 Are you referring to the fact that it'd be nice to be able to reschedule reconciliation?

[metacosm]

I've updated the PR to avoid crashing if the build time is not available.

[steknowis]

With the new version of the PR it worked! But I also needed to updated the fabric8-client version to 5.7.0. I can not exclude that the original issue was coming from using version 5.4.0 of fabric8-client. I know that in the release note it is stated which fabric8 version to use, I will fix that on our side.

Thanks for the fix!

Not sure I understand the "complete native approach" in the context of the referenced issue… 🤔 Are you referring to the fact that it'd be nice to be able to reschedule reconciliation?

Yes some easier approach to reschedule reconciliation would be nice, but TimerEventSource also works fine for the moment.

[metacosm]

Glad to hear it worked!

[metacosm]

Fixed by #517.