K8S probe timing out when using ManagedExecutor a lot

[manofthepeace]

Hi all,

this possibly relates to #23429 and this comment on another related issue; #7893 (comment)

We do not use much resteasy-reactive, just few custom management apis, but we do an extensive use of the ManagedExecutor. At one point kubernetes killed our application because of a timeout in the liveness probe. This happens when we run with a certain load (managed executor running async about 5k+ somewhat short tasks), called from a quartz @Scheduled method

we got this exception when it happened before everything died and got restarted again. Unsure if it is actually related to the actual issue since k8s killed the pod.

2022-07-26 18:00:48,596 ERROR [io.qua.ver.cor.run.VertxCoreRecorder] (executor-thread-39) Uncaught exception received by Vert.x: java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:923)
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:350)
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:343)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:825)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815)
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:83)
        at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:173)
        at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:66)
        at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)
        at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23)
        at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:161)
        at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:555)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1462)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:833)
        at com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:704)
        at com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:202)

2022-07-26 18:00:48,596 INFO  [io.quarkus] (Shutdown thread) eae stopped in 23.672s

It would be nice I think for that reason, to get the management apis on another port, if it could solve that particular issue. I wonder if anybody had that kind of thing, and any ways to solve it with the current quarkus implementation. I am sure I can bump the timeout, but it seems like pushing the problem, until a next time.

[quarkus-bot[bot]]

/cc @geoand, @iocanel

[geoand]

Could you show an example of your Quartz usage?

[manofthepeace]

Thanks for the confirmation. So yes seems to be 2 things in the end because with @ManagedExecutorConfig(maxAsync = 150) seems like it alleviated the problem as we cannot see it anymore. So looks like exhausting the executor threads is not that good of a thing i.e. 150/200. Is there facilities to created our own threadpool in quarkus? I did not find any besides via vertx.

Also yes our pod cpu is 500m at the moment, which seem ok for most of the time, but when our scheduled task run it seems to peak and render the processing of other things not really efficient.

[geoand]

Is there facilities to created our own threadpool in quarkus?

We don't provide anything other than the ManagedExecutor and the built in executors for Vert.x. But if we did, I don't think it would solve your problem.

Also yes our pod cpu is 500m at the moment

I would advise against allocating less than 2 CPUs to JVM processes. See this and this for more information.

[manofthepeace]

thank you, just to be sure that 2cpu recommendation is also for native application right?

[geoand]

Good question. @galderz any specific recommendation for ^

[galderz]

@geoand Sorry I missed this. @manofthepeace GraalVM CE only has a serial GC right now, so giving it 2 CPUs won't have the same effect as in JVM mode, where G1 can take advantage of the extra CPU to do some of the work concurrently. There is a G1 GC for GraalVM but that's only available in their enterprise version.