Task Queue time for Tasks executed using KubernetesExecutor

[stephsi]

Hello everyone!

I have not found much information regarding my question, thus reaching out here:
We are running Airflow 2.9.2 on Openshift using the KubernetesExecutor. Many of our tasks are quite short around 5-15 seconds, but the queue time still is around 25-30 seconds for every task execution, even after tweaking various settings. The airflow pods have limits of 500m CPU and 512MB of memory and of course the airflow image is cached on the nodes and we use the ifnotpresent pull policy.

Thus my question: is 25-30 seconds normal for the queueing time of airflow tasks executed by the KubernetesExecutor?
Due to the lack of good answers, I questioned chatgpt as well and got a wide estimate of 10-50 seconds.

I know that the queing time can depend on many aspects, but with this poll I want to get a feeling what queueing times other people are able to achieve. If possible, also leave a short comment with a short description of your environment, so others have an idea where to look for better performance. Edit: I am totally aware that CeleryExecutor would be a better choice for many smaller tasks, but I am specifically interested what is possible using the KubernetesExecutor.

Hopefully the results also help others when they try to optimize their airflow setup.

Thanks

[potiuk]

One comment. If you care about latency of tasks and you have many small tasks you should look at CeleryKubernetesExecutor (Celery deployed on K8S) - and in Airflow 3 this setup is possible with Multi-Executor setup.