Controller loses API connection after token expiry on Azure Kubernetes Service (AKS) 1.30 due to `kopf` bug

[creste]

dask-operator fails to create Dask Jobs on Azure Kubernetes Service (AKS) 1.30:

See this kopf bug report for details.

Minimal Complete Verifiable Example:

1. Install dask-operator on AKS 1.30.
2. Wait an hour for the authentication token to expire.
3. Create a DaskJob resource.

dask-operator will not create the DaskJob because dask-operator's kubernetes authentication token has expired and kopf's watchers are no longer connected to kubeapi. A bug in kopf prevents kopf from refreshing the authentication token.

This only occurs on AKS 1.30+ because that is the first AKS version that sets --service-account-extend-token-expiration to false.

Environment:

• Dask operator version: 2024.9.0

[jacobtomlinson]

Thanks for flagging this here. I don't see any immediate solution we can implement in dask-kubernetes to work around this so I expect we will need to wait for a fix in nolar/kopf#980

[creste]

@jacobtomlinson - nolar/kopf#980 was fixed in kopf 1.38.0. Updating dask-kubernetes to use the latest version of kopf should fix this issue.

[jacobtomlinson]

Thanks for the update. I'll close this out as fixed upstream.

[guozhans]

@jacobtomlinson will you update the minimum version? and make a new release for operator?

dask-kubernetes/pyproject.toml

Line 17 in b2099bc

"kopf>=1.35.3",