Skip to content

Conversation

@desertaxle
Copy link
Member

closes #19937

Summary

  • Adds a configurable semaphore to rate-limit concurrent API calls during Kubernetes observer startup
  • Prevents the "thundering herd" problem when there are many existing pods/jobs in the cluster
  • Default concurrency limit is 5, configurable via PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_STARTUP_EVENT_CONCURRENCY
Changes
  • Added startup_event_concurrency setting to KubernetesObserverSettings with default value of 5
  • Added module-level semaphore _startup_event_semaphore initialized during observer startup
  • Wrapped the startup event deduplication API call with the semaphore to limit concurrent requests
  • Added test to verify semaphore correctly limits concurrency

🤖 Generated with Claude Code

…r startup

closes #19937

When the observer starts, it receives events for all existing pods/jobs
and checks for duplicates via API calls. This can overwhelm the server
when there are many pods. This change adds a configurable semaphore
(default: 5 concurrent calls) to rate-limit these startup checks.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@github-actions github-actions bot added the enhancement An improvement of an existing feature label Dec 23, 2025
@desertaxle desertaxle marked this pull request as ready for review December 23, 2025 21:20
@desertaxle desertaxle merged commit 156d775 into main Dec 23, 2025
19 checks passed
@desertaxle desertaxle deleted the fix-k8s-events-thundering-herd branch December 23, 2025 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement An improvement of an existing feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ignore past k8s events in worker's observer

3 participants