Skip to content

⚡ Switch from prefork to threads for Celery concurrency#375

Closed
stevenbal wants to merge 3 commits into
mainfrom
feature/concurrency-improvements
Closed

⚡ Switch from prefork to threads for Celery concurrency#375
stevenbal wants to merge 3 commits into
mainfrom
feature/concurrency-improvements

Conversation

@stevenbal

@stevenbal stevenbal commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Because Open Notificaties is primarily I/O bound, the Celery workers could spend a decent amount of time waiting, and in this scenario it is more efficient to use threads because they have a lower memory footprint than processes

TODO:

  • Push custom docker tag and let DH loadtest with this
  • adjust default concurrency numbers in charts?
  • Should we apply the same for apps that send to open notificaties?
  • reuse session in tasks

Changes

  • Switch from prefork to threads for Celery concurrency
  • Disable DB connection pooling and increasy Celery concurrency for docker-compose
  • Update webhook demo setup to use fastAPI

Checklist

Check off the items that are completed or not relevant.

  • Experimental features/changes

    • Any experimental features added in this PR are backwards compatible
    • Any experimental features added in this PR are documented in the docs/api/experimental.rst page
  • Commit hygiene

    • Commit messages refer to the relevant Github issue
    • Commit messages explain the "why" of change, not the how

because Open Notificaties is primarily I/O bound, the Celery workers could spend a decent amount of time waiting, and in this scenario it is more efficient to use threads because they have a lower memory footprint than processes
@stevenbal stevenbal marked this pull request as draft March 9, 2026 16:12
@codecov

codecov Bot commented Mar 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.68%. Comparing base (8bef94b) to head (957c662).
⚠️ Report is 27 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #375   +/-   ##
=======================================
  Coverage   93.68%   93.68%           
=======================================
  Files         124      124           
  Lines        4231     4231           
  Branches      666      666           
=======================================
  Hits         3964     3964           
  Misses        205      205           
  Partials       62       62           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

this allows more proper simulation of timeouts, without limiting the concurrency of the webhook itself
@stevenbal stevenbal force-pushed the feature/concurrency-improvements branch from 952bbd0 to 957c662 Compare March 10, 2026 08:40
@stevenbal stevenbal changed the title Feature/concurrency improvements ⚡ Switch from prefork to threads for Celery concurrency Mar 10, 2026
@stevenbal stevenbal mentioned this pull request Mar 30, 2026
20 tasks
@stevenbal

Copy link
Copy Markdown
Contributor Author

Closing this in favor of #364 since that also introduces the same change

@stevenbal stevenbal closed this Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant