Skip to content

Refact: Graceful shutdown and signal handling in Gunicorn Mode#2280

Merged
danielaskdd merged 5 commits intoHKUDS:mainfrom
danielaskdd:fix-exit-handler
Oct 29, 2025
Merged

Refact: Graceful shutdown and signal handling in Gunicorn Mode#2280
danielaskdd merged 5 commits intoHKUDS:mainfrom
danielaskdd:fix-exit-handler

Conversation

@danielaskdd
Copy link
Collaborator

Refact: Graceful shutdown and signal handling in Gunicorn Mode

This branch fixes critical issues with graceful shutdown and signal handling in LightRAG's API server, particularly for systemd service deployments and Gunicorn multi-process setups.

Problem: The original implementation had custom signal handlers that competed with Uvicorn/Gunicorn's built-in signal handling, causing:

  • Resources not being properly cleaned up
  • Race conditions during shutdown
  • Systemd service stop failures or timeouts

Solution: Leverage the framework's built-in mechanisms:

  • Uvicorn: Use lifespan context manager for cleanup
  • Gunicorn: Use on_exit hook in master process for shared resource cleanup
  • Let each framework handle its own signal processing

This ensures clean, predictable shutdowns in all deployment scenarios (standalone Uvicorn, Gunicorn multi-process, and systemd service management).

Key Changes

1. Signal Handler Removal

  • Removed custom signal handlers that were interfering with built-in shutdown mechanisms
  • Both Uvicorn and Gunicorn have their own signal handling - custom handlers were causing conflicts

2. Improved Shutdown Logic

  • Added detection for Gunicorn vs Uvicorn mode using LIGHTRAG_GUNICORN_MODE environment variable
  • Cleanup logic now properly delegates to the appropriate context (worker vs master process)
  • In Gunicorn mode, shared storage finalization deferred to master process via on_exit hook
  • In Uvicorn mode, cleanup handled directly in lifespan context manager

3. Systemd Service Configuration

  • Updated lightrag.service.example with proper process management settings
  • Added KillMode=process to target main process only
  • Added explicit ExecStop with proper timeout (60s)
  • Removed wrapper script requirement - now directly executes lightrag-gunicorn or lightrag-server
  • Added environment variable setup for virtual environment PATH
  • Increased restart delay from 10s to 30s

4. File Cleanup

  • Removed lightrag-api wrapper script (no longer needed)
  • Simplified deployment by eliminating unnecessary indirection

5. Documentation Updates

  • Updated both English and Chinese READMEs with new systemd service setup instructions
  • Clarified that ExecStart must directly call the main process (no wrapper scripts)
  • Removed confusing wrapper script documentation

• Add worker_exit hook in gunicorn config
• Add shutdown_manager parameter in finalize_share_data of share_storage
• Prevent Manager shutdown in workers
• Remove custom signal handlers
• Document UvicornWorker hook limitations
• Add GUNICORN_CMD_ARGS cleanup guard
• Prevent double cleanup in workers
• Use custom env var for mode detection
• Improve Gunicorn mode reliability
• Remove unreliable worker_exit function
• Add debug logs for cleanup modes
• Move DEBUG_LOCKS to top of file
- Remove bash wrapper script
- Update systemd service configuration
- Improve process management for gunicorn
- Simplify shared storage cleanup logic
- Update documentation for deployment
@danielaskdd
Copy link
Collaborator Author

@codex review

@chatgpt-codex-connector
Copy link

Codex Review: Didn't find any major issues. Nice work!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danielaskdd danielaskdd merged commit a1cf01d into HKUDS:main Oct 29, 2025
1 check passed
@danielaskdd danielaskdd deleted the fix-exit-handler branch October 29, 2025 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant