Skip to content

Feat: Add Pipeline Cancellation Feature with Enhanced Reliability#2258

Merged
danielaskdd merged 9 commits intoHKUDS:mainfrom
danielaskdd:pipeline-cancelllation
Oct 24, 2025
Merged

Feat: Add Pipeline Cancellation Feature with Enhanced Reliability#2258
danielaskdd merged 9 commits intoHKUDS:mainfrom
danielaskdd:pipeline-cancelllation

Conversation

@danielaskdd
Copy link
Collaborator

@danielaskdd danielaskdd commented Oct 24, 2025

Feat: Add Pipeline Cancellation Feature with Enhanced Reliability

Overview

This PR introduces a graceful pipeline cancellation mechanism for document processing operations, along with critical fixes for lock leakage issues that could cause resource exhaustion in production environments.

iShot_2025-10-25_10 28 49

Key Features

1. Pipeline Cancellation System

  • User-Initiated Cancellation: Added ability for users to cancel long-running document processing operations through the UI
  • Graceful Termination: Processes stop cleanly at safe checkpoints without data corruption
  • Progress Preservation: Successfully processed documents are retained while only incomplete operations are cancelled
  • Status Tracking: Real-time feedback showing which documents were processed before cancellation

2. Critical Lock Management Fixes

  • Lock Leakage Prevention: Implemented comprehensive rollback protection to prevent lock leaks during cancellation or failures
  • Atomic Lock Operations: Enhanced lock acquisition/release with proper state tracking using dictionaries
  • Shield Protection: Used asyncio.shield() to ensure cleanup completes even under cancellation
  • Reference Counting: Fixed reference count management to prevent memory leaks

3. Enhanced Error Handling

  • Differentiated Logging: User cancellations log as warnings without stack traces, while genuine errors log full context
  • LLM Cache Persistence: Added error handling for cache persistence failures to prevent pipeline crashes
  • Variable Initialization: Fixed UnboundLocalError issues by initializing variables at function start

4. Frontend Changes

UI Components:

  • Added Cancel button to Pipeline Status Dialog
  • Implemented confirmation dialog for cancellation requests
  • Removed separate retry button and merged functionality into scan button
  • Simplified status dialog layout with consolidated message sections
  • Improved text overflow handling with overflow-x-hidden and break-all

API Integration (lightrag_webui/src/api/lightrag.ts):

  • Added cancelPipeline API method

Breaking Changes

None. All changes are backwards compatible.

• Add cancel_pipeline API endpoint
• Implement PipelineCancelledException
• Add cancellation checks in main loop
• Handle task cancellation gracefully
• Mark cancelled docs as FAILED
- Add cancelPipeline API endpoint
- Add cancel button to status dialog
- Update status response type
- Add cancellation UI translations
- Handle cancellation request states
• Change default log level to INFO
• Force enable error logging output
• Add lock cleanup rollback protection
• Handle LLM cache persistence errors
• Fix async task exception handling
• Remove separate latest message section
• Combine into single pipeline messages area
• Add overflow-x-hidden for better display
• Change break-words to break-all
• Update translations across all locales
@danielaskdd danielaskdd merged commit e2ec1cd into HKUDS:main Oct 24, 2025
1 check passed
@danielaskdd danielaskdd deleted the pipeline-cancelllation branch October 25, 2025 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant