Refact: Limit Vector Database Metadata Size

[danielaskdd]

Refact: Limit Vector Database Metadata Size

Problem Statement

In production deployments, entity and relation metadata can grow unbounded as documents are continuously ingested. The source_id (chunk IDs) and file_path fields in entities and relations can accumulate thousands of entries, leading to:

• Performance degradation in vector database operations
• Increased storage costs
• Memory pressure during query operations
• Slower merge operations when processing new documents

Solution Overview

This PR implements a configurable metadata size control system with two key features:

1. Source ID limiting: Controls the maximum number of chunk IDs stored per entity/relation
2. File path limiting: Controls the maximum number of file paths displayed in metadata (display-only, doesn't affect query performance)

Both features support two strategies:

• FIFO (First In First Out): Removes oldest entries when limit is reached. Best for evolving knowledge bases, keeps most recent information.
• KEEP: Keeps oldest entries, skips new ones when limit is reached. Best for stable knowledge bases, faster (fewer merge operations)

Key Features

1. Chunk Tracking System

• New storage fields: entity_chunks and relation_chunks
• Tracks which chunks reference each entity/relation
• Enables accurate deduplication during merging
• Supports data migration from existing deployments

2. Configurable Limits

New environment variables:

# Source ID limits (affects query performance)
MAX_SOURCE_IDS_PER_ENTITY=300
MAX_SOURCE_IDS_PER_RELATION=300
SOURCE_IDS_LIMIT_METHOD=FIFO

# File path limits (display only)
MAX_FILE_PATHS=100

3. Visual Indicators

• Truncation indicator (†) in graph UI properties view
• Tooltip showing truncation method (FIFO/KEEP)

4. Storage Backend Support

Implemented across all storage backends:

• PostgreSQL
• MongoDB
• Redis
• JSON (default)
• All support data migration and backward compatibility

5. Enhanced Logging

• Shows source ID ratios when skipping entities/edges
• Informative messages about limit methods during rebuild
• Better debugging information for production deployments

Configuration Details

Default Values

DEFAULT_MAX_SOURCE_IDS_PER_ENTITY = 300
DEFAULT_MAX_SOURCE_IDS_PER_RELATION = 300
DEFAULT_SOURCE_IDS_LIMIT_METHOD = "FIFO"
DEFAULT_MAX_FILE_PATHS = 100

Limit Strategies

• FIFO: Best for evolving knowledge bases, keeps most recent information
• KEEP: Best for stable knowledge bases, faster (fewer merge operations)

Breaking Changes

None. The changes are fully backward compatible:

• Existing data is automatically migrated on first access
• Default limits are high enough to not affect small deployments
• Old behavior can be approximated by setting very high limits

Performance Impact

Positive impacts:

• Reduced vector database query overhead
• Lower memory usage during entity/relation merging
• Faster metadata deserialization

Considerations:

• Initial migration may take time on large existing databases
• FIFO strategy requires more merge operations than KEEP

Migration Path

For existing deployments:

1. Update code to this branch
2. Update .env with new configuration variables (optional)
3. Restart LightRAG service
4. Storage backends will auto-migrate on first access
5. Monitor logs for migration status

[danielaskdd]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".