feat: Add Automatic Text Truncation Support for Embedding Functions

[danielaskdd]

Add Automatic Text Truncation Support for Embedding Functions

Summary

This PR enhances the EmbeddingFunc wrapper to automatically inject max_token_size parameter to underlying embedding functions that support it. This enables automatic text truncation for embedding operations, preventing API errors caused by texts exceeding model token limits.

Changes

Core Enhancement: lightrag/utils.py

• Added import inspect for function signature introspection
• Added automatic max_token_size injection logic in EmbeddingFunc.__call__:
  • Uses inspect.signature() to check if the underlying function supports max_token_size
  • Only injects when the parameter is supported (avoids TypeError for unsupported functions)

OpenAI Embedding: lightrag/llm/openai.py

• Added import tiktoken for tokenization
• Added _TIKTOKEN_ENCODING_CACHE module-level cache and _get_tiktoken_encoding_for_model() helper
• Added max_token_size parameter to openai_embed() function
• Implemented client-side text truncation using tiktoken (OpenAI API may return errors for over-limit texts)

Gemini Embedding: lightrag/llm/gemini.py

• Added max_token_size parameter to gemini_embed() function
• No client-side truncation - Gemini API handles truncation automatically (autoTruncate=True by default)

Ollama Embedding: lightrag/llm/ollama.py

• Added max_token_size parameter to ollama_embed() function
• Added comprehensive docstring
• No client-side truncation - Ollama API handles truncation automatically based on num_ctx setting

Minor Updates

• lightrag/api/lightrag_server.py: Updated log message for clarity
• lightrag/operate.py: Changed token threshold from 90% to 100% and improved warning message

Truncation Strategies by Provider

Provider	Truncation Strategy	Reason
OpenAI	Client-side (tiktoken)	API may return errors for over-limit texts
Gemini	Server-side (autoTruncate)	API automatically truncates
Ollama	Server-side (num_ctx)	API automatically truncates

How It Works

User calls: await embedding_func(texts)
    ↓
EmbeddingFunc.__call__:
    1. inspect.signature() checks if func supports max_token_size
    2. If supported → inject max_token_size from decorator
    ↓
embedding_func(texts, max_token_size=...):
    - openai_embed: Client-side truncation with tiktoken
    - gemini_embed / ollama_embed: Server-side automatic truncation

Backward Compatibility

• ✅ Fully backward compatible
• Embedding functions without max_token_size parameter continue to work (signature check prevents injection)
• No breaking changes to existing API

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.