Refact: Separate Configuration of RAGAS for LLM and Embeddings

[danielaskdd]

📊 Separate Configuration of RAGAS for LLM and Embeddings

Overview

This PR enhances the RAGAS evaluation system with more flexible configuration options, improved user experience, and better documentation. The changes primarily focus on supporting custom OpenAI-compatible endpoints for both LLM and embedding models while improving the evaluation workflow.

Key Features

🔧 Separate Endpoint Configuration for LLM and Embeddings

• New Environment Variables:
  • EVAL_EMBEDDING_BINDING_API_KEY: Dedicated API key for embedding models
  • EVAL_EMBEDDING_BINDING_HOST: Dedicated endpoint URL for embedding models
• Smart Fallback Chain:
  • Embedding API key: EVAL_EMBEDDING_BINDING_API_KEY → EVAL_LLM_BINDING_API_KEY → OPENAI_API_KEY
  • Embedding endpoint: EVAL_EMBEDDING_BINDING_HOST → EVAL_LLM_BINDING_HOST → None
• Use Case: Enables using different providers or endpoints for LLM (e.g., vLLM) and embeddings (e.g., custom embedding service)

🔇 Suppress Non-Critical Warnings

• Suppresses token usage warnings for custom OpenAI-compatible endpoints (vLLM, SGLang, etc.)
• Improves log clarity without affecting functionality
• Token tracking is non-critical for RAGAS evaluation

📈 Enhanced Progress Display

• Zero-padded evaluation index display for better readability
• Improved results display format with clearer sectioning
• Better structured logging output

📚 Documentation Improvements

• Refactored evaluation README with clearer structure and sections
• Added examples for the new configuration options
• Updated news section in main README files

Changes Summary

• lightrag/evaluation/eval_rag_quality.py: Core evaluation improvements (97 lines changed)
• lightrag/evaluation/README.md: Documentation refactor (186 lines changed)
• env.example: New configuration examples (18 lines changed)
• README.md & README-zh.md: News updates

Benefits

1. Flexibility: Support for heterogeneous evaluation setups (different LLM and embedding providers)
2. Better UX: Cleaner output with improved progress tracking and logging
3. Production-Ready: Suppress non-critical warnings for cleaner production logs
4. Better Documentation: Easier for users to understand and configure RAGAS evaluation

Testing

• ✅ Tested with custom OpenAI-compatible endpoints
• ✅ Verified fallback chain behavior
• ✅ Confirmed warning suppression works correctly
• ✅ Validated improved progress display formatting

Breaking Changes

None - all changes are backward compatible with existing configurations.

[danielaskdd]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Hooray!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".