Fix Gemini driver retry mechanism

[danielaskdd]

🔧 Fix Gemini driver retry mechanism

Problem

The Gemini LLM driver had two critical issues:

1. Missing Retry Mechanism: When concurrent requests reached the 8th request, the system would fail with RuntimeError: Gemini response did not contain any text content without any retry attempts, causing document processing to crash.

2. Missing Dependency: The code imported google.api_core.exceptions but google-api-core was not listed as a dependency, causing ModuleNotFoundError on fresh installations.

3. Overly Broad Retry Logic: The initial fix used a too-broad Exception retry that would retry non-recoverable errors like authentication failures and configuration errors, wasting time and resources.

Changes Made

1. Enhanced Retry Logic (lightrag/llm/gemini.py)

• Added InvalidResponseError exception class for empty response scenarios
• Implemented precise exception-based retry using google.api_core.exceptions:
  • InternalServerError (500) - Server errors
  • ServiceUnavailable (503) - Service unavailable
  • ResourceExhausted (429) - Rate limiting
  • GatewayTimeout (504) - Gateway timeout
  • BadGateway (502) - Bad gateway
  • DeadlineExceeded - Request timeout
  • Aborted (409) - Concurrency conflicts
  • Unknown (500) - Unknown server errors
  • InvalidResponseError - Empty response errors
• Automatic dependency installation via pipmaster for google-api-core
• 3 retry attempts with exponential backoff (4-60 seconds)
• Applied to both gemini_complete_if_cache and gemini_embed functions

2. Dependency Management

pyproject.toml:

• Added google-api-core>=2.0.0,<3.0.0 to core dependencies
• Added to api optional dependencies
• Added to offline-llm optional dependencies

requirements-offline-llm.txt:

• Added google-api-core>=2.0.0,<3.0.0

Benefits

✅ Fixes concurrent request failures - Properly handles transient errors on the 8th concurrent request
✅ Smart retry strategy - Only retries recoverable errors, fails fast on permanent errors
✅ Prevents dependency issues - Automatic installation ensures google-api-core is always available
✅ Handles rate limiting - Exponential backoff for 429 errors
✅ Production-ready - Enterprise-grade error handling matching industry best practices
✅ Consistent with OpenAI driver - Similar retry logic across LLM bindings

Errors That Will Be Retried (Max 3 attempts)

• Server errors (500, 503)
• Rate limiting (429)
• Gateway errors (502, 504)
• Timeouts
• Empty responses

Errors That Will Fail Immediately (No retry)

• Missing API key (ValueError)
• Authentication errors (401)
• Invalid configuration (400)
• Programming errors (TypeError, AttributeError)

Testing Recommendations

1. Test concurrent requests: Run 10+ concurrent document processing requests
2. Test empty response handling: Verify retry behavior on empty responses
3. Test dependency installation: Fresh install to confirm google-api-core auto-installs
4. Test rate limiting: Verify exponential backoff on 429 errors
5. Test fast-fail: Confirm authentication errors don't trigger retries

Files Changed

• lightrag/llm/gemini.py - Enhanced retry logic and dependency installation
• pyproject.toml - Added google-api-core dependency (3 locations)
• requirements-offline-llm.txt - Added google-api-core for offline installs

Breaking Changes

None. This is a backward-compatible improvement.

Related Issues

Fixes: ModuleNotFoundError: No module named 'google.api_core'