feat: Add token tracking support to openai_embed function

[yrangana]

Description

This PR adds optional token tracking support to the openai_embed() function, enabling comprehensive monitoring of embedding token usage alongside existing LLM token tracking capabilities.

Related Issues

N/A - Enhancement to improve observability of API token usage for embeddings.

Changes Made

• Add optional token_tracker parameter to openai_embed() function in lightrag/llm/openai.py
• Track prompt_tokens and total_tokens for embedding API calls
• Update function docstring to document the new token_tracker parameter
• Enables monitoring of embedding token usage alongside LLM calls
• Maintains backward compatibility with existing code (parameter is optional)

Checklist

• Changes tested locally
• Code reviewed
• Documentation updated

Additional Notes

Implementation Details:

• Follows the same pattern used in openai_complete_if_cache() for consistency
• Token tracking only occurs when token_tracker is provided and response contains usage data
• No breaking changes - existing code continues to work without modifications

Usage Example:

from lightrag.utils import TokenTracker
from lightrag.llm.openai import openai_embed

token_tracker = TokenTracker()
embeddings = await openai_embed(
    texts=["example text"],
    token_tracker=token_tracker
)
print(token_tracker)  # Shows token usage statistics

[danielaskdd]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Hooray!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting

• @codex fix this CI failure
• @codex address that feedback

[danielaskdd]

Thanks for sharing.