[Bug]: LangChain usage extractors drop token usage when model metadata is absent

[feiiiiii5]

Describe the bug

The LangChain usage extractors (provider_usage_extractors in the Python SDK) discard an already-extracted usage payload whenever the model-name or provider lookup raises. In try_extract_provider_usage_data, the orchestrator catches exceptions from get_llm_usage_info and returns None, so the span is emitted with no usage/cost even though the tokens were extractable.

Affected lookups (all unguarded against missing keys):

• openai_usage_extractor: run_dict["extra"], run_dict["outputs"]["generations"][-1][-1]["generation_info"], and _get_provider's run_dict["extra"]
• groq_usage_extractor, vertexai_usage_extractor, anthropic_vertexai_usage_extractor, google_generative_ai_usage_extractor: run_dict["extra"]
• anthropic_usage_extractor: streaming path raises IndexError on empty generations (only KeyError is caught)

Reproduction: a streaming-shape OpenAI run whose usage sits in the message usage_metadata but whose generation dict has no generation_info key — try_extract_provider_usage_data returns None instead of the usage.

Expected behavior

Usage extraction should not depend on model metadata being present: extract the usage and return it with model=None when the model name cannot be resolved.

[github-actions]

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing solution or workaround found. Related issue #4474 (closed) touched streaming/usage-pydantic failures for OpenAI but did not address the root cause described here.

Code Investigation

The bug has two distinct failure modes in the same pipeline:

1. try_extract_provider_usage_data swallows partial success — usage_extractor.py

# sdks/python/src/opik/integrations/langchain/provider_usage_extractors/usage_extractor.py
def try_extract_provider_usage_data(run_dict):
    for extractor in _REGISTERED_PROVIDER_USAGE_EXTRACTORS:
        if not extractor.is_provider_run(run_dict):
            continue
        try:
            return extractor.get_llm_usage_info(run_dict)
        except Exception:
            LOGGER.warning(...)
            return None   # ← drops everything, even if usage was already extracted

If get_llm_usage_info raises (e.g., due to a missing key inside _try_get_model_name), the whole result is discarded, even though each extractor already calls _try_get_token_usage internally (which itself is guarded and returns None on failure).

2. _try_get_model_name in openai_usage_extractor.py performs unguarded key access

# sdks/python/src/opik/integrations/langchain/provider_usage_extractors/openai_usage_extractor.py

def _try_get_model_name(run_dict):
    model = None
    if metadata := run_dict["extra"].get("metadata"):   # KeyError if "extra" absent
        model = metadata.get("ls_model_name")

    if llm_output := run_dict["outputs"].get("llm_output"):
        model = llm_output.get("model_name", model)
    elif generation_info := run_dict["outputs"]["generations"][-1][-1]["generation_info"]:
        # ↑ KeyError if "generation_info" is absent (streaming without that key),
        #   or IndexError if generations list is empty
        model = generation_info.get("model_name", model)

    return model

def _get_provider(self, run_dict):
    if base_url := run_dict["extra"].get("invocation_params", {}).get("base_url"):
        # ↑ KeyError if "extra" key itself is absent
        ...

3. _try_get_model_name in anthropic_usage_extractor.py only catches KeyError, missing IndexError

# sdks/python/src/opik/integrations/langchain/provider_usage_extractors/anthropic_usage_extractor.py
for model_name_key in POSSIBLE_MODEL_NAME_KEYS:
    try:
        ...
        model = run_dict["outputs"]["generations"][-1][-1]["message"]["kwargs"][
            "response_metadata"
        ][model_name_key]
    except KeyError:  # ← IndexError on empty generations list is NOT caught
        continue

4. _try_get_model_name in groq_usage_extractor.py — unguarded "extra" key access

def _try_get_model_name(run_dict):
    model = run_dict["extra"].get("metadata", {}).get("ls_model_name")  # KeyError if "extra" absent

Test coverage: sdks/python/tests/unit/integrations/ contains only an otel/ subdirectory — there are no unit tests for any of the LangChain provider usage extractors, which explains why these edge cases have gone undetected.

Next Steps

There are two complementary fixes needed:

Fix 1 — Harden try_extract_provider_usage_data (usage_extractor.py): On exception, instead of returning None, attempt to salvage the partial result by calling usage/model extraction individually. Alternatively, restructure each extractor so get_llm_usage_info never raises (both _try_get_token_usage and _try_get_model_name already use try/except; the issue is they aren't always applied consistently):

# Proposed change in try_extract_provider_usage_data:
try:
    return extractor.get_llm_usage_info(run_dict)
except Exception:
    LOGGER.warning(...)
    # Don't return None here — fall through to try the next extractor,
    # or return LLMUsageInfo(provider=extractor.PROVIDER, model=None, usage=None)
    return llm_usage.LLMUsageInfo(provider=extractor.PROVIDER)

Fix 2 — Guard all run_dict["extra"] direct accesses in openai_usage_extractor.py, groq_usage_extractor.py, anthropic_vertexai_usage_extractor.py, and _try_get_model_name in openai_usage_extractor.py:

# openai_usage_extractor.py — _try_get_model_name streaming branch:
elif generations := run_dict["outputs"].get("generations"):
    generation_info = (generations[-1][-1] if generations and generations[-1] else {}).get("generation_info") or {}
    model = generation_info.get("model_name", model)

# _get_provider:
if extra := run_dict.get("extra"):
    if base_url := extra.get("invocation_params", {}).get("base_url"):
        ...

# groq_usage_extractor.py:
model = run_dict.get("extra", {}).get("metadata", {}).get("ls_model_name")

Fix 3 — anthropic_usage_extractor.py: Broaden the caught exception type from KeyError to (KeyError, IndexError) in _try_get_model_name.

The core principle across all fixes: model-name resolution should never propagate an exception into get_llm_usage_info, and get_llm_usage_info should never propagate into try_extract_provider_usage_data. Usage data should always be returned even when model=None.

Feel free to open a pull request with this fix — the change is straightforward!

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik

[Harsh23Kashyap]

I'd like to work on this. The fix is surgical: harden the model-name/provider extraction helpers in the 6 extractors so they return None instead of raising on missing run_dict['extra'] / run_dict['outputs'] / empty generations, plus a defense-in-depth wrapper in get_llm_usage_info so a model-extraction failure never drops an already-extracted usage. Adding a unit test that asserts the partial-success case (usage present, model=None) is preserved end-to-end. PR coming up.