[SDK] fix: keep LangChain usage extraction when model metadata is absent - #7861
Open
feiiiiii5 wants to merge 3 commits into
Open
[SDK] fix: keep LangChain usage extraction when model metadata is absent#7861feiiiiii5 wants to merge 3 commits into
feiiiiii5 wants to merge 3 commits into
Conversation
try_extract_provider_usage_data discards an already-extracted usage payload whenever the model-name or provider lookup raises (unguarded run_dict["extra"] / generations[-1][-1]["generation_info"] accesses), so spans lose their usage and cost when model metadata is missing. Guard the lookups so usage survives with model=None. Resolves comet-ml#7860 Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
github-actions
Bot
added
python
feiiiiii5
marked this pull request as ready for review
August 16, 2026 07:53
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
Follow-up from review: explicit None in extra.metadata or extra.invocation_params still aborted usage extraction (AttributeError on the None result, or is_provider_run returning False when try_get_ls_metadata raised on a missing extra). Normalize non-mapping metadata/invocation_params to absent and guard the shared ls-metadata helper, so usage survives with model=None in every shape. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
Consolidate the three OpenAI resilience tests into a single pytest.mark.parametrize case named test_try_extract_provider_usage_data__missing_model_metadata__returns_usage_with_null_model and annotate the fixture builders as Dict[str, Any], per review. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details
try_extract_provider_usage_datadiscards an already-extracted usage payload whenever the model-name or provider lookup raises: the orchestrator catches the exception and returnsNone, so the span is emitted with no usage/cost even though the tokens were extractable. The lookups assume keys that are not guaranteed by LangChain run dicts (extra,outputs.generations[-1][-1].generation_info).Change: guard every model-name/provider lookup across the six provider extractors so missing keys resolve to
Noneinstead of raising; usage is returned withmodel=Nonewhen model metadata is absent.Issues
Resolves #7860
Testing
tests/unit/integrations/langchain/provider_usage_extractors/test_model_name_resilience.py: streaming-shape OpenAI run withoutgeneration_infoand invoke-shape run withoutextraboth return usage (prompt_tokens=10) withmodel=None. Pre-fix both fail (usage dropped); post-fix 2/2 pass.pytest tests/unit/llm_usage— 65 passed.ruff checkandruff formatclean on all touched files.Documentation
No docs change: behavior change is internal to usage extraction.