[FR] Token Usage for Eval Agents

[uttamdeb]

Currently, I am unable to find token usage details of the eval agents that run against a trace. I can find only the token usage usage details of my app's actual agent as belows. If I could get the details of the eval agent's token usage, then I could keep track of how much my eval agent is costing me.

{
  "completion_tokens": 389,
  "original_usage.completion_tokens": 389,
  "original_usage.completion_tokens_details.audio_tokens": 0,
  "original_usage.completion_tokens_details.reasoning_tokens": 192,
  "original_usage.prompt_tokens": 6472,
  "original_usage.prompt_tokens_details.audio_tokens": 0,
  "original_usage.prompt_tokens_details.cached_tokens": 1152,
  "original_usage.total_tokens": 6861,
  "prompt_tokens": 6472,
  "total_tokens": 6861
}

[alexkuzmik]

Hi @uttamdeb!

Are you using our opik.evaluate function for performing evaluations? If yes - then inside every evaluation trace there is a span "metrics_calculation" containing all the spans created during the task evaluation, including the LLM spans with their token usage information.

If you're not using opik.evaluate, I'd ask you to share more details because from your message it's not clear how exactly you're running evals and why you can't get token usage for the evals agent.

[uttamdeb]

I am using it via OpikTracer in LangChain. Stores traces with opik_trace_id. Fetches feedback_scores from traces (evaluation results). Source is "online_scoring": The evaluations are marked as coming from Opik's "online_scoring" system. But can't find "metrics_calculation" spans.

from opik.integrations.langchain import OpikTracer

[alexkuzmik]

@uttamdeb I see now, thank you. Let me discuss it with the team.