[FR]: Add a provider-free OpenAI Agents 0.20/0.21 tracing regression

[FU-max-boop]

Proposal summary

Add an isolated Python 3.12 compatibility job that installs agentrunproof==0.2.0 with exact openai-agents==0.20.0 and ==0.21.0 cells.

A single test would drive the real public Runner through one local function tool with DeterministicModel(..., emit_traces=True), export through Opik's existing OpikTracingProcessor into the existing fake backend, and assert:

• one exact Task → Agent → (Turn → Generation + Tool, Turn → Generation) tree;
• two model calls and one tool invocation; and
• successful final output with the scripted model fully consumed.

The job would use no provider API key or model call. AgentRunProof would be test-only and isolated from Opik's runtime dependencies and existing unpinned/latest integration job.

Motivation

The current OpenAI Agents integration suite intentionally follows an unpinned latest SDK and calls the live OpenAI API. That is valuable coverage, but it does not provide a deterministic released-version boundary or a provider-free structural regression. PR #6320 documents a previous upstream tracing-shape change that broke this CI surface and required a large expectation update.

I prototyped the focused test against current main (73b2c959946e08bcab5996deec94eabef8c8464b):

• exact OpenAI Agents 0.20.0: 1 passed in 0.66s;
• exact OpenAI Agents 0.21.0: 1 passed in 0.74s;
• Ruff format/check, actionlint, zizmor high/offline, and git diff --check: all clean.

The existing job also remains clean when AgentRunProof is absent; the proposed exact cells would not constrain its latest-SDK resolution.

Would maintainers accept a small draft PR containing only this test and its isolated two-cell workflow job?

Disclosure: I maintain AgentRunProof. This proposal is deliberately limited to an optional test-only CI dependency; it does not add AgentRunProof to Opik's runtime dependencies.

[github-actions]

I now have a comprehensive picture. Let me write the response.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing solution or workaround found for provider-free pinned-version regression testing. The current test suite requires live API keys and an unpinned openai-agents version.

Code Investigation

The relevant files are:

• sdks/python/tests/library_integration/openai/agents_tests/test_opik_tracing_processor.py — the existing agents integration test. It uses real OpenAI API calls (MODEL_FOR_TESTS) and relies on ANY_BUT_NONE / ANY_LIST matchers rather than asserting exact structural shapes, precisely because it can't control the model's response.
• sdks/python/tests/library_integration/openai/requirements.txt — currently pins nothing: openai, openai-agents, litellm[google] — all unpinned.
• sdks/python/src/opik/integrations/openai/agents/opik_tracing_processor.py — the OpikTracingProcessor that maps openai-agents Trace/Span events to Opik spans. This is the code under test.
• sdks/python/src/opik/integrations/openai/agents/span_data_parsers.py — span parsers that specifically handle span types that changed in 0.14.0 (task, turn, TaskSpanData, TurnSpanData). This is the most fragile surface for version-to-version breakage, as documented by PR [NA] [SDK] fix: support openai-agents 0.14.0 TaskSpanData and TurnSpanData #6320.
• .github/workflows/lib-openai-tests.yml — the existing CI job. It uses OPENAI_API_KEY and OPENAI_ORG_ID secrets and calls the live API.

The proposal is technically sound in terms of problem statement: span_data_parsers.py uses openai_span_data.type == "task" and "turn" string duck-typing precisely because those types were added at 0.14.0 and the exact class names are unstable. A regression at 0.20.0 / 0.21.0 is plausible. The existing tests do not catch structural shape regressions deterministically.

However, the proposal introduces a meaningful concern: it asks Opik to take a test-only CI dependency on agentrunproof, a third-party package maintained by the issue reporter themselves. This is the key blocker, not the idea itself.

The existing test infrastructure already has a fake_backend fixture that captures all spans without a real server. The missing piece — a deterministic, provider-free model — could be implemented using the openai-agents SDK's own FakeModel or FakeLLM test utilities (available in the SDK's own test suite), or by mocking Runner at the ModelProvider level directly in pytest, without any external package.

Next Steps

The core idea (a pinned-version, provider-free structural regression test for OpikTracingProcessor) is valuable and well-motivated by the history documented in PR #6320. However, the team should not accept agentrunproof as a CI dependency, for these reasons:

1. External package risk: A test-only third-party package maintained by the FR author is a supply-chain concern and an ongoing maintenance burden. If agentrunproof drops support for a new openai-agents version, Opik's CI breaks with no clear ownership.
2. Unnecessary abstraction: The openai-agents SDK already exposes the internal test primitives (FakeModel, set_trace_processors, DeterministicModel patterns used in its own test suite) that are sufficient to write this test directly, without wrapping them in agentrunproof.
3. Isolation via a separate workflow cell is independently achievable by adding a second job to lib-openai-tests.yml (or a new lib-openai-pinned-tests.yml) with a requirements_pinned.txt that pins openai-agents==0.20.0 and ==0.21.0, and driving the test using only openai-agents-internal test utilities.

Recommended path for a PR:

• Add sdks/python/tests/library_integration/openai/agents_tests/test_opik_tracing_processor_deterministic.py using openai-agents' own FakeModel (or equivalent mock at the ModelProvider level) — no agentrunproof.
• Add sdks/python/tests/library_integration/openai/requirements_pinned.txt with pinned openai-agents==0.20.0 (and separately ==0.21.0 via matrix).
• Add a new workflow job (or extend lib-openai-tests.yml) that installs from requirements_pinned.txt and runs only the deterministic test, with no API key required.

This achieves the stated goal (pinned-version structural regression guard) without the third-party dependency risk.

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik

[FU-max-boop]

Thanks — the supply-chain/ownership concern is reasonable, and I will wait for a human maintainer's preference before opening any PR.

One version-boundary correction may help that decision: released openai-agents==0.20.0 has no packaged agents.testing module or public FakeModel; agents.testing.ScriptedModel first appears in 0.21.0. AgentRunProof's purpose in this proposal is therefore not to wrap an SDK helper that exists in both cells: its 0.20 path implements the public Model interface, while its 0.21 path delegates to the SDK's public ScriptedModel, preserving one fixture across that compatibility boundary.

The proposed dependency would be exact (agentrunproof==0.2.0), test-only, isolated to these two cells, and sourced from the immutable 0.2.0 release. That said, if maintainers prefer either a small vendored public-Model fixture or a 0.21-only test using the SDK helper, I will follow that direction and will not push the external dependency. I will not open a PR without explicit confirmation.