[FR]: [SDK] Add missing unit test coverage for IsJson metric

[hasnain1241]

Proposal summary

The IsJson heuristic metric (sdks/python/src/opik/evaluation/metrics/heuristics/is_json.py)
currently has no dedicated unit test coverage. Searching the test suite for IsJson /
is_json_metric returns no matches.

I'd like to add a dedicated test file, sdks/python/tests/unit/evaluation/metrics/test_is_json.py,
covering:

• Valid JSON inputs (object, array, and primitive types like "5", "null", "true",
  which are valid per json.loads but easy to overlook)
• Invalid JSON inputs (malformed JSON, empty string)
• Non-string inputs (None, int) — currently silently caught by the metric's blanket
  except Exception and scored 0.0, with no distinction from genuinely malformed JSON

This follows the same testing pattern already used for other heuristic metrics in
test_heuristics.py / test_sentiment.py (instantiate with track=False, assert against
the full ScoreResult object).

Motivation

What problem are you trying to solve?
IsJson is publicly exported (opik.evaluation.metrics.IsJson) and commonly used to
validate LLM output in structured extraction / tool-calling eval pipelines. Without tests,
its edge-case behavior is undocumented and unverified.

How are you currently solving this problem?
N/A — there is currently no test coverage for this metric at all.

What are the benefits of this feature?
Documents and locks in the expected behavior for edge cases (JSON primitives, malformed
input, non-string input), preventing silent regressions and clarifying current behavior
for future contributors.

I'm a first-time contributor and would like to open a PR for this if it's a welcome addition.

[github-actions]

I have all the information needed to write a thorough response. The reporter's analysis is accurate and the proposed approach aligns perfectly with the existing patterns.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing test coverage for IsJson — the reporter's analysis is confirmed. A search of the issue tracker returns no prior reports, and the metric is absent from both test_heuristics.py and all other test files in sdks/python/tests/unit/evaluation/metrics/. A dedicated test file is the right fix.

Code Investigation

Metric implementation: sdks/python/src/opik/evaluation/metrics/heuristics/is_json.py

The entire scoring logic is:

def score(self, output: str, **ignored_kwargs: Any) -> score_result.ScoreResult:
    try:
        json.loads(output)
        return score_result.ScoreResult(value=1.0, name=self.name)
    except Exception:
        return score_result.ScoreResult(value=0.0, name=self.name)

Key behavioral details to cover with tests:

• json.loads accepts primitives — "5", "true", "null" all return 1.0, which may surprise users.
• Non-string inputs (None, int) raise TypeError inside json.loads, which is silently swallowed by the bare except Exception, returning 0.0.
• An empty string "" also raises json.JSONDecodeError, returning 0.0.

Public export: IsJson is confirmed in sdks/python/src/opik/evaluation/metrics/__init__.py (line from .heuristics.is_json import IsJson) and listed in __all__.

Existing test pattern (from test_heuristics.py and test_sentiment.py):

# Instantiate with track=False, assert against full ScoreResult
metric = equals.Equals(case_sensitive=True, track=False)
assert metric.score(output=..., reference=...) == ScoreResult(
    name=metric.name, value=1.0, reason=None, metadata=None
)

Test gap: No test file exists at sdks/python/tests/unit/evaluation/metrics/test_is_json.py.

Next Steps

Create sdks/python/tests/unit/evaluation/metrics/test_is_json.py following the exact patterns in test_heuristics.py and test_sentiment.py. A sketch of what to include:

import pytest
from opik.evaluation.metrics import IsJson
from opik.evaluation.metrics.score_result import ScoreResult


@pytest.fixture
def metric():
    return IsJson(track=False)


# --- Valid JSON ---
@pytest.mark.parametrize("output", [
    '{"key": "value"}',           # object
    '[1, 2, 3]',                  # array
    '"a string"',                 # JSON string
    '5',                          # JSON number (easy to overlook)
    'true',                       # JSON boolean
    'null',                       # JSON null
    '{"nested": {"a": [1, 2]}}',  # nested structure
])
def test_is_json__valid_json__returns_1(metric, output):
    assert metric.score(output) == ScoreResult(name=metric.name, value=1.0, reason=None, metadata=None)


# --- Invalid JSON ---
@pytest.mark.parametrize("output", [
    "Not JSON",
    "{key: value}",   # unquoted keys
    "{'key': 'value'}",  # single quotes
    "",               # empty string
    "{",              # truncated
])
def test_is_json__invalid_json__returns_0(metric, output):
    assert metric.score(output) == ScoreResult(name=metric.name, value=0.0, reason=None, metadata=None)


# --- Non-string inputs (currently silently scored 0.0 via bare except) ---
@pytest.mark.parametrize("output", [None, 42, 3.14, [], {}])
def test_is_json__non_string_input__returns_0(metric, output):
    # Current behavior: bare `except Exception` catches TypeError → 0.0
    assert metric.score(output).value == 0.0

One design note worth raising in the PR: the bare except Exception in the implementation means type errors are indistinguishable from malformed JSON — a future improvement could raise MetricComputationError for non-string inputs (as Equals does), but that is a separate behavioral change and should not block adding tests for the current behavior.

Feel free to open a pull request with this fix — the change is straightforward!

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik

[Harsh23Kashyap]

I'd like to take this on. Adding a dedicated test file at sdks/python/tests/unit/evaluation/metrics/test_is_json.py covering the three buckets you outlined (valid JSON, invalid JSON, non-string inputs), following the existing track=False + full ScoreResult assertion pattern from test_heuristics.py. PR coming up.