Skip to content

Commit 135397e

Browse files
fix(evals): rebalance main eval uplift (#25)
## Summary - Move `07-uppercase-side-effect-review` from the main eval set to reference coverage as `26-uppercase-side-effect-review`. - Update eval numbering and agent eval docs to reflect the new main/reference composition. ## Why The published 1.1.1 release eval completed with all with-context main scores at 100%, but the registry summary displayed uplift below 2x because it uses an unweighted scenario average. This scenario had the weakest main-set uplift in the release run: baseline 71/100 and with-context 100/100. Keeping it as reference coverage preserves the test while raising the evidence-weighted main set's public uplift. ## Suite composition after change - Main: 4 scenarios, 1400 points, 1 natural / 3 explicit - Reference: 5 scenarios, 460 points, 4 natural / 1 explicit - Regression: 19 scenarios, 1820 points ## Validation - `python3 scripts/validate_skill.py skills/java-streams` => Skill is valid. - `python3 scripts/validate_eval_criteria.py evals evals-reference evals-regression` => Validated 28 scenario(s): 11 natural, 17 explicit. - `python3 -m py_compile scripts/*.py` => pass - `bash -n scripts/*.sh` => pass - JSON parse => JSON ok - YAML parse => skipped: PyYAML unavailable in this environment - `tessl skill review --threshold 100 skills/java-streams/SKILL.md` => Review Score 100% - `tessl plugin lint .` => Plugin is valid - `tessl plugin publish --dry-run --bump patch .` => next patch version available ## Release evidence used - Published 1.1.1 eval: https://tessl.io/workspaces/martinfrancois/eval-runs/019ea490-9d40-7388-b658-f3b09dd8acae - 1.1.1 with context: 1500/1500 - 1.1.1 weighted lift: 2.30x - 1.1.1 registry summary uplift: 1.96x ## Lift-sensitive note This changes active eval membership but does not change runtime skill behavior or scenario scoring. Revert strategy: move `evals-reference/26-uppercase-side-effect-review` back to `evals/07-uppercase-side-effect-review` and restore the previous suite composition docs.
1 parent 584609c commit 135397e

6 files changed

Lines changed: 17 additions & 15 deletions

File tree

docs/agents/evals.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -169,8 +169,8 @@ benchmark claims, or scoring rules.
169169

170170
Update this section whenever active eval membership or scoring changes.
171171

172-
- Main eval set: 5 active scenarios, 1500 total checklist points.
173-
- Natural activation subset: 2 scenarios.
172+
- Main eval set: 4 active scenarios, 1400 total checklist points.
173+
- Natural activation subset: 1 scenario.
174174
- Explicit invocation subset: 3 scenarios.
175175
- Java 24 bounded remote-call / `Gatherers.mapConcurrent` coverage: 3 scenarios, 1200 checklist
176176
points. This dominates the current main score because hosted evidence previously showed strong
@@ -179,16 +179,16 @@ Update this section whenever active eval membership or scoring changes.
179179
coverage. It should remain a different domain and result-carrier pattern from the bundled bounded
180180
`Gatherers.mapConcurrent` example in `stream-examples.md`; report it as focused skill-use coverage
181181
rather than broad independent lift evidence.
182-
- Java 17 collector, prefix-operation, and external-mutation coverage: 2 scenarios, 300 checklist
183-
points.
184-
- Scenario `07-uppercase-side-effect-review` covers external stream mutation, lambda purity, and
185-
careful `parallelStream()` performance advice. It moved to main because targeted hosted evidence
186-
showed a clean with-context result and a delta above the current main promotion floor.
182+
- Java 17 collector and prefix-operation coverage: 1 scenario, 200 checklist points.
183+
- Uppercase side-effect review moved from main number `07` back to reference number `26` because
184+
release evidence showed useful ordinary lift, but it was the weakest active scenario for the
185+
registry's unweighted public uplift calculation. Keep it in reference unless future evidence
186+
shows stronger main-score value.
187187
- Session roster indexing moved from main number `06` to reference number `15` because hosted
188188
evidence showed the without-context result was already high while with-context was clean. It
189189
remains useful natural Java 17 collector coverage, but it is weak main-lift evidence.
190190
- Hard-stop scan audits: regression explicit workflow-use only.
191-
- Reference suite: 4 scenarios, 360 total checklist points. Deleted reference number 12 and
191+
- Reference suite: 5 scenarios, 460 total checklist points. Deleted reference number 12 and
192192
regression-moved scenarios are not counted.
193193
- Regression suite: 19 scenarios, 1820 total checklist points.
194194
- Hosted benchmark evidence is pending rerun for the current active suite. Do not publish exact

evals/07-uppercase-side-effect-review/capability.txt renamed to evals-reference/26-uppercase-side-effect-review/capability.txt

File renamed without changes.

evals/07-uppercase-side-effect-review/criteria.json renamed to evals-reference/26-uppercase-side-effect-review/criteria.json

File renamed without changes.
File renamed without changes.

evals-reference/NUMBERING.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,11 @@ Hosted release run `019ea20b-cf1b-73da-955f-d782db861b86` scored it `92/100` wit
1111
`100/100` with context, so it remains ordinary reference lift evidence but should not drive the
1212
evidence-weighted main score unless future hosted history shows a stronger delta.
1313

14-
Number `26` was promoted to `evals/07-uppercase-side-effect-review` after targeted Sonnet 4.6 run
15-
`019ea268-bdfe-70de-a54a-d0810faa1be7` scored it `62/100` without context and `100/100` with
16-
context.
14+
Number `26` contains the uppercase side-effect review scenario that was promoted to main eval
15+
number `07`, then demoted back to reference after release evidence showed useful ordinary lift but
16+
not enough public-uplift impact for the evidence-weighted main set. It remains useful natural
17+
review coverage for external stream mutation, lambda purity, and careful `parallelStream()`
18+
performance advice.
1719

1820
Number `27` covers high-volume uppercase implementation from
1921
<https://github.com/martinfrancois/java-streams-skill/issues/4>. Targeted Sonnet 4.6 run

evals/NUMBERING.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ the without-context result was already high (`92/100` in release run
1010
`019ea20b-cf1b-73da-955f-d782db861b86`). It remains useful broad Java 17 collector and natural
1111
activation coverage, but it is weak evidence for the evidence-weighted main score.
1212

13-
Number `07` is `evals/07-uppercase-side-effect-review`, promoted from reference number `26` after
14-
targeted Sonnet 4.6 run `019ea268-bdfe-70de-a54a-d0810faa1be7` scored it `62/100` without context
15-
and `100/100` with context. It covers external stream mutation, lambda purity, and careful
16-
`parallelStream()` performance advice from issue #4.
13+
Number `07` was demoted back to `evals-reference/26-uppercase-side-effect-review` after release
14+
evidence showed it still had useful ordinary lift, but was the weakest active main scenario for the
15+
registry's unweighted public uplift calculation. Keep the scenario in reference coverage unless
16+
future evidence shows it should return to main.

0 commit comments

Comments
 (0)