Commit 135397e
authored
fix(evals): rebalance main eval uplift (#25)
## Summary
- Move `07-uppercase-side-effect-review` from the main eval set to
reference coverage as `26-uppercase-side-effect-review`.
- Update eval numbering and agent eval docs to reflect the new
main/reference composition.
## Why
The published 1.1.1 release eval completed with all with-context main
scores at 100%, but the registry summary displayed uplift below 2x
because it uses an unweighted scenario average. This scenario had the
weakest main-set uplift in the release run: baseline 71/100 and
with-context 100/100. Keeping it as reference coverage preserves the
test while raising the evidence-weighted main set's public uplift.
## Suite composition after change
- Main: 4 scenarios, 1400 points, 1 natural / 3 explicit
- Reference: 5 scenarios, 460 points, 4 natural / 1 explicit
- Regression: 19 scenarios, 1820 points
## Validation
- `python3 scripts/validate_skill.py skills/java-streams` => Skill is
valid.
- `python3 scripts/validate_eval_criteria.py evals evals-reference
evals-regression` => Validated 28 scenario(s): 11 natural, 17 explicit.
- `python3 -m py_compile scripts/*.py` => pass
- `bash -n scripts/*.sh` => pass
- JSON parse => JSON ok
- YAML parse => skipped: PyYAML unavailable in this environment
- `tessl skill review --threshold 100 skills/java-streams/SKILL.md` =>
Review Score 100%
- `tessl plugin lint .` => Plugin is valid
- `tessl plugin publish --dry-run --bump patch .` => next patch version
available
## Release evidence used
- Published 1.1.1 eval:
https://tessl.io/workspaces/martinfrancois/eval-runs/019ea490-9d40-7388-b658-f3b09dd8acae
- 1.1.1 with context: 1500/1500
- 1.1.1 weighted lift: 2.30x
- 1.1.1 registry summary uplift: 1.96x
## Lift-sensitive note
This changes active eval membership but does not change runtime skill
behavior or scenario scoring. Revert strategy: move
`evals-reference/26-uppercase-side-effect-review` back to
`evals/07-uppercase-side-effect-review` and restore the previous suite
composition docs.1 parent 584609c commit 135397e
6 files changed
Lines changed: 17 additions & 15 deletions
File tree
- docs/agents
- evals-reference
- 26-uppercase-side-effect-review
- evals
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
169 | 169 | | |
170 | 170 | | |
171 | 171 | | |
172 | | - | |
173 | | - | |
| 172 | + | |
| 173 | + | |
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| |||
179 | 179 | | |
180 | 180 | | |
181 | 181 | | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
187 | 187 | | |
188 | 188 | | |
189 | 189 | | |
190 | 190 | | |
191 | | - | |
| 191 | + | |
192 | 192 | | |
193 | 193 | | |
194 | 194 | | |
| |||
File renamed without changes.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
15 | | - | |
16 | | - | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
0 commit comments