fix: harden eval integrity and release safety

Author: martinfrancois

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -24,7 +24,7 @@ body:
         - Skill behavior
         - README or docs
         - Evals or scoring
-        - Tessl install or tile metadata
+        - Tessl install or plugin metadata
         - Tessl publish or registry display
         - CI, release, or dependency automation
         - Not sure
@@ -78,17 +78,17 @@ body:
       description: Paste sanitized command output, eval output, CI links, or Tessl output if useful.
       render: text
   - type: input
-    id: tile-version
+    id: plugin-version
     attributes:
-      label: Tessl tile version
-      description: Paste the published tile version, `tessl` output, or the commit SHA if known.
+      label: Tessl plugin version
+      description: Paste the published plugin version, `tessl` output, or the commit SHA if known.
       placeholder: "martinfrancois/java-optionals version or commit SHA: "
   - type: input
     id: model
     attributes:
       label: Model
       description: For agent-output bugs, name the model that produced the behavior.
-      placeholder: "For example: Claude Opus 4.8 or GPT-5.5"
+      placeholder: "For example: model name and version"
   - type: dropdown
     id: reasoning-effort
     attributes:

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -19,7 +19,7 @@ body:
         - Optional example or reference material
         - Evals or scoring
         - README or docs
-        - Tessl install or tile metadata
+        - Tessl install or plugin metadata
         - CI, release, or dependency automation
         - Other
     validations:

.github/pull_request_template.md

@@ -46,14 +46,17 @@ Checks most contributors can run:
 
 - [ ] `python3 scripts/validate_skill.py skills/java-optionals`
 - [ ] `python3 scripts/validate_eval_criteria.py evals evals-reference`
+- [ ] `python3 -m py_compile scripts/validate_skill.py scripts/validate_eval_criteria.py`
+- [ ] `bash -n scripts/check_publish_dry_run.sh`
 - [ ] `tessl plugin lint .`
 - [ ] `markdownlint`, if Markdown changed
 - [ ] Manual rendered-doc or example review, if docs or examples changed
 
 Tessl-authenticated checks:
 
 - [ ] `bash scripts/check_publish_dry_run.sh .`
-- [ ] `tessl skill review --threshold 100 skills/java-optionals/SKILL.md`, if skill text or references changed
+- [ ] `tessl plugin publish --dry-run --bump patch .`
+- [ ] `tessl skill review --threshold 90 skills/java-optionals/SKILL.md`, if skill text or references changed
 - [ ] `tessl eval run --variant with-context --variant without-context .`, if skill behavior,
       evals, or benchmark claims changed
 
@@ -80,6 +83,22 @@ explain why.
 
 - [ ] Docs updated, or N/A
 - [ ] Evals updated, or N/A
+- [ ] Scenario directories include `task.md`, `criteria.json`, and `capability.txt`, or N/A
+- [ ] Scenario invocation style is classified as natural or explicit, or N/A
+- [ ] Natural activation prompts don't explicitly invoke the skill, or N/A
+- [ ] Explicit invocation prompts are labeled as explicit, or N/A
+- [ ] Headline criteria include compile/artifact checks, or N/A
+- [ ] Headline criteria include behavior correctness checks, or N/A
+- [ ] Runtime references contain no eval answer keys, scenario inventory, hosted run IDs, or fixed
+      score claims
+- [ ] Java baseline compatibility has been considered, or N/A
+- [ ] `OptionalInt`, `OptionalLong`, and `OptionalDouble` guidance has been considered, or N/A
+- [ ] Optional-producing stream terminals and collectors are covered, or N/A
+- [ ] Java 26 Javadocs were checked for Optional-family coverage, or N/A
+- [ ] Valid README package-runner instructions were preserved, or N/A
+- [ ] Tessl package commands match the verified plugin package format
+- [ ] Full/reference eval reporting is not hidden or cherry-picked
+- [ ] Tessl checks were run, or unavailability is documented
 - [ ] PR title or squash title uses Conventional Commits
 - [ ] Redaction checked: no Tessl tokens, GitHub tokens, package manager tokens, private repository
       links, private eval artifacts, private registry/workspace links, local host paths, or

.github/workflows/ci.yml

@@ -28,7 +28,7 @@ jobs:
       - name: Setup Tessl CLI
         uses: tesslio/setup-tessl@25ec223fc0da33b41b8044ff5ab2b85235f4f91e # v2
         with:
-          version: "0.81.1"
+          version: "0.81.2"
           token: ${{ env.TESSL_TOKEN }}
 
       - name: Validate skill metadata
@@ -37,13 +37,47 @@ jobs:
       - name: Validate eval criteria
         run: python3 scripts/validate_eval_criteria.py evals evals-reference
 
+      - name: Compile validation scripts
+        run: python3 -m py_compile scripts/validate_skill.py scripts/validate_eval_criteria.py
+
+      - name: Check shell scripts
+        run: bash -n scripts/check_publish_dry_run.sh
+
+      - name: Parse JSON files
+        run: |
+          python3 - <<'PY'
+          import json
+          import pathlib
+          for path in pathlib.Path('.').rglob('*.json'):
+              json.load(open(path, encoding='utf-8'))
+          print('JSON ok')
+          PY
+
+      - name: Parse YAML files
+        run: |
+          python3 - <<'PY'
+          import pathlib
+          try:
+              import yaml
+          except ImportError:
+              print('PyYAML unavailable; skipping YAML parse')
+              raise SystemExit(0)
+          for path in list(pathlib.Path('.').rglob('*.yml')) + list(pathlib.Path('.').rglob('*.yaml')):
+              yaml.safe_load(open(path, encoding='utf-8'))
+          print('YAML ok')
+          PY
+
       - name: Lint Tessl plugin
         run: tessl plugin lint .
 
-      - name: Check publish dry-run
+      - name: Check fast publish dry-run
         if: ${{ env.TESSL_TOKEN != '' }}
         run: bash scripts/check_publish_dry_run.sh .
 
+      - name: Check eval publish dry-run
+        if: ${{ env.TESSL_TOKEN != '' }}
+        run: tessl plugin publish --dry-run --bump patch .
+
       - name: Skip publish dry-run when Tessl token is unavailable
         if: ${{ env.TESSL_TOKEN == '' }}
-        run: echo "TESSL_TOKEN isn't configured; skipping publish dry-run."
+        run: echo "TESSL_TOKEN isn't configured; skipping Tessl publish dry-runs."

.github/workflows/commitlint.yml

@@ -22,7 +22,7 @@ permissions:
   pull-requests: read
 
 concurrency:
-  group: commitlint-${{ github.workflow }}-${{ github.event.pull_request.number }}
+  group: commitlint-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
 jobs:
@@ -33,7 +33,7 @@ jobs:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
         with:
           fetch-depth: 0
-          ref: ${{ github.event.pull_request.head.sha }}
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
 
       - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
         with:

.github/workflows/publish-tessl.yml

@@ -9,7 +9,7 @@ permissions:
   contents: read
 
 concurrency:
-  group: publish-tessl-${{ github.event.release.tag_name }}
+  group: publish-tessl-${{ github.event.release.tag_name || github.run_id }}
   cancel-in-progress: false
 
 jobs:
@@ -21,11 +21,27 @@ jobs:
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
 
+      - name: Require Tessl token
+        run: |
+          if [ -z "${TESSL_TOKEN:-}" ]; then
+            echo "TESSL_TOKEN is required to publish the Tessl plugin." >&2
+            exit 1
+          fi
+
       - name: Setup Tessl CLI
         uses: tesslio/setup-tessl@25ec223fc0da33b41b8044ff5ab2b85235f4f91e # v2
         with:
-          version: "0.81.1"
+          version: "0.81.2"
           token: ${{ env.TESSL_TOKEN }}
 
+      - name: Validate plugin before publish
+        run: |
+          python3 scripts/validate_skill.py skills/java-optionals
+          python3 scripts/validate_eval_criteria.py evals evals-reference
+          python3 -m py_compile scripts/validate_skill.py scripts/validate_eval_criteria.py
+          bash -n scripts/check_publish_dry_run.sh
+          tessl plugin lint .
+          tessl plugin publish --dry-run .
+
       - name: Publish plugin
         run: tessl plugin publish .

.github/workflows/release-please.yml

@@ -27,20 +27,6 @@ jobs:
           config-file: release-please-config.json
           manifest-file: .release-please-manifest.json
 
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
-        if: ${{ steps.release.outputs.release_created == 'true' }}
-
-      - name: Setup Tessl CLI for release publish
-        if: ${{ steps.release.outputs.release_created == 'true' }}
-        uses: tesslio/setup-tessl@25ec223fc0da33b41b8044ff5ab2b85235f4f91e # v2
-        with:
-          version: "0.81.1"
-          token: ${{ secrets.TESSL_TOKEN }}
-
-      - name: Publish Tessl plugin
-        if: ${{ steps.release.outputs.release_created == 'true' }}
-        run: tessl plugin publish .
-
       - name: Find release PR
         id: release-pr
         env:
@@ -78,7 +64,7 @@ jobs:
         if: ${{ steps.release-pr.outputs.found == 'true' }}
         uses: tesslio/setup-tessl@25ec223fc0da33b41b8044ff5ab2b85235f4f91e # v2
         with:
-          version: "0.81.1"
+          version: "0.81.2"
           token: ${{ secrets.TESSL_TOKEN }}
 
       - name: Validate release PR
@@ -109,10 +95,13 @@ jobs:
 
           python3 scripts/validate_skill.py skills/java-optionals
           python3 scripts/validate_eval_criteria.py evals evals-reference
+          python3 -m py_compile scripts/validate_skill.py scripts/validate_eval_criteria.py
+          bash -n scripts/check_publish_dry_run.sh
           tessl plugin lint .
 
           if [ -n "${TESSL_TOKEN:-}" ]; then
             bash scripts/check_publish_dry_run.sh .
+            tessl plugin publish --dry-run .
           else
             echo "TESSL_TOKEN isn't configured; skipping publish dry-run."
           fi

.github/workflows/skill-review.yml

@@ -9,8 +9,13 @@ on:
       - main
     paths:
       - ".github/workflows/skill-review.yml"
-      - "skills/java-optionals/SKILL.md"
-      - "skills/java-optionals/references/**"
+      - ".tessl-plugin/plugin.json"
+      - "tile.json"
+      - "tessl.json"
+      - "README.md"
+      - "CONTRIBUTING.md"
+      - "docs/agents/**"
+      - "skills/java-optionals/**"
 
 concurrency:
   group: skill-review-${{ github.workflow }}-${{ github.ref }}
@@ -33,9 +38,9 @@ jobs:
         if: ${{ env.TESSL_TOKEN != '' }}
         uses: tesslio/setup-tessl@25ec223fc0da33b41b8044ff5ab2b85235f4f91e # v2
         with:
-          version: "0.81.1"
+          version: "0.81.2"
           token: ${{ env.TESSL_TOKEN }}
 
       - name: Review skill
         if: ${{ env.TESSL_TOKEN != '' }}
-        run: tessl skill review --threshold 100 skills/java-optionals/SKILL.md
+        run: tessl skill review --threshold 90 skills/java-optionals/SKILL.md

.tessl-plugin/plugin.json

@@ -3,6 +3,9 @@
   "version": "0.1.24",
   "description": "Help AI coding agents use Java Optional well in new code and cleanups, without replacing one antipattern with another.",
   "summary": "Help AI coding agents use Java Optional well in new code and cleanups, without replacing one antipattern with another.",
+  "repository": "https://github.com/martinfrancois/java-optionals-skill",
+  "homepage": "https://github.com/martinfrancois/java-optionals-skill#readme",
+  "license": "MIT",
   "entrypoint": "README.md",
   "private": false
 }

CONTRIBUTING.md

@@ -35,10 +35,13 @@ For suspected vulnerabilities, don't open a public issue. Follow the private rep
 ├── skills/java-optionals/
 │   ├── SKILL.md
 │   ├── agents/openai.yaml
-│   ├── evals/evals.json
 │   └── references/
-│       ├── optional-examples.md
-│       └── source-notes.md
+│       ├── java-optional-api.md
+│       └── optional-examples.md
+├── docs/agents/
+│   ├── eval-case-inventory.md
+│   ├── legacy-evals.json
+│   └── source-notes.md
 ├── AI_CONTRIBUTION_POLICY.md
 ├── CODE_OF_CONDUCT.md
 ├── CONTRIBUTING.md
@@ -50,12 +53,14 @@ For suspected vulnerabilities, don't open a public issue. Follow the private rep
   requests.
 - `skills/java-optionals/SKILL.md` is the runtime instruction file loaded by agents.
 - `skills/java-optionals/agents/openai.yaml` provides display metadata.
-- `skills/java-optionals/references/optional-examples.md` contains larger examples and eval case
-  notes.
-- `skills/java-optionals/references/source-notes.md` records where the skill came from and why it
-  changed over time.
-- `evals/` contains the hosted Tessl implementation-regression benchmark used for the headline
-  README score.
+- `skills/java-optionals/references/optional-examples.md` contains runtime-safe examples.
+- `skills/java-optionals/references/java-optional-api.md` records Java 8 through Java 26
+  Optional-family compatibility guidance.
+- `docs/agents/eval-case-inventory.md`, `docs/agents/legacy-evals.json`, and
+  `docs/agents/source-notes.md` are maintainer-only material and must not be linked from runtime
+  skill references.
+- `evals/` contains the hosted Tessl implementation-regression benchmark used for headline
+  reporting.
 - `evals-reference/` keeps extra review and test scenarios that are useful during development but
   aren't part of the headline benchmark.
 - `scripts/` contains portable validation checks used by CI.
@@ -82,23 +87,32 @@ Run these before committing skill, eval, README, package, script, or CI changes:
 ```bash
 python3 scripts/validate_skill.py skills/java-optionals
 python3 scripts/validate_eval_criteria.py evals evals-reference
+python3 -m py_compile scripts/validate_skill.py scripts/validate_eval_criteria.py
+bash -n scripts/check_publish_dry_run.sh
 tessl plugin lint .
 ```
 
 If you change the skill text or reference files, also run:
 
 ```bash
-tessl skill review --threshold 100 skills/java-optionals/SKILL.md
+tessl skill review --threshold 90 skills/java-optionals/SKILL.md
 ```
 
+The threshold is intentionally below 100 so useful, specific guidance doesn't get removed only to
+make the review score look cleaner. Treat the review output as a quality signal and address valid
+feedback.
+
 If you have Tessl access, you can also run the publish dry-run:
 
 ```bash
 bash scripts/check_publish_dry_run.sh .
+tessl plugin publish --dry-run --bump patch .
 ```
 
-That dry-run may fail because the current version already exists in the registry. That's expected
-after a version has already been published; any other failure needs investigation.
+The script runs the fast skipped-eval package smoke check and retries with a patch bump if the
+current version already exists. The full `tessl plugin publish --dry-run --bump patch .` command is
+a PR-safe full eval-ingesting dry-run. Release publishing uses an exact-version dry-run before the
+real publish command.
 
 `tessl skill review`, `bash scripts/check_publish_dry_run.sh .`, and hosted evals require Tessl
 authentication. Hosted evals also require a linked Tessl project. If you don't have access, include
@@ -186,14 +200,18 @@ tessl eval run --variant with-context --variant without-context .
 If you don't have a Tessl workspace, that's fine. Open the pull request with the local check results,
 and a maintainer can run the hosted evals before release.
 
-The headline benchmark should stay focused on implementation tasks that mirror the motivating
-failures. Broad review and smoke scenarios can live in `evals-reference/` unless they're part of the
-headline measurement.
+The headline benchmark should stay focused on realistic tasks that mirror the motivating failures.
+It must include a documented mix of natural activation prompts and explicit invocation prompts.
+Natural scenarios must not mention `$java-optionals` or ask to use the skill. Explicit scenarios may
+name the skill and must be labeled as explicit in `criteria.json`.
+
+Every scenario directory must contain `task.md`, `criteria.json`, and `capability.txt`. Headline
+implementation criteria must include compile/artifact checks and behavior correctness checks before
+Optional style checks. Do not move or hide baseline-solved scenarios just to improve lift; keep broad
+coverage in `evals-reference/` and report it separately.
 
-The broader review scenarios in `evals-reference/` are useful while developing the skill, but many
-are small snippets that a strong generic model can already solve without the skill. Keep them as
-reference coverage unless they reveal a real implementation failure that belongs in the headline
-benchmark.
+Runtime skill references must not contain eval inventories, expected answers, score rubrics, hosted
+run IDs, or benchmark claims. Put maintainer-only eval history in `docs/agents/`.
 
 ## Benchmark Updates
 
@@ -203,7 +221,9 @@ When the hosted benchmark changes:
 - record the content commit;
 - update baseline and skill scores;
 - update lift, raw score ratio, and missed-point reduction;
-- keep the README wording clear about what the benchmark measures.
+- report natural activation, explicit invocation, headline combined, and reference/full results
+  separately when available;
+- keep the README wording clear about what the benchmark measures and avoid stale fixed claims.
 
 ## Release Checklist