Problem
Nothing asserts that the helm binary on PATH matches the version pinned in
.tool-versions. Every golden-regeneration target shells out to whichever helm
resolves first, so a contributor with a different Helm silently produces goldens
that disagree with CI — and the output looks entirely plausible.
Affected targets (Makefile):
go.update-golden-only (L123) and go.update-golden-only-lite (L109)go.update-registry-golden (L117)go.test (L88), go.test-golden-updated (L96), go.update-golden-only-cleanup (L100)helm.schema-update (L277) and precommit.chores (L381), which calls the -lite variant
tools.asdf-install installs the pinned version, but installing it is not the
same as using it: a Homebrew (or other) helm earlier on PATH wins over the
asdf shim, and asdf gives no warning.
How this surfaced
.tool-versions pins helm 4.2.4. /opt/homebrew/bin/helm was 4.2.3 and
shadowed the asdf shim. Helm 4.2.3 drops a trailing-whitespace line at the end of
the connectors application.yaml block that 4.2.4 emits, so
TestGoldenDefaultsTemplateConnectors failed locally. Regenerating with
make go.update-golden-only "fixed" the test by deleting a line CI legitimately
renders.
The failure mode is what makes this worth fixing: the local test went green, the
diff was one plausible line in a generated file, and there was no signal pointing
at the toolchain. It took a distro-ci[bot] chores commit re-adding the line —
reducing the PR to changed_files=0 — to expose it. See #6901, closed as invalid.
A patch-level Helm difference was enough. This is not an exotic mismatch.
Suggested fix
A .helm-version-check prerequisite on the targets above: compare
helm version --short against the helm line in .tool-versions and fail with
the two versions and a pointer to make tools.asdf-install when they differ.
Two details worth deciding deliberately:
- Fail vs warn. Fail for the golden/schema-regeneration targets, where wrong
output gets committed. go.test could arguably warn instead, so a contributor
on a mismatched Helm can still run tests — though a mismatch there produces
false failures, which is how this started.
- Where the pin is read. Parse
.tool-versions rather than hardcoding, so the
check cannot drift from the pin it is enforcing.
Worth checking whether the same gap applies to the other pinned tools the
generation targets depend on (yq, jq, go).
Problem
Nothing asserts that the
helmbinary onPATHmatches the version pinned in.tool-versions. Every golden-regeneration target shells out to whicheverhelmresolves first, so a contributor with a different Helm silently produces goldens
that disagree with CI — and the output looks entirely plausible.
Affected targets (
Makefile):go.update-golden-only(L123) andgo.update-golden-only-lite(L109)go.update-registry-golden(L117)go.test(L88),go.test-golden-updated(L96),go.update-golden-only-cleanup(L100)helm.schema-update(L277) andprecommit.chores(L381), which calls the-litevarianttools.asdf-installinstalls the pinned version, but installing it is not thesame as using it: a Homebrew (or other)
helmearlier onPATHwins over theasdf shim, and asdf gives no warning.
How this surfaced
.tool-versionspins helm 4.2.4./opt/homebrew/bin/helmwas 4.2.3 andshadowed the asdf shim. Helm 4.2.3 drops a trailing-whitespace line at the end of
the connectors
application.yamlblock that 4.2.4 emits, soTestGoldenDefaultsTemplateConnectorsfailed locally. Regenerating withmake go.update-golden-only"fixed" the test by deleting a line CI legitimatelyrenders.
The failure mode is what makes this worth fixing: the local test went green, the
diff was one plausible line in a generated file, and there was no signal pointing
at the toolchain. It took a
distro-ci[bot]chores commit re-adding the line —reducing the PR to
changed_files=0— to expose it. See #6901, closed as invalid.A patch-level Helm difference was enough. This is not an exotic mismatch.
Suggested fix
A
.helm-version-checkprerequisite on the targets above: comparehelm version --shortagainst thehelmline in.tool-versionsand fail withthe two versions and a pointer to
make tools.asdf-installwhen they differ.Two details worth deciding deliberately:
output gets committed.
go.testcould arguably warn instead, so a contributoron a mismatched Helm can still run tests — though a mismatch there produces
false failures, which is how this started.
.tool-versionsrather than hardcoding, so thecheck cannot drift from the pin it is enforcing.
Worth checking whether the same gap applies to the other pinned tools the
generation targets depend on (
yq,jq,go).