check-values-enterprise validates only the multi-arch index, never the per-platform child manifests

[leiicamundi]

Description

scripts/check-values-enterprise.sh is the guardrail that asserts every image referenced by
values-enterprise.yaml is pullable from registry.camunda.cloud. It resolves availability with
docker manifest inspect (manifest_exists(), scripts/check-values-enterprise.sh:35-48), which
fetches the multi-arch index only. It never fetches the per-platform child manifests that the
index references.

That is precisely the failure mode vendor-ee/* has exhibited repeatedly since July (#6804): the
index resolves fine while the linux/amd64 child returns 404. The check is therefore structurally
incapable of detecting the only enterprise-image defect this repository has actually been hit by,
and it reports a green ✓ for images that cannot be pulled on the amd64 nodes CI runs on.

There is a second reason a naive fix will not work either: a registry-protocol GET self-heals on a
cache miss by re-proxying from upstream, so it returns 200 whether or not the local copy is intact.
camunda/infra-core#13884 hit exactly this while writing its own test and had to move the assertion
off the registry path onto Harbor's metadata API. Any child-manifest probe added here needs to
account for that, otherwise it will be green for the same wrong reason.

This issue is scoped to detection in this repository. The registry-side defect is #6804.

Expected vs Actual Behavior

Expected: validation fails when an image referenced by values-enterprise.yaml cannot be pulled
on linux/amd64.

Actual: validation passes as long as the index resolves, regardless of whether any child
manifest is fetchable.

Evidence, scheduled run 32005401695 on 2026-08-17:

✓ registry.camunda.cloud/vendor-ee/postgresql:15.18.0-debian-12-r17
✓ registry.camunda.cloud/vendor-ee/postgresql:14.23.0-debian-12-r19
✗ registry.camunda.cloud/vendor-ee/elasticsearch:8.19.20 (not resolvable)

Two days later those exact two images failed every 8.7 entv install with a 404 on the amd64
child manifest:

Failed to pull image "registry.camunda.cloud/vendor-ee/postgresql:15.18.0-debian-12-r17":
rpc error: code = NotFound desc = failed to pull and unpack image ...: failed to copy:
httpReadSeeker: failed open: content at
https://registry.camunda.cloud/v2/vendor-ee/postgresql/manifests/
sha256:1a9ef74da62314cdc642c7d9635f50c6a00fd96241172ed35c0b2867c4d51bb8
not found: not found

Four consecutive merge-queue runs, identical child digests, 12:38 → 16:43 UTC on 2026-08-19:

Run	Job	Started (UTC)
32250601958	96070402102	12:38
32257752735	96096088336	14:02
32265701668	96119912881	15:14
32272410706	96141556284	16:21

The next scheduled guardrail run, 32343262770
on 2026-08-20, reported success for all charts.

So the guardrail was green before, during and after a four-hour outage of two of the images it
exists to guard.

Steps to Reproduce

The blind spot is demonstrable without waiting for the broken state:

docker manifest inspect registry.camunda.cloud/vendor-ee/postgresql:15.18.0-debian-12-r17

This returns the index and exits 0. Nothing in that call dereferences .manifests[].digest.
manifest_exists() treats that exit code as proof of pullability, so any state in which the index
is present but a child is not is reported as ✓.

Contrast with what a pull actually does:

# what the script checks — the index
docker manifest inspect <image> | jq -r '.manifests[] | select(.platform.os=="linux") | "\(.platform.architecture) \(.digest)"'
# what the script never checks — each child, which is what containerd fetches next

Motivation / Use Case

entv is the only scenario exercising the enterprise-values overlay. When these images break they
break the merge queue, and the failure is attributed to whatever unrelated PR happens to be in it —
which is exactly what happened to #6522 in run 32272410706, a PR touching only
charts/camunda-platform-8.10/**.

The 07:00 cron exists so that this is caught a day early instead of in the merge queue. Today it
cannot be.

Note that reverting the pin is not the remedy — that was established in #6804 and is why #6803
was closed. The defect is not tied to a tag; any cached multi-arch index can enter this state.
Detection is the part that is missing.

Acceptance Criteria

• Validation resolves the index and asserts every referenced platform child manifest, at minimum
  linux/amd64 (the architecture CI and the large majority of customers run on).
• The assertion cannot be satisfied by a re-proxying registry GET; see camunda/infra-core#13884
  for why a plain /v2/.../manifests/<digest> read is not sufficient evidence.
• A missing amd64 child makes the run fail, verified against a deliberately constructed or
  known-bad reference rather than assumed.
• Failure output names the image, the platform and the child digest, so triage does not require
  re-deriving them from a kubelet event.
• Single-arch images and plain (non-index) manifests do not produce false positives.
• The existing behaviour of reporting all failing images per chart, rather than stopping at the
  first, is preserved.

Additional Context

Mechanism, for context only — the registry side is tracked in #6804 and by the infra team:
vendor-ee/* is served from a Harbor proxy-cache project whose retention policy runs every five
minutes. The per-platform children of a multi-arch index are untagged artifacts, and are eligible to
be reaped independently of the index that references them. Once the index is cached without its
child, Harbor short-circuits to the local copy and returns a hard MANIFEST_UNKNOWN instead of
re-proxying.

The upstream content was intact throughout: both tags are listed as present in the published feed
Renovate consumes (https://camunda.github.io/camunda-deployment-references/bitnami_postgresql.json),
so this was never an upstream deletion. vendor-ee/elasticsearch pulled normally in the same runs
(integration-elasticsearch-master-0 1/1 Running), and 8.9 entv, which pins different PostgreSQL
tags, passed at 13:24 and 14:45 UTC inside the same window. The failure is scoped to the cached
artifact, not to the cluster or the network — which is exactly the class of problem an index-only
probe cannot see.

References

• registry.camunda.cloud: vendor-ee proxy cache intermittently serves multi-arch indexes without their amd64 child manifest #6804 — registry-side defect, open, owned by infra
• QA gap: validate enterprise values image availability during chart release #6214 — the QA gap that this check was created to close
• [ENHACEMENT] Integration Test: Values Enterprise #6220 — the entv scenario
• Investigate entv scenario breakage (broken upstream enterprise images) and re-enable #6627 — earlier entv breakage, closed on the incorrect conclusion that upstream had fixed its images
• fix: revert vendor-ee/postgresql to -r2 to restore amd64 image pulls #6622, fix: revert vendor-ee/postgresql tag back to -r14 #6632, fix: revert vendor-ee/elasticsearch to 8.19.18 to restore amd64 image pulls #6803 — pin-revert attempts; fix: revert vendor-ee/elasticsearch to 8.19.18 to restore amd64 image pulls #6803 closed once the pin was shown to protect nothing
• camunda/infra-core#13884 — Harbor retention fix, and the source of the "a registry GET self-heals" finding
• scripts/check-values-enterprise.sh, .github/workflows/check-values-enterprise.yaml

Environment

• Platform: GKE, linux/amd64 nodes, distroci CI infrastructure
• Chart version: observed on 8.7; the check covers values-enterprise.yaml for 8.5 → 8.10
• Registry: registry.camunda.cloud, vendor-ee/*

Not specific to a Helm CLI or Kubernetes version.

[distro-ci]

🧭 Triage Assessment

Field	Value
Work type	BUG
Kind	bug
Severity	🔴 High
Likelihood	🔥 High
Confidence	0.97

📝 Rationale

Within SM Platform Distribution scope: the team boundaries assign ownership of Helm Charts, camunda/camunda-platform-helm, and automated scenario testing/deployments to Distribution. This is a BUG because the existing guardrail is intended to validate enterprise image pullability but only checks whether the manifest index resolves; the implementation does not dereference child manifests. That is incorrect existing behavior under the Distro definition of kind/bug. Severity is high: missing child manifests have already caused amd64 ImagePullBackOff failures, blocked enterprise integration installs and merge-queue runs, and affected multiple chart versions. The validator's false green result removes the intended early-detection control, while a durable workaround is not available because pin reverts do not prevent the same cache inconsistency on another tag. The high-severity classification aligns with noticeable production impact where no simple workaround exists. Likelihood is high because the same index-present/amd64-child-missing pattern recurred for PostgreSQL and Elasticsearch, with affected scope potentially broader than the observed incidents. The issue has sufficient triage information: clear problem statement, expected versus actual behavior, reproduction evidence, logs, environment, impact, and acceptance criteria. The closest issues are related but not strict duplicates: #6804 tracks the registry-side defect, #6622 and #6803 document prior image-pull incidents/workarounds, while #6214 and #6220 track the broader enterprise-values validation and integration-test gaps. The requested change is specifically the consumer-side child-manifest detection logic.

🔍 Similar Issues

#6804, #6622, #6803, #6214, #6220

---

🤖 This issue was automatically triaged by an AI-based system.
✋ If you believe this assessment is incorrect, update the labels or add the triage:manual label for human review.