Integration tests gate retry can re-run e2e jobs without their install job, masking the real failure with 'namespace not found'

[eamonnmoloney]

Description

When the integration tests gate retries a failed matrix run, the retried e2e jobs can run without their install job re-executing. The namespace created by the original install has already been cleaned up by then, so every retried e2e job fails with:

Error: namespace 'camunda-pr-<pr>-intg-<version>-gke-<flow>-<hash>' not found in the current Kubernetes context

This is worse than a plain retry failure: it replaces the original error with a misleading one. The real cause is no longer visible in the final check output, and anyone reading the PR sees a Kubernetes namespace problem that does not exist.

Observed on PR #6823, run 32011352361.

Expected vs Actual Behavior

Expected: a gate retry either re-establishes the preconditions the retried jobs need (i.e. re-runs install), or does not retry jobs whose preconditions are gone.

Actual: the e2e job re-runs alone against a deleted namespace and fails with an unrelated error, and the gate reports that second failure as the run's conclusion.

Steps to Reproduce

1. Get a transient infrastructure failure in a Playwright e2e smoke after install job — in the observed case, codeload.github.com returned HTTP 429 while downloading docker/setup-buildx-action@bb05f3f, failing 2× Local cluster - KIND and 2× Playwright e2e smoke ... eske (8.8 and 8.9).
2. Let the gate detect the failure and trigger its one-shot retry (gate.go:221, triggering retry of failed jobs).
3. Observe the retried e2e job fail with namespace ... not found.

Additional Context

Evidence from run 32011352361:

Attempt	Trigger	install job	e2e outcome
1	original	ran 08:39:57→08:44:18	failed — codeload 429 (real cause)
2	gate retry	not re-executed — record still shows started_at 08:39:57	failed — namespace ... not found
3	manual gh run rerun <id>	re-executed 09:29:45	passed

Attempt 1's failure was purely transient:

##[error]Response status code does not indicate success: 429 (Too Many Requests).
##[error]Failed to download archive '.../docker/setup-buildx-action/tar.gz/bb05f3f...' after 3 attempts.

Cleanup is not the culprit in an obvious way — the Cleanup job for the affected flow ran at 08:59:57, after the attempt-2 e2e failure at 08:58:37. The namespace was already gone from attempt 1's own cleanup.

Namespace collision across runs is ruled out: ciworkflow/workflowvars.go:175 hashes namespace + RunID, so namespaces are per-run.

Unexplained discrepancy — worth confirming before choosing a fix. The gate invokes gh run rerun <runID> with no --failed flag (gh.go:103-105), which should re-run all jobs. That is the same command used manually for attempt 3, where install did re-execute. Yet in attempt 2 install retained its attempt-1 started_at. Same command, different observed behaviour. I have not been able to explain this from the run data alone, so the mechanism should be confirmed rather than assumed.

Possible directions (not a recommendation until the above is understood):

• Ensure the retry re-runs the install job for any flow whose e2e jobs are being retried.
• Defer namespace cleanup until the gate has reached a final conclusion, so a retry still has its namespace.
• Fail fast with a clear message when a retried e2e job finds its namespace missing, so the misleading error never becomes the reported conclusion.

The comment at workflowvars.go:171-173 states the run attempt is deliberately excluded from the namespace hash so that "re-run failed jobs" computes the same namespace as the original install. That intent only holds while the namespace still exists; this issue is the case where it does not.

Acceptance Criteria

• The mechanism behind the attempt-2 vs attempt-3 difference is confirmed.
• A gate retry of an e2e job either has a valid namespace or fails with an error naming the real cause.
• The original failure remains visible in the gate's reported conclusion rather than being masked by namespace ... not found.
• Covered by a test in scripts/integration-tests-gate/gate_test.go.

References

• PR ci: harden checkout credential handling for INC-7027 #6823 — where this was observed
• Run 32011352361 (attempts 1–3)
• scripts/integration-tests-gate/gate.go:221, scripts/integration-tests-gate/gh.go:103
• scripts/camunda-core/pkg/ciworkflow/workflowvars.go:165-195

Environment

• Platform: GKE

[distro-ci]

🧭 Triage Assessment

Field	Value
Work type	BUG
Kind	bug
Severity	⚠️ Mid
Likelihood	🔶 Mid
Confidence	0.9

📝 Rationale

Within SM Platform/Distribution scope: team boundaries assign the Helm Charts repository and automated scenario testing/deployments to this team. This is a BUG because existing CI retry behavior is not working as intended: a retried e2e job can lose its namespace and mask the original failure. Severity is mid: it can block or mislead PR validation and requires manual intervention, but it has no customer production impact, data loss, or security impact, and a manual rerun is available. Likelihood is mid: the failure is reproducible under transient integration-test failures, but only one concrete occurrence is provided. The report is sufficiently detailed for triage; the attempt-2 versus attempt-3 mechanism is an investigation acceptance criterion, not a blocker. The closest related item is #6063, which fixed a different namespace mismatch caused by including run_attempt in the hash; it is not a strict duplicate. #6441 introduced the one-shot failed-job gate retry, while #6643 concerns a separate merge-queue race. Current repository code documents/use gh run rerun --failed, which should be confirmed against the affected historical run because it differs from part of the issue description.

🔍 Similar Issues

#6063, #6441, #6643

---

🤖 This issue was automatically triaged by an AI-based system.
✋ If you believe this assessment is incorrect, update the labels or add the triage:manual label for human review.