test(e2e): move flaky retry/matrix tests to no-ci temporarily by vdemeester · Pull Request #9242 · tektoncd/pipeline

vdemeester · 2025-12-17T09:28:19Z

Changes

Move two consistently flaky e2e tests to no-ci/ directory to unblock CI while we investigate and fix the root cause:

pipelinerun-with-matrix (examples/v1/pipelineruns/beta/pipelinerun-with-matrix.yaml)
- Times out at ~900 seconds on k8s-oldest + alpha features
- Multiple failures: Dec 16 (run 20265445080), Dec 15 (runs 20242616379, 20231100917)
- Suspected issue: matrix-with-task-retries task may have bug in context variable substitution
using-retries-and-retry-count-variables (examples/v1/pipelineruns/using-retries-and-retry-count-variables.yaml)
- Times out at ~900 seconds on k8s-oldest + alpha features
- Failure: Dec 16 (run 20276197820)
- Similar retry logic that may be affected by same root cause

Analysis

Both tests timeout at exactly 15 minutes (global test timeout in test/wait.go:64) when running on k8s-oldest with alpha features enabled. The common pattern:

✅ Pass on k8s-latest
❌ Timeout on k8s-oldest + alpha
Both involve retry logic with context variables
Timeout is consistent (~900-903 seconds)

Hypothesis: The retry logic with $(context.pipelineTask.retries) or $(context.task.retry-count) may not work correctly on older Kubernetes versions with alpha features, causing infinite retry loops until timeout.

Next Steps

Reproduce locally with k8s 1.28.0 + alpha features
Investigate context variable substitution with retries
Identify and fix root cause
Move tests back to their original locations
Verify tests run reliably in CI

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
Has Tests included if any functionality added or changed
pre-commit Passed
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

NONE

- Add startupProbe to accommodate slow Docker daemon initialization - Allow up to 30 seconds for daemon startup before marking as failed - Verify daemon functionality with 'docker info' instead of cert existence - Tries to address flaky test failures in CI with k8s native sidecar support Signed-off-by: Vincent Demeester <[email protected]>

Move two consistently flaky e2e tests to no-ci to unblock CI while we investigate and fix the root cause: 1. pipelinerun-with-matrix - Times out at 900s on k8s-oldest + alpha 2. using-retries-and-retry-count-variables - Times out at 900s on k8s-oldest + alpha Both tests timeout at exactly 15 minutes (global test timeout) when running on k8s-oldest with alpha features enabled. The issue appears to be related to retry logic with matrix expansion or context variable substitution. Related to tektoncd#9201, tektoncd#9062

vdemeester · 2025-12-17T09:30:15Z

/kind flake

afrittoli

/approve

tekton-robot · 2025-12-17T11:17:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afrittoli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [afrittoli]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

khrm

/lgtm

vdemeester · 2026-01-27T10:33:40Z

/cherry-pick release-v1.0.x

tekton-robot · 2026-01-27T10:37:01Z

✅ Cherry-pick to release-v1.0.x successful!

A new pull request has been created to cherry-pick this change to release-v1.0.x.

Please review and merge the cherry-pick PR.

vdemeester · 2026-01-28T09:47:36Z

/cherry-pick release-v1.0.x

tekton-robot · 2026-01-28T09:48:08Z

ℹ️ Cherry-pick to release-v1.0.x already exists!

A pull request for this cherry-pick already exists: #9315

PR: #9315

Add startupProbe with failureThreshold of 30 to allow more time for Docker daemon initialization. Change readinessProbe to use 'docker info' command which verifies the daemon is actually ready to accept commands. This is a cherry-pick of the probe changes from tektoncd#9242 on main.

Add startupProbe with failureThreshold of 30 to allow more time for Docker daemon initialization. Change readinessProbe to use 'docker info' command which verifies the daemon is actually ready to accept commands. This is a cherry-pick of the probe changes from #9242 on main.

vdemeester · 2026-01-30T15:18:26Z

/cherry-pick release-v1.0.x

tekton-robot · 2026-01-30T16:05:43Z

❌ Cherry-pick to release-v1.0.x failed!

The automatic cherry-pick to release-v1.0.x failed.

Output:

🤖 Starting cherry-pick process...
Fetching PR #9242 information...
Found merge commit: 57927d74a611802726dcc68505eb60ac5078291d
PR title: test(e2e): move flaky retry/matrix tests to no-ci temporarily
Fetching target branch: release-v1.0.x...
From https://github.com/tektoncd/pipeline
 * branch                release-v1.0.x -> FETCH_HEAD
Checking for existing cherry-pick PR...
Creating cherry-pick branch: cherry-pick-9242-to-release-v1.0.x...
Switched to a new branch 'cherry-pick-9242-to-release-v1.0.x'
branch 'cherry-pick-9242-to-release-v1.0.x' set up to track 'origin/release-v1.0.x'.
Fetching commits from PR #9242...
Found 2 commit(s) to cherry-pick
Cherry-picking commit 1/2: 209e1b55ebddad2e6728f31f40dc95f39aad2c48...
fatal: bad object 209e1b55ebddad2e6728f31f40dc95f39aad2c48
❌ ERROR: Cherry-pick failed for commit 209e1b55ebddad2e6728f31f40dc95f39aad2c48 due to conflicts or other errors

Next steps:

Check the action logs for complete details
If the PR is not merged, merge it first and try again
If there are conflicts, you'll need to manually cherry-pick this PR

vdemeester added 2 commits December 17, 2025 09:31

github-project-automation bot added this to Tekton Community Roadmap Dec 17, 2025

github-project-automation bot moved this to Todo in Tekton Community Roadmap Dec 17, 2025

tekton-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Dec 17, 2025

tekton-robot requested review from abayer and dibyom December 17, 2025 09:28

tekton-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Dec 17, 2025

vdemeester mentioned this pull request Dec 17, 2025

e2e: Re-enable and fix flaky pipelinerun-with-matrix example test #9201

Open

5 tasks

tekton-robot added release-note-none Denotes a PR that doesnt merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Dec 17, 2025

tekton-robot added the kind/flake Categorizes issue or PR as related to a flakey test label Dec 17, 2025

afrittoli reviewed Dec 17, 2025

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 17, 2025

khrm reviewed Dec 17, 2025

View reviewed changes

tekton-robot assigned khrm Dec 17, 2025

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 17, 2025

tekton-robot merged commit 57927d7 into tektoncd:main Dec 17, 2025
24 of 27 checks passed

github-project-automation bot moved this from Todo to Done in Tekton Community Roadmap Dec 17, 2025

vdemeester deleted the fix-matrix-timeout branch December 17, 2025 11:34

This was referenced Dec 17, 2025

ci: cherry pick CI fixes to 1.3.0 and 1.6.0 #9243

Closed

Backport test reliability fixes to release-v1.3.x #9250

Merged

Backport test reliability fixes to release-v1.6.x #9251

Merged

tekton-robot mentioned this pull request Jan 27, 2026

[cherry-pick: release-v1.0.x] test(e2e): move flaky retry/matrix tests to no-ci temporarily #9315

Closed

13 tasks

vdemeester mentioned this pull request Jan 29, 2026

[release-v1.0.x] build: bump Go version to 1.24.0 and fix e2e test infrastructure #9318

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(e2e): move flaky retry/matrix tests to no-ci temporarily#9242

test(e2e): move flaky retry/matrix tests to no-ci temporarily#9242
tekton-robot merged 2 commits intotektoncd:mainfrom
vdemeester:fix-matrix-timeout

vdemeester commented Dec 17, 2025 •

edited

Loading

Uh oh!

vdemeester commented Dec 17, 2025

Uh oh!

afrittoli left a comment

Uh oh!

tekton-robot commented Dec 17, 2025

Uh oh!

khrm left a comment

Uh oh!

Uh oh!

vdemeester commented Jan 27, 2026

Uh oh!

tekton-robot commented Jan 27, 2026

Uh oh!

vdemeester commented Jan 28, 2026

Uh oh!

tekton-robot commented Jan 28, 2026

Uh oh!

vdemeester commented Jan 30, 2026

Uh oh!

tekton-robot commented Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

vdemeester commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Analysis

Next Steps

Related

Submitter Checklist

Release Notes

Uh oh!

vdemeester commented Dec 17, 2025

Uh oh!

afrittoli left a comment

Choose a reason for hiding this comment

Uh oh!

tekton-robot commented Dec 17, 2025

Uh oh!

khrm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vdemeester commented Jan 27, 2026

Uh oh!

tekton-robot commented Jan 27, 2026

Uh oh!

vdemeester commented Jan 28, 2026

Uh oh!

tekton-robot commented Jan 28, 2026

Uh oh!

vdemeester commented Jan 30, 2026

Uh oh!

tekton-robot commented Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vdemeester commented Dec 17, 2025 •

edited

Loading