Proposal: test: implement parallel/serial test categorization system by vdemeester · Pull Request #9224 · tektoncd/pipeline

vdemeester · 2025-12-09T14:09:59Z

Changes

Add a test categorization system that enables selective execution of e2e tests based on whether they can safely run in parallel or must run serially. This optimizes test execution time while preserving safety for tests that modify shared cluster state.

Uses comment annotations to mark tests:

@test:execution=parallel - Safe for concurrent execution
@test:execution=serial - Must run sequentially (modifies ConfigMaps)
@test:reason - Documents why serial execution is required

The TestMain function in init_test.go parses these annotations using Go's AST parser and orchestrates test execution:

-category=parallel - Run only parallel tests
-category=serial - Run only serial tests
-category=all - Run serial first, then parallel (with fail-fast)
-show-tests - Display test categorization
Faster local development: run parallel tests concurrently
CI optimization: separate parallel and serial test jobs
Safety: prevents test interference from ConfigMap modifications
Self-documenting: annotations explain execution requirements

Actions

Does this approach make sense to you @tektoncd/core-maintainers ?
Should we consider unannotated test and parallel the same, and thus only require an annotation for parallel test
Should I split this ? The most interested changes is in the init_test.go file - I could split that change from the one that adds the annotations

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
Has Tests included if any functionality added or changed
pre-commit Passed
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

NONE

vdemeester · 2025-12-09T14:16:20Z

/kind misc

vdemeester · 2025-12-10T09:58:59Z

/cherry-pick foo

tekton-robot · 2025-12-10T09:59:35Z

❌ Cherry-pick to foo failed!

The automatic cherry-pick to foo failed.

Output:

🤖 Starting cherry-pick process...
Fetching PR #9224 information...
❌ ERROR: PR #9224 is not merged yet (state: OPEN). Cherry-pick requires merged PRs.

Next steps:

Check the action logs for complete details
If the PR is not merged, merge it first and try again
If there are conflicts, you'll need to manually cherry-pick this PR

afrittoli · 2025-12-10T09:59:49Z

Thanks for this. I think that having @test:execution=parallel as default probably makes sense, but I don't mind the extra annotations, they mean that someone took the time to verify that a test can be executed in parallel.

vdemeester · 2025-12-10T09:59:55Z

Alright, the cherry-pick command is back !

vdemeester · 2025-12-10T12:12:19Z

Thanks for this. I think that having @test:execution=parallel as default probably makes sense, but I don't mind the extra annotations, they mean that someone took the time to verify that a test can be executed in parallel.

That's kind of my approach on this. I am even willing to make the CI fails if there is an unannotated test (to make sure we keep it organized).

twoGiants · 2025-12-16T16:03:06Z

That's kind of my approach on this. I am even willing to make the CI fails if there is an unannotated test (to make sure we keep it organized).

@vdemeester Yes, just wanted to mention this and then saw your comment. It must be consistent, if the annotations are introduced, they are supposed to be used on every test.

Can be a separate follow up PR though.

twoGiants

Thank you for this great improvement! I bet the e2e tests are less flaky and way faster now 🥇

I have a few suggestions below:

extract filtering utils
add unit tests
drop Unknown and maybe tags

twoGiants · 2025-12-16T16:06:45Z

test/init_test.go

+		filterTests(manifest.Parallel)
+		fmt.Fprintf(os.Stderr, "Running %d parallel tests\n", len(manifest.Parallel))
+		if len(manifest.Unknown) > 0 {
+			fmt.Fprintf(os.Stderr, "WARNING: %d test(s) without @test:execution annotation\n", len(manifest.Unknown))


I would fail (but earlier in scan test annotations) when Unknown > 0. Same below.

In general I would not allow Unknown and update that below everywhere. 😸

twoGiants · 2025-12-16T16:19:42Z

test/init_test.go

+}
+
+// scanTestAnnotations parses all *_test.go files and extracts test metadata
+func scanTestAnnotations(dir string) (*TestManifest, error) {


This is a nice little test utility library you wrote which could benefit from some unit (or better integration) tests itself. I would not leave it here but only use here. Put it into /test/annotations or /test/categorization. The functions are independent of *testing.M, which is great. Put also the structs there TestInfo and TestManifest.

I would also add some unit(or actual integration) tests without abstracting the filesystem. It shouldn't be to many and should run quick.

twoGiants · 2025-12-16T16:25:47Z

test/init_test.go

+		switch key {
+		case "execution":
+			execution = value
+		case "reason":


Maybe you want to enforce reason if execution is serial?

twoGiants · 2025-12-16T17:40:32Z

test/init_test.go

+	}
+	fmt.Fprintf(os.Stdout, "\n")
+
+	fmt.Fprintf(os.Stdout, "Serial Tests (%d):\n", len(manifest.Serial))


If serial tests are run first, maybe list them first?

twoGiants · 2025-12-16T17:44:46Z

test/larger_results_sidecar_logs_test.go


+// @test:execution=serial
+// @test:reason=modifies results-from field in feature-flags ConfigMap
+// @test:tags=results,sidecar,featureflags,stateful


Not sure about the tags. What is the specific use case for them?

And if we use them, I'd prefer to have to pick them from an enum or there could be to many similar ones at some point => featureflags,feature-flags,feature_flags, ...

twoGiants · 2025-12-16T17:48:41Z

test/README.md

-Just note that the build tags should be `-tags=featureflags`.
+Just note that the build tags should be `-tags=featureflags`.
+
+## Test Categorization: Parallel vs Serial Execution


This document should be linked somewhere more visible. Here in the main DEVELOPMENT.md the link to the testing docs README.md is here under point number 5: https://github.com/tektoncd/pipeline/blob/main/DEVELOPMENT.md#developing-and-testing

Very difficult to find. 🙈

I would change it up a bit. Remove point 5 and add a subsection under: https://github.com/tektoncd/pipeline/blob/main/DEVELOPMENT.md#iterating-on-code-changes => named Testing and there I would put the link to the actual testing doc. Something like that.

tekton-robot · 2025-12-16T17:58:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: twoGiants

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [twoGiants]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

afrittoli · 2025-12-17T11:48:29Z

@twoGiants are you happy for your latest comments to be implemented as follow-ups? If so, I'll lgtm this.

afrittoli · 2025-12-17T11:49:34Z

/retest

vdemeester · 2025-12-17T12:11:00Z

ah lol, I messed up something 😂

Add a test categorization system that enables selective execution of e2e tests based on whether they can safely run in parallel or must run serially. This optimizes test execution time while preserving safety for tests that modify shared cluster state. Uses comment annotations to mark tests: - `@test:execution=parallel` - Safe for concurrent execution - `@test:execution=serial` - Must run sequentially (modifies ConfigMaps) - `@test:reason` - Documents why serial execution is required The TestMain function in init_test.go parses these annotations using Go's AST parser and orchestrates test execution: - `-category=parallel` - Run only parallel tests - `-category=serial` - Run only serial tests - `-category=all` - Run serial first, then parallel (with fail-fast) - `-show-tests` - Display test categorization - Faster local development: run parallel tests concurrently - CI optimization: separate parallel and serial test jobs - Safety: prevents test interference from ConfigMap modifications - Self-documenting: annotations explain execution requirements Co-Authored-By: Claude <[email protected]> Signed-off-by: Vincent Demeester <[email protected]>

vdemeester · 2025-12-18T10:36:29Z

/retest

twoGiants · 2025-12-19T10:28:34Z

@twoGiants are you happy for your latest comments to be implemented as follow-ups? If so, I'll lgtm this.

Yes of course 😸 @afrittoli

afrittoli · 2025-12-19T11:31:03Z

/retest

afrittoli · 2025-12-19T11:31:34Z

/lgtm

github-project-automation bot added this to Tekton Community Roadmap Dec 9, 2025

tekton-robot added the release-note-none Denotes a PR that doesnt merit a release note. label Dec 9, 2025

github-project-automation bot moved this to Todo in Tekton Community Roadmap Dec 9, 2025

tekton-robot requested review from jerop and pritidesai December 9, 2025 14:10

tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 9, 2025

tekton-robot added the kind/misc Categorizes issue or PR as a miscellaneuous one. label Dec 9, 2025

vdemeester force-pushed the test/e2e-configmap-isolation branch 2 times, most recently from 9e6545a to 0c46b6b Compare December 10, 2025 09:41

zakisk mentioned this pull request Dec 11, 2025

feat: Add digest validation support to HTTP resolver #9171

Merged

8 tasks

twoGiants approved these changes Dec 16, 2025

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 16, 2025

vdemeester force-pushed the test/e2e-configmap-isolation branch 2 times, most recently from 7d8ac0f to 712aadc Compare December 17, 2025 09:29

vdemeester force-pushed the test/e2e-configmap-isolation branch from 712aadc to 9d89b01 Compare December 17, 2025 12:14

tekton-robot assigned afrittoli Dec 19, 2025

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 19, 2025

tekton-robot merged commit 791c3a6 into tektoncd:main Dec 19, 2025
49 of 51 checks passed

github-project-automation bot moved this from Todo to Done in Tekton Community Roadmap Dec 19, 2025

vdemeester deleted the test/e2e-configmap-isolation branch December 19, 2025 15:45

Conversation

vdemeester commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Actions

Submitter Checklist

Release Notes

Uh oh!

vdemeester commented Dec 9, 2025

Uh oh!

vdemeester commented Dec 10, 2025

Uh oh!

tekton-robot commented Dec 10, 2025

Uh oh!

afrittoli commented Dec 10, 2025

Uh oh!

vdemeester commented Dec 10, 2025

Uh oh!

vdemeester commented Dec 10, 2025

Uh oh!

twoGiants commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

twoGiants left a comment

Choose a reason for hiding this comment

Uh oh!

twoGiants Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

twoGiants Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

twoGiants Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

twoGiants Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

twoGiants Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

twoGiants Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

tekton-robot commented Dec 16, 2025

Uh oh!

afrittoli commented Dec 17, 2025

Uh oh!

afrittoli commented Dec 17, 2025

Uh oh!

vdemeester commented Dec 17, 2025

Uh oh!

vdemeester commented Dec 18, 2025

Uh oh!

twoGiants commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afrittoli commented Dec 19, 2025

Uh oh!

afrittoli commented Dec 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vdemeester commented Dec 9, 2025 •

edited

Loading

twoGiants commented Dec 16, 2025 •

edited

Loading

twoGiants commented Dec 19, 2025 •

edited

Loading