Skip to content

fix(unit): stabilize backend unit tests#13525

Open
ntny wants to merge 1 commit into
kubeflow:masterfrom
ntny:fix-backend-unit-flaky-tests
Open

fix(unit): stabilize backend unit tests#13525
ntny wants to merge 1 commit into
kubeflow:masterfrom
ntny:fix-backend-unit-flaky-tests

Conversation

@ntny

@ntny ntny commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

The Test_executeV2_publishLogs/retry_required_-_component_success case had an implicit dependency on artifact map iteration order. When executor-logs was not processed in the expected order, the retry path could miss uploading logs and fail randomly with: blob (key "executor-logs-0") (code=NotFound): blob not found

The fake metadata client now fails the executor-logs artifact explicitly, so the test no longer depends on which artifact is processed first.

failed tests:

testoutput
E0612 21:06:16.100891   27196 launcher_v2.go:449] Component failed to execute successfully: exit status 1
=== RUN   Test_executeV2_publishLogs/retry_required_-_component_success
testoutput
    launcher_v2_test.go:312: 
        	Error Trace:	/home/runner/work/pipelines/pipelines/backend/src/v2/component/launcher_v2_test.go:312
        	Error:      	Expected nil, but got: blob (key "executor-logs-0") (code=NotFound): blob not found
        	Test:       	Test_executeV2_publishLogs/retry_required_-_component_success
        	Messages:   	Expected executor-logs to be readable at key "executor-logs-0"
    launcher_v2_test.go:313: 
        	Error Trace:	/home/runner/work/pipelines/pipelines/backend/src/v2/component/launcher_v2_test.go:313
        	Error:      	Not equal: 
        	            	expected: "testoutput\n"
        	            	actual  : ""
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1 @@
        	            	-testoutput
        	            	 
        	Test:       	Test_executeV2_publishLogs/retry_required_-_component_success

https://github.com/kubeflow/pipelines/actions/runs/27442866956/job/81120877056?pr=13171

Copilot AI review requested due to automatic review settings June 15, 2026 11:25
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hbelmiro for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR updates the v2 metadata fake client and related launcher tests to make RecordArtifact failures deterministic per output name (avoiding flaky behavior due to map iteration order).

Changes:

  • Extend RecordArtifactFailureFakeClient to track call counts per outputName and optionally fail only for a specific output.
  • Add a new constructor to configure failures for a single output name.
  • Update Test_executeV2_publishLogs assertions to validate retries specifically for executor-logs.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
backend/src/v2/metadata/client_fake.go Adds per-output failure configuration and per-output call counting in the fake metadata client.
backend/src/v2/component/launcher_v2_test.go Updates tests to use output-specific failure fake and assert retry counts deterministically.

Comment on lines 142 to +144
func (c *RecordArtifactFailureFakeClient) RecordArtifact(ctx context.Context, outputName, schema string, runtimeArtifact *pipelinespec.RuntimeArtifact, state pb.Artifact_State, bucketConfig *objectstore.Config) (*OutputArtifact, error) {
c.RecordArtifactCalls++
c.OutputNameCalls[outputName]++

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NewRecordArtifactFailureFakeClientForOutput is the sole constructor that always initializes OutputNameCalls

Comment thread backend/src/v2/metadata/client_fake.go Outdated
Comment thread backend/src/v2/metadata/client_fake.go Outdated
// and succeed the second time, to test retry behavior without depending on map iteration order.
if test.uploadFailure {
countingFakeMetadataClient = metadata.NewRecordArtifactFailureFakeClient(1)
countingFakeMetadataClient = metadata.NewRecordArtifactFailureFakeClientForOutput("executor-logs", 1)
@ntny ntny force-pushed the fix-backend-unit-flaky-tests branch from 1ca9758 to 87dba7f Compare June 15, 2026 11:38
@google-oss-prow google-oss-prow Bot added size/S and removed size/M labels Jun 15, 2026
@ntny ntny requested a review from Copilot June 15, 2026 19:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comment thread backend/src/v2/metadata/client_fake.go
Comment thread backend/src/v2/metadata/client_fake.go
Comment on lines +121 to +122
FailOutputName string
OutputNameCalls map[string]int
@ntny ntny force-pushed the fix-backend-unit-flaky-tests branch from 87dba7f to ad0a059 Compare June 15, 2026 20:01
@google-oss-prow google-oss-prow Bot added size/M and removed size/S labels Jun 15, 2026
…now targets the `executor-logs` artifact explicitly, avoiding random failures when artifacts are processed in a different map order.

Signed-off-by: arpechenin <arpechenin@avito.ru>
@ntny ntny force-pushed the fix-backend-unit-flaky-tests branch from ad0a059 to e311e0f Compare June 15, 2026 20:04
@ntny

ntny commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@ntny ntny changed the title fix(unit): deflake backend unit tests fix(unit): stabilize backend unit tests Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants