Skip to content

🌱 test: Improve registry+v1 render regression test #2103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

camilamacedo86
Copy link
Contributor

@camilamacedo86 camilamacedo86 commented Jul 16, 2025

Simplify and fix render regression test

  • Old test made it easy to overwrite expected files via make verify.
  • In all Operator Framework projects, make verify is used for codegen (go fmt, controller-gen, etc), not for regenerating testdata that should never change.
    See context
  • Expected manifests should only change when renderer logic changes on purpose.

This PR:

  • Moves bundles and expected output under testdata/, following Go test conventions.
  • Makes it clear the ArgoCD bundle is used only for this regression test.
  • Adds a proper test that runs the renderer and checks the output matches the expected YAMLs.

Safer, clearer, and avoids mistakes in reviews.

@camilamacedo86 camilamacedo86 requested a review from a team as a code owner July 16, 2025 02:10
@openshift-ci openshift-ci bot requested review from bentito and oceanc80 July 16, 2025 02:11
Copy link

openshift-ci bot commented Jul 16, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign camilamacedo86 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

netlify bot commented Jul 16, 2025

Deploy Preview for olmv1 ready!

Name Link
🔨 Latest commit 8a66074
🔍 Latest deploy log https://app.netlify.com/projects/olmv1/deploys/687edf7291ec6a000865464a
😎 Deploy Preview https://deploy-preview-2103--olmv1.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Copy link

codecov bot commented Jul 16, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.63%. Comparing base (b5a475a) to head (8a66074).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2103      +/-   ##
==========================================
+ Coverage   73.60%   73.63%   +0.02%     
==========================================
  Files          78       78              
  Lines        7260     7260              
==========================================
+ Hits         5344     5346       +2     
+ Misses       1566     1565       -1     
+ Partials      350      349       -1     
Flag Coverage Δ
e2e 43.72% <ø> (+0.04%) ⬆️
experimental-e2e 56.47% <ø> (+0.09%) ⬆️
unit 58.87% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@camilamacedo86 camilamacedo86 force-pushed the remove-bundles-argo branch 7 times, most recently from 311dab9 to 36ae501 Compare July 16, 2025 07:17
@camilamacedo86 camilamacedo86 force-pushed the remove-bundles-argo branch 3 times, most recently from f7a2e82 to 0abdf8f Compare July 16, 2025 13:12
@camilamacedo86 camilamacedo86 changed the title (NOT MERGE) - Test 🌱 test: replace ArgoCD manifest regression test with mock-based resource validation Jul 16, 2025
@camilamacedo86 camilamacedo86 changed the title 🌱 test: replace ArgoCD manifest regression test with mock-based resource validation 🌱 test: Improve registry+v1 render regression test Jul 22, 2025
@@ -1,7 +0,0 @@
## registry+v1 bundle generation regression tests
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do not need the readme anymore
now, the code is documented and we can clarify on it directly to simplify

@@ -252,6 +246,10 @@ test-unit: $(SETUP_ENVTEST) envtest-k8s-bins #HELP Run the unit tests
$(UNIT_TEST_DIRS) \
-test.gocoverdir=$(COVERAGE_UNIT_DIR)

.PHONY: test-regression
test-regression: #HELP Run regression test
go test -count=1 -v ./test/regression/...
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now have a dedicated directory for regression tests, so it's easy to add more cases in the future.
It also makes it very clear what this test does and why it's there.

@@ -172,15 +172,9 @@ generate: $(CONTROLLER_GEN) #EXHELP Generate code containing DeepCopy, DeepCopyI
$(CONTROLLER_GEN) --load-build-tags=$(GO_BUILD_TAGS) object:headerFile="hack/boilerplate.go.txt" paths="./..."

.PHONY: verify
verify: k8s-pin kind-verify-versions fmt generate manifests crd-ref-docs generate-test-data #HELP Verify all generated code is up-to-date. Runs k8s-pin instead of just tidy.
Copy link
Contributor Author

@camilamacedo86 camilamacedo86 Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If make verify failed in CI, most people would just run it locally and push the changes — that’s normal since verify usually runs things like go fmt, vet, and controller-gen (as in all OF projects). But it was also re-generating regression test output — which should never change or at least very unlikely change. Then:

  • If that output changed because of a bug, we could accidentally miss it and merge a regression.
  • It also wasn’t clear to reviewers that files in testdata/expected-manifests/ should never be edited or auto-regenerated.
  • Expecting everyone to remember to check that manually is risky and easy to miss.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants