Implement a retry mechanism with non-spot instance in `jenkinsci/jenkins` pipeline

[lemeurherve]

Service(s)

ci.jenkins.io

Summary

When working on core releases like jenkins-infra/release#744 (or any other core work) and to avoid wasting contributor time, having the retry executed on non-spot instances for Windows (VMs) could improve the process.

In order to reproduce the same mechanism as implemented on ATH (jenkinsci/acceptance-test-harness#1352), new -nonspot agent definitions for maven-<jdk>-windows (with <jdk> = 17 | 21 | 25) should be added to https://github.com/jenkins-infra/jenkins-infra/blob/f670c9fe1caf720c384a21e10d2e3254bf22bc84/hieradata/clients/aws.ci.jenkins.io.yaml#L357 cf https://github.com/jenkinsci/jenkins/blob/65e05c0755125722a8f9dab2af4e20dc879cc633/Jenkinsfile#L66-L69

Reproduction steps

1. Open a pull request on https://github.com/jenkinsci/jenkins/
2. Notice the build runs first on Windows spot instances
3. If the build failed, notice it restarts on the same kind of node/agent than the first time

Having the build restarting on non-spot/"on demand" instances would avoid aborted builds due to spot instance reclaim.

Ref:

• https://matrix.to/#/!JlkqzpdEnsUUuVtjgE:matrix.org/$Q9EGkf0KynT3NBCIL2ZqS6ShIRtpJUJcAfH870ZFwWA?via=gitter.im&via=matrix.org

[lemeurherve]

Note: I'm willing to work on this issue.

[dduportal]

Notes (which might require updating the issue body):

• It's only needed for maven-xx-windows which are VMs. The maven-XX are Kubernetes (Linux) agents for which the concept of retrying to non spot makes no sense as we use Karpenter for managing nodes.
• The pipeline library buildPlugin() already has an algorithm for this: https://github.com/jenkins-infra/pipeline-library/blob/2cb8931abde48f96a55f98af46432306f8287e36/vars/buildPlugin.groovy#L46-L58. I believe we can totally add a new function in the infra.groovy to reuse this behavior

[lemeurherve]

It's only needed for maven-xx-windows which are VMs. The maven-XX are Kubernetes (Linux) agents for which the concept of retrying to non spot makes no sense as we use Karpenter for managing nodes.

Thanks, integrated into the issue body.

The pipeline library buildPlugin() already has an algorithm for this: https://github.com/jenkins-infra/pipeline-library/blob/2cb8931abde48f96a55f98af46432306f8287e36/vars/buildPlugin.groovy#L46-L58. I believe we can totally add a new function in the infra.groovy to reuse this behavior

You're suggesting to add that mechanism as an infra.groovy function then used in jenkinsci/jenkins pipeline instead of implementing it directly in the pipeline?
Where would it be reused? If there is no other immediate candidate, does it worth the effort?

[dduportal]

The pipeline library buildPlugin() already has an algorithm for this: https://github.com/jenkins-infra/pipeline-library/blob/2cb8931abde48f96a55f98af46432306f8287e36/vars/buildPlugin.groovy#L46-L58. I believe we can totally add a new function in the infra.groovy to reuse this behavior

You're suggesting to add that mechanism as an infra.groovy function then used in jenkinsci/jenkins pipeline instead of implementing it directly in the pipeline?

Exactly, thanks for clarifying!

Where would it be reused? If there is no other immediate candidate, does it worth the effort?

All "non plugin" jobs: could benefit from this centralized logic (ATH, BOM, infra jobs, docker builds etc.), along with buildGroovy. But I see your point: you can start with the Jenkins Core pipeline only as first viable step.

[dduportal]

ping @lemeurherve what is the status on this issue?

[lemeurherve]

Opened jenkinsci/jenkins#11052

I'll need someone to approve my Jenkinsfile changes on ci.jenkins.io as I don't have any permission on jenkinsci/jenkins repo:

Loading trusted files from base branch master at c64384a8671b03b0541342b678d414ae51e308be rather than 38cb6348383891c14a201d7e984072e271ee277a