LTS 2.541.1 release checklist

[shalinisudarsan]

Previous .1 release check list: #762

Next LTS release

More information about the release process is available on the release guide.

Release Lead

@shalinisudarsan

Prep work

• LTS baseline discussed and selected in the Jenkins developers mailing list.
  If the last release of the preceding LTS line is a security release, consider making the matching weekly release the new LTS baseline.
  For example, 2.462.3 LTS and 2.479 were security releases and it is simpler to use 2.479 as baseline than 2.477.

• Create or update the release branches in all the repositories below, e.g. stable-2.387 with the init-lts-line script or carry out the equivalent steps therein. For more info, refer to stable.

  • jenkinsci/jenkins - https://github.com/jenkinsci/jenkins/tree/stable-2.541
  • jenkinsci/packaging - https://github.com/jenkinsci/packaging/tree/stable-2.541
  • jenkins-infra/release - https://github.com/jenkins-infra/release/tree/stable-2.541
  • jenkins-infra/docker - https://github.com/jenkins-infra/docker/tree/stable-2.541

• Check with the Jenkins Infrastructure team for backports on repositories jenkinsci/packaging, jenkins-infra/release and jenkins-infra/docker as per https://github.com/jenkins-infra/release/blob/master/docs/releases.md#open-a-backporting-pr.
  A message in the Matrix channel #jenkins-infra mentioning this issue and this item is enough: they will own the backports

  • Packaging backport: Backport of packaging in stable-2.541 for LTS 2.541.1 jenkinsci/packaging#728
  • Release backport: Backport from master (Weekly) in stable-2.541 for LTS 2.541.1 #833
  • Docker controller backport: LTS 2.541.1 release checklist #811 (comment)

• Create a pull request to update bom to the weekly version that will be the base of the release line (and strike this out for new point release).
  Assure that the bom-weekly version number is already testing the base of the release line or a version newer than the base of the release line.

• Review recent security advisories for fixes in Jenkins weeklies after the LTS baseline, and ensure there are Jira issues for their backport.

• Review GitHub issues and pull requests for additional LTS candidates, adding the lts-candidate label, and ensure that all tickets are resolved.

  • https://github.com/jenkinsci/jenkins/issues?q=is%3Aissue%20state%3Aclosed%20label%3Alts-candidate

• Send a backporting announcement email to the jenkinsci-dev mailing list, using the default template.
  Remember to exchange the LTS version, release date and Jira URLs.

  • https://groups.google.com/g/jenkinsci-dev/c/Rherp2AmNZI/m/80E80TRvBwAJ

• Update labels for lts-candidate issues, either add 2.541.1-fixed and remove lts-candidate or add 2.541.1-rejected, and retain lts-candidate.

• Backport changes, run the list-issue-commits script to locate commits via GItHub PR number, some manual work is required to locate them if the issue ID wasn't present at merge time, backport with git cherry-pick -x $commit.

• Open backporting PR with into-lts label and summary of changes in description from lts-candidate-stats script and:

  • the selected lts-candidates. Backport 2.541.1 jenkinsci/jenkins#26070
  • possible LTS candidates in the release repository.
  • possible LTS candidates in the packaging repository.

• Open a pull request towards the acceptance test harness and plugin compatibility test to confirm the incremental produced by the backporting PR doesn't contain regressions.
  The documentation explains which profiles you have to modify in your PR.

  Add 2.541.x line, remove 2.504.x line jenkinsci/bom#6198
  Test - updated jenkins version - 2.541.1-rc jenkinsci/acceptance-test-harness#2579

• Prepare LTS changelog based on the style guide using the changelog generator - This is normally done by the docs team, ask in gitter.

• Prepare LTS upgrade guide based on previous upgrade guides - This is normally done by the docs team, ask in gitter.

RC creation

• Merge backporting PR in jenkinci/jenkins using a merge commit (and do not squash).

• Retrieve the URL for the RC from the commit status (Jenkins Incrementals Publisher / Incrementals) of the last build on the stable branch (requires a passing build). Visit the jenkins-war URL and copy the URL of the war file, which would be something like https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/jenkins-war/2.387.1-rc32701.b_06d9cef554c/jenkins-war-2.387.1-rc32701.b_06d9cef554c.war. If the incrementals are broken you can deploy a build from your own machine with mvn -e clean deploy -DskipTests=true.

  • https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/jenkins-war/2.541.1-rc37778.03f54f77f4cd/jenkins-war-2.541.1-rc37778.03f54f77f4cd.war

• Publish a pre-release Github release, e.g. sample currently we don't have a changelog for RCs.

  • https://github.com/jenkinsci/jenkins/releases/tag/jenkins-2.541.1-rc

• Confirm the automatic announcement has been sent to the jenkinsci-dev mailing list and community forums. If the automatic announcement is not sent, compose and send the announcement yourself.

• Check with security team that no security update is planned. If a security update is planned, revise the checklist after the public pre-announcement to the jenkinsci-advisories mailing list.

• For a new LTS baseline's ".1" release, if there were recent security advisories for fixes in Jenkins weeklies after the LTS baseline that had to be backported:

  • Update those advisories to mention the new 2.xxx.1 LTS release as an additional fix version (example)
  • Update warnings metadata to exclude the ".1" release (example)
  • Inform the Jenkins security team about the need to update CVE metadata to exclude the new LTS line from affected version ranges.

LTS release

• Publish changelog (one day prior to the release in case of a security update).

• Announce the start of the LTS release process in the #jenkins-release:matrix.org channel.

• Launch job on release.ci.jenkins.io if no security release for Jenkins is planned.

  • Manually review and approve the child release job after carefully checking the "Plan" stage (you can compare with previous stable line).
  • If this is the first release of a new LTS line, the packaging job will fail on its first run. Either run the packaging job once and cancel it before the primary release job is run or accept that the packaging job on the first release of a new LTS line will need to be run a second time after it fails the initial run.
  • ~3 to 4 hours after the beginning of release job, manually review and approve the child packaging job.

• Wait for successful job completion (release: ~3 to 4 hours, packaging ~30 minutes).

• Check LTS changelog is visible on the downloads site.

• Publish GitHub release pointing to LTS changelog, sample.

• Confirm that all Packages are available on the Datadog page.

• Confirm the Debian installer acceptance test is passing.
  For good measures, check the console log to confirm that the correct release package was used (e.g. search for 2.387. If not, launch tests again).

• Confirm the Red Hat installer acceptance test is passing.
  For good measures, check the console log to confirm that the correct release package was used (e.g. search for 2.387. If not, launch tests again).

• Adjust state and Released As of Jira issues fixed in the release (see the changelog for issue links).

• Create pull request to update the jenkins.version in the most recent release profile in plugin BOM to the newly released version.
  Refer to first step before the release and second step after the release for examples
  Add 2.541.x line and remove 2.504.x jenkinsci/bom#6285

• Create a tag matching the LTS release you create in the docker repository and publish a GitHub release.

• Confirm that the images are available at Docker hub.

• Merge the PR generated by the jenkins-dependency-updater bot in the jenkinsci/helm-charts repository.

• Create a helpdesk ticket to update ci.jenkins.io, trusted.ci, cert.ci and release.ci to the new LTS release, example.
  Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.541.1 helpdesk#4965

• Send email asking for the next release lead, example, dates for the next one can be found on the Jenkins calendar.

[dduportal]

Note: updated the body following #831 and #832.

[lemeurherve]

Backport of packaging:

• Backport of packaging in stable-2.541 for LTS 2.541.1 jenkinsci/packaging#728

[lemeurherve]

• Check with the Jenkins Infrastructure team for backports on both repositories jenkinsci/packaging and jenkins-infra/release

I'm surprised to see this item already ticked as jenkinsci/packaging and jenkins-infra/release backports are not completed yet.
I would have expected it to be ticked after completion, then the current formulation is a bit ambiguous.

We may want to add items to the checklist to mark the actual completion of these backports, and/or merge/integrate them with the "Create or update the release branches in all the repositories below" section above.

WDYT?

cc @krisstern @shalinisudarsan @dduportal

[krisstern]

Yes, we checked the box since we were informed the Infra team is working on it and takes ownership of the tasks and we are keeping taps on the progress