[ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries

[dduportal]

Follow up of #4688

• [ci.jenkins.io] migrate from Azure (sponsored) subscription to AWS (sponsored) account #4688 (comment)
• [ci.jenkins.io] migrate from Azure (sponsored) subscription to AWS (sponsored) account #4688 (comment)
• [ci.jenkins.io] migrate from Azure (sponsored) subscription to AWS (sponsored) account #4688 (comment)

=> The EC2 instances we are using for Windows seems way slower than when we were in Azure.

Multiple analysis and remediation here:

• Use Windows 2022 over 2019 by default (requires label fixups for the Docker jobs)
• Check the I/O performances
  • Usage of the mounted Z: drive (local NVMe), is it used for all (temp, other?)
  • Increase EBS performances for the Windows C: drive (rootfs)
  • Check the Windows performance tuning (does EC2 AMI needs something specific?)
• Evaluate the instance size (CPU/memory, spot rate reclaim, disk size, etc.)

[dduportal]

Interesting new feature for Windows: a new windows-ssh AMi type has been introduced in https://github.com/jenkinsci/ec2-plugin/releases/tag/1948.v72ee2dd75cf0

See https://issues.jenkins.io/browse/JENKINS-70093

=> could improve the whole Windows experience \o/

[dduportal]

With jenkins-infra/jenkins-infra#4251 applied (thanks @MarkEWaite ), Windows agents and their java CLI now utilizes Z: for temp dirs (ref. jenkins-infra/pipeline-library#947 but generalized).

Ref. jenkins-infra/jenkins-infra#4251 (comment)

=> build commands should now be using Z:. However, the agent process is still using C: (needs investigation even though impact need to be evaluated).

[dduportal]

Next step is to change the Windows agent setup from current AMI Type unix to the brand new windows-ssh to improve the startup behavior (before we try to setup the java.io.tmpdir to avoid path backslash escaping hell).

[dduportal]

As a prerequisite: applied https://plugins.jenkins.io/ec2/releases/#version_1949.vf7e0fa_c3b_5b_c to ci.jenkins.io

[dduportal]

(Windows VM agents only) After a few changes, we now have both the agent process's temp directory and the process environment using the specified Z: drive (Note that C:\\Windows\\Temp\\/ is still required for uploading the remoting/jar file until we can successfully use the windows-ssh launcher):

INFO: Launching remoting agent (via SSH2 Connection):  C:/tools/jdk-21/bin/java -XX:+PrintCommandLineFlags -Djava.io.tmpdir=Z:/Temp -jar C:\\Windows\\Temp\\/remoting.jar -workDir Z:/jenkins