MacStadium maintenance window on January 23rd

[UlisesGascon]

As described in ticket: SERVICE-176962

Dear OpenJS

On Tuesday, January 23rd, 2024, at 9 AM ET, we need to conduct a one-hour maintenance in our ATL data center that will impact your ORKA cluster for one hour. We apologize in advance.

Before the start of the maintenance, please save and shut down any VMs in advance of the maintenance start.

We will notify you once the nodes are back up here in the ticket. Again, we apologize in advance for any inconvenience this may cause. Thank you for your understanding.

Potential affected machines:

• release-orka-macos11-x64-1 (tracked on Jenkins)
• test-orka-macos10.15-x64-1 (test-orka-macos10.15-x64-1 is DOWN jenkins-alerts#1237)
• test-orka-macos10.15-x64-2 (test-orka-macos10.15-x64-2 is DOWN jenkins-alerts#703)
• test-orka-macos11-x64-1 (test-orka-macos11-x64-1 is DOWN jenkins-alerts#1238)
• test-orka-macos11-x64-2 (test-orka-macos11-x64-2 is DOWN jenkins-alerts#1239)

Next steps

I am not sure that I will be able to manage the "save and shut down" for the VMs before the deadline (tomorrow), anyone is available to do it (@nodejs/build)?

test-orka-macos10.15-x64-1:

• Restore test-orka-macos10.15-x64-1 in Orka cluster
• Reansible test-orka-macos10.15-x64-1
• re-enable test-orka-macos10.15-x64-1 in Jenkins
• Save the state
• commit the image changes

test-orka-macos10.15-x64-2:

• Restore test-orka-macos10.15-x64-2 in Orka cluster
• Reansible test-orka-macos10.15-x64-2
• re-enable test-orka-macos10.15-x64-2 in Jenkins
• Save the state
• commit the image changes

test-orka-macos11-x64-1:

• Restore test-orka-macos11-x64-1 in Orka cluster
• Reansible test-orka-macos11-x64-1
• re-enable test-orka-macos11-x64-1 in Jenkins
• Save the state
• commit the image changes

test-orka-macos11-x64-2:

• Restore test-orka-macos11-x64-2 in Orka cluster
• Reansible test-orka-macos11-x64-2
• re-enable test-orka-macos11-x64-2 in Jenkins
• Save the state
• commit the image changes

release-orka-macos11-x64-1:

• Restore release-orka-macos11-x64-1 in Orka cluster
• Reansible release-orka-macos11-x64-1
• Manual steps on release-orka-macos11-x64-1
• re-enable release-orka-macos11-x64-1 in Jenkins
• Save the state
• commit the image changes

[UlisesGascon]

Update (9 AM ET):

We are beginning the maintenance and will update you once completed.

[UlisesGascon]

Update (10 AM ET):

The maintenance is now completed. Thank you.

[UlisesGascon]

We will need to recover the machines manually in order to make the Orka cluster working again. cc: @nodejs/build.

I am not available today, but I can try to work on it tomorrow (potentially), but feel free to take leadership if you want.

IMPORTANT: You can use this table (#3240 (comment)) as a reference to know where to locate the vms within the cluster in order to align the VMs with the inventory

[UlisesGascon]

I am not available today, but I can try to work on it tomorrow (potentially), but feel free to take leadership if you want.

I am afraid that I won't be able to work on it today, I will start to work on it only from next Monday. 😓

[mhdawson]

@UlisesGascon thanks for working on it. One question is if the machine recovery is needed because they were not shut down properly (I only noticed the original issue too late to help out) or if that would have been required regardless?

[UlisesGascon]

One question is if the machine recovery is needed because they were not shut down properly (I only noticed the original issue too late to help out) or if that would have been required regardless?

This situation is a bit tricky, drawing from past experiences such as #3112. The VMs allocated in specific slots, including port mapping, are expected to be shut down and effectively 'removed' from the Orka cluster nodes.

Once the cluster is back, a manual relocation process is necessary to create new VMs using the images. This ensures the correct slots are filled, maintaining the expected mapping from the inventory and Jenkins (IPs and ports).

In this case, we didn't save and shut down the VMs before the process. Consequently, I suspect that the images, due to the destruction of VMs, might be an older version of the existing VMs. This will require re-ansibleing each VM once deployed and some manual configuration, particularly with the Jenkins tokens, depending on the state of the images.

I'll have a clearer picture on Monday. Unfortunately, I haven't been able to connect yet to check the status of the cluster or the nodes after the upgrade.