baremetal: integrate automated (re-)provisioning logic

The bare-metal platform lacked any understanding of whether an instance
is actually running the configuration that lokoctl put into matchbox,
because, when the configuration was updated, there was no notification
to the user that PXE booting has to be done again for this instance.
Also, it was not clear to the user when to boot from PXE because the
PXE boot must happen after lokoctl populated matchbox with the (new)
configuration but before any other steps are timing out. The goal of
this patch is to bring the baremetal platform to an actually usable
level and support automated provisioning and reprovisioning, in a
configurable way regardless if IPMI is used or VMs are created.
In addition, we don't want to require a complicated PXE boot for each
configuration update because it is slow, fragile and needs a special
DHCP infrastructure.
Add user-defined commands to perform automated PXE provisioning at the
right time, i.e. initally at the first run or when recreating a node.
However, PXE booting is a long and maybe even manual process, or even
impossible at production side due to the lack of an appropriate DHCP
server. We can rely on Ignition to simulate reprovisioning by creating
the first_boot flag file via SSH and issuing a reboot, which makes
Ignition fetch the configuration from matchbox, and if we make sure to
clean the root filesystem by formatting it, the result is the same as
if reprovisioned was done with a PXE boot. This is achieved by a null
resource in Terraform that executes a helper script which either does
a PXE boot or uses SSH to trigger a reprovisioning with Ignition. It
also handles the case of ignoring userdata changes for controller
nodes to prevent losing etcd state. Since there is no notion of a
baremetal node on the Terraform level (reminder: all this exercise here
is done because we don't have a Terraform provider doing this for us)
a local flag file is created under the asset folder. If it exists, the
node was provisioned with PXE and SSH will be used for reprovisioning,
if it does not exist, it will be provisioned with PXE during inital
setup and for the next reprovisioning because the user forced
recreating the node by deleting the flag file. Another flag file on the
node is used to check whether a node was successfully reprovisioned.
When SSH is used to reprovision, the kernel parameters for GRUB are
updated directly because they are not part of the Ignition
configuration. The copy-controller-secrets step is run after recreating
a controller node, again since there is no notion of a node object by
depending on the variables itself which define the node state.
Also add a user-defined command to run after the PXE OS installation
and before booting into the final OS. This is needed to set up
persistent booting from disk after the PXE booting was configured.
The whole patch is used by Racker, and can be tested either with the
bootstrap/prepare.sh script to create VMs with lokoctl or by running
Racker in the QEMU IPMI simulator environment through the
racker-sim/ipmi-env.sh script and a Racker Docker image built with
installer/conf.yaml pointing to this Lokomotive branch.

Author: pothos

assets/terraform-modules/bare-metal/flatcar-linux/kubernetes/controller.tf

@@ -13,8 +13,22 @@ module "controller" {
   set_standard_hostname  = false
   clc_snippets = concat(lookup(var.clc_snippets, var.controller_names[count.index], []), [
     <<EOF
+filesystems:
+  - name: root
+    mount:
+      device: /dev/disk/by-label/ROOT
+      format: ext4
+      wipe_filesystem: true
+      label: ROOT
 storage:
   files:
+    - path: /ignition_ran
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          Flag file indicating that Ignition ran.
+          Should be deleted by the SSH step that checks it.
     - path: /etc/hostname
       filesystem: root
       mode: 0644

assets/terraform-modules/bare-metal/flatcar-linux/kubernetes/controller_profiles.tf

@@ -1,8 +1,10 @@
 module "controller_profile" {
   source                   = "../../../matchbox-flatcar"
   count                    = length(var.controller_names)
+  asset_dir                = var.asset_dir
   node_name                = var.controller_names[count.index]
   node_mac                 = var.controller_macs[count.index]
+  node_domain              = var.controller_domains[count.index]
   download_protocol        = var.download_protocol
   os_channel               = var.os_channel
   os_version               = var.os_version
@@ -17,4 +19,7 @@ module "controller_profile" {
   ignition_clc_config      = module.controller[count.index].clc_config
   cached_install           = var.cached_install
   wipe_additional_disks    = var.wipe_additional_disks
+  ignore_changes           = true
+  pxe_commands             = var.pxe_commands
+  install_pre_reboot_cmds  = var.install_pre_reboot_cmds
 }

assets/terraform-modules/bare-metal/flatcar-linux/kubernetes/ssh.tf

@@ -81,7 +81,12 @@ resource "null_resource" "copy-controller-secrets" {
     ]
   }
 
+
+  # Triggered when the Ignition Config changes (used to recreate a controller)
   triggers = {
+    clc_config       = module.controller[count.index].clc_config
+    kernel_console   = join(" ", var.kernel_console)
+    kernel_args      = join(" ", var.kernel_args)
     etcd_ca_cert     = module.bootkube.etcd_ca_cert
     etcd_server_cert = module.bootkube.etcd_server_cert
     etcd_peer_cert   = module.bootkube.etcd_peer_cert

assets/terraform-modules/bare-metal/flatcar-linux/kubernetes/variables.tf

@@ -227,3 +227,15 @@ variable "wipe_additional_disks" {
   description = "Wipes any additional disks attached, if set to true"
   default     = false
 }
+
+variable "pxe_commands" {
+  type        = string
+  default     = "echo 'you must (re)provision the node by booting via iPXE from http://MATCHBOX/boot.ipxe'; exit 1"
+  description = "shell commands to execute for PXE (re)provisioning, with access to the variables $mac (the MAC address), $name (the node name), and $domain (the domain name), e.g., 'bmc=bmc-$domain; ipmitool -H $bmc power off; ipmitool -H $bmc chassis bootdev pxe; ipmitool -H $bmc power on'"
+}
+
+variable "install_pre_reboot_cmds" {
+  type        = string
+  default     = "true"
+  description = "shell commands to execute on the provisioned host after installation finished and before reboot, e.g., docker run --privileged --net host --rm debian sh -c 'apt update && apt install -y ipmitool && ipmitool chassis bootdev disk options=persistent'"
+}

assets/terraform-modules/bare-metal/flatcar-linux/kubernetes/worker.tf

@@ -12,8 +12,22 @@ module "worker" {
   set_standard_hostname  = false
   clc_snippets = concat(lookup(var.clc_snippets, var.worker_names[count.index], []), [
     <<EOF
+filesystems:
+  - name: root
+    mount:
+      device: /dev/disk/by-label/ROOT
+      format: ext4
+      wipe_filesystem: true
+      label: ROOT
 storage:
   files:
+    - path: /ignition_ran
+      filesystem: root
+      mode: 0644
+      contents:
+        inline: |
+          Flag file indicating that Ignition ran.
+          Should be deleted by the SSH step that checks it.
     - path: /etc/hostname
       filesystem: root
       mode: 0644

assets/terraform-modules/bare-metal/flatcar-linux/kubernetes/worker_profiles.tf

@@ -1,8 +1,10 @@
 module "worker_profile" {
   source                   = "../../../matchbox-flatcar"
   count                    = length(var.worker_names)
+  asset_dir                = var.asset_dir
   node_name                = var.worker_names[count.index]
   node_mac                 = var.worker_macs[count.index]
+  node_domain              = var.worker_domains[count.index]
   download_protocol        = var.download_protocol
   os_channel               = var.os_channel
   os_version               = var.os_version
@@ -17,4 +19,6 @@ module "worker_profile" {
   ignition_clc_config      = module.worker[count.index].clc_config
   cached_install           = var.cached_install
   wipe_additional_disks    = var.wipe_additional_disks
+  pxe_commands             = var.pxe_commands
+  install_pre_reboot_cmds  = var.install_pre_reboot_cmds
 }

assets/terraform-modules/matchbox-flatcar/profiles.tf

@@ -33,6 +33,7 @@ data "ct_config" "install-ignitions" {
     kernel_console           = join(" ", var.kernel_console)
     kernel_args              = join(" ", var.kernel_args)
     wipe_additional_disks    = var.wipe_additional_disks
+    install_pre_reboot_cmds  = var.install_pre_reboot_cmds
     # only cached-container-linux profile adds -b baseurl
     baseurl_flag = ""
     mac_address  = var.node_mac
@@ -80,6 +81,7 @@ data "ct_config" "cached-install-ignitions" {
     kernel_console           = join(" ", var.kernel_console)
     kernel_args              = join(" ", var.kernel_args)
     wipe_additional_disks    = var.wipe_additional_disks
+    install_pre_reboot_cmds  = var.install_pre_reboot_cmds
     # profile uses -b baseurl to install from matchbox cache
     baseurl_flag = "-b ${var.http_endpoint}/assets/flatcar"
     mac_address  = var.node_mac

assets/terraform-modules/matchbox-flatcar/pxe-helper.sh.tmpl

@@ -0,0 +1,87 @@
+# (executed in-line, #!/... would be ignored)
+# Terraform template variable substitution:
+name=${name}
+domain=${domain}
+mac=${mac}
+asset_dir=${asset_dir}
+ignore_changes=${ignore_changes}
+kernel_args="${kernel_args}"
+kernel_console="${kernel_console}"
+ignition_endpoint="${ignition_endpoint}"
+# From now on use $var for dynamic shell substitution
+
+if test -f "$asset_dir/$mac" && [ "$(cat "$asset_dir/$mac")" = "$domain" ]; then
+  echo "found $asset_dir/$mac containing $domain, skipping PXE install"
+  node_exists=yes
+else
+  echo "$asset_dir/$mac does not contain $domain, forcing PXE install"
+  node_exists=no
+fi
+
+if [ $node_exists = yes ]; then
+  if $ignore_changes ; then
+    echo "Keeping old config because 'ignore_changes' is set."
+    exit 0
+  else
+    # run single commands that can be retried without a side effect in case the connection got disrupted
+    count=30
+    while [ $count -gt 0 ] && ! ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NumberOfPasswordPrompts=0 core@$domain sudo touch /boot/flatcar/first_boot; do
+      sleep 1
+      count=$((count - 1))
+    done
+    if [ $count -eq 0 ]; then
+      echo "error reaching $domain via SSH, please remove the $asset_dir/$mac file to force a PXE install"
+      exit 1
+    fi
+    echo "created the first_boot flag file to reprovision $domain"
+    count=5
+    while [ $count -gt 0 ] && ! ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NumberOfPasswordPrompts=0 core@$domain "printf 'set linux_append=\"$kernel_args ignition.config.url=$ignition_endpoint?mac=$mac&os=installed\"\\nset linux_console=\"$kernel_console\"\\n' | sudo tee /usr/share/oem/grub.cfg"; do
+      sleep 1
+      count=$((count - 1))
+    done
+    if [ $count -eq 0 ]; then
+      echo "error reaching $domain via SSH, please retry"
+      exit 1
+    fi
+    count=5
+    while [ $count -gt 0 ] && ! ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NumberOfPasswordPrompts=0 core@$domain sudo systemctl reboot; do
+      sleep 1
+      count=$((count - 1))
+    done
+    if [ $count -eq 0 ]; then
+      echo "error reaching $domain via SSH, please reboot manually"
+      exit 1
+    fi
+    echo "rebooted the $domain"
+  fi
+else
+  # the user may provide ipmitool commands or any other logic for forcing a PXE boot
+  ${pxe_commands}
+fi
+
+echo "checking that $domain comes up"
+count=600
+# check that we can reach the node and that it has the flag file which we remove here, indicating a reboot happened which prevents a race when issuing the reboot takes longer (both the systemctl reboot and PXE case)
+# Just in case the connection breaks and SSH may report an error code but still execute successfully, we will first check file existence and then delete with "rm -f" to be able to rerun both commands.
+# This sequence gives us the same error reporting as just running "rm" once.
+while [ $count -gt 0 ] && ! ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NumberOfPasswordPrompts=0 core@$domain test -f /ignition_ran; do
+  sleep 1
+  count=$((count - 1))
+done
+if [ $count -eq 0 ]; then
+  echo "error: failed verifying with SSH if $domain came up by checking the /ignition_ran flag file"
+  exit 1
+fi
+count=5
+while [ $count -gt 0 ] && ! ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NumberOfPasswordPrompts=0 core@$domain sudo rm -f /ignition_ran; do
+  sleep 1
+  count=$((count - 1))
+done
+if [ $count -eq 0 ]; then
+  echo "error: failed to remove the /ignition_ran flag file on $domain"
+  exit 1
+else
+  echo "$domain came up again"
+fi
+# only write the state file once the system is up, this allows to rerun lokoctl if the first PXE boot did not work and it will try again
+echo $domain > "$asset_dir/$mac"

assets/terraform-modules/matchbox-flatcar/ssh.tf

@@ -0,0 +1,14 @@
+resource "null_resource" "reprovision-node-when-ignition-changes" {
+  # Triggered when the Ignition Config changes
+  triggers = {
+    ignition_config = matchbox_profile.node.raw_ignition
+    kernel_args     = join(" ", var.kernel_args)
+    kernel_console  = join(" ", var.kernel_console)
+  }
+  # Wait for the new Ignition config object to be ready before rebooting
+  depends_on = [matchbox_group.node]
+  # Trigger running Ignition on the next reboot (first_boot flag file) and reboot the instance, or, if the instance needs to be (re)provisioned, run external commands for PXE booting (also runs on the first provisioning)
+  provisioner "local-exec" {
+    command = templatefile("${path.module}/pxe-helper.sh.tmpl", { domain = var.node_domain, name = var.node_name, mac = var.node_mac, pxe_commands = var.pxe_commands, asset_dir = var.asset_dir, kernel_args = join(" ", var.kernel_args), kernel_console = join(" ", var.kernel_console), ignition_endpoint = format("%s/ignition", var.http_endpoint), ignore_changes = var.ignore_changes })
+  }
+}

assets/terraform-modules/matchbox-flatcar/templates/install.yaml.tmpl

@@ -67,6 +67,7 @@ storage:
           echo 'set linux_append="${kernel_args} ignition.config.url=${ignition_endpoint}?mac=${mac_address}&os=installed"' >> /tmp/oemfs/grub.cfg
           echo 'set linux_console="${kernel_console}"' >> /tmp/oemfs/grub.cfg
           umount /tmp/oemfs
+          ${install_pre_reboot_cmds}
           systemctl reboot
 passwd:
   users: