When the host CPI property is set (along with username and private_key), the CPI acts on a distant VitrualBox installation connecting with SSH.
This features doesn't work out-of-the-box anymore. Indeed the vbox CPI v0.4.2 uses a deprecated ssh-rsa singing algorithm when authenticating to the distant SSH server. Whereas recent OpenSSH installations are setup to refuse that signing algorithm by default, which leads to connection failures.
Example of failure
Here is an example with a bosh upload-stemcell operation made against a Bosh Director, with properly-configured VirtualBox CPI (named parent-host-vbox) that tries to connect to the “example-vbox-host” distant VirtualBox host:
$ bosh upload-stemcell --sha1 "7724ce4272dd8f19b44584a17d31595eac7595e5" "https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=621.125"
...
...
Task 982 | 15:19:56 | Update stemcell: Checking if this stemcell already exists (cpi: parent-host-vbox) (00:00:00)
Task 982 | 15:19:56 | Update stemcell: Uploading stemcell bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125 to the cloud (cpi: parent-host-vbox) (00:00:11)
L Error: CPI error 'Bosh::Clouds::CloudError' with message 'Importing stemcell from '/var/vcap/data/director/tmp/stemcell20240211-11466-yi90el/image': Creating stemcell parent: Connecting via SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain' in 'create_stemcell' CPI method (CPI request ID: 'cpi-307703')
Task 982 | 15:20:07 | Error: CPI error 'Bosh::Clouds::CloudError' with message 'Importing stemcell from '/var/vcap/data/director/tmp/stemcell20240211-11466-yi90el/image': Creating stemcell parent: Connecting via SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain' in 'create_stemcell' CPI method (CPI request ID: 'cpi-307703')
Task 982 Started Sun Feb 11 15:19:30 UTC 2024
Task 982 Finished Sun Feb 11 15:20:07 UTC 2024
Task 982 Duration 00:00:37
Task 982 error
Uploading remote stemcell 'https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=621.125':
Expected task '982' to succeed but state is 'error'
Exit code 1
It fails and on the “example-vbox-host” the logs show that the ssh-rsa key algorithm is refused:
benjamin@example-vbox-host:~$ sudo tail -F -n0 /var/log/auth.log
Feb 11 15:20:07 example-vbox-host sshd[356145]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Feb 11 15:20:07 example-vbox-host sshd[356145]: Connection closed by authenticating user benjamin <REDACTED-DIRECTOR-IP> port 60962 [preauth]
CPI SSH settings are fine, though
When connecting with OpenSSH from the Bosh Director, using the CPI settings, it works. But the signing algorithm is more recent: rsa-sha2-512.
bosh/0:~# apt update -qq && apt install -y -qq jq
bosh/0:~# jq -r .PrivateKey /var/vcap/jobs/virtualbox_cpi/config/cpi.json > ssh-key-from-config
bosh/0:~# chmod 600 ssh-key-from-config
bosh/0:~# ssh -vvvv -i ssh-key-from-config $(jq -r .Username /var/vcap/jobs/virtualbox_cpi/config/cpi.json)@$(jq -r .Host /var/vcap/jobs/virtualbox_cpi/config/cpi.json) cat /etc/hostname
...
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:PP64/8ntZnYRHjjBFkIx8QNf2roVy0YrApSPLCPwNi8
...
example-vbox-host
...
Workaround relaxing the ssh config
This StackExchange answer is a great source of understanding. Based on the suggested solution, we can modify the sshd config on the vbox host to accept the old signing algorithm:
# cat > /etc/ssh/sshd_config.d/accept-ssh-rsa-algorithm.conf
PubkeyAcceptedAlgorithms +ssh-rsa
# systemctl restart sshd
# systemctl status sshd
Then the same bosh upload-stemcell operation just works fine:
$ bosh upload-stemcell --sha1 "7724ce4272dd8f19b44584a17d31595eac7595e5" "https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=621.125"
...
...
Task 1112 | 14:42:21 | Update stemcell: Uploading stemcell bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125 to the cloud (cpi: parent-host-vbox) (00:00:25)
Task 1112 | 14:42:46 | Update stemcell: Save stemcell bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125 (sc-d7340402-90bb-44dd-4ea0-dfd8dfd35472) (cpi: parent-host-vbox) (00:00:00)
Task 1112 Started Tue Feb 13 14:42:01 UTC 2024
Task 1112 Finished Tue Feb 13 14:42:46 UTC 2024
Task 1112 Duration 00:00:45
Task 1112 done
Succeeded
Conclusion
The VitualBox CPI v0.4.2 is compiled using Golang v1.17.2, which old SSH libraries are still using by default a deprecated ssh-rsa singing algorithm when authenticating to the distant VirtualBox host.
Recent OpenSSH server installation expect more secure rsa-sha2-256 or rsa-sha2-512 algorithms instead. These are certainly the default in recent Golang SSH libraries.
In order for the SSH feature to work again out-of-the-box, we need the VirtualBox CPI to be recompiled using a recent Go version.
When the
hostCPI property is set (along withusernameandprivate_key), the CPI acts on a distant VitrualBox installation connecting with SSH.This features doesn't work out-of-the-box anymore. Indeed the vbox CPI v0.4.2 uses a deprecated
ssh-rsasinging algorithm when authenticating to the distant SSH server. Whereas recent OpenSSH installations are setup to refuse that signing algorithm by default, which leads to connection failures.Example of failure
Here is an example with a
bosh upload-stemcelloperation made against a Bosh Director, with properly-configured VirtualBox CPI (namedparent-host-vbox) that tries to connect to the “example-vbox-host” distant VirtualBox host:It fails and on the “example-vbox-host” the logs show that the
ssh-rsakey algorithm is refused:CPI SSH settings are fine, though
When connecting with OpenSSH from the Bosh Director, using the CPI settings, it works. But the signing algorithm is more recent:
rsa-sha2-512.Workaround relaxing the ssh config
This StackExchange answer is a great source of understanding. Based on the suggested solution, we can modify the sshd config on the vbox host to accept the old signing algorithm:
Then the same
bosh upload-stemcelloperation just works fine:Conclusion
The VitualBox CPI v0.4.2 is compiled using Golang v1.17.2, which old SSH libraries are still using by default a deprecated
ssh-rsasinging algorithm when authenticating to the distant VirtualBox host.Recent OpenSSH server installation expect more secure
rsa-sha2-256orrsa-sha2-512algorithms instead. These are certainly the default in recent Golang SSH libraries.In order for the SSH feature to work again out-of-the-box, we need the VirtualBox CPI to be recompiled using a recent Go version.