Unable to set up Singularity runtime on Fred Hutch's `rhino`

[victorlin]

Error

$ nextstrain setup --set-default singularity
Setting up singularity…
Updating Singularity image docker://nextstrain/base:build-20240130T210714Z…

…

Checking setup…
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
FATAL:   Unable to handle docker://busybox uri: unable to build: conveyor failed to get: Error initializing source oci:/home/vlin/.nextstrain/runtimes/singularity/cache/cache/oci:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74: Error initializing image from source docker://busybox:latest: unsupported schema version 2
✔ yes: singularity is installed
✔ yes: singularity version 3.5.3 ≥ 3.0.0 (1.0.0 for Apptainer)
✘ no: singularity works

Checks failed!  Setup is unlikely to be fully functional.

Details

First, there was an extra step to access singularity.

$ singularity --version
Singularity is not allowed on Rhino nodes as it may cause 
stability problems.
Please run singularity via sbatch or run grabnode and request at least 4 cores.

I used grabnode to enter an environment with access to singularity, then ran module load Singularity/3.5.3.

$ singularity --version
singularity version 3.5.3

nextstrain setup does not complete successfully:

$ nextstrain setup --set-default singularity
Setting up singularity…
Updating Singularity image docker://nextstrain/base:build-20240130T210714Z…

INFO:    Starting build...
Getting image source signatures
Copying blob 0e0969fcaa82 done
…
Copying config a736e69756 done
Writing manifest to image destination
Storing signatures
2024/02/02 10:47:27  info unpack layer: sha256:0e0969fcaa8240e1eeb53f9f5d4ddd1bf89a2c9971c9cbe455eba0e66eeefb53
…
2024/02/02 10:47:48  info unpack layer: sha256:37780fb472b6b29396d6794364b768a2e7c8c4bbfe51d592bfcf6a825382bc2d
INFO:    Creating SIF file...
INFO:    Build complete: /home/vlin/.nextstrain/runtimes/singularity/images/nextstrain/base/build-20240130T210714Z.sif

Cleaning up…

Removing /home/vlin/.nextstrain/runtimes/singularity/cache/cache/library
Removing /home/vlin/.nextstrain/runtimes/singularity/cache/cache/oci-tmp
Removing /home/vlin/.nextstrain/runtimes/singularity/cache/cache/shub
Removing /home/vlin/.nextstrain/runtimes/singularity/cache/cache/oci
Removing /home/vlin/.nextstrain/runtimes/singularity/cache/cache/net
Removing /home/vlin/.nextstrain/runtimes/singularity/cache/cache/oras

Pruning old images…


Checking setup…
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
FATAL:   Unable to handle docker://busybox uri: unable to build: conveyor failed to get: Error initializing source oci:/home/vlin/.nextstrain/runtimes/singularity/cache/cache/oci:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74: Error initializing image from source docker://busybox:latest: unsupported schema version 2
✔ yes: singularity is installed
✔ yes: singularity version 3.5.3 ≥ 3.0.0 (1.0.0 for Apptainer)
✘ no: singularity works

Checks failed!  Setup is unlikely to be fully functional.

[victorlin]

Solution: Use Apptainer

$ module load Apptainer

$ singularity --version
apptainer version 1.1.6

$ nextstrain setup --set-default singularity
Setting up singularity…
Using existing local copy of Singularity image docker://nextstrain/base:build-20240130T210714Z.
  Hint: if you want to ignore this existing local copy, re-run `nextstrain setup` with --force.

Checking setup…
INFO:    Detected Singularity user configuration directory
INFO:    Environment variable SINGULARITY_CACHEDIR is set, but APPTAINER_CACHEDIR is preferred
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 9ad63333ebc9 done  
Copying config 3f57d9401f done  
Writing manifest to image destination
Storing signatures
2024/02/02 11:21:11  info unpack layer: sha256:9ad63333ebc97e32b987ae66aa3cff81300e4c2e6d2f2395cef8a3ae18b249fe
INFO:    Creating SIF file...
INFO:    Environment variable SINGULARITYENV_PROMPT_COMMAND is set, but APPTAINERENV_PROMPT_COMMAND is preferred
✔ yes: singularity is installed
✔ yes: singularity version 1.1.6 ≥ 3.0.0 (1.0.0 for Apptainer)
✔ yes: singularity works

Setting default runtime to singularity.

All good!  Set up of singularity complete.

[tsibley]

I think that Singularity is having issues with the docker://busybox image that's used in the ✔ yes: singularity works check. This may be due to the upstream busybox image changing to a newer format underneath us?