Problem running with podman on HPC

[benz0li]

You could give glcr.b-data.ch/jupyterlab/r/geospatial a try, i.e.

podman run --rm \
  -p 8888:8888 \
  glcr.b-data.ch/jupyterlab/r/geospatial

This just gives me: Error: OCI runtime error: crun: cannot setresgid to `100`: Invalid argument

podman run --rm \
  -p 8888:8888 \
  -u root \
  -e NB_USER=root \
  -e NB_UID=0 \
  -e NB_GID=0 \
  -e NOTEBOOK_ARGS="--allow-root" \
  glcr.b-data.ch/jupyterlab/r/geospatial

This seems to do much more:

Entered start.sh with args: start-notebook.sh
Running hooks in: /usr/local/bin/start-notebook.d as uid: 0 gid: 0
Sourcing shell script: /usr/local/bin/start-notebook.d/10-populate.sh
Done running hooks in: /usr/local/bin/start-notebook.d
Updated the jovyan user:
- username: jovyan       -> root
- home dir: /home/jovyan -> /home/root
Attempting to copy /home/jovyan to /home/root...
Success!
Changing working directory to /home/root/
Running hooks in: /usr/local/bin/before-notebook.d as uid: 0 gid: 0
Sourcing shell script: /usr/local/bin/before-notebook.d/10-env.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/11-home.sh

But there is an error in the "end": runuser: cannot set groups: Operation not permitted

Originally posted by @bernt-matthias in rocker-org/rocker-versioned2#838 (comment)

[benz0li]

Most likely due to run_user_group, i.e.

jupyterlab-r-docker-stack/base/scripts/usr/local/bin/before-notebook.d/11-home.sh

Lines 9 to 10 in bfcd656

	run_user_group mkdir -p "/home/$NB_USER${DOMAIN:+@$DOMAIN}/projects"
	run_user_group mkdir -p "/home/$NB_USER${DOMAIN:+@$DOMAIN}/workspaces"

[benz0li]

jupyterlab-r-docker-stack/base/scripts/usr/local/bin/start.sh

Lines 13 to 15 in bfcd656

	function run_user_group() {
	runuser -u "${NB_USER}" -g "$(id -gn "${NB_USER}")" -G "users" -- "$@"
	}

This might not be required when run as root with -e NB_USER=root -e NB_UID=0 -e NB_GID=0.

@bernt-matthias I will look into this and provide a patched image so you can test it on the HPC.

[bernt-matthias]

Excellent. Thanks a lot

[benz0li]

@bernt-matthias Try with

podman run --rm \
  -p 8888:8888 \
  -u root \
  -e NB_USER=root \
  -e NB_UID=0 \
  -e NB_GID=0 \
  -e NOTEBOOK_ARGS="--allow-root" \
  glcr.b-data.ch/jupyterlab/r/base:test-hpc

Thanks for your feedback.

[bernt-matthias]

Seems that it gets closer, but there is still some error:

Entered start.sh with args: start-notebook.sh
Running hooks in: /usr/local/bin/start-notebook.d as uid: 0 gid: 0
Sourcing shell script: /usr/local/bin/start-notebook.d/10-populate.sh
Done running hooks in: /usr/local/bin/start-notebook.d
Updated the jovyan user:
- username: jovyan       -> root
- home dir: /home/jovyan -> /home/root
Attempting to copy /home/jovyan to /home/root...
Success!
Changing working directory to /home/root/
Running hooks in: /usr/local/bin/before-notebook.d as uid: 0 gid: 0
Sourcing shell script: /usr/local/bin/before-notebook.d/10-env.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/11-home.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/12-r.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/13-update-cran.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/30-code-server.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/50-rstudio.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/71-tensorboard.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/90-limits.sh
Sourcing shell script: /usr/local/bin/before-notebook.d/95-misc.sh
Done running hooks in: /usr/local/bin/before-notebook.d
Running as root: start-notebook.sh
sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Invalid argument
sudo: no valid sudoers sources found, quitting
sudo: error initializing audit plugin sudoers_audit

[benz0li]

That is due to sudo in start.sh:

jupyterlab-r-docker-stack/base/scripts/usr/local/bin/start.sh

Lines 240 to 244 in bfcd656

	exec sudo --preserve-env --set-home --user "${NB_USER}" \
	LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" \
	PATH="${PATH}" \
	PYTHONPATH="${PYTHONPATH:-}" \
	"${cmd[@]}"

sudo is not required when run as root with -e NB_USER=root -e NB_UID=0 -e NB_GID=0.

Let me see what I can do...

[benz0li]

@bernt-matthias I patched the image once more.

Please pull glcr.b-data.ch/jupyterlab/r/base:test-hpc and try again.

[bernt-matthias]

There seem to be two files with unexpected permissions:

Executing: jupyter lab --allow-root
[I 2024-08-08 11:59:57.298 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-08-08 11:59:57.310 ServerApp] jupyter_server_mathjax | extension was successfully linked.
[I 2024-08-08 11:59:57.310 ServerApp] jupyter_server_proxy | extension was successfully linked.
[I 2024-08-08 11:59:57.317 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-08-08 11:59:57.324 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-08-08 11:59:57.324 ServerApp] jupyterlab_git | extension was successfully linked.
[I 2024-08-08 11:59:57.329 ServerApp] nbclassic | extension was successfully linked.
[I 2024-08-08 11:59:57.329 ServerApp] nbdime | extension was successfully linked.
[I 2024-08-08 11:59:57.334 ServerApp] notebook | extension was successfully linked.
[I 2024-08-08 11:59:57.343 ServerApp] Writing Jupyter server cookie secret to /home/root/.local/share/jupyter/runtime/jupyter_cookie_secret
[W 2024-08-08 11:59:57.344 ServerApp] notebook_shim | error linking extension: Permissions assignment failed for secure file: '/home/root/.local/share/jupyter/runtime/jupyter_cookie_secret'. Got '0o655' instead of '0o0600'.
    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/site-packages/traitlets/traitlets.py", line 632, in get
        value = obj._trait_values[self.name]
                ~~~~~~~~~~~~~~~~~^^^^^^^^^^^
    KeyError: 'cookie_secret'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/site-packages/jupyter_server/extension/manager.py", line 346, in link_extension
        extension.link_all_points(self.serverapp)
      File "/usr/local/lib/python3.12/site-packages/jupyter_server/extension/manager.py", line 228, in link_all_points
        self.link_point(point_name, serverapp)
      File "/usr/local/lib/python3.12/site-packages/jupyter_server/extension/manager.py", line 218, in link_point
        point.link(serverapp)
      File "/usr/local/lib/python3.12/site-packages/jupyter_server/extension/manager.py", line 140, in link
        linker(serverapp)
      File "/usr/local/lib/python3.12/site-packages/notebook_shim/nbserver.py", line 109, in _link_jupyter_server_extension
        members = diff_members(serverapp, nbapp)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/notebook_shim/nbserver.py", line 62, in diff_members
        m1 = public_members(obj1)
             ^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/notebook_shim/nbserver.py", line 56, in public_members
        members = inspect.getmembers(obj)
                  ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/inspect.py", line 608, in getmembers
        return _getmembers(object, predicate, getattr)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/inspect.py", line 586, in _getmembers
        value = getter(object, key)
                ^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/traitlets/traitlets.py", line 687, in __get__
        return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
                         ^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/traitlets/traitlets.py", line 635, in get
        default = obj.trait_defaults(self.name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/traitlets/traitlets.py", line 1897, in trait_defaults
        return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/traitlets/traitlets.py", line 1241, in __call__
        return self.func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyter_server/serverapp.py", line 1158, in _default_cookie_secret
        self._write_cookie_secret_file(key)
      File "/usr/local/lib/python3.12/site-packages/jupyter_server/serverapp.py", line 1167, in _write_cookie_secret_file
        with secure_write(self.cookie_secret_file, True) as f:
      File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
        return next(self.gen)
               ^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/site-packages/jupyter_core/paths.py", line 1007, in secure_write
        raise RuntimeError(msg)
    RuntimeError: Permissions assignment failed for secure file: '/home/root/.local/share/jupyter/runtime/jupyter_cookie_secret'. Got '0o655' instead of '0o0600'.
[I 2024-08-08 11:59:57.464 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-08-08 11:59:57.469 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-08-08 11:59:57.469 ServerApp] jupyter_server_mathjax | extension was successfully loaded.
[I 2024-08-08 11:59:57.512 ServerApp] jupyter_server_proxy | extension was successfully loaded.
[I 2024-08-08 11:59:57.516 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-08-08 11:59:57.618 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.12/site-packages/jupyterlab
[I 2024-08-08 11:59:57.618 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-08-08 11:59:57.620 LabApp] Extension Manager is 'pypi'.
[I 2024-08-08 11:59:57.674 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-08-08 11:59:57.678 ServerApp] jupyterlab_git | extension was successfully loaded.
[I 2024-08-08 11:59:57.691 ServerApp] nbclassic | extension was successfully loaded.
[I 2024-08-08 11:59:58.207 ServerApp] nbdime | extension was successfully loaded.
[I 2024-08-08 11:59:58.219 ServerApp] notebook | extension was successfully loaded.
[I 2024-08-08 11:59:58.220 ServerApp] Serving notebooks from local directory: /home/root
[I 2024-08-08 11:59:58.220 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-08-08 11:59:58.220 ServerApp] http://fee50fb4c9d3:8888/lab?token=3f9ffcf78e92c325035d3a158fc13976b6b8f4e03e1debf3
[I 2024-08-08 11:59:58.220 ServerApp]     http://127.0.0.1:8888/lab?token=3f9ffcf78e92c325035d3a158fc13976b6b8f4e03e1debf3
[I 2024-08-08 11:59:58.220 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|
                                                                           
Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

Traceback (most recent call last):
  File "/usr/local/bin/jupyter-lab", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/jupyter_server/extension/application.py", line 623, in launch_instance
    serverapp.start()
  File "/usr/local/lib/python3.12/site-packages/jupyter_server/serverapp.py", line 3120, in start
    self.start_app()
  File "/usr/local/lib/python3.12/site-packages/jupyter_server/serverapp.py", line 3021, in start_app
    self.write_server_info_file()
  File "/usr/local/lib/python3.12/site-packages/jupyter_server/serverapp.py", line 2827, in write_server_info_file
    with secure_write(self.info_file) as f:
  File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/jupyter_core/paths.py", line 1007, in secure_write
    raise RuntimeError(msg)
RuntimeError: Permissions assignment failed for secure file: '/home/root/.local/share/jupyter/runtime/jpserver-2.json'. Got '0o655' instead of '0o0600'.

[benz0li]

@bernt-matthias Try with

podman run --rm \
  -p 8888:8888 \
  -u root \
  -e NB_UMASK=0077 \
  -e NB_USER=root \
  -e NB_UID=0 \
  -e NB_GID=0 \
  -e NOTEBOOK_ARGS="--allow-root" \
  glcr.b-data.ch/jupyterlab/r/base:test-hpc

[benz0li]

Or (better)

podman run --rm \
  -p 8888:8888 \
  -u root \
  -e NB_USER=root \
  -e NB_UID=0 \
  -e NB_GID=0 \
  -e JUPYTER_ALLOW_INSECURE_WRITES=true \
  -e NOTEBOOK_ARGS="--allow-root" \
  glcr.b-data.ch/jupyterlab/r/base:test-hpc

[benz0li]

Or (alternatively)

podman run --rm \
  -p 8888:8888 \
  -u root \
  -e NB_USER=root \
  -e NB_UID=0 \
  -e NB_GID=0 \
  -e JUPYTER_RUNTIME_DIR=/directory/not/in/mounted/filesystem \
  -e NOTEBOOK_ARGS="--allow-root" \
  glcr.b-data.ch/jupyterlab/r/base:test-hpc

should /home/root be mounted somehow.

Cross reference:

• Kernel error when creating new jupyter notebook jupyter/notebook#5536 (comment)

[benz0li]

You have quite an edge case here... (Openshift?)

Because on a regular Linux machine, the mentioned files have permission 0o0600 inside the container – with both Docker and Podman.

[bernt-matthias]

Or (better)

podman run --rm \
  -p 8888:8888 \
  -u root \
  -e NB_USER=root \
  -e NB_UID=0 \
  -e NB_GID=0 \
  -e JUPYTER_ALLOW_INSECURE_WRITES=true \
  -e NOTEBOOK_ARGS="--allow-root" \
  glcr.b-data.ch/jupyterlab/r/base:test-hpc

This worked :)