Add fast-track to skip uninstall/install if NVIDIA driver modules present #454

karthikvetrivel · 2025-10-02T15:55:48Z

This PR is a part of this endeavor:

GPU Driver container should avoid re-installing drivers on spurious container restarts

Relevant PRs:

Check if NVIDIA kernel modules are loaded to avoid modprobe gpu-operator#1746
Add shouldSkipUninstall to avoid GPU driver teardown on restart k8s-driver-manager#103

…sent Signed-off-by: Karthik Vetrivel <[email protected]>

…s and fix scenario handling Signed-off-by: Karthik Vetrivel <[email protected]>

…nge detection Signed-off-by: Karthik Vetrivel <[email protected]>

ubuntu22.04/nvidia-driver

tariq1890 · 2025-11-20T16:11:54Z

Thanks @karthikvetrivel, this looks very promising! I was curious as to why ubuntu 24.04 wasn't included in the changeset

tariq1890 · 2025-11-20T16:13:47Z

ubuntu22.04/nvidia-driver

+		_mount_rootfs
+
+		# Ensure persistence daemon is running
+		_ensure_persistence_running


The name of this method can be shortened to this

Suggested change

_ensure_persistence_running

_ensure_persistenced

karthikvetrivel · 2025-11-20T16:21:23Z

Thanks @karthikvetrivel, this looks very promising! I was curious as to why ubuntu 24.04 wasn't included in the changeset

Thanks! Yes, I picked one Ubuntu nvidia-driver version to test on and get feedback before porting to others.

rhel9/nvidia-driver

Signed-off-by: Karthik Vetrivel <[email protected]>

cdesiniotis

Just reviewed the rhel9 directory.

rhel9/nvidia-driver

rhel9/ocp_dtk_entrypoint

ubuntu22.04/nvidia-driver

Signed-off-by: Karthik Vetrivel <[email protected]>

rhel9/common.sh

cdesiniotis · 2025-12-11T19:53:20Z

rhel9/nvidia-driver


-    trap "make -s -j ${MAX_THREADS} SYSSRC=/lib/modules/${KERNEL_VERSION}/build clean > /dev/null" EXIT
+    # Skip cleanup trap for DTK builds - modules are copied after this function returns
+    if [ "${PACKAGE_TAG:-}" != "builtin" ]; then


Question - could you elaborate on why this change is needed? What exactly does the clean make target do?

To my understanding, make clean removes the compiled .ko module files. For DTK builds, the modules need to persist after _create_driver_package() returns because ocp_dtk_entrypoint copies them to a shared volume afterward. If the cleanup trap runs, the modules are deleted before the copy happens.

Feel free to push back on my understanding here.

Are you suggesting that this change is needed in general and is not related to the fast-track install code path?

Yes, this isn't related to a fast track optimization. This path isn't hit on a fast track reinstall. I can move these changes to a follow-up PR if preferred!

Yes, I would prefer if this change (and any other changes not related to the fast-track optimization) were in a separate PR with a clear description of the problem addressed.

rhel9/nvidia-driver

cdesiniotis · 2025-12-11T20:08:53Z

rhel9/nvidia-driver

    nvidia-topologyd
 }

+_ensure_persistence() {


Where is this being called? I don't see this method being invoked in the rhel9 scripts.

Updated, replaced a prior _start_daemons method.

ubuntu22.04/nvidia-driver

cdesiniotis · 2025-12-11T20:12:29Z

ubuntu22.04/nvidia-driver

+            _unmount_rootfs
+            _update_package_cache
+            _resolve_kernel_version || exit 1
+            _install_prerequisites


Question -- is installing prerequisite packages actually required in this case? If yes, which packages are required for the userspace-only install?

You're right, the prerequisites (kernel headers, kernel image, kernel modules) are specifically for kernel module compilation, which doesn't happen here—we run nvidia-installer --no-kernel-module. Will remove.

Let's also remove the call to _update_package_cache while we are at it. If we aren't installing any new packages then we don't need to update the package cache.

Updated. Also removed in RHEL9.

ubuntu22.04/nvidia-driver

cdesiniotis · 2025-12-12T02:25:12Z

rhel9/nvidia-driver

+    fi
+
+    _mount_rootfs
+    _ensure_persistence


Why are we only starting the nvidia-persistenced daemon for this code path? Shouldn't we be restarting all of the daemons that we bring up in _start_daemons()?

My intuition tells me that none of the daemons should still be running at this point -- that is, none of the daemons should still be running after a previously running driver container terminates / restarts. If my intuition is correct, then couldn't we unconditionally call _start_daemons() here? Let me know if you have observed different behavior.

i think you're right. Whenever we hit _ensure_persistence in this code path, we log persistence not found. I'll start the daemons here (and mirror the changes to Ubuntu 22.04).

Speaking of daemons, but I noticed that we don't ever stop nvidia-topologyd in _unload_driver() but we stop every other daemon. Is there a reason why?

Speaking of daemons, but I noticed that we don't ever stop nvidia-topologyd in _unload_driver() but we stop every other daemon. Is there a reason why?

No reason AFAIK. We should also stop it in _unload_driver().

cdesiniotis · 2025-12-12T02:35:06Z

ubuntu22.04/nvidia-driver

+            _unmount_rootfs
+            _update_package_cache
+            _resolve_kernel_version || exit 1
+            _install_prerequisites


Let's also remove the call to _update_package_cache while we are at it. If we aren't installing any new packages then we don't need to update the package cache.

Signed-off-by: Karthik Vetrivel <[email protected]>

Add fast-track to skip uninstall/install if NVIDIA driver modules pre…

7c178f6

…sent Signed-off-by: Karthik Vetrivel <[email protected]>

This was referenced Oct 2, 2025

Add shouldSkipUninstall to avoid GPU driver teardown on restart NVIDIA/k8s-driver-manager#103

Open

Check if NVIDIA kernel modules are loaded to avoid modprobe NVIDIA/gpu-operator#1746

Open

karthikvetrivel marked this pull request as draft October 2, 2025 16:06

feat: implement userspace-only reinstall for non-clean driver restart…

a8dbb15

…s and fix scenario handling Signed-off-by: Karthik Vetrivel <[email protected]>

karthikvetrivel marked this pull request as ready for review October 17, 2025 15:13

karthikvetrivel marked this pull request as draft November 6, 2025 14:18

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from 0bb2e4c to b107ac5 Compare November 7, 2025 14:00

Store driver config state and compare on restart to enable config cha…

ba7e6de

…nge detection Signed-off-by: Karthik Vetrivel <[email protected]>

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from b107ac5 to ba7e6de Compare November 7, 2025 20:35

tariq1890 reviewed Nov 20, 2025

View reviewed changes

ubuntu22.04/nvidia-driver Outdated Show resolved Hide resolved

tariq1890 reviewed Nov 20, 2025

View reviewed changes

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from 4650182 to bf0bb88 Compare November 20, 2025 16:48

tariq1890 reviewed Nov 20, 2025

View reviewed changes

rhel9/nvidia-driver Outdated Show resolved Hide resolved

Add support for OpenShift 14.04

0a036ed

Signed-off-by: Karthik Vetrivel <[email protected]>

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from bf0bb88 to 0a036ed Compare November 20, 2025 17:46

cdesiniotis reviewed Nov 20, 2025

View reviewed changes

rhel9/nvidia-driver Outdated Show resolved Hide resolved

rhel9/nvidia-driver Outdated Show resolved Hide resolved

rhel9/ocp_dtk_entrypoint Outdated Show resolved Hide resolved

rhel9/ocp_dtk_entrypoint Show resolved Hide resolved

karthikvetrivel requested review from cdesiniotis and tariq1890 November 20, 2025 19:32

shivamerla reviewed Nov 21, 2025

View reviewed changes

ubuntu22.04/nvidia-driver Outdated Show resolved Hide resolved

Refactor fast-path logic in OpenShift

d4a6dff

Signed-off-by: Karthik Vetrivel <[email protected]>

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from 0fb8195 to d4a6dff Compare December 1, 2025 15:20

Extract common code into reusable helper functions

b660caa

Signed-off-by: Karthik Vetrivel <[email protected]>

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from f0d0d41 to b660caa Compare December 8, 2025 19:55

karthikvetrivel marked this pull request as ready for review December 8, 2025 20:58

cdesiniotis reviewed Dec 11, 2025

View reviewed changes

cdesiniotis reviewed Dec 12, 2025

View reviewed changes

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from 4d6bb0f to aae8035 Compare December 12, 2025 14:51

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from aae8035 to 883b5a9 Compare December 12, 2025 18:44

refactor: address PR review comments for driver install scripts

34e89fc

Signed-off-by: Karthik Vetrivel <[email protected]>

karthikvetrivel force-pushed the eat/avoid-reinstall-gpu-container branch from 883b5a9 to 34e89fc Compare December 16, 2025 21:11

karthikvetrivel requested a review from cdesiniotis December 17, 2025 19:26

Add fast-track to skip uninstall/install if NVIDIA driver modules present #454

Are you sure you want to change the base?

Add fast-track to skip uninstall/install if NVIDIA driver modules present #454

Conversation

karthikvetrivel commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tariq1890 commented Nov 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karthikvetrivel commented Nov 20, 2025

Uh oh!

Uh oh!

cdesiniotis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karthikvetrivel Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karthikvetrivel Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

karthikvetrivel commented Oct 2, 2025 •

edited

Loading

karthikvetrivel Dec 11, 2025 •

edited

Loading

karthikvetrivel Dec 11, 2025 •

edited

Loading