Nvidia GPU passthrough issue

[luxnr]

Qubes OS release

Qubes OS 4.2

Brief summary

Trying to setup GPU passthrough on my newly built desktop computer (motherboard : MSI B650, GPU: MSI RTX 5070). The qube boots and the GPU is listed in lspci, but nvidia-smi shows "no devices were found" error.

I've checked that the passthrough works with KVM (same guest, Fedora host).

Steps to reproduce

1. Install Qubes 4.2.4 and update dom0
2. Hide GPU and associated audio device from dom0
3. Create a fedora HVM and attach both devices (with permissive=True and no-strict-reset=True)
4. Install akmod-nvidia and xorg-x11-drv-nvidia-cuda in HVM
5. Reboot
6. Run nvidia-smi

Expected behavior

Print GPU informations

Actual behavior

"No devices were found"

Additional information

In dom0

$ cat /proc/cmdline
placeholder root=/dev/mapper/qubes_dom0-root ro rd.luks.uuid=luks-21e9460e-7bff-4bf9-8b96-541bacb7b6eb rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles 6.6.77-1.qubes.fc37.x86_64 x86_64 rhgb quiet usbcore.authorized_default=0 rd.qubes.hide_pci=01:00.0,01:00.1,02:00.0,11:00.3,11:00.4,12:00.0

$ lspci -nnk
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2f04] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:5323]
	Kernel driver in use: pciback
	Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:2f80] (rev a1)
	Subsystem: NVIDIA Corporation Device [10de:0000]
	Kernel driver in use: pciback
	Kernel modules: snd_hda_intel

In gpu qube

$ lspci -nnk
00:06.0 VGA compatible controller [0300]: NVIDIA Corporation GB205 [GeForce RTX 5070] [10de:2f04] (rev a1)
    Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:5323]
    Kernel driver in use: nvidia
    Kernel modules: nouveau, nova_core, nvidia_drm, nvidia
00:07.0 Audio device [0403]: NVIDIA Corporation Device [10de:2f80] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:0000]
    Kernel driver in use: snd_hda_intel
    Kernel modules: nova_core, snd_hda_intel

$ nvidia-smi
No devices were found

$ sudo dmesg
[    4.639647] nvidia: loading out-of-tree module taints kernel.
[    4.639655] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    4.742397] vif vif-0 enX0: renamed from eth0
[    4.742708] nvidia-nvlink: Nvlink Core is being initialized, major device number 237

[    4.745428] nvidia 0000:00:06.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=io+mem
[    4.809931] snd_hda_intel 0000:00:07.0: Disabling MSI
[    4.843360] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:07.0/sound/card0/input7
[    4.845393] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:07.0/sound/card0/input8
[    4.848111] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:07.0/sound/card0/input9
[    4.848885] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:07.0/sound/card0/input10
[    4.924466] EXT4-fs (xvda2): mounted filesystem b4503358-f7db-42e5-9e6f-5afcccdb7b61 r/w with ordered data mode. Quota mode: none.
[    5.684786] NET: Registered PF_QIPCRTR protocol family
[    6.026721] RPC: Registered named UNIX socket transport module.
[    6.026724] RPC: Registered udp transport module.
[    6.026726] RPC: Registered tcp transport module.
[    6.026727] RPC: Registered tcp-with-tls transport module.
[    6.026728] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    7.656386] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  575.64  Release Build  (dvs-builder@U22-I3-AF02-18-4)  Tue Jun 10 18:53:09 UTC 2025
[    7.815475] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  575.64  Release Build  (dvs-builder@U22-I3-AF02-18-4)  Tue Jun 10 18:43:58 UTC 2025
[    8.029491] NVRM: kfspProcessCommandResponse_GH100: FSP response reported error. Task ID: 0x1 Command type: 0x14 Error code: 0x177
[    8.029503] NVRM: kfspSendBootCommands_GH100: Sent following content to FSP: 
[    8.029505] NVRM: kfspSendBootCommands_GH100: version=0x2, size=0x35c, gspFmcSysmemOffset=0x129a80000
[    8.029506] NVRM: kfspSendBootCommands_GH100: frtsSysmemOffset=0x0, frtsSysmemSize=0x0
[    8.029508] NVRM: kfspSendBootCommands_GH100: frtsVidmemOffset=0x1c00000, frtsVidmemSize=0x100000
[    8.029509] NVRM: kfspSendBootCommands_GH100: gspBootArgsSysmemOffset=0x1418b4000
[    8.029510] NVRM: kfspSendBootCommands_GH100: FSP boot cmds failed. RM cannot boot.
[    8.029513] NVRM: kfspDumpDebugState_GB202: FSP microcode v4.76
[    8.029515] NVRM: kfspDumpDebugState_GB202: GPU 0000:00:06
[    8.029518] NVRM: kfspDumpDebugState_GB202: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(0) = 0x177
[    8.029520] NVRM: kfspDumpDebugState_GB202: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(1) = 0x0
[    8.029523] NVRM: kfspDumpDebugState_GB202: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(2) = 0x0
[    8.029525] NVRM: kfspDumpDebugState_GB202: NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_2(3) = 0xa
[    8.030705] NVRM: nvCheckOkFailedNoLog: Check failed: Failure: Generic Error [NV_ERR_GENERIC] (0x0000FFFF) returned from kfspSendBootCommands_HAL(pGpu, pKernelFsp) @ kernel_gsp_gh100.c:856
[    8.031121] NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[    8.031124] NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[    8.031131] NVRM: gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110094,  regvalue: 0xbadf4100,  error code: Unknown SYS_PRI_ERROR_CODE
[    8.031147] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[    8.032726] NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x62:0x40:1941)
[    8.037384] NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0
[    8.048182] [drm] [nvidia-drm] [GPU ID 0x00000006] Loading driver
[    8.112082] NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[    8.112088] NVRM: _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
[    8.112095] NVRM: gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110094,  regvalue: 0xbadf4100,  error code: Unknown SYS_PRI_ERROR_CODE
[    8.112105] NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[    8.113160] NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x62:0x40:1941)
[    8.117994] NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0

(the last part repeats)

[marmarek]

Check if you get any messages in dom0 about this device, especially something about "write to read-only configuration space". If so, post that message here for further debugging, but also you can try if "permissive" option helps: https://www.qubes-os.org/doc/how-to-use-pci-devices/#permissive

[luxnr]

Here are messages about the device :

$ sudo dmesg | grep "0000:01:00"
[    0.645850] pci 0000:01:00.0: [10de:2f04] type 00 class 0x030000 PCIe Legacy Endpoint
[    0.645861] pci 0000:01:00.0: BAR 0 [mem 0xf0000000-0xf3ffffff]
[    0.645871] pci 0000:01:00.0: BAR 1 [mem 0xf800000000-0xfbffffffff 64bit pref]
[    0.645881] pci 0000:01:00.0: BAR 3 [mem 0xfc00000000-0xfc01ffffff 64bit pref]
[    0.645888] pci 0000:01:00.0: BAR 5 [io  0xf000-0xf07f]
[    0.645894] pci 0000:01:00.0: ROM [mem 0xf4000000-0xf407ffff pref]
[    0.645916] pci 0000:01:00.0: Enabling HDA controller
[    0.645969] pci 0000:01:00.0: PME# supported from D0 D3hot
[    0.646003] pci 0000:01:00.0: VF BAR 0 [mem 0x00000000-0x0003ffff 64bit pref]
[    0.646004] pci 0000:01:00.0: VF BAR 0 [mem 0x00000000-0x0003ffff 64bit pref]: contains BAR 0 for 1 VFs
[    0.646012] pci 0000:01:00.0: VF BAR 2 [mem 0x00000000-0x0fffffff 64bit pref]
[    0.646012] pci 0000:01:00.0: VF BAR 2 [mem 0x00000000-0x0fffffff 64bit pref]: contains BAR 2 for 1 VFs
[    0.646019] pci 0000:01:00.0: VF BAR 4 [mem 0x00000000-0x01ffffff 64bit pref]
[    0.646020] pci 0000:01:00.0: VF BAR 4 [mem 0x00000000-0x01ffffff 64bit pref]: contains BAR 4 for 1 VFs
[    0.646083] pci 0000:01:00.0: 252.048 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x16 link at 0000:00:01.1 (capable of 504.112 Gb/s with 32.0 GT/s PCIe x16 link)
[    0.646174] pci 0000:01:00.1: [10de:2f80] type 00 class 0x040300 PCIe Endpoint
[    0.646185] pci 0000:01:00.1: BAR 0 [mem 0xf4080000-0xf4083fff]
[    2.678658] pci 0000:01:00.0: vgaarb: setting as boot VGA device
[    2.678658] pci 0000:01:00.0: vgaarb: bridge control possible
[    2.678658] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    2.685718] pci 0000:01:00.0: VF BAR 2 [mem size 0x10000000 64bit pref]: can't assign; no space
[    2.685720] pci 0000:01:00.0: VF BAR 2 [mem size 0x10000000 64bit pref]: failed to assign
[    2.685721] pci 0000:01:00.0: VF BAR 4 [mem size 0x02000000 64bit pref]: can't assign; no space
[    2.685721] pci 0000:01:00.0: VF BAR 4 [mem size 0x02000000 64bit pref]: failed to assign
[    2.685722] pci 0000:01:00.0: VF BAR 0 [mem 0xf40c0000-0xf40fffff 64bit pref]: assigned
[    2.686008] pci 0000:01:00.1: extending delay after power-on from D3hot to 20 msec
[    2.686023] pci 0000:01:00.1: D0 power state depends on 0000:01:00.0
[    4.832369] pciback 0000:01:00.0: xen_pciback: seizing device
[    4.936373] pciback 0000:01:00.1: xen_pciback: seizing device
[    4.936418] pciback 0000:01:00.1: enabling device (0000 -> 0002)
[ 1268.567525] pciback 0000:01:00.0: xen_pciback: enabling permissive mode configuration space accesses!
[ 1268.567533] pciback 0000:01:00.0: xen_pciback: permissive mode is potentially unsafe!
[ 1268.571564] pciback 0000:01:00.1: xen_pciback: enabling permissive mode configuration space accesses!
[ 1268.571566] pciback 0000:01:00.1: xen_pciback: permissive mode is potentially unsafe!
[ 1268.571782] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[ 1268.571896] pciback 0000:01:00.0: registering for 15
[ 1268.574204] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[ 1268.574323] pciback 0000:01:00.1: registering for 15
[ 1268.856765] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[ 1608.635568] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[ 1608.635688] pciback 0000:01:00.0: registering for 17
[ 1608.637865] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[ 1608.637981] pciback 0000:01:00.1: registering for 17
[ 1608.936820] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[ 6473.192130] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[ 6473.192228] pciback 0000:01:00.0: registering for 20
[ 6473.194343] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[ 6473.194451] pciback 0000:01:00.1: registering for 20
[ 6473.486850] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[ 6594.839510] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[ 6594.839636] pciback 0000:01:00.0: registering for 22
[ 6594.841926] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[ 6594.842052] pciback 0000:01:00.1: registering for 22
[ 6595.137474] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[ 6640.255835] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[ 6640.255938] pciback 0000:01:00.0: registering for 24
[ 6640.257956] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[ 6640.258067] pciback 0000:01:00.1: registering for 24
[ 6640.548748] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[ 6696.918895] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[ 6696.919012] pciback 0000:01:00.0: registering for 26
[ 6696.921068] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[ 6696.921159] pciback 0000:01:00.1: registering for 26
[ 6697.223927] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[ 7849.430603] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[ 7849.430726] pciback 0000:01:00.0: registering for 28
[ 7849.432909] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[ 7849.433012] pciback 0000:01:00.1: registering for 28
[ 7849.734339] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[11597.603466] pciback 0000:01:00.0: xen_pciback: vpci: assign to virtual slot 0
[11597.603556] pciback 0000:01:00.0: registering for 32
[11597.605652] pciback 0000:01:00.1: xen_pciback: vpci: assign to virtual slot 0 func 1
[11597.605767] pciback 0000:01:00.1: registering for 32
[11597.882954] pciback 0000:01:00.0: enabling device (0000 -> 0003)

[edit : after enabling SR-IOV in the BIOS, the "can't assign / failed to assign" messages no longer appear (but the issue is unchanged)]

I have already set permissive=True.

$ xenstore-ls -fp /local/domain/0/backend/pci
[...]
/local/domain/0/backend/pci/32 = ""   (n0)
/local/domain/0/backend/pci/32/0 = ""   (n0,r32)
/local/domain/0/backend/pci/32/0/frontend = "/local/domain/32/device/pci/0"   (n0,r32)
/local/domain/0/backend/pci/32/0/frontend-id = "32"   (n0,r32)
/local/domain/0/backend/pci/32/0/online = "1"   (n0,r32)
/local/domain/0/backend/pci/32/0/state = "4"   (n0,r32)
/local/domain/0/backend/pci/32/0/domain = "gpu_fedora-dm"   (n0,r32)
/local/domain/0/backend/pci/32/0/key-0 = "0000:01:00.0"   (n0,r32)
/local/domain/0/backend/pci/32/0/dev-0 = "0000:01:00.0"   (n0,r32)
/local/domain/0/backend/pci/32/0/opts-0 = "msitranslate=0,power_mgmt=0,permissive=1,rdm_policy=relaxed"   (n0,r32)
/local/domain/0/backend/pci/32/0/state-0 = "3"   (n0,r32)
/local/domain/0/backend/pci/32/0/num_devs = "2"   (n0,r32)
/local/domain/0/backend/pci/32/0/vdev-0 = "0000:00:00.00"   (n0,r32)
/local/domain/0/backend/pci/32/0/root-0 = "0000:00"   (n0,r32)
/local/domain/0/backend/pci/32/0/root_num = "1"   (n0,r32)
/local/domain/0/backend/pci/32/0/key-1 = "0000:01:00.1"   (n0,r32)
/local/domain/0/backend/pci/32/0/dev-1 = "0000:01:00.1"   (n0,r32)
/local/domain/0/backend/pci/32/0/opts-1 = "msitranslate=0,power_mgmt=0,permissive=1,rdm_policy=relaxed"   (n0,r32)
/local/domain/0/backend/pci/32/0/state-1 = "3"   (n0,r32)
/local/domain/0/backend/pci/32/0/vdev-1 = "0000:00:00.01"   (n0,r32)

[Atrate]

Install akmod-nvidia and xorg-x11-drv-nvidia-cuda in HVM

How did you manage to install akmod-nvidia? Last time I checked on Fedora 42, the compilation just failed completely on Linux 6.15.

[luxnr]

How did you manage to install akmod-nvidia? Last time I checked on Fedora 42, the compilation just failed completely on Linux 6.15.

Did you use a standalone cloned from the template or installed from scratch ? It works when installed from scratch. I also tried cloning the Fedora 41 template, I think I remember akmods indeed failed to build so I used official Nvidia installer instead, and had the same issue as described above.

[3nprob]

I am not familiar with akmod but fwiw passthrough works fine (older nvidia cards though) here on fedora-41-based with:

• nvidia-gpu-firmware
• xorg-x11-drv-nouveau
• libva-nvidia-driver (probably irrelevant)

(installed from scratch using qusal)

Other things you can try:

• Disable/enable resize-BAR in firmware
• Try different PCI slots if you have more

[luxnr]

I am not familiar with akmod but fwiw passthrough works fine (older nvidia cards though) here on fedora-41-based with:

• nvidia-gpu-firmware
• xorg-x11-drv-nouveau
• libva-nvidia-driver (probably irrelevant)

nouveau is not yet compatible with 5000 series : https://nouveau.freedesktop.org/CodeNames.html/

• Disable/enable resize-BAR in firmware

I've tried that.

• Try different PCI slots if you have more

I will try the other PCI x16 slot (which is limited to x4) when I have access to the computer again.