Test GPU (Zotac Nvidia GeForce GT 710)

[geerlingguy]

I'm currently testing this GPU (Zotac Nvidia GeForce GT 710), so until that's done, I wanted an issue open to track where that work is going on.

Currently I have the driver installed, and everything sorta works, but I've had to do a lot to get it to that state. Right now I'm installing the CUDA tools to see if I can get some programming done on the GPU. The X window system did not seem to like the driver for some reason.

Anyways, some relevant links to follow:

• Gist: Increase the BAR memory address space for PCIe devices on CM4
• Gist: Setting up the Nvidia GeForce GT 710 on Raspberry Pi Compute Module 4
• Pi Forum Topic: BAR space for PCIe allocation on CM4?
• Nvidia bug report: Drivers fail to load with NVRM: GPU 0004:01:00.0: RmInitAdapter failed! (0x25:0x54:1211)

[geerlingguy]

Can't get simple CUDA examples to work either, as they trigger dmesg failure notices—see https://gist.github.com/geerlingguy/9f1510ab028e68b712381520308db2af#gistcomment-3500278

[geerlingguy]

Just posting as a point of reference, 96Boards mentioned getting an Nvidia 710 working with one of their own boards: https://www.96boards.org/blog/oxalis-gpu/

The reason behind choosing the Nvidia GT710 was that it was the only GPU I had ever seen running on Arm on the 96Boards DeveloperBox. Part of the reason it works well is that the DeveloperBox as SBBA/SBSA compliant UEFI bios that supports ACPI, and runs qemu to for emulating x86 and running the driver for the Nvidia GPU.

Looking at the DeveloperBox specs, though, it is a vastly different architecture than the current Pi:

PCIe: Two of 1 Lane per slot, one 16 Lanes slot: 1 for PCIe Graphics card, 1 for PCIe IO Extension, 2x USB3.0/2.0 + 2x SATA

And here's the time (15:14) in their Oxalis Enterprise Demo where Sahaj revealed the GT 710 working on their board: https://youtu.be/rPHahahbheo?t=914

[geerlingguy]

One other thing I'd like to try at some point—I just received a 1x to 16x adapter with external power, so I can supply power to the board without leeching off the IO Board's 12v 2A power supply. Might work better, probably won't :P

[geerlingguy]

It looks like some users of the ROCKPRO64 (using the RK3399, which has x4 PCIe) also attempted and failed to use a GPU on that flavor of the ARM SoC: Considering buying and question on the PCIe slot

[geerlingguy]

Some good reading here: PCI BARs and other means of accessing the GPU

[NostalgiaRunner]

Greetings Jeff,
I recently saw your GPU on CM4 video and blog post and had an idea for a way to get either dGPU working. I don't know if this would be worth the rabbit trail, but If you're willing to try something a little less than on-metal, my idea is to use QEMU. QEMU utilizes the Linux kernel's KVM hypervisor and is able to emulate X86 and X86_64 operating systems on ARM64 hardware. In my previous ventures with ProxMox (VMWare ESXI's open-source QEMU based competition), I was able to passthrough a cheap GT 430 GPU to my very old BIOS age Xeon server. If you wish to try it, I think it'd be possible. Atleast on paper it is. Run QEMU on a headless minimal Pi image, create an x86 VM with your choice of x86 image, and configure the QEMU VM to passthrough the GPU device. Here's a link to the guide I used for ProxMox. Though I have my doubts that it'll be 100% the same process or success level. https://pve.proxmox.com/wiki/Pci_passthrough#GPU_Passthrough While this wouldn't be running the GPU on-metal, I think it would still be an incredible feat if the VM is able to utilize the card, as this would make the idea of custom ARM based desktop motherboards for x86 work a little more appealing. Some final notes are... there's a potential software dilemma. QEMU is and has been able to run x86 VMs on the RPi. However, it is not able to address all 8GB of RAM for the 8GB model, for use across all VMs. ESXI ARM Fling on the other hand is a free??? beta testing product at this time, and is able to address all 8GB on the host. However, it is my recollection that ESXI ARM Fling is unable to run x86 VMs, and potentially unable to passthrough PCIe devices. These are my thoughts and findings, and I hope they may be of use to you. Cheers!

[geerlingguy]

Over in the raspberrypi/linux project, it looks like this commit (raspberrypi/linux@54db4b2) has increased the default BAR allocation to 1GB by default—nice!

[geerlingguy]

Before I throw in the towel on this thing, I'm going to try to use the external powered PCIe switch to see if it makes any difference. I'll first try with Nouveau, by cross-compiling the kernel from my Mac.

[geerlingguy]

I'm not even seeing the card when it's plugged into the switch:

$ lspci
00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 20)
01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G304 EL/SL PCIe2 3-Port/4-Lane Packet Switch (rev 05)

No errors in dmesg.

[geerlingguy]

And if I plug it straight in, I get pcie link down each time.

[geerlingguy]

And using my other powered PCIe adapter I get pcie link down as well. Darn. No dice with this card currently.

[geerlingguy]

All right, spinning this card up again once more using the expanded BAR space suggested in this comment: #6 (comment)

And to make it so I can monitor what happens, I created /etc/modprobe.d/blacklist-nouveau.conf with the contents blacklist nouveau. That way I can power on the Pi, wait for successful boot, then run sudo modprobe nouveau.