Test cases hang when creating CUDA streams

[Minamoto25]

When testing LibreCuda with a 4090 GPU, each test case in the tests folder hangs when creating CUDA streams. The call stack obtained using gdb is as follows:

(gdb) r
Starting program: /home/minamoto/LibreCuda/build/tests/memcopy/test_memcopy 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Device count: 2
^C
Program received signal SIGINT, Interrupt.
0x00007ffff7c35c9b in sched_yield () at ../sysdeps/unix/syscall-template.S:120
120     ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x00007ffff7c35c9b in sched_yield () at ../sysdeps/unix/syscall-template.S:120
#1  0x000055555558ee5f in __gthread_yield () at /usr/include/x86_64-linux-gnu/c++/10/bits/gthr-default.h:693
#2  0x000055555558eeee in std::this_thread::yield () at /usr/include/c++/10/thread:379
#3  0x000055555558c5e2 in NvCommandQueue::signalWaitCpu (this=0x5555555cddb0, pSignal=0x100f402000, 
    signalTarget=1) at /home/minamoto/LibreCuda/driverapi/src/cmdqueue.cpp:351
#4  0x000055555558c229 in NvCommandQueue::awaitExecution (this=0x5555555cddb0)
    at /home/minamoto/LibreCuda/driverapi/src/cmdqueue.cpp:274
#5  0x000055555558ba1c in NvCommandQueue::initializeQueue (this=0x5555555cddb0)
    at /home/minamoto/LibreCuda/driverapi/src/cmdqueue.cpp:154
#6  0x0000555555562981 in libreCuStreamCreate (pStreamOut=0x7fffffffdb68, flags=0)
    at /home/minamoto/LibreCuda/driverapi/src/librecuda.cpp:1388
#7  0x0000555555559e26 in main () at /home/minamoto/LibreCuda/tests/memcopy/main.cpp:30

Environment:

• GPU driver version: 535.183.01
• Operating System: Ubuntu 22.04.4 LTS, kernel version: 5.15.0-119-generic
• GPU Model: Two 4090 GPUs

Steps to Reproduce:

• Build LibreCuda
• change directory to build/tests/memcopy and run the test case
• Observe that test case hangs when creating CUDA streams

Is it possible that this is due to differences in the ioctl command for creating streams between the Ada and Ampere architectures?

I would appreciate your assistance with this matter.

[mikex86]

This might sound strange, but have you tried restarting your computer?
There is an unrecoverable state that you can get your GPU into with LibreCuda when you kill a program mid-execution. (probably some unclean shutdown that I haven't quite figured out how to do for real).
In this state, you should see a UVM-GPU1 process consume about 100% cpu when opening up top. In this state programs will no longer exit properly.

I cannot reproduce your issue with 550.54.15 (open version) and one RTX 4090.
Do all tests happen to fail this way?
If you just want the problem gone, I would recommend you install a newer open driver and see if the problem goes away.

If you want the problem fixed, there is a few things I can tell you that might to go wrong here if its not the rather typical unrecoverable GPU crash:

The commandqueue is built around

1. executing some commands
2. awaiting the execution of the commands.
   This works by incrementing the timelineSignal.value integer gpu-side via signalNotify().
   This enqueues a semaphore set command. The cpu will spin-wait / yield-wait until this value has reached its target.

When the GPU cannot execute some commands for whatever reason, the subsequent semaphore command will never execute, resulting in the cpu waiting forever.
So, the fact that it hangs in signalWaitCpu() is not surprising. It also does so at the first opportunity it gets (while creating the command queue).

This means any of these commands fail for whatever reason:

    // setup compute queue
    {
        LIBRECUDA_ERR_PROPAGATE(
                enqueue(makeNvMethod(1, NVC6C0_SET_OBJECT, 1), {ctx->device->compute_class}, COMPUTE)
        );
        LIBRECUDA_ERR_PROPAGATE(
                enqueue(
                        makeNvMethod(1, NVC6C0_SET_SHADER_LOCAL_MEMORY_WINDOW_A, 2),
                        {U64_HI_32_BITS(local_mem_window), U64_LO_32_BITS(local_mem_window)},
                        COMPUTE
                )
        );
        LIBRECUDA_ERR_PROPAGATE(
                enqueue(
                        makeNvMethod(1, NVC6C0_SET_SHADER_SHARED_MEMORY_WINDOW_A, 2),
                        {U64_HI_32_BITS(shared_mem_window), U64_LO_32_BITS(shared_mem_window)},
                        COMPUTE
                )
        );
        timelineCtr++; // note how this integer is incremented first cpu side to serve as the target
        LIBRECUDA_ERR_PROPAGATE(signalNotify(timelineSignal, timelineCtr, COMPUTE)); // then the gpu is instructed to increment the actual signal to this target.
    }
    LIBRECUDA_ERR_PROPAGATE(startExecution(COMPUTE)); // then the command buffer then gets dispatched for execution
    LIBRECUDA_ERR_PROPAGATE(awaitExecution()); // await execution waits for timelineSignal.value to reach timelineCtr. This does not happen for some reason and it hangs forever.

Here's a few known values when running test_memcpy and placing a breakpoint where the command queue is being setup.
ctx->device->compute_class should be 51648 on the RTX 4090. If this is 0 for some reason, the problem is in libreCuCtxCreate_v2 where the compute class selection fails for some reason. See librecuda.cpp:502 for the compute class selection. This is my top suspicion.
local_mem_window should be 4278190080
and shared_mem_window should be 4261412864.
However local_mem_window and shared_mem_window are effectively constant at this point, so I highly doubt they will differ.

My bet is on the compute_class selection. If this fails, this I know there is something different with respect to how the NV0080_CTRL_CMD_GPU_GET_CLASSLIST command behaves.

If you find the time, I would appreciate if you could share your findings.

[Minamoto25]

I've made the following additional attempts:

1. I used the top command and found that all my CPUs are idle.
2. Using gdb, I set a breakpoint at cmdqueue.cpp:153 (i.e., LIBRECUDA_ERR_PROPAGATE(startExecution(COMPUTE));) and printed the command parameters. The result was:

(gdb) b cmdqueue.cpp:153
Breakpoint 1 at 0x3892f: file /home/minamoto/LibreCuda/driverapi/src/cmdqueue.cpp, line 153.
(gdb) r
Starting program: /home/minamoto/LibreCuda/build/tests/compute_chronological_consistency/test_chronological_consistency 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Device count: 2
Device Name: NVIDIA GeForce RTX 4090
Num functions: 1
  function "write_float"

Breakpoint 1, NvCommandQueue::initializeQueue (this=0x5555555cbcb0) at /home/minamoto/LibreCuda/driverapi/src/cmdqueue.cpp:153
153         LIBRECUDA_ERR_PROPAGATE(startExecution(COMPUTE));
(gdb) p ctx->device->compute_class
$1 = 51648
(gdb) p local_mem_window
$2 = 4278190080
(gdb) p shared_mem_window
$3 = 4261412864

The parameters seem normal.
3. Continuing with single-step execution, the program still hangs after reaching cmdqueue.cpp:154 (i.e., LIBRECUDA_ERR_PROPAGATE(awaitExecution());).
Is it possible that there might be an issue with sending commands to the GPU? I will try restarting our server tomorrow, but I suspect that may not be the cause.

[mikex86]

Ok, that seems interesting. One way you can check whether any command actually gets executed by the GPU is by moving the following lines before the first command is enqueued:

        timelineCtr++;
        LIBRECUDA_ERR_PROPAGATE(signalNotify(timelineSignal, timelineCtr, COMPUTE));

Even though that introduces a bug, that way there is no command that could fail before the semaphore set command and it should run immediately, resulting in the signal reaching its target and the cpu should not hang.
If it still hangs, there is something going on with the GPU FIFO and how commands are submitted to it...

Your code should look like this afterwards:

    // setup compute queue
    {
        // insert begin
        timelineCtr++;
        LIBRECUDA_ERR_PROPAGATE(signalNotify(timelineSignal, timelineCtr, COMPUTE));
        // insert end

        LIBRECUDA_ERR_PROPAGATE(
                enqueue(makeNvMethod(1, NVC6C0_SET_OBJECT, 1), {ctx->device->compute_class}, COMPUTE)
        );
        LIBRECUDA_ERR_PROPAGATE(
                enqueue(
                        makeNvMethod(1, NVC6C0_SET_SHADER_LOCAL_MEMORY_WINDOW_A, 2),
                        {U64_HI_32_BITS(local_mem_window), U64_LO_32_BITS(local_mem_window)},
                        COMPUTE
                )
        );
        LIBRECUDA_ERR_PROPAGATE(
                enqueue(
                        makeNvMethod(1, NVC6C0_SET_SHADER_SHARED_MEMORY_WINDOW_A, 2),
                        {U64_HI_32_BITS(shared_mem_window), U64_LO_32_BITS(shared_mem_window)},
                        COMPUTE
                )
        );
    }
    LIBRECUDA_ERR_PROPAGATE(startExecution(COMPUTE));
    LIBRECUDA_ERR_PROPAGATE(awaitExecution());

[Minamoto25]

I modified my code according to the suggested method and recompiled it. I found that the program didn’t hang at the place where I inserted the code. Instead, it still hangs at awaitExecution after enqueuing the three commands. Does this indicate that there might be an issue with executing these three GPU commands? How should I further investigate this issue?

[mikex86]

Yes, this means something is wrong with how commands are enqueued.
The fact that it doesn't block where commands are enqueued is not surprising. All this does is basically construct a giant vector<uin32_t>. This data is then copied into the computeQueuePage.commandQueueSpace after submission and a specific address is written to to commence execution.
You can check a few more things:

• if computeQueuePage.commandQueueSpace is getting correctly allocated.

This happens here:

libreCudaStatus_t NvCommandQueue::initializeQueue() {
    // allocate compute command queue page
    {
        NvU64 cmdq_va{};
        LIBRECUDA_ERR_PROPAGATE(
                gpuAlloc(ctx, MAX_CMD_QUEUE_PAGE_SIZE * sizeof(NvU32), false, false, true, 0, &cmdq_va));
        computeQueuePage.commandQueueSpace = reinterpret_cast<NvU32 *>(cmdq_va);
    }

The error propagation should aready return if this fails, but you can check for obviously suspicious things like if the virtual address (cmdq_va) is still 0 after gpuAlloc should have already populated it.

But I suspect everything is fine here.

ctx->device->gpu_mmio also needs to get correctly created, but considering that none of the ioctls in libreCuCtxCreate_v2 that do that fail, this is a very strange bug.

I'm going to make a wild hypothesis here:
What if the driver doesn't like that the GPU mmio space is in the address space I declared as the heap?
Maybe the proprietary driver doesn't like this.
I have pushed some changes to master which moves the mmio space wherever mmap decides. Maybe this fixes your problem.

This is a wild guess, so if you find the time please confirm if that did something.

[Minamoto25]

I have pulled the latest code and rebuilt it, but the program still stops while waiting for the execution of the three commands.

• After the computeQueuePage allocation, I printed the value of cmdq_va, which was 68956463104.
• I set a breakpoint in the NvCommandQueue::submitToFifo function and printed the value of ctx->device->gpu_mmio, which was (NvU32 *) 0x7ffff7fab000.
• Additionally, when the program hangs, I checked nvidia-smi and found that memory has been successfully allocated, with the following output:

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   1251906      C   ./test_memcopy                               18MiB |
+---------------------------------------------------------------------------------------+

I hope this information helps with further analysis. Thank you!

[mikex86]

At this point I would recommend you upgrade to the at least 550.54.15 of the open-gpu-kernel-modules driver.
Given that you have 2 RTX 4090s, you could even install https://github.com/tinygrad/open-gpu-kernel-modules
This additionally implements P2P support even though it is not officially supported by Nvidia.
This is the driver I use and test on.
I unfortunately can't re-create every environment to see if something breaks and debugging this via GitHub issues is a bit too low bandwidth to be efficient...
I think its better to just say minimum driver supported driver is open-gpu-kernel-modules version 550.54.15
I know people have also had success with the proprietary drivers, but this was with version 555.58.02.
If it still fails with 550.54.15, please let me know because that would be highly unexpected.

[mikex86]

has this issue been resolved?

[Minamoto25]

We have switched to the 550.90.07-p2p version of the open-source GPU driver and are using the latest version of LibreCuda (commit aaf6bfb), but the test case still fails to run. The error message is as follows:

~/LibreCuda/build/tests$ ./async_kernels/test_async_kernels 
[LibreCuda Debug]: rm_ctrl failed with status: 31
[LibreCuda Debug]: /home/minamoto/LibreCuda/driverapi/src/librecuda.cpp:181: rm_ctrl failed with status 999
[CUDA ERROR] at file /home/minamoto/LibreCuda/tests/async_kernels/main.cpp:20: LIBRECUDA_ERROR_UNKNOWN

./compute_chronological_consistency/test_chronological_consistency 
[LibreCuda Debug]: rm_ctrl failed with status: 31
[LibreCuda Debug]: /home/minamoto/LibreCuda/driverapi/src/librecuda.cpp:181: rm_ctrl failed with status 999
[CUDA ERROR] at file /home/minamoto/LibreCuda/tests/compute_chronological_consistency/main.cpp:19: LIBRECUDA_ERROR_UNKNOWN

./memcopy/test_memcopy 
[LibreCuda Debug]: rm_ctrl failed with status: 31
[LibreCuda Debug]: /home/minamoto/LibreCuda/driverapi/src/librecuda.cpp:181: rm_ctrl failed with status 999
[CUDA ERROR] at file /home/minamoto/LibreCuda/tests/memcopy/main.cpp:16: LIBRECUDA_ERROR_UNKNOWN

It seems that the NV0000_CTRL_CMD_GPU_GET_ID_INFO_V2 ioctl call has failed, which is very confusing to me. Do you have any debugging ideas to help pinpoint the specific issue?

[mikex86]

try running nvidia-smi first. This looks very similar to the failure you get when the nvidia driver isn't initialized yet.

[Minamoto25]

I checked, and our drivers should be installed.

~/LibreCuda$ nvidia-smi 
Fri Oct 18 07:14:01 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:B8:00.0 Off |                  Off |
| 30%   29C    P8              4W /  450W |   21947MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:D8:00.0 Off |                  Off |
| 30%   30C    P8              9W /  450W |       4MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   3048366      C   .../usr/anaconda3/envs/vllm/bin/python      21930MiB |
+-----------------------------------------------------------------------------------------+

I also tested it on another server, but it still didn't work. The output is as follows:

~/LibreCuda/build/tests/async_kernels$ ./test_async_kernels 
Device count: 2
Device Name: NVIDIA GeForce RTX 3080
Num functions: 0
[CUDA ERROR] at file /home/minamoto/LibreCuda/tests/async_kernels/main.cpp:72: LIBRECUDA_ERROR_NOT_FOUND

This server has the proprietary driver installed, with the following version:

~/LibreCuda/build/tests/async_kernels$ nvidia-smi 
Fri Oct 18 15:23:16 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        Off |   00000000:04:00.0 Off |                  N/A |
|  0%   31C    P8             18W /  320W |       2MiB /  10240MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3080        Off |   00000000:83:00.0 Off |                  N/A |
|  0%   35C    P8             10W /  320W |       2MiB /  10240MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[mikex86]

The ./test_async_kernels error is a small screw up from moving files around. It's fixed now.
Most tests should pass on the 3080, however some DMA related things do not work correctly with the proprietary driver and might result in hangs. This is a known issue and one of the reasons why you generally shouldn't use the propriery driver.
With regards to the 4090 error, this error is basically nonsensical.
It's not actually the driver >initialization error<, that one would be a LIBRECUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE.
But I have seen this error before. I've managed to get my GPU into this exact state before, where all RM api calls would fail with status 31 (=NV_ERR_INVALID_ARGUMENT).
It was caused by me trying out some things on the command queue that crashed the GPU a bit too hard, and the result was this error. A reboot fixed it. Surprisingly, normal CUDA applications still worked fine, but LibreCuda always failed to initialize while the GPU was in this state.

If a reboot doesn't help, I would suggest you do yourself a favor and create a fresh linux installation, preferably with 550.54.15-p2p as opposed to what you have now. That is just a step of caution that I would really recommend here. I would consider it unlikely to be the cause, but when debugging you want to eliminate as many variables as possible.

Unfortunately there is not much else I can do here.

[mikex86]

I just had your exact issue, which seems to happen randomly on longrunning systems unfortunately, where the driver just de-initialized and nvidia-smi was no longer working. LibreCUDA will throw an LIBRECUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE error in this situtation because the driver is not running.
This unfortunately seems to be a driver issue. After reinstalling the driver, nvidia-smi works, but librecuda still fails with your error:
rm_ctrl status failed with 31, resulting in LIBRECUDA_ERROR_UNKNOWN = 999.
What fixed it was a reboot. After said reboot, I ran nvidia-smi to initialize the driver first, then ran a librecuda application and then it worked again.
This is seemingly an unfortunate driver issue, where the driver will de-intialize/crash for strange reasons, even while the system is doing nothing.
I suspect this issue to be exclusive to self built nvidia drivers however. Need to investigate more what is causing this, but for now, this is a way to solve the issue.