Crazy Taxi 3: High Roller - Inconsistent performance issues especially in the Vulkan backend (excessive shader generation)

[ehw]

Title

https://xemu.app/titles/53450004/#Crazy-Taxi-3-High-Roller

SE-004 [53450004]

Bug Description

This title suffers from some odd performance issues while in any stage (particularly the Glitter Oasis and Small Apple stages). Depending on where you are in the stage and what's occurring on the screen, the game starts losing performance. The problem might be remedied if you have a great CPU, but is more evident in slower CPUs (like my 5800x). As you drive around the stages, the game will start dropping frames every few seconds or so, sometimes at a second or two at a time. This is significantly worse with the Vulkan backend, where the framerates is some areas are halved. With the vulkan backend, QUEUE_SUBMIT_AUX, TEX_UPLOAD, and QUEUE_SUBMIT_4 shoot up from 0 to 500 or so rapidly within every second.

This is NOT remedied with the shader cache, as the game appears to be generating way too many shaders that any game should be creating.

To recreate:
1.) Load the game
2.) Wait past the company logos for the title screen, then press start.
3.) Select "Glitter Oasis"
4.) Pick "Play By Normal Rules"
5.) Select a character
6.) Wait for the stage to load up
7.) Drive a little bit toward the center of the stage, around the four way. Depending on your PC specs, you might see performance issues right away.
8.) Change the video backend to the Vulkan one.

Once you switch to the Vulkan backend, performance tanks considerably.

OpenGL:

Vulkan:

Here's a video showing OpenGL performance on a PC spec similar to mine:
https://youtu.be/BMl8HDkmg7Y?t=254

Expected Behavior

As far as I know, the game should not have any performance/frame time issues. The game on real hardware doesn't display any kind of frame dip and seems to always run at a consistent 60fps. See a video below of the game running on real hardware, presumably:

https://www.youtube.com/watch?v=9Dn_-0cNxGc

Some people with higher spec'd PCs seem to be able to handle the game a bit better. Here's a video of someone playing with a Intel i9-13900K. It's not perfect, but performs better than the 5800X video:

https://youtu.be/77OvR5p1zng?t=238

xemu Version

Version: 0.8.78
Branch: master
Commit: 7d9739d
Date: Sat Jun 21 00:45:10 UTC 2025

System Information

CPU: AMD Ryzen 7 5800X 8-Core Processor
OS Platform: Windows
OS Version: 22H2
Manufacturer: NVIDIA Corporation
GPU Model: NVIDIA GeForce RTX 3080/PCIe/SSE2
Driver: 4.0.0 NVIDIA 576.80
Shader: 4.00 NVIDIA via Cg compiler

Additional Context

While I know this may seem to indicate that the performance issues are because of the user's PC specs and just in general Vulkan related (since Vulkan does have some performance issues in some games already), this seems to be different. The issue might seem to be indirectly related to the extremely large amount of shaders that Xemu wants to create for this game. It creates literally thousands of shaders at least for the OpenGL backend as evident by the large amount of folders in the shader cache that are made for this game alone. It doesn't seem coincidental that most of the folders are made specifically when playing Glitter Oasis and Small Apple. Based on what I see from the geometry pop in, it's almost as if Xemu is creating a shader for almost every 'piece' of geometry on the map for some reason. Xemu shouldn't be doing this, and I can't imagine that even if all the shaders are compiled that Xemu is performant with handling thousands of shaders at the same time.

[abaire]

Any chance you could get a pgraph trace (the nv2a events) when it starts generating tons of shaders? https://xemu.app/docs/dev/debug/guest/#tracing

That would hopefully tell us if the game itself is generating tons of different programmable shaders or if it's changing parameters that are causing xemu to generate unique shaders.

[ehw]

Here you go, I ran this for about 8 hours as soon as the game started up Glitter Oasis. In that time it generated 35 shaders. The log is big so hopefully it answers some things...

ct3_nv2a_pgraph_ogl.zip

[abaire]

A quick first pass analyis:

Typical frame results indicate that the game primarily uses the fixed function pipeline and is not doing anything obviously pathological with the limited programmable pipe use. It does use a fair number of pixel shaders, but it's still only around 30 of them (almost certainly with a high cache hit rate)

== Frame summary: ================
  1145 draws
    Fixed function: 1115  Programmable: 30
      1 unique vertex shader programs
    27 unique combiners

I'll take a look at how it's configuring the fixed function pipeline to see if it's doing something that results in tons of unique xemu shaders.

[abaire]

After processing the log, I'm not seeing an obvious smoking gun.

-- File summary --------
38 frames total
Unique combiners: 28
Unique programmable shaders: at least 1
Unique fixed function shaders: at least 32

It's entirely possible that I've missed cases that generate additional shaders, but it seems like it'd have to be something that would generate unique shaders on every frame to add up to a very large number of shaders.

I do own this particular game; can you share a save game or instructions on the fastest way to reproduce so I can try to step through it in the debugger?

[ehw]

After processing the log, I'm not seeing an obvious smoking gun.

-- File summary --------
38 frames total
Unique combiners: 28
Unique programmable shaders: at least 1
Unique fixed function shaders: at least 32

It's entirely possible that I've missed cases that generate additional shaders, but it seems like it'd have to be something that would generate unique shaders on every frame to add up to a very large number of shaders.

I do own this particular game; can you share a save game or instructions on the fastest way to reproduce so I can try to step through it in the debugger?

For me you just have to load up Glitter Oasis on the main menu and drive around. I was capturing the trace when the stage initially loaded so it might've been just building the shaders for just the starting area. But if you drive just a little bit up the road it just starts building massive amounts of shaders.

[abaire]

Ah, thanks, I didn't realize it was just a main menu entry :)

I'm not seeing the issue on M3 mac (which does not support shader caching). I see some shaders compiled when entering the stage, but driving around I'm not seeing any additional ones generated. I'll be able to check on Linux/Windows in a few days to see if there's something wrong with the cache check that's causing it to re-save (rather than re-compile) the shaders unnecessarily.

If you completely nuke the cache (or just rename it so you can move it back), and only run CT3, does the problem still happen?

[RetroGamingX]

This game won't start on my system (windows)...

[ehw]

Ah, thanks, I didn't realize it was just a main menu entry :)

I'm not seeing the issue on M3 mac (which does not support shader caching). I see some shaders compiled when entering the stage, but driving around I'm not seeing any additional ones generated. I'll be able to check on Linux/Windows in a few days to see if there's something wrong with the cache check that's causing it to re-save (rather than re-compile) the shaders unnecessarily.

If you completely nuke the cache (or just rename it so you can move it back), and only run CT3, does the problem still happen?

Yep, the problem still occurs. I thought maybe this might be exclusive to NVidia but it's also happening with AMD drivers as well on my ROG Ally. Just running the game for about a half an hour letting it go through the post title screen demo sequences generated 3000+ shader folders for both NVidia/AMD. :(

System Information (ROG Ally X):
CPU: AMD Ryzen Z1 Extreme
OS Platform: Windows
OS Version: 24H2
Manufacturer: ATI Technologies Inc.
GPU Model: AMD Radeon Graphics
Driver: 4.0.0 Core Profile Context 25.5.1.250417
Shader: 4.60

[abaire]

If you look at the Debug > Video > Advanced section, does the SHADER_GEN line show a non-zero number continuously?

[ehw]

If you look at the Debug > Video > Advanced section, does the SHADER_GEN line show a non-zero number continuously?

Nope, it seems to be generating shaders upon loading up a stage on a clean cache and then small handfuls as you run through the stage. The SHADER_BIND seems consistently large though? Always hovering around 300-500.

[abaire]

SHADER_BIND is normal, it just means that the active shader has been switched. In my tests it's generally in the same range, which seems reasonable given your pgraph log indicating ~1000 draws or so per frame. It doesn't explain why you'd end up with thousands of cached shaders though.

Is there anything interesting in your xemu.log file or printed to the console? In particular things like nv2a: Failed to write shader list entry or nv2a: Failed to write shader binary file.

I'm also curious if performance improves dramatically if you disable shader caching, though that would not be a reasonable solution.

[medievil1]

lots more debug info on vulkan versus ogl (windows)
did notice que submit 4 is active in vulkan and not ogl....
fps was pretty much 60 in ogl and 27-30'ish in vulkan....

of course, render passes could be the issue too, on that level for vulkan it is in the 290 range (first level is 2)

edit
noticed in the original vulkan screen cap up top renderpass is 340

[mborgerson]

I've debugged the main Vulkan issue, fixed in #2294.

@ehw What is the evidence for the claim of excessive shader generation? Is it just the number of files listed in the xemu on-disk shader cache? I don't see excessive active shader generation in the profiles shown in the issue description, though the number of binds is quite high.

[Adad-G]

I just took a look at it, and after several minutes and driving through much of that scenario, I can say that it's currently just the opposite. Performance improves with time, and except for the very first few moments in each session and occasional slight frame drops, performance is almost perfect.

Is Perhaps this update the cause: #2324?

Version: 0.8.96
Branch: master
CPU: AMD Ryzen 5 5600X 6-Core Processor
OS Platform: Linux
OS Version: Linux Mint 22.1
Manufacturer: NVIDIA Corporation
GPU Model: NVIDIA GeForce RTX 3070/PCIe/SSE2
Driver: 4.0.0 NVIDIA 575.64.03
API: Vulkan