RPi5: banklow 1 causes massive performance regression with non-numa kernels

[HiassofT]

Describe the bug

Software decoding H264 videos on a non-numa kernel performs really poorly (fps almost halved) with current rpi-eeprom versions (having SDRAM_BANKLOW=1 as default) compared to earlier ones which used SDRAM_BANKLOW=3.

While that is no huge problem for HD videos it is for 4k where previously lots of 4kp24 videos could be software decoded smoothly.

Manually adding SDRAM_BANKLOW=3 to the eeprom config mitigates that.

Steps to reproduce the behaviour

Use ffmpeg to software decode a video, eg Big Buck Bunny, check the decoding time and observe the reported framerate and decoding speed:

time ffmpeg -i bbb_sunflower_1080p_30fps_normal.mp4 -an -f null /dev/null

To (temporarily) disable numa add numa=fake=1 to cmdline.txt

Using SDRAM_BANKLOW=3 and numa disabled (i.e. the state of about half a year ago) as a baseline I get
fps=217 and 1m29s real time reported

Using SDRAM_BANKLOW=3 and numa enabled I get
fps=253 and 1m16s (that's +16% more fps, about the expected performance increase from fake numa)

Using default rpi-eeprom-config (i.e. SDRAM_BANKLOW=1) and numa enabled I also get
fps=253 and 1m16s

Using default rpi-eeprom-config (i.e. SDRAM_BANKLOW=1) and numa disabled I only get
fps=121 and 2m37s (that's -45% compared to baseline)

Device (s)

Raspberry Pi 5

Bootloader configuration.

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=1
BOOT_ORDER=0xf421
NET_INSTALL_ENABLED=0

System

RPiOS Bookworm lite, fully upated.

/etc/rpi-issue:

Raspberry Pi reference 2023-10-10
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 962bf483c8f326405794827cce8c0313fd5880a8, stage4

bootloader_version:

2025/07/17 17:25:12
version a668b6e6edce3274de221324b93cb8741e4a7f7c (release)
timestamp 1752769512
update-time 1754066025
capabilities 0x0000007f

kernel:

Linux raspberrypi 6.12.34+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.12.34-1+rpt1~bookworm (2025-06-26) aarch64 GNU/Linux

As a side note:
I initially noticed this on LibreELEC (running kernel 6.12.40) where we currently use 512MB CMA (which in turn disables numa on 4GB and smaller models) to avoid display corruption issues.
I suspect though other distributions or workloads might be affected as well

Bootloader logs

No response

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

No response

[popcornmix]

This is expected.
BANKLOW=1 + NUMA gives the best performance and is the default and recommended.
BANKLOW=1 + no NUMA results in bad performance and should not be used.

It is unavoidable - there is nothing there to fix.
If NUMA is not possible then BANKLOW=3 is a workaround, but not recommended.

Large CMA doesn't play nicely with NUMA and shouldn't be used on a Pi5 - the iommus should allow system memory to be used in all cases (I know it is currently being used to work around a bug, but that is the real issue that needs solving).

[HiassofT]

The impact of this change in bootloader default config is that running (a bit) older distributions on an RPi5 with current bootloader can result in a potentially massive performance degradation.

Think about multiboot systems, people doing regression tests with older releases etc.

At the very least this should be properly documented and explicitly warned about.