SDL apps not working when using SDL_VIDEODRIVER=wayland

[infapi00]

Describe the bug

With all the applications we tried, SDL applications are not working when using SDL_VIDEODRIVER=wayland.

This is desiderable as in some cases it works better when the application tries to set a fullscreen resolution different to the desktop resolution. We also understand that it should be the preferred option, as wayfire is a Wayland compositor.

We initially suspected of a problem with the SDL wayland backend, but then we tested other wayland compositors (weston, sway on rpi4, gnome-shell on a laptop) and it was working fine there.

To Reproduce

This was initially detected with openarena, as part of a work to test different opengl/vulkan applications performance on different scenarios (I'm one of the developers of the rpi4 v3d/v3dv driver). openarena is a well known ioquake3-based project, and usually part of the official repositories of most distributions. So in order to test it:

1. Install openarena: apt-get install openarena
2. Execute it using SDL_VIDEODRIVER=wayland. For convenience this is the script we use (it automatically loads a demo, and shows the final FPS);

#!/bin/bash

# Removing ioq3.pid, to avoid the "Looks like a failure on last execution" modal dialog
rm -rf /home/pi/.openarena/baseoa/ioq3.pid
export WAYLAND_DISPLAY="wayland-1"
export SDL_VIDEODRIVER=wayland
openarena +set cl_renderer opengl1 +set r_mode -1 +set r_customwidth 1280 + set r_customheight 720 +set r_fullscreen 1 +timedemo 1 +set demodone "quit" +set demoloop1 "demo demo088-test1.dm_71; set nextdemo vstr demodone" +vstr demoloop1

Then I started to search for smaller SDL applications. So for example:

• https://github.com/xyproto/sdl2-examples: this repository has basically the same SDL app on different languages.
• https://github.com/libsdl-org/SDL/tree/main/test: test directory on SDL main repository. I executed testgl2 and testgles2

Expected behavior

• With openarena the expected outcome is the game properly loading a recorded game, setting it on fullscreen, see during some minutes people shot each other, and finish with a line showing the full run FPS. Based on gdb (backtrace below), what happens is that it gets blocked
• With sdl2-examples it is expected to get a window showing a grumpy cat for some seconds and close. What happens is that nothing appears on screen, but at least it properly closes.
• With testgl2 and testgles2 from the SDL repository, it is expected to get a colored cube rotating. Nothing appears on screen, but at least it doesn't seems to get blocked, because using gdb -p, it stops at different places all the time.

Screenshots or stacktrace

As mentioned, on the case of openarena it seems to get blocked, because it doesn't show anything, and using gdb -p, it is always at the same place. The backtrace at that moment:

(gdb) bt
#0  0x0000007f87acde94 in __GI___poll (fds=fds@entry=0x7fedcbf378, nfds=nfds@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000007f7c677830 in wl_display_poll (display=0x559a50dd30, events=1) at ../src/wayland-client.c:1914
#2  wl_display_dispatch_queue (queue=<optimized out>, display=<optimized out>) at ../src/wayland-client.c:1987
#3  wl_display_dispatch_queue (display=0x559a50dd30, queue=0x559a5814f0) at ../src/wayland-client.c:1960
#4  0x0000007f7beb17ec in dri2_wl_swap_buffers_with_damage (disp=0x559a51fba0, draw=0x559a5c3c80, rects=0x0, n_rects=0) at ../src/egl/drivers/dri2/platform_wayland.c:1570
#5  0x0000007f7beb1bd8 in dri2_wl_swap_buffers (disp=0x559a51fba0, draw=0x559a5c3c80) at ../src/egl/drivers/dri2/platform_wayland.c:1671
#6  0x0000007f7bea17ac in dri2_swap_buffers (disp=0x559a51fba0, surf=0x559a5c3c80) at ../src/egl/drivers/dri2/egl_dri2.c:1890
#7  0x0000007f7be8fb54 in eglSwapBuffers (dpy=0x559a51fba0, surface=0x559a5c3c80) at ../src/egl/main/eglapi.c:1436
#8  0x0000007f88055450 in Wayland_GLES_SwapWindow (_this=0x559a512010, window=<optimized out>) at /home/pi/mesa/source/SDL/src/video/wayland/SDL_waylandopengles.c:166
#9  0x0000007f7cb030a0 in GLimp_EndFrame () at code/sdl/sdl_glimp.c:1102
#10 0x0000007f7cac6adc in RB_SwapBuffers (data=data@entry=0x7f7ce60810) at code/renderergl1/tr_backend.c:1094
#11 0x0000007f7cac6c28 in RB_ExecuteRenderCommands (data=0x7f7ce60810, data@entry=0x7f7ce5d4f0) at code/renderergl1/tr_backend.c:1128
#12 0x0000007f7caccfb4 in R_IssueRenderCommands (runPerformanceCounters=qtrue) at code/renderergl1/tr_cmds.c:96
#13 RE_EndFrame (frontEndMsec=0x0, backEndMsec=0x0) at code/renderergl1/tr_cmds.c:474
#14 0x0000005573f0988c in SCR_UpdateScreen () at code/client/cl_scrn.c:588
#15 0x0000005573eebe60 in CL_CgameSystemCalls (args=0x7fedcbf728) at code/client/cl_cgame.c:471
#16 0x0000005573f5d728 in VM_DllSyscall (arg=<optimized out>) at code/qcommon/vm.c:352
#17 0x0000007f6a665e44 in CG_Init (serverMessageNum=<optimized out>, serverCommandSequence=<optimized out>, clientNum=<optimized out>) at code/cgame/cg_main.c:2137
#18 0x0000007f6a666678 in vmMain
    (command=<optimized out>, arg0=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=<optimized out>, arg5=<optimized out>, arg6=<optimized out>, arg7=1945118872, arg8=1949833200, arg9=1950811408, arg10=-305399408, arg11=1945119692) at code/cgame/cg_main.c:50
#19 0x0000005573f5ea14 in VM_Call (vm=0x5575a668b8 <vmTable+216>, callnum=callnum@entry=0) at code/qcommon/vm.c:891
#20 0x0000005573eecd08 in CL_InitCGame () at code/client/cl_cgame.c:745
#21 0x0000005573f02898 in CL_DownloadsComplete () at code/client/cl_main.c:2145
#22 0x0000005573f02bcc in CL_InitDownloads () at code/client/cl_main.c:2327
#23 0x0000005573f07d38 in CL_ParseGamestate (msg=msg@entry=0x7fedcbfff0) at code/client/cl_parse.c:549
#24 0x0000005573f0838c in CL_ParseServerMessage (msg=msg@entry=0x7fedcbfff0) at code/client/cl_parse.c:911
#25 0x0000005573f005e4 in CL_ReadDemoMessage () at code/client/cl_main.c:980
#26 CL_PlayDemo_f () at code/client/cl_main.c:1152
#27 0x0000005573f19834 in Cbuf_Execute () at code/qcommon/cmd.c:248
#28 0x0000005573f1df7c in Com_Frame () at code/qcommon/common.c:3157
#29 0x0000005573eeaf4c in main (argc=<optimized out>, argv=<optimized out>) at code/sys/sys_main.c:767

Wayfire version

All this was detected using system Wayfire (0.7.5-1), but hoping that this issue were fixed upstream, I also tested it from the git repository (so version 0.8.0, HEAD at becc957)

Just in case it was an SDL issue, I tested with SDL from system (2.26.5), and from the git repository (SDL2 branch, HEAD at f032e8c19165aeee6f9b47e82fb56809d4aeae1e)

Something else?

We are willing to debug, and even try to modify/fix the code ourselves, but we would really appreciate some guidance.

[soreau]

I can reproduce this with testgl2 and testgles2 from the SDL repository at least, works fine in sway. The curious thing I notice is from wayfire log, it says

II 27-08-23 19:35:22.606 - [src/core/seat/input-method-relay.cpp:226] Enabling text input, but input method is gone
II 27-08-23 19:35:22.607 - [src/core/seat/input-method-relay.cpp:249] Committing text input, but input method is gone
II 27-08-23 19:35:22.607 - [src/core/seat/input-method-relay.cpp:249] Committing text input, but input method is gone
II 27-08-23 19:35:22.646 - [src/view/xdg-shell/xdg-toplevel-view.cpp:31] new xdg_shell_stable surface: /opt/sdl/libexec/installed-tests/SDL3/testgl app-id: testgl
II 27-08-23 19:35:23.365 - [src/core/seat/input-method-relay.cpp:124] Disabling text input, but input method is gone

and no window appears. A hunch is that SDL might be waiting for some protocol call that wayfire isn't making related to these messages. A quick look shows that wayfire callback for on_input_method_new in wayfire/src/core/seat/input-method-relay.cpp isn't firing but I'm not sure why. It seems sway connects to the same signal, and it apparently gets called there.

[ammen99]

The two SDL demo clients you linked work on my laptop, I wonder what is different in our setups (and I have verified they are running native wayland).

As a first step, it would be interesting to see the output of the SDL app with WAYLAND_DEBUG=1, to see what is happening. Also, try starting Wayfire with -d txn -d txni -d views and share the log (it would show us whether there is any transactions going on for the new view).

Also, try disabling as many Wayfire plugins as possible to make sure this is not caused by a specific plugin.

[ammen99]

@soreau Provided additional logs on IRC, so this issue has now been narrowed down to decorations. The bug reproduces only with the following conditions:

• SDL needs to be compiled with libdecor support (for client-side decorations)
• Wayfire needs to be configured with preferred_decoration_mode=client

It seems that first SDL tries to create a window for SSD. When Wayfire communicates that it prefers CSD, SDL destroys its toplevel and attempts to use Libdecor .. which hangs. I'm still trying to figure out why.

[ammen99]

I figured it out, the bug is indeed in Wayfire. The following patch is enough to fix it: http://ix.io/4EKv

I'll merge it in master shortly.

The problem was that SDL says a hard 'I need SSD' but Wayfire does not take that into account and forces CSD.

[ammen99]

With the fix, SDL apps will work correctly, however, it may be worth reporting this to the SDL devs as well. SDL appears to have a code path for this which seemingly does not work .. Of course, I'm not sure how to trigger that path without having a bug in the compositor.

[soreau]

Also, @wb9688 pointed out that another workaround is setting SDL_VIDEO_WAYLAND_ALLOW_LIBDECOR=0

[infapi00]

Hey people!

Thanks a lot!! it was a really quick fix. I was able to confirm both the Wayfire fix (#1871) and the workaround. FWIW, the possibility of having a workaround is really important, so again, thanks a lot.

With the fix, SDL apps will work correctly, however, it may be worth reporting this to the SDL devs as well. SDL appears to have a code path for this which seemingly does not work .. Of course, I'm not sure how to trigger that path without having a bug in the compositor.

Ok, will try to create an issue about that on SDL.

@soreau Provided additional logs on IRC, so this issue has now been narrowed down to decorations.

FWIW, sorry for not interacting more on IRC. Hard to do that with the summer holidays and three kids at home.

[infapi00]

Ok, will try to create an issue about that on SDL.

Done:

libsdl-org/SDL#8173