[Wayland] Crash On Pipewire Screen Capture: obs-studio 31.0.0-1, pipewire 1.2.7
Hello Arch Linux GitLab Package Maintainers and Community, thank you for your consideration and time! I have done my utmost to be comprehensive, my apologies for any mistakes.
Description:
When trying to use OBS for screen capture with Wayland, the 'Screen Capture (PipeWire)' mode will cause the app to crash ("SIGTRAP") under Wayland or XWayland when a screen/app capture source is selected. This is caused by OBS Studio's libobs calling query_dmabuf_modifiers or query_dmabuf_formats (EGL-Registry) which returns 0 modifiers, and then trying to allocate 0 bytes with bzalloc or bmalloc. Which then triggers the call to bcrash added by OBS Studio to libobs in: GitHub obsproject/obs-studio "libobs/util: Crash on bmalloc(0) #11181". This behavior was primarily observed when using an AMD Radeon RX 400-series or 500-series GPU.
Additional info:
-
package version(s):
-
"obs-studio" (31.0.0-1), -
"pipewire" (1:1.2.7-1), -
"wireplumber" (0.5.7-1), -
"ffmpeg" (2:7.1-3), -
"glibc" (2.40+r16+gaa533d58ff-2), -
"gcc-libs" (14.2.1+r134+gab884fffe3fc-1), -
"mesa" (1:24.3.1-3), -
"pam" (1.7.0-1), -
"systemd-libs" (257-1), -
"wayland" (1.23.1-1), -
"plasma-desktop" (6.2.4-1), -
"plasma-meta" (6.1-1), -
"gnome-desktop" (1:44.1-1), -
"gnome-session" (47.0.1-1), -
"gnome-shell" (1:47.2-1), -
"gdm" (47.0-2), "linux" (6.12.4.arch1-1)
-
-
config and/or log files:
- The ALPM log file (filtered) to show the packages upgraded when the breaking changes in OBS Studio occured: breaking-pacman-Syu-ALPM-log.txt,
- Running OBS Studio in GDB with "SIGTRAP" crash (Wayland): zsh_gdb_obs_run_--verbose.txt,
- Running OBS Studio in GDB with "SIGTRAP" crash (XWayland): zsh__env__QT_QPA_PLATFORM_xcb__GDK_BACKEND_x11__gdb__obs.txt,
- Coredump file stack trace with GDB: zsh_coredumpctl_-1_gdb.txt,
- Running OBS Studio with
obs --verbosein Zsh (without trying to use Screen Capture, so it can exit normally): zsh_obs-studio_--verbose.txt, - OBS Studio
obs-studiopackage and dependency versions: zsh_pactree_-u_obs-studio_expac_-Q_n_v_-.txt,
-
link to upstream bug report, if any:
- Upstream issue: GitHub obsproject/obs-studio "Crash when initializing PipeWire screen sharing since 31.0.0 (GNOME Wayland) #11617".
- It's confirmed to be fixed with the upstream patch: GitHub obsproject/obs-studio "libobs-opengl: Avoid trying to allocate 0 byte on Linux #11618", but GitHub obsproject/obs-studio "libobs-opengl: Fix for formats with zero modifiers #11638" looks like a smaller patch file (untested) solution.
Steps to reproduce:
- Arch Linux (Installed according to Arch Wiki),
- Linux kernel (linux 6.12.4.arch1-1),
- Desktop Manager (gdm 47.0-2) and Wayland (wayland 1.23.1-1), with KDE Plasma 6 (plasma-desktop 6.2.4-1, plasma-meta 6.1-1) or GNOME 47 (gnome-desktop 1:44.1-1, gnome-session 47.0.1-1, gnome-shell 1:47.2-1)
- Pipewire (libpipewire 1:1.2.7-1, pipewire 1:1.2.7-1) with WirePlummer (libwireplumber 0.5.7-1, wireplumber 0.5.7-1)
- OBS Studio (obs-studio 31.0.0-1, ffmpeg 2:7.1-3, alsa-lib 1.2.13-1, glibc 2.40+r16+gaa533d58ff-2, gcc-libs 14.2.1+r134+gab884fffe3fc-1, systemd-libs 257-1, pam 1.7.0-1, pipewire-audio 1:1.2.7-1, mesa 1:24.3.1-3)
- Start a Wayland desktop session,
- Open OBS (through .desktop file, terminal
obs, or with GDB) - Click the "+" ("Add Source") button in the "Sources" section, and select "Screen Capture (PipeWire)" option,
- In the "Properties for 'Screen Capture (PipeWire)'" OBS pop-up window, click the "Open Selector" button. (This step will fail under an X.Org desktop session.)
- In the "Screen Sharing" GNOME/KDE pop-up dialog menu window, select any of the options under either the "screens" tab or the "windows" tab (for attached/virtual monitors or open application windows respectively),
- Immediate crash (SIGTRAP under GNU Debugger)
- Get a SystemD coredump file added to /var/lib/systemd/coredump