[Wayland] Crash On Pipewire Screen Capture: obs-studio 31.0.0-1, pipewire 1.2.7
Hello Arch Linux GitLab Package Maintainers and Community, thank you for your consideration and time! I have done my utmost to be comprehensive, my apologies for any mistakes.
# Description:
When trying to use OBS for screen capture with Wayland, the 'Screen Capture (PipeWire)' mode will cause the app to crash ("SIGTRAP") under Wayland or XWayland when a screen/app capture source is selected. This is caused by OBS Studio's `libobs` calling `query_dmabuf_modifiers` or `query_dmabuf_formats` (EGL-Registry) which returns `0` modifiers, and then trying to allocate `0` bytes with `bzalloc` or `bmalloc`. Which then triggers the call to `bcrash` added by OBS Studio to `libobs` in: [GitHub obsproject/obs-studio "libobs/util: Crash on bmalloc(0) #11181"](https://github.com/obsproject/obs-studio/pull/11181). This behavior was primarily observed when using an AMD Radeon RX 400-series or 500-series GPU.
# Additional info:
See: [Arch Linux Forums Index » Applications & Desktop Environments » "[Wayland] Crash On Screen Capture: obs-studio 31.0.0-1, pipewire 1.2.7"](https://bbs.archlinux.org/viewtopic.php?id=301775)
* ##### package version(s):
* `"obs-studio" (31.0.0-1)`,
* `"pipewire" (1:1.2.7-1)`,
* `"wireplumber" (0.5.7-1)`,
* `"ffmpeg" (2:7.1-3)`,
* `"glibc" (2.40+r16+gaa533d58ff-2)`,
* `"gcc-libs" (14.2.1+r134+gab884fffe3fc-1)`,
* `"mesa" (1:24.3.1-3)`,
* `"pam" (1.7.0-1)`,
* `"systemd-libs" (257-1)`,
* `"wayland" (1.23.1-1)`,
* `"plasma-desktop" (6.2.4-1)`,
* `"plasma-meta" (6.1-1)`,
* `"gnome-desktop" (1:44.1-1)`,
* `"gnome-session" (47.0.1-1)`,
* `"gnome-shell" (1:47.2-1)`,
* `"gdm" (47.0-2)`,
* `"linux" (6.12.4.arch1-1)`
* ##### config and/or log files:
* The ALPM log file (filtered) to show the packages upgraded when the breaking changes in OBS Studio occured: [breaking-pacman-Syu-ALPM-log.txt](/uploads/030f411a6989f5371e3497b408f5cfd4/breaking-pacman-Syu-ALPM-log.txt),
* Running OBS Studio in GDB with "SIGTRAP" crash (Wayland): [zsh_gdb_obs_run_--verbose.txt](/uploads/b1d58f2b28a48a3a033f5829f3fdad15/zsh_gdb_obs_run_--verbose.txt),
* Running OBS Studio in GDB with "SIGTRAP" crash (XWayland): [zsh__env__QT_QPA_PLATFORM_xcb__GDK_BACKEND_x11__gdb__obs.txt](/uploads/895104f18b9080a5806a9c605a88e34d/zsh__env__QT_QPA_PLATFORM_xcb__GDK_BACKEND_x11__gdb__obs.txt),
* Coredump file stack trace with GDB: [zsh_coredumpctl_-1_gdb.txt](/uploads/792f14a199d45324d89b4304e5f0acae/zsh_coredumpctl_-1_gdb.txt),
* Running OBS Studio with `obs --verbose` in Zsh (without trying to use Screen Capture, so it can exit normally): [zsh_obs-studio_--verbose.txt](/uploads/9f7e8de58092ef355016d25ee66f5e75/zsh_obs-studio_--verbose.txt),
* OBS Studio `obs-studio` package and dependency versions: [zsh_pactree_-u_obs-studio_expac_-Q_n_v_-.txt](/uploads/7661d8e9277f136b89b70f0a8f06a69b/zsh_pactree_-u_obs-studio_expac_-Q_n_v_-.txt),
* ##### link to upstream bug report, if any:
* Upstream issue: [GitHub obsproject/obs-studio "Crash when initializing PipeWire screen sharing since 31.0.0 (GNOME Wayland) #11617"](https://github.com/obsproject/obs-studio/issues/11617).
* It's [confirmed](https://github.com/obsproject/obs-studio/issues/11617#issuecomment-2545999324) to be fixed with the upstream patch: [GitHub obsproject/obs-studio "libobs-opengl: Avoid trying to allocate 0 byte on Linux #11618"](https://github.com/obsproject/obs-studio/pull/11618), but [GitHub obsproject/obs-studio "libobs-opengl: Fix for formats with zero modifiers #11638"](https://github.com/obsproject/obs-studio/pull/11638) looks like a smaller patch file (untested) solution.
# Steps to reproduce:
* Arch Linux (Installed according to Arch Wiki),
* Linux kernel (linux 6.12.4.arch1-1),
* Desktop Manager (gdm 47.0-2) and Wayland (wayland 1.23.1-1), with KDE Plasma 6 (plasma-desktop 6.2.4-1, plasma-meta 6.1-1) or GNOME 47 (gnome-desktop 1:44.1-1, gnome-session 47.0.1-1, gnome-shell 1:47.2-1)
* Pipewire (libpipewire 1:1.2.7-1, pipewire 1:1.2.7-1) with WirePlummer (libwireplumber 0.5.7-1, wireplumber 0.5.7-1)
* OBS Studio (obs-studio 31.0.0-1, ffmpeg 2:7.1-3, alsa-lib 1.2.13-1, glibc 2.40+r16+gaa533d58ff-2, gcc-libs 14.2.1+r134+gab884fffe3fc-1, systemd-libs 257-1, pam 1.7.0-1, pipewire-audio 1:1.2.7-1, mesa 1:24.3.1-3)
* Start a Wayland desktop session,
* Open OBS (through .desktop file, terminal `obs`, or with GDB)
* Click the "+" ("Add Source") button in the "Sources" section, and select "Screen Capture (PipeWire)" option,
* In the "Properties for 'Screen Capture (PipeWire)'" OBS pop-up window, click the "Open Selector" button. (This step will fail under an X.Org desktop session.)
* In the "Screen Sharing" GNOME/KDE pop-up dialog menu window, select any of the options under either the "screens" tab or the "windows" tab (for attached/virtual monitors or open application windows respectively),
* Immediate crash (SIGTRAP under GNU Debugger)
* Get a SystemD coredump file added to /var/lib/systemd/coredump
issue