AMD GPU & Cycles: Memory access fault by GPU node-1 Reason: Page not present or supervisor privilege
Description:
Blender 4.0.2-10 crashes when using HIP Cycles (rx 6800, rocm 5.6.1) The same does not happen with blender 4.0.2-9, same file, same version of rocm, same configurations.
The error in the terminal is:
Memory access fault by GPU node-1 (Agent handle: 0x7092c2daa300) on address 0x708fde7ac000. Reason: Page not present or supervisor privilege. IOT instruction (core dumped) blender
Additional info:
tested with 'classroom' blender sample file from:
https://download.blender.org/demo/test/classroom.zip
with regards to rocm 5.6.1, all new versions (5.7.1 and 6.0) have caused problems and crashes for users -- rocm 5.6.1 is stable across multiple applications (tested pytorch and blender up to 4.0.2-9). See below recent forum post on the matter:
https://bbs.archlinux.org/viewtopic.php?id=290424
Also see bug reports on rocm:
https://bugs.archlinux.org/task/80326#comment223998
- package version(s): blender 4.0.2-10
Steps to reproduce:
- open classroom.blend file
- go to 'Render' Panel
- set Render Engine to Cycles
- set Feature set to Supported
- set Device to GPU Compute (make sure to select HIP back end using the AMD GPU in the preferences)
- go to 'Sampling' options (within the 'Render' panel)
- check Noise Threshold check box in Sampling-> Viewport
- check Denoise checkbox in Sampling-> Viewport
- check Noise Threshold check box in Sampling-> Render
- check Denoise checkbox in Sampling-> Render
- Change 'Viewport Shading' to 'Rendered'
- if the segfault does not trigger immediately, try switching between 'Viewport Shading' 'Rendered' and 'Material Preview'
- The segfault triggers for me after switching between 'Viewport Shading' 'Rendered' and 'Material Preview' a couple of times
- the issue triggers for me straightforwardly on a number of other personal files I am not in the position of sharing. File classroom.blend is reproducing the issue as described in points above. Possibly other files from Blender Files do the same (not the standard start-up cube)
Activity
added priority3-normal severity4-low statusunconfirmed labels
assigned to @morganamilo, @toolybird, and @thurstylark
crashes
When reporting crashes, nobody can help unless you provide a backtrace that includes debugging information. A simplified guide is here.
BTW, this error message is all over the web, so it could easily be an upstream problem due to the recent ROCm 6 upgrade, and not an Arch packaging issue.
added statuswaiting-input label and removed statusunconfirmed label
assigned to @svenstaro, @freswa, and @ffy00 and unassigned @morganamilo, @toolybird, and @thurstylark
If it is not an Arch packaging issue, why 4.0.2-9 does not crash using exactly the same configurations (rocm 5.6.1, etc)?
Note that Blender 4.0 requirements for AMD GPU are:
Minimum driver versions:
Windows: Radeon Software 21.12.1 or Radeon PRO Software 21.Q4 Linux: Radeon Software 22.10 or ROCm 5.3https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html#supported-hardware
Edited by cellstijeBelow debug trace:
Memory access fault by GPU node-1 (Agent handle: 0x7fff2c9ce600) on address 0x7ffc057ac000. Reason: Page not present or supervisor privilege. Thread 103 "blender" received signal SIGABRT, Aborted. [Switching to Thread 0x7fff1f800000 (LWP 20840)] __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 Downloading source file /usr/src/debug/glibc/glibc/nptl/pthread_kill.c 44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0; (gdb) bt #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007fffe38ab393 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78 #2 0x00007fffe385a6c8 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x00007fffe38424b8 in __GI_abort () at abort.c:79 #4 0x00007fff1fa2132a in ??? () at /opt/rocm/lib/libhsa-runtime64.so.1 #5 0x00007fff1fa745d2 in ??? () at /opt/rocm/lib/libhsa-runtime64.so.1 #6 0x00007fff1fa257cc in ??? () at /opt/rocm/lib/libhsa-runtime64.so.1 #7 0x00007fffe38a955a in start_thread (arg=<optimized out>) at pthread_create.c:447 #8 0x00007fffe3926a3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 (gdb)Note that installing rocm 6.0 as currently in arch repos is not an option for many AMD GPU users that want to use GPGPU capabilities. It may not be accepted as bug report, but please see what it is reported in the forums:
-
Thanks for the trace. But I think you're going to have to experiment with some environment variables to try and get some better debug output from HIP. See for example here and here. Time to bring in @tpkessler methinks.
-
@bugbuddy assign @tpkessler
assigned to @tpkessler
Blender 4.0.2 downloaded from blender.org works absolutely fine
I tested using the same methodology as in my original bug report, same input file (classroom) and following same steps.
I switched between 'Viewport Shading' 'Rendered' and 'Material Preview' at least 10 times, no crash.
Edit -- again I am using rocm 5.6.1 as was provided by Arch repos few months ago. I had to roll-back from 5.7.1 as that version was not working with blender and pytorch.
In the forum thread I referenced in previous post, many users have flagged that rocm 6.0 is still not working properly (ppl with rx 5700 and rx 6600 from a quick scan).
1 person has reported that switching from ROCM to opencl-rusticl-mesa has solved issues with darktable (but rusticl is not supported by blender or pytorch -- AFAIK)
Edited by cellstijeNo, I use my computer for work, I need something that I know works with pytorch -- I have hold-off the upgrade of a number of packages due to the status of rocm since 5.7.1
In the forum linked there are enough ppl flagging the issue, also with rocm 6.0
Edited by cellstije... and blender 4.0.2-9 from Arch works no problem. I will hold off the upgrade of blender until there is a new version (4.0.2-11?) to test
Edited by cellstije4.0.2-10 was a rebuild against ROCm 6 which appears to be the culprit. I read the entire thread you linked and there doesn't appear to be anyone pointing out what the result of ROCm 6 with Blender from blender.org is. Maybe @tpkessler can do a quick test? I don't have hardware to test this with.
-
No, I use my computer for work
Well, please don't be surprised if the ticket is closed with "@reporter uncooperative". Honestly, this is how Arch works. If you report a bug, you are required to do what is asked of you. How else will it get fixed?
Additionally, I didn't cotton on that you were in a "partial updates" situation which is completely unsupported! Ughh.
blender works fine on my machine. Here's the output of
hipconfig --fullHIP version : 6.0.32830- == hipconfig HIP_PATH : /opt/rocm ROCM_PATH : /opt/rocm HIP_COMPILER : clang HIP_PLATFORM : amd HIP_RUNTIME : rocclr CPP_CONFIG : -D__HIP_PLATFORM_HCC__= -D__HIP_PLATFORM_AMD__= -I/opt/rocm/include -I/opt/rocm/lib/llvm/lib/clang/17.0.0 == hip-clang HIP_CLANG_PATH : /opt/rocm/llvm/bin clang version 17.0.0 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /opt/rocm/llvm/bin AOMP-16.0-45 (http://github.com/ROCm-Developer-Tools/aomp): Source ID:16.0-45-6b875fb548b9ded0f07df02bc2af6e12568504a9 LLVM version 17.0.0git Optimized build with assertions. Default target: x86_64-pc-linux-gnu Host CPU: znver4 Registered Targets: amdgcn - AMD GCN GPUs nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit r600 - AMD GPUs HD2XXX-HD6XXX x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 hip-clang-cxxflags : -isystem "/opt/rocm/include" -O3 hip-clang-ldflags : --driver-mode=g++ -O3 --hip-link --rtlib=compiler-rt -unwindlib=libgcc === Environment Variables PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/rocm/bin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/rustup/bin egrep: warning: egrep is obsolescent; using grep -E CUDA_PATH=/opt/cuda == Linux Kernel Hostname : Can't exec "hostname": No such file or directory at /opt/rocm/bin//hipconfig.pl line 211. Linux ideapad 6.7.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 05 Feb 2024 22:07:49 +0000 x86_64 GNU/Linux LSB Version: n/a Distributor ID: Arch Description: Arch Linux Release: rolling Codename: n/aNote that the users' issues in the linked thread are likely caused by a driver bug that was fixed with a recent kernel update.
then please close the bug report.
On my end I did what I felt was right -- reporting the issue and provide debug info as requested for a software that has no hard dependency on rocm6 and its upstream version works fine -- aka it is an Arch package issue
Edited by cellstijeHere is my output for
hipconfig --fullI note that compared with output from @tpkessler there is no CUDA variables set -- I have no NVIDIA hardware or related packages installed on my machine
HIP version : 5.6.31062- == hipconfig HIP_PATH : /opt/rocm ROCM_PATH : /opt/rocm HIP_COMPILER : clang HIP_PLATFORM : amd HIP_RUNTIME : rocclr CPP_CONFIG : -D__HIP_PLATFORM_HCC__= -D__HIP_PLATFORM_AMD__= -I/opt/rocm/include -I/opt/rocm/llvm/bin/../lib/clang/16.0.0 == hip-clang HIP_CLANG_PATH : /opt/rocm/llvm/bin clang version 16.0.0 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /opt/rocm/llvm/bin LLVM (http://llvm.org/): LLVM version 16.0.0git Optimized build with assertions. Default target: x86_64-pc-linux-gnu Host CPU: znver3 Registered Targets: amdgcn - AMD GCN GPUs nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit r600 - AMD GPUs HD2XXX-HD6XXX x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 hip-clang-cxxflags : -isystem "/opt/rocm/bin/include" -O3 hip-clang-ldflags : -O3 --hip-link --rtlib=compiler-rt -unwindlib=libgcc === Environment Variables PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/apache-spark/bin:/opt/apache-spark/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/rocm/bin egrep: warning: egrep is obsolescent; using grep -E HIP_ROCCLR_HOME=/opt/rocm/bin == Linux Kernel Hostname : XXXXX Linux gemini 6.7.4-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Mon, 05 Feb 2024 22:07:37 +0000 x86_64 GNU/Linux LSB Version: n/a Distributor ID: Arch Description: Arch Linux Release: rolling Codename: n/aBelow the output of rocminfo (it is not clear what AMD GPU @tpkessler is using) -- I am on rx 6800
ROCk module is loaded ===================== HSA System Attributes ===================== Runtime Version: 1.1 System Timestamp Freq.: 1000.000000MHz Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model: LARGE System Endianness: LITTLE ========== HSA Agents ========== ******* Agent 1 ******* Name: AMD Ryzen 9 5900X 12-Core Processor Uuid: CPU-XX Marketing Name: AMD Ryzen 9 5900X 12-Core Processor Vendor Name: CPU Feature: None specified Profile: FULL_PROFILE Float Round Mode: NEAR Max Queue Number: 0(0x0) Queue Min Size: 0(0x0) Queue Max Size: 0(0x0) Queue Type: MULTI Node: 0 Device Type: CPU Cache Info: L1: 32768(0x8000) KB Chip ID: 0(0x0) ASIC Revision: 0(0x0) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 4951 BDFID: 0 Internal Node ID: 0 Compute Unit: 24 SIMDs per CU: 0 Shader Engines: 0 Shader Arrs. per Eng.: 0 WatchPts on Addr. Ranges:1 Features: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: FINE GRAINED Size: 32766556(0x1f3fa5c) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 2 Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 32766556(0x1f3fa5c) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 3 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 32766556(0x1f3fa5c) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: TRUE ISA Info: ******* Agent 2 ******* Name: gfx1030 Uuid: GPU-52f35cc74f3c7772 Marketing Name: AMD Radeon RX 6800 Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 1 Device Type: GPU Cache Info: L1: 16(0x10) KB L2: 4096(0x1000) KB L3: 131072(0x20000) KB Chip ID: 29631(0x73bf) ASIC Revision: 1(0x1) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 2475 BDFID: 3328 Internal Node ID: 1 Compute Unit: 60 SIMDs per CU: 2 Shader Engines: 4 Shader Arrs. per Eng.: 2 WatchPts on Addr. Ranges:4 Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 32(0x20) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 32(0x20) Max Work-item Per CU: 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) Max fbarriers/Workgrp: 32 Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 16760832(0xffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx1030 Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) FBarrier Max Size: 32 *** Done ***All,
taking a third look @tpkessler
hipconfig --full, noting that a bunch of NVIDIA related env variables are set:PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/rocm/bin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/rustup/bin egrep: warning: egrep is obsolescent; using grep -E CUDA_PATH=/opt/cudaand the following string seems to indicate that the hardware is a lenovo 'idepad'
== Linux Kernel Hostname : Can't exec "hostname": No such file or directory at /opt/rocm/bin//hipconfig.pl line 211. Linux ideapad 6.7.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 05 Feb 2024 22:07:49 +0000 x86_64 GNU/Linux LSB Version: n/aI wonder if there is some interaction with the NVIDIA hardware possibly installed in his machine. To the best of my knowledge Lenovo IdeaPads have options with NIVIDIA RTX video cards dGPU -- no AMD dGPU (AFAIK)
https://www.lenovo.com/us/en/c/laptops/ideapad
... maybe a red herring
-
I tested it on my laptop with integrated GPU,
gfx1103(no dGPU). As explained above, your blender crashes as the newest version is compiled with the recent version of ROCm, ROCm 6.0.0. Since this is a major release (from 5.7.1 to 6.0.0) it includes breaking API changes. So it's not surprising that your setup breaks. If you decide to use an older version of ROCm then you also have to roll back all packages that use it. In your case this means: Install an older (working) version of blender and add it toIgnorePkgin/etc/pacman.confor locally recompile blender with your old ROCm version. The support of blender for ROCm 5+ doesn't mean that you can run it with any ROCm version but linked against one particular version of ROCm it will work. As we use ROCm 6 but you have ROCm 5.6.1 installed, blender will break. So, you are saying that Arch is artificially imposing an external library minimum requirement (rocm 6.0) to a software (blender) that has no such minimum requirement (blender states rocm >= 5.3) and its upstream version (blender 4.0.2) works fine (tested) with rocm 5.6.1 as was available in Arch repos.
ok, I take note of this.
However consider that the above imposes essentially an extra dependency (unnecessary) on blender on whatever status of rocm stack is. To date rocm in Arch has been problematic (either with pytorch, blender, darktable etc). And many end users have flagged the issues both on the old Arch bug report portal and in forums.
If the desire of the maintainer is to close this bug report, please go ahead. On my end I will block the upgrade to blender 4.0.2-10
We never supported partial upgrades in Arch because they are unsupportable. As such, we always have the implicit need for users to upgrade to the newest available versions of everything. Consider also that Arch is entirely volunteer run and we all do this in our spare time. If you need commercial level support for deprecated versions of some software, it might be better to look into an enterprise distro.
I understand your point and I appreciate all the effort from all volunteers (including end users providing feedback in their own spare time)
However there is a requirement to provide software packages in Arch official repos that are tested and stable -- otherwise everyone here is wasting his/her own time
Unfortunately rocm is demonstrably not stable as in Arch. Maybe including it in the official repos is not a good idea. And it is not a good idea to impose artificially dependencies on specific versions of external libraries to a software that has no such dependecies
Edited by cellstije
If you are interested: https://wiki.archlinux.org/title/Arch_Testing_Team
-
then please close the bug report
Gladly. Hopefully you can now appreciate why partial upgrades are unsupportable. Another point worth a mention: it ruins backtraces via our
debuginfodserver i.e. the traces become ineffective due to "holes" in the output.PMs - I'm very sorry for allowing this "partial upgrade" report through thus wasting your time!
added resolutioninvalid label
Sorry to necrobump, but I'm getting the same issue on a fully upgraded system with ROCm 6.0 and Linux 6.7.4:gdb.txt
hipconfig:
% LANG=C /opt/rocm/bin/hipconfig --full HIP version : 6.0.32830- == hipconfig HIP_PATH : /opt/rocm ROCM_PATH : /opt/rocm HIP_COMPILER : clang HIP_PLATFORM : amd HIP_RUNTIME : rocclr CPP_CONFIG : -D__HIP_PLATFORM_HCC__= -D__HIP_PLATFORM_AMD__= -I/opt/rocm/include -I/opt/rocm/lib/llvm/lib/clang/17.0.0 == hip-clang HIP_CLANG_PATH : /opt/rocm/llvm/bin clang version 17.0.0 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /opt/rocm/llvm/bin AOMP-16.0-45 (http://github.com/ROCm-Developer-Tools/aomp): Source ID:16.0-45-6b875fb548b9ded0f07df02bc2af6e12568504a9 LLVM version 17.0.0git Optimized build with assertions. Default target: x86_64-pc-linux-gnu Host CPU: znver3 Registered Targets: amdgcn - AMD GCN GPUs nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit r600 - AMD GPUs HD2XXX-HD6XXX x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 hip-clang-cxxflags : -isystem "/opt/rocm/include" -O3 hip-clang-ldflags : --driver-mode=g++ -O3 --hip-link --rtlib=compiler-rt -unwindlib=libgcc === Environment Variables PATH=/home/kouta/.nvm/versions/node/v16.15.0/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/opt/android-sdk/platform-tools:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/rustup/bin:/home/kouta/.local/bin/:/opt/ps2dev/bin:/opt/ps2dev/ee/bin:/opt/ps2dev/iop/bin:/opt/ps2dev/dvp/bin:/opt/ps2dev/ps2sdk/bin egrep: warning: egrep is obsolescent; using grep -E == Linux Kernel Hostname : Can't exec "hostname": No such file or directory at /opt/rocm/bin//hipconfig.pl line 211. Linux arch-kouta 6.7.4-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 05 Feb 2024 22:07:49 +0000 x86_64 GNU/Linux LSB Version: n/a Distributor ID: Arch Description: Arch Linux Release: rolling Codename: n/aUnlike the OG reporter however, I get another (not the same) issue with Blender's official packaging:
kouta@arch-kouta ~/Descargas/blender-4.0.2-linux-x64 % ./blender Read prefs: "/home/kouta/.config/blender/4.0/config/userpref.blend" Read blend: "/home/kouta/riffi/untitled.blend" blender: /usr/src/debug/hip-runtime-amd/clr-rocm-6.0.0/hipamd/src/hip_memory.cpp:2194: hipError_t ihipGetMemcpyParam3DCommand(amd::Command*&, const HIP_MEMCPY3D*, hip::Stream*): Assertion `false && "ShouldNotReachHere()"' failed. [1] 6127 IOT instruction (core dumped) ./blender ./blender 3,39s user 2,13s system 34% cpu 16,143 totalFollow-up: Daily builds of 4.1.0 do work correctly so it's probably not an Arch issue :)
Edited by Salvador Pardiñas
mentioned in issue hip-runtime-amd#2 (closed)
mentioned in issue #12 (moved)
mentioned in issue python-pytorch#9 (moved)
mentioned in issue #13 (closed)
