Boot hangs early unless using "iommu=off"
Description:
My PC hangs when booting from linux-6.9.7.arch1-1. The console displays:
Booting `Arch Linux' Loading Linux linux ... Loading initial ramdisk ...
Then goes no further.
Booting with "iommu=off" or "amd_iommu=off" works as expected.
Additional info:
This affects the following kernels:
linux-6.9.7.arch1-1 linux-6.9.6.arch1-1 linux-6.9.5.arch1-1 linux-6.9.4.arch1-1 linux-6.9.3.arch1-1 linux-6.9.2.arch1-1 linux-6.9.1.arch1-2 linux-6.9.1.arch1-1
linux-6.8.9.arch1-2 and earlier are unaffected.
Also see https://bbs.archlinux.org/viewtopic.php?id=297469
Steps to reproduce:
- Boot PC using one of the affected kernels above
- Watch in dismay
Activity
added priority3-normal severity4-low statusunconfirmed labels
assigned to @freswa, @anthraxx, @toolybird, and @gromit
Did you already have a chance to test 6.9.8 from
[core]?
Also it would be good to check if the bug is present in the latest mainline kernel release candidate:
sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.10rc6-1-x86_64.pkg.tar.zsEdited by Christian Heuseladded scoperegression label
-
This very much looks like kernel regression, which should be bisected and reported upstream to the kernel developers and the regression mailing list
Are you confident to do the bisection on your own or do you need some help?
If you want we could also provide you with prebuilt kernel images for you to test
Good info to get you started is:
-
To start off it would be good if you could verify that 6.8 works:
sudo pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.8.arch1-1-x86_64.pkg.tar.zstAnd that the 6.9 kernel does not:
sudo pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.9.arch1-1-x86_64.pkg.tar.zst
The next package is the first one from the bisection itself:
sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r8073.g480e035-1-x86_64.pkg.tar.zstNote that this installs as pkgbase
linux-mainline(/boot/vmlinuz-linux-mainline) so you might need to teach your bootloader how to boot that (i.e. by runninggrub-mkconfigor creating the needed loader entry forsystemd-boot)After each test it would be good to get a quick message with the feedback on what worked or what was broken
Edited by Christian Heusel >Note that this installs as pkgbase linux-mainline (/boot/vmlinuz-linux-mainline) so you might need to teach your bootloader how to boot that (i.e. by running grub-mkconfig or creating the needed loader entry for systemd-boot)
I didn't realise this so I re-tested mainline-6.10rc6-1 - still hangs
linux-6.8.arch1-1 - boots
linux-6.9.arch1-1 - hangs
I'll test the bisection next
-
Please test
sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.8.r1336.g691632f-1-x86_64.pkg.tar.zstAh and because I didn't mention it earlier, we got around 10 more images to test (according to
git bisect's estimate)Edited by Christian Heusel -
We're being pointed to the following commit:
f0551af021308a2a1163dc63d1f1bba3594208bd is the first bad commit commit f0551af021308a2a1163dc63d1f1bba3594208bd Author: Thomas Gleixner <tglx@linutronix.de> Date: Wed Mar 6 12:17:02 2024 +0100 x86/topology: Ignore non-present APIC IDs in a present package Borislav reported that one of his systems has a broken MADT table which advertises eight present APICs and 24 non-present APICs in the same package. The non-present ones are considered hot-pluggable by the topology evaluation code, which is obviously bogus as there is no way to hot-plug within the same package. As the topology evaluation code accounts for hot-pluggable CPUs in a package, the maximum number of cores per package is computed wrong, which in turn causes the uncore performance counter driver to access non-existing MSRs. It will probably confuse other entities which rely on the maximum number of cores and threads per package too. Cure this by ignoring hot-pluggable APIC IDs within a present package. In theory it would be reasonable to just do this unconditionally, but then there is this thing called reality^Wvirtualization which ruins everything. Virtualization is the only existing user of "physical" hotplug and the virtualization tools allow the above scenario. Whether that is actually in use or not is unknown. As it can be argued that the virtualization case is not affected by the issues which exposed the reported problem, allow the bogosity if the kernel determined that it is running in a VM for now. Fixes: 89b0f15f408f ("x86/cpu/topology: Get rid of cpuinfo::x86_max_cores") Reported-by: Borislav Petkov (AMD) <bp@alien8.de> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Tested-by: Borislav Petkov (AMD) <bp@alien8.de> Link: https://lore.kernel.org/r/87a5nbvccx.ffs@tglx arch/x86/kernel/cpu/topology.c | 39 ++++++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 9 deletions(-)Could you please check if a kernel built with the above commit reverted works:
sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.10rc7-1.1-x86_64.pkg.tar.zstIf not I fear that we may have marked one of the revisions wrong, or what do you think @loqs?
-
We will now start to collect all the necessary information for the bug report
- Could you install
aur/cpuidand provide its output? - Could you provide the dmesg of a working boot?
- Could you provide the output of
cat /sys/kernel/debug/x86/topo/cpus/*from a working boot?
Also do you want to write the bug report to the upstream kernel devs yourself or should I do it?
Reported-by: Rob Newcater <rob@example.org>.Edited by Christian Heusel - Could you install
cpuid.txt attached
dmesg.txt attached
$ ls /sys/kernel/debug/x86/topo/ ls: cannot access '/sys/kernel/debug/x86/topo/': No such file or directory
Re: the bug report I think it be better coming from you
Just wondering how far back I'll have to go as LTS doesn't seem to have this 'todo' entry:
# uname -a Linux aquilonia 6.6.39-1-lts #1 SMP PREEMPT_DYNAMIC Thu, 11 Jul 2024 15:06:47 +0000 x86_64 GNU/Linux # ls -la /sys/kernel/debug/x86/ total 0 drwxr-xr-x 3 root root 0 Jul 14 16:22 . drwx------ 48 root root 0 Jul 14 16:22 .. drwxr-xr-x 3 root root 0 Jul 14 16:22 boot_params -r-------- 1 root root 0 Jul 14 16:22 cpa_stats -rw-r--r-- 1 root root 0 Jul 14 16:22 nmi_longest_ns -r-------- 1 root root 0 Jul 14 16:22 pat_memtype_list -rw------- 1 root root 0 Jul 14 16:22 tlb_single_page_flush_ceiling48525fd x86/cpu: Provide debug interface added the interface in 6.7 and the interface is still present. Perhaps the bug is causing the interface to not be created or some hardening is hiding it?
\> ... added the interface in 6.7 ...
Apologies, I went too far back. Just booted off 6.9.9 (requires "iommu=off") and was able to get the required data. I have also attached an updated dmesg from 6.9.9 in case it is required as previous dmesg was from LTS.
-
Report sent out https://lore.kernel.org/all/361b2ba5-7fc3-4e9a-af63-adfd4bc20304@heusel.eu/, now we wait
-
Hey @painless, could you please provide the output of
dmesgwith the mainline kernel that has the problematic commit reverted?sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.10rc7-1.1-x86_64.pkg.tar.zstEdited by Christian Heusel
added statuswaiting-upstream label and removed statusunconfirmed label
