[6.9.1-arch1-1] [6.9.1-arch1-2] Firmware Bug
Firmware Bug after upgrading to linux 6.9.1 with intel core i7-7700k.
Output from dmesg says:
[Firmware Bug]: CPU0: Topology domain 0 shift 1 != 4
The lm_sensors don't show me any temperatures from the CPU
Activity
added priority3-normal severity4-low statusunconfirmed labels
assigned to @freswa, @anthraxx, @toolybird, and @gromit
The error message is from introduced in
ebdb20361059 x86/cpu: Provide cpu_init/parse_topology(). Is the issue still present in the upstream git tree? Is the message correct that there is a firmware bug? What is the connection with lm_sesnsors?Edit:
https://bugzilla.kernel.org/show_bug.cgi?id=218879 Which does not contain a bisection.
Edited by loqsadded statusconfirmed label and removed statusunconfirmed label
added scoperegression label
Thanks for reporting this issue! This could possibly be a kernel regression, which should be bisected and reported upstream to the kernel developers and the regression mailing list
Are you confident to do the bisection on your own or do you need some help?
If you want we could also provide you with prebuilt kernel images for you to test
Good info to get you started is:
- https://docs.kernel.org/admin-guide/reporting-regressions.html
- https://wiki.archlinux.org/title/Kernel#Debugging_regressions
In order for the issue to be properly bisectable it would need to be reproducible, so do you already know how you can trigger the bug reliably?
Perhaps try to shortcut the bisection to
ebdb20361059 x86/cpu: Provide cpu_init/parse_topology()and its parent?Edit:
- linux-6.8rc4.r2.gebdb20361059-1-x86_64.pkg.tar.zst
- linux-headers-6.8rc4.r2.gebdb20361059-1-x86_64.pkg.tar.zst
If the kernel above does not have at least one of the issues there is no need to test the kernel below:
Edited by loqslinux-6.8rc4.r2 works well without problems.
The problem with sensors is the coretemp-isa-0000
output with linux-6.8.9 and earlier
coretemp-isa-0000 Adapter: ISA adapter Package id 0: +28.0°C (high = +80.0°C, crit = +100.0°C) Core 0: +26.0°C (high = +80.0°C, crit = +100.0°C) Core 1: +28.0°C (high = +80.0°C, crit = +100.0°C) Core 2: +27.0°C (high = +80.0°C, crit = +100.0°C) Core 3: +26.0°C (high = +80.0°C, crit = +100.0°C)output since 6.9.1
coretemp-isa-0000 Adapter: ISA adapter Package id 0: +29.0°C (high = +80.0°C, crit = +100.0°C) Core 0: +29.0°C (high = +80.0°C, crit = +100.0°C)Does
linux-6.8rc4.r2produce the\[Firmware Bug\]: CPU0: Topology domain 0 shift 1 != 4message? If not it would seem both issues are introduced by later commits.Edited by loqs@igorog please consider also testing the two kernels from #57 (comment 188608)
This kernel works well, no spam and all sensors work.
-
@andy_on_archlinux could also please share the full output of dmesg on booting the linux-6.9.2.arch1-1.5-x86_64.pkg.tar.zst kernel?
Edited by Christian Heusel Here is the log dmesg.log
Linux FIRE 6.9.2-arch1-1archlinux kernel: [Firmware Bug]: CPU4: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU4: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU4: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU4: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU4: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU4: Topology domain 6 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU5: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU5: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU5: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU5: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU5: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU5: Topology domain 6 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU6: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU6: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU6: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU6: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU6: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU6: Topology domain 6 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU7: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU7: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU7: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU7: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU7: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU7: Topology domain 6 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU8: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU8: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU8: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU8: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU8: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU8: Topology domain 6 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU9: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU9: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU9: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU9: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU9: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU9: Topology domain 6 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU10: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU10: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU10: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU10: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU10: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU10: Topology domain 6 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU11: Topology domain 1 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU11: Topology domain 2 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU11: Topology domain 3 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU11: Topology domain 4 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU11: Topology domain 5 shift 7 != 6 archlinux kernel: [Firmware Bug]: CPU11: Topology domain 6 shift 7 != 6@gromit to me that confirms the error message has been triggering ever since it was introduced in
ebdb20361059 x86/cpu: Provide cpu_init/parse_topology(). If it is considered a legitimate firmware error that the kernel should be warning about then perhaps ask the kernel developers to rate cap the message? Outstanding is if there is any connection with sensors.@igorog does the patch below which is intended to only print the error once and the kernel with it applied below that help?
diff --git a/arch/x86/kernel/cpu/topology_common.c b/arch/x86/kernel/cpu/topology_common.c index 9a6069e7133c..492a105e943d 100644 --- a/arch/x86/kernel/cpu/topology_common.c +++ b/arch/x86/kernel/cpu/topology_common.c @@ -172,12 +172,12 @@ void cpu_parse_topology(struct cpuinfo_x86 *c) if (IS_ENABLED(CONFIG_X86_LOCAL_APIC)) { if (c->topo.initial_apicid != c->topo.apicid) { - pr_err(FW_BUG "CPU%4u: APIC ID mismatch. CPUID: 0x%04x APIC: 0x%04x\n", + pr_err_once(FW_BUG "CPU%4u: APIC ID mismatch. CPUID: 0x%04x APIC: 0x%04x\n", cpu, c->topo.initial_apicid, c->topo.apicid); } if (c->topo.apicid != cpuid_to_apicid[cpu]) { - pr_err(FW_BUG "CPU%4u: APIC ID mismatch. Firmware: 0x%04x APIC: 0x%04x\n", + pr_err_once(FW_BUG "CPU%4u: APIC ID mismatch. Firmware: 0x%04x APIC: 0x%04x\n", cpu, cpuid_to_apicid[cpu], c->topo.apicid); } } @@ -185,7 +185,7 @@ void cpu_parse_topology(struct cpuinfo_x86 *c) for (dom = TOPO_SMT_DOMAIN; dom < TOPO_MAX_DOMAIN; dom++) { if (tscan.dom_shifts[dom] == x86_topo_system.dom_shifts[dom]) continue; - pr_err(FW_BUG "CPU%d: Topology domain %u shift %u != %u\n", cpu, dom, + pr_err_once(FW_BUG "CPU%d: Topology domain %u shift %u != %u\n", cpu, dom, tscan.dom_shifts[dom], x86_topo_system.dom_shifts[dom]); }With this kernel, from all the previous spam, only one line remained in the log:
archlinux kernel: [Firmware Bug]: CPU4: Topology domain 1 shift 7 != 6For some reason, it was CPU4, the rest of the processor cores disappeared from the log
Edited by igor
-
This is currently being investigated: https://lore.kernel.org/all/87zfs78zxq.ffs@tglx/
6.9.2.arch1 with the second debug patch applied:
Please post the dmesg generated with the second debug patch applied if no one has already done so in the linked mailing list thread.
Edit:
No need the cause has been determined https://lore.kernel.org/all/87o78n8fe2.ffs@tglx/
Edited by loqs-
There just was a fix proposed: https://lore.kernel.org/all/87le3r8dyw.ffs@tglx
Amended patch:
--- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1589,6 +1589,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c) if (have_cpuid_p()) { cpu_detect(c); get_cpu_vendor(c); + intel_unlock_cpuid_leafs(c); get_cpu_cap(c); setup_force_cpu_cap(X86_FEATURE_CPUID); get_cpu_address_sizes(c); @@ -1748,7 +1749,7 @@ static void generic_identify(struct cpuinfo_x86 *c) cpu_detect(c); get_cpu_vendor(c); - + intel_unlock_cpuid_leafs(c); get_cpu_cap(c); get_cpu_address_sizes(c); diff --git a/arch/x86/kernel/cpu/cpu.h b/arch/x86/kernel/cpu/cpu.h index ea9e07d57..1beccefba 100644 --- a/arch/x86/kernel/cpu/cpu.h +++ b/arch/x86/kernel/cpu/cpu.h @@ -61,9 +61,11 @@ extern __ro_after_init enum tsx_ctrl_states tsx_ctrl_state; extern void __init tsx_init(void); void tsx_ap_init(void); +void intel_unlock_cpuid_leafs(struct cpuinfo_x86 *c); #else static inline void tsx_init(void) { } static inline void tsx_ap_init(void) { } +static inline void intel_unlock_cpuid_leafs(struct cpuinfo_x86 *c) { } #endif /* CONFIG_CPU_SUP_INTEL */ extern void init_spectral_chicken(struct cpuinfo_x86 *c); diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c index be30d7fa2..93efd9edc 100644 --- a/arch/x86/kernel/cpu/intel.c +++ b/arch/x86/kernel/cpu/intel.c @@ -268,19 +268,26 @@ static void detect_tme_early(struct cpuinfo_x86 *c) c->x86_phys_bits -= keyid_bits; } +void intel_unlock_cpuid_leafs(struct cpuinfo_x86 *c) +{ + if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL) + return; + + if (c->x86 < 6 || (c->x86 == 6 && c->x86_model < 0xd)) + return; + + /* + * The BIOS can have limited CPUID to leaf 2, which breaks feature + * enumeration. Unlock it and update the maximum leaf info. + */ + if (msr_clear_bit(MSR_IA32_MISC_ENABLE, MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) + c->cpuid_level = cpuid_eax(0); +} + static void early_init_intel(struct cpuinfo_x86 *c) { u64 misc_enable; - /* Unmask CPUID levels if masked: */ - if (c->x86 > 6 || (c->x86 == 6 && c->x86_model >= 0xd)) { - if (msr_clear_bit(MSR_IA32_MISC_ENABLE, - MSR_IA32_MISC_ENABLE_LIMIT_CPUID_BIT) > 0) { - c->cpuid_level = cpuid_eax(0); - get_cpu_cap(c); - } - } - if ((c->x86 == 0xf && c->x86_model >= 0x03) || (c->x86 == 0x6 && c->x86_model >= 0x0e)) set_cpu_cap(c, X86_FEATURE_CONSTANT_TSC);-
I'll wait for the dmesg from @igorog and then send out my mail
Sorry for not answering yesterday. It was already late at night. I post the logs: dmesg6.9.2-1.5.log dmesg6.9.2-1.log
Edited by igor-
Thanks for providing these outputs, the upstream developers are asking about more information about your system:
Can I get: - dmesg from 6.8.y kernel - output of cpuid -r - content of /sys/kernel/debug/x86/topo/cpus/* (on 6.9.y) please?You can get these i.e. by installing the following kernel from the archive
sudo pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.8.9.arch1-2-x86_64.pkg.tar.zstRunning the following pre-compiled program: kcpuid
$ ./kcpuid -rI'll get
cpuidpackaged some time soon
Edit: now available as
extra/kcpuid
Does somebody know how we get
/sys/kernel/debug/x86/topo/cpus/*to show up? Its not present on my systemEdited by Christian Heusel
- Edited by igor
-
I seems like upstream thinks something might not be right with the output from
kcpuid
Could you install it from the repos and re-send its output @igorog? I have packaged it in the meantime$ sudo pacman -Syu kcpuid # make sure that its the right kcpuid and i.e. not something in /usr/local/ $ type -a kcpuid kcpuid is /sbin/kcpuid kcpuid is /bin/kcpuid kcpuid is /usr/bin/kcpuid kcpuid is /usr/sbin/kcpuid $ kcpuid -rEdited by Christian Heusel https://www.etallen.com/cpuid.html is the cpuid used by Fedora and Debian it does produce per cpu output for
cpuid -rso I suspect this is the variant upstream wants.Edited by loqskcpuid /usr/
kcpuid /usr/bin/
kcpuid /usr/bin/kcpuid
kcpuid /usr/share/
kcpuid /usr/share/misc/
kcpuid /usr/share/misc/cpuid.csv
Edited by igorigor can you build https://aur.archlinux.org/packages/cpuid and post the output from
cpuid -r.cpuid /usr/
cpuid /usr/bin/
cpuid /usr/bin/cpuid
cpuid /usr/bin/cpuinfo2cpuid
cpuid /usr/share/
cpuid /usr/share/man/
cpuid /usr/share/man/man1/
cpuid /usr/share/man/man1/cpuid.1.gz
cpuid /usr/share/man/man1/cpuinfo2cpuid.1.gz
https://lore.kernel.org/regressions/87wmna6y0l.ffs@tglx/T/#md23c8f658e3c72bbf5461c2ec140f2ad5e599fea is there a firmware update for your system's mainboard?
I don't quite understand, if this is a firmware error, why errors in the boot log appeared only on the new 6.9 kernels. After all, on previous kernels such dmesg-spam was not observed in the logs.
PS: unfortunately, I can’t update the UEFI-BIOS firmware, since they simply aren’t available for my china laptop. ))
Edited by igorAnd even on this kernel (linux-6.9.2.arch1-1.1-x86_64.pkg.tar.zst) the situation is much better, because there is one line about a firmware error. ))
Edited by igor-
I don't quite understand, if this is a firmware error, why errors in the boot log appeared only on the new 6.9 kernels. After all, on previous kernels such dmesg-spam was not observed in the logs.
The commit
ebdb20361059 x86/cpu: Provide cpu_init/parse_topology()mentioned by loqs in #57 (comment 188608) was first introduced in v6.9on this kernel (linux-6.9.2.arch1-1.1-x86_64.pkg.tar.zst) the situation is much better, because there is one line about a firmware error. ))
Yes this is because the applied patch makes it print the firmware only once, the behaviour itself is unchanged
"Yes this is because the applied patch makes it print the firmware only once, the behaviour itself is unchanged" )))
Do you think it is possible to correct the spam situation on my configuration? Or will you have to live with this forever? Okay, this can still be overcome, but I’m afraid that the situation will get worse in further implementations of the Linux kernel. What if the system suddenly stops booting completely? ))
PS: and one more thing: it seems to me that (purely subjective feelings) that on 6.9.. the system has become a little slower.
Should be fixed in
6.9.4with commit f7c6d909a2d4.Edited by Jan Alexander Steffens (heftig)
added resolutioncompleted label
Yes, spam still remains when the system boots, as in this [message] (#57 (comment 188783)) of mine. I'm sorry.
