Popping in USB headphones connected through a UGREEN USB-C hub since kernel 6.10
Description:
On Dell XPS 15 9520, the Corsair HS80 USB headphones make an annoying "pop" every ~10 minutes when audio is playing. The problem occurs only when the headphones are connected through UGREEN's 5-in-1 USB-C Hub, or through another hub connected to that hub transitively - this would generally indicate to me that the issue is with the hub, not anything in the software. However, I am able to point out the time the issue started occurring, down to the specific kernel (sub)version, meaning a change in the kernel somehow triggered this behavior.
I would be happy to test specific commits from the kernel to pinpoint the source of the issue, if a precompiled binaries and/or a PKGBUILDs are available, as I understand the issue is somehow specific. Even though I have not found any similar reports online, I think it might potentially cause undesired behavior on other devices as well at some point.
Additional info:
The issue occurs from the kernel 6.10 on (inclusive). The zen kernel is also affected. Since the lts kernel is (currently) on a "lower" version, the issue does not occur there, so using that is my current "workaround" at least till the next large lts version comes out.
- tested affected versions:
6.10.arch1-2,zen 6.10.10.zen1-1,6.11rc7-1 - tested unaffected versions:
6.9.10.arch1-1,lts 6.6.51-1
Activity
added priority3-normal severity4-low statusunconfirmed labels
assigned to @freswa, @anthraxx, @toolybird, and @gromit
Thanks for reporting this issue! This looks like kernel regression, which should be bisected and reported upstream to the kernel developers and the regression mailing list
Are you confident to do the bisection on your own or do you need some help?
If you want we could also provide you with prebuilt kernel images for you to test
Good info to get you started is:
- https://docs.kernel.org/admin-guide/reporting-regressions.html
- https://wiki.archlinux.org/title/Kernel#Debugging_regressions
In order for the issue to be properly bisectable it would need to be reproducible, so do you already know how you can trigger the bug reliably?
I can do the bisection as long as there is a PKGBUILD (but preferably prebuilt binary) of the specific kernel commit I should test.
As said, the issue is narrowed down to the switch from
6.9.10to6.10, but I am not sufficiently familiar with the kernel development to be able to point to the specific commit that would be useful to test.
added scoperegression label
-
Since you already have tested the good/bad commits we can directly start with the bisection:
sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-v6.9.r7370.g33e02dc-1-x86_64.pkg.tar.zstPlease give a quick feedback on whether a kernel is broken or not, compare Issue #78 (closed) for the process.
Note that this installs a new kernel as "linux-mainline" so you'll have to teach your bootloader how to start that (i.e. "grub-mkconfig .." or adding a new loader file for systemd-boot)
This is the first of approximately 13 steps to test
Edited by Christian Heusel
No issue
Here I took a bit more time to double check that I did not mishear anything and the sound did actually not occur.
Edited by Adam Posker
-
I marked the previous commit as good:
git bisect good cd97950cbcabe662cd8a9fd0a08a247c1ea1fb28This gives us commit
033ee84e5f01to test which translates to the above version string:$ git describe --long --abbrev=7 | sed 's/\([^-]*-g\)/r\1/;s/-/./g' v6.9.rc7.r74.g033ee84So I guess everything is alright
-
We have landed on the following commit:
$ git bisect bad 1b6d7f9eb150305dcb0da4f7101a8d30dcdf0497 is the first bad commit commit 1b6d7f9eb150305dcb0da4f7101a8d30dcdf0497 (HEAD) Author: James Bottomley <James.Bottomley@HansenPartnership.com> Date: Mon Apr 29 16:28:07 2024 -0400 tpm: add session encryption protection to tpm2_get_random() If some entity is snooping the TPM bus, they can see the random numbers we're extracting from the TPM and do prediction attacks against their consumers. Foil this attack by using response encryption to prevent the attacker from seeing the random sequence. Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com> Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org> Tested-by: Jarkko Sakkinen <jarkko@kernel.org> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org> drivers/char/tpm/tpm2-cmd.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)Click to see bisect log
git bisect start # status: waiting for both good and bad commits # bad: [0c3836482481200ead7b416ca80c68a29cfdaabd] Linux 6.10 git bisect bad 0c3836482481200ead7b416ca80c68a29cfdaabd # good: [a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6] Linux 6.9 git bisect good a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6 # bad: [33e02dc69afbd8f1b85a51d74d72f139ba4ca623] Merge tag 'sound-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound git bisect bad 33e02dc69afbd8f1b85a51d74d72f139ba4ca623 # bad: [b850dc206a57ae272c639e31ac202ec0c2f46960] Merge tag 'firewire-updates-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/ieee1394/linux1394 git bisect bad b850dc206a57ae272c639e31ac202ec0c2f46960 # bad: [59729c8a76544d9d7651287a5d28c5bf7fc9fccc] Merge tag 'tag-chrome-platform-for-v6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/chrome-platform/linux git bisect bad 59729c8a76544d9d7651287a5d28c5bf7fc9fccc # good: [14a60290edf6d947b9e2210f7a223bcc6af1716a] Merge tag 'soc-drivers-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc git bisect good 14a60290edf6d947b9e2210f7a223bcc6af1716a # bad: [f4e8d80292859809ea135e9f4c43bae47e4f58bc] Merge tag 'vfs-6.10.rw' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs git bisect bad f4e8d80292859809ea135e9f4c43bae47e4f58bc # bad: [b19239143e393d4b52b3b9a17c7ac07138f2cfd4] Merge tag 'tpmdd-next-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd git bisect bad b19239143e393d4b52b3b9a17c7ac07138f2cfd4 # good: [c0b9620bc3f0a0f914996cc6631522d41870a9e0] Merge tag 'rcu.next.v6.10' of https://github.com/urezki/linux git bisect good c0b9620bc3f0a0f914996cc6631522d41870a9e0 # good: [cd97950cbcabe662cd8a9fd0a08a247c1ea1fb28] Merge tag 'slab-for-6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab git bisect good cd97950cbcabe662cd8a9fd0a08a247c1ea1fb28 # good: [033ee84e5f01c86997cde29947805e9781ddf233] tpm: Add TCG mandated Key Derivation Functions (KDFs) git bisect good 033ee84e5f01c86997cde29947805e9781ddf233 # bad: [eb24c9788cd90db397b3e41322aff4a5557623b4] tpm: disable the TPM if NULL name changes git bisect bad eb24c9788cd90db397b3e41322aff4a5557623b4 # good: [6519fea6fd372b2247a48d72dcb23e14de70b4ea] tpm: add hmac checks to tpm2_pcr_extend() git bisect good 6519fea6fd372b2247a48d72dcb23e14de70b4ea # bad: [52ce7d9731ed8fada505b5ac33fb1df0190fb8c3] KEYS: trusted: Add session encryption protection to the seal/unseal path git bisect bad 52ce7d9731ed8fada505b5ac33fb1df0190fb8c3 # bad: [1b6d7f9eb150305dcb0da4f7101a8d30dcdf0497] tpm: add session encryption protection to tpm2_get_random() git bisect bad 1b6d7f9eb150305dcb0da4f7101a8d30dcdf0497 # first bad commit: [1b6d7f9eb150305dcb0da4f7101a8d30dcdf0497] tpm: add session encryption protection to tpm2_get_random()Since it seems quite unlikely for me that the commit in question causes this type of issue we should maybe verify/double-check our results
I would like to test that reverting the patch on top of the latest mainline kernel actually solves the issue, but it seems like I get conflicts on both 6.10 and 6.11
@loqs could you also have a look for good measure?
Okay to get back to this:
- I have been on the
v6.9.rc7.r78.g6519feaversion since my last comment, and the issue has not appeared. With this, I would be very confident that this commit is not problematic, as I have never been on a kernel that had the problem, without it showing up for so long. - I have briefly re-tested
v6.9.rc7.r79.g1b6d7f9, and the problem occurred again within half an hour, which is about the usual time for it to appear. - For illustration, in case it helped anything, I have noticed that the popping in the headphones is very similar to a sound that can be heard in the latest Jeff Geerling video, around the 5:22-5:23 timestamp. It's just about twice as loud compared to the sound heard in the video.
- I can confirm and emphasize that the problematic popping we are dealing with is not part of any media I am consuming.
- I have been on the
added statuswaiting-input label and removed statusunconfirmed label
-
@posker hm, this sounds somewhat similar: https://lore.kernel.org/all/ZvgCdYfKgwHpJXGE@eldamar.lan
Definitely very similar in terms of the manifestation yes, though for me the crack occurs once every 15-30 minutes, not "every few seconds" like in that issue.
I am not sure whether it is also similar in terms of the underlying issue however - the XPS does not have AMD soundcard, and the sound is not going through HDMI.
The recent regression report revealed that the use of WC pages for AMD HDMI device together with AMD IOMMU leads to unexpected truncation or noises. The issue seems triggered by the change in the kernel core memory allocation that enables IOMMU driver to use always S/G buffers.
Is it possible that the tweaks with the TPM in the bisected commit could be causing something similar in my case?
-
Could you try if there is any difference between the latest mainline release candidate:
sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.12rc1-1-x86_64.pkg.tar.zstAnd a build of it that has the above commit (
478689b5990d ("ALSA: hda: Conditionally use snooping for AMD HDMI")) reverted?sudo pacman -U https://pkgbuild.com/\~gromit/linux-bisection-kernels/linux-mainline-6.12rc1-1.1-x86_64.pkg.tar.zst No worries! The
6.12rc1-1.1with the commit reverted has the issue present, unfortunately - I will test the stock6.12now.Edited by Adam Posker
Would it still be possible to report the issue upstream bisected like this? Or is it generally also necessary to be able to revert the bisected commit on a newer version (say
6.10,6.11or6.12) and confirm the problem is not there (which I assume we just cannot do because of the conflicts)? Is there any other viable way this could be resolved?-
No this is not generally a requirement, but in this case as I already wrote in one of the comments above I find it quite unlikely that the commit in question causes the reported behaviour
Leaving just at where we are right now also sounds like the wrong move though, so we indeed should decide on how we go from here
I have chatted a bit with other people that frequently deal with regressions and they said that we should report the issue but also express our concerns that we're not sure if the commit is actually what's causing the issue. Do you want to do that or should I write the report and CC you?
It is indeed strange, although I am quite confident in terms of the bisecting and landing on this commit, since we even re-tested the adjacent commits with the same result. There might just be something to it, I have heard of the weirdest behaviors coming from the TPM, but I definitely do not have sufficient knowledge to look into the inner workings of it.
If I could please ask you to write the report, that would be great, as I have not submitted similar report before so I am not familiar with the process - is it possible they would at some point reach out for further questions/testing or something like that?
added statuswaiting-upstream label and removed statuswaiting-input label
My understanding from the discussion linked above was that the fix has been merged in time for the 6.12 release. So now that it has been released as stable in the repos, I installed
6.12.1-arch1-1to test it, but unfortunately it seems like the popping is still there. Is the fix indeed supposed to exist in this version? Otherwise I assume this would mean that the fix has somehow not resolved my particular problem.-
Yeah maybe this is a feedback that we should give to upstream
Edit: https://lore.kernel.org/all/b3a01060-f59b-430d-afcc-48c5ec628bcb@heusel.eu
Edited by Christian Heusel Sorry for the delay - I have only now gotten to testing the workaround mentioned in the linked discussion.
My understanding is that I need to add
tpm.disable_pcr_integrity_protection=1andtpm.disable_random_integrity=1toGRUB_CMDLINE_LINUX_DEFAULTin/etc/default/gruband then rerungrub-mkconfig.Having done that, I unfortunately have to say that the problem persists, both with either option enabled individually, and both enabled at once. This is on
6.12.8-arch1-1.Please let me know if I'm doing something wrong or if I should test something else.
added resolutionupstream label
