Long timeout (more than a minute) on the connected USB mass storage device (Garmin)

added priority3-normal severity4-low statusunconfirmed labels

assigned to @freswa, @anthraxx, @toolybird, and @gromit

changed the description

Thanks for reporting this issue! This very much looks like kernel a regression, which should be bisected and reported upstream to the kernel developers and the regression mailing list

Do you want to do the bisection on your own or do should I provide you with prebuilt kernel images for you to test?

Did you already get the chance to test this with the latest mainline kernel?

$ sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-6.10rc2-1-x86_64.pkg.tar.zst

added scoperegression label

Much, much worse. It doesn't progress at all. It fell into RCU stalls after these errors (amount of which is doubled or so) and I have got no shell, no GUI, no nothing exept repeating stall messages from the kernel. TL;DR: v6.10-rc2 is an unusable disaster in this sense.

Are these extra errors also related to the handling of the Garmin device? Or does it boot properly if you remove the watch from the usb connector?

I believe so: the tasks stalled are kworker, blkid, scsi_eh.
Yes, it boots if I disconnect the device

I'm writing now from v6.10-rc2 kernel, but it required to disconnect device for a couple of seconds or so.

FWIW, I can bisect, but I may not as it's my home laptom, the only one, and I prefer to spend time more effectively. If you have a wild guesses and some kernels to test, I might do that, but I will be for a long vacation soon, so I would be able to do something real somewhere in August.

To start off it would be good if you could verify that 6.8 works:

sudo pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.8.arch1-1-x86_64.pkg.tar.zst

And that the 6.9 kernel does not:

sudo pacman -U https://archive.archlinux.org/packages/l/linux/linux-6.9.arch1-1-x86_64.pkg.tar.zst

The next package is the first one from the bisection itself:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r8073.g480e035-1-x86_64.pkg.tar.zst

Yes, v6.8 works like a charm (also display suspend works, in v6.9 the xscreensaver never able to actually suspend the graphics).

I'll test the other two later on. I need to do something useful on my laptop.

v6.9 is bad, v6.8.r8073.g480e035 is good. Can you generata next one and then for bad and good (so 3 images altogether)? It will speed up a bit this process.

The next step is

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r11837.g2ac2b16-1-x86_64.pkg.tar.zst

If thats good:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.9.rc2.r115.ge858bee-1-x86_64.pkg.tar.zst

If thats bad:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r9944.g68bf6bf-1-x86_64.pkg.tar.zst

v6.8.r11837.g2ac2b16 is good, v6.9.rc2.r115.ge858bee is bad.

The next step is

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r12818.gcba9ffd-1-x86_64.pkg.tar.zst

If thats good:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.9.rc1.r40.g27f58c0-1-x86_64.pkg.tar.zst

If thats bad:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r12355.g0a59b3f-1-x86_64.pkg.tar.zst

v6.8.r12818.gcba9ffd is bad, v6.8.r12355.g0a59b3f is good.

Note, starting from tomorrow I will lose access to the HW in question for 2+ months. Hence, if it's possible I would really like to speed up tonight the bisection. I'm fully allocated for that.

It looks like 10 steps left. Should be five messages from you with up to 3 kernels to test :-)

Alrighty, no problem from my side! I'll try to be responsive about this issue

The next step is

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.rc5.r226.g37efe11-1-x86_64.pkg.tar.zst

v6.8.rc5.r226.g37efe11 is good. So, let's start next with good/bad ones.

(just needed a moment to finish the builds)

If thats good:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r12680.g88d92fb-1-x86_64.pkg.tar.zst

If thats bad:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.rc5.r118.gd4551c1-1-x86_64.pkg.tar.zst

v6.8.r12680.g88d92fb is good.

Nice, the good build is already finished:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r5271.gf99c5f5-1-x86_64.pkg.tar.zst

I'm wondering if that rXXXX are continuous.

What do you mean? Its just the output of git describe --long --abbrev=7 | sed 's/\([^-]*-g\)/r\1/;s/-/./g'

v6.8.r5271.gf99c5f5 is good. Hmm... Probably I never tried --long.

How long does it take to get a build done on your side?

Roughly 10 minutes I'm just shooting it off to our build box and did not setup caching, so its taking longer than it should I'll tweak my setup for that in the coming days

For the last iteration I started out with the bad build, so the good build is almost done

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.rc5.r34.g5e3cf30-1-x86_64.pkg.tar.zst

I can not see anything USB/block/FS related left in the bisection range. If the builds keep testing good I suspect a previous step is wrong.

v6.8.rc5.r34.g5e3cf30 is good. @loqs which one do you want me retest? Or is it a mistake on your side?

@gromit It seems there is no need to spin two builds as my testing cycle is shorter than your build cycle.

My current bisect log is the following:

$ git bisect log | grep -v "#" 
git bisect start
git bisect good e8f897f4afef0031fe618a8e94127a0934896aba
git bisect bad a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
git bisect good 480e035fc4c714fb5536e64ab9db04fedc89e910
git bisect good 2ac2b1665d3fbec6ca709dd6ef3ea05f4a51ee4c
git bisect bad e858beeddfa3a400844c0e22d2118b3b52f1ea5e
git bisect bad cba9ffdb9913dfe6be29f049ce920ce451ce7cc4
git bisect good 0a59b3f42e5703a89dd6ddf5bc818a4cff975302
git bisect good 37efe116bea197c17cdae0861e0004ca70f10d5c
git bisect good 88d92fb1c034922572bab93482ac9cc61d4ba43c
git bisect good f99c5f563c174a49ea1cbf4754539b05cfde40c4
git bisect good 5e3cf304a0bdc7a3d7ba4805bb2bb05c5e6916ff

I think this matches what you reported here Maybe we should re-test 0a59b3f42e5703a89dd6ddf5bc818a4cff975302 or cba9ffdb9913dfe6be29f049ce920ce451ce7cc4..? What do you think @loqs

Next step would be the following, but I suspect #60 (comment 190807) is correct

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.rc5.r51.g5270316-1-x86_64.pkg.tar.zst

Okay, let me test in this order: the latest from the comment 190810, followed by 0a... followed by cba... (in case it follows your log).

The last block and USB commits were removed from the bisect when cba9ffdb9913 was marked bad so that we be my candidate to retest as well.

Yeah, sorry guys, my mistake. I checked even more now:

v6.8.r12818.gcba9ffd is actually good
v6.9.rc2.r115.ge858bee is bad (as it was)
v6.9.rc1.r40.g27f58c0 is bad (this is new test in accordance with the above correction)

I believe I chose wrong entry in a bootloader since I have official kernel installed which is bad.

Please test the following:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-headers-v6.8.r13022.g1e3cd03-1-x86_64.pkg.tar.zst

Goddamnit, I had added the wrong link in #60 (comment 190819)

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r13022.g1e3cd03-1-x86_64.pkg.tar.zst

No problem. Now I have tested it and v6.8.r13022.g1e3cd03 is good.

Please test the following:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r13149.ge3111d9-1-x86_64.pkg.tar.zst

v6.8.r13149.ge3111d9 is good.

Please test the following:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r13213.g7029324-1-x86_64.pkg.tar.zst

v6.8.r13213.g7029324 is bad. Does it correlate with the range contents? I suppose we are somewhere in USB merge.

While at it, can you tell what the vanilla SHAs are for these r13149 and r13213?

The last part of the version is the short sum (minus the g prefix):

$ git show --pretty=oneline 7029324 e3111d9

70293240c5ce675a67bfc48f419b093023b862b3 Merge tag 'timers-urgent-2024-03-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

e3111d9c3f7250309f451cfbf55845a74e692d41 Merge tag 'block-6.9-20240322' of git://git.kernel.dk/linux

Bisect log is the following:

git bisect start
git bisect good e8f897f4afef0031fe618a8e94127a0934896aba
git bisect bad a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
git bisect good 480e035fc4c714fb5536e64ab9db04fedc89e910
git bisect good 2ac2b1665d3fbec6ca709dd6ef3ea05f4a51ee4c
git bisect bad e858beeddfa3a400844c0e22d2118b3b52f1ea5e
git bisect good cba9ffdb9913dfe6be29f049ce920ce451ce7cc4
git bisect bad 27f58c04a8f438078583041468ec60597841284d
git bisect good 1e3cd03c54b76b4cbc8b31256dc3f18c417a6876
git bisect good e3111d9c3f7250309f451cfbf55845a74e692d41
git bisect bad 70293240c5ce675a67bfc48f419b093023b862b3

Looks like these

4f53138fffc2 scsi: sd: Translate data lifetime information
4977c0f4523e scsi: scsi_proto: Add structures and constants related to I/O groups and streams
96b171d6dba6 scsi: core: Query the Block Limits Extension VPD page
c819dbd07832 dt-bindings: timer: Add support for cadence TTC PWM
b9920fdd5a75

are the only candidates to test.

Looking into contents I bet on the first one in the above list. There is no gurantee old devices like this Garmin watch may support all new features.

#define SD_TIMEOUT		(30 * HZ)

Oh, yeah, this correlates with the 1 minute I see (like two attempts to ask for CDB that makes device crazy...

Please test the following:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r13182.g484193f-1-x86_64.pkg.tar.zst

Should we just go on with the bisection or would you like to short-circuit it to the commit mentioned above?

I'm going to test the above, but would it be possible to build v6.10-rc3 with that patch reverted? (If the revert is smooth)

Almost: You need to revert these two

$ git revert --no-edit -1 961990efc608  
Auto-merging drivers/scsi/sd.c                                                              
[] Revert "scsi: sd: Only print updates to permanent stream count"
 Date: Tue Jun 11 23:33:54 2024 +0300
 1 file changed, 1 insertion(+), 3 deletions(-)
$ git revert --no-edit -1 4f53138fffc2  
Auto-merging drivers/scsi/sd.c
Auto-merging drivers/scsi/sd.h
[] Revert "scsi: sd: Translate data lifetime information"
 Date: Tue Jun 11 23:33:58 2024 +0300
 2 files changed, 4 insertions(+), 98 deletions(-)

I'm about to reboot, but I believe it (484193f...) will be bad as it includes that bfa... which is merge of scsi-misc. Nevertheless, please try that revert first.

The kernel with the reverts from #60 (comment 190839) is currently building

Yeah, as I predicted, v6.8.r13182.g484193f is bad.

Here is the mainline rc with the request commits removed:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-6.10rc3-1.1-x86_64.pkg.tar.zst

Edit: If it 404's give it a few seconds, I posted a bit early and the upload should now be finished

Everything is fine and bingo! That's the culprit. Thanks a lot for helping bisecting this beast. I informed upstream already: https://lore.kernel.org/linux-scsi/20240130214911.1863909-1-bvanassche@acm.org/T/#mf4e3410d8f210454d7e4c3d1fb5c0f41e651b85f

Here is the next step for the bisection:

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-v6.8.r13167.gb718713-1-x86_64.pkg.tar.zst

Great! So we stop with the bisection I guess?

If you want to give me credit feel free to throw a Reported-by: Christian Heusel <christian@heusel.eu> or Bisected-by: Christian Heusel <christian@heusel.eu> in the loop

I'll also add this to regzbot

Yes for stopping bisection and yes for regzbot.

Oh, the credit can be discussed rather when the fix is ready. Maybe you can answer to that thread (you can do it easily with b4 mbox ... and then using your mail client against it) with your email and wishes to be Cc'ed on the progress?

@andy-shev I have applied the proposed fix to the following package, could you test it?

sudo pacman -U https://pkgbuild.com/~gromit/linux-bisection-kernels/linux-mainline-6.10rc3-1.2-x86_64.pkg.tar.zst

It is a clean v6.10-rc3 + only fix, correct? Then it fixes the issue. Thank you for the prompt kernel image build!

Yes, I have disabled the reverts from #60 (comment 190845) again and applied the patch from the email

Really nice that Bart had an idea for a fix so quickly

---
 drivers/scsi/sd.c              | 4 ++++
 drivers/usb/storage/scsiglue.c | 2 ++
 include/scsi/scsi_devinfo.h    | 4 +++-
 3 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index f6c822c9cbd2d..43b7309d5ee5b 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -63,6 +63,7 @@
 #include <scsi/scsi_cmnd.h>
 #include <scsi/scsi_dbg.h>
 #include <scsi/scsi_device.h>
+#include <scsi/scsi_devinfo.h>
 #include <scsi/scsi_driver.h>
 #include <scsi/scsi_eh.h>
 #include <scsi/scsi_host.h>
@@ -3118,6 +3119,9 @@ static void sd_read_io_hints(struct scsi_disk *sdkp, unsigned char *buffer)
 	struct scsi_mode_data data;
 	int res;
 
+	if (sdp->sdev_bflags & BLIST_SKIP_IO_HINTS)
+		return;
+
 	res = scsi_mode_sense(sdp, /*dbd=*/0x8, /*modepage=*/0x0a,
 			      /*subpage=*/0x05, buffer, SD_BUF_SIZE, SD_TIMEOUT,
 			      sdkp->max_retries, &data, &sshdr);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index b31464740f6c8..9a7185c688728 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -79,6 +79,8 @@ static int slave_alloc (struct scsi_device *sdev)
 	if (us->protocol == USB_PR_BULK && us->max_lun > 0)
 		sdev->sdev_bflags |= BLIST_FORCELUN;
 
+	sdev->sdev_bflags |= BLIST_SKIP_IO_HINTS;
+
 	return 0;
 }
 
diff --git a/include/scsi/scsi_devinfo.h b/include/scsi/scsi_devinfo.h
index 6b548dc2c4965..fa8721e49dec2 100644
--- a/include/scsi/scsi_devinfo.h
+++ b/include/scsi/scsi_devinfo.h
@@ -69,8 +69,10 @@
 #define BLIST_RETRY_ITF		((__force blist_flags_t)(1ULL << 32))
 /* Always retry ABORTED_COMMAND with ASC 0xc1 */
 #define BLIST_RETRY_ASC_C1	((__force blist_flags_t)(1ULL << 33))
+/* Do not read the I/O hints mode page */
+#define BLIST_SKIP_IO_HINTS	((__force blist_flags_t)(1ULL << 34))
 
-#define __BLIST_LAST_USED BLIST_RETRY_ASC_C1
+#define __BLIST_LAST_USED BLIST_SKIP_IO_HINTS
 
 #define __BLIST_HIGH_UNUSED (~(__BLIST_LAST_USED | \
 			       (__force blist_flags_t) \
-- 
2.45.2

mentioned in issue #61 (closed)

added statusconfirmed label and removed statusunconfirmed label

assigned to @heftig, @dvzrv, and @tpowa and unassigned @freswa, @toolybird, and @gromit

Cherry-picked the latest patches into 6.9.5.arch1.

https://lore.kernel.org/linux-scsi/20240613211828.2077477-2-bvanassche@acm.org
https://lore.kernel.org/linux-scsi/20240613211828.2077477-3-bvanassche@acm.org

closed

added resolutioncompleted label

Long timeout (more than a minute) on the connected USB mass storage device (Garmin)

Description:

Additional info:

Steps to reproduce:

Child items ...

Activity

Admin message

Long timeout (more than a minute) on the connected USB mass storage device (Garmin)

Description:

Additional info:

Steps to reproduce:

Activity