Intermittent "unexpected EOF" while downloading container layers when built with go 1.24

Description:

When pulling images, there occasionally occurs a segmentation fault in dockerd. As this is built with Go 1.24, please refer to the upstream issue: https://github.com/moby/moby/issues/49513. Until this issue is resolved, it is recommended to avoid packaging with Go 1.24 and temporarily switch to using Go 1.23 as a workaround.

Additional info:

• package version(s): docker 28.0.1 with go1.24.1 at git commit bbd0a17ccc
• config and/or log files:
• link to upstream bug report, if any: https://github.com/moby/moby/issues/49513

Steps to reproduce:

1. Install the latest Docker version via pacman -S docker.
2. Repeatedly attempt to pull images using docker pull SOME_IMAGE
3. When encountering an unexpected EOF error, inspect Docker service logs with journalctl -xeu docker.
4. Identify crash reports similar to the following:

Process 36200 (dockerd) of user 0 dumped core.

                                                   Stack trace of thread 36220:
                                                   #0  0x00005f257ce81098 n/a (/usr/bin/dockerd + 0x39098)
                                                   #1  0x00005f257cedfbd9 n/a (/usr/bin/dockerd + 0x97bd9)
                                                   #2  0x00005f257cee5029 n/a (/usr/bin/dockerd + 0x9d029)
                                                   #3  0x00005f257cedc516 n/a (/usr/bin/dockerd + 0x94516)
                                                   #4  0x00005f257ceb0625 n/a (/usr/bin/dockerd + 0x68625)
                                                   #5  0x00005f257ceb02df n/a (/usr/bin/dockerd + 0x682df)
                                                   ELF object binary architecture: AMD x86-64

added priority3-normal severity4-low statusunconfirmed labels

assigned to @freswa, @foxboron, @toolybird, and @gromit

I believe I am having the same issue. I didn't check the journal, but the symptom is the same.

I downgraded the a docker package built with Go 1.23 instead and the issue went away.

https://github.com/moby/moby/issues/49513#issuecomment-2744164263

If I could reproduce it, I would git-bisect golang between v1.23.0 to v1.24.0 and see where it breaks.

Is anyone who can reproduce the issue bisecting or need help with that?

Thanks for the report.

switch to using Go 1.23 as a workaround

Eww, not realistic. As per comment from @loqs, someone really needs to identify the bad commit and report it upstream..

$ git bisect start
status: waiting for both good and bad commits
$ git bisect bad go1.24.0
status: waiting for good commit(s), bad commit known
$ git bisect good go1.23.0
Bisecting: a merge base must be tested
[5e8a7316658c2f300a375041b6e0a606fec4c5f2] README: fix CC BY license name
$ git describe 
fatal: No annotated tags can describe '5e8a7316658c2f300a375041b6e0a606fec4c5f2'.
However, there were unannotated tags: try --tags.
$ git describe --tags
go1.19beta1-7546-g5e8a7316658

@toolybird does that merge base seem reasonable?

added scoperegression statuswaiting-upstream labels and removed statusunconfirmed label

assigned to @seblu and @lfleischer and unassigned @freswa, @toolybird, and @gromit

If it helps docker-1:28.0.1-1.1-x86_64.pkg.tar.zst built with go 2:1.19beta1.r7546.g5e8a731-1 which is the merge base mentioned above.

@loqs I've just attempted testing with this binary.

So far, looking fine. Do you have the PKGBUILD handy you're utilising to build Docker with the compiled version of Go, and is the bisected version/commit of Go something that you're providing in said PKGBUILD?

Nevermind I don't need it I've started bisecting and will post back here and upstream with my findings.

So far, looking fine. Do you have the PKGBUILD handy you're utilising to build Docker with the compiled version of Go, and is the bisected version/commit of Go something that you're providing in said PKGBUILD?

The PKGBUILD I am using for docker is 1:28.0.1-1 but with the pkgrel incremented to 1.1 so you can tell the versions apart. The bisection build for go is bisecting.

$ git bisect good
Bisecting: 829 revisions left to test after this (roughly 10 steps)
[87a89fa45130d4406fa4d9f0882b9c5014240d03] runtime: add the checkPtraceScope to skip certain tests
$ git describe --tags
go1.19beta1-8376-g87a89fa4513

The current command I am using to build docker is pkgctl build -I ~/builds/go/go-2\:1.19beta1.r8376.g87a89fa-1-x86_64.pkg.tar.zst -I /var/cache/pacman/pkg/cmake-3.31.6-1-x86_64.pkg.tar.zst which includes the go that is being bisected and cmake 3.31.6 as cmake 4 is not support by docker 28.0.1.

The next package to test would be docker-1:28.0.1-1.2-x86_64.pkg.tar.zst built by go-2:1.19beta1.r8376.g87a89fa-1.

Great, I didn't know about pkgctl build.

I've just been manually building Go from the source of the bisected commits and running makepkg -s with said version of Go.

I've also filed away each package with the version of Go it's built with, incase I need to go back to it for whatever reason.

-rw-r--r-- 1 amir amir  24M 2025-04-01 10:36 go1.19beta1-7546-g5e8a7316658-docker-1_28.0.1-1.1-x86_64.pkg.tar.zst
-rw-r--r-- 1 amir amir  24M 2025-04-01 13:23 go1.19beta1-7961-g1b5ae45181-docker-1:28.0.4-1-x86_64.pkg.tar.zst
-rw-r--r-- 1 amir amir  24M 2025-04-01 13:05 go1.19beta1-8376-g87a89fa451-docker-1:28.0.4-1-x86_64.pkg.tar.zst

I'm up to:

Bisecting: 207 revisions left to test after this (roughly 8 steps)
[ba42120723a8bb4161c4f54c93f7ab3234923473] runtime: properly compute whether PC is inside vDSO pages

I'm having the same issues but it's taken me a few days to get signed up to this site.

Is upstream here?: https://github.com/moby/moby/issues/49513

I'd like to pitch in with this to get a resolution and agree this isn't a distro-specific issue, but would like direction as to where to report this if the above link isn't correct.

@toolybird please see the discussion at: https://github.com/moby/moby/issues/49513

One of the maintainers on there has been trying to replicate the issue on different systems but has failed to do so suggesting that this in fact may be an Arch specific issue. In which case how to remedy?

Do as that maintainer suggested in https://github.com/moby/moby/issues/49513#issuecomment-2744164263 and bisect the issue?

@loqs I've no idea how to do that I'm afraid. Discussion on moby now seems to be pointing towards a go bug in 1.24 that's affecting other things such as podman and dagger.

That is why the suggestion is to bisect go between 1.23 and 1.24 to identify the bad commit in go. #3 (comment 257237) contains docker built with the first bisection point of go between 1.24 and 1.24. If you do not want to use a binary I built I can provide you step by step instructions to build it yourself. @strit would be willing to perform the bisection?

Okay so based on my testing, the following is the output of the bisecting:

git bisect start
# status: waiting for both good and bad commits
# bad: [3901409b5d0fb7c85a3e6730a59943cc93b2835c] [release-branch.go1.24] go1.24.0
git bisect bad 3901409b5d0fb7c85a3e6730a59943cc93b2835c
# good: [6885bad7dd86880be6929c02085e5c7a67ff2887] [release-branch.go1.23] go1.23.0
git bisect good 6885bad7dd86880be6929c02085e5c7a67ff2887
# good: [5e8a7316658c2f300a375041b6e0a606fec4c5f2] README: fix CC BY license name
git bisect good 5e8a7316658c2f300a375041b6e0a606fec4c5f2
# bad: [87a89fa45130d4406fa4d9f0882b9c5014240d03] runtime: add the checkPtraceScope to skip certain tests
git bisect bad 87a89fa45130d4406fa4d9f0882b9c5014240d03
# good: [1b5ae45181ef5274045b9b93ae0603ebb34fa811] os/user: User.GroupIds shouldn't error on users with no groups
git bisect good 1b5ae45181ef5274045b9b93ae0603ebb34fa811
# good: [ba42120723a8bb4161c4f54c93f7ab3234923473] runtime: properly compute whether PC is inside vDSO pages
git bisect good ba42120723a8bb4161c4f54c93f7ab3234923473
# bad: [0733682e5ff4cd294f5eccb31cbe87a543147bc6] internal/runtime/maps: initial swiss table map implementation
git bisect bad 0733682e5ff4cd294f5eccb31cbe87a543147bc6
# bad: [e86982c515ba4a494fb1f8e1367f4238a2b59c2e] encoding/json: add omitzero option
git bisect bad e86982c515ba4a494fb1f8e1367f4238a2b59c2e
# good: [6536c207c2309da7c1c21e3669f8ddf491e31f5b] net: improve GODEBUG=netdns=1 debug messages
git bisect good 6536c207c2309da7c1c21e3669f8ddf491e31f5b
# bad: [0206eb9679dd3819c44912e6ebdba3b0fbf959a3] README.vendor: add note about GOROOT, recommend fresh go
git bisect bad 0206eb9679dd3819c44912e6ebdba3b0fbf959a3
# bad: [f4b7291d73c40af90fdbc5fa0d72a26c5f127931] crypto/internal/randutil: use math/rand/v2
git bisect bad f4b7291d73c40af90fdbc5fa0d72a26c5f127931
# good: [327074551a2f22f2c0b8e444d1673c86f77ca745] cmd/go: exit non-zero if version -m is used with a non-Go file
git bisect good 327074551a2f22f2c0b8e444d1673c86f77ca745
# bad: [3637ebf1e0d833234c1025db1faceeae6fe6ecbb] crypto/internal/cryptotest: fix typos in AEAD test comments
git bisect bad 3637ebf1e0d833234c1025db1faceeae6fe6ecbb
# bad: [86781963cc2bf0d3dbd1eccebd8a2f080846f3ee] runtime: align vgetrandom states to cache line
git bisect bad 86781963cc2bf0d3dbd1eccebd8a2f080846f3ee
# first bad commit: [86781963cc2bf0d3dbd1eccebd8a2f080846f3ee] runtime: align vgetrandom states to cache line

I'm just going back and confirming that all the "good" passes pass with at least 50 runs of the test. This is because I wasn't able to find a way to reproduce the issue reliably; I assumed 10 successful runs of the test for said commit as a pass.

Once I've confirmed this locally I'll post upstream with my bisect results.

Results from the additional testing seem to suggest the bisect was correct.

I've attached Docker built against the respective version of Go for the last known good and bad commit below. It would be useful for those with affected machines to also confirm the findings.

Good commit: go1.19beta1-8197-g327074551a-docker-1_28.0.4-1-x86_64.pkg.tar.zst

Bad commit: go1.19beta1-8198-g86781963cc-docker-1_28.0.4-1-x86_64.pkg.tar.zst

Have you tried as a cross check go1.24.1 with 86781963cc2bf0d3dbd1eccebd8a2f080846f3ee reverted?

I have not, but I have time to do that now. I'll report back shortly.

Yep, looks like no crashes with that commit reverted.

For clarity, I utilised go1.24.2 and reverted 86781963cc2bf0d3dbd1eccebd8a2f080846f3ee.

Attached is the compiled version of Docker with those changes:

go1.24.2-revert86781963cc2bf0d3dbd1eccebd8a2f080846f3ee-docker-1_28.0.4-1-x86_64.pkg.tar.zst

I'm running the above version on both my desktop and server, given that I was able to reliably trigger the issue on my server.

I'll report back first thing tomorrow morning, or earlier if I somehow manage to experience any crashes.

So I had left the test running in the background on my desktop and managed to get a crash with go1.24.2 and 86781963cc2bf0d3dbd1eccebd8a2f080846f3ee reverted.

I've gone back and retraced my steps and am currently running more tests against both the good and bad commit packages above.

Bad commit, I can reproduce on almost every first run. Good commit, I've gone back to running several rounds of 50 tests (but given each test takes between 3-5 minutes, this will take some time to confirm).

Can others who seem to be able to reproduce the behaviour please try the good commit package and report back?

I guess it's possible that something between 1.23.0 and 1.24.2 after 86781963cc2bf0d3dbd1eccebd8a2f080846f3ee could be contributing to the behaviour, or whatever race that appears to be occurring seems less prevalent with 327074551a2f22f2c0b8e444d1673c86f77ca745.

Latest findings have been shared upstream with both Docker and Go.

This commit being identified as the culprit.

I've built Docker with Go 1.24.2 and eb6f2c24cd17c0ca1df7e343f8d9187eef7d6e13 reverted and attached below.

go1.24.2-revert-eb6f2c24cd17c0ca1df7e343f8d9187eef7d6e13-docker-1_28.0.4-1-x86_64.pkg.tar.zst

I'm running this on both my server and desktop, and going to try several hundred runs to make sure that I don't trigger the rare crash with the commit included.

@toolybird can you please move the issue to go as @nightah linked to there is an open issue there, upstream go can reproduce has marked the issue as critical and is aiming to get the fix into the next patch release.

closed

moved to go#4 (closed)