Mirroring aur.git to github broken since 2024-06-30

Jun 30 08:58:51 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 08:59:16 aur.archlinux.org git[4176274]: Everything up-to-date
Jun 30 08:59:17 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Deactivated successfully.
Jun 30 08:59:17 aur.archlinux.org systemd[1]: Finished aurweb-github-mirror.service.
Jun 30 08:59:17 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 21.821s CPU time.
Jun 30 08:59:52 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 09:00:05 aur.archlinux.org git[4178471]: Everything up-to-date
Jun 30 09:00:05 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Deactivated successfully.
Jun 30 09:00:05 aur.archlinux.org systemd[1]: Finished aurweb-github-mirror.service.
Jun 30 09:00:05 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 3.867s CPU time.
Jun 30 09:00:58 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 09:01:30 aur.archlinux.org git[4182909]: To github.com:archlinux/aur.git
Jun 30 09:01:30 aur.archlinux.org git[4182909]:    9743eb08844a..24160a68b575  yandex-messenger -> yandex-messenger
Jun 30 09:01:30 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Deactivated successfully.
Jun 30 09:01:30 aur.archlinux.org systemd[1]: Finished aurweb-github-mirror.service.
Jun 30 09:01:30 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 21.750s CPU time.
Jun 30 09:01:38 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 09:02:10 aur.archlinux.org git[4184311]: Everything up-to-date
Jun 30 09:02:10 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Deactivated successfully.
Jun 30 09:02:10 aur.archlinux.org systemd[1]: Finished aurweb-github-mirror.service.
Jun 30 09:02:10 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 23.618s CPU time.
Jun 30 09:02:38 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 09:03:10 aur.archlinux.org git[4186920]: Everything up-to-date
Jun 30 09:03:10 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Deactivated successfully.
Jun 30 09:03:10 aur.archlinux.org systemd[1]: Finished aurweb-github-mirror.service.
Jun 30 09:03:10 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 18.990s CPU time.
Jun 30 09:03:22 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 09:09:51 aur.archlinux.org git[4188956]: send-pack: unexpected disconnect while reading sideband packet
Jun 30 09:09:52 aur.archlinux.org git[4188959]: Connection to github.com closed by remote host.
Jun 30 09:13:55 aur.archlinux.org git[4188956]: fatal: the remote end hung up unexpectedly
Jun 30 09:13:55 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Main process exited, code=exited, status=128/n/a
Jun 30 09:13:55 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Failed with result 'exit-code'.
Jun 30 09:13:55 aur.archlinux.org systemd[1]: Failed to start aurweb-github-mirror.service.
Jun 30 09:13:55 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 10min 6.418s CPU time.
Jun 30 09:13:55 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 09:20:19 aur.archlinux.org git[19500]: send-pack: unexpected disconnect while reading sideband packet
Jun 30 09:20:20 aur.archlinux.org git[19511]: Connection to github.com closed by remote host.
Jun 30 09:25:36 aur.archlinux.org git[19500]: fatal: the remote end hung up unexpectedly
Jun 30 09:25:36 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Main process exited, code=exited, status=128/n/a
Jun 30 09:25:36 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Failed with result 'exit-code'.
Jun 30 09:25:36 aur.archlinux.org systemd[1]: Failed to start aurweb-github-mirror.service.
Jun 30 09:25:36 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 11min 13.341s CPU time.
Jun 30 09:25:36 aur.archlinux.org systemd[1]: Starting aurweb-github-mirror.service...
Jun 30 09:31:54 aur.archlinux.org git[45543]: send-pack: unexpected disconnect while reading sideband packet
Jun 30 09:31:54 aur.archlinux.org git[45568]: Connection to github.com closed by remote host.
Jun 30 09:35:48 aur.archlinux.org git[45543]: fatal: the remote end hung up unexpectedly
Jun 30 09:35:49 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Main process exited, code=exited, status=128/n/a
Jun 30 09:35:49 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Failed with result 'exit-code'.
Jun 30 09:35:49 aur.archlinux.org systemd[1]: Failed to start aurweb-github-mirror.service.
Jun 30 09:35:49 aur.archlinux.org systemd[1]: aurweb-github-mirror.service: Consumed 9min 45.934s CPU time.

I'm not sure what changed around that time. It looks like git pack-objects consumes a lot of CPU for roughly 10 minutes and then the push fails.

I tried experimented a bit with git multi-pack-index and pushing only a few refs with the following commands, but even that is unreliable.

[aur@aur aur.git]$ comm -13 <(git ls-remote https://github.com/archlinux/aur.git | tr '\t' ' ' | sort) <(git show-ref | grep -v ' refs/namespaces/' | sort) | awk '{print $2}' | xargs -P1 -n1 git push git@github.com:archlinux/aur.git
end-pack: unexpected disconnect while reading sideband packet
Connection to github.com closed by remote host.
fatal: the remote end hung up unexpectedly
To github.com:archlinux/aur.git
   21240c502e79..053e1fbf3c07  tonelib-noise-reducer-bin -> tonelib-noise-reducer-bin
To github.com:archlinux/aur.git
   46097ba53eae..166a1085c5f0  mingw-w64-p11-kit -> mingw-w64-p11-kit
To github.com:archlinux/aur.git
   f1b404e3892a..1f80636c9e59  python-yaql -> python-yaql
To github.com:archlinux/aur.git
   34487538a3f3..23a31a5519fb  mingw-w64-rust-bin -> mingw-w64-rust-bin
To github.com:archlinux/aur.git
   2fa2d7509521..2c3ef087a4c5  mingw-w64-rust -> mingw-w64-rust
To github.com:archlinux/aur.git
   87933f242209..327b4a1f9103  nfpm-bin -> nfpm-bin
To github.com:archlinux/aur.git
   3a82b24ecd67..36485ddd9628  tonelib-bass-drive-bin -> tonelib-bass-drive-bin
To github.com:archlinux/aur.git
   7d011440aa4d..3897c5cc7ff3  c-intro-and-ref -> c-intro-and-ref
To github.com:archlinux/aur.git
   3b1f33a4f201..3aeb9d3b6ccd  keeper-secrets-manager-core -> keeper-secrets-manager-core
To github.com:archlinux/aur.git
   d2cd25d20b4f..43c0d3be7a56  ferdium-git -> ferdium-git
To github.com:archlinux/aur.git
   c917b532df05..51a154f443c3  musikcube-bin -> musikcube-bin
To github.com:archlinux/aur.git
   764c953bd2a4..5fab6deec681  iptables-fullconenat -> iptables-fullconenat
To github.com:archlinux/aur.git
   1f9c1cea1eb0..65a0a20a266b  mingw-w64-harfbuzz -> mingw-w64-harfbuzz
To github.com:archlinux/aur.git
   dc883bff7456..7455312c3b92  mingw-w64-poppler -> mingw-w64-poppler
To github.com:archlinux/aur.git
   44e31baab396..760ea027986b  octave-statistics -> octave-statistics
To github.com:archlinux/aur.git
   096c93443762..77378c6d0751  emacs-prop-menu-git -> emacs-prop-menu-git
To github.com:archlinux/aur.git
   0130e8e418bd..8d8d7a17d4dd  python-jenkinsapi -> python-jenkinsapi
To github.com:archlinux/aur.git
   e0d4432b1b2e..a1604e45eace  clash-verge -> clash-verge
To github.com:archlinux/aur.git
   acda6dcd3f67..a19fb3ed86c4  kde-service-menu-reimage-mod -> kde-service-menu-reimage-mod
To github.com:archlinux/aur.git
   5199e75f9052..a469936ac7a1  tonelib-tube-warmth-bin -> tonelib-tube-warmth-bin
To github.com:archlinux/aur.git
   b77b39512519..a47737c8c261  yandex-messenger -> yandex-messenger
To github.com:archlinux/aur.git
   5201eeba09a1..a4adc3f96bf8  mingw-w64-pango -> mingw-w64-pango
To github.com:archlinux/aur.git
   6bc3f96577ff..acdd9ef1eaec  iso-commander -> iso-commander
To github.com:archlinux/aur.git
   02470146a843..b178a078278e  github-desktop-bin -> github-desktop-bin
To github.com:archlinux/aur.git
   38650614195d..bca3dfad85c6  shedskin-git -> shedskin-git
To github.com:archlinux/aur.git
   fdcb46b1df68..cd4357f00045  cmake-pkg -> cmake-pkg
To github.com:archlinux/aur.git
   bc9172048785..d1ddd3e9ade7  yandex-music -> yandex-music
To github.com:archlinux/aur.git
   a7a559448dfd..d2817f44d97b  musikcube -> musikcube
To github.com:archlinux/aur.git
   883b04e1f273..d6fee1e42145  python-paramiko-expect -> python-paramiko-expect
To github.com:archlinux/aur.git
   7942a1588880..ecd4a3aa05a1  nvim-lazy -> nvim-lazy
send-pack: unexpected disconnect while reading sideband packet
Connection to github.com closed by remote host.
fatal: the remote end hung up unexpectedly
send-pack: unexpected disconnect while reading sideband packet
Connection to github.com closed by remote host.
fatal: the remote end hung up unexpectedly

I have stopped aurweb-github-mirror.timer on the AUR server for now, so we don't hit GitHub unnecessarily.