Processing arch-commits mails is slow (1-2 mails per minute)

https://lists.archlinux.org/pipermail/arch-commits/2020-December/thread.html has been rewritten every second for hours now. At the same time arch-commits has processed 1-2 mails per minute (the backlog is hopefully empty soon ~1 hour).

Something is clearly not working at intended.

assigned to @freswa

changed the description

This seems to be related but we are at a version which should include the fix. https://mail.python.org/pipermail/mailman-users/2003-October/032532.html

One issue I observed is that mail from mail.archlinux.org is being ratelimitted most likely because arch-commits receives a ton of issues, can we white list it?

I noticed the same issue this morning; mailman's ArchRunner process was using 100% CPU with 196h of total CPU time. strace showed it was rewriting the January archives (adding lots of newlines, changing previous/next thread links). In the previous days I had also noticed that the arch-dev-public archives were taking several tens of minutes to update. @jelle sounds right about possible rate limits, because my understanding is that ArchRunner only handles archive updates and cannot cause delays in email delivery.

While investigating, I managed to mess up some of the older thread archives (and also kind of deleted the 2021 January ones :P). I opted to recreate the whole arch-commits archive with /usr/lib/mailman/bin/arch --wipe. Unfortunately this renumbers most of the threads so it breaks links like arch-commits/2021-January/884377.html, but that shouldn't matter too much.

I had stopped ArchRunner while recreating the arch-commits archive, and restarting mailman (and thus ArchRunner) happily processed some 120+ new messages that were queued for archival in /var/lib/mailman/qfiles/archive. It will be interesting to see if this issue will resurface, but I think we can close this for now.

arch-commits is not archived anymore as part of the mailman3 migration (4d8dfb6a, https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/thread/FEPUJM4GRXYWFVAQQE2ELZZ6S6O5HPXL/#OKVGUCSUAFVS2VO33HKX2BUJG5N7QVXU).

closed