Split repos server to repos and archive

Split repos.archlinux.org to repos.archlinux.org and archive.archlinux.org

The current repos.al.org server has 4x hdd setup in RAID10 to support both archive and repos. With the merge of extra and community the extra folder in /srv/pool has many files in there and when mirrors request to rsync from it it takes some seconds to generate the list to send before it starts syncing. During that period there's a high chance a new package to be updated resulting in rewrite of the db.ta.gz files which lead to deleting those from the mirrors (this is an edge case in POSIX which btrfs actually doesn't handle but not a bug). By splitting the server and moving the repos into a 2x SSD raid1 setup it will make it much faster to operate and less likely to trigger the bug. Better technical explanation in btrfs ML 1, 2
The second reason to split it is for seperation of concerns of current server for repos and archive.

the move to the archive can happen async (every 5 mins) on a systemd timer which will rsync from the repos /srv/archive to the actual archive (addition only, not delete)
every day a separate systemd timer will clean up old pkg files from /src/archive older than 24h
mirrors which mirror archive need to include change their scripts to additionally mirror from the new server
borg backups needs to be updated to handle new server

Trigger backup
Create new repos server with 2x SSD raid1
Mirror the current repos.al.org except archive
Downtime for Package Maintainers for 2 hours to change the DNS (no change in dbscripts)
Repurpose current repos.al.org to archive.al.org
Deploy the two systemd timers/services to async copy and cleanup packages to archive in repos.al.org
Deploy mirror archive scripts to rsync from archive.al.org
Inform other mirrors who sync archive about the change
Trigger backup

@klausenbusk: archive-tools: ensure that the daily snapshot does not run while syncing
TBA: write sync script to sync from repo to archive box
TBA: write cleanup tools for the "archive" on the new repo server
@gromit: Check what needs changing in https://github.com/archlinux/archlinux-repro -> https://github.com/archlinux/archlinux-repro/issues/131
Migrate the Data from gemini to repos (Careful: ACL's and groups)
- state repos
- ftp dir
TBA: Inform the packagers & mirror-admins when we are ready to do the migration
- New hostkeys: https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/269895fcf2998a66f7e785c6e5be68806b769f74/docs/ssh-hostkeys.txt#L288-295
- Downtime window
- Home directories

Edited Jun 23, 2024 by Christian Heusel

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information