Clear up some backup space

We're currently using 9.5 TB (out of 10 TB) on the storage box. There isn't a bigger storage box plan, so we got to clean up whatever is unneeded and takes up a lot of space.

At first glance, gitlab likely makes poor use of deduplication (due to its backup strategy?).

We could also remove the orion backups for some quick gains. Not great, not terrible.

Disk usage per server:

Backup Size	Hostname
3.5T	gemini.archlinux.org
3.0T	gitlab.archlinux.org
2.3T	orion.archlinux.org
149G	apollo.archlinux.org
136G	aur.archlinux.org
98G	matrix.archlinux.org
65G	luna.archlinux.org
54G	homedir.archlinux.org
35G	bbs.archlinux.org
27G	archlinux.org
22G	wiki.archlinux.org
15G	monitoring.archlinux.org
13G	reproducible.archlinux.org
13G	mail.archlinux.org
7.4G	bugs.archlinux.org
6.3G	quassel.archlinux.org
5.0G	accounts.archlinux.org
4.8G	state.archlinux.org
4.8G	aur-dev.archlinux.org
4.6G	phrik.archlinux.org
4.3G	security.archlinux.org
4.0G	patchwork.archlinux.org
3.5G	md.archlinux.org
863M	lists.archlinux.org

changed the description

changed the description

marked this issue as related to #359 (closed)

mentioned in merge request !429 (merged)

This also fixes #339 (closed) I think :)

Perhaps! We'll know when/if the total backup space used by gitlab.archlinux.org drops to more manageable sizes. Concurrency-wise it's not doing too bad even without it; only takes about 1.5 hours for borg-backup.service to do its thing.

Scratch the above concurrency statement, 1.5 hours was the duration of the actual borg command; gitlab-backup itself takes about 4 hours to prepare its tarballs.

That said, the borg archive creation/upload will be significantly faster than 1.5 hours if deduplication ends up working well (though we get hit by a ~35-40% slower gzip compression with --rsyncable).

Created a dashboard to (hopefully) see the size reduction from !429 (merged) in two days!

I'm seeing conflicting results with !429 (merged); the latest backup archive of gitlab.archlinux.org is 207 GB on Hetzner and just 18.6 GB on rsync.net. (Previous archives in both borg repos are around 200 GB.) Both backup destinations have had exactly two archives with GZIP_RSYNCABLE=yes so there must be some non-obvious explanation why Hetzner's archive hasn't shrunk.

Ended up removing orion's borg repo (we still have it stored on rsync.net which is roomier). After doing this, we now have 2.7 TB of free space (up from 399 GB).

Still unsure about why !429 (merged) only worked on rsync.net, but I'll go ahead and close this. Will monitor the gitlab backups on Hetzner to see if the situation improves.

closed