Verified Commit 2222767c authored by Evangelos Foutras's avatar Evangelos Foutras 🐱
Browse files

gitlab-backup: Set GZIP_RSYNCABLE=yes so borg can dedup

gitlab-backup produces gzipped tarballs that cannot be meaningfully
deduplicated by borg.  This can be mitigated by passing --rsyncable
to gzip.

The above is verified by creating two new borg repositories, adding
the two most recent gitlab.archlinux.org archives to both, with the
difference of re-compressing the tarballs with `gzip -1 --rsyncable`
before adding them to the second repository.

In the first case, the 215.97 GB backup archive gets compressed and
deduplicated down to 176.24 GB. With --rsyncable it gets reduced to
just 12.79 GB. These numbers are for /srv/gitlab/data/backups only,
but the other non-tarballed files get sufficiently deduped already.

Based on the above, I am hoping to see the borg repository for gitlab
shrink over time from the current 3 TB to around 600 GB which is more
manageable.
parent 1ed1ee0f
Pipeline #8900 passed with stage
in 57 seconds
......@@ -12,4 +12,4 @@ echo "emptying backup directory ${backupdir}"
# Verify that the gitlab_backupdir in ansible was defined, otherwise we will rm /* and remove the previous backup
rm -r "${backupdir:?backup dir unset}/"*
docker exec gitlab gitlab-backup create STRATEGY=copy SKIP=tar
docker exec gitlab gitlab-backup create STRATEGY=copy SKIP=tar GZIP_RSYNCABLE=yes
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment