Only doing this on the Hetzner storage box for now; waiting for
rsync.net to upgrade to borg 1.2 so we can enable it there too.

Using GitLab's official backup tool takes too much time and, more
importantly, space; /srv/gitlab is a bit over 430G but backing it
up nearly exhausts its 1TB volume.

As we're creating btrfs snapshots and backing those up with borg, it
seems unnecessary to also create tarballs of the same data. GitLab's
documentation mentions snapshots as a viable backup strategy, and to
the restored system it should seem like recovering from a power loss.

[1] https://docs.gitlab.com/ee/raketasks/backup_restore#alternative-backup-strategies

New username; separate and longer account manager + storage passwords.

Also, have to use --remote-path=borg1 when interacting with rsync.net.

The official backup tool for GitLab takes many hours to run because it
puts everything inside tarballs and then gzips each one. It seems safe
and much more efficient to skip this step for the offsite backup while
reusing the tarballs generated by the first backup to the Storage Box.

Should save ~5 hours from the borg-backup-offsite.service execution.

The helper scripts that create mysql/postgres database dumps as well as
the script running gitlab-backup were executed after the btrfs snapshot
was taken. This resulted in stale db and gitlab backups (from last run).

Move execution of these helper scripts further up so their outputs get
included in the btrfs snapshot.

Reported-by: Kristian Klausen <kristian@klausen.dk>

Now that our textcollector depends on the condition of /backup being
available, we should be extra careful that we get rid of the directory.
If anything in the script fails, a trap now umounts and removes the
directory automatically if on btrfs.being available, we should be extra
careful that we get rid of the directory. If anything in the script
fails, a trap now umounts and removes the directory automatically if on
btrfs.

Sven-Hendrik Haase authored 4 years ago and Frederik Schwan committed 4 years ago

The perl script is currently breaking and there's not really an advantage to keeping it around.
We've never used it.

The backup script fails on our Gitlab machine as it tries to remove a
snapshot of the docker submount, which we do not want to backup anyway.

btrfs subvolume snapshot -r /var/lib/docker/btrfs /var/lib/docker/btrfs/backup-snapshot
ERROR: Not a Btrfs subvolume: Invalid argument

Use Gitlab's official backup tool for dumping the database, git
repositories and other data with a wrapper script similiar to the
postgres and mysql backup scripts.

Signed-off-by: Jelle van der Waa <jelle@archlinux.org>

When nothing is mounted, umount will error out failing the backup, so
handle this case.

Earlier it would only snapshot the top dir which resulted in some awkward moments when another btrfs drive that was mounted to /home was empty in the backups because snapshots only snapshot the topmost device in the tree.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

$PATH doesn't include that directory.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Also add mariadb support

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

The snapshot is not actually a mountpoint, but rather looks like a
directory.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

This reverts commit 5ec3d1a8.

The server uses forced commands so this doesn't work.

This allows to easily check for backups with no update to the timestamp
on the backup host.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>