Also alphabetically sort the servers in this group.

The default sslmode is require which doesn't protect against MITM
attacks (the certificate isn't verified). The different modes are
explained here [1].

[1] https://www.postgresql.org/docs/current/libpq-ssl.html

It was using a nonexistent target path when copying the renewed cert and
was not reloading postgresql.service in order for it to reload the certs.

matrix: use C locale for the synapse database

See merge request !559

Synapse needs the database to be in C locale. Since v1.56.0, it refuses
to start when this is not the case, see [upgrade.md][1].

[1]: https://github.com/matrix-org/synapse/blob/v1.56.0/docs/upgrade.md#change-in-behaviour-for-postgresql-databases-with-unsafe-locale

/srv/gitlab has been moved to local (NVMe SSD) storage; hopefully it
won't grow too large and thus require transferring back to a volume.

nginx: remove a few fields from json_reduced

See merge request !557

This brings it in line with the non-JSON "reduced" log format.

prometheus: monitor geomirror nameservers

See merge request !556

Ensure all the nameservers serving the geo.mirror.pkgbuild.com subzone
are working and respond with a valid resource record to A/AAAA queries.

Makes prometheus.yml less noisy and doesn't break when non-vhost
mappings are added to blackbox_targets.

Make sure lists are indented correctly and enable lstrip_blocks to get
consistent indendation of lines inside Jinja blocks.

prometheus: Monitor all the mirrors backing our GeoIP mirror

Closes #443

See merge request !555

Monitoring just geo.mirror.pkgbuild.com would only monitor the mirror
returned for that geographical area, with this commit we monitor all the
potential returned mirrors.

Co-authored-by: Evangelos Foutras <evangelos@foutrelis.com>

Fix #443

arch_boxes_sync: Create predictable symlinks for latest image files

See merge request !552

This is needed so we can provide stable links for libosinfo (used by
gnome-boxes, virt-install and virt-manager for easy installation).

Potentially useful to quickly know which mirror we're talking to.

Using paccache for this which keeps up to 3 versions per package, and
also instructing it to keep any packages accessed in the past 2 weeks.

rebuilderd_worker: start removing stale chroots

See merge request !554

On some build failures, or perhaps when rebooting the boxes while a
build is ongoing, repro chroots can be left behind taking up space.

Add a service with a daily timer to remove week old chroots.

Avoid single point-of-failure for our GeoIP domain

See merge request !553

Fixes: 9f65f99c ("Add GeoIP domain for our sponsored mirros")

We don't want mirror.pkgbuild.com's DNS server to be a
single-point-of-failure, so this commit adds multiple authoritative DNS
servers for the zone. The extra DNS servers are run on the geomirror
servers.

The _acme-challenge zone, used for obtaining certificates, is run solely
on mirror.pkgbuild.com's DNS server, to avoid syncing DNS records
between the servers (KISS).

We want the GeoIP databases to be ready for use by software installed
by dependant roles; run geoipupdate when it's installed or configured.

mirror.pkgbuild.com doesn't need it.

Add GeoIP domain for our sponsored mirros

Closes #101

See merge request !522

We had a GeoIP mirror in the past based on nginx and its GeoIP module,
but it didn't perform very well, due to the high latency (asking a
central server for the package and then redirected to the closest
mirror).

One of the reasons for offering this service, is so we can relieve
mirror.pkgbuild.com which is burning a ton of traffic (50TB/month),
likely due to it being the default mirror in our Docker image. Another
reason is so we can offer a link to our arch-boxes images in libosinfo
(used by gnome-boxes, virt-install and virt-manager), with good enough
performance for most users.

This time we take a different approach and use a DNS based solution,
which means the latency penalty is only paid once (the first DNS
request). The downside is that the mirrors must have a valid certificate
for the same domain name, which makes using third-party mirrors a
challenge. So for now, we are just using the sponsored mirorrs
controlled by the DevOps team.

Fix #101

It's on github not gitlab

Ref #448

This hasn't seen much growth in the past two months and is chilling
around 13G. We can easily bump it once we have more debug packages.

With the PHP->Python port done[1][2], there isn't much need for aur-dev
anynmore. Most things can also be tested locally and aur-dev haven't got
any love since the port (ex: allowing the aurweb maintainers to deploy
without asking DevOps).

[1] https://lists.archlinux.org/pipermail/aur-general/2022-February/036786.html
[2] !525

Fix #441