PowerDNS provides a neat way to implement GeoIP-based redirection and
automatic failover. With GeoLite2-City database, it is able to select
the closest mirror from a list of IPs we provide. Every 60 seconds it
also checks if the mirror's HTTPS URL is working as expected; if that
check fails, it stops giving it out (this acts as automatic failover).

archbuild: Distribute CPU and IO resources equally among users

See merge request !564

archbuild: Turn off Git's safe.directory

See merge request !561

Without this setting, Git exits with an error when the repository is not
owned by the current user. This messes with our shared srcdest.

Packer bootstrap tweaks

See merge request !562

New hcloud adds protection fields to servers, volumes and floating IPs.

Since we are now using the local disk instead of a volume (which can be
scaled up easily) it helps to have a more consistent view of free space.

All database user passwords have been updated to use scram-sha-256, so
there's no need for backward compatibility with md5.

Also remove the suggestion to call delete_old_cluster.sh; it's now being
created under /tmp and it only contains a command to remove the old data
directory. (We can do the latter ourselves after some time has passed.)

Ensure the correct version is installed and matches $FROM_VERSION.

Commit 8f113698b63b15a4e0a4b15d3ee37238c1d1821d upstream:

  Remove analyze_new_cluster script from pg_upgrade

  Since this script just runs vacuumdb anyway, remove the script and
  replace the instructions to run it with instructions to run vacuumdb
  directly.

Not much point in vimdiff'ing pg_hba.conf and postgresql.conf.

Adapt upgrade_pg.sh to avoid manipulating /var/lib/postgres' structure
as the postgres user. Instead, create a new empty data directory owned
by postgres for initdb to use.

postgres: rebase config to postgresql 14.2-1

See merge request !560

Also alphabetically sort the servers in this group.

The default sslmode is require which doesn't protect against MITM
attacks (the certificate isn't verified). The different modes are
explained here [1].

[1] https://www.postgresql.org/docs/current/libpq-ssl.html

It was using a nonexistent target path when copying the renewed cert and
was not reloading postgresql.service in order for it to reload the certs.

matrix: use C locale for the synapse database

See merge request !559

Synapse needs the database to be in C locale. Since v1.56.0, it refuses
to start when this is not the case, see [upgrade.md][1].

[1]: https://github.com/matrix-org/synapse/blob/v1.56.0/docs/upgrade.md#change-in-behaviour-for-postgresql-databases-with-unsafe-locale

/srv/gitlab has been moved to local (NVMe SSD) storage; hopefully it
won't grow too large and thus require transferring back to a volume.

nginx: remove a few fields from json_reduced

See merge request !557

This brings it in line with the non-JSON "reduced" log format.

prometheus: monitor geomirror nameservers

See merge request !556

Ensure all the nameservers serving the geo.mirror.pkgbuild.com subzone
are working and respond with a valid resource record to A/AAAA queries.

Makes prometheus.yml less noisy and doesn't break when non-vhost
mappings are added to blackbox_targets.

Make sure lists are indented correctly and enable lstrip_blocks to get
consistent indendation of lines inside Jinja blocks.

prometheus: Monitor all the mirrors backing our GeoIP mirror

Closes #443

See merge request !555

Monitoring just geo.mirror.pkgbuild.com would only monitor the mirror
returned for that geographical area, with this commit we monitor all the
potential returned mirrors.

Co-authored-by: Evangelos Foutras <evangelos@foutrelis.com>

Fix #443