Currently our textcollector can sometimes fail with 'Failed to
create/acquire the lock /home/backup/$server/lock.exclusive (timeout)."
Instead of checking on a borg lock file, check if our backup snapshot
dir exists which the backup script creates and removes. This should give
less false positives then our current method.

Rate limit our archweb RSS feeds

See merge request archlinux/infrastructure!319

Cache urls which urls marks as can be cached by nginx. This offloads our
uwsgi workers and allows for speedier delivery of RSS feeds and other
cached routes.

Due to users misconfiguring their conky to query for rss updates every
second add proper rate limitting to all rss endpoints in nginx.

The websocket support always 400'd as upgrade headers where missing for
/socket.io

arch_boxes_sync: Override the "latest" link instead of following it

See merge request archlinux/infrastructure!318

Configure network correctly for Kape servers

See merge request archlinux/infrastructure!320

Previously we configured our network conf to all interfaces, which
shouldn't be done as not all our routed to the internet and this causes
systemd-network-online target to fail.

Update ServiceDown rule to 10 minutes from 5 min.

See merge request archlinux/infrastructure!321

Our dedicated servers are fairly slow when rebooting and are then not
available for 5 minutes, which means a ServiceDown notification is send
for a normal reboot.

keycloak: Add "Well-Known URL for Changing Passwords"[1]

See merge request archlinux/infrastructure!310

More and more browsers and password managers support this[2].

[1] https://w3c.github.io/webappsec-change-password-url/
[2] https://github.com/w3c/webappsec-change-password-url/issues/16

Prometheus repo exporter

See merge request archlinux/infrastructure!314

Export the repository size of our repositories so they can be monitored
and we have some useful data for repository growth in the future.

The value of the expr is not really useful as of now, but if we show the
value of probe_ssl_earliest_cert_expiry it should show the date when the
cert expires.

Prometheus alerts $value is the result of the expression, so it will be
the amount of seconds since the last backup and not the last backup
date.

postfix: Install PCRE explicit after package split[1]

See merge request archlinux/infrastructure!317

[1] https://lists.archlinux.org/pipermail/arch-dev-public/2021-January/030284.html

gemini takes a long time to run backups and it would sometimes produce false positives for not having backed up for some time.
The higher threshold should help with those false positives.

AUR systemd hardening

See merge request archlinux/infrastructure!287

This service only requires MySQL access and ability to submit an email.

This service only requires MySQL access and access to the aur home
directory (/srv/http/aurweb) to write the package/pkgbase/user files.

This service only requires MySQL access, network connection access and
the ability to create an pyalpm handle to sync the pacman db's to update
the blacklist.

This service only requires MySQL access for deleting old, empty reserved
aurweb pkgbases.

This service only requires MySQL access for updating the per-package
popularity counts.

arch_boxes_sync: Quick service filesystem hardening

See merge request archlinux/infrastructure!315

Should protect the filesystem from the script going amok and nuking the
filesystem (shouldn't happen).

Add new role which sync arch-boxes images to the repos

Closes #272

See merge request archlinux/infrastructure!296

Fix #272

Without a location nginx will always return

See merge request archlinux/infrastructure!313

Without a location block, nginx always executes the return block causing
a 302 which makes certbot unable to renew as it follows redirects.