Record the rebuilderd queue length in prometheus so we can generate an
alert for when the queue length keeps rising. As this could be an
indication that the rebuilders have builds which are stuck.

Add a new role called prometheus_exporters which should be run on every
machine we have and starts different collectors depending on what group
the machine is in. Currently supported our the gitlab runner exporter,
rebuilder textcollector, mysqld-exporter, borg textcollector and an
node/arch exporter. The arch exporter monitors the security status and
pacman out of date packages gauge.

Introduce a new monitoring server with prometheus and alertmanager for
monitoring all our boxes.

reproducible.archlinux.org has no memcached installed.

Added a new server for the mail migration.

Both boxes died and are no longer responsive, retire them.

Changed the playbook to add the missing php modules and add the AUR role. Also set the
domain and the git branch variables so the right version of the AUR is installed.

Apparently the host without prio is the traffic shaped one which has been very slow for us.

It's an old box that's died and we have no contacts for the box. We haven't been able to reach it for a long time. Let's consider it dead and move on. It hosted no critical infrastructure.

Create a new VPS for running the master rebuilderd daemon which
coordinates the rebuild tasks to rebuilderd ndoes.

Signed-off-by: Jelle van der Waa <jelle@vdwaa.nl>

sgp and mex both have disk errors and died, so remove them to hosts.

Signed-off-by: Jelle van der Waa <jelle@vdwaa.nl>

Given flyspray is not maintained at all and that it is a security liability,
we are creating a VPS for it and isolating it from the rest of the resources.

This took quite a bit of manual work on the particular systems as they were partitioned in all sorts of ways that didn't allow for a big continuous chunk of storage such as required by the archive. I've manually repartitioned everything and it seems to work just fine. Now all systems have a big 4TB / and that's that.

add to mysql-servers for monitoring and fix the php-fpm extensions.