Closes: #166

Kristian Klausen authored Sep 16, 2020 and Sven-Hendrik Haase committed Sep 18, 2020

3 days is a bit too late. Certbot renews the certificate 30 days before,
so 25 days should be safe and shouldn't cause any "false positives" due
to transient errors.

Record the rebuilderd queue length in prometheus so we can generate an
alert for when the queue length keeps rising. As this could be an
indication that the rebuilders have builds which are stuck.

Run the blackbox exporter on monitoring.archlinux.org to monitor other
machines http status for public services we provide. Also has an alert
for when a certificate is about to expire in 3 days.

Introduce a new monitoring server with prometheus and alertmanager for
monitoring all our boxes.