GitLab

Closes: #166

Start monitoring prometheus to keep check of the database growth rate
and retain data for a longer preriod in prometheus.

Extend the memcached service for the AUR to allow the memcached group to
read the socket to obtain statistics.

Kristian Klausen authored Sep 16, 2020 and Sven-Hendrik Haase committed Sep 18, 2020

3 days is a bit too late. Certbot renews the certificate 30 days before,
so 25 days should be safe and shouldn't cause any "false positives" due
to transient errors.

Record the rebuilderd queue length in prometheus so we can generate an
alert for when the queue length keeps rising. As this could be an
indication that the rebuilders have builds which are stuck.

Run the blackbox exporter on monitoring.archlinux.org to monitor other
machines http status for public services we provide. Also has an alert
for when a certificate is about to expire in 3 days.

Add a new role called prometheus_exporters which should be run on every
machine we have and starts different collectors depending on what group
the machine is in. Currently supported our the gitlab runner exporter,
rebuilder textcollector, mysqld-exporter, borg textcollector and an
node/arch exporter. The arch exporter monitors the security status and
pacman out of date packages gauge.

Introduce a new monitoring server with prometheus and alertmanager for
monitoring all our boxes.