Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
Co-authored-by: Kristian Klausen <kristian@klausen.dk>
Co-authored-by: moson-mo <mo-son@mailbox.org>

WireGuard was setup to provide a internal network with confidentiality,
authenticity and integrity[1]. This migrate the remaining Prometheus
exporters to use the internal WireGuard network.

[1] 664deb67 ("WireGuard all hosts")

Fix #384

- add the new role to redirect.archlinux.org
- release mirror.pkgbuild.com of all DNS duties

All servers are part of these groups which makes them redundant.

The idea bebind this is to be able to give vault access to new DevOps
members without giving away more important credentials like Hetzner's.

Makes prometheus.yml less noisy and doesn't break when non-vhost
mappings are added to blackbox_targets.

Make sure lists are indented correctly and enable lstrip_blocks to get
consistent indendation of lines inside Jinja blocks.

Monitoring just geo.mirror.pkgbuild.com would only monitor the mirror
returned for that geographical area, with this commit we monitor all the
potential returned mirrors.

Co-authored-by: Evangelos Foutras <evangelos@foutrelis.com>

Fix #443

We don't want mirror.pkgbuild.com's DNS server to be a
single-point-of-failure, so this commit adds multiple authoritative DNS
servers for the zone. The extra DNS servers are run on the geomirror
servers.

The _acme-challenge zone, used for obtaining certificates, is run solely
on mirror.pkgbuild.com's DNS server, to avoid syncing DNS records
between the servers (KISS).

We had a GeoIP mirror in the past based on nginx and its GeoIP module,
but it didn't perform very well, due to the high latency (asking a
central server for the package and then redirected to the closest
mirror).

One of the reasons for offering this service, is so we can relieve
mirror.pkgbuild.com which is burning a ton of traffic (50TB/month),
likely due to it being the default mirror in our Docker image. Another
reason is so we can offer a link to our arch-boxes images in libosinfo
(used by gnome-boxes, virt-install and virt-manager), with good enough
performance for most users.

This time we take a different approach and use a DNS based solution,
which means the latency penalty is only paid once (the first DNS
request). The downside is that the mirrors must have a valid certificate
for the same domain name, which makes using third-party mirrors a
challenge. So for now, we are just using the sponsored mirorrs
controlled by the DevOps team.

Fix #101

A dashboard will be added at a later date.

Leonidas Spyropoulos authored 3 years ago and Kristian Klausen committed 3 years ago

The dashboards is from [1] and fixed with:
sed 's/${DS_PROMETHEUS}/$datasource/g' -i synapse.json

[1] https://github.com/matrix-org/synapse/tree/c167e09fe58d3a256fb1c763b391ad6633d2507d/contrib/grafana



Closes: #290

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

Also send the metrics to dashboards.archlinux.org.

Fix #323

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Fix #263

Extend the memcached service for the AUR to allow the memcached group to
read the socket to obtain statistics.

Add a new role called prometheus_exporters which should be run on every
machine we have and starts different collectors depending on what group
the machine is in. Currently supported our the gitlab runner exporter,
rebuilder textcollector, mysqld-exporter, borg textcollector and an
node/arch exporter. The arch exporter monitors the security status and
pacman out of date packages gauge.

Introduce a new monitoring server with prometheus and alertmanager for
monitoring all our boxes.