Sven-Hendrik Haase authored 3 years ago and Kristian Klausen committed 3 years ago

Co-authored-by: Kristian Klausen <kristian@klausen.dk>

I think this is all that is missing to actually start monitoring
man.archlinux.org.

This adds a collaborative markdown editor as newly offered service which
is available via login for all Arch Linux Staff with an option to allow
anonymous edits by users (not default). Users are managed via keycloak
and require the Staff role to be allowed in, non staff keycloak users
currently will receive an internal server error due to an upstream
issue.

Closes: #231

Remove apollo from hosts.

Added security.archlinux.org to the relevant groups on hosts and created a host_vars
so we can run the all-hosts-basic.

Added patchwork to the relevant groups on hosts and created a new host_vars file
for it.

Added the wiki.archlinux.org machine to the hosts file as part of the respective
groups.

Setup Kape servers as archive mirrors (asia,europe,america), Gitlab
runner and Rebuilderd worker. All machines except runner1 are EFI
machines with grub setup and a EFI parition which is not supported by
our ansible install role and is manually rolled out.

Jelle van der Waa authored 4 years ago and Jelle van der Waa committed 4 years ago

The repro3.pkgbuild.com machine was a packet.net box with an Ubuntu
installation. Now converted to an Arch Linux installation managed by
ansible with a new rebuilderd_worker role.

Jelle van der Waa authored 4 years ago and Jelle van der Waa committed 4 years ago

Orion has been replaced by gemini and for mail by mail.archlinux.org

As it turns out, secure-runner2 isn't fast enough to serve as CI/CD and if we keep rescaling it to be
large enough, it'll be more expensive than secure-runner1 which is a lot faster. So, it'd be most
useful to just get rid of this VPS.

Jelle van der Waa authored 4 years ago and Sven-Hendrik Haase committed 4 years ago

The WKD webservice ran on orion, but as we want to retire it, we will
move it to it's own CX11 VPS. As it's just a simple web page.

Extend the memcached service for the AUR to allow the memcached group to
read the socket to obtain statistics.

Jelle van der Waa authored 4 years ago and Sven-Hendrik Haase committed 4 years ago

The PIA boxes are retired.

Record the rebuilderd queue length in prometheus so we can generate an
alert for when the queue length keeps rising. As this could be an
indication that the rebuilders have builds which are stuck.

Add a new role called prometheus_exporters which should be run on every
machine we have and starts different collectors depending on what group
the machine is in. Currently supported our the gitlab runner exporter,
rebuilder textcollector, mysqld-exporter, borg textcollector and an
node/arch exporter. The arch exporter monitors the security status and
pacman out of date packages gauge.

Introduce a new monitoring server with prometheus and alertmanager for
monitoring all our boxes.

reproducible.archlinux.org has no memcached installed.

Added a new server for the mail migration.

Both boxes died and are no longer responsive, retire them.