Add a new role called prometheus_exporters which should be run on every
machine we have and starts different collectors depending on what group
the machine is in. Currently supported our the gitlab runner exporter,
rebuilder textcollector, mysqld-exporter, borg textcollector and an
node/arch exporter. The arch exporter monitors the security status and
pacman out of date packages gauge.

- home directory needs 751 - nginx accesses it to serve static files
- cache and sessions directories are used only by PHP -> 750
- uploads is public -> 755

Note that the "fix home permissions" task was duplicated. Other tasks
fixing permissions were moved above.

This is much cleaner because the nginx role does not have to set the
fastcgi_cache variable to "false" by default, which was overridden by
host_vars/apollo.archlinux.org to "wiki", but the value was still
hardcoded in the config.

At first, I was wondering that the cache "zone" name should be
generalized to improve the configuration (from the original per-host to
per-service), but that would be an overkill since the fastcgi cache is
used only for the wiki...

This is needed for the role to work in containers. The option will also
be applied upstream, see the upstream ticket:
https://github.com/smartmontools/smartmontools/issues/62

The previous task creating the "zabbix_agent" user in the database stays
here as it actually needs the mariadb role. But note that it uses a
hardcoded name "zabbix_agent" for setting the password. The zabbix_agent
uses a different variable ("{{zabbix_agent_mysql_password}}") in the
my.cnf.j2 template, but I don't see where the variable is defined...

This role actually uses a handler from nginx to reload nginx.service.

Otherwise the timer may be started before mysqld and the service would
fail at the first start.

archwiki-runjobs.service is one-shot and timer-activated, it is not
supposed to be enabled.

This happens in the local Docker container, not sure about the
production environment...

The backup script fails on our Gitlab machine as it tries to remove a
snapshot of the docker submount, which we do not want to backup anyway.

btrfs subvolume snapshot -r /var/lib/docker/btrfs /var/lib/docker/btrfs/backup-snapshot
ERROR: Not a Btrfs subvolume: Invalid argument

It's per worker, and we have 5 of them.

Introduce a new monitoring server with prometheus and alertmanager for
monitoring all our boxes.

This is now built enitrely in GitLab CI in the arch-boxes repo so this is no longer required.

As all these services do maintenance tasks and query the database so
it's better to spread out the load.

Allow the http user to access the conference home directory.

Signed-off-by: Morten Linderud <morten@linderud.pw>