Move the 'sshd_enable_includes' override to aur's host vars instead of
specifying it as part of playbooks/aur.archlinux.org. Otherwise, would
break the AUR's SSH auth if ssh.d/aurweb_config does not get included.

group_vars/all was enabling just the sshd jail so move this into the
fail2ban role defaults. patchwork, security and wiki were redefining
fail2ban_jails without deviating from the group_vars/all default and
can therefore be dropped.

The traffic hitting ping.archlinux.org has lately been exhausting its
default nf_conntrack_max limit of 64k. Bump it to 256k (which is also
the default limit found on systems with more than 4G of memory).

Suggested-by: Kristian Klausen <kristian@klausen.dk>

With the final lists migrated to mailman3[1], the mailman2 server can
finally be killed.

When the mailman3 server was initially setup[2], it was done on a
separate server because the mailman and mailman3 packages conflicted,
and the traffic was routed over wireguard (HTTP, LMTP and SMTP).

Instead of installing mailman3 on the original lists.al.org server and
transferring the data, it was easier just to install the missing pieces
(basically Postfix and adjusting the Nginx configuration) on the ml3
server and move the IPs (to keep the IP mail reputation).

So basically the following was done:
- The IPs for the original lists.al.org was moved to the mailman3.al.org
  server
- The mailman2 datadir was transferred to mailman3.al.org server, so we
  can keep the pipermail links alive, and import missing mails if needed
- The original lists.al.org server was decommissioned
- The mailman3.al.org server was renamed to lists.al.org
- The missing pieces was added to the mailman3 role (basically Postfix +
  Nginx adjustments)
- The mailman role was deleted and the mailman3 role renamed to mailman

[1] 75ac7d09 ("mailman: Fourth and final batch of mailman3 migrated lists")
[2] 9294828f ("Setup mailman3 server")

Fix #59

These roles are very similar and can be merged into a single new role.

Note: The archive mirror is changed from a 4-hour sync to minutely for
conformity with the other two mirrors. In practice this doesn't matter
as it was already taking over 4 hours to finish and was starting again
right after its previous run.

The default of 0.5 has proven insufficient on at least 3 boxes so far.

We moved away from raid6 a while back; update the host var to reflect
the current configuration.

This box is very sussy and really likes to fill up its zram swap:

  [root@reproducible ~]# zramctl
  NAME       ALGORITHM DISKSIZE  DATA  COMPR TOTAL STREAMS MOUNTPOINT
  /dev/zram0 lzo-rle       1.9G  1.5G 183.4M  196M       1 [SWAP]

  [root@reproducible ~]# free -m
                 total        used        free      shared  buff/cache   available
  Mem:            1928         529          73           5        1325        1236
  Swap:           1927        1543         384

Fixes: 4a5748ea ("Bump zram-fraction to 1.0 for reproducible.archlinux.org")

Its disks were migrated to a new server (prompted by an unsolvable issue
with the previous box's network interface; might have been a mobo issue).

The runner was accidentally made "specific", which can't be reverted[1].

[1] https://gitlab.com/gitlab-org/gitlab/-/issues/16167

For some workloads running in a container is too restrictive, ex:
arch-boxes (loop device, filesystem mount, pacstrap) and archiso
(pacstrap). Currently they both run a TCG accelerated QEMU VM, which is
very slow and painful to work with. We should provide a better option to
our users!

This adds a hardware accelerated VM for this kinds of workloads, which
is way faster and you can do whatever you like (mostly)!

Fix #283

This box somehow gets a compression ratio of over 12; bump its zram
fraction accordingly, to stop getting alerts about high swap usage.

debuginfod.archlinux.org runs into high swap often, but also gets good
compression ratio; try upping the zram size to 100% of RAM (from 50%).

Extend the removal of the dashes from unencrypted YAML documents to
encrypted ones as well.

Fixes: a9e0790f ("Remove the three dashes from all YAML documents")

These are used to signal the start of the document in a stream of many
documents. As Ansible only supports one YAML document per file this is
unnecessary. About a third of our YAML documents already lacked these.

Nobody uses this for alert management and we also have Grafana showing
the alerts now.

We want to migrate to mailman3 as mailman2 is basically unmaintained and
requires Python 2 which is EOL.

Because the mailman and mailman3 packages conflict and we don't want to
perform a big bang migration, mailman3 must be deployed on a separate
server. mailman-web (mailman3's web interface) hasn't been packaged yet,
so for now we are using my homebrewed PKGBUILD[1].

[1] https://gist.github.com/klausenbusk/5982063f95c503754a51ed2fefb8915e

Ref #59

Fixes: afb582b1 ("geomirror: extract acme dns challenge into new role")

- add the new role to redirect.archlinux.org
- release mirror.pkgbuild.com of all DNS duties

- group_vars/all/vault_mariadb.yml: remove 'zabbix' database user
- misc/vaults/additional-credentials.vault: remove zabbix irc bot
- roles/dbscripts/tasks/main.yml: drop unused tier0 mirror access

The idea bebind this is to be able to give vault access to new DevOps
members without giving away more important credentials like Hetzner's.

We don't want mirror.pkgbuild.com's DNS server to be a
single-point-of-failure, so this commit adds multiple authoritative DNS
servers for the zone. The extra DNS servers are run on the geomirror
servers.

The _acme-challenge zone, used for obtaining certificates, is run solely
on mirror.pkgbuild.com's DNS server, to avoid syncing DNS records
between the servers (KISS).

With the PHP->Python port done[1][2], there isn't much need for aur-dev
anynmore. Most things can also be tested locally and aur-dev haven't got
any love since the port (ex: allowing the aurweb maintainers to deploy
without asking DevOps).

[1] https://lists.archlinux.org/pipermail/aur-general/2022-February/036786.html
[2] !525

This is needed for prometheus memcached exporter to work.

(AUR doesn't seem to use memcached anymore, but changed it for
consistency.)

Does not seem possible to communicate with hosts in the same subnet
without going through the gateway. Matches the configuration of our
other dedicated servers at Hetzner.

Add vault variables described in !532 (for aur-dev this time).

Kind of sensitive information that doesn't need to be available to all
hosts.

Change docs/ssh-known_hosts.txt to be partially managed by Ansible, so
custom entries can be added to the top of the file. Use the new format
to write down the host keys of our two borg hosts.

Kevin Morris authored 3 years ago and Evangelos Foutras committed 3 years ago

[foutrelis: add vault variables described in !532]

Signed-off-by: Kevin Morris <kevr@0cost.org>
Signed-off-by: Evangelos Foutras <evangelos@foutrelis.com>

The two secrets: vault_aurweb_{secret,postmaster}

Using GitLab's official backup tool takes too much time and, more
importantly, space; /srv/gitlab is a bit over 430G but backing it
up nearly exhausts its 1TB volume.

As we're creating btrfs snapshots and backing those up with borg, it
seems unnecessary to also create tarballs of the same data. GitLab's
documentation mentions snapshots as a viable backup strategy, and to
the restored system it should seem like recovering from a power loss.

[1] https://docs.gitlab.com/ee/raketasks/backup_restore#alternative-backup-strategies

The sponsored mirrors have a ton of storage, but mirror.pkgbuild.com
doesn't, so debug packages aren't synced to it.

[1] {america,asia,europe}.mirror.pkgbuild.com

Fixes: 91f9df69 ("Add missing wireguard for gluebuddy")

Fixes: d88c0b95 ("Initialize gluebuddy host")