With the support for network.wireguard.* credentials[1] in systemd
v256[2], we can now easily avoid storing the credentials centrally in
our ansible vault, which is preferable as it makes the private keys less
exposed. It may also make fine-grained access easier in the future[3] as
there is no longer a vault file for each server.

All the keys have been rotated and the new private keys are only stored
on the servers.

[1] https://github.com/systemd/systemd/pull/30826
[2] https://github.com/systemd/systemd/releases/tag/v256
[3] #64

As per my announcement to arch-devops[1] and staff, this adds a Mumble
server for Arch Linux.

The password for the special root user SuperAdmin is automatically
generated on first launch and printed to the logs. I went ahead and
added it to the vault. It should not usually be required to login as
SuperAdmin though as long as there are user admins around.

This uses certbot for local certificates.

[1] https://lists.archlinux.org/archives/list/arch-devops@lists.archlinux.org/thread/AHAOSTGFJTLQDSXLWFORDKGR6RDVHYEI/

It failed to reboot during the last upgrade procedure. Upon logging into
the Equinix Metal console, we discovered that we lack access to all 4 of
the servers sponsored by Equinix Metal. They are under the CNCF account,
and it's not possible to transfer them to our organization.

Equinix Metal is being sunset, and the remaining 3 servers will also go
away on June 30th 2026. We can keep them until then, or until they fail
to boot like seoul.mirror.pkgbuild.com.

It has been disabled client side since 7.0[1] (2015-08-11), server side
since 7.7[2][3] (2018-04-02), default DSA host key generation has been
disabled since 9.1[4] (2022-10-04) and with 9.8[5] (2024-07-01) DSA
support is disabled by default at compile time. In other words, DSA has
de facto been disabled (by default) for years.

From the 9.8 release notes[5]:
"OpenSSH plans to remove support for the DSA signature algorithm in
early 2025"

The DSA host keys have been removed on our servers by running[6]:
ansible all -a "rm /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_dsa_key.pub"

[1] https://www.openssh.com/txt/release-7.0
[2] https://bugzilla.mindrot.org/show_bug.cgi?id=2662
[3] https://github.com/openssh/openssh-portable/commit/88c50a5ae20902715f0fca306bb9c38514f71679
[4] https://www.openssh.com/txt/release-9.1
[5] https://www.openssh.com/txt/release-9.8
[6] #596 (comment 203938)

Fix #596

When you become full DevOps you are basically handed the keys to the
kingdom, for this reason alone, access should not be given too easily.
Making the Junior DevOps program mandatory will ensure access is given
out in incremental steps.

The pair programming requirement has been reword to reflect the reality,
as we never really did pair programming.

Using a cert named after the primary domain with `_legacy` appended.
However, the cert is only issued for the legacy domains, not the primary
domain.

Deploy for `ipxe.archlinux.org`.

Fixes: archlinux/releng#22

CX11  -> CX22: Same price and better specs!
CX21  -> CX22: Cheaper price and same specs!
CX31  -> CX32: Cheaper price and better specs!
CPX11 -> CX22: Same price and similar spec! (for consistency)
CPX21 -> CX32: Cheaper price and better specs!
CPX41 -> CX42: Cheaper price and similar specs!

Some of the CX11 servers are not rescaled as they are on older, cheaper
grandfathered plans.

[1] https://www.hetzner.com/news/new-cx-plans/

This was done via the sync-ssh-hostkeys.yml playbook.

Signed-off-by: Christian Heusel <christian@heusel.eu>

Move backup-related variable defaults from the database roles into the
borg_client role. Also check group membership to guard installation of
database backup helper scripts.

As announced[2][3] the bugtracker has been migrated to gitlab, so
bugs.a.o can be decommissioned and replaced with a static copy[1](to
avoid link rot).

[1] https://gitlab.archlinux.org/archlinux/bugs-archive/
[2] https://archlinux.org/news/bugtracker-migration-to-gitlab-completed/
[3] https://lists.archlinux.org/hyperkitty/list/arch-dev-public@lists.archlinux.org/thread/WYXDTJ3TR2DWRQCDZK44BQDH67IDVGTS/

Fix #550
Fix #551

Extend the role (previously used for ACME DNS verifications only) to
support dynamic DNS functionality planned for sandbox.archlinux.page.

Bugbuddy is the upcoming tool for assigning package bugs to the proper
folks. The bugbuddy role will be created at a later date when the tool
is ready.

Fixes: ae53da35 ("Setup OpenSearch server for GitLab's advanced search feature[1]")
Fixes: b892c0e8 ("geomirror: new uk based mirror sponsored from jump.net.uk")

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

gitlab.archlinux.org's host SSH daemon now listens on port 2222. Adjust
the sync-ssh-hostkeys task to take this into account. Port 22 is for GL.

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Added hosts:

- repro3.pkgbuild.com
- runner3.archlinux.org
- seoul.mirror.pkgbuild.com
- sydney.mirror.pkgbuild.com

Removed hosts:

- repro1.pkgbuild.com
- runner2.archlinux.org

Equinix's AMS1 DC is being shut down so we need to recreate this box.

For Geo variety, this one is created in Frankfurt instead of Amsterdam.

Ref #495

Equinix's AMS1 DC is being shut down so we need to recreate this box.

Ref #495

As announced on the mailing list[2] pacman has been migrated to gitlab
and there is no real use for patchwork left, so it can be
decommissioned. A static copy[1] is kept around for the time being to
avoid link rot.

[1] https://gitlab.archlinux.org/archlinux/patchwork-archive
[2] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/message/7B6R5HVEC67U7B2VQ3SKUVXU4RDCRRMM/

Fix #487

With the final lists migrated to mailman3[1], the mailman2 server can
finally be killed.

When the mailman3 server was initially setup[2], it was done on a
separate server because the mailman and mailman3 packages conflicted,
and the traffic was routed over wireguard (HTTP, LMTP and SMTP).

Instead of installing mailman3 on the original lists.al.org server and
transferring the data, it was easier just to install the missing pieces
(basically Postfix and adjusting the Nginx configuration) on the ml3
server and move the IPs (to keep the IP mail reputation).

So basically the following was done:
- The IPs for the original lists.al.org was moved to the mailman3.al.org
  server
- The mailman2 datadir was transferred to mailman3.al.org server, so we
  can keep the pipermail links alive, and import missing mails if needed
- The original lists.al.org server was decommissioned
- The mailman3.al.org server was renamed to lists.al.org
- The missing pieces was added to the mailman3 role (basically Postfix +
  Nginx adjustments)
- The mailman role was deleted and the mailman3 role renamed to mailman

[1] 75ac7d09 ("mailman: Fourth and final batch of mailman3 migrated lists")
[2] 9294828f ("Setup mailman3 server")

Fix #59

This avoids triggering a GitLab push rule which rejects files that look
like secrets.

Fixes: 26f289b7 ("Capitalize the first letter of all task names")

Also tweak the documentation on rebuilderd workers and add runner1.

Vagrant Cloud has been used for years by arch-boxes[1] for publishing
Vagrant boxes. Access to the organization[2] was handed out to a few
members of the DevOps team and the creator of the organization
(arch-boxes maintainer at the time).

With this commit the control of the organization is handed over to the
DevOps team through a new Vagrant Cloud account.

[1] https://gitlab.archlinux.org/archlinux/arch-boxes
[2] https://app.vagrantup.com/archlinux/

The server has been reimaged to be sure the playbook and roles work as
intended.

We want to migrate to mailman3 as mailman2 is basically unmaintained and
requires Python 2 which is EOL.

Because the mailman and mailman3 packages conflict and we don't want to
perform a big bang migration, mailman3 must be deployed on a separate
server. mailman-web (mailman3's web interface) hasn't been packaged yet,
so for now we are using my homebrewed PKGBUILD[1].

[1] https://gist.github.com/klausenbusk/5982063f95c503754a51ed2fefb8915e

Ref #59

- add the new role to redirect.archlinux.org
- release mirror.pkgbuild.com of all DNS duties

All servers are part of these groups which makes them redundant.

The idea bebind this is to be able to give vault access to new DevOps
members without giving away more important credentials like Hetzner's.

We had a GeoIP mirror in the past based on nginx and its GeoIP module,
but it didn't perform very well, due to the high latency (asking a
central server for the package and then redirected to the closest
mirror).

One of the reasons for offering this service, is so we can relieve
mirror.pkgbuild.com which is burning a ton of traffic (50TB/month),
likely due to it being the default mirror in our Docker image. Another
reason is so we can offer a link to our arch-boxes images in libosinfo
(used by gnome-boxes, virt-install and virt-manager), with good enough
performance for most users.

This time we take a different approach and use a DNS based solution,
which means the latency penalty is only paid once (the first DNS
request). The downside is that the mirrors must have a valid certificate
for the same domain name, which makes using third-party mirrors a
challenge. So for now, we are just using the sponsored mirorrs
controlled by the DevOps team.

Fix #101

With the PHP->Python port done[1][2], there isn't much need for aur-dev
anynmore. Most things can also be tested locally and aur-dev haven't got
any love since the port (ex: allowing the aurweb maintainers to deploy
without asking DevOps).

[1] https://lists.archlinux.org/pipermail/aur-general/2022-February/036786.html
[2] !525

Kind of sensitive information that doesn't need to be available to all
hosts.

Change docs/ssh-known_hosts.txt to be partially managed by Ansible, so
custom entries can be added to the top of the file. Use the new format
to write down the host keys of our two borg hosts.

Using GitLab's official backup tool takes too much time and, more
importantly, space; /srv/gitlab is a bit over 430G but backing it
up nearly exhausts its 1TB volume.

As we're creating btrfs snapshots and backing those up with borg, it
seems unnecessary to also create tarballs of the same data. GitLab's
documentation mentions snapshots as a viable backup strategy, and to
the restored system it should seem like recovering from a power loss.

[1] https://docs.gitlab.com/ee/raketasks/backup_restore#alternative-backup-strategies