We want to have the repos and the archive on different servers, to
ensure separation of concerns.

The I/O on gemini is also very sluggish, which is likely related to the
archive design (millions of files on spinning rust), and this has
resulted in rsync deleting the package database files due to a race
condtion[1][2]. The does not solve the race condition, but it should
make it less likely.

We also need to rethink the archive design, but for now we are just
isolating the problem.

There is some "brokenness" related to the archive no longer being on the
same server as the repos, which will be fixed in subsequent commits.

[1] https://lore.kernel.org/linux-btrfs/00ed09b9-d60c-4605-b3b6-f4e79bf92fca@foutras.com/
[2] https://lore.kernel.org/linux-btrfs/ZP8AWKMVYOY0mAwq@debian0.Home/

Relates to #531

As per my announcement to arch-devops[1] and staff, this adds a Mumble
server for Arch Linux.

The password for the special root user SuperAdmin is automatically
generated on first launch and printed to the logs. I went ahead and
added it to the vault. It should not usually be required to login as
SuperAdmin though as long as there are user admins around.

This uses certbot for local certificates.

[1] https://lists.archlinux.org/archives/list/arch-devops@lists.archlinux.org/thread/AHAOSTGFJTLQDSXLWFORDKGR6RDVHYEI/

It failed to reboot during the last upgrade procedure. Upon logging into
the Equinix Metal console, we discovered that we lack access to all 4 of
the servers sponsored by Equinix Metal. They are under the CNCF account,
and it's not possible to transfer them to our organization.

Equinix Metal is being sunset, and the remaining 3 servers will also go
away on June 30th 2026. We can keep them until then, or until they fail
to boot like seoul.mirror.pkgbuild.com.

As discussed in archlinux/infrastructure#531


we want to split the repo and the archive server and as a first step of
that we're commissioning this AX41-NVME server from hetzner to serve as
a future repo host.

Signed-off-by: Christian Heusel <christian@heusel.eu>

As announced[2][3] the bugtracker has been migrated to gitlab, so
bugs.a.o can be decommissioned and replaced with a static copy[1](to
avoid link rot).

[1] https://gitlab.archlinux.org/archlinux/bugs-archive/
[2] https://archlinux.org/news/bugtracker-migration-to-gitlab-completed/
[3] https://lists.archlinux.org/hyperkitty/list/arch-dev-public@lists.archlinux.org/thread/WYXDTJ3TR2DWRQCDZK44BQDH67IDVGTS/

Fix #550
Fix #551

Bugbuddy is the upcoming tool for assigning package bugs to the proper
folks. The bugbuddy role will be created at a later date when the tool
is ready.

Gluebuddy does not keep state so there is no reason for backing up the
server.

Fixes: d88c0b95 ("Initialize gluebuddy host")

Initial setup bootstrapped from arch-boxes repo [1], default user 'arch'
removed after.

https://gitlab.archlinux.org/archlinux/arch-boxes/-/jobs/157024/artifacts/browse/output



Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

GitLab is configured to use OpenSearch from its admin panel[2].

[1] https://docs.gitlab.com/ee/user/search/advanced_search.html
[2] https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html#enable-advanced-search

Fix #159

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Equinix's AMS1 DC is being shut down so we need to recreate this box.

For Geo variety, this one is created in Frankfurt instead of Amsterdam.

Ref #495

Equinix's AMS1 DC is being shut down so we need to recreate this box.

Ref #495

As announced on the mailing list[2] pacman has been migrated to gitlab
and there is no real use for patchwork left, so it can be
decommissioned. A static copy[1] is kept around for the time being to
avoid link rot.

[1] https://gitlab.archlinux.org/archlinux/patchwork-archive
[2] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/message/7B6R5HVEC67U7B2VQ3SKUVXU4RDCRRMM/

Fix #487

This has become outdated (missing new dedicated servers) and its usage
can be replicated by checking if ansible_virtualization_role == "host".

For Ansible ad hoc commands, '!hcloud' can be used to the same effect.

Repro machines are now placed in the rebuilderd_workers group.

This group is unused and contains fewer than half of our web servers.

With the final lists migrated to mailman3[1], the mailman2 server can
finally be killed.

When the mailman3 server was initially setup[2], it was done on a
separate server because the mailman and mailman3 packages conflicted,
and the traffic was routed over wireguard (HTTP, LMTP and SMTP).

Instead of installing mailman3 on the original lists.al.org server and
transferring the data, it was easier just to install the missing pieces
(basically Postfix and adjusting the Nginx configuration) on the ml3
server and move the IPs (to keep the IP mail reputation).

So basically the following was done:
- The IPs for the original lists.al.org was moved to the mailman3.al.org
  server
- The mailman2 datadir was transferred to mailman3.al.org server, so we
  can keep the pipermail links alive, and import missing mails if needed
- The original lists.al.org server was decommissioned
- The mailman3.al.org server was renamed to lists.al.org
- The missing pieces was added to the mailman3 role (basically Postfix +
  Nginx adjustments)
- The mailman role was deleted and the mailman3 role renamed to mailman

[1] 75ac7d09 ("mailman: Fourth and final batch of mailman3 migrated lists")
[2] 9294828f ("Setup mailman3 server")

Fix #59

Its disks were migrated to a new server (prompted by an unsolvable issue
with the previous box's network interface; might have been a mobo issue).

For some workloads running in a container is too restrictive, ex:
arch-boxes (loop device, filesystem mount, pacstrap) and archiso
(pacstrap). Currently they both run a TCG accelerated QEMU VM, which is
very slow and painful to work with. We should provide a better option to
our users!

This adds a hardware accelerated VM for this kinds of workloads, which
is way faster and you can do whatever you like (mostly)!

Fix #283

We took it out of Geo duties two months ago, but it's still offline and
it gets annoying having to exclude it from all Ansible executions we do.

Also tweak the documentation on rebuilderd workers and add runner1.

asia.mirror.pkgbuild.com has been offline for 12 days so far while we
wait for a NIC replacement. Should have taken it out of DNS NS duties
earlier but better late than never.

We want to migrate to mailman3 as mailman2 is basically unmaintained and
requires Python 2 which is EOL.

Because the mailman and mailman3 packages conflict and we don't want to
perform a big bang migration, mailman3 must be deployed on a separate
server. mailman-web (mailman3's web interface) hasn't been packaged yet,
so for now we are using my homebrewed PKGBUILD[1].

[1] https://gist.github.com/klausenbusk/5982063f95c503754a51ed2fefb8915e

Ref #59

We make almost no use of the dynamic properties of the hcloud inventory,
so we can simplify this by declaring all cloud servers in the main hosts
inventory.

The main benefit of this change is that temporary and experimental cloud
servers are not automatically included in the Ansible playbooks. In such
cases it is usually incorrect to deploy changes to these unknown servers.

A smaller side benefit is that Ansible will now use hostnames to connect
to cloud servers, whereas the dynamic inventory provided IPv4 addresses.
This results in more meaningful ~/.ssh/known_hosts entries.

All servers are part of these groups which makes them redundant.

Also alphabetically sort the servers in this group.

We had a GeoIP mirror in the past based on nginx and its GeoIP module,
but it didn't perform very well, due to the high latency (asking a
central server for the package and then redirected to the closest
mirror).

One of the reasons for offering this service, is so we can relieve
mirror.pkgbuild.com which is burning a ton of traffic (50TB/month),
likely due to it being the default mirror in our Docker image. Another
reason is so we can offer a link to our arch-boxes images in libosinfo
(used by gnome-boxes, virt-install and virt-manager), with good enough
performance for most users.

This time we take a different approach and use a DNS based solution,
which means the latency penalty is only paid once (the first DNS
request). The downside is that the mirrors must have a valid certificate
for the same domain name, which makes using third-party mirrors a
challenge. So for now, we are just using the sponsored mirorrs
controlled by the DevOps team.

Fix #101

With the PHP->Python port done[1][2], there isn't much need for aur-dev
anynmore. Most things can also be tested locally and aur-dev haven't got
any love since the port (ex: allowing the aurweb maintainers to deploy
without asking DevOps).

[1] https://lists.archlinux.org/pipermail/aur-general/2022-February/036786.html
[2] !525

Indirect way to get "configure_network: true".

These are managed services and Ansible doesn't run on them. It got
boring writing 'all,!rsync_net,!hetzner_storageboxes' in playbooks
and ad-hoc commands, so remove these borg hosts from our inventory.

Kevin Morris authored 3 years ago and Kristian Klausen committed 3 years ago

Signed-off-by: Kevin Morris <kevr@0cost.org>

Fixes: d88c0b95 ("Initialize gluebuddy host")

Fixes: d88c0b95 ("Initialize gluebuddy host")

Collects the smart data using smartctl and outputs them in the
textcollector dir. This expects smartd to be configured to regularly
self tests on a regular interval to detect if a disk is broken.

We sadly run all that stuff in docker now.