If the cost exceeds $0, it indicates that we have run out of credit
and/or are doing something wrong, in either case we want to be alerted.

With the support for network.wireguard.* credentials[1] in systemd
v256[2], we can now easily avoid storing the credentials centrally in
our ansible vault, which is preferable as it makes the private keys less
exposed. It may also make fine-grained access easier in the future[3] as
there is no longer a vault file for each server.

All the keys have been rotated and the new private keys are only stored
on the servers.

[1] https://github.com/systemd/systemd/pull/30826
[2] https://github.com/systemd/systemd/releases/tag/v256
[3] #64

With the last commit[1], we now lint the misc host_vars files and the
indentation is off for some of the files.

[1] b0f46412 ("Add missing .yml suffix to the misc host_vars files")

The naming of yaml files should be consistent.

As per my announcement to arch-devops[1] and staff, this adds a Mumble
server for Arch Linux.

The password for the special root user SuperAdmin is automatically
generated on first launch and printed to the logs. I went ahead and
added it to the vault. It should not usually be required to login as
SuperAdmin though as long as there are user admins around.

This uses certbot for local certificates.

[1] https://lists.archlinux.org/archives/list/arch-devops@lists.archlinux.org/thread/AHAOSTGFJTLQDSXLWFORDKGR6RDVHYEI/

It failed to reboot during the last upgrade procedure. Upon logging into
the Equinix Metal console, we discovered that we lack access to all 4 of
the servers sponsored by Equinix Metal. They are under the CNCF account,
and it's not possible to transfer them to our organization.

Equinix Metal is being sunset, and the remaining 3 servers will also go
away on June 30th 2026. We can keep them until then, or until they fail
to boot like seoul.mirror.pkgbuild.com.

They are our HTTP/3 guinea pigs for now. HTTP/3 has been enabled on
archlinux.org since 2024-07-22, so I do not expect any issues.

$http_host is changed to $host for aurweb, as HTTP/3 uses the
":authority" pseudo-header instead of the "Host" header[1][2].

[1] https://trac.nginx.org/nginx/ticket/2281
[2] https://mailman.nginx.org/pipermail/nginx-devel/2024-January/LCIUMLKCM2EBMEMTU3KXMW74AP2C4FYZ.html

Ref #606

The wiki has been hammered with requests from some stupid Chinese
bots/crawlers. Adding a simple challenge (requiring a cookie to be set),
seems to be enough to throw them off.

This was initially added for all pages, but as that could affect Chinese
search engines (concern raised on the forum[1]), it was changed to only
affect "action views", which search engines are not supposed to crawl.

[1] https://bbs.archlinux.org/viewtopic.php?pid=2185963#p2185963

New server; same CPU and RAM as previous one, hopefully more stable.

As discussed in #531


we want to split the repo and the archive server and as a first step of
that we're commissioning this AX41-NVME server from hetzner to serve as
a future repo host.

Signed-off-by: Christian Heusel <christian@heusel.eu>

As announced[2][3] the bugtracker has been migrated to gitlab, so
bugs.a.o can be decommissioned and replaced with a static copy[1](to
avoid link rot).

[1] https://gitlab.archlinux.org/archlinux/bugs-archive/
[2] https://archlinux.org/news/bugtracker-migration-to-gitlab-completed/
[3] https://lists.archlinux.org/hyperkitty/list/arch-dev-public@lists.archlinux.org/thread/WYXDTJ3TR2DWRQCDZK44BQDH67IDVGTS/

Fix #550
Fix #551

10.0.0.43 had already been allocated to london.mirror.pkgbuild.com
creating a conflict in Prometheus. Pick the next available address.

Extend the role (previously used for ACME DNS verifications only) to
support dynamic DNS functionality planned for sandbox.archlinux.page.

Bugbuddy is the upcoming tool for assigning package bugs to the proper
folks. The bugbuddy role will be created at a later date when the tool
is ready.

The same drop-in functionality is now provided by the openssh package
via /etc/ssh/sshd_config.d/.

Jelle van der Waa authored 1 year ago and Leonidas Spyropoulos committed 1 year ago

Apply the same rate limitting and fail2ban rules for aur.archlinux.org

Boxy seems to randomly restart after running out of memory. We do not
yet know the reason behind the increased memory usage, but zram might
help a bit.

Initial setup bootstrapped from arch-boxes repo [1], default user 'arch'
removed after.

https://gitlab.archlinux.org/archlinux/arch-boxes/-/jobs/157024/artifacts/browse/output



Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

GitLab is configured to use OpenSearch from its admin panel[2].

[1] https://docs.gitlab.com/ee/user/search/advanced_search.html
[2] https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html#enable-advanced-search

Fix #159

With the ongoing git migration[1] our GitLab will gain a lot more usage,
so GitLab should get the default ssh port and then DevOps can use a
non-standard port.

We will redirect the old port (222) to the new port for some time, so
fetching won't break for existing local repositories.

[1] https://archlinux.org/news/git-migration-announcement/

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Jelle van der Waa authored 2 years ago and Jelle van der Waa committed 2 years ago

Currently the tracker is hammered by a scraper which causes high load if
we don't actually ban the user.

Equinix's AMS1 DC is being shut down so we need to recreate this box.

For Geo variety, this one is created in Frankfurt instead of Amsterdam.

Ref #495

Equinix's AMS1 DC is being shut down so we need to recreate this box.

Ref #495

As announced on the mailing list[2] pacman has been migrated to gitlab
and there is no real use for patchwork left, so it can be
decommissioned. A static copy[1] is kept around for the time being to
avoid link rot.

[1] https://gitlab.archlinux.org/archlinux/patchwork-archive
[2] https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/message/7B6R5HVEC67U7B2VQ3SKUVXU4RDCRRMM/

Fix #487

This role will still handle setting up nginx and rsyncd, due to specific
configuration requirements these services have.

We're also effectively relieving build.archlinux.org of rsyncd duties as
it is not something it should be doing anyway.

Move the 'sshd_enable_includes' override to aur's host vars instead of
specifying it as part of playbooks/aur.archlinux.org. Otherwise, would
break the AUR's SSH auth if ssh.d/aurweb_config does not get included.

group_vars/all was enabling just the sshd jail so move this into the
fail2ban role defaults. patchwork, security and wiki were redefining
fail2ban_jails without deviating from the group_vars/all default and
can therefore be dropped.

The traffic hitting ping.archlinux.org has lately been exhausting its
default nf_conntrack_max limit of 64k. Bump it to 256k (which is also
the default limit found on systems with more than 4G of memory).

Suggested-by: Kristian Klausen <kristian@klausen.dk>

With the final lists migrated to mailman3[1], the mailman2 server can
finally be killed.

When the mailman3 server was initially setup[2], it was done on a
separate server because the mailman and mailman3 packages conflicted,
and the traffic was routed over wireguard (HTTP, LMTP and SMTP).

Instead of installing mailman3 on the original lists.al.org server and
transferring the data, it was easier just to install the missing pieces
(basically Postfix and adjusting the Nginx configuration) on the ml3
server and move the IPs (to keep the IP mail reputation).

So basically the following was done:
- The IPs for the original lists.al.org was moved to the mailman3.al.org
  server
- The mailman2 datadir was transferred to mailman3.al.org server, so we
  can keep the pipermail links alive, and import missing mails if needed
- The original lists.al.org server was decommissioned
- The mailman3.al.org server was renamed to lists.al.org
- The missing pieces was added to the mailman3 role (basically Postfix +
  Nginx adjustments)
- The mailman role was deleted and the mailman3 role renamed to mailman

[1] 75ac7d09 ("mailman: Fourth and final batch of mailman3 migrated lists")
[2] 9294828f ("Setup mailman3 server")

Fix #59

These roles are very similar and can be merged into a single new role.

Note: The archive mirror is changed from a 4-hour sync to minutely for
conformity with the other two mirrors. In practice this doesn't matter
as it was already taking over 4 hours to finish and was starting again
right after its previous run.

The default of 0.5 has proven insufficient on at least 3 boxes so far.

We moved away from raid6 a while back; update the host var to reflect
the current configuration.

This box is very sussy and really likes to fill up its zram swap:

  [root@reproducible ~]# zramctl
  NAME       ALGORITHM DISKSIZE  DATA  COMPR TOTAL STREAMS MOUNTPOINT
  /dev/zram0 lzo-rle       1.9G  1.5G 183.4M  196M       1 [SWAP]

  [root@reproducible ~]# free -m
                 total        used        free      shared  buff/cache   available
  Mem:            1928         529          73           5        1325        1236
  Swap:           1927        1543         384

Fixes: 4a5748ea ("Bump zram-fraction to 1.0 for reproducible.archlinux.org")

Its disks were migrated to a new server (prompted by an unsolvable issue
with the previous box's network interface; might have been a mobo issue).

The runner was accidentally made "specific", which can't be reverted[1].

[1] https://gitlab.com/gitlab-org/gitlab/-/issues/16167

For some workloads running in a container is too restrictive, ex:
arch-boxes (loop device, filesystem mount, pacstrap) and archiso
(pacstrap). Currently they both run a TCG accelerated QEMU VM, which is
very slow and painful to work with. We should provide a better option to
our users!

This adds a hardware accelerated VM for this kinds of workloads, which
is way faster and you can do whatever you like (mostly)!

Fix #283

This box somehow gets a compression ratio of over 12; bump its zram
fraction accordingly, to stop getting alerts about high swap usage.