tasks/upgrade-servers: upgrade Keycloak box first

Closes #496

See merge request !685

matrix.archlinux.org doesn't like it when accounts.archlinux.org is
unavailable at start-up. Try to work around this by upgrading
accounts.archlinux.org first and doing a health check before proceeding
to update the rest of the servers.

Fixes: #496

90 seconds isn't enough time for keycloak's ExecStartPre cmd to finish.

Enabling debug packages by default has resulted in more debug packages.

Nobody could have foreseen this.

flyspray: add removed packages script

See merge request !681

Jelle van der Waa authored 2 years ago and Jelle van der Waa committed 2 years ago

This script goes through all open bugs in the Arch Linux and Community
tracker and extracts the packagename from "[$pkgname]" and tries match
it to the list of packages in the repo. If there is no match the package
is assumed to be dropped from the repo and printed.

This script will give false positives, but not enough which requires
some extra filtering.

When debuginfod processes big debug packages it can run out of memory
and get killed. Then, it gets restarted by systemd only to OOM again.

Let's try doubling the memory to ~4G.

Onboard pitastrudl as support staff (mirror admin)

See merge request archlinux/infrastructure!679

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Liberally add "noqa no-changed-when" tags to the problematic tasks,
except for two "systemd-tmpfiles --create" calls. For these we can
simply include the creates= parameter in the command module's call.

Leonidas Spyropoulos authored 2 years ago and Evangelos Foutras committed 2 years ago

Convert the permissions to strings to avoid octal interpretation.

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Deleting events causes database corruption, see e.g.
https://github.com/matrix-org/synapse/issues/11738 and
https://github.com/matrix-org/synapse/issues/13476.

Broken servers are common making these loggers very spammy.

gitlab: rate-limit gitlab-pages

See merge request !678

Implement ip based rate limiting on gitlab-pages. This ensures we avoid
slowing down the gitlab server when extensive requests are made against
our gitlab-pages like WKD sync from Archlinux keyring service

https://docs.gitlab.com/ee/administration/pages/#rate-limits



Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Fix some ansible-lint warnings

See merge request archlinux/infrastructure!677

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

The daemon_reload is specific to systemd and produce a warning in
ansible-lint

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Decommission Equinix Metal servers at legacy AMS1 DC

Closes #495

See merge request archlinux/infrastructure!676

Added hosts:

- repro3.pkgbuild.com
- runner3.archlinux.org
- seoul.mirror.pkgbuild.com
- sydney.mirror.pkgbuild.com

Removed hosts:

- repro1.pkgbuild.com
- runner2.archlinux.org

Equinix's AMS1 DC is being shut down so we need to recreate this box.

For Geo variety, this one is created in Frankfurt instead of Amsterdam.

Ref #495

Equinix's AMS1 DC is being shut down so we need to recreate this box.

Ref #495

Pass --target=i386-pc to grub-install to stick with legacy BIOS boot.

Our partitioning scheme is not suitable for EFI boot.

/etc/motd now contains "OS Rescue Environment" when in rescue mode.

This directory isn't part of the docker package so we need to create it.

aurweb: set receive.fsckobjects to true

See merge request archlinux/infrastructure!675

https://git-scm.com/docs/git-config#Documentation/git-config.txt-receivefsckObjects
Related: aurweb#426



Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

mailman: Patch out CC modification breaking DKIM/DMARC

See merge request !673

The bug[1] would explain all the bounces and unsubscriptions, so patch
it manually until upstream gets it sorted.

[1] https://gitlab.com/mailman/mailman/-/issues/636

Follow-up to merge request archlinux/infrastructure!671.

Give Leonidas full DevOps access

See merge request archlinux/infrastructure!671

We've been seeing CPU spikes that a single vCPU can't handle, resulting
in alerts about high CPU usage. Let's try two cores to see how they do.