.ansible-lint

@@ -4,15 +4,19 @@ exclude_paths:
   - playbooks/tasks
   - roles/prometheus/files/node.rules.yml
 skip_list:
-  # line too long (x > 80 characters) (line-length)
-  - 'line-length'
-  # yaml: too many spaces inside braces (braces)
-  - 'braces'
+  # yaml: line too long (x > 160 characters) (yaml[line-length])
+  - yaml[line-length]
+  # yaml: too many spaces inside braces (yaml[braces])
+  - yaml[braces]
   # Do not recommend running tasks as handlers
-  - 'no-handler'
+  - no-handler
   # Do not force galaxy info in meta/main.yml
-  - 'meta-no-info'
+  - meta-no-info
   # Allow package versions to be specified as 'latest'
-  - 'package-latest'
-  # Don't require FQCN for builtin actions
-  - 'fqcn-builtins'
+  - package-latest
+  # Don't require fully-qualified collection names
+  - fqcn
+  # Allow free-form module calling syntax
+  - no-free-form
+  # Allow role includes with unprefixed role variables
+  - var-naming[no-role-prefix]

.gitlab-ci.yml

@@ -2,14 +2,14 @@ image: "archlinux:latest"
 
 ansible-lint:
   before_script:
-    - pacman -Syu --needed --noconfirm ansible-lint ansible
+    - pacman -Syu --needed --noconfirm ansible-lint ansible python-jmespath
   script:
     # Fix weird ansible bug: https://github.com/trailofbits/algo/issues/1637
     # This probably happens due to gitlab-runner mounting the git repo into the container
     - chmod o-w .
     # Fix syntax-check rule (https://github.com/ansible-community/ansible-lint/issues/1350#issuecomment-778764110)
-    - sed "s/,hcloud_inventory.py//" -i ansible.cfg
-    - sed "/^vault_password_file/d" -i ansible.cfg
+    - sed -i "/^vault_identity_list/d" ansible.cfg
+    - sed -i -e "/vars_files:/d" -e "/misc\/vaults\/vault_/d" playbooks/*.yml
     # Fix load-failure: Failed to load or parse file
     - ansible-lint $(printf -- "--exclude %s " */*/vault_*)
 

.gitlab/issue_templates/New Official Project.md

@@ -28,19 +28,19 @@ If you want to add a new official project, here are some guidelines to follow:
        to a protected branch of the project.
 1. [ ] If a secure runner is used, create an MR to make sure the project's `.gitlab-ci.yml` specifies
        `tags: secure`.
-1. [ ] Make sure that the *Push Rules* in https://gitlab.archlinux.org/archlinux/arch-boxes/-/settings/repository
+1. [ ] Make sure that the *Push Rules* in https://gitlab.archlinux.org/archlinux/my-example/-/settings/repository
        reflect these values:
-   - `Committer restriction`: `on`
+   - `Reject unverified users`: `on`
    - `Reject unsigned commits`: `on`
    - `Do not allow users to remove tags with git push`: `on`
-   - `Check whether author is a gitlab user`: `on`
-   - `Prevent committing secrets to git`: `on`
+   - `Check whether author is a GitLab user`: `on`
+   - `Prevent pushing secret files`: `on`
    - All of these should be activated by default as per group rules but it's good to check.
 1. [ ] The *Protected Branches* in https://gitlab.archlinux.org/archlinux/my-example/-/settings/repository should specify
        `Allowed to merge` and `Allowed to push` as `Developers + Maintainers.`
-1. [ ] Disable unneeded project features under *Visibility, project features, permissions* (https://gitlab.archlinux.org/archlinux/my-example/edit)  
+1. [ ] Disable unneeded project features under *Visibility, project features, permissions* (https://gitlab.archlinux.org/archlinux/my-example/edit)
    Always:
-   - `Users can request access`: `off`  
+   - `Users can request access`: `off`
    Often, but not always:
    - Repository -> Container registry
    - Repository -> Git Large File Storage (LFS)
@@ -86,7 +86,7 @@ If you want to add a new official project, here are some guidelines to follow:
    - `Issues`
    - `Projects`
 1. [ ] Go to https://github.com/archlinux/my-example/settings/hooks and add a new webhook
-   - `Payload URL`: `$(misc/get_key.py misc/vault_github.yml github_pull_closer_webhook_url)`
+   - `Payload URL`: `$(misc/get_key.py misc/vaults/vault_github.yml github_pull_closer_webhook_url)`
    - `Content type`: `application/json`
    - `Which events would you like to trigger this webhook?`
      - `Let me select individual events.`: `Pull requests`

.gitlab/issue_templates/Offboarding.md

@@ -7,43 +7,51 @@ This template should be used for offboarding Arch Linux team members.
 ## Details
 
 - **Team member username**:
-- **Currently held roles**: <!-- Add known roles here like TU, DevOps, etc -->
+- **Currently held roles**: <!-- Add known roles here like Package Maintainer, DevOps, etc -->
 - **Removal request**: <!-- Add link to relevant mailing list mail -->
   - **Voting result**: <!-- Add link to relevant mailing list mail -->
 
 ## All roles checklist
 
-- [ ] Remove user email by reverting instructions from `docs/email.md`.
-  - [ ] Setup forwarding if desired (please add the current date as a comment above the mail address in Postfix's `users` file).
-  - [ ] Inform the user of the conditions for forwarding.
-    - In most cases we only offer forwarding for 6 months.
-    - We will inform the user prior to disabling the forwarding.
-    - The forwarding can be extended if there are good reasons for doing so.
+- [ ] Remove user email by reverting instructions from [`docs/email.md`](docs/email.md).
+  - [ ] Remove entry in [`group_vars/all/archusers.yml`](group_vars/all/archusers.yml).
+  - [ ] Remove SSH pubkey from `pubkeys/<username>.pub`.
+  - [ ] Run `ansible-playbook -t archusers  $(git grep -l archusers playbooks/ | grep -v phrik)`.
+  - [ ] Setup forwarding if requested (please add the current date as a comment above the mail address in Postfix's `users` file).
+    - [ ] Inform the user of the conditions for forwarding.
+      - In most cases we only offer forwarding for 6 months.
+      - We will inform the user prior to disabling the forwarding.
+      - The forwarding can be extended if there are good reasons for doing so.
 - [ ] Set user to inactive in archweb: https://www.archlinux.org/admin/auth/user/
-- [ ] Remove member from [staff mailing list](https://lists.archlinux.org/admin/staff/members)
-- [ ] Ask the user to leave `#archlinux-staff` on Libera Chat and forget the password
-- [ ] Remove staff cloak on Libera Chat ([Group contacts](https://wiki.archlinux.org/title/Arch_IRC_channels#Libera_Chat_group_contacts))
+- [ ] Remove member from [staff mailing list](https://lists.archlinux.org/mailman3/lists/staff.lists.archlinux.org/members/member/).
+- [ ] Moderate email address on [arch-dev-public](https://lists.archlinux.org/mailman3/lists/arch-dev-public.lists.archlinux.org/members/member/) (find member and moderate).
+- [ ] Ask the user to leave `#archlinux-staff` on Libera Chat and forget the password.
+- [ ] Remove staff cloak on Libera Chat ([Group contacts](https://wiki.archlinux.org/title/Arch_IRC_channels#Libera_Chat_group_contacts)). cc @archlinux/teams/irc/group-contacts
+- [ ] Remove the user from relevant staff groups on Keycloak.
+- [ ] Move the user from the public list of their usergroup on archweb ([support staff](https://archlinux.org/people/support-staff/) / [TUs](https://archlinux.org/people/trusted-users/) / [devs](https://archlinux.org/people/developers/)) to the respective fellow site ([fellow support staff](https://archlinux.org/people/support-staff-fellows/) / [fellow TUs](https://archlinux.org/people/trusted-user-fellows/) / [fellow devs](https://archlinux.org/people/developer-fellows/))
+- [ ] Remove the user from the Arch Linux GitHub organisation
 
-## TU/Developer offboarding checklist
+## Main key offboarding checklist
 
-- [ ] Remove entry in `group_vars/all/archusers.yml`.
-- [ ] Remove SSH pubkey from `pubkeys/<username>.pub`.
-- [ ] Run `ansible-playbook -t archusers  $(git grep -l archusers playbooks/ | grep -v phrik)`.
-- [ ] Remove the user from the `Trusted Users`/`Developers` groups on Keycloak.
-- [ ] Moderate email address on [arch-dev-public](https://lists.archlinux.org/admin/arch-dev-public/members) (find member and moderate)
-- [ ] Remove member from [arch-tu](https://lists.archlinux.org/admin/arch-tu/members) and/or [arch-dev](https://lists.archlinux.org/admin/arch-dev/members) mailing lists
-- [ ] Create [issue in archlinux-keyring](https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/new) (choose *"Remove Packager Key"* and/or *"Remove Main Key"* template)
+- [ ] Remove user email for the `master-key.archlinux.org` subdomain by reverting instructions from [`docs/email.md`](docs/email.md).
+- [ ] Create an issue in [archlinux-keyring](https://gitlab.archlinux.org/archlinux/archlinux-keyring) using the [*"Remove Main Key"*](https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/new?issuable_template=Remove%20Main%20Key) template.
+
+## Package Maintainer/Developer offboarding checklist
+
+- [ ] Remove member from [arch-tu](https://lists.archlinux.org/mailman3/lists/arch-tu.lists.archlinux.org/members/member/) and/or [arch-dev](https://lists.archlinux.org/mailman3/lists/arch-dev.lists.archlinux.org/members/member/) mailing lists.
+- [ ] Ask the user to leave `#archlinux-tu` and/or `#archlinux-dev` aswell as `#archlinux-packaging` on Libera Chat and forget the password(s).
+- [ ] Create an issue in [archlinux-keyring](https://gitlab.archlinux.org/archlinux/archlinux-keyring) using the [*"Remove Packager Key"*](https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/new?issuable_template=Remove%20Packager%20Key) template.
+- [ ] Remove [stale package relations](https://archlinux.org/packages/stale_relations/) for the now inactive user.
+- [ ] Remove their extended permissions on AURweb
 
 ## DevOps offboarding checklist
 
-- [ ] Remove entries in `group_vars/all/root_access.yml`.
-- [ ] Run `ansible-playbook -t root_ssh playbooks/*.yml`.
+- [ ] Remove entries in [`group_vars/all/root_access.yml`](group_vars/all/root_access.yml).
+- [ ] Run `ansible-playbook -t root_ssh playbooks/all-hosts-basic.yml`.
 - [ ] Run `ansible-playbook playbooks/hetzner_storagebox.yml playbooks/rsync.net.yml`.
-- [ ] Remove the user from the `DevOps` group on Keycloak.
-- [ ] Remove member from [arch-devops-private mailing lists](https://lists.archlinux.org/admin/arch-devops-private/members)
-- [ ] Remove pubkey from [Hetzner's key management](https://robot.your-server.de/key/index)
+- [ ] Remove member from [arch-devops-private mailing lists](https://lists.archlinux.org/mailman3/lists/arch-devops-private.lists.archlinux.org/members/member/).
+- [ ] Remove pubkey from [Hetzner's key management](https://robot.your-server.de/key/index).
 
 ## Wiki Administrator checklist
 
-- [ ] Remove the user from the `Wiki Admins` group on Keycloak.
-- [ ] Remove member from [arch-wiki-admins mailing list](https://lists.archlinux.org/admin/arch-wiki-admins/members).
+- [ ] Remove member from [arch-wiki-admins mailing list](https://lists.archlinux.org/mailman3/lists/arch-wiki-admins.lists.archlinux.org/members/member/).

.gitlab/issue_templates/Onboarding.md

@@ -30,62 +30,52 @@ https://www.gnupg.org/gph/en/manual/x135.html
 -->
 
 ## All roles checklist
-The mailing list password can be found in [`misc/additional-credentials.vault`](misc/additional-credentials.vault).
 
-- [ ] Add new user email as per [`docs/email.md`](docs/email.md).
+- [ ] Add user mail if TU or developer, or support staff and **communication e-mail address** is arch.
+  - [ ] Add new user email as per [`docs/email.md`](docs/email.md).
+  - [ ] Add entry in [`group_vars/all/archusers.yml`](group_vars/all/archusers.yml).
+    - If support staff `hosts` should be set to `mail.archlinux.org`.
+    - `homedir.archlinux.org` is also allowed for support staff, but it is opt-in.
+  - [ ] Add SSH pubkey to `pubkeys/<username>.pub`.
+  - [ ] Run `ansible-playbook -t archusers $(git grep -l archusers playbooks/ | grep -v phrik)`.
 - [ ] Create a new user in [archweb](https://www.archlinux.org/devel/newuser/). Select the appropriate group membership and allowed repos (if applicable).
-- [ ] Subscribe **communication e-mail address** to internal [staff mailing list](https://lists.archlinux.org/admin/staff/members/add).
+- [ ] Subscribe **communication e-mail address** to internal [staff mailing list](https://lists.archlinux.org/mailman3/lists/staff.lists.archlinux.org/mass_subscribe/).
+- [ ] Allow sending from **communication e-mail address** on [arch-dev-public](https://lists.archlinux.org/mailman3/lists/arch-dev-public.lists.archlinux.org/members/member/) (subscribe and/or find address and remove moderation).
 - [ ] Give the user access to `#archlinux-staff` on Libera Chat.
 - [ ] Give the user a link to our [staff services page](https://wiki.archlinux.org/title/DeveloperWiki:Staff_Services).
 - [ ] Replace the **Team member username** with the @-prefixed username on Gitlab.
 - [ ] Remove personal information (such as **Full Name** and **Personal e-mail
   address**, as well as the clearsigned representation of this data), remove
   the description history and make the issue non-confidential.
-- [ ] Request staff cloak on Libera Chat ([Group contacts](https://wiki.archlinux.org/title/Arch_IRC_channels#Libera_Chat_group_contacts))
-- [ ] Go to [Arch Linux group](https://gitlab.archlinux.org/groups/archlinux/-/group_members) -> Enter Admin mode -> go to members -> add username as "minimal access"
-- [ ] Go to [Arch Staff group](https://gitlab.archlinux.org/groups/archlinux/teams/staff/-/group_members) -> Enter Admin mode -> go to members -> add username as "reporter"
-
-## Packager onboarding checklist
-
-<!-- The ticket should be created by a sponsor of the new packager -->
-- [ ] Create [issue in archlinux-keyring using the *"New Packager Key"* template](https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/new?issuable_template=New%20Packager%20Key).
+- [ ] Request staff cloak on Libera Chat ([Group contacts](https://wiki.archlinux.org/title/Arch_IRC_channels#Libera_Chat_group_contacts)) cc @archlinux/teams/irc/group-contacts
 
 ## Main key onboarding checklist
 
 - [ ] Add new user email for the `master-key.archlinux.org` subdomain as per [`docs/email.md`](docs/email.md).
-<!-- The ticket should be created by the developer becoming a new main key holder -->
-- [ ] Create [issue in archlinux-keyring using the *"New Main Key"* template](https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/new?issuable_template=New%20Main%20Key).
+  <!-- The ticket should be created by the developer becoming a new main key holder -->
+- [ ] Create an issue in [archlinux-keyring](https://gitlab.archlinux.org/archlinux/archlinux-keyring) using the [*"New Main Key"*](https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/new?issuable_template=New%20Main%20Key) template.
 
-## Developer onboarding checklist
+## Package Maintainer/Developer onboarding checklist
 
-- [ ] Add entry in [`group_vars/all/archusers.yml`](group_vars/all/archusers.yml).
-- [ ] Add SSH pubkey to `pubkeys/<username>.pub`.
-- [ ] Run `ansible-playbook -t archusers  $(git grep -l archusers playbooks/ | grep -v phrik)`.
-- [ ] Assign the user to the `Developers` groups on Keycloak.
-- [ ] Assign the user to the `Developers` group on [archlinux.org](https://archlinux.org/admin/auth/user/).
-- [ ] Subscribe **communication e-mail address** to internal [arch-dev](https://lists.archlinux.org/admin/arch-dev/members/add) mailing list.
-- [ ] Allow sending from **communication e-mail address** on [arch-dev-public](https://lists.archlinux.org/admin/arch-dev-public/members) (subscribe and/or find address and remove moderation).
+<!-- The ticket should be created by a sponsor of the new packager -->
+- [ ] Create an issue in [archlinux-keyring](https://gitlab.archlinux.org/archlinux/archlinux-keyring) using the [*"New Packager Key"*](https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/new?issuable_template=New%20Packager%20Key) template.
+- [ ] Assign the user to the correct group in the `Arch Linux Staff/Package Maintainer Team/` group on Keycloak.
+- [ ] Assign the user to the `Package Maintainers` or `Developers` group on [archlinux.org](https://archlinux.org/admin/auth/user/).
+- [ ] Subscribe **communication e-mail address** to internal [arch-tu](https://lists.archlinux.org/mailman3/lists/arch-tu.lists.archlinux.org/mass_subscribe/) or [arch-dev](https://lists.archlinux.org/mailman3/lists/arch-dev.lists.archlinux.org/mass_subscribe/) mailing list.
+- [ ] Give the user access to `#archlinux-tu` or `#archlinux-dev` aswell as `#archlinux-packaging` on Libera Chat.
 
-## TU onboarding checklist
+## Support staff checklist
 
-- [ ] Add entry in [`group_vars/all/archusers.yml`](group_vars/all/archusers.yml).
-- [ ] Add SSH pubkey to `pubkeys/<username>.pub`.
-- [ ] Run `ansible-playbook -t archusers  $(git grep -l archusers playbooks/ | grep -v phrik)`.
-- [ ] Assign the user to the `Trusted Users` groups on Keycloak.
-- [ ] Assign the user to the `Trusted Users` group on [archlinux.org](https://archlinux.org/admin/auth/user/).
-- [ ] Subscribe **communication e-mail address** to internal [arch-tu](https://lists.archlinux.org/admin/arch-tu/members/add) mailing list.
-- [ ] Allow sending from **communication e-mail address** on [arch-dev-public](https://lists.archlinux.org/admin/arch-dev-public/members) (subscribe and/or find address and remove moderation).
+- [ ] Assign the user to the proper support staff group on Keycloak.
 
 ## DevOps onboarding checklist
 
 - [ ] Add entries in [`group_vars/all/root_access.yml`](group_vars/all/root_access.yml).
 - [ ] Run `ansible-playbook -t root_ssh playbooks/all-hosts-basic.yml`.
 - [ ] Run `ansible-playbook playbooks/hetzner_storagebox.yml playbooks/rsync.net.yml`.
-- [ ] Assign the user to the `DevOps` group on Keycloak.
-- [ ] Subscribe **communication e-mail address** to internal [arch-devops-private](https://lists.archlinux.org/admin/arch-devops-private/members/add) mailing list.
+- [ ] Subscribe **communication e-mail address** to internal [arch-devops-private](https://lists.archlinux.org/mailman3/lists/arch-devops-private.lists.archlinux.org/mass_subscribe/) mailing list.
 - [ ] Add pubkey to [Hetzner's key management](https://robot.your-server.de/key/index) for Dedicated server rescue system.
 
 ## Wiki Administrator checklist
 
-- [ ] Assign the user to the `Wiki Admins` group on Keycloak.
-- [ ] Subscribe **communication e-mail address** to the [arch-wiki-admins](https://lists.archlinux.org/admin/arch-wiki-admins/members/add) mailing list.
+- [ ] Subscribe **communication e-mail address** to the [arch-wiki-admins](https://lists.archlinux.org/mailman3/lists/arch-wiki-admins.lists.archlinux.org/mass_subscribe/) mailing list.

README.md

@@ -16,11 +16,11 @@ Install these packages:
 ### Instructions
 
 All systems are set up the same way. For the first time setup in the Hetzner rescue system,
-run the provisioning script: `ansible-playbook playbooks/tasks/install-arch.yml -l $host`.
+run the provisioning script: `ansible-playbook playbooks/tasks/install_arch.yml -l $host`.
 The provisioning script configures a sane basic systemd with sshd. By design, it is NOT idempotent.
 After the provisioning script has run, it is safe to reboot.
 
-Once in the new system, run the regular playbook: `HCLOUD_TOKEN=$(misc/get_key.py misc/vault_hetzner.yml hetzner_cloud_api_key) ansible-playbook playbooks/$hostname.yml`.
+Once in the new system, run the regular playbook: `HCLOUD_TOKEN=$(misc/get_key.py misc/vaults/vault_hetzner.yml hetzner_cloud_api_key) ansible-playbook playbooks/$hostname.yml`.
 This playbook is the one regularity used for administrating the server and is entirely idempotent.
 
 When adding a new machine you should also deploy our SSH known_hosts file and update the SSH hostkeys file in this git repo.
@@ -29,26 +29,30 @@ It will also deploy any new SSH host keys to all our machines.
 
 #### Note about GPG keys
 
-The `root_access.yml` file contains the `root_gpgkeys` variable that determine the users that have access to the vault, as well as the borg backup keys.
-All the keys should be on the local user gpg keyring and at **minimum** be locally signed with `--lsign-key`. This is necessary for running either the reencrypt-vault-key
-or the fetch-borg-keys tasks.
+The `root_access.yml` file contains the `vault_default_pgpkeys` variable which
+determines the users that have access to the `default` vault, as well as the
+borg backup keys. A separate `super` vault exists for storing highly sensitive
+secrets like Hetzner credentials; access to the `super` vault is controlled by
+the `vault_super_pgpkeys` variable.
 
-#### Note about Ansible dynamic inventories
-
-We use a dynamic inventory script in order to automatically get information for
-all servers directly from hcloud. You don't really have to do anything to make
-this work but you should keep in mind to NOT add hcloud servers to `hosts`!
-They'll be available automatically.
+All the keys should be on the local user gpg keyring and at **minimum** be
+locally signed with `--lsign-key` (or if you use TOFU, have `--tofu-policy
+good`). This is necessary for running any of the `reencrypt-vault-default-key`,
+`reencrypt-vault-super-key `or `fetch-borg-keys` tasks.
 
 #### Note about packer
 
 We use packer to build snapshots on hcloud to use as server base images.
 In order to use this, you need to install packer and then run
 
-    packer build -var $(misc/get_key.py misc/vault_hetzner.yml hetzner_cloud_api_key --format env) packer/archlinux.json
+    packer build -var $(misc/get_key.py misc/vaults/vault_hetzner.yml hetzner_cloud_api_key --format env) packer/archlinux.pkr.hcl
 
 This will take some time after which a new snapshot will have been created on the primary hcloud archlinux project.
 
+For the sandbox project please run
+
+    packer build -var $(misc/get_key.py misc/vaults/vault_hetzner.yml hetzner_cloud_sandbox_infrastructure_api_key --format env | sed 's/_sandbox_infrastructure//') -var install_ec2_public_keys_service=true packer/archlinux.pkr.hcl
+
 #### Note about terraform
 
 We use terraform in two ways:
@@ -65,7 +69,7 @@ but for the time being, this is what we're stuck with.
 The very first time you run terraform on your system, you'll have to init it:
 
     cd tf-stage1  # and also tf-stage2
-    terraform init -backend-config="conn_str=postgres://terraform:$(../misc/get_key.py ../group_vars/all/vault_terraform.yml vault_terraform_db_password)@state.archlinux.org"
+    terraform init -backend-config="conn_str=postgres://terraform:$(../misc/get_key.py ../group_vars/all/vault_terraform.yml vault_terraform_db_password)@state.archlinux.org?sslmode=verify-full"
 
 After making changes to the infrastructure in `tf-stage1/archlinux.tf`, run
 
@@ -119,19 +123,10 @@ Arch-audit can be used to find servers in need of updates for security issues.
 
     ansible all -a "arch-audit -u"
 
-#### Updating servers
-
-The following steps should be used to update our managed servers:
+### Semi-automated server upgrades
 
-  * pacman -Syu
-  * sync
-  * checkservices
-  * reboot
-
-##### Semi-automated server updates (experimental)
-
-For updating a lot of servers in a more unattended manner, the following
-playbook can be used:
+For updating all servers in a mostly unattented manner, the following playbook
+can be used:
 
     ansible-playbook playbooks/tasks/upgrade-servers.yml [-l SUBSET]
 
@@ -151,26 +146,20 @@ This section has been moved to [docs/servers.md](docs/servers.md).
 
 ## Ansible repo workflows
 
-### Replace vault password and change vaulted passwords
-
-  - Generate a new key and save it as ./new-vault-pw: `pwgen -s 64 1 > new-vault-pw`
-  - `for i in $(ag ANSIBLE_VAULT -l); do ansible-vault rekey --new-vault-password-file new-vault-pw $i; done`
-  - Change the key in misc/vault-password.gpg
-  - `rm new-vault-pw`
-
-### Re-encrypting the vault after adding or removing a new GPG key
-
-  - Make sure you have all the GPG keys **at least** locally signed
-  - Run the `playbooks/tasks/reencrypt-vault-key.yml` playbook and make sure it does not have **any** failed task
-  - Test that the vault is working by running ansible-vault view on any encrypted vault file
-  - Commit and push your changes
-
 ### Fetching the borg keys for local storage
 
   - Make sure you have all the GPG keys **at least** locally signed
   - Run the `playbooks/tasks/fetch-borg-keys.yml` playbook
   - Make sure the playbook runs successfully and check the keys under the borg-keys directory
 
+### Re-encrypting the vaults after adding a new PGP key
+
+Follow the instructions in [group_vars/all/root_access.yml](group_vars/all/root_access.yml).
+
+### Changing the vault password on encrypted files
+
+See [docs/vault-rekeying.md](docs/vault-rekeying.md).
+
 ## Backup documentation
 
 We use BorgBackup for all of our backup needs. We have a primary backup storage as well as an

ansible.cfg

 [defaults]
-inventory = hosts,hcloud_inventory.py
+inventory = hosts
 library = library
 remote_tmp = $HOME/.ansible/tmp
 remote_user = root
 nocows = 1
 roles_path = roles
-vault_password_file = misc/get-vault-pass.sh
+vault_id_match = True
+vault_identity_list = default@misc/vault-keyring-client.sh,super@misc/vault-keyring-client.sh
 retry_files_enabled = False
 callback_plugins = plugins/callback
 callbacks_enabled = profile_tasks
 max_diff_size = 250000
+stdout_callback = debug
+interpreter_python = /usr/bin/python
 
 [ssh_connection]
 pipelining = True

docs/backups.md

@@ -76,12 +76,13 @@ or just a sub-directory:
 
 ### Mariadb
 
-For Mariadb backups are made using mariabackup to `mysql_backup_dir`.Backups can are made and
-restored using the `mariabackup` tool. See also [official MariaDB docs](https://mariadb.com/kb/en/full-backup-and-restore-with-mariabackup/).
+For Mariadb backups are made using mariabackup to `backup_mysql_dir`. Backups
+are made and can be restored using the `mariadb-backup` tool. See also
+[official MariaDB docs](https://mariadb.com/kb/en/full-backup-and-restore-with-mariabackup/).
 
 ### PostgreSQL
 
-For PostgreSQL backups are made using pg_dump to `postgres_backup_dir`.
+For PostgreSQL backups are made using pg_dump to `backup_postgres_dir`.
 
 Restoring backups can be done with `pg_restore`. See also [official PostgreSQL docs](https://www.postgresql.org/docs/current/app-pgrestore.html).
 

docs/banning.md

@@ -33,3 +33,10 @@ To remove a banned IP Address:
 ```
 firewall-cmd --remove-rich-rule='rule family="ipv6" source address="1:2:3:4:6::" reject' --zone=public
 ```
+
+## Applying your changes
+
+After adding new rules you need to reload the firewall:
+```
+firewall-cmd --reload
+```

docs/becoming-devops.md

@@ -6,17 +6,21 @@ members.
 
 ## Junior DevOps program
 
-In order be able to onboard lesser-known members of the community who still want to help out with
-DevOps topics, we started the Junior DevOps program. This program requires applicants to
+In order to become a full DevOps, the applicant must first join the Junior DevOps program. This
+program requires applicants to
 
 0) have contributed to Arch multiple times in some meaningful ways,
 1) find two sponsors, and
 2) write an application to the arch-devops mailing list.
 
 The idea of Junior DevOps is that they don't get full access to all secrets and machines as opposed
-to full DevOps and have to make operational changes in pairing session with a full DevOps.
+to full DevOps but access within the limited scope on which they have been assigned rights to work
+on. As trust grows the scope on which the Junior DevOps operates may be extended, while their
+sponsors are expected to help them learn and should feel responsible to review any meaningful
+changes.
 
 However, Junior DevOps can already help with many tasks and are expected to take charge of a given
 topic.
 
-After a lot of trust is built up, Junior DevOps may graduate to become full DevOps.
+After a lot of trust is built up, Junior DevOps may graduate to become full DevOps. This usually
+takes 3-9 months.

docs/email.md

@@ -62,16 +62,16 @@ rspamadm dkim_keygen -s dkim-rsa -b 4096 -d archlinux.org -t rsa -k archlinux.or
 the ouput gives you the DNS entries to add to the terraform files.
 The keys generated need to go to the vault:
 ```
-roles/rspamd/files/archlinux.org.dkim-rsa.key
-roles/rspamd/files/archlinux.org.dkim-ed25519.key
+roles/rspamd/files/archlinux.org.dkim-rsa.key.vault
+roles/rspamd/files/archlinux.org.dkim-ed25519.key.vault
 ```
 
-# Gitlab servicedesk
+# GitLab Service Desk
 
-Gitlab has a [servicedesk
+GitLab has a [Service Desk
 feature](https://docs.gitlab.com/ee/user/project/service_desk.html) which
-creates issues for incomding emails and allows multiple people to reply via
-Gitlab on those issues and assign issues. Gitlab generates a default email
+creates issues for incoming emails and allows multiple people to reply via
+GitLab on those issues and assign issues. GitLab generates a default email
 address with the following logic:
 
 ```
@@ -80,7 +80,7 @@ gitlab+<group>-<project>-<project-id>-issue-@archlinux.org
 
 As we prefer to use user friendly addresses such as `privacy@archlinux.org` for communication a postfix alias is configured in `/etc/postix/aliases`.
 
-For a new Gitlab service desk project, add a new alias to `/etc/postfix/aliases` as:
+For a new GitLab Service Desk project, add a new alias to `/etc/postfix/aliases` as:
 
 ```
 foobar: gitlab+<group>-<project>-<project-id>-issue-@archlinux.org

docs/geomirrors.md

+# Geo mirrors
+
+DevOps team maintain a geo mirror across the world. The Geo mirror is public facing on geo.mirror.pkgbuild.com domain and it will resolve the closest to the location of the requester mirror.
+
+## Locations
+
+| Mirror                                    | Location                    |
+| ----------------------------------------- | --------------------------- |
+| https://america.mirror.pkgbuild.com/      | Miami (United States)       |
+| https://asia.mirror.pkgbuild.com/         | Hong Kong                   |
+| https://berlin.mirror.pkgbuild.com/       | Berlin (Germany)            |
+| https://europe.mirror.pkgbuild.com/       | Prague (Czechia)            |
+| https://johannesburg.mirror.pkgbuild.com/ | Johannesburg (South Africa) |
+| https://london.mirror.pkgbuild.com/       | London (United Kingdom)     |
+| https://losangeles.mirror.pkgbuild.com/   | Los Angeles (United States) |
+| https://singapore.mirror.pkgbuild.com/    | Singapore                   |
+| https://sydney.mirror.pkgbuild.com/       | Sydney (Australia)          |
+| https://taipei.mirror.pkgbuild.com/       | Taipei (Taiwan)             |
+
+### Logical split
+The continent mirrors america, asia and europe contain the archive mirrors as well as repository mirrors. The city mirrors have just the repositories hosted.
+
+## Requirements
+- Host with Arch Linux installed
+- root access provided
+- Enough storage to host repos / debugrepos (at least)
+- Bandwidth (depends on location)
+
+## Adding a new mirror box
+- Add new entries in `hosts` file under `mirrors` and `geo_mirrors` sections
+- Adjust terraform `tf-stage1/archlinux.tf` to include the IPv4 and IPv6 entries of the new server
+- Adjust terraform `tf-stage1/templates.tf` to include the IPv4 and IPv6 entries of the new server as a `NS` record for `geo.mirror.pkgbuild.com`
+- Add a new files in `host_vars`
+    - `host_vars/<fqdn>/misc`
+        Containing all the information for the mirror itself
+    - `host_vars/<fqdn>/vault_wireguard.yml`
+        Containing the wireguard private key in encrypted vault
+
+## Ansible Playbooks execution
+
+| Playbook | Roles | Reason | Hosts (limits) |Comments |
+| ----------- | ----------- | ----------- | ----------- |  ----------- |
+| install_arch | All | Install Arch | | Optional if you can |
+| mirrors.yml | All | Setup mirror | `<fqdn>` | |
+| redirect.archlinux.org.yml | dyn_dns | Make TXT records | | |
+| repos.archlinux.org.yml | dbscripts | Allow debug repo syncing | | |
+| mirrors.yml | geo_dns | Add new domain to DNS | All other mirrors from geo.mirror | |
+| monitoring.archlinux.org.yml | wireguard,prometheus | Allow loki and prometheus to fetch data | | |
+| archlinux.org.yml | postgres,wireguard | Allow wireguard IP to connect for Mirror check | | Optional see Check Location below |
+
+### Add mirror in geo.mirror.pkgbuild.com
+
+Add mirror IP and FQDN in archweb admin https://archlinux.org/admin/mirrors/mirror/ under the `geo.mirror.pkgbuild.com` entry.
+
+### Check Location (optional)
+
+If you want the server to check for ping and stats create an entry in:
+ https://archlinux.org/admin/mirrors/checklocation/

docs/grow-disks.md

 # Growing (partitioned) Disks
 
-Our VPS are provisioned with 20G as CX11 by default. When one is resized the disk size usually changes.
+Our VPS are provisioned with 40G as CX22 by default. When one is resized the disk size usually changes.
 To use the additional space, one needs to grow the partition and the filesystem.
 
 ## Resizing partition

docs/ipmi.md

-# IPMI
-
-One of our servers has IPMI access, to use it install ipmitool and active an
-IPMI remote shell with:
-
-```
-impitool -C3 -I lanplus -H $ip -U $username -P $password sol active
-```

docs/maintenance.md

@@ -27,6 +27,7 @@ The basic configuration looks like this:
     service_name: "<service name>"
     service_domain: "{{ service_domain }}"
     service_alternate_domains: []
+    service_legacy_domains: []
     service_nginx_conf: "{{ service_nginx_conf }}"
   when: maintenance is defined
 ```
@@ -39,7 +40,7 @@ as a variable, to make sure the right file is used.
 - name: set up nginx
   template: src=nginx.d.conf.j2 dest="{{ service_nginx_conf }}" owner=root group=root mode=644
   notify:
-    - reload nginx
+    - Reload nginx
   when: maintenance is not defined
   tags: ['nginx']
 ```

docs/matrix.md

@@ -11,37 +11,47 @@ For the initial sign-in you need to use a client that supports OpenID Single-Sig
 [Element Web](https://app.element.io/). Enter `@username:archlinux.org` as the username and Element
 should offer to sign into our homeserver.
 
-You will be automatically invited to several rooms:
-  - `#archlinux:archlinux.org`: A public room for Arch Linux users.
-  - `#internal:archlinux.org`: A staff-only room with end-to-end encryption.
+You will be automatically invited to several spaces and rooms:
+  - `#public-space:archlinux.org`: A public space for Arch Linux users.
+    - `#archlinux:archlinux.org`: A public room for Arch Linux users.
+  - `#staff-space:archlinux.org`: A staff-only space for Arch Linux staff.
+    - `#internal:archlinux.org`: A staff-only room with end-to-end encryption.
 
 Password login is currently disabled, which might exclude some clients. It can be re-enabled should
 demand exist.
 
 If you need to provide your client with a homeserver address, use `https://matrix.archlinux.org`.
 
-## IRC bridges
+## Our rooms bridged to IRC
 
-### Our bridge
+We bridge several of our private IRC channels on Libera.Chat to Matrix.
 
-We bridge several of our private IRC channels on Libera Chat to Matrix, which you need to be invited
-into:
+These rooms are open to all staff-space members:
+  - `#packaging:archlinux.org`: Bridged with `#archlinux-packaging`.
+  - `#staff:archlinux.org`: Bridged with `#archlinux-staff`.
+
+The following rooms are not open to all staff, so you need to be invited:
   - `#developers:archlinux.org`: Bridged with `#archlinux-dev`.
   - `#trusted-users:archlinux.org`: Bridged with `#archlinux-tu`.
-  - `#staff:archlinux.org`: Bridged with `#archlinux-staff`.
 
 Please request an invitation in `#internal:archlinux.org` for the rooms you need to be in.
 
-### Matrix.org bridge
-
-Channels without keys are available via the official Libera Chat bridge. For example:
-  - `#archlinux-devops:libera.chat`: Bridged with `#archlinux-devops`.
-  - `#archlinux-projects:libera.chat`: Bridged with `#archlinux-projects`.
-
-**Please avoid joining large bridged rooms (such as `#archlinux:libera.chat`), as these slow down
-the server immensely.**
-
-Libera Chat may require you to have a registered nick to join certain channels. Once
-`@appservice:libera.chat` contacts you, tell it `!username <username>`, then `!storepass <password>`
-with the username and the password of your Libera Chat NickServ account. Then `!reconnect` and it
-will reconnect you as registered.
+These rooms are bridged to public channels, for which you should log into Libera.Chat via SASL:
+  - `#aurweb:archlinux.org`: Bridged with `#archlinux-aurweb`.
+  - `#bugs:archlinux.org`: Bridged with `#archlinux-bugs`.
+  - `#devops:archlinux.org`: Bridged with `#archlinux-devops`.
+  - `#pacman:archlinux.org`: Bridged with `#archlinux-pacman`.
+  - `#projects:archlinux.org`: Bridged with `#archlinux-projects`.
+  - `#reproducible:archlinux.org`: Bridged with `#archlinux-reproducible`.
+  - `#security:archlinux.org`: Bridged with `#archlinux-security`.
+  - `#testing:archlinux.org`: Bridged with `#archlinux-testing`.
+  - `#wiki:archlinux.org`: Bridged with `#archlinux-wiki`.
+
+If you fail to do so, your bridged IRC user cannot join the channels, meaning your messages won't be
+bridged. See [Libera.Chat's guide](https://libera.chat/guides/registration) on how to register a
+nickname. Afterwards, contact `@irc-bridge:archlinux.org` and send it the folllowing commands:
+  - `!username <username>`, with the primary nickname you registered with, then
+  - `!storepass <password>`, with your password for NickServ, and then
+  - `!reconnect` to reconnect and attempt the SASL login.
+
+If this worked, `@liberachat_SaslServ:archlinux.org` should contact you after the reconnect.

docs/monitoring.md

@@ -5,7 +5,6 @@ To access our monitoring system, go to https://monitoring.archlinux and log in v
 
 ## Adding a new host to monitoring
 
-* Add $host to node_exporters in `hosts`
 * Rollout exporter on host: `ansible-playbook playbooks/host.yml -t prometheus_exporters`
 * Rollout changes on monitoring host: `ansible-playbook playbooks/monitoring.archlinux.org.yml -t prometheus`
 

docs/otp.md

@@ -14,7 +14,7 @@ Run
 
     pass otp insert -i GitHub -a archlinux-master-token github.com/archlinux-master-token -s
 
-When asked for a secret, provide the `github_master_seed` from `misc/vault_github.yml`.
+When asked for a secret, provide the `github_master_seed` from `misc/vaults/vault_github.yml`.
 You can then run
 
     pass otp code github.com/archlinux-master-token
@@ -30,7 +30,7 @@ Run
 
     pass otp insert -i Hetzner -a archlinux-master-token Hetzner/archlinux-master-token -s
 
-When asked for a secret, provide the `hetzner_master_seed` from `misc/vault_hetzner.yml`.
+When asked for a secret, provide the `hetzner_master_seed` from `misc/vaults/vault_hetzner.yml`.
 You can then run
 
     pass otp code Hetzner/archlinux-master-token
@@ -43,7 +43,7 @@ Run
 
     pass otp insert -i UptimeRobot -a archlinux UptimeRobot/archlinux-master-token -s
 
-When asked for a secret, provide the `2FA token seed` from `misc/additional-credentials.vault`.
+When asked for a secret, provide the `2FA token seed` from `misc/vaults/additional-credentials.vault`.
 You can then run
 
     pass otp code UptimeRobot/archlinux-master-token
@@ -63,6 +63,19 @@ You can then run
 
 to generate a token to log in.
 
+## Vagrant Cloud
+
+Run
+
+    pass otp insert -i VagrantCloud -a archlinux VagrantCloud/archlinux-master-token -s
+
+When asked for a secret, provide the `vagrant_cloud_seed` from `misc/vaults/vault_vagrant_cloud.yml`.
+You can then run
+
+    pass otp code VagrantCloud/archlinux-master-token
+
+to generate a token to log in.
+
 ### Adding your own account
 
 Hetzner supports multiple 2FA devices at once which allows you to add your own 2FA app of choice

docs/quassel.md

@@ -14,10 +14,10 @@ support@libera.chat.
 
 ## Migration quassel
 
-Stop the quassel service:
+Stop the quassel service, then:
 
 `sudo -u postgres pg_dump -F c quassel >quassel.dump`
 
 Restore the data:
 
-`sudo -u postgres pg_restore -d quassel --clean --exit-on-error <quassel.dump`
+`sudo -u postgres pg_restore -d quassel --clean --if-exists --exit-on-error <quassel.dump`

docs/servers.md

@@ -3,11 +3,9 @@
 ## Table of contents
 [[_TOC_]]
 
-## gemini
+## archive.archlinux.org
 
 ### Services
-  - repos/sync (repos.archlinux.org)
-  - sources (sources.archlinux.org)
   - archive (archive.archlinux.org)
 
 ## lists.archlinux.org
@@ -26,19 +24,6 @@
 ### Services
   - aurweb
 
-## aur-dev.archlinux.org
-
-### Services
-  - aurweb deployed with the `pu` branch
-
-  The database is filled using the `./schema/gendummydata.py` script in the aurweb repository with a
-  modification to create suspended users. Test users can sign up using the normal registration flow.
-
-## bugs.archlinux.org
-
-### Services
-  - flyspray
-
 ## bbs.archlinux.org
 
 ### Services
@@ -99,12 +84,20 @@ So to set up this server from scratch, run:
 [Rebuilderd docs](./docs/rebuilderd.md)
 
 ### Services
-  - Runs a master [rebuilderd](https://reproducible.archlinux.org) instance two workers:
-    - repro1.pkgbuild.com (packet.net Arch Linux box)
+  - Runs a master [rebuilderd](https://reproducible.archlinux.org) instance
+    with these workers:
+    - repro2.pkgbuild.com (Kape server with an EPYC 7702P and 256G RAM - 4 workers)
+    - repro3.pkgbuild.com (Equinix Metal box with a Xeon E-2278G and 64G RAM - 2 workers)
+    - repro4.pkgbuild.com (Proxmox VM with 16vCores and 192G RAM - 2 workers)
+
 
-## runner2.archlinux.org
+## runner1.archlinux.org
 
-Medium-fast-ish packet.net Arch Linux box.
+Medium-fast-ish Kape Arch Linux box.
+
+## runner3.archlinux.org
+
+Medium-fast-ish Equinix Metal Arch Linux box.
 
 ### Services
   - GitLab runner
@@ -125,6 +118,11 @@ Medium-fast-ish packet.net Arch Linux box.
   - [Grafana](https://monitoring.archlinux.org) and [docs/grafana.md](./docs/grafana.md)
   - Prometheus
 
+## mumble.archlinux.org
+
+### Services
+  - Mumble
+
 ## dashboards.archlinux.org
 
 Prometheus, and Grafana server which receives selected performance/metrics from monitoring.archlinux.org and make them public accessible.
@@ -133,17 +131,19 @@ Prometheus, and Grafana server which receives selected performance/metrics from
   - [Grafana](https://dashboards.archlinux.org) and [docs/grafana.md](./docs/grafana.md)
   - Prometheus
 
-## patchwork.archlinux.org
-
-### Services
-  - patchwork
-
 ## redirect.archlinux.org
 
 ### Services
   - Redirects (nginx redirects)
+  - Authoritative DNS server (PowerDNS) for ACME DNS challenges
   - ping
 
+## repos.archlinux.org
+
+### Services
+  - repos/sync (repos.archlinux.org)
+  - sources (sources.archlinux.org)
+
 ## security.archlinux.org
 
 ### Services
@@ -169,3 +169,8 @@ The [Arch Linux Archive](https://archive.archlinux.org) is mirrored to three ded
   - https://america.archive.pkgbuild.com
   - https://asia.archive.pkgbuild.com
   - https://europe.archive.pkgbuild.com
+
+## gitlab.archlinux.org
+
+### Services
+  - GitLab