Commits (595)
......@@ -15,6 +15,8 @@ terraform-validate:
- cd tf-stage1
- terraform init -backend=false
- terraform validate
- terraform fmt --check
- cd ../tf-stage2
- terraform init -backend=false
- terraform validate
- terraform fmt --check --diff
<!--
This template should be used for decommissioning Arch Linux servers.
-->
# Decommissioning an Arch Linux server
## Details
- **Server name**: <!-- Put server hostname here -->
## Steps
- [ ] Notified arch-devops about decommissioning plan (at least a week prior)
- [ ] (Optional) Notified arch-staff about decommissioning plan (at least two weeks prior)
- [ ] Make sure data is backed up and migrated
- [ ] Wait until decommissioning date
- [ ] Boot server in rescue system
- [ ] Securely wipe disks
- [ ] Schedule server to be cancelled in provider interface
- [ ] Report back to arch-devops
- [ ] Report back to arch-staff
<!--
This template should be used for adding a new Arch Linux Archive Mirror.
-->
# Add a new Archive Mirror Server
## Details
- **Server name**: <!-- Put server hostname here -->
## Steps
- [ ] Verify if the new mirror has enough diskspace for hosting the archive
- [ ] Add a new domain in terraform
- [ ] Add the new server to `archive_mirrors` in `hosts`
- [ ] Run the dbscripts role to allow the new archive mirror to sync
- [ ] Run the `archive-mirrors.yml` playbook
- [ ] Run `certbot certonly --email webmaster@archlinux.org --agree-tos --rsa-key-size 4096 --renew-by-default --webroot -w /var/lib/letsencrypt/ -d <domain-name>`
<!--
This template should be used by DevOps members when adding a GitLab Pages project to GitLab.
In order to use GitLab Pages with Arch Linux, you'll have to specifically request a custom subdomain
below `pkgbuild.com` or `archlinux.org` to be assigned. We don't allow random projects to use Pages
because of legal and safety reasons (we don't want people to be able to trick others into thinking
something hosted below one of our domains is official).
-->
# Procedure for adding a GitLab Pages project to GitLab
## Details
- **Project name**: hello
- **Desired subdomain**: hello.archlinux.org
## New Pages site checklist
1. [ ] Have a pipeline that outputs some static HTML to `public/` during the build.
1. [ ] Specify this path (`public/`) as an artifact path.
1. [ ] GitLab should now recognize that you're trying to use Pages and will show some relevant
information at https://gitlab.archlinux.org/your-namespace/your-project/pages
1. [ ] At this page, you'll also need to add your custom domain. Add the custom domain you requested earlier.
GitLab will then show domain verification information which you'll need in the next step.
1. [ ] At this point, we'll need to add some stuff to the `archlinux_org_gitlab_pages` variable in `archlinux.tf`. It should look something like this.
Make sure to substitute the `your-domain` and `your-code-shown-by-gitlab` strings accordingly:
"your-domain" = "your-code-shown-by-gitlab"
1. [ ] Run `terraform apply` and go back to GitLab. Hit `Verify` and it should pick up the new domain
verification code. It should then also automatically begin fetching a certificate via Let's
Encrypt. That should take roughly 10min.
......@@ -17,7 +17,8 @@ If you want to add a new official project, here are some guidelines to follow:
1. [ ] Evaluate whether the project can sit in the official [GitLab Arch Linux group](https://gitlab.archlinux.org/archlinux)
or whether it needs its own group. It only needs its own group if the primary
development group is somehow detached from Arch Linux and only losely related (for instance: [pacman](https://gitlab.archlinux.org/pacman))
1. [ ] After project creation, add the responsible people to the project in the *Members* page (https://gitlab.archlinux.org/archlinux/my-example/-/project_members)
1. [ ] After project creation (use the GitLab import function if you migrate a repo), add the responsible people to the project
in the *Members* page (https://gitlab.archlinux.org/archlinux/my-example/-/project_members)
and give them the `Developer` role. The idea is to let these people mostly manage their own project while not giving them
enough permissions to be able to misconfigure the project.
1. [ ] If mirroring to github.com is desired, work through the **GitHub.com mirroring checklist**
......@@ -25,7 +26,7 @@ If you want to add a new official project, here are some guidelines to follow:
1. [ ] If the project needs a secure runner to build trusted artifacts, coordinate with
the rest of the DevOps team and if found to be reasonable, assign a secure runner
to a protected branch of the project.
1. [ ] If a secure runner is used, make sure the project's `.gitlab-ci.yml` specifies
1. [ ] If a secure runner is used, create an MR to make sure the project's `.gitlab-ci.yml` specifies
`tags: secure`.
1. [ ] Make sure that the *Push Rules* in https://gitlab.archlinux.org/archlinux/arch-boxes/-/settings/repository
reflect these values:
......
......@@ -8,6 +8,8 @@ This template should be used for offboarding Arch Linux team members.
- **Team member username**:
- **Currently held roles**: <!-- Add known roles here like TU, DevOps, etc -->
- **Removal request**: <!-- Add link to relevant mailing list mail -->
- **Voting result**: <!-- Add link to relevant mailing list mail -->
## All roles checklist
......@@ -20,6 +22,9 @@ This template should be used for offboarding Arch Linux team members.
- [ ] Remove SSH pubkey from `pubkeys/<username>.pub`.
- [ ] Run `ansible-playbook -t archusers playbooks/*.yml`.
- [ ] Remove the user from the `Trusted Users`/`Developers` groups on Keycloak.
- [ ] Moderate email address on [arch-dev-public](https://lists.archlinux.org/admin/arch-dev-public/members) (find member and moderate)
- [ ] Remove member from [arch-tu mailing lists](https://lists.archlinux.org/admin/arch-tu/members)
- [ ] Remove member from [staff mailing lists](https://lists.archlinux.org/admin/staff/members)
## DevOps offboarding checklist
......@@ -27,3 +32,10 @@ This template should be used for offboarding Arch Linux team members.
- [ ] Run `ansible-playbook -t root_ssh playbooks/*.yml`.
- [ ] Run `ansible-playbook playbooks/hetzner_storagebox.yml playbooks/rsync.net.yml`.
- [ ] Remove the user from the `DevOps` group on Keycloak.
- [ ] Remove member from [arch-devops-private mailing lists](https://lists.archlinux.org/admin/arch-devops-private/members)
- [ ] Remove pubkey from [Hetzner's key management](https://robot.your-server.de/key/index)
## Wiki Administrator checklist
- [ ] Remove the user from the `Wiki Admins` group on Keycloak.
- [ ] Remove member from [arch-wiki-admins mailing list](https://lists.archlinux.org/admin/arch-wiki-admins/members).
......@@ -8,19 +8,33 @@ It can also be used as a reference for adding new roles to an existing team memb
## Details
- **Team member username**:
- **Application**: <!-- Add link to relevant mailing list mail -->
- **Voting result**: <!-- Add link to relevant mailing list mail -->
## All roles checklist
- [ ] Add new user email as per `docs/email.md`.
- [ ] Create a new user in archweb: https://www.archlinux.org/devel/newuser/
This is also linked in the django admin backend at the top
- [ ] Subscribe user to internal [staff mailing list](https://lists.archlinux.org/admin/staff/members/add)
## TU/Developer onboarding checklist
## Developer onboarding checklist
- [ ] Add entry in `group_vars/all/archusers.yml`.
- [ ] Add SSH pubkey to `pubkeys/<username>.pub`.
- [ ] Run `ansible-playbook -t archusers playbooks/*.yml`.
- [ ] Assign the user to the `Trusted Users`/`Developers` groups on Keycloak.
- [ ] Assign the user to the `Developers` groups on Keycloak.
- [ ] Subscribe user to internal [arch-dev mailing list](https://lists.archlinux.org/admin/arch-dev/members/add)
- [ ] Whitelist email address on [arch-dev-public](https://lists.archlinux.org/admin/arch-dev-public/members) (find member and unmoderate)
## TU onboarding checklist
- [ ] Add entry in `group_vars/all/archusers.yml`.
- [ ] Add SSH pubkey to `pubkeys/<username>.pub`.
- [ ] Run `ansible-playbook -t archusers playbooks/*.yml`.
- [ ] Assign the user to the `Trusted Users` groups on Keycloak.
- [ ] Whitelist email address on [arch-dev-public](https://lists.archlinux.org/admin/arch-dev-public/members) (find member and unmoderate)
- [ ] Subscribe user to internal [arch-tu mailing list](https://lists.archlinux.org/admin/arch-tu/members/add)
## DevOps onboarding checklist
......@@ -28,7 +42,10 @@ It can also be used as a reference for adding new roles to an existing team memb
- [ ] Run `ansible-playbook -t root_ssh playbooks/*.yml`.
- [ ] Run `ansible-playbook playbooks/hetzner_storagebox.yml playbooks/rsync.net.yml`.
- [ ] Assign the user to the `DevOps` group on Keycloak.
- [ ] Subscribe user to [arch-devops-private mailing lists](https://lists.archlinux.org/admin/arch-devops-private/members/add)
- [ ] Add pubkey to [Hetzner's key management](https://robot.your-server.de/key/index) for Dedicated server rescue system.
## Wiki Administrator checklist
- [ ] Assign the user to the `Wiki Admins` group on Keycloak.
- [ ] Subscribe the user to the [arch-wiki-admins mailing list](https://lists.archlinux.org/admin/arch-wiki-admins/members/add).
......@@ -2,6 +2,9 @@
This repository contains the complete collection of ansible playbooks and roles for the Arch Linux infrastructure.
## Table of contents
[[_TOC_]]
## Requirements
Install these packages:
......@@ -37,16 +40,6 @@ all servers directly from hcloud. You don't really have to do anything to make
this work but you should keep in mind to NOT add hcloud servers to `hosts`!
They'll be available automatically.
#### Note about first time certificates
The first time a certificate is issued, you'll have to do this manually by yourself. First, configure the DNS to
point to the new server and then run a playbook onto the server which includes the nginx role. Then on the server,
it is necessary to run the following once:
certbot certonly --email webmaster@archlinux.org --agree-tos --rsa-key-size 4096 --renew-by-default --webroot -w /var/lib/letsencrypt/ -d <domain-name>
Note that some roles already run this automatically.
#### Note about packer
We use packer to build snapshots on hcloud to use as server base images.
......@@ -96,7 +89,7 @@ set up.
#### SMTP Configuration
All hosts should be relaying email through our primary mx host (currently 'orion'). See [docs/email.md](./docs/email.md) for full details.
All hosts should be relaying email through our primary mx host (currently 'mail.archlinux.org'). See [docs/email.md](./docs/email.md) for full details.
#### Note about opendkim
......@@ -144,127 +137,7 @@ The following steps should be used to update our managed servers:
## Servers
### orion
#### Services
- repos/sync (repos.archlinux.org)
- sources (sources.archlinux.org)
- archive (archive.archlinux.org)
### luna
#### Services
- mailman
- projects (projects.archlinux.org)
### apollo
#### Services
- wiki (wiki.archlinux.org)
- archweb
- patchwork
### aur.archlinux.org
#### Services
- aurweb
### bugs.archlinux.org
#### Services
- flyspray
### bbs.archlinux.org
#### Services
- bbs
### phrik.archlinux.org
#### Services
- phrik (irc bot) users in the phrik group defined in
the hosts vars and re-used the archusers role. Users
in the phrik group are allowed to restar the irc bot.
### dragon
#### Services
- build server
- sogrep
### state.archlinux.org
#### Services
- postgres server for terraform state
### quassel.archlinux.org
#### Services
- quassel core
### matrix.archlinux.org
#### Services
- Matrix homeserver (Synapse)
- Matrix ↔ IRC bridge
### homedir.archlinux.org
#### Services
- ~/user/ webhost
### accounts.archlinux.org
This server is _special_. It runs keycloak and is central to our unified Arch Linux account management world.
It has an Ansible playbook for the keycloak service but that only installs the package and starts it but it's configured via a secondary Terraform file only for keycloak `keycloak.tf`.
The reason for doing it this way is that Terraform support for Keycloak is much superior and it's declarative too which is great for making sure that no old config remains in the case of config changes.
So to set up this server from scratch, run:
- `cd tf-stage1`
- `terraform apply`
- `cd ../tf-stage2`
- `terraform import keycloak_realm.master master`
- `terraform apply`
#### Services
- keycloak
### mirror.pkgbuild.com
#### Services
- Regular mirror.
### reproducible.archlinux.org
#### Services
- Runs a master rebuilderd instance two workers:
- repro1.pkgbuild.com (PIA worker)
- repro3.pkgbuild.com (packet.net machine which runs Ubuntu)
### runner1.archlinux.org
Slow-ish PIA box with spinning disks.
#### Services
- GitLab runner
### runner2.archlinux.org
Medium-fast-ish packet.net box with Debian on it. Is currently maintained manually.
#### Services
- GitLab runner
### monitoring.archlinux.org
Prometheus server which collects performance/metrics from our services and runs alertmanager
### Services
- Prometheus
- Alertmanager
This section has been moved to [docs/servers.md](docs/servers.md).
## Ansible repo workflows
......@@ -290,45 +163,10 @@ Medium-fast-ish packet.net box with Debian on it. Is currently maintained manual
## Backup documentation
Adding a new server to be backed up goes as following:
* Make sure the new servers host key is synced to `docs/ssh-known_hosts.txt` if not run:
ansible-playbook playbooks/tasks/sync-ssh-hostkeys.yml
* Add the server to [borg-clients] in hosts
* Run the borg role on u236610.your-storagebox.de to allow the new machine to create backups
ansibe-playbook playbooks/hetzner_storagebox.yml
* Run the borg role for rsync.net to allow the new machine to create backups
ansibe-playbook playbooks/rsync.net.yml
* Run the borg role on the new machine to initialize the repository
ansibe-playbook playbooks/$machine.yml -t borg
Backups should be checked now and then. Some common tasks are listed below.
You'll have to get the correct username from the vault.
### Listing current backups per server
borg list ssh://<hetzner_storagebox_username>@u236610.your-storagebox.de:23/~/backup/<hostname>
borg list ssh://<rsync_net_username>@prio.ch-s012.rsync.net:22/~/backup/<hostname>
Example
borg list ssh://<hetzner_storagebox_username>@u236610.your-storagebox.de:23/~/backup/homedir.archlinux.org
### Listing files in a backup
borg list ssh://<hetzner_storagebox_username>@u236610.your-storagebox.de:23/~/backup/<hostname>::<archive name>
Example
We use BorgBackup for all of our backup needs. We have a primary backup storage as well as an
additional offsite backup.
borg list ssh://<hetzner_storagebox_username>@u236610.your-storagebox.de:23/~/backup/homedir.archlinux.org::20191127-084357
See [docs/backups.md](./docs/backups.md) for detailed backup information.
## Updating Gitlab
......
# Backups
Backups should be checked now and then. Some common tasks are listed below.
You'll have to get the correct username from the vault.
## Accessing backup hosts
We use two different borg backup hosts: A primary one and an offsite one.
The URL format for the primary one is
ssh://<hetzner_storagebox_username>@u236610.your-storagebox.de:23/~/backup/<hostname>
while for the offsite one it's
ssh://<rsync_net_username>@prio.ch-s012.rsync.net:22/~/backup/<hostname>
In the examples below, we'll just abbreviate the full address as `<backup_address>`.
If you want to use one of the examples below, you'll have to fill in the
placeholder with your desired full address to the backup repository. For instance,
misc/borg.sh list <backup_address>::20191127-084357
becomes
misc/borg.sh ssh://<hetzner_storagebox_username>@u236610.your-storagebox.de:23/~/backup/homedir.archlinux.org::20191127-084357
A convenience wrapper script is available at `misc/borg.sh` which makes sure you
use the correct keyfile for the given server.
## Listing backups in repository
This allows you to check which backups are currently available for the given server:
misc/borg.sh list <backup_address>
## Listing files in a specific backup
Once you figured out which backup you want to use, you can list the files inside via:
misc/borg.sh list <backup_address>::<archive_name>
## Getting info for a repository
Check how large all backups for a server are:
misc/borg.sh info <backup_address>
## Getting info for a specific backup
Check how large a single backup is and how long it took to perform:
misc/borg.sh info <backup_address>::<archive_name>
## Mounting a backup
One convenient way to access the files inside an archive is to mount it:
mkdir mnt
misc/borg.sh mount <backup_address>::<archive_name> mnt
You might want to mount it with `-o ignore_permissions` depending on which user
you're using to access the backup.
## Extracing files from a backup
Alternatively, if you don't want to mount it and instead want to extract files directly, you can
do so. Either extract the whole backup:
misc/borg.sh extract <backup_address>::<archive_name>
or just a sub-directory:
misc/borg.sh extract <backup_address>::<archive_name> backup/srv/gitlab
## Adding a new server
Adding a new server to be backed up goes as follows:
* Make sure the new servers host key is synced to `docs/ssh-known_hosts.txt` if not run:
ansible-playbook playbooks/tasks/sync-ssh-hostkeys.yml
* Add the server to [borg-clients] in hosts
* Run the borg role on u236610.your-storagebox.de to allow the new machine to create backups
ansible-playbook playbooks/hetzner_storagebox.yml
* Run the borg role for rsync.net to allow the new machine to create backups
ansible-playbook playbooks/rsync.net.yml
* Run the borg role on the new machine to initialize the repository
ansible-playbook playbooks/$machine.yml -t borg
# Becoming Arch Linux DevOps
In Arch Linux, DevOps are expected to be reliable, trusthworthy, and self-directed.
DevOps should be known and trusted by the community beforehand or be recommended by previous
members.
## Junior DevOps program
In order be able to onboard lesser-known members of the community who still want to help out with
DevOps topics, we started the Junior DevOps program. This program requires applicants to
0) have contributed to Arch multiple times in some meaningful ways,
1) find two sponsors, and
2) write an application to the arch-devops mailing list.
The idea of Junior DevOps is that they don't get full access to all secrets and machines as opposed
to full DevOps and have to make operational changes in pairing session with a full DevOps.
However, Junior DevOps can already help with many tasks and are expected to take charge of a given
topic.
After a lot of trust is built up, Junior DevOps may graduate to become full DevOps.
# Configuration for users
SMTP/IMAP server: mail.archlinux.org
SMTP port: 587 STARTTLS
IMAP port: 143 (STARTTLS), 993 (TLS)
SMTP port: 465 (TLS), [deprecated: 587 STARTTLS]
IMAP port: 993 (TLS)
username: the system account name
password: set by each user themselves with `passwd` on orion
password: set by each user themselves with `passwd` on mail.archlinux.org
# Adding new archlinux.org email addresses
Login to orion and edit `/etc/postfix/users`, add the new email address in the
Login to mail.archlinux.org and edit `/etc/postfix/users`, add the new email address in the
appropriate category and run `postmap /etc/postfix/users`.
If the user wants to forward email, either enter the destination directly in
......@@ -19,7 +19,7 @@ into `~username/.forward` so that they can edit it themselves.
# SMTP Architecture
All hosts should be relaying outbound SMTP traffic via our primary MX server
(currently 'orion'). Each hosts authenticates using SASL over a TLS connection
(currently 'mail.archlinux.org'). Each hosts authenticates using SASL over a TLS connection
to the server. This gives us several benefits:
1. DKIM signing can be done centrally.
......@@ -31,16 +31,30 @@ to the server. This gives us several benefits:
When a new host is provisioned:
- The *postfix* role has a task delegated to 'orion' to create a local user
on 'orion' that is used for the new server to authenticate against. The user
- The *postfix* role has a task delegated to 'mail.archlinux.org' to create a local user
on 'mail.archlinux.org' that is used for the new server to authenticate against. The user
name is the shortname of the new servers hostname (ie, "foobar.archlinux.org"
will authenticate with the username "foobar")
- You will need to run the *postfwd* role against orion to update the
- You will need to run the *postfwd* role against mail.archlinux.org to update the
rate-limiting it performs (servers are given higher rate-limits than normal
users - see `/etc/postfwd/postfwd.cf` for exact limits). This *should*
happen automatically as the *postfwd* role is a dependency of the *postfix*
role (using `delegate_to` to run it against 'orion' regardless of the target
role (using `delegate_to` to run it against 'mail.archlinux.org' regardless of the target
host that the postfix role is being run on)
- Any services on the new host that need to relay mail should relay using SMTP
to `localhost` on port 10027 which bypasses any filtering/restrictions that
are applied by postfix to port 25 traffic.
# Create new DKIM keys
The rspamd role expects the key to exist in the vault. To generate new keys, run
```
rspamadm dkim_keygen -s dkim-ed25519 -b 0 -d archlinux.org -t ed25519 -k archlinux.org.dkim-ed25519.key
rspamadm dkim_keygen -s dkim-rsa -b 4096 -d archlinux.org -t rsa -k archlinux.org.dkim-rsa.key
```
the ouput gives you the DNS entries to add to the terraform files.
The keys generated need to go to the vault:
```
roles/rspamd/files/archlinux.org.dkim-rsa.key
roles/rspamd/files/archlinux.org.dkim-ed25519.key
```
# fail2ban
Fail2Ban is an intrusion prevention software framework that protects computer servers from brute-force attacks.
By default all our servers playbook should include the `fail2ban` role which enables the `sshd` jail by default through `groups_vars`. The default `/etc/fail2ban/jail.local` configuration whitelists all the servers in `hosts`, take note that when adding a new postfix relayhost the `fail2ban` role has to be run on the postfix server to update the whitelist.
## Jails
Fail2ban can provide multiple jails for different services, to check the status of for example the `sshd` jail:
```
fail2ban-client status sshd
```
To unblock an IP Address:
```
fail2ban-client set sshd unbanip 8.8.8.8
```
### sshd
The sshd jail should be enabled for every host we have, to block brute force ssh attacks.
### postfix
The postfix jail not enabled on any server. Adding it to a host:
Add `fail2ban_jails` dict with `postfix: true` to the host's `host_vars`.
### dovecot
The dovecot jail is enabled for our mail server, blocking failed logins. Adding it to a host:
Add `fail2ban_jails` dict with `dovecot: true` to the host's `host_vars`.
# Grafana
Our Grafana is hosted on https://monitoring.archlinux.org and is accessible for
all Arch Linux Staff, editing rights are restricted to users with the Devops
Role.
Dashboards and datasources are automatically provisioned by Grafana with Grafana's built-in [provisioning configuration](https://grafana.com/docs/grafana/latest/administration/provisioning/).
## Adding a new Dashboard
A new dashboard can be configured in our Grafana instance to try it out and if satisfactory checked in to Git as following:
* Export the dashboard to json (top left, share dashboard => exporter => save to file).
* Save the json file in `roles/grafana/files/dashboards'
* Git add the file and run the grafana playbook
# Growing (partitioned) Disks
Our VPS are provisioned with 20G as CX11 by default. When one is resized the disk size usually changes.
To use the additional space, one needs to grow the partition and the filesystem.
## Resizing partition
Grow the partition with a tool called growpart
growpart /dev/sdX <partnum>
So for most of our machines this is:
growpart /dev/sda 1
## Resizing filesystem
This is straight forward
btrfs fi res max <mountpoint>
For most of our setups, being in the root homedir:
btrfs fi res max .
## Kape Servers
All donated servers are with EFI except runner1.archlinux.org.
### Archive mirrors
Three servers have set up as archive mirrors:
* america.mirror.pkgbuild.com
* asia.mirror.pkgbuild.com
* europe.mirror.pkgbuild.com
The servers have been setup as RAID 5 with 3 x 10TB disks.
### Gitlab runner
A runner is setup on a 2xE5-2620v4 - 64GB - 2x 1TB SSD as runner1.archlinux.org.
### Rebuilderd worker
A rebuilderd worker is setup on a EPYC - 256GB - 2x 500GB SSD as repro2.pkgbuild.com
......@@ -19,6 +19,7 @@ mode configures nginx. There are a few examples of roles that can be used, like
The basic configuration looks like this:
```
- name: run maintenance mode
include_role:
name: maintenance
......@@ -28,17 +29,29 @@ The basic configuration looks like this:
service_alternate_domains: []
service_nginx_conf: "{{ service_nginx_conf }}"
when: maintenance is defined
```
This is best placed at the top of the tasks main file for the role, to make sure it is ran first.
Replace <service_name> with the name of the web service. The nginx configuration is best to be set
as a variable, to make sure the right file is used.
```
- name: set up nginx
template: src=nginx.d.conf.j2 dest="{{ service_nginx_conf }}" owner=root group=root mode=644
notify:
- reload nginx
when: maintenance is not defined
tags: ['nginx']
```
This causes the regular nginx configuration to only be applied when there is no maintenance variable
on the command line.
# Adding a custom maintenance mode nginx template
The maintenance role can also use a custom nginx template, if the service_nginx_template variable is
set alongside the other vars when including the maintenance role, it will look up first on the maintenance
role template directory and then on the calling role template directory for the specified template.
Since this is a completely custom file, it is the job of this file of putting the service into maintenance
mode. The maintenance role will provide the 503 file and create the directories.
# Monitoring
All of our servers are monitored using Prometheus, exporters on the to be monitored machines have a firewall rule configured to allow connections from monitoring.archlinux.org for the specific exporter port.
To access our monitoring system, go to https://monitoring.archlinux and log in via your Arch Linux SSO credentials.
## Adding a new host to monitoring
* Add $host to node_exporters in `hosts`
* Rollout exporter on host: `ansible-playbook playbooks/host.yml -t prometheus_exporters`
* Rollout changes on monitoring host: `ansible-playbook playbooks/monitoring.archlinux.org.yml -t prometheus`
### System
For general system performance monitoring [prometheus-node-exporter](https://github.com/prometheus/node_exporter) is used in combination with the textfile collector for Arch Linux specific metrics. A systemd service/timer 'prometheus-arch-textcollector' writes the amount of out of date packages and security updates. When running the prometheus_exporters role the node-exporter and arch textcollector is automatically added.
### memcached
[prometheus-memcached-exporter](https://github.com/prometheus/memcached_exporter) is used for monitoring. Adding memcached monitoring to a host is as simple as:
* Add the host to the `memcached` group
* Add `memcached_socket` to the `host_vars` of the machine with the location of the memcached socket
* Rollout exporter on host: `ansible-playbook playbooks/host.yml -t prometheus_exporters`
### Borg
For monitoring our borg backups prometheus-node-exporter's textfile collector feature is used, the textfile is written by a systemd service run periodically by a systemd timer called prometheus-borg-textcollector. Borg's last backup time is recorded for our Hetzner and rsync.net backups. Adding monitoring to a system is as simple as:
* Add the host to the `borg_clients` group
* Rollout exporter on host: `ansible-playbook playbooks/host.yml -t prometheus_exporters`
### rebuilderd
The rebuilderd instance Arch Linux hosts is monitored using prometheus-node-exporter's textfile collector feature which periodically collects data using a prometheus-rebuilderd-textcollector.timer. The 'rebuilderd-textcollector.sh' script collects the queue length and amount of working rebuilders to monitor if the rebuilderd queue keeps growing forever or rebuilderd workers stopped working. The 'rebuilderd-status-textcollector.py' script collects the rebuilderd status good, bad and unknown packages per repository for keeping tracking of the reproducible builds progress. Adding monitoring for rebuilderd:
* Add the rebuilderd instance to the `rebuilderd` group
* Rollout exporter on host: `ansible-playbook playbooks/host.yml -t prometheus_exporters`
### MySQL
For monitoring MySQL [prometheus-mysqld-exporter](https://github.com/prometheus/mysqld_exporter) configured to use a separate user for obtaining MySQL statistics.
* Add the host to the `mysql_servers` group
* Rollout exporter on host: `ansible-playbook playbooks/host.yml -t prometheus_exporters`
### Keycloak
For monitoring Keycloak [keycloak-metrics-spi](https://github.com/aerogear/keycloak-metrics-spi) is used, which exports basic Keycloak user events such as logins, errors and registration errors. The exporter is automatically configured when running the keycloak role and it's hardcoded in our prometheus configuration. The prometheus endpoint is protected with basic auth configured in the role and the endpoint is hardcoded in our prometheus configuration.
### Gitlab
Gitlab has a built-in prometheus endpoint available which requires a token to access it which can be found [here](https://gitlab.archlinux.org/admin/health_check). The Gitlab endpoint is hardcoded in our prometheus configuration.
### Gitlab runners
Gitlab runners export a [prometheus endpoint](https://docs.gitlab.com/runner/monitoring/), adding them to monitoring:
* Add the host to the `gitlab_runners` group
* Rollout exporter on host: `ansible-playbook playbooks/host.yml -t prometheus_exporters`
### Network monitoring
For http(s)/icmp monitoring [prometheus-black-exporter](https://github.com/prometheus/blackbox_exporter) is used, which currently has alerts configured for https and SSL certificate expiry monitoring. The web endpoints to monitor are configured in `roles/prometheus/defaults/main.yml`.
### Archive monitoring
The [Archive](https://archive.archlinux.org) and its mirrors defined in `archive_mirrors` are monitored using a textcollector which monitors the archive size in bytes.
## Rebuilderd
We host a [Rebuilderd](https://github.com/kpcyrd/rebuilderd) instance on reproducible.archlinux.org which rebuilds Arch packages from repositories defined in `rebuilderd-sync.conf`. Workers automatically connect to the configured rebuilderd instance and query it for work and publish results to the rebuilderd instance.
Results are shown on [our website](https://reproducible.archlinux.org) which is a [rebuilderd-website](https://gitlab.archlinux.org/archlinux/rebuilderd-website) instance.
## Configuration
Setting up rebuilderd-workers requires adding the new machine under `rebuilderd_workers` in hosts
and adding `rebuilderd_workers` with a list of rebuilderd-worker names for example too it's host_vars:
```
rebuilderd_workers:
- repro11
- repro12
```
Then run the rebuilderd-workers playbook.
## Monitoring
The rebuilderd workers and queue are monitored by Prometheus.
## Common commands
Checking rebuilderd-workers status on reproducible.archlinux.org:
```
rebuildctl status
```
Checking rebuilderd queue length:
```
rebuildctl queue ls
```
# Servers
## Table of contents
[[_TOC_]]
## gemini
### Services
- repos/sync (repos.archlinux.org)
- sources (sources.archlinux.org)
- archive (archive.archlinux.org)
## luna
### Services
- mailman
- projects (projects.archlinux.org)
## archlinux.org
### Services
- archweb (Arch's site)
## aur.archlinux.org
### Services
- aurweb
## bugs.archlinux.org
### Services
- flyspray
## bbs.archlinux.org
### Services
- bbs
## phrik.archlinux.org
### Services
- phrik (irc bot) users in the phrik group defined in
the hosts vars and re-used the archusers role. Users
in the phrik group are allowed to restar the irc bot.
## dragon
### Services
- build server
- sogrep
## state.archlinux.org
### Services
- postgres server for terraform state
## quassel.archlinux.org
### Services
- quassel core
## matrix.archlinux.org
### Services
- Matrix homeserver (Synapse)
- Matrix ↔ IRC bridge
## homedir.archlinux.org
### Services
- ~/user/ webhost
## accounts.archlinux.org
This server is _special_. It runs keycloak and is central to our unified Arch Linux account management world.
It has an Ansible playbook for the keycloak service but that only installs the package and starts it but it's configured via a secondary Terraform file only for keycloak `keycloak.tf`.
The reason for doing it this way is that Terraform support for Keycloak is much superior and it's declarative too which is great for making sure that no old config remains in the case of config changes.
So to set up this server from scratch, run:
- `cd tf-stage1`
- `terraform apply`
- `cd ../tf-stage2`
- `terraform import keycloak_realm.master master`
- `terraform apply`
### Services
- keycloak
## mirror.pkgbuild.com
### Services
- Regular mirror.
## reproducible.archlinux.org
[Rebuilderd docs](./docs/rebuilderd.md)
### Services
- Runs a master [rebuilderd](https://reproducible.archlinux.org) instance two workers:
- repro1.pkgbuild.com (packet.net Arch Linux box)
## runner2.archlinux.org
Medium-fast-ish packet.net Arch Linux box.
### Services
- GitLab runner
## mail.archlinux.org
### Services
- postfix (mail server)
- rspamd
- dovecot (imap)
## monitoring.archlinux.org
Prometheus and Grafana server which collects performance/metrics from our services and runs alertmanager.
### Services
- Alertmanager
- [Grafana](https://monitoring.archlinux.org) and [docs/grafana.md](./docs/grafana.md)
- Prometheus
## patchwork.archlinux.org
### Services
- patchwork
## redirect.archlinux.org
### Services
- Redirects (nginx redirects)
## security.archlinux.org
### Services
- security tracker
## wiki.archlinux.org
### Services
- archwiki
## Archive Mirrors
The [Arch Linux Archive](https://archive.archlinux.org) is mirrored to three dedicated servers to help aid global availability.
### Servers
- https://america.archive.pkgbuild.com
- https://asia.archive.pkgbuild.com
- https://europe.archive.pkgbuild.com
......@@ -9,27 +9,60 @@
256 MD5:e7:c7:62:0a:d1:b7:24:62:08:15:73:18:9c:9e:5e:a7 root@archlinux-packer (ED25519)
3072 MD5:9b:04:00:69:a4:f2:62:c3:80:8e:a8:b1:80:ad:a7:9d root@archlinux-packer (RSA)
# apollo.archlinux.org
1024 SHA256:WArxFzvhf5HknYxil2EQSHHRirM2cyjqbtLvhbQAYC8 root@apollo (DSA)
256 SHA256:sYJfY17PE0kJ4K8fbkPK/XqRQjY1+g6hmIF7dvTbZoo root@apollo (ECDSA)
256 SHA256:owwpolkJxPyUmmfJMfFeYIdDXiruwzaEw3bS+q6k97Q root@apollo (ED25519)
2048 SHA256:JW9dUO95gxGJRTkV/V/1HtmLfLq8uztbWc5KAOg8Blc root@apollo (RSA)
1024 MD5:90:46:7f:8e:1e:79:17:10:1e:32:79:a7:69:c6:4b:a4 root@apollo (DSA)
256 MD5:4b:52:61:77:f7:f8:4e:75:ca:83:e6:ae:fc:6e:77:67 root@apollo (ECDSA)
256 MD5:a7:84:8b:95:4f:53:ac:b6:9d:24:79:79:fc:c7:bf:1f root@apollo (ED25519)
2048 MD5:77:b0:17:18:57:74:38:91:47:31:43:04:47:e9:9e:30 root@apollo (RSA)
# america.mirror.pkgbuild.com
1024 SHA256:pycjsXlenFbGqHMp2C5tJZRKJnxCZ1usCux9NOJVTQA root@america.mirror.pkgbuild.com (DSA)
256 SHA256:cmT+nhDEvcuWeZhc5q8GVv6xuxmtS5PkL1ehsilU0C0 root@america.mirror.pkgbuild.com (ECDSA)
256 SHA256:046/o/xSGSruNAEhIMaW2E4a56i7l2jZe13nJADwczo root@america.mirror.pkgbuild.com (ED25519)
3072 SHA256:c/0AQtR2RlNTNI4fHdki6ef7/fWekT87sJ1B8ODHc/Q root@america.mirror.pkgbuild.com (RSA)
1024 MD5:24:c0:a6:27:87:f5:04:c5:e5:89:58:1c:e8:a9:06:9d root@america.mirror.pkgbuild.com (DSA)
256 MD5:86:d5:e0:ed:d7:3e:56:50:0a:92:60:21:53:24:4d:0f root@america.mirror.pkgbuild.com (ECDSA)
256 MD5:4b:0b:1c:81:27:81:7a:22:b4:48:88:75:69:a5:b4:4e root@america.mirror.pkgbuild.com (ED25519)
3072 MD5:a2:41:dc:97:5a:ae:89:7a:4f:69:f7:ec:a0:d4:67:b6 root@america.mirror.pkgbuild.com (RSA)
# archlinux.org
1024 SHA256:7jLDIo/l9ngy+KcC2Yh2yCE+gSVix4VmZVaVTMLOiEg root@archlinux-packer (DSA)
256 SHA256:9nc3jaxyh21w+HVT1Xo0/ujMx7/qWKguqcSiDX7jrA0 root@archlinux-packer (ECDSA)
256 SHA256:nxDSSxkjiccOuzBmqSvsd07WIO/ySIlOMlBxQiTWFaE root@archlinux-packer (ED25519)
3072 SHA256:JrVqWHWZHttME6OE+NNp6ZY+v3rE0W2AwNuZlH8Lghc root@archlinux-packer (RSA)
1024 MD5:57:c1:f0:c8:61:7f:5a:a6:df:ce:10:3c:ee:cb:c1:ad root@archlinux-packer (DSA)
256 MD5:81:86:7f:cf:87:66:59:78:17:a5:c3:03:ad:70:24:9c root@archlinux-packer (ECDSA)
256 MD5:ed:cf:e6:86:fa:8c:96:a2:b4:ce:bd:c3:73:9f:f9:fb root@archlinux-packer (ED25519)
3072 MD5:26:d2:ca:46:64:20:69:1d:f2:e2:80:95:84:c2:9b:7e root@archlinux-packer (RSA)
# asia.mirror.pkgbuild.com
1024 SHA256:NZilDXhhVEFsT7JPcB6APY8HhiO7RgyRMyX3pL+zDik root@archive1.mirror.pkgbuild.com (DSA)
256 SHA256:gMJUYOIH8zdYa1x92WnrlLkxZtTf99Na+ESnZ+Kvk2E root@archive1.mirror.pkgbuild.com (ECDSA)
256 SHA256:aKSZxnj43Q0c3CZ82KOBzV6/I6xH1K0SEg2l3nTpbB4 root@archive1.mirror.pkgbuild.com (ED25519)
3072 SHA256:xJG12dFONxe7TNST9oogoO4nEWprHV2o/92FbPT4E6I root@archive1.mirror.pkgbuild.com (RSA)
1024 MD5:16:e8:82:51:1f:cd:5d:bf:08:13:68:40:37:bc:e0:fa root@archive1.mirror.pkgbuild.com (DSA)
256 MD5:c0:3a:eb:cb:b7:47:52:01:e3:cb:ab:40:94:b3:a4:21 root@archive1.mirror.pkgbuild.com (ECDSA)
256 MD5:f9:3b:1f:ac:be:b6:15:67:07:02:30:48:eb:c0:30:eb root@archive1.mirror.pkgbuild.com (ED25519)
3072 MD5:84:04:71:14:38:34:e0:c4:a3:fa:7c:3f:ee:e2:ed:59 root@archive1.mirror.pkgbuild.com (RSA)
# aur-dev.archlinux.org
1024 SHA256:UPSaOwfVUU5XnBARVikOGxksKlZx48aUyIPjZE9zpAc root@archlinux-packer (DSA)
256 SHA256:b1/sK6szU73jV3XdtoWFgXcSN3FP4QQEBPdw+g0KMro root@archlinux-packer (ECDSA)
256 SHA256:eSsnneEKh60EYqc08//of2SrdbL3tg1y07XSNF25ZwA root@archlinux-packer (ED25519)
3072 SHA256:4yKdHD71M5yxsu2LiLKaOYfFzoFStwgF+HP4stk0/nI root@archlinux-packer (RSA)
1024 MD5:12:c4:cb:12:cf:9e:4d:13:f6:9b:9c:8d:a0:9f:ef:2c root@archlinux-packer (DSA)
256 MD5:5b:c6:00:09:3b:e4:ec:f3:e6:87:a8:0d:ce:69:c7:13 root@archlinux-packer (ECDSA)
256 MD5:fd:d3:a4:31:64:9b:4b:51:b8:89:dc:05:76:a9:49:84 root@archlinux-packer (ED25519)
3072 MD5:0a:58:e7:1e:a1:05:22:8f:c7:f4:2c:c4:ea:91:78:8c root@archlinux-packer (RSA)
1024 SHA256:VzUmG0B+Yb1mrcXVnJI0dMECOgi+7oIwW2PhPvOUhkw root@archlinux-packer (DSA)
256 SHA256:KFBhxP4afMhI5dqHMJwIrbuJ9/EzlHsdXxJ5cwiTMbg root@archlinux-packer (ECDSA)
256 SHA256:sXSwFaVRLiHi1081XoEZg9fE1t0ZOQxWcMocNdWETAA root@archlinux-packer (ED25519)
3072 SHA256:yQCBknJ9mSz2QC1jwWTToGHgtwwT/kgvWKHqWkk/NyQ root@archlinux-packer (RSA)
1024 MD5:c7:7b:16:f0:1d:13:04:2c:49:d6:0a:48:d7:6f:d7:25 root@archlinux-packer (DSA)
256 MD5:ad:6f:76:8c:bb:ca:b0:3d:f4:6d:d4:23:87:c5:98:6e root@archlinux-packer (ECDSA)
256 MD5:9f:97:8b:35:ba:10:3a:4a:b2:c4:6f:0d:e2:77:58:ed root@archlinux-packer (ED25519)
3072 MD5:cc:4b:ef:cb:3f:4a:23:02:6f:c4:25:cc:b4:e1:a0:a8 root@archlinux-packer (RSA)
# aur.archlinux.org
1024 SHA256:kFn1IwQmUEVtiiBLYyShUr/H1614PXs49jM2dXDp5z4 root@archlinux-packer (DSA)
256 SHA256:uTa/0PndEgPZTf76e1DFqXKJEXKsn7m9ivhLQtzGOCI root@archlinux-packer (ECDSA)
256 SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4 root@archlinux-packer (ED25519)
3072 SHA256:5s5cIyReIfNNVGRFdDbe3hdYiI5OelHGpw2rOUud3Q8 root@archlinux-packer (RSA)
1024 MD5:bf:d8:fd:62:91:bc:f0:ab:15:c4:ff:fc:0e:f7:7b:89 root@archlinux-packer (DSA)
256 MD5:22:13:f2:18:8e:d7:b5:a9:35:1f:cb:08:36:32:e6:89 root@archlinux-packer (ECDSA)
256 MD5:f6:38:f7:d3:26:dd:8f:70:fb:7e:59:5b:52:54:5f:d6 root@archlinux-packer (ED25519)
3072 MD5:f7:3e:6c:e7:8d:8e:f3:30:b4:a9:3d:ff:04:1a:65:76 root@archlinux-packer (RSA)
# bbs.archlinux.org
1024 SHA256:8D8LNOrQ4wByBgNJ3n19B7SH7OF1CONh1rU5wbEd53w root@archlinux-packer (DSA)
......@@ -64,6 +97,17 @@
256 MD5:54:23:82:0d:e6:da:6c:d7:09:f1:f0:0f:49:5a:64:04 root@dragon (ED25519)
2048 MD5:e1:e6:4f:72:31:a3:9a:2c:af:e0:0f:53:43:27:6b:df root@dragon (RSA)
# europe.mirror.pkgbuild.com
1024 SHA256:Oq3eikchfo8Wt6AUzWAiU1mDR24rXudJR/zqKBFnrMo root@europe.mirror.pkgbuild.com (DSA)
256 SHA256:3S0HuO72jHUUrPM8BjfcjsB0FNXkubxovc7Sm5jZBjc root@europe.mirror.pkgbuild.com (ECDSA)
256 SHA256:aqnPnq4WG/3xNuKOJlsuCGgPiH0RWavcQi/n/HO9h6Y root@europe.mirror.pkgbuild.com (ED25519)
3072 SHA256:cJGscbI/w0iINNBpU+Q6jLtSlF2Y3hLPs/By8CzX4tM root@europe.mirror.pkgbuild.com (RSA)
1024 MD5:f3:da:87:c4:b2:bc:da:be:1c:ce:a3:73:3c:da:ff:f4 root@europe.mirror.pkgbuild.com (DSA)
256 MD5:44:ef:66:dc:e2:68:86:69:ad:74:22:a4:92:c6:5b:e1 root@europe.mirror.pkgbuild.com (ECDSA)
256 MD5:bd:af:e2:cb:6b:fe:b6:60:73:b0:ba:7b:db:af:21:b7 root@europe.mirror.pkgbuild.com (ED25519)
3072 MD5:57:a2:59:db:c7:07:4f:ac:91:9a:f8:db:7f:16:a7:d4 root@europe.mirror.pkgbuild.com (RSA)
# gemini.archlinux.org
1024 SHA256:F1Corf6i2U72yub+CIzzGHLOMVKVnjALh1YHM8gBjxE root@gemini.archlinux.org (DSA)
256 SHA256:If51DkTftUpDAFz65totgDfTd/ddu/2w/RBZIHtY74U root@gemini.archlinux.org (ECDSA)
......@@ -75,17 +119,6 @@
256 MD5:44:f5:60:54:d7:a7:b7:6d:fd:69:35:05:8f:4e:a5:0f root@gemini.archlinux.org (ED25519)
3072 MD5:20:2f:93:37:ae:33:e6:3e:9f:74:b6:57:c9:f3:58:9e root@gemini.archlinux.org (RSA)
# ger.mirror.pkgbuild.com
1024 SHA256:5yrR2CAO4IoA5zK/u/2bNXV/vjJCdDIou4cHrPy6GA4 root@archlinux (DSA)
256 SHA256:7MwUKiox99hR1EEfpK6ZWtrhJ54klqCG9Y5NR8ZPfJY root@archlinux (ECDSA)
256 SHA256:eJo6fJCfC0Uj1u5v4UAXNuXStyv+k9Mt0Cc7iJyricc root@archlinux (ED25519)
2048 SHA256:5IMb6Dawsj68orPeRS+y9TgkodNu2auYXjj2DyrfopY root@archlinux (RSA)
1024 MD5:55:9f:14:8e:ad:1e:0d:2d:08:19:e6:cb:ad:d2:c5:92 root@archlinux (DSA)
256 MD5:b9:d2:ea:ea:0e:7b:dd:db:b6:bb:59:e4:d0:d7:aa:a4 root@archlinux (ECDSA)
256 MD5:4a:01:62:ce:4c:f6:3a:93:24:7b:e2:d3:b7:ea:d3:f8 root@archlinux (ED25519)
2048 MD5:29:c9:81:09:23:03:1c:26:af:50:9c:4a:fa:b5:da:29 root@archlinux (RSA)
# gitlab.archlinux.org
1024 SHA256:mbHLMb1i7JBTytqWxpu8bWz7suiSNXpuTdla6/l6yK8 root@archlinux-packer (DSA)
256 SHA256:0OBfrHiu/X7HcECLaOQFY3XElaiH3qxcltK6kjH9PRI root@archlinux-packer (ECDSA)
......@@ -119,6 +152,17 @@
256 MD5:1d:92:08:da:8e:a1:fb:1c:c5:65:00:c8:15:a4:87:32 root@alderaan.archlinux.org (ED25519)
2048 MD5:c4:7f:00:d4:5e:c7:23:45:97:bb:40:ec:15:ce:7c:a9 root@alderaan (RSA)
# mail.archlinux.org
1024 SHA256:/d3MC4NoQbPSNgNebFyzNCze4HVHPhITVWy9vWdZUp4 root@archlinux-packer (DSA)
256 SHA256:IbQnu28PPf6iZnr6DPwzITD4o2DznYMO6j0mkjZXasE root@archlinux-packer (ECDSA)
256 SHA256:O+88oCLCsdC0DWs6TY7IABiPRyrnh60XUPIzFRSatqE root@archlinux-packer (ED25519)
3072 SHA256:9+28nPjF/dqmWnwuubJ3/9qLERhNTK6Kewj5XvoXPOk root@archlinux-packer (RSA)
1024 MD5:6e:0f:bb:1f:a8:78:5b:b4:48:df:c6:ae:6b:41:4b:03 root@archlinux-packer (DSA)
256 MD5:14:36:a5:f5:92:18:b6:c2:7e:20:30:e7:12:db:8e:d3 root@archlinux-packer (ECDSA)
256 MD5:dd:20:c1:f1:f2:fa:70:86:3a:e2:39:86:b1:01:2f:61 root@archlinux-packer (ED25519)
3072 MD5:b6:14:30:bd:fe:43:46:6a:20:a2:8b:b0:aa:d4:35:19 root@archlinux-packer (RSA)
# mailman3.archlinux.org
1024 SHA256:Vs/PxyU74qe6uR5EUUMWhDLA+B8lBQO2PEbRSmZwzYA root@archlinux-packer (DSA)
256 SHA256:ARXQTmcvjHISznthbjI04GBOUEuQAIT2v/fRdAg3Zqw root@archlinux-packer (ECDSA)
......@@ -130,6 +174,17 @@
256 MD5:91:95:e9:e2:1f:17:24:66:10:ae:29:ea:90:41:d9:fb root@archlinux-packer (ED25519)
3072 MD5:97:9f:77:0e:f5:99:44:f3:ab:db:4b:f4:4a:98:cd:dc root@archlinux-packer (RSA)
# man.archlinux.org
1024 SHA256:11C7Qa1GSNBBspSlber3Sp+LEMRpfr/VWkypfu6OnhA root@archlinux-packer (DSA)
256 SHA256:fL79NVaEiwXGfUhTXWLkue/D1seSADYbui+jwQ2dvW0 root@archlinux-packer (ECDSA)
256 SHA256:qnuyQJXOuk5VuN7xfainNcgyAzCc1rKjYKyTKvEd0HE root@archlinux-packer (ED25519)
3072 SHA256:mI+a0Bi94vDqlXC8jPQToFriA9NwB2YkKsVtcjFceUE root@archlinux-packer (RSA)
1024 MD5:68:9b:b0:97:76:d0:71:28:10:c1:ea:d0:1a:f7:1d:99 root@archlinux-packer (DSA)
256 MD5:23:b4:2d:ff:10:b1:80:43:52:d0:8d:9f:ae:dd:36:0d root@archlinux-packer (ECDSA)
256 MD5:d1:af:34:47:0c:90:9d:d7:fb:fd:47:e1:b3:97:ac:9b root@archlinux-packer (ED25519)
3072 MD5:56:0e:71:f1:5f:73:7b:e9:0e:b8:06:60:03:ec:a0:52 root@archlinux-packer (RSA)
# matrix.archlinux.org
1024 SHA256:4xl3Vzj2VTffMV6zCiAx0DSrsYIBmMnWo41kjR4ZWUo root@archlinux-packer (DSA)
256 SHA256:+v4KFzSadzQmENY2HvHpn8Zse0opJc7FaixR7/K3y0Y root@archlinux-packer (ECDSA)
......@@ -163,16 +218,16 @@
256 MD5:fe:a1:ab:4d:f6:5d:76:f9:a3:99:be:fd:51:ee:77:ed root@archlinux-packer (ED25519)
3072 MD5:ad:ee:a6:6d:b7:9b:f0:f7:78:9f:df:b4:53:2e:5f:9f root@archlinux-packer (RSA)
# orion.archlinux.org
1024 SHA256:Y7XP+fTQZAEDgmCHuSqFc0MmNUmCPJYRZs/7iq6viK8 root@mnt (DSA)
256 SHA256:2gH/IGaZ/pOnpt4+VY0twd4+hUOraUWRceJiNQxnbxs root@mnt (ECDSA)
256 SHA256:G4mz3jsK8XZymCDjUE7TKhA3Kz/eC+q4gHlnhCWyVB4 root@mnt (ED25519)
2048 SHA256:PxFPKc82M5wShxNX+62FmZPKJBACz4n7epevqEDOUUw root@mnt (RSA)
# patchwork.archlinux.org
1024 SHA256:FRn2yrB4ABYzLFs9muVWw/4PRq3ahNoV0EnweAip87g root@archlinux-packer (DSA)
256 SHA256:Zmox+d8LXNaUo4d8ywmPXTHyjV7mCCAFYycEZd73c6w root@archlinux-packer (ECDSA)
256 SHA256:xdY/tDi/x0z2SDTxzCrrt+Oygn5IYI0kMyN9JlPesw4 root@archlinux-packer (ED25519)
3072 SHA256:erHz5RJTWqaKPUCB/bZMBxQfXQIi23rrrOOTPH+KUtc root@archlinux-packer (RSA)
1024 MD5:67:a7:23:42:0c:22:74:30:ea:e2:89:4a:68:8c:a7:d6 root@mnt (DSA)
256 MD5:47:ce:6f:89:fa:06:ab:d5:94:e1:e1:95:94:40:68:5c root@mnt (ECDSA)
256 MD5:95:53:ec:52:c3:78:e8:5d:43:c6:2f:bc:d9:7e:9a:4c root@mnt (ED25519)
2048 MD5:ff:9d:c3:b0:ee:c9:89:32:72:0c:d8:fb:cc:5d:ae:75 root@mnt (RSA)
1024 MD5:44:f4:b3:50:0e:ef:bb:ee:b6:85:06:b6:fb:bd:02:6f root@archlinux-packer (DSA)
256 MD5:6c:f2:1b:97:21:77:f2:b2:03:e7:9a:8f:aa:0d:c6:6c root@archlinux-packer (ECDSA)
256 MD5:6b:03:f5:0e:0f:4c:50:34:7a:19:f8:28:48:5b:93:09 root@archlinux-packer (ED25519)
3072 MD5:0a:03:c6:ef:a1:9d:24:a7:59:ce:bb:41:87:6a:e0:af root@archlinux-packer (RSA)
# phrik.archlinux.org
1024 SHA256:+482UWH5/pSMZ8VoIgkGZxGOm1tZ72rI5RrZsnQHDVk root@archlinux-packer (DSA)
......@@ -196,16 +251,38 @@
256 MD5:15:45:eb:91:69:df:c3:6d:9f:99:b9:13:02:94:a6:ac root@archlinux-packer (ED25519)
2048 MD5:ca:2f:cf:5c:4d:ec:75:c3:71:76:d6:b7:b9:fa:aa:32 root@archlinux-packer (RSA)
# repro1.pkgbuild.com
1024 SHA256:L16HBYUCY04dcc0A+pSZBYwktwhFkk7jZIyKWlEAByA root@archlinux (DSA)
256 SHA256:C7dfr0woXiasu5JcW0nXLA4/d2ERePlqNs4+ezuRkR4 root@archlinux (ECDSA)
256 SHA256:jNrWFp/hSt22kyRAoRAYvga2YG+6aS0DEIDCPbIxcTY root@archlinux (ED25519)
2048 SHA256:2t7LJ6uSQpk6eVhivIu1c5qw659wNKXCXOn0wVC7tEc root@archlinux (RSA)
# redirect.archlinux.org
1024 SHA256:hqw3Wmif3BUI9VLcNnvcB3I+M9f5OUtDjRT8H6tAuEU root@archlinux-packer (DSA)
256 SHA256:JaUkz0eOofslq9BVifMx8c6sapM/DSig9zrVyFqrHD4 root@archlinux-packer (ECDSA)
256 SHA256:sUcgzScFlMByQKLW2IDYBc2m6EvLXzM6KVa2mzls3TA root@archlinux-packer (ED25519)
3072 SHA256:yUn8pVpioFsltzFKA2cImHb6UnD63pCOCiJsP5OFLBQ root@archlinux-packer (RSA)
1024 MD5:d0:82:2d:aa:c0:ac:b9:07:e7:b8:91:0d:92:b2:ae:38 root@archlinux (DSA)
256 MD5:c1:40:f1:c7:cd:c0:08:bd:dc:a0:01:66:f3:ef:a0:d4 root@archlinux (ECDSA)
256 MD5:9c:47:85:f4:e4:19:42:92:7a:b1:dd:de:f5:3e:74:d6 root@archlinux (ED25519)
2048 MD5:e4:e1:d2:00:1b:96:9c:60:60:ac:a1:32:97:3b:1d:7c root@archlinux (RSA)
1024 MD5:a8:f9:dd:2a:79:ca:3a:ef:b5:24:49:6b:61:1f:bb:07 root@archlinux-packer (DSA)
256 MD5:b1:f5:78:51:c5:50:5e:25:73:68:fc:80:53:25:94:ba root@archlinux-packer (ECDSA)
256 MD5:5a:49:d5:f3:00:ca:49:17:d8:cc:3e:84:1d:60:be:06 root@archlinux-packer (ED25519)
3072 MD5:1e:52:48:56:d3:13:20:e5:02:4f:10:1b:af:27:e5:c7 root@archlinux-packer (RSA)
# repro1.pkgbuild.com
1024 SHA256:K2RjAgIzlRrSkqdf3vqwfXOOg0oEMt/AwAT2Gmt2wpA root@repro3.pkgbuild.com (DSA)
256 SHA256:H/7en8S/UqQ+llIDPyCIn9sYHjiEU6L+Myu0MpmoDsE root@repro3.pkgbuild.com (ECDSA)
256 SHA256:NaiYaen6WM78LFaRn4MUnb450sWjiac0rZjy84CTd+U root@repro3.pkgbuild.com (ED25519)
3072 SHA256:lsXZROhzM0y5kxQ6cBnbm7IGFN+Um3cjqhZGJt36cHM root@repro3.pkgbuild.com (RSA)
1024 MD5:a4:45:6e:b1:92:5d:c0:c4:1e:22:3f:13:ef:3a:24:16 root@repro3.pkgbuild.com (DSA)
256 MD5:23:e1:f9:16:1f:17:f2:c5:7e:7a:28:36:33:3c:79:d5 root@repro3.pkgbuild.com (ECDSA)
256 MD5:15:31:37:f4:f2:e5:93:e7:28:8f:e2:db:6e:6d:24:e2 root@repro3.pkgbuild.com (ED25519)
3072 MD5:d7:f7:8e:0b:bf:8f:58:83:33:ba:c8:22:4b:3c:50:96 root@repro3.pkgbuild.com (RSA)
# repro2.pkgbuild.com
1024 SHA256:sppthtBQD60z8f0bDUnoMUesg55M7/ez4qGXVUUDtRQ root@repro2.pkgbuild.com (DSA)
256 SHA256:enqq08K6vQV8CcISu1upR3Ooa63HD6Z+PtRzMVArnTk root@repro2.pkgbuild.com (ECDSA)
256 SHA256:CA71k+BRGrEEcLLVKqtUBU55th2W12Emq/x++zGtoH0 root@repro2.pkgbuild.com (ED25519)
3072 SHA256:zQy/zasnSYXF5h863hxxjKy7xqw2HifboYGBb59g9Vg root@repro2.pkgbuild.com (RSA)
1024 MD5:2b:3b:9a:9f:b0:fc:d4:20:8b:21:67:bf:f7:a4:c8:e7 root@repro2.pkgbuild.com (DSA)
256 MD5:95:c1:25:1a:b3:46:f3:d2:9f:19:21:02:9b:e7:5f:9e root@repro2.pkgbuild.com (ECDSA)
256 MD5:21:76:73:3b:ac:30:6d:f5:a5:f6:52:2e:13:dc:b4:cb root@repro2.pkgbuild.com (ED25519)
3072 MD5:12:70:8a:d4:ef:a9:43:6e:6b:53:46:71:f7:96:ec:fb root@repro2.pkgbuild.com (RSA)
# reproducible.archlinux.org
1024 SHA256:3HoA8rGGureKWKaIZst+Dc6f7yrf3Wfn5PO1HFMl35E root@archlinux-packer (DSA)
......@@ -219,26 +296,26 @@
3072 MD5:42:0a:57:89:2a:ae:e6:c5:c1:ae:6a:a2:bd:3d:5f:dc root@archlinux-packer (RSA)
# runner1.archlinux.org
1024 SHA256:v11IkBs2iNsgfaOOXoY4Yzk+UZohGS8RzApHmKKA6tM root@archlinux (DSA)
256 SHA256:p4wjp/S6+nO9lyoPvpNc4sY97jJg6RXvYW3ozBe3yco root@archlinux (ECDSA)
256 SHA256:YXm13437g8Dn1H0/5UZVs9FQuInHuae8JQaaMxu5dNM root@archlinux (ED25519)
2048 SHA256:fT3bmQCw3h2QOGWz9HAfOxNZu7NDAM1dex2k75rU6n0 root@archlinux (RSA)
1024 SHA256:TgWGBKCSo3rkdYuP4mJzGtH+Ucub8dmEJXnnzrhUSwM root@84.17.49.250 (DSA)
256 SHA256:2OEO/Pv1OxCLaVg9r8wHzTR7V3YTWEw//g7LJgIBMGs root@84.17.49.250 (ECDSA)
256 SHA256:nXpZ/R3IuOEAasoIvCCTkdMlyNjCnWVTvQBh6xvycIg root@84.17.49.250 (ED25519)
3072 SHA256:Ii5JwUoEKXV37rdjnsts4k3ppbEX+GHYjBC9mygdwY4 root@84.17.49.250 (RSA)
1024 MD5:0d:40:31:b2:00:ab:aa:24:c0:79:57:f0:ba:53:99:72 root@archlinux (DSA)
256 MD5:57:7c:bf:8a:2e:d5:e4:4e:e2:13:33:47:ab:e1:13:6e root@archlinux (ECDSA)
256 MD5:7f:a9:c8:d7:af:e4:b4:7f:fa:ba:7f:09:85:b0:13:23 root@archlinux (ED25519)
2048 MD5:4e:53:05:87:1b:98:8b:d0:6c:af:63:97:94:04:aa:ba root@archlinux (RSA)
1024 MD5:31:f6:9c:50:d2:80:f1:fa:b5:50:c5:06:3a:9e:80:cc root@84.17.49.250 (DSA)
256 MD5:ad:cb:d6:53:cb:42:8d:33:26:99:33:fd:c9:19:a0:ea root@84.17.49.250 (ECDSA)
256 MD5:c1:7f:43:ad:dc:69:17:b4:33:df:bb:94:00:22:33:5a root@84.17.49.250 (ED25519)
3072 MD5:da:9f:ec:c4:f3:c1:63:6e:ca:f4:9b:ac:32:23:d9:74 root@84.17.49.250 (RSA)
# runner2.archlinux.org
1024 SHA256:nZ+E6sBUchtBDhBt2BKapEj9+82/56G8KIUadCXFW6s root@runner2.archlinux.org (DSA)
256 SHA256:vj8lRTIbehnv1jVZwSkU30kCzLMcIJgHuReLNCLfS1Y root@runner2.archlinux.org (ECDSA)
256 SHA256:PvjRyx7S+1bVkNXT+ra7gpEsDc171H2e6Ng6I/eL2AM root@runner2.archlinux.org (ED25519)
2048 SHA256:nfJ1ifz/iHYSWkN+M1iLCEBHUIf0fssy4GRzNh3KK3M root@runner2.archlinux.org (RSA)
1024 SHA256:tK3XeUrt/ahlj1fdl5tQgeVXticllUIaewK2y86Jvzc root@runner2.archlinux.org (DSA)
256 SHA256:iJGAgTQWSQk97dlNpkBMEW3peysCDnNFFZO5TyEguvs root@runner2.archlinux.org (ECDSA)
256 SHA256:x9wziZt9b8qP7jgGPwlXAHI3aGC72xeoBg3pabGFhC0 root@runner2.archlinux.org (ED25519)
3072 SHA256:D5gonGPebFRludfXdD1QNkb9CFE1/7hWWIKgq26b8pw root@runner2.archlinux.org (RSA)
1024 MD5:b7:cd:1a:bc:cc:a0:69:57:56:55:45:0c:5d:d4:50:75 root@runner2.archlinux.org (DSA)
256 MD5:90:93:75:65:f4:db:73:f0:f7:9f:62:9e:5a:42:9d:fb root@runner2.archlinux.org (ECDSA)
256 MD5:38:8e:ea:b6:e5:f7:ea:25:c3:e1:92:01:5b:5c:ff:5b root@runner2.archlinux.org (ED25519)
2048 MD5:15:6c:a7:5a:a8:32:97:9f:69:a9:a7:c0:00:da:03:2f root@runner2.archlinux.org (RSA)
1024 MD5:20:b1:f6:81:e8:ff:3e:f4:01:ab:b5:6d:42:87:7a:2d root@runner2.archlinux.org (DSA)
256 MD5:60:3c:e7:96:b4:38:0a:96:b5:82:3f:ba:5d:3b:8d:17 root@runner2.archlinux.org (ECDSA)
256 MD5:84:2a:9a:5b:41:d9:a6:d5:94:5b:29:16:dc:12:46:3c root@runner2.archlinux.org (ED25519)
3072 MD5:01:5f:4b:37:f7:da:4c:56:8d:d8:c1:6c:a9:0e:33:fb root@runner2.archlinux.org (RSA)
# secure-runner1.archlinux.org
1024 SHA256:9R7X3mEZFVnTChSgjX2TKu50/+oyeQSiR2dkdBgl6+4 root@secure-runner1.archlinux.org (DSA)
......@@ -251,6 +328,17 @@
256 MD5:ae:29:1d:79:35:95:58:df:fe:0b:3a:d7:0e:78:21:83 root@secure-runner1.archlinux.org (ED25519)
3072 MD5:e2:d2:34:cb:4d:d4:03:da:02:00:14:79:25:03:0c:00 root@secure-runner1.archlinux.org (RSA)
# security.archlinux.org
1024 SHA256:Z9lziuoL5tom8LWYSyf7hWntrjPW9LtDusL7NNmQuGM root@archlinux-packer (DSA)
256 SHA256:vIZnpAn/xjyw0tHPImNWvpEf27FaaGVVfvc7PQBpQHQ root@archlinux-packer (ECDSA)
256 SHA256:Ly8nOHcI1YL0XHZLVk0nznT3ReISvLNRG2oNYCnnpd4 root@archlinux-packer (ED25519)
3072 SHA256:xrzF3yYdzkzMZzK8AKrs8Bkk+MglQdDSOSJ8phrLQW0 root@archlinux-packer (RSA)
1024 MD5:44:bd:32:11:bc:ce:21:de:eb:ed:d9:70:9e:0a:2a:e4 root@archlinux-packer (DSA)
256 MD5:f7:aa:5b:b2:2c:49:3c:03:9c:35:c7:5f:4d:50:52:a6 root@archlinux-packer (ECDSA)
256 MD5:9e:30:b4:b8:91:f0:e2:4c:ff:c5:54:9a:73:b0:17:76 root@archlinux-packer (ED25519)
3072 MD5:bf:05:ec:33:54:26:58:51:a8:20:a5:c2:35:55:f8:bd root@archlinux-packer (RSA)
# state.archlinux.org
1024 SHA256:4oNX8CksPEgIzibu+ETa2OVVPBX2pzcvcVUa60NbHiQ root@archlinux-packer (DSA)
256 SHA256:uR7EDdVrvkZf43eNmumOeu2MeZn4oMB39ad9kHoobkk root@archlinux-packer (ECDSA)
......@@ -262,7 +350,7 @@
256 MD5:2b:7f:a8:75:ef:38:e3:c3:f7:2e:ea:9e:73:fd:3e:d5 root@archlinux-packer (ED25519)
2048 MD5:f8:a9:75:e2:99:4f:ae:2b:70:72:a2:ae:9e:fb:f1:a2 root@archlinux-packer (RSA)
# svn2gittest
# svn2gittest.archlinux.org
1024 SHA256:R5uXRsoC0CXMxAE+dV6Ola8K1amyK84VFknjM9QgonI root@archlinux-packer (DSA)
256 SHA256:CVc0FXXE1DY1wmwoHCseGg5TnzYOgbf6adTbgstVWx0 root@archlinux-packer (ECDSA)
256 SHA256:kcuaxYVB/oCraE0q+ZsnUeozpVJYYDZ24tW5MEObj1E root@archlinux-packer (ED25519)
......@@ -273,3 +361,14 @@
256 MD5:9a:97:48:f7:11:b3:32:ba:fa:ab:9f:0c:41:41:da:e4 root@archlinux-packer (ED25519)
3072 MD5:f3:11:d6:58:f9:32:d1:34:fa:4e:d9:e3:d7:c8:6b:f2 root@archlinux-packer (RSA)
# wiki.archlinux.org
1024 SHA256:MnCkxFpWB/mTDRHPVB4RLuSPMNfPQyotpFaWuc55DCk root@archlinux-packer (DSA)
256 SHA256:26K98Dg4laIWFt++vxGPiANR6w+AvxgQUTb1TzeLilY root@archlinux-packer (ECDSA)
256 SHA256:rRzytaydRgwVjifkE+QURI9ezl9JnRRjmXMjLKfzPO4 root@archlinux-packer (ED25519)
3072 SHA256:kvE+19HTCY7D3ZdVN/VpPIKJywe3zE27H2Me98NMmq8 root@archlinux-packer (RSA)
1024 MD5:c1:f7:eb:89:35:8f:1c:3a:8d:13:5a:fc:94:4e:83:12 root@archlinux-packer (DSA)
256 MD5:66:2d:77:84:ad:e4:9d:ef:2e:5e:50:41:f3:67:f1:f6 root@archlinux-packer (ECDSA)
256 MD5:9b:ea:d8:3a:1a:54:48:36:f5:90:06:b8:10:f7:62:0f root@archlinux-packer (ED25519)
3072 MD5:59:c0:3d:76:36:73:87:f6:f6:37:64:17:0f:ea:8c:7b root@archlinux-packer (RSA)