infrastructure issueshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues2022-10-22T23:50:48Zhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/50Modernize mail server setup2022-10-22T23:50:48ZSven-Hendrik Haasesvenstaro@archlinux.orgModernize mail server setup### Prelude
- [x] Switch SPF to softfail (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/6278f6688a5c4ecdf74a4b1fd4b5bbfa69ce4b01)
- [x] Remove Postgrey (!43)
- [x] Setup SPF for HELO name ([RFC 7208 section 10.1.3](https...### Prelude
- [x] Switch SPF to softfail (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/6278f6688a5c4ecdf74a4b1fd4b5bbfa69ce4b01)
- [x] Remove Postgrey (!43)
- [x] Setup SPF for HELO name ([RFC 7208 section 10.1.3](https://tools.ietf.org/html/rfc7208#section-10.1.3)) (!122)
- [x] Switch to Rspamd (!42)
- [x] Use Rspamd DKIM signing module for signing instead of OpenDKIM (#213, !147)
- [x] Stop relaying of luna via mail.
- [ ] create main opensmtpd config (#215)
- [ ] create opensmtpd config for relayhosts (#216)
- [ ] Prepare virtual user setup for dovecot and OpenSMTPD (#214)
- [ ] Store the (virtual) mail password in keycloak (#217)
- [x] harden used IMAP and SMTP ports ([RFC 8314](https://tools.ietf.org/html/rfc8314), #219)
- [x] migrate existing services to use implicit TLS for SMTP Submission (!207)
- [ ] store alias and sender file in Ansible (encrypted in the vault)
- [ ] store keycloak UUID -> arch mail address mapping in ansible (encrypted in the vault)
- [ ] write export tool to automatically pull password hashes from keycloak (#218)
- [ ] create keycloak client with minimal permissions for the export tool (https://gitlab.archlinux.org/archlinux/mail-credential-syncer/-/issues/3)
- [x] Setup MTA-STS in testing mode and SMTP TLS Reporting ([RFC 8460](https://tools.ietf.org/html/rfc8460)) (!191, !231)
- [x] Setup monitoring (!206)
- [ ] https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/365
### Main part
- [ ] Rollout the export tool from #218
- [ ] Replace Postfix by OpenSMTPD on our relaying hosts
- [ ] Switch Dovecot to virtual users
- [ ] Replace Postfix by OpenSMTPD on our main mail server
### Aftermath
- [ ] Switch DMARC to reject
- [ ] Add archlinux.org to rspamd whitelistes: https://github.com/rspamd/maps/tree/master/rspamd (`spf_dkim` + `dmarc`)
- [ ] Setup ARC
- [ ] Cleanup OpenSMTPD and Dovecot config if possible
- [ ] remove ssh access for users on the mail host
- [x] Cleanup SPF record (#197, !229)
- [x] Deprecate STARTTLS on Port 587 (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/0ae67c4a64e2892eef58285f39e9575bdbe38268)
- [ ] use floating IPs to keep the spam reputation case we need to migrate the mail server
- [x] Remove old ip addresses from DNSWL
- [ ] Process DMARC and TLS-RPT reports automatically (#241)
- [x] Switch MTA-STS to enforce mode (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/0b87cbfd062d31814c877614a5c2e388ee5eb416)
- [ ] Setup blacklist monitoring
- [ ] Setup mails sent, received, bounced monitoring
- [x] Deprecate POP3 (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/cf9c92fd346a6f832e5057e305759c35d59692e8)
---
**removed Tasks**
- [ ] Pull users from Keycloak ([somehow](https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/50#note_2248))
- [ ] Dovecot: Switch passdb from [pam](https://doc.dovecot.org/configuration_manual/authentication/pam/) to [passwd-file](https://doc.dovecot.org/configuration_manual/authentication/passwd_file/)
**Original description:**
Our mail server isn't ansibled and it's fairly opaque how everything is setup. Ansible the whole thing, put it on a separate box and modernize it in the process.
Some guides to follow along with a similar stack: https://prefetch.eu/blog/2020/email-server/ and https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/
There's also this: https://wiki.dovecot.org/PasswordDatabase/oauth2
@foxboron mentioned that perhaps OpenSMTPD is inadvisable due to its fairly bad security track record.Frederik SchwanKristian KlausenFrederik Schwanhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/14Use Prometheus to monitor Gitlab2022-01-04T00:27:02ZJelle van der WaaUse Prometheus to monitor GitlabSetup Prometheus and configure it to pull data from Gitlab. Investigate how we can set this up in a secure fashion either with a VPN or HTTPS and firewalld rules.
Also monitor docker with Prometheus as Gitlab runs on Docker.
* [ ] Moni...Setup Prometheus and configure it to pull data from Gitlab. Investigate how we can set this up in a secure fashion either with a VPN or HTTPS and firewalld rules.
Also monitor docker with Prometheus as Gitlab runs on Docker.
* [ ] Monitor GitLab Artifact Size
* [ ] Monitor Gitlab Number of Projects
* [ ] Monitor Gitlab Number of Users
* [ ] Monitor Gitlab Number of Namespaces
* [ ] Monitor Gitlab Project Repository Size, Docker size, Artefact Size (node-exporter to monitor Gitlab directories?)
* [x] Monitor GitLab Runners - too many jobs queue / overloaded runners
https://docs.gitlab.com/ee/administration/monitoring/prometheus/https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/23Setup monitoring for keycloak2020-09-09T19:26:27ZJelle van der WaaSetup monitoring for keycloak* General host monitoring
* Investigate keycloak monitoring options
- [x] package https://github.com/aerogear/keycloak-metrics-spi
- [x] write ansible task for exporter
- [ ] setup prometheus alerts
- [x] setup [grafana dashboard](https...* General host monitoring
* Investigate keycloak monitoring options
- [x] package https://github.com/aerogear/keycloak-metrics-spi
- [x] write ansible task for exporter
- [ ] setup prometheus alerts
- [x] setup [grafana dashboard](https://grafana.com/grafana/dashboards/10441)Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/17Investigate giving out SSH access via Keycloak2021-08-02T13:37:22ZJelle van der WaaInvestigate giving out SSH access via KeycloakAs we are moving to SSO, we want ssh keys to be handled by keycloak as this will help us with onboarding new folks and removing access when a user.
#### Requirements:
* Restrict ssh keys (with certain keysize/algo)
* Allow Developers/TU...As we are moving to SSO, we want ssh keys to be handled by keycloak as this will help us with onboarding new folks and removing access when a user.
#### Requirements:
* Restrict ssh keys (with certain keysize/algo)
* Allow Developers/TU's to upload their keys (and no one else)
#### Potential solutions
* [pam oidc](https://github.com/CyberDem0n/pam-oauth2)
* [smallstep](https://smallstep.com/certificates/)
* [teleport](https://github.com/gravitational/teleport)
* [vault](https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates)
### Uploading keys
For solutions such as PAM OIDC and vault, we need to provide our own ssh keys
* [extend keycloak profile](https://wjw465150.gitbooks.io/keycloak-documentation/content/server_development/topics/custom-attributes.html)
* [separate app which uses the keycloak API](https://keycloak.discourse.group/t/update-user-profile-through-api-without-admin-rights/1457)
#### Note
Keep root ssh keys as is for now, so we can log in when keycloak fails to start on a reboot. (Note hetzner VPS'es allow login via the web console)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/25Migrate archweb to Keycloak2020-08-06T21:04:54ZJelle van der WaaMigrate archweb to KeycloakAdd keycloak openid/saml support in archweb and connect it to keycloak.
Groups on archweb:
* Developer
* Trusted Users
* Mirror administrator
* Testers
* Release engineering
* Retired Support Staff
* Support Staff
* Retired Developers
*...Add keycloak openid/saml support in archweb and connect it to keycloak.
Groups on archweb:
* Developer
* Trusted Users
* Mirror administrator
* Testers
* Release engineering
* Retired Support Staff
* Support Staff
* Retired Developers
* Retired Trusted Users
Additionally there are some special permissions for some groups:
* everyone from the DevOps team get's Staff status and Superuser status on archweb
* everyone from Mirror maintainers/administrators get's Staff status.
Furthermore we want to be able to write customer user attributes in keycloak via OIDC/SAML, so this must be supported.AndrewAndrewhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/31Setup a prometheus box with alertmanager2020-12-22T14:35:02ZJelle van der WaaSetup a prometheus box with alertmanagerSetup a box with prometheus and alertmanager ( requires mail setup ). For collecting our monitoring data from gitlab and other sources:
## Systemd ( packaged )
* failed systemd unit
* CPU resource accounting research
https://github.co...Setup a box with prometheus and alertmanager ( requires mail setup ). For collecting our monitoring data from gitlab and other sources:
## Systemd ( packaged )
* failed systemd unit
* CPU resource accounting research
https://github.com/prometheus/node_exporter
( not sure if it does cpu accounting)
## Prometheus-postgresql-exporter
- [ ] package exporter
- [ ] add ansible role for enabling exporter when postgresql_servers
- [ ] setup alerts for postgresql
## Prometheus-mysql-exporter
- [x] package exporter
- [x] add ansible role for enabling exporter when mysql_servers
- [ ] setup alerts for mysql
- [ ] setup [grafana dashboard](https://devconnected.com/complete-mysql-dashboard-with-grafana-prometheus/) [percona dashboards](https://github.com/percona/grafana-dashboards)
## Nginx
* 200,400,500's etc. (parses logs)
Use loki instead, with [alerting](https://www.infracloud.io/blogs/grafana-loki-log-monitoring-alerting/)
## Memcached - packaged
* Cache/hit ratio's etc
prometheus-memcached-exporter
## General
CPU/Load status
https://github.com/prometheus/node_exporter
## UWSGI
Not monitored but status are exported, needs packaging
## Btrfs
Btrfs errors can be monitored using a [custom textcollector](https://github.com/prometheus-community/node-exporter-textfile-collector-scripts/blob/master/btrfs_stats.py)Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/34Move utilities out of infrastructure.git2020-09-09T19:13:27ZJelle van der WaaMove utilities out of infrastructure.gitWe have several utilities which live in the repository but should be moved to their own repo or packaged in Arch Linux
Under roles/archbuild/files:
- [ ] gitpkg
- [ ] diffpkg
- [ ] diffrepo
- [ ] pkgdiffrepo
- [x] A git submodule for *...We have several utilities which live in the repository but should be moved to their own repo or packaged in Arch Linux
Under roles/archbuild/files:
- [ ] gitpkg
- [ ] diffpkg
- [ ] diffrepo
- [ ] pkgdiffrepo
- [x] A git submodule for **checkservices** which should be packaged in Arch Linux.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/38Migrate the wiki to keycloak2024-03-13T15:02:24ZJelle van der WaaMigrate the wiki to keycloakIn the future we want to use keycloak for authentication mediawiki:
- https://www.mediawiki.org/wiki/Extension:OpenID_Connect
- https://stackoverflow.com/questions/16893589/prevent-users-from-changing-their-passwords-in-mediawiki
- http...In the future we want to use keycloak for authentication mediawiki:
- https://www.mediawiki.org/wiki/Extension:OpenID_Connect
- https://stackoverflow.com/questions/16893589/prevent-users-from-changing-their-passwords-in-mediawiki
- https://wiki.archlinux.org/index.php/ArchWiki:Access_levels_and_roles
/cc @pierre @archlinux/teams/wiki/adminshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/47archweb role can't be deployed on a new box2021-01-10T22:31:17ZJelle van der Waaarchweb role can't be deployed on a new boxThere is a chicken and egg problem in that the role creates /srv/http/archweb and then tries to clone archweb.git in that directory.
```
TASK [clone archweb repo] *************************************************************************...There is a chicken and egg problem in that the role creates /srv/http/archweb and then tries to clone archweb.git in that directory.
```
TASK [clone archweb repo] ***********************************************************************************************************************************************
Thursday 25 June 2020 21:57:48 +0200 (0:00:00.050) 0:00:05.443 *********
fatal: [gemini.archlinux.org]: FAILED! => {"changed": false, "cmd": "/usr/bin/git clone --origin origin https://github.com/archlinux/archweb.git /srv/http/archweb", "msg": "fatal: destination path '/srv/http/archweb' already exists and is not an empty directory.", "rc": 128, "stderr": "fatal: destination path '/srv/http/archweb' already exists and is not an empty directory.\n", "stderr_lines": ["fatal: destination path '/srv/http/archweb' already exists and is not an empty directory."], "stdout": "", "stdout_lines": []}
PLAY RECAP **************************************************************************************************************************************************************
gemini.archlinux.org : ok=4 changed=0 unreachable=0 failed=1 skipped=5 rescued=0 ignored=0
```
And
```
TASK [clone archweb repo] ***********************************************************************************************************************************************
Thursday 25 June 2020 22:04:52 +0200 (0:00:00.044) 0:00:05.342 *********
fatal: [gemini.archlinux.org]: FAILED! => {"changed": false, "msg": "Failed to verify GPG signature of commit/tag \"release_2020-06-05\"", "rc": 1, "stderr": "[GNUPG:] NEWSIG\n[GNUPG:] ERRSIG C06086337C50773E 1 8 00 1591391236 9 E499C79F53C96A54E572FEE1C06086337C50773E\n[GNUPG:] NO_PUBKEY C06086337C50773E\n", "stderr_lines": ["[GNUPG:] NEWSIG", "[GNUPG:] ERRSIG C06086337C50773E 1 8 00 1591391236 9 E499C79F53C96A54E572FEE1C06086337C50773E", "[GNUPG:] NO_PUBKEY C06086337C50773E"], "stdout": "", "stdout_lines": []}
PLAY RECAP **************************************************************************************************************************************************************
gemini.archlinux.org : ok=4 changed=0 unreachable=0 failed=1 skipped=5 rescued=0 ignored=0
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/49use irker to post notifications to IRC channels2020-11-26T11:57:45ZDavid Rungeuse irker to post notifications to IRC channelsWe now have [irker in [community]](https://www.archlinux.org/packages/community/any/irker/). It can be used to [integrate notification messages with IRC channels](https://docs.gitlab.com/ee/user/project/integrations/irker.html).
Unfortu...We now have [irker in [community]](https://www.archlinux.org/packages/community/any/irker/). It can be used to [integrate notification messages with IRC channels](https://docs.gitlab.com/ee/user/project/integrations/irker.html).
Unfortunately irker does not deal well with secrets (it exposes the password on the commandline via a flag), which is why deploying it in a container or on a special host would make most sense.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/57Apply more kernel hardening2020-10-18T02:39:32ZJelle van der WaaApply more kernel hardeningAt the moment we have a hardening role which applies a few things, this can be extended with:
- [ ] Set lockdown in kernel command line (requires setting GRUB_CMDLINE_LINUX_DEFAULT="rootflags=compress=lzo,lsm=lockdown,yama" in /etc/defa...At the moment we have a hardening role which applies a few things, this can be extended with:
- [ ] Set lockdown in kernel command line (requires setting GRUB_CMDLINE_LINUX_DEFAULT="rootflags=compress=lzo,lsm=lockdown,yama" in /etc/default/grub)
- [ ] Set module.sig_enforce=1 for boxes (verify that no machines use external modules like virtualbox)
- [x] Set net.core.bpf_jit_harden=2 (any impact for us @anthraxx)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/63Fix dbscripts role to symlink the dbscripts config.local for community and pa...2020-07-12T11:44:36ZJelle van der WaaFix dbscripts role to symlink the dbscripts config.local for community and packages repository.config.local is not deployed by ansible in the role in /srv/repos/svn-community/dbscripts and for svn-packages. The file should be the same as config.local.svn-community as a symlink. Check how it is on gemini currently.config.local is not deployed by ansible in the role in /srv/repos/svn-community/dbscripts and for svn-packages. The file should be the same as config.local.svn-community as a symlink. Check how it is on gemini currently.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/64Think about finely grained secret access2022-05-07T16:33:54ZSven-Hendrik Haasesvenstaro@archlinux.orgThink about finely grained secret accessWe currently slap everything into the Ansible Vault. This is not optimal. We need more finely grained access while still allowing people to get their work done. Investigate which tools allow for that still of managing credentials. Perhap...We currently slap everything into the Ansible Vault. This is not optimal. We need more finely grained access while still allowing people to get their work done. Investigate which tools allow for that still of managing credentials. Perhaps something like bitwarden-rs?https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/65Decide on a password manager2021-06-05T02:06:42ZGiancarlo RazzoliniDecide on a password managerWe need to take out of the vault the passwords that are not required for roles and playbooks to run, and that we could use to store passwords in the future, while also having granularity on who has access to which password.
* Requiremen...We need to take out of the vault the passwords that are not required for roles and playbooks to run, and that we could use to store passwords in the future, while also having granularity on who has access to which password.
* Requirements:
* has to be used remote
* entry level per people
* sso integration or gpg?
* for storing "team" credentials such as PyPi, keycloak admin creds
* Solutions to look into:
* password-storage
* bitwarden/bitwarden_rs
* keepass
* gopasshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/66Investigate using Tower/AWX2020-11-16T01:54:56ZGiancarlo RazzoliniInvestigate using Tower/AWXThis issue is to investigate implementing a central place for running books while also solving the [finer grained control](#64).This issue is to investigate implementing a central place for running books while also solving the [finer grained control](#64).https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/67Keycloak shortcomings2021-08-03T19:59:39ZSven-Hendrik Haasesvenstaro@archlinux.orgKeycloak shortcomingsThis issue serves to track Keycloak shortcomings that we've found.
- Email-squatting is possible as users can change their email to any random other email which allows you to block emails. This is not a security issue but it's pretty an...This issue serves to track Keycloak shortcomings that we've found.
- Email-squatting is possible as users can change their email to any random other email which allows you to block emails. This is not a security issue but it's pretty annoying. https://issues.redhat.com/browse/KEYCLOAK-6455
- ~~No audit logs. These exist but we have to set them up properly: https://www.keycloak.org/docs/latest/server_admin/#auditing-and-events~~
- ~~Doesn't allow multiple OTP devices. https://issues.redhat.com/browse/KEYCLOAK-14297~~
- ~~Users can't add WebAuthn providers in the account management page. https://issues.redhat.com/browse/KEYCLOAK-14298~~
- Users should be forced to an OTP before removing an OTP device. https://issues.redhat.com/browse/KEYCLOAK-14296
- Allow users to have multiple emails. https://issues.redhat.com/browse/KEYCLOAK-14295
- ~~Check that "forgot password" via email does not reset MFA which makes MFA basically useless https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/112~~
- Not being able to set your default OTP/Security key https://issues.redhat.com/browse/KEYCLOAK-18957Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/69Notes for GitLab + Keycloak announcement2020-07-30T20:08:13ZSven-Hendrik Haasesvenstaro@archlinux.orgNotes for GitLab + Keycloak announcementWe should properly document how we announce the official use of GitLab + Keycloak along with any notes for our staff and what the change means to them. We should also note how NOT to use it.
Notes in no particular order:
- Do not upload...We should properly document how we announce the official use of GitLab + Keycloak along with any notes for our staff and what the change means to them. We should also note how NOT to use it.
Notes in no particular order:
- Do not upload secrets.
- Do not use GitLab to automatically build packages.
- Secure runners are trusted and can be used to create official artifacts (such as archiso, archboxes, etc). They can only be hand-assigned by DevOps to specific projects and projects needs will be discussed on a one-by-one basis with project owners.
- all CI variables that contain secrets like credentials must be marked as "protect variable" and "mask variable"
- DevOps must review a projects .gitlab-ci.yml and also make the project owner aware that all jobs on protected branches/tags must always have a runner tag selector "secure" declared to avoid protected branches to be run on none secure runners which would allow shared runners to access the secrets we try to separate from such runners. Also only jobs should select the secure runners that actually need to access secure credentials in the CI variables. Default jobs that do not need any secrets and don't publish any artifacts must not be run on the secure runners.
- define modus operandi for secret rotation etc in case a secure runner is once assigned to unsecure jobs (f.e. failure of a .gitlab-ci.yml)
- if we need a privileged runner, we should have both, an unprivileged that is used for everything that doesn't need privileged docker and a special one with the privileged tag that only runs jobs that need privileged docker.Sven-Hendrik Haasesvenstaro@archlinux.orgLevente Polyakanthraxx@archlinux.orgSven-Hendrik Haasesvenstaro@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/73Improve mysqld server configuration2020-09-30T05:27:23ZJelle van der WaaImprove mysqld server configurationOur current innodb_buffer_pool_size is set to 64 MB in the role, while the [upstream default size](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size) is 128 MB and according to some [posts](htt...Our current innodb_buffer_pool_size is set to 64 MB in the role, while the [upstream default size](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size) is 128 MB and according to some [posts](https://dba.stackexchange.com/a/27341) this value can easily be set higher if enough ram is available.
In short, we should re-evaluate our defaults:
- [ ] innodb_data_file_path = ibdata1:10M:autoextend
- [ ] innodb_buffer_pool_size = 64M
- [ ] innodb_log_file_size = 64M
- [ ] innodb_log_buffer_size = 8M
- [ ] innodb_flush_log_at_trx_commit = 1
- [ ] innodb_stats_sample_pages = 32
- [ ] innodb_thread_concurrency = 8
- [ ] innodb_file_per_table
- [ ] table_open_cache = 64
An idea is to dump the flyspray db (400MB) in size and benchmark it with different settings in a VM to replicate the ram values we have in a live system.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/79Disable 2FA options in Gitlab2023-05-19T18:23:45ZCaleb Maclennancaleb@alerque.comDisable 2FA options in GitlabSince this Gitlab instance *only* accepts logins via SSO from Keycloak there is no point in Gitlab prompting for (or even allowing) adding 2FA to Gitlab accounts. That's only going to confuse and confiscate things.Since this Gitlab instance *only* accepts logins via SSO from Keycloak there is no point in Gitlab prompting for (or even allowing) adding 2FA to Gitlab accounts. That's only going to confuse and confiscate things.Caleb Maclennancaleb@alerque.comCaleb Maclennancaleb@alerque.comhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/82Follow-up from "Add description templates for user onboarding and offboarding"2020-07-27T21:51:57ZSven-Hendrik Haasesvenstaro@archlinux.orgFollow-up from "Add description templates for user onboarding and offboarding"The following discussions from !38 should be addressed:
- [ ] @jelle started a [discussion](https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/38#note_1968):
> This would lack some things as:
>
> * bbs...The following discussions from !38 should be addressed:
- [ ] @jelle started a [discussion](https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/38#note_1968):
> This would lack some things as:
>
> * bbs (you get some flair)
> * bug tracker
> * wiki
> * gitlab? (Since we have no gluebuddy yet?)
- [ ] @jelle started a [discussion](https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/38#note_1969):
> This lacks:
> * running playbooks/tasks/reencrypt-vault-key.yml
> * removing the user from email aliasses on orion.
- [ ] @jelle started a [discussion](https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/38#note_1970):
> Missing:
> * playbooks/tasks/reencrypt-vault-key.ym
- [ ] https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/38#note_1964
- [ ] https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/38#note_1965
- [ ] https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/38#note_1966Jelle van der Waahashworkshashworks@archlinux.orgJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/83Document keycloak debugging2020-09-09T19:20:05ZSven-Hendrik Haasesvenstaro@archlinux.orgDocument keycloak debuggingMight be helpful to document how to debug Keycloak as it's a fairly important part of our infra.
- Logs
- [Debugging an OIDC client](https://lists.jboss.org/pipermail/keycloak-user/2016-July/006793.html)
- `misc/kcadm_wrapper.sh`
- Ment...Might be helpful to document how to debug Keycloak as it's a fairly important part of our infra.
- Logs
- [Debugging an OIDC client](https://lists.jboss.org/pipermail/keycloak-user/2016-July/006793.html)
- `misc/kcadm_wrapper.sh`
- Mention stuff like https://accounts.archlinux.org/auth/realms/archlinux/protocol/openid-connect/userinfo and https://accounts.archlinux.org/auth/realms/archlinux/protocol/saml/descriptorhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/91Create template for adding external contributor2020-07-31T13:07:29ZSven-Hendrik Haasesvenstaro@archlinux.orgCreate template for adding external contributorWe should add a template for adding external contributors so we have some paper trail and so that we don't make mistakes.
- [ ] Add them to the `External Contributor`s group in Keycloak
- [ ] Assign them as `Developer` in GitLab to the ...We should add a template for adding external contributors so we have some paper trail and so that we don't make mistakes.
- [ ] Add them to the `External Contributor`s group in Keycloak
- [ ] Assign them as `Developer` in GitLab to the proper projecthttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/104Investigate building containers to test setups locally2020-10-18T14:42:03ZJelle van der WaaInvestigate building containers to test setups locally@ffy00 and @lahwaacz have looked into building container images to run a playbook deployed machine locally for testing. We should investigate if adding the ability to build containers locally aids development, debugging issues locally.
...@ffy00 and @lahwaacz have looked into building container images to run a playbook deployed machine locally for testing. We should investigate if adding the ability to build containers locally aids development, debugging issues locally.
https://gitlab.archlinux.org/ffy00/infrastructure/-/commit/c4fbad31a256ae6767e5796e2faa0115d20d2c99
https://ansible-community.github.io/ansible-bender/build/html/index.htmlhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/109offload-build should create subvolumes in /var/lib/archbuild2020-09-06T13:04:49ZJelle van der Waaoffload-build should create subvolumes in /var/lib/archbuildCurrently we have ~ 162 subvolumes from offload-build (most likely as the build was aborted
```
[root@dragon ~]# btrfs subvolume list / | grep offload | wc -l
162
```
These should be cleaned up automatically using clean-chroots, by ch...Currently we have ~ 162 subvolumes from offload-build (most likely as the build was aborted
```
[root@dragon ~]# btrfs subvolume list / | grep offload | wc -l
162
```
These should be cleaned up automatically using clean-chroots, by changing offload-build to create subvolumes in /var/lib/archbuild.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/111Store GitLab database on its own filesystem2020-09-28T07:17:46ZKristian KlausenStore GitLab database on its own filesystemIf a malicious user \<somehow> manages to fill up the filesystem, we risk the database goes down or enter read-only mode.
We should try to avoid that situation by storing the database on its own filesystem. We should also consider stori...If a malicious user \<somehow> manages to fill up the filesystem, we risk the database goes down or enter read-only mode.
We should try to avoid that situation by storing the database on its own filesystem. We should also consider storing [other type of GitLab "content"](https://docs.gitlab.com/ee/administration/object_storage.html) on independent filesystems.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/117sourceballs.service systemd unit should check if repos are locked2020-09-09T19:08:50ZJelle van der Waasourceballs.service systemd unit should check if repos are lockedwhen the database is locked, sourceballs can't run and fails.
```
Sep 05 04:37:06 gemini.archlinux.org systemd[1]: Starting Sourceballs...
Sep 05 04:37:08 gemini.archlinux.org sourceballs[920185]: ==> ERROR: Repo [community] (x86_64) is ...when the database is locked, sourceballs can't run and fails.
```
Sep 05 04:37:06 gemini.archlinux.org systemd[1]: Starting Sourceballs...
Sep 05 04:37:08 gemini.archlinux.org sourceballs[920185]: ==> ERROR: Repo [community] (x86_64) is already locked by repo-{add,remove} process 920080
Sep 05 04:37:09 gemini.archlinux.org sourceballs[920185]: ==> Removing left over lock from sourceballs
Sep 05 04:37:08 gemini.archlinux.org systemd[1]: sourceballs.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 04:37:08 gemini.archlinux.org systemd[1]: sourceballs.service: Failed with result 'exit-code'.
Sep 05 04:37:08 gemini.archlinux.org systemd[1]: Failed to start Sourceballs.
```
systemd has ```ConditionPathExistsGlob``` in the ```[Unit]``` section which can be used to postpone the service from starting until the the lock file is gone.
The lock file is created by [dbscripts](https://github.com/archlinux/dbscripts/blob/eb1a090c5bfb9a1620abc21f8b81da510f965c4d/db-functions#L45) and located [here](https://github.com/archlinux/dbscripts/blob/d5970df3a7051b7ceb5d69a345c763e9b5ed8052/config.local.svn-packages#L14)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/124Add reset password gauge to keycloak metrics and monitoring2020-09-09T19:26:08ZJelle van der WaaAdd reset password gauge to keycloak metrics and monitoringIt would be good to monitor excessive attempts to reset passwords attempts and errors:
Keycloak defines the following two events for it:
* SEND_RESET_PASSWORD
* SEND_RESET_PASSWORD_ERROR
* RESET_PASSWORD
* RESET_PASSWORD_ERROR (for exa...It would be good to monitor excessive attempts to reset passwords attempts and errors:
Keycloak defines the following two events for it:
* SEND_RESET_PASSWORD
* SEND_RESET_PASSWORD_ERROR
* RESET_PASSWORD
* RESET_PASSWORD_ERROR (for example invalid code, expired code)
The list of events which are record are defined in [keycloak-metrics-spi](https://github.com/aerogear/keycloak-metrics-spi/blob/master/src/main/java/org/jboss/aerogear/keycloak/metrics/MetricsEventListener.java#L18)
And needs to be first implemented [upstream](https://github.com/aerogear/keycloak-metrics-spi/issues/78) before we can add monitoring and an alert for it.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/126Add mysql dashboard to Grafana2020-09-09T19:27:40ZJelle van der WaaAdd mysql dashboard to Grafanahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/128Document monitoring solution2020-10-13T20:26:58ZSven-Hendrik Haasesvenstaro@archlinux.orgDocument monitoring solution* [x] Using Grafana / where to find Grafana
* [x] Adding Grafana dashboard
* [x] Adding prometheus monitoring / different types of collectors (mysqld, memcached, borg, etc.)
* [ ] Adding a new Alert on Alertmanager* [x] Using Grafana / where to find Grafana
* [x] Adding Grafana dashboard
* [x] Adding prometheus monitoring / different types of collectors (mysqld, memcached, borg, etc.)
* [ ] Adding a new Alert on AlertmanagerJelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/129Add a Gitlab dashboard for Grafana2020-09-10T18:49:53ZJelle van der WaaAdd a Gitlab dashboard for Grafanahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/130Consider making a staging Keycloak for toying around2021-06-08T12:26:49ZSven-Hendrik Haasesvenstaro@archlinux.orgConsider making a staging Keycloak for toying aroundTesting/rolling out new login flows should be tested in a staging Keycloak env. before rolling them out in production. After #39 we can't just keep toying around with the production instance.Testing/rolling out new login flows should be tested in a staging Keycloak env. before rolling them out in production. After #39 we can't just keep toying around with the production instance.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/132Adjust the Keycloak theme to warn about removing all MFA2020-09-19T08:36:38ZJelle van der WaaAdjust the Keycloak theme to warn about removing all MFAKeycloak currently allows removing all MFA's from the [configuration page](https://accounts.archlinux.org/auth/realms/archlinux/account/#/security/signingin). We should warn users that they shouldn't remove all their MFA authenticators.
...Keycloak currently allows removing all MFA's from the [configuration page](https://accounts.archlinux.org/auth/realms/archlinux/account/#/security/signingin). We should warn users that they shouldn't remove all their MFA authenticators.
@lambdaclan can Keycloak be themed to achieve this?Ira ¯\_(ツ)_/¯Ira ¯\_(ツ)_/¯https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/137Add alert for Prometheus mysqld exporter being up2020-09-20T18:27:19ZJelle van der WaaAdd alert for Prometheus mysqld exporter being upWhen prometheus can scrape the mysqld exporter but there is no data, an alert should be generated.When prometheus can scrape the mysqld exporter but there is no data, an alert should be generated.Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/147Add Prometheus metrics dashboard2020-09-21T18:33:39ZJelle van der WaaAdd Prometheus metrics dashboardAdd a Grafana dashboard for Prometheus itself, to keep track of the data growth.
* ingestion_samples_per_second: rate(prometheus_tsdb_head_samples_appended_total[2h])
* bytes_per_sample: rate(prometheus_tsdb_compaction_chunk_size_sum[2...Add a Grafana dashboard for Prometheus itself, to keep track of the data growth.
* ingestion_samples_per_second: rate(prometheus_tsdb_head_samples_appended_total[2h])
* bytes_per_sample: rate(prometheus_tsdb_compaction_chunk_size_sum[2h]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[2h])Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/150Add total Keycloak users on Grafana2020-10-03T16:03:50ZSven-Hendrik Haasesvenstaro@archlinux.orgAdd total Keycloak users on GrafanaA graph over time showing total registered users on KeycloakA graph over time showing total registered users on KeycloakJelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/156Setup custom pre-receive hook for important repositories2022-04-15T17:56:23ZKristian KlausenSetup custom pre-receive hook for important repositoriesWe should require every merge commit to be signed by a key in the [archlinux-keyring](https://git.archlinux.org/archlinux-keyring.git/).
See: https://docs.gitlab.com/ee/administration/server_hooks.htmlWe should require every merge commit to be signed by a key in the [archlinux-keyring](https://git.archlinux.org/archlinux-keyring.git/).
See: https://docs.gitlab.com/ee/administration/server_hooks.htmlhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/167Monitor nginx active/max connections and total memory usage2021-07-22T10:11:21ZFrederik SchwanMonitor nginx active/max connections and total memory usageMigrated from https://kanboard.archlinux.org/project/1/task/68Migrated from https://kanboard.archlinux.org/project/1/task/68https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/171Test our roles with testkitchen/inspec2020-10-21T12:04:32ZFrederik SchwanTest our roles with testkitchen/inspecWe should consider using inspec/testkitchen to test if our roles behave properly.
Very low priority as there is way more things to do.
Migrated from: https://kanboard.archlinux.org/project/1/task/56We should consider using inspec/testkitchen to test if our roles behave properly.
Very low priority as there is way more things to do.
Migrated from: https://kanboard.archlinux.org/project/1/task/56https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/174Determine why wiki exception are not logged to syslog2020-10-18T02:30:33ZFrederik SchwanDetermine why wiki exception are not logged to syslogThe exceptions in this bug[1] were not logged. I had to set up a debug log in mediawiki itself which should be unnecessary AFAICT from the docs.
[1] https://bugs.archlinux.org/task/57944
Migrated from: https://kanboard.archlinux.org/pr...The exceptions in this bug[1] were not logged. I had to set up a debug log in mediawiki itself which should be unnecessary AFAICT from the docs.
[1] https://bugs.archlinux.org/task/57944
Migrated from: https://kanboard.archlinux.org/project/1/task/80https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/175Check if ARA works for us2020-10-18T02:31:56ZFrederik SchwanCheck if ARA works for usCheck how it can connect to the DB, if it supports multiple DBs or how we can use it if our DB might be down. Also check if it would work for us in general (somehow encrypted db connection for example)
https://github.com/openstack/ara
...Check how it can connect to the DB, if it supports multiple DBs or how we can use it if our DB might be down. Also check if it would work for us in general (somehow encrypted db connection for example)
https://github.com/openstack/ara
Migrated from: https://kanboard.archlinux.org/project/1/task/87https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/176Think about removing firewall rules2022-10-22T00:56:15ZFrederik SchwanThink about removing firewall rulesWe may no longer need rules if a service is moved or removed so the firewall rules should go away too. Could be difficult with the ansible module since they are added incrementally so we have to explicitly remove them again. Figure out h...We may no longer need rules if a service is moved or removed so the firewall rules should go away too. Could be difficult with the ansible module since they are added incrementally so we have to explicitly remove them again. Figure out how to best integrate that into our playbooks or if there is a better solution.
Migrated from: https://kanboard.archlinux.org/project/1/task/93https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/178Setup security@archlinux.org mailing list for the Security Team2021-10-27T21:43:24ZFrederik SchwanSetup security@archlinux.org mailing list for the Security TeamThe security team wants to offer a way to report security issues related to arch securely using PGP encrypted mails. This uses a publicly known PGP key for the security@archlinux.org email address and forwards the email by re-encrypting ...The security team wants to offer a way to report security issues related to arch securely using PGP encrypted mails. This uses a publicly known PGP key for the security@archlinux.org email address and forwards the email by re-encrypting it to the security team.
Schleuder is a piece of software which offers this functionality, it is to be packaged in the repos to be used in our infrastructure.
Since this requires a private key on our server, we should consider using a new server or a server with limited access.
Migrated from: https://kanboard.archlinux.org/project/1/task/98https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/184Secure SSH setup, especially on build servers2020-11-02T10:34:55ZFrederik SchwanSecure SSH setup, especially on build servers* https://matrix.org/blog/2019/04/11/security-incident/
* https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg
* https://web.archive.org/web/20190412143901/https://github.com/matrix-org/matrix.org/issues/
* https://doi...* https://matrix.org/blog/2019/04/11/security-incident/
* https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg
* https://web.archive.org/web/20190412143901/https://github.com/matrix-org/matrix.org/issues/
* https://doi.org/10.6028/NIST.IR.7966
- determine needs of our users
- copy packages from build server to orion
- sign packages on build server
- svn?
- anything else?
- implement solutions
- disable all unneeded access; implement all useful security ideas from the NIST paper
- verify that new setup is secure
Migrated from: https://kanboard.archlinux.org/project/1/task/132https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/185Monitor postgres2021-04-27T16:48:43ZFrederik SchwanMonitor postgresselect/update/insert/delete/... per second
Migrated from: https://kanboard.archlinux.org/project/1/task/51select/update/insert/delete/... per second
Migrated from: https://kanboard.archlinux.org/project/1/task/51https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/186Create playbook for user removal2021-01-05T21:28:48ZFrederik SchwanCreate playbook for user removalCreate a playbook to deactivate/remove a user from our servers. For example when a Trusted User or Developer resigns.
- Deactive user in Archweb and move it to the corresponding fellows group and disable the user (this requires a manage...Create a playbook to deactivate/remove a user from our servers. For example when a Trusted User or Developer resigns.
- Deactive user in Archweb and move it to the corresponding fellows group and disable the user (this requires a management command from archweb)
- Remove the pubkey from the infrastructure repo
- Remove the user from all servers
- Run the relevant role to remove the users pubkey from our servers
- Optional Create a ticket for keyring
- Optional Update BBS/Bugtracker role (if this is easily scriptable)
- Optional: List packages still signed by this user in the repos (could be an archweb management command as well, combine with deactivation)
Migrated from: https://kanboard.archlinux.org/project/1/task/97https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/187Add Secure Header for all our sites2021-08-27T19:31:48ZFrederik SchwanAdd Secure Header for all our sitesA good example of a site with correct secure headers is security.archlinux.org and we want to do the same for other sites.
Sites needing extra headers:
- bbs.archlinux.org
- wiki.archlinux.org
- aur.archlinux.org, needs patches in aurw...A good example of a site with correct secure headers is security.archlinux.org and we want to do the same for other sites.
Sites needing extra headers:
- bbs.archlinux.org
- wiki.archlinux.org
- aur.archlinux.org, needs patches in aurweb.
https://securityheaders.com/:
![image](/uploads/3820ca04ebaa3f8b40b325f41fe064ce/image.png)
![image](/uploads/262d6a35ecd72f09040135a6d752d15a/image.png)
![image](/uploads/f93c7ef25316275427b941a3ff1c086a/image.png)
Migrated from: https://kanboard.archlinux.org/project/1/task/120https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/188Harden services with systemd sandboxing2020-10-18T13:09:03ZFrederik SchwanHarden services with systemd sandboxingWe can further harden our own systemd services with some options from man systemd.exec
For example:
```
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKern...We can further harden our own systemd services with some options from man systemd.exec
For example:
```
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
```
Later systemd will have an option to show hints about it: https://github.com/systemd/systemd/pull/10701
Migrated from https://kanboard.archlinux.org/project/1/task/103https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/190Add the pacman public keyring to root user gpg.conf2021-04-28T02:04:15ZGiancarlo RazzoliniAdd the pacman public keyring to root user gpg.confSome operations require the keys of our staff and since the root user keyring is used for ansible, let's add archlinux-keyrings's file /etc/pacman.d/gnupg/pubring.gpg to /root/.gnupg/gpg.conf.Some operations require the keys of our staff and since the root user keyring is used for ansible, let's add archlinux-keyrings's file /etc/pacman.d/gnupg/pubring.gpg to /root/.gnupg/gpg.conf.Giancarlo RazzoliniGiancarlo Razzolinihttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/202postfix relayhost configuration prints warnings in log2020-10-26T19:50:59ZJelle van der Waapostfix relayhost configuration prints warnings in log```
/usr/bin/postconf: warning: /etc/postfix/master.cf: undefined parameter: post_queue_smtpd_recipient_restrictions
``````
/usr/bin/postconf: warning: /etc/postfix/master.cf: undefined parameter: post_queue_smtpd_recipient_restrictions
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/206Let the world mirror our (encrypted) backup2020-12-20T16:35:39ZKristian KlausenLet the world mirror our (encrypted) backup> “Only wimps use tape backup. REAL men just upload their important stuff on ftp and let the rest of the world mirror it.”
> \- Linus Torvalds
Should protect against "rogue employee"..> “Only wimps use tape backup. REAL men just upload their important stuff on ftp and let the rest of the world mirror it.”
> \- Linus Torvalds
Should protect against "rogue employee"..https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/208Switch to ECDSA certificate or provide dual ECDSA/RSA certificates2021-05-01T12:48:37ZKristian KlausenSwitch to ECDSA certificate or provide dual ECDSA/RSA certificatesWith the ECDSA support in Certbot slowly moving forward, I think it is time to create a issue: https://github.com/certbot/certbot/pull/8431 https://github.com/certbot/certbot/pull/8254#pullrequestreview-515564860With the ECDSA support in Certbot slowly moving forward, I think it is time to create a issue: https://github.com/certbot/certbot/pull/8431 https://github.com/certbot/certbot/pull/8254#pullrequestreview-515564860https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/214Prepare virtual user setup for dovecot and OpenSMTPD2022-04-11T13:14:46ZFrederik SchwanPrepare virtual user setup for dovecot and OpenSMTPDAt the moment we use one mailbox per unix user on the mail machine. This should be changed to a virtual setup where no users login to the mail machine anymore.
- [ ] create MR for new dovecot and OpenSMTPD config
- [ ] create migration ...At the moment we use one mailbox per unix user on the mail machine. This should be changed to a virtual setup where no users login to the mail machine anymore.
- [ ] create MR for new dovecot and OpenSMTPD config
- [ ] create migration script for dovecot mailboxeshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/217Store the (virtual) mail password in keycloak2023-02-28T21:46:08ZFrederik SchwanStore the (virtual) mail password in keycloakWe will store a password hash in the Keycloak user attributes. The attrs are writable <=> the user can manage his own account. This implies that we shall not save any data to the attrs that the user is not allowed to change. But his own ...We will store a password hash in the Keycloak user attributes. The attrs are writable <=> the user can manage his own account. This implies that we shall not save any data to the attrs that the user is not allowed to change. But his own mail password is fine.
- attribute to use: `mail_password_hash`
- no other attribute is saved
- preferably use the java on the server side for hashing
We also want to do some basic password validation (length > X). Keycloak already has something builtin which we should be able to use. We just need to expose it.
The custom attribute needs to be added to the Account Console. The new account console is written in React and use the REST endpoints so we will need to edit that to use our custom endpoints.
For custom provider deployment see [here](https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/5ac750c909357f3cda5223b475643fc164410a1d) for reference.
TODOs:
- [ ] Modify theme to add a custom attribute "mail_password_hash" to the account management console
- [ ] Implement a domain extension to provide custom REST endpoint for bcrypt with cost 12 and 2b variant (use Bouncy Castle library - make sure to use the crypt encoded version implemented in OpenBSD)
- [ ] Implement an add "password-validate" REST endpoint which use the internal Keycloak API
- [ ] Ensure the password hash attribute in Keycloak can be modified (via the templating engine)
- [x] Discuss which pw hash algo to use (bcrypt)
- [ ] Update [mail credential syncer script](https://gitlab.archlinux.org/archlinux/mail-credential-syncer/-/blob/master/src/main.rs#L160) to use 2b variant
- [ ] write manual how to change the mail pw
- [ ] ping all arch mail users to store their passwords
References:
- https://www.keycloak.org/docs/latest/server_development/#_extensions_rest
- https://github.com/keycloak/keycloak/tree/master/examples/providers/rest
- https://github.com/keycloak/keycloak/tree/master/examples/providers/domain-extension
- https://www.keycloak.org/docs/latest/server_development/#account-management-console
- https://www.bouncycastle.org/docs/docs1.5on/org/bouncycastle/crypto/generators/OpenBSDBCrypt.htmlIra ¯\_(ツ)_/¯Ira ¯\_(ツ)_/¯https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/218write export tool to automatically pull password hashes from keycloak2020-12-29T21:35:01ZFrederik Schwanwrite export tool to automatically pull password hashes from keycloakMoved from: #50 and #210
Project repo: https://gitlab.archlinux.org/archlinux/mail-credential-syncer
Since many members of arch-devops work with rust and it's good security characteristics, this tool shall be written in rust (https://...Moved from: #50 and #210
Project repo: https://gitlab.archlinux.org/archlinux/mail-credential-syncer
Since many members of arch-devops work with rust and it's good security characteristics, this tool shall be written in rust (https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/210#note_6535 contains a very dirty POC in Go).
Three config parameters:
- Path to mapping file for keycloak UUID -> arch mail address on local FS
- Keycloak hostname
- Post-receive script
Implementation:
- [ ] Use inotify to receive events when the mapping file changes
- [ ] Subscribe to pw change events for the keycloak user attribute `mail_password_hash`
Whenever an event fires:
- iterate over mapping
- get pw hash from keycloak
- check if the hash is valid and contains no malicious input (probably with a regex)
- export dovecot and opensmtpd version of virtual user file
- backup old config files
- run post receive script
- when the post receive script failes, restore old config files
- report error via e-mail (or prometheus?)Kristian KlausenKristian Klausenhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/225Improve offboarding2021-01-14T19:31:48ZFrederik SchwanImprove offboardingAs mentioned [here](https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/221#note_8080),
the offboarding doesn't work yet.
* homdir on gemini is not deleted
* user on mail is not removed
* user still can login to imap
* home...As mentioned [here](https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/221#note_8080),
the offboarding doesn't work yet.
* homdir on gemini is not deleted
* user on mail is not removed
* user still can login to imap
* homedir on mail is not removed
* if the user used the IMAP server, there is still a mailbox
* it's unclear what to do with the existing data and mailbox (delete? forward?)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/241Process DMARC and TLS-RPT reports automatically2020-12-17T12:33:17ZKristian KlausenProcess DMARC and TLS-RPT reports automaticallyWe should process all the DMARC and TLS-RPT reports automatically. Maybe we can use node_exporter textfile collector?
Inspiration for the DMARC logic: https://github.com/domainaware/parsedmarcWe should process all the DMARC and TLS-RPT reports automatically. Maybe we can use node_exporter textfile collector?
Inspiration for the DMARC logic: https://github.com/domainaware/parsedmarchttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/242Setup ICMP monitorings of all hosts2022-04-10T20:51:42ZKristian KlausenSetup ICMP monitorings of all hostsWe can use the [blackbox exporter](https://github.com/prometheus/blackbox_exporter).
This should help detecting misconfiguration (!199).We can use the [blackbox exporter](https://github.com/prometheus/blackbox_exporter).
This should help detecting misconfiguration (!199).https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/243Stop archweb services connecting over postgresql SSL2020-12-18T22:11:15ZJelle van der WaaStop archweb services connecting over postgresql SSLCurrently we have postgresql over SSL to allow archweb services on mirrors and on repos.archlinux.org for:
* mirrorcheck
* mirrorresolv
* reporead
* updating rsyncd whitelist from archweb db
To simplify things we can consider adding a ...Currently we have postgresql over SSL to allow archweb services on mirrors and on repos.archlinux.org for:
* mirrorcheck
* mirrorresolv
* reporead
* updating rsyncd whitelist from archweb db
To simplify things we can consider adding a full fledged API to archweb to be able to POST mirror results and GET allowed ips for whitelisting.
For reporead we could rsync the databases to archlinux.org, if this doesn't cause any issues with archweb reading the db while it's being rsync'd?https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/250Create prometheus mailq exporter + alert2022-02-26T22:17:19ZKristian KlausenCreate prometheus mailq exporter + alertSomeone (*cough*) broke Postfix (!249) on at least aur.archlinux.org, we should setup monitoring of the mail queue length.Someone (*cough*) broke Postfix (!249) on at least aur.archlinux.org, we should setup monitoring of the mail queue length.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/252Use terraform for managing GitLab/GitHub projects2021-01-15T00:48:17ZKristian KlausenUse terraform for managing GitLab/GitHub projectsSee the ["New Official Project"](.gitlab/issue_templates/New Official Project.md) issue template, for what needs to be configured.
https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs/resources/project
https://registry.t...See the ["New Official Project"](.gitlab/issue_templates/New Official Project.md) issue template, for what needs to be configured.
https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs/resources/project
https://registry.terraform.io/providers/hashicorp/github/latest/docs/resources/repositoryhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/257FluxBB replacement2024-01-08T16:28:21ZKristian KlausenFluxBB replacementFluxBB isn't maintained so we need to migrate to something else.
https://lists.archlinux.org/pipermail/arch-devops/2019-October/000297.html
```
The below feedback has been solicited from the current Arch Linux Forum
team in October 2019...FluxBB isn't maintained so we need to migrate to something else.
https://lists.archlinux.org/pipermail/arch-devops/2019-October/000297.html
```
The below feedback has been solicited from the current Arch Linux Forum
team in October 2019, based on the premise that the DevOps team desires to
replace the current forum software.
The points have been summarized for readability, and anonymized to ensure
the points are considered based on their merit alone.
Our preference is that a demo system is set up prior to a final decision
being made so we can evaluate the moderation tools available to assess
their suitability for our community behaviors.
General Criteria
===============
* I hope that we will stick to simple and not deviate too far from the
functionality of a basic forum.
* I would opt for disabling all unnecessary features
* I am very skeptical of any system of community voting for relevance.
* The forum should remain a technical resource and not devolve into
karma farming.
* I would also strongly oppose any sort of obligatory 2-factor
authentication.
* Reasonably good functionality on limited bandwidth and/or text-mode
browsers.
* I would like better tools to help identify duplicate accounts
* Perhaps tools that can auto-detect behavior such as post blanking when
it starts.
* I'm not a fan of endless scrolling or javascript "features" either,
and I don't see any point of social media things like "likes" on a
technical support forum.
* KISS.
* One of the things that I hate most in the world is continuous scroll
down rather than pagination.
* Functionally, FluxBB does everything I expect a forum software to do.
* It shouldn't contain post-voting facilities
* I dislike sub-threading where one can reply to other posts inline. If
reddit is any indication it leads to incredibly messy and hard to follow
discussions.
* I liked myBB and misago the most from some quick browsing through the
examples.
* Based on several criteria above, Discourse seems like a particularly
poor fit for our community.
* "Oh dear god, anything but discourse."
Alternatives
===============
phpBB
* https://www.phpbb.com/
* Seems still actively developed.
* Is listed on some "top 10" forum software for 2019 sites.
* It is very similar to fluxBB.
Thredded
* https://thredded.org/
* It is actively developed
* Free & open-source
* Has a 'modern' visual style (while being readily
customizable/themeable)
Can be used with a SSO
Paginates
Demos I've found load quickly and are easy to read on a TUI.
MyBB
* https://mybb.com/
* Maintained
* Has an LDAP plugin.
Farum
* https://flarum.org/features/
* Seems to be similar to discourse; claims to be lightweight.
* Has SSO, API, and Anti-spam.
* Still beta release.
Vanilla
* https://open.vanillaforums.com/
* Maintained
* Open-core/Freemium model.
* Has SSO.
Misago
* https://misago-project.org/
* "Looks OK from a bit of playing around on their site. Not sure what
moderation tools look like."
Discourse
* https://www.discourse.org/
* There seems to be a lot of bells and whistles that increase its
complexity but I have no experience with it.
* Discourse ... design philosophy seems to be in the mindset of catering
to the majority who have high powered systems, good network connections,
and "modern" (GUI) browsers. The Arch community has always seemed to work
hard to not needlessly marginalize those who have limited bandwidth, old
hardware, or text-mode browsers; so the use of a tool like discourse
strikes me as either antithetical to our existing community ethos, or a
sign of a significant change in direction.
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/258Borg alertmanager notification doesn't show last backup2021-01-02T14:36:11ZJelle van der WaaBorg alertmanager notification doesn't show last backup```
description = Borg has not backuped for more than 24 hours. Last backup made on 1970-01-25 02:26:42.552 +0000 UTC
``````
description = Borg has not backuped for more than 24 hours. Last backup made on 1970-01-25 02:26:42.552 +0000 UTC
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/266Enable GPG verification for archmanweb2021-01-10T22:31:16ZJakub KlinkovskýEnable GPG verification for archmanwebFollow-up from !159
Figure out and enable verification of GPG signatures with Ansible. The problem is related to the second one in https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/47Follow-up from !159
Figure out and enable verification of GPG signatures with Ansible. The problem is related to the second one in https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/47Jakub KlinkovskýJakub Klinkovskýhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/269Disable unneeded project features2021-03-28T22:56:24ZKristian KlausenDisable unneeded project featuresRelated: https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/252
I only had a quick look on at few projects:
- [x] https://gitlab.archlinux.org/archlinux/wkd (disable Container registry + Packages + Requirements + Wiki + Sni...Related: https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/252
I only had a quick look on at few projects:
- [x] https://gitlab.archlinux.org/archlinux/wkd (disable Container registry + Packages + Requirements + Wiki + Snippets (?))
- [x] https://gitlab.archlinux.org/archlinux/archmanweb (disable Container registry + Packages + Requirements + Wiki + Snippets (?))
- [x] https://gitlab.archlinux.org/archlinux/asknot-ng (disable Container registry + Packages + Requirements + Wiki + Snippets (?))
- [x] https://gitlab.archlinux.org/archlinux/archiso (disable Container registry + Packages + Requirements + Snippets (?))
- [x] https://gitlab.archlinux.org/archlinux/arch-boxes (disable Requirements + Operations + Analytics)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/273man.archlinux.org should be linked from top banner2021-05-11T14:37:14ZMichael Vorburger.chman.archlinux.org should be linked from top bannerI've just noticed https://archlinux.org/news/manual-pages-indexing-service/ (while staring at https://man.archlinux.org/man/community/cloud-image-utils/cloud-localds.1.en, in the context of https://gitlab.archlinux.org/archlinux/archiso/...I've just noticed https://archlinux.org/news/manual-pages-indexing-service/ (while staring at https://man.archlinux.org/man/community/cloud-image-utils/cloud-localds.1.en, in the context of https://gitlab.archlinux.org/archlinux/archiso/-/merge_requests/117), and it occurred to me that it could be neat to have a "man" link in that top banner that appears to be included everywhere (I mean the thing showing "Home Packages Forums Wiki Bugs Security AUR Download").
I'm not sure this is the best place to file an issue re. the website itself, but it's the best I could think of.
If the HTML template for that is in a Git repo, I'm happy to raise an MR suggesting it.
@dvzrv @nl6720Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/274initial playbook run of build.archlinux.org led to [testing] packages being i...2023-03-22T19:39:51ZLevente Polyakanthraxx@archlinux.orginitial playbook run of build.archlinux.org led to [testing] packages being installedinitial playbook run of build.archlinux.org led to [testing] packages being installed
We should investigate what kind of order or problem the root of this cause is.
I suspect afterwards it may be fine, but initially while setting up `/et...initial playbook run of build.archlinux.org led to [testing] packages being installed
We should investigate what kind of order or problem the root of this cause is.
I suspect afterwards it may be fine, but initially while setting up `/etc/pacman.conf` it somehow upgrades from [testing]https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/280Setup signing server2022-05-29T16:33:17ZKristian KlausenSetup signing serverWe are automating more and more stuff (ex: https://gitlab.archlinux.org/archlinux/arch-boxes/-/issues/132), so we need a signing server sooner rather than later.
Spec: https://gitlab.archlinux.org/archlinux/signstar
Alternatives:
* Fed...We are automating more and more stuff (ex: https://gitlab.archlinux.org/archlinux/arch-boxes/-/issues/132), so we need a signing server sooner rather than later.
Spec: https://gitlab.archlinux.org/archlinux/signstar
Alternatives:
* Fedora / Red Hat use https://pagure.io/sigul
* openSUSE uses https://github.com/openSUSE/obs-sign
* Mozilla uses XX
* https://github.com/coreos/ferohttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/299Harden borg-backup.service2021-04-28T02:11:13ZKristian KlausenHarden borg-backup.serviceWe [received](https://lists.archlinux.org/pipermail/arch-devops/2021-March/000509.html) a bug fix for `backup-mysql.sh.j2`, that under the right circumstances it could run `rm -rf /*` (see also !339).
From https://gitlab.archlinux.org/a...We [received](https://lists.archlinux.org/pipermail/arch-devops/2021-March/000509.html) a bug fix for `backup-mysql.sh.j2`, that under the right circumstances it could run `rm -rf /*` (see also !339).
From https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/339#note_16314:
> We should harden [borg-backup.service](/archlinux/infrastructure/-/blob/master/roles/borg_client/files/borg-backup.service), so at worst the scripts can't delete any (important) files. Perhaps `ProtectSystem=strict` and `ReadWritePaths=<something>`.
>
> I'm not 100% sure it will work though:
>
> > [ReadWritePaths=, ReadOnlyPaths=, InaccessiblePaths=, ExecPaths=, NoExecPaths=](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#ReadWritePaths=)
> >
> > \[...\]
> >
> > Note that these settings will disconnect propagation of mounts from the unit's processes to the host. This means that this setting may not be used for services which shall be able to install mount points in the main mount namespace.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/301Deduplicate the syncarchive, syncdebug and syncrepo role2022-11-12T18:30:57ZKristian KlausenDeduplicate the syncarchive, syncdebug and syncrepo rolePerhaps we can create a separate role and use `include_role`.Perhaps we can create a separate role and use `include_role`.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/308Implement rate limiting2021-05-18T12:58:05ZKristian KlausenImplement rate limitingWe should implement rate limiting for the most important service (bbs, wiki, aur, al.og etc.), so a single user can't take them down with a 5$ VPS.We should implement rate limiting for the most important service (bbs, wiki, aur, al.og etc.), so a single user can't take them down with a 5$ VPS.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/315Enable (more) caching for Archwiki2021-04-23T12:24:35ZKristian KlausenEnable (more) caching for Archwikihttps://www.mediawiki.org/wiki/Manual:$wgUseCdn first and perhaps https://www.mediawiki.org/wiki/Manual:File_cache in the future.https://www.mediawiki.org/wiki/Manual:$wgUseCdn first and perhaps https://www.mediawiki.org/wiki/Manual:File_cache in the future.Kristian KlausenKristian Klausenhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/324Create a nice-looking home dashboard for dashboards.archlinux.org2021-05-07T00:27:48ZKristian KlausenCreate a nice-looking home dashboard for dashboards.archlinux.orghttps://dashboards.archlinux.orghttps://dashboards.archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/327Enable KSM for the runners2021-05-21T22:20:30ZKristian KlausenEnable KSM for the runnersarchiso and arch-boxes is using a ton of memory, perhaps we should enable KSM?
https://wiki.archlinux.org/title/QEMU#Enabling_KSMarchiso and arch-boxes is using a ton of memory, perhaps we should enable KSM?
https://wiki.archlinux.org/title/QEMU#Enabling_KSMhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/328Add RateLimit headers for selected endpoints2021-05-11T23:17:30ZKristian KlausenAdd RateLimit headers for selected endpointshttps://datatracker.ietf.org/doc/html/draft-ietf-httpapi-ratelimit-headers-00
Ref: https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/378#note_23518https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-ratelimit-headers-00
Ref: https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/378#note_23518https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/330Add watchdog2021-05-14T23:26:43ZKristian KlausenAdd watchdogWe enabled `RuntimeWatchdogSec=5min` in https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/304, but `systemd` is apparently still "pinging" the watchdog if the system is trashing heavily.
Perhaps we should switch to...We enabled `RuntimeWatchdogSec=5min` in https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/304, but `systemd` is apparently still "pinging" the watchdog if the system is trashing heavily.
Perhaps we should switch to [watchdog<sup>AUR</sup>](https://aur.archlinux.org/packages/watchdog/)? or something more configurable? or https://github.com/troglobit/watchdogd (I also created a issue for PSI support: https://github.com/troglobit/watchdogd/issues/25)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/335Send blackbox exporter metrics to dashboards.al.org2021-05-13T21:58:20ZKristian KlausenSend blackbox exporter metrics to dashboards.al.orgWe need to commit the dashboard first.We need to commit the dashboard first.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/337Gitlab promtail logs2021-05-15T20:36:26ZJelle van der WaaGitlab promtail logsOur gitlab instance runs nginx in the gitlab container which means the logs are not in their usual place, but in `/srv/gitlab/logs/nginx`.Our gitlab instance runs nginx in the gitlab container which means the logs are not in their usual place, but in `/srv/gitlab/logs/nginx`.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/343Allow dynamically setting the iso location with the install_arch role2021-05-18T21:18:53ZDavid RungeAllow dynamically setting the iso location with the install_arch roleIn https://gitlab.archlinux.org/archlinux/releng/-/issues/11 we are discussing the restructuring of our release artifacts.
One of the projects affected by this, is this repository, as [the install_arch role makes use of the bootstrap im...In https://gitlab.archlinux.org/archlinux/releng/-/issues/11 we are discussing the restructuring of our release artifacts.
One of the projects affected by this, is this repository, as [the install_arch role makes use of the bootstrap image](https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/128edca7ea73992c3cf857366bb4b9b37a6751fa/roles/install_arch/tasks/main.yml#L47) (which will change locations).
After introducing the new directory structure, the role needs to be adapted, before the old release artifacts are phased out after three months.David RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/352Modernize secrets management2021-06-05T18:11:16ZKristian KlausenModernize secrets management[[_TOC_]]
### Intro
On the [2021-05-20 DevOps meeting](https://gitlab.archlinux.org/archlinux/infrastructure/-/wikis/meetings/2021-05-20#segmented-vault-access) segmenting the vault by using [Ansible's vault IDs](https://docs.ansible.co...[[_TOC_]]
### Intro
On the [2021-05-20 DevOps meeting](https://gitlab.archlinux.org/archlinux/infrastructure/-/wikis/meetings/2021-05-20#segmented-vault-access) segmenting the vault by using [Ansible's vault IDs](https://docs.ansible.com/ansible/latest/user_guide/vault.html#managing-multiple-passwords-with-vault-ids) was discussed, so access could be handed out to Junior DevOps. The idea was scratched as the tooling isn't ideal and we would end up with a lot of vaults.
### Alternatives
Instead it was decided to look at more "modern" alternatives like [Vault](https://www.vaultproject.io/) and [vaultwarden](https://github.com/dani-garcia/vaultwarden) (*Unofficial Bitwarden compatible server written in Rust*).
#### Vaultwarden
[vaultwarden](https://github.com/dani-garcia/vaultwarden) was ditched very early on as it lacks some critical features:
* No OIDC support
* Custom tooling is required for integrating with Ansible
* No Terraform provider
#### Vault
[Vault](https://www.vaultproject.io/) on the other hand, looks like a much better solution:
* Built for secrets
* [Supports OIDC](https://www.vaultproject.io/docs/auth/jwt)
* [Supported by Ansible](https://docs.ansible.com/ansible/latest/collections/community/hashi_vault/hashi_vault_lookup.html)
```yml
{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/hello:value') }}
```
* [Policy support](https://www.vaultproject.io/docs/concepts/policies)
```sh
# This section grants all access on "secret/*". Further restrictions can be
# applied to this broad policy, as shown below.
path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Even though we allowed secret/*, this line explicitly denies
# secret/super-secret. This takes precedence.
path "secret/super-secret" {
capabilities = ["deny"]
}
```
* [Group support](https://www.vaultproject.io/docs/secrets/identity#identity-groups) (ex: assign every user group X if they are part of Keycloak group Y)
* [Audit logging](https://www.vaultproject.io/docs/audit)
* [Web UI](https://learn.hashicorp.com/tutorials/vault/getting-started-ui)
* [Terraform provider](https://registry.terraform.io/providers/hashicorp/vault/latest/docs)
* [Encrypted at rest](https://www.vaultproject.io/docs/concepts/seal)
##### Vault Workflow
<details><summary>Click to expand</summary>
```sh
$ vault login
Complete the login via your OIDC provider. Launching browser to:
https://dev-2i513orw.auth0.com/authorize?client_id=FFXlsY2atr_wfNaF_hMtsE-zTAeTZnu8&nonce=73fe4c11828d3c4eb8c2c270aa1b3ab45ebda490&redirect_uri=http%3A%2F%2Flocalhost%3A8250%2Foidc%2Fcallback&response_type=code&scope=openid&state=965d81adcfad9d83a29659891cee37358c8d45cc
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.mefxZpkwUzGirhakGtQZoez0
token_accessor oGB9LqtSGxiEWp2zz4DLlIBD
token_duration 768h
token_renewable true
token_policies ["default" "reader"]
identity_policies []
policies ["default" "reader"]
token_meta_role reader
$ ansible-playbook playbooks/....
```
Example var:
```yml
vault_monitoring_grafana_client_secret: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/roles/grafana:client_secret') }}"
```
Adding/reading a secret (or use the [web UI](https://learn.hashicorp.com/tutorials/vault/getting-started-ui)):
```shell
$ vault kv put secret/my-secret my-value=s3cr3t
Key Value
--- -----
created_time 2019-06-19T17:20:22.985303Z
deletion_time n/a
destroyed false
version 1
$ vault kv get secret/my-secret
====== Metadata ======
Key Value
--- -----
created_time 2019-06-19T17:20:22.985303Z
deletion_time n/a
destroyed false
version 1
====== Data ======
Key Value
--- -----
my-value s3cr3t
```
</details>
### TODO/questions
- [ ] Is Vault the best solution?
- [ ] Package https://aur.archlinux.org/packages/python-hvac/ ~~and https://github.com/ansible-collections/community.hashi_vault~~ (**Edit**: hashi_vault is installed by the ansible package)
- [x] What do we store in Vault and how do we store it?
- **Decision:** Everything used by Ansible or Terraform (please be aware of the Keycloak credentials)
- [ ] How should we split the secrets? Per role and per host?
- Do we have any secrets not tied to a role or host? Perhaps secrets used by Terraform?
- What secrets do we have?
- Credentials (Hetzner, GitHub etc.)
- Secrets:
- database passwords
- nginx htpasswd (usually metrics)
- internet archive password ? (2FA?)
- more?
- [x] How do we handle credentials? (related #65)
- **Decision:** Postponed for now, use `ansible-vault` in the meantime
- This is not a blocker and we can handle it later
- They should be stored locally\* and we should be able to revoke access (catch-22)
- \* we can't risk losing access due to a broken server
- [x] How often should you be required to reauth with Keycloak for accessing the vault?
- **Decision**: Use a low value for now (`default_lease_ttl=1h`) but allow renewing the vault token for a bit longer (`max_lease_ttl=3h`)
- https://learn.hashicorp.com/tutorials/vault/tokens#ttl-and-max-ttl
- [x] Manual or automatic unsealing?
- **Decision:** Use a script (`misc/unseal-the-vault.sh`) initially and if it is a hassle, reevaluate
- > When a Vault server is started, it starts in a sealed state. In this state, Vault is configured to know where and how to access the physical storage, but doesn't know how to decrypt any of it.
- https://www.vaultproject.io/docs/concepts/seal
- Should we create a script which SSH to the server and unseal the vault?
- or a script which run on the server at boot and unseal the vault automatically?
- [ ] Do we make Vault publicly accessible or via a VPN / ssh tunnel?
- [ ] For vault, check if we can also use it for ssh access via Keycloak?!https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/382Restrict mail service accounts to mail2021-08-02T12:30:15ZKristian KlausenRestrict mail service accounts to mailhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/385Add documentation for adding a new server2021-08-01T17:07:10ZKristian KlausenAdd documentation for adding a new server1. terraform
1. Add playbook
1. Add to `hosts` (add to the correct groups)
1. Add wireguard keys and address
1. Run `prometheus` role on `monitoring.archlinux.org`
1. mail?1. terraform
1. Add playbook
1. Add to `hosts` (add to the correct groups)
1. Add wireguard keys and address
1. Run `prometheus` role on `monitoring.archlinux.org`
1. mail?https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/393Implement the arch-release-promotion2021-09-09T18:36:34ZJelle van der WaaImplement the arch-release-promotion
* install arch-release-promotion
* Create directory `/srv/ftp/releases`
* Override systemd unit to adjust ReadWritePaths to `/srv/ftp/releases`
```
[[projects]]
name = "dvzrv/test"
job_name = "build"
#name = "archlinux/releng"
#job_nam...
* install arch-release-promotion
* Create directory `/srv/ftp/releases`
* Override systemd unit to adjust ReadWritePaths to `/srv/ftp/releases`
```
[[projects]]
name = "dvzrv/test"
job_name = "build"
#name = "archlinux/releng"
#job_name = "secure_build"
metrics_file = "metrics.txt"
output_dir = "output"
releases = [
]
[projects.sync_config]
#directory = "/srv/ftp/releases/"
directory = "/srv/ftp/TESTDIRECTORY/"
backlog = 4
temp_in_sync_dir = true
```
Call arch-release-sync -p dvzrv/testhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/404Ansible /etc/postfix/users on mail.archlinux.org2021-10-24T14:52:24ZKristian KlausenAnsible /etc/postfix/users on mail.archlinux.orgAll the mailboxes are currently managed manually on mail.al.org, we should manage them with Ansible.All the mailboxes are currently managed manually on mail.al.org, we should manage them with Ansible.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/412Migrate forwards in postfix/users to Sieve2023-01-21T08:06:35ZKristian KlausenMigrate forwards in postfix/users to SieveWe have some staff acc on mail.archlinux.org using `/etc/postfix/users` for forwarding mails to their personal mail address.
Long-term we want to Ansible the `users` file so they can't stay there and staff can't change the forwarding.
...We have some staff acc on mail.archlinux.org using `/etc/postfix/users` for forwarding mails to their personal mail address.
Long-term we want to Ansible the `users` file so they can't stay there and staff can't change the forwarding.
Using Sieve would also solve the issue of probably-spam getting forwarded and affecting our mail reputation.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/417Update access to (password protected) staff-only channels via ansible2021-11-13T12:31:11ZDavid RungeUpdate access to (password protected) staff-only channels via ansibleThe current way of dealing with access to staff-only channels on libera.chat is very static (i.e. password protected channels).
It would be beneficial to add and revoke access for these channels based on an "infrastructure as code" appr...The current way of dealing with access to staff-only channels on libera.chat is very static (i.e. password protected channels).
It would be beneficial to add and revoke access for these channels based on an "infrastructure as code" approach, so that this may be updated in the regular onboarding/offboarding workflow, as well as "on demand" (e.g. additional nicks, changes in nicks, etc.).
The upside to this is to have one place where staff users may be added/ add themselves to a channel based on a simple merge request and not requiring a password protection anymore. Changes to per channel files could be auto-assigned to the specific founders or members of a given channel for acknowledgement.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/450Use Loki's recording rules to create fancy graphs for the mirros2022-04-18T20:02:27ZKristian KlausenUse Loki's recording rules to create fancy graphs for the mirroshttps://grafana.com/docs/loki/latest/rules/#recording-rules
Recording rules can be used to "parse" the nginx access logs for the mirror and create fancy graphs. Ex: number of packages downloaded split by repository (core, extra and comm...https://grafana.com/docs/loki/latest/rules/#recording-rules
Recording rules can be used to "parse" the nginx access logs for the mirror and create fancy graphs. Ex: number of packages downloaded split by repository (core, extra and community), traffic ratio for the mirrors backing our geo mirror etc..https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/473Create dedicated keycloak/GitLab user for gluebuddy2022-10-24T21:58:52ZLevente Polyakanthraxx@archlinux.orgCreate dedicated keycloak/GitLab user for gluebuddyWe should follow principle of least privilege and single purpose service accounts. Which means we should create a dedciated user `gluebuddy` for keycloak/GitLab instead of reuse the GitLab token of `arch-packaging-bot`.
In case any token...We should follow principle of least privilege and single purpose service accounts. Which means we should create a dedciated user `gluebuddy` for keycloak/GitLab instead of reuse the GitLab token of `arch-packaging-bot`.
In case any token/credentials get leaked anywhere, this would make it tremendously easier for incident response analysis to understand where its coming from compared to one service account's token being used all around the places.Levente Polyakanthraxx@archlinux.orgLevente Polyakanthraxx@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/476Allow users to delete their own Keycloak account2023-02-06T22:47:35ZJelle van der WaaAllow users to delete their own Keycloak accountAs the GDPR tells us a user should be able to delete their own account, we should allow this in keycloak. See the keyclaok docs:
https://www.keycloak.org/docs/latest/server_admin/#proc-allow-user-to-delete-account_server_administration_...As the GDPR tells us a user should be able to delete their own account, we should allow this in keycloak. See the keyclaok docs:
https://www.keycloak.org/docs/latest/server_admin/#proc-allow-user-to-delete-account_server_administration_guidehttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/484Decommission svn2git jobs, checkouts and github repository2022-11-20T02:38:09ZLevente Polyakanthraxx@archlinux.orgDecommission svn2git jobs, checkouts and github repositoryhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/497Improve syncrepo-template.sh to allow rsync for lastupdate check2023-03-04T17:11:04ZAnton Hvornumtorxed@archlinux.orgImprove syncrepo-template.sh to allow rsync for lastupdate checkAs suggested in https://bugs.archlinux.org/task/71617.
I'm transferring the suggestion here as it's more relevant to keep track of the code changes via GitLab issues.
It also allows tagging the change to a recorded issue here.
https://...As suggested in https://bugs.archlinux.org/task/71617.
I'm transferring the suggestion here as it's more relevant to keep track of the code changes via GitLab issues.
It also allows tagging the change to a recorded issue here.
https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/roles/syncrepo/files/syncrepo-template.sh currently requires a http/https URL to enable lastupdate awareness.
Nikke suggests to enhance the script to allow using rsync to check lastupdate freshness.
With a reasonably modern rsync, this is IMHO easiest done with something similar to:
```bash
needupd="$(rsync -n -R -t --no-motd --out-format='%n' --timeout=60 rsync://source.site/lastupdate /destination/dir/)"
if [ $? = 0 -a -z "$needupd" ]; then
echo "Up 2 date, only rsync lastsync"
else
echo "Need update, do full rsync"
fi
```
Granted, it doesn't do the full compare of the lastupdate content, but it's good enough as a freshness check in the general use case.
With this addition, the script would work out of the box by only supplying it with an rsync URL making setup easier for mirror admins, and removes the requirement that the master site provides lastupdate via http/https. While the latter might not be stricly needed as the Arch Linux master provides it via http(s), it might help lastupdate/lastsync adoption by other projects which would be a Good Thing for us mirror admins.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/502Spam Filter2023-03-25T21:32:59ZEric WallerSpam FilterIs there any chance to get a spam filter for the github projects? I am sick of wading through things like this
https://gitlab.archlinux.org/archlinux/service-desks/forum/-/issues/779Is there any chance to get a spam filter for the github projects? I am sick of wading through things like this
https://gitlab.archlinux.org/archlinux/service-desks/forum/-/issues/779https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/504Setup crates.io account for Arch Linux organization2023-10-02T04:36:51ZLevente Polyakanthraxx@archlinux.orgSetup crates.io account for Arch Linux organizationWe want to be the owner of some crates on crates.io, hence setup an account for Arch Linux organization and store access tokens in the vault.
Crates:
- [ ] alpm, alpm-sys and almp-utils
- [x] alpm-types (https://gitlab.archlinux.org/arc...We want to be the owner of some crates on crates.io, hence setup an account for Arch Linux organization and store access tokens in the vault.
Crates:
- [ ] alpm, alpm-sys and almp-utils
- [x] alpm-types (https://gitlab.archlinux.org/archlinux/alpm/alpm-types/-/merge_requests/26)
- [ ] arch-audit
- [x] arch-repro-status (https://gitlab.archlinux.org/archlinux/arch-repro-status/-/commit/b0df09edb9b660813c0e1be31bed19cdc60e407e)Levente Polyakanthraxx@archlinux.orgLevente Polyakanthraxx@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/506Assign keycloak packager group to devs/tus2023-05-25T15:50:22ZLevente Polyakanthraxx@archlinux.orgAssign keycloak packager group to devs/tusWe need to:
- [ ] extend the onboarding/offboarding templates to add/remove keycloak packager group
- [x] assign all devs to the "Core Package Maintainers" keycloak group
- [x] assign all TUs to the "Package Maintainers" keycloak groupWe need to:
- [ ] extend the onboarding/offboarding templates to add/remove keycloak packager group
- [x] assign all devs to the "Core Package Maintainers" keycloak group
- [x] assign all TUs to the "Package Maintainers" keycloak groupLeonidas SpyropoulosLeonidas Spyropouloshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/507Enable nested virtualization for the VM runners2023-04-12T14:15:47ZKristian KlausenEnable nested virtualization for the VM runnersFor faster performance when testing VMs with QEMU it could be beneficial to enable nested virtualization for the VM runners.
At the time of writing the following projects have a use case for this:
- arch-boxes (for testing the built ima...For faster performance when testing VMs with QEMU it could be beneficial to enable nested virtualization for the VM runners.
At the time of writing the following projects have a use case for this:
- arch-boxes (for testing the built images and testing with VirtualBox in the future)
- archiso (perhaps for testing the ISOs in the future)
- mkinitcpio (perhaps for testing mkinitcpio in the future)
- infrastructure (@dvzrv is working on some Ansible Molecule stuff where VMs could be useful)
Some concerns have been raised earlier about the safety of this, so let's discuss it :)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/512GitLab Sourcegraph integration2023-05-20T19:58:42ZLevente Polyakanthraxx@archlinux.orgGitLab Sourcegraph integrationWe'd like to have search across all our packages and all our GitLab projects (GitLab advanced search is tracked in #159). One of the more advanced options out there would be sourcegraph which provides both, a free basic community version...We'd like to have search across all our packages and all our GitLab projects (GitLab advanced search is tracked in #159). One of the more advanced options out there would be sourcegraph which provides both, a free basic community version as well as an advanced enterprise version.
- example search: https://sourcegraph.com/search
- Tour: https://docs.sourcegraph.com/getting-started/tour
sourcegraph.com [Features](https://docs.sourcegraph.com/code_search/explanations/features):
- Use regular expressions and exact queries to perform full-text searches.
- Perform language-aware structural search on code structure.
- Search any branch and commit, with no indexing required.
- Search commit diffs and commit messages to see how code has changed.
- Narrow your search by repository and file pattern.
- Smart Search query assistant.
- Use search contexts to search across a set of repositories at specific revisions.
- Curate saved searches for yourself or your org.
- Use code monitoring to set up notifications for code changes that match a query.
- View language statistics for search results.
Tasks
- [ ] investigate into sourcegraph and its setup plus peak into the enterprise features.
- [ ] If we are happy about that solution, the PL should reach out to sourcegraph and try to secure some sponsoring.
- https://about.sourcegraph.com/pricinghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/514createlinks runs with an error2023-05-20T14:18:31ZJelle van der Waacreatelinks runs with an errorAfter the extra/community merge `createlinks` now fails on this big package.
```
May 20 11:36:07 gemini.archlinux.org createlinks[1895389]: extra/x86_64: intel-oneapi-basekit-2023.1.0.46401-1
May 20 11:38:22 gemini.archlinux.org createl...After the extra/community merge `createlinks` now fails on this big package.
```
May 20 11:36:07 gemini.archlinux.org createlinks[1895389]: extra/x86_64: intel-oneapi-basekit-2023.1.0.46401-1
May 20 11:38:22 gemini.archlinux.org createlinks[2281454]: sort: write failed: 'standard output': No space left on device
May 20 11:38:22 gemini.archlinux.org createlinks[2281454]: sort: write error
```
Previously it worked:
```
May 16 00:07:02 gemini.archlinux.org createlinks[1562202]: community/x86_64: mupdf-tools-1.22.1-2
May 16 00:07:03 gemini.archlinux.org createlinks[1562202]: community/x86_64: dev86-0.16.21-7
May 16 00:07:03 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-basekit-2023.1.0.46401-1
May 16 00:09:20 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-compiler-dpcpp-cpp-runtime-2023.1.0-1
May 16 00:09:24 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-compiler-dpcpp-cpp-runtime-libs-2023.1.0-1
May 16 00:09:24 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-compiler-shared-2023.1.0-1
May 16 00:09:25 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-compiler-shared-runtime-2023.1.0-1
May 16 00:09:28 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-compiler-shared-runtime-libs-2023.1.0-1
May 16 00:09:29 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-dev-utilities-2021.9.0_44447-2
May 16 00:09:29 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-dpcpp-debugger-2023.1.0_43513-1
May 16 00:09:40 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-mkl-2023.1.0_46342-1
May 16 00:09:55 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-openmp-2023.1.0-1
May 16 00:09:55 gemini.archlinux.org createlinks[1562202]: community/x86_64: intel-oneapi-tbb-2021.9.0-1
May 16 00:09:56 gemini.archlinux.org createlinks[1562202]: community/x86_64: libdwarf-1:0.6.0-2
May 16 00:09:56 gemini.archlinux.org createlinks[1562202]: community/x86_64: coin-or-lemon-1.3.1-4
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/516uptimerobot seems broken2023-05-28T19:48:14ZLeonidas Spyropoulosuptimerobot seems brokenWe got no uptimerobot alerts for some time now.We got no uptimerobot alerts for some time now.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/517Allow all packagers access to [mutlilib]2023-07-23T10:37:21ZJelle van der WaaAllow all packagers access to [mutlilib]Agreed upon here: https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/thread/MWJ4CR32RFIIZJRJ5J72HOZVJLGM4WKF/
- [x] Roll out dbscripts changes https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/...Agreed upon here: https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/thread/MWJ4CR32RFIIZJRJ5J72HOZVJLGM4WKF/
- [x] Roll out dbscripts changes https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/40
- [x] Infra changes https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/723
- [ ] Drop multilib group from gemini
- [ ] Configure allowed_repos in Archweb @jellehttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/520Leftovers from the Git Migration2023-11-16T16:49:25ZChristian HeuselLeftovers from the Git MigrationThis issue is created so that the leftover points are not lost in hedgedocs somewhere.
Points taken from:
- https://md.archlinux.org/kF4a9cx2Sqe5ESusZw85zg?both#leftover-todo%E2%80%99s-from-migration
- https://md.archlinux.org/utjjQ-bQTs...This issue is created so that the leftover points are not lost in hedgedocs somewhere.
Points taken from:
- https://md.archlinux.org/kF4a9cx2Sqe5ESusZw85zg?both#leftover-todo%E2%80%99s-from-migration
- https://md.archlinux.org/utjjQ-bQTsipIKntPrpf8g#3-post-rollout
@jelle wanted to have a look at the stuff that still needs doing. :rocket:
---
- [ ] Check package push rules: `ipxe` `wireless-regdb`
- [ ] repos.archlinux.org: drop tu group
- [ ] state repo: https://gitlab.archlinux.org/archlinux/packaging/state
- [ ] include README in the repository https://md.archlinux.org/QX8kjaPyTbWdaoaFgWE_-Q#
- [x] set the logo: https://mathphys.info/~chris/state_repo.png
- [x] remove the community package pool
- [x] drop community sources `/srv/ftp/sources/community`
- [x] Remove `/srv/repos/svn-{community,packages}`
- [x] Drop community-debug pools `/srv/ftp/community-debug`
- [x] Remove `/srv/svn`
- [x] check svntogit user and owner files and remove it
```
svn-packages:x:1080:1080::/home/svn-packages:/bin/bash
svn-community:x:1081:1081::/home/svn-community:/bin/bash
svntogit:x:1084:1084::/srv/svntogit:/sbin/nologin
```
- [x] Drop this after all packages sources have been updated /srv/ftp/other/community/
https://gitlab.archlinux.org/search?group_id=11323&scope=blobs&search=sources.archlinux.org%2Fother%2Fcommunity Create to do list.
To-Do https://archlinux.org/todo/move-sources-from-srvftpothercommunity-to-extra/
- [ ] Resolve broken symlinks
`find /srv/ftp -xtype l`
https://gitlab.archlinux.org/archlinux/infrastructure/-/work_items/484
- [x] check svntogit user and owner files and remove it
- [ ] asp:
- [x] drop to AUR
- [ ] archive https://github.com/archlinux/asphttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/531Split repos server to repos and archive2024-03-27T22:47:50ZLeonidas SpyropoulosSplit repos server to repos and archive# Split repos.archlinux.org to repos.archlinux.org and archive.archlinux.org
## Rationale
- The current repos.al.org server has 4x hdd setup in RAID10 to support both archive and repos. With the merge of extra and community the extra fo...# Split repos.archlinux.org to repos.archlinux.org and archive.archlinux.org
## Rationale
- The current repos.al.org server has 4x hdd setup in RAID10 to support both archive and repos. With the merge of extra and community the extra folder in /srv/pool has many files in there and when mirrors request to rsync from it it takes some seconds to generate the list to send before it starts syncing. During that period there's a high chance a new package to be updated resulting in rewrite of the db.ta.gz files which lead to deleting those from the mirrors (this is an edge case in POSIX which btrfs actually doesn't handle but not a bug). By splitting the server and moving the repos into a 2x SSD raid1 setup it will make it much faster to operate and less likely to trigger the bug. Better technical explanation in btrfs ML [1], [2]
- The second reason to split it is for seperation of concerns of current server for repos and archive.
## Considerations
- the move to the archive can happen async (every 5 mins) on a systemd timer which will rsync from the repos /srv/archive to the actual archive (addition only, not delete)
- every day a separate systemd timer will clean up old pkg files from /src/archive older than 24h
- mirrors which mirror archive need to include change their scripts to additionally mirror from the new server
- borg backups needs to be updated to handle new server
## Plan
- Trigger backup
- Create new repos server with 2x SSD raid1
- Mirror the current repos.al.org except archive
- Downtime for Package Maintainers for 2 hours to change the DNS (no change in dbscripts)
- Repurpose current repos.al.org to archive.al.org
- Deploy the two systemd timers/services to async copy and cleanup packages to archive in repos.al.org
- Deploy mirror archive scripts to rsync from archive.al.org
- Inform other mirrors who sync archive about the change
- Trigger backup
## Diagram
![repos-migration](/uploads/eb31abf30054661a5a940b72796f087f/repos-migration.png)
[1]: https://lore.kernel.org/linux-btrfs/00ed09b9-d60c-4605-b3b6-f4e79bf92fca@foutras.com/
[2]: https://lore.kernel.org/linux-btrfs/ZP8AWKMVYOY0mAwq@debian0.Home/#t
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=9b378f6ad48c