infrastructure issueshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues2021-06-08T12:26:49Zhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/130Consider making a staging Keycloak for toying around2021-06-08T12:26:49ZSven-Hendrik Haasesvenstaro@archlinux.orgConsider making a staging Keycloak for toying aroundTesting/rolling out new login flows should be tested in a staging Keycloak env. before rolling them out in production. After #39 we can't just keep toying around with the production instance.Testing/rolling out new login flows should be tested in a staging Keycloak env. before rolling them out in production. After #39 we can't just keep toying around with the production instance.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/120Add Webauthn policy to Keycloak2020-09-30T17:01:46ZSven-Hendrik Haasesvenstaro@archlinux.orgAdd Webauthn policy to KeycloakCurrently, the Webauthn policy gets overridden on every single `terraform apply` which sucks a lot!
We urgently need to manage this via Terraform properly. It needs these PRs to be merged prior:
- https://github.com/mrparkers/terraform-...Currently, the Webauthn policy gets overridden on every single `terraform apply` which sucks a lot!
We urgently need to manage this via Terraform properly. It needs these PRs to be merged prior:
- https://github.com/mrparkers/terraform-provider-keycloak/pull/356
- ~~https://github.com/mrparkers/terraform-provider-keycloak/pull/357~~
- https://github.com/mrparkers/terraform-provider-keycloak/pull/385 (not as urgent as 356)
@klausenbusk Could you take a look at your PRs over at the provider's side?Kristian KlausenKristian Klausenhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/112Figure out how to handle Keycloak and 2FA resets2023-07-03T23:47:27ZKristian KlausenFigure out how to handle Keycloak and 2FA resetsResetting the password allows the user (or attacker) to add a new 2FA device without 2FA, which makes 2FA basically useless. We can fix the issue, by disabling the `Reset OTP` element in the `Reset Credentials` flow ([relevant doc](https...Resetting the password allows the user (or attacker) to add a new 2FA device without 2FA, which makes 2FA basically useless. We can fix the issue, by disabling the `Reset OTP` element in the `Reset Credentials` flow ([relevant doc](https://www.keycloak.org/docs/latest/server_admin/#forgot-password)), but it breaks `Forgot Password` for some(all?) users.
We also need to set a policy for 2FA reset. Is it just bad luck?
Relevant upstream issues:
- https://issues.redhat.com/browse/KEYCLOAK-13134
- https://issues.redhat.com/browse/KEYCLOAK-14640
Relevant meeting notes: https://gitlab.archlinux.org/archlinux/infrastructure/-/wikis/meetings/2020-09-05Sven-Hendrik Haasesvenstaro@archlinux.orgLevente Polyakanthraxx@archlinux.orgSven-Hendrik Haasesvenstaro@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/94Switch matrix authentication to Keycloak2021-04-15T12:38:33ZJelle van der WaaSwitch matrix authentication to KeycloakUpstream has [documentation](https://github.com/matrix-org/synapse/blob/5c5516f80ef08dc07c1a7c297614f455c1bc75d4/docs/openid.md#keycloak) for setting up Matrix with Keycloak.
- [ ] How do we migrate/switch over existing users?
- [ ] Imp...Upstream has [documentation](https://github.com/matrix-org/synapse/blob/5c5516f80ef08dc07c1a7c297614f455c1bc75d4/docs/openid.md#keycloak) for setting up Matrix with Keycloak.
- [ ] How do we migrate/switch over existing users?
- [ ] Implement Keycloak/Matrix integrationIra ¯\_(ツ)_/¯Ira ¯\_(ツ)_/¯https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/74review new secure runner setup2020-10-22T23:36:21ZSven-Hendrik Haasesvenstaro@archlinux.orgreview new secure runner setupWe now have a hardware-based secure runner with virtualization support for virtual box and kvm and a secondary virtualized secure runner with no special access.
secure-runner1.archlinux.org (tags secure-virtualbox and secure-kvm) is the...We now have a hardware-based secure runner with virtualization support for virtual box and kvm and a secondary virtualized secure runner with no special access.
secure-runner1.archlinux.org (tags secure-virtualbox and secure-kvm) is the virtualization-support enabled one while secure-runner2.archlinux.org (tag secure-general) is the generic one. The latter has less attack surface and should be preferred for most projects.
This needs to be reviewed.Jelle van der WaaLevente Polyakanthraxx@archlinux.orgJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/72Harden AUR systemd services and fpm configuration2023-02-19T13:25:34ZJelle van der WaaHarden AUR systemd services and fpm configurationThe systemd unit's for the AUR can be hardened to sandbox them further:
- [ ] aurweb-popupdate
- [ ] aurweb-tuvotereminder
- [ ] aurweb-pkgmaint
- [ ] aurweb-mkpkglists
- [ ] aurweb-git (does not require any network interaction!)
- [ ] a...The systemd unit's for the AUR can be hardened to sandbox them further:
- [ ] aurweb-popupdate
- [ ] aurweb-tuvotereminder
- [ ] aurweb-pkgmaint
- [ ] aurweb-mkpkglists
- [ ] aurweb-git (does not require any network interaction!)
- [ ] aurweb-aurblup
- [ ] aurweb-memcached
The php-fpm configuration can disable some functions which can be used by attackers to execute arbitrary commands:
`php_admin_value[disable_functions] = passthru, exec, proc_open, shell_exec, system, popen`https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/69Notes for GitLab + Keycloak announcement2020-07-30T20:08:13ZSven-Hendrik Haasesvenstaro@archlinux.orgNotes for GitLab + Keycloak announcementWe should properly document how we announce the official use of GitLab + Keycloak along with any notes for our staff and what the change means to them. We should also note how NOT to use it.
Notes in no particular order:
- Do not upload...We should properly document how we announce the official use of GitLab + Keycloak along with any notes for our staff and what the change means to them. We should also note how NOT to use it.
Notes in no particular order:
- Do not upload secrets.
- Do not use GitLab to automatically build packages.
- Secure runners are trusted and can be used to create official artifacts (such as archiso, archboxes, etc). They can only be hand-assigned by DevOps to specific projects and projects needs will be discussed on a one-by-one basis with project owners.
- all CI variables that contain secrets like credentials must be marked as "protect variable" and "mask variable"
- DevOps must review a projects .gitlab-ci.yml and also make the project owner aware that all jobs on protected branches/tags must always have a runner tag selector "secure" declared to avoid protected branches to be run on none secure runners which would allow shared runners to access the secrets we try to separate from such runners. Also only jobs should select the secure runners that actually need to access secure credentials in the CI variables. Default jobs that do not need any secrets and don't publish any artifacts must not be run on the secure runners.
- define modus operandi for secret rotation etc in case a secure runner is once assigned to unsecure jobs (f.e. failure of a .gitlab-ci.yml)
- if we need a privileged runner, we should have both, an unprivileged that is used for everything that doesn't need privileged docker and a special one with the privileged tag that only runs jobs that need privileged docker.Sven-Hendrik Haasesvenstaro@archlinux.orgLevente Polyakanthraxx@archlinux.orgSven-Hendrik Haasesvenstaro@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/68Activate Keycloak logging2020-07-27T08:50:15ZSven-Hendrik Haasesvenstaro@archlinux.orgActivate Keycloak loggingPretty easy. Just enable saving of events via Terraform. See https://www.keycloak.org/docs/latest/server_admin/#auditing-and-eventsPretty easy. Just enable saving of events via Terraform. See https://www.keycloak.org/docs/latest/server_admin/#auditing-and-eventsSven-Hendrik Haasesvenstaro@archlinux.orgSven-Hendrik Haasesvenstaro@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/67Keycloak shortcomings2021-08-03T19:59:39ZSven-Hendrik Haasesvenstaro@archlinux.orgKeycloak shortcomingsThis issue serves to track Keycloak shortcomings that we've found.
- Email-squatting is possible as users can change their email to any random other email which allows you to block emails. This is not a security issue but it's pretty an...This issue serves to track Keycloak shortcomings that we've found.
- Email-squatting is possible as users can change their email to any random other email which allows you to block emails. This is not a security issue but it's pretty annoying. https://issues.redhat.com/browse/KEYCLOAK-6455
- ~~No audit logs. These exist but we have to set them up properly: https://www.keycloak.org/docs/latest/server_admin/#auditing-and-events~~
- ~~Doesn't allow multiple OTP devices. https://issues.redhat.com/browse/KEYCLOAK-14297~~
- ~~Users can't add WebAuthn providers in the account management page. https://issues.redhat.com/browse/KEYCLOAK-14298~~
- Users should be forced to an OTP before removing an OTP device. https://issues.redhat.com/browse/KEYCLOAK-14296
- Allow users to have multiple emails. https://issues.redhat.com/browse/KEYCLOAK-14295
- ~~Check that "forgot password" via email does not reset MFA which makes MFA basically useless https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/112~~
- Not being able to set your default OTP/Security key https://issues.redhat.com/browse/KEYCLOAK-18957Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/66Investigate using Tower/AWX2020-11-16T01:54:56ZGiancarlo RazzoliniInvestigate using Tower/AWXThis issue is to investigate implementing a central place for running books while also solving the [finer grained control](#64).This issue is to investigate implementing a central place for running books while also solving the [finer grained control](#64).https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/65Decide on a password manager2021-06-05T02:06:42ZGiancarlo RazzoliniDecide on a password managerWe need to take out of the vault the passwords that are not required for roles and playbooks to run, and that we could use to store passwords in the future, while also having granularity on who has access to which password.
* Requiremen...We need to take out of the vault the passwords that are not required for roles and playbooks to run, and that we could use to store passwords in the future, while also having granularity on who has access to which password.
* Requirements:
* has to be used remote
* entry level per people
* sso integration or gpg?
* for storing "team" credentials such as PyPi, keycloak admin creds
* Solutions to look into:
* password-storage
* bitwarden/bitwarden_rs
* keepass
* gopasshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/64Think about finely grained secret access2022-05-07T16:33:54ZSven-Hendrik Haasesvenstaro@archlinux.orgThink about finely grained secret accessWe currently slap everything into the Ansible Vault. This is not optimal. We need more finely grained access while still allowing people to get their work done. Investigate which tools allow for that still of managing credentials. Perhap...We currently slap everything into the Ansible Vault. This is not optimal. We need more finely grained access while still allowing people to get their work done. Investigate which tools allow for that still of managing credentials. Perhaps something like bitwarden-rs?https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/55Consider removing firewall rules and re-applying them for postgresql2022-10-22T00:56:15ZJelle van der WaaConsider removing firewall rules and re-applying them for postgresqlWe open Postgresql firewall ports for specific ports, but never remove them. Research whether it's possible to remove all postgresql related firewall rules and re-add them.We open Postgresql firewall ports for specific ports, but never remove them. Research whether it's possible to remove all postgresql related firewall rules and re-add them.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/54Remove fukawi access2021-01-14T19:17:24ZSven-Hendrik Haasesvenstaro@archlinux.orgRemove fukawi accessfukawi resigned. Let's remove access.fukawi resigned. Let's remove access.Jelle van der WaaJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/53Write onboarding/offboarding guidelines/checklists for various roles2022-04-10T20:57:19ZSven-Hendrik Haasesvenstaro@archlinux.orgWrite onboarding/offboarding guidelines/checklists for various rolesWe currently have no onboarding or offboarding guidelines nor checklists for our roles. This is bad. Write guidelines for these roles:
- [ ] DevOps
- [ ] TU
- [ ] Dev
- [ ] Security Team
- [ ] Bug Wrangler
- [ ] Forum Admin
- [ ] Wiki A...We currently have no onboarding or offboarding guidelines nor checklists for our roles. This is bad. Write guidelines for these roles:
- [ ] DevOps
- [ ] TU
- [ ] Dev
- [ ] Security Team
- [ ] Bug Wrangler
- [ ] Forum Admin
- [ ] Wiki Admin
- [ ] IRC Operators/Mods
Ask people who are currently involved in those roles to contribute some roles.
Previous work:
* [Developer Checklist](https://wiki.archlinux.org/index.php/DeveloperWiki:Developer_Checklist)
* [TODO list for a new TU](https://wiki.archlinux.org/index.php/AUR_Trusted_User_Guidelines#TODO_list_for_new_Trusted_Users)
* [TODO retiring a TU](https://wiki.archlinux.org/index.php/AUR_Trusted_User_Guidelines#TODO_list_retiring_a_Trusted_User)
We need onboarding automation as this is currently painful:
* Does mailman allow automation of onboarding/adding users (staff list for example)hashworkshashworks@archlinux.orghashworkshashworks@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/50Modernize mail server setup2022-10-22T23:50:48ZSven-Hendrik Haasesvenstaro@archlinux.orgModernize mail server setup### Prelude
- [x] Switch SPF to softfail (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/6278f6688a5c4ecdf74a4b1fd4b5bbfa69ce4b01)
- [x] Remove Postgrey (!43)
- [x] Setup SPF for HELO name ([RFC 7208 section 10.1.3](https...### Prelude
- [x] Switch SPF to softfail (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/6278f6688a5c4ecdf74a4b1fd4b5bbfa69ce4b01)
- [x] Remove Postgrey (!43)
- [x] Setup SPF for HELO name ([RFC 7208 section 10.1.3](https://tools.ietf.org/html/rfc7208#section-10.1.3)) (!122)
- [x] Switch to Rspamd (!42)
- [x] Use Rspamd DKIM signing module for signing instead of OpenDKIM (#213, !147)
- [x] Stop relaying of luna via mail.
- [ ] create main opensmtpd config (#215)
- [ ] create opensmtpd config for relayhosts (#216)
- [ ] Prepare virtual user setup for dovecot and OpenSMTPD (#214)
- [ ] Store the (virtual) mail password in keycloak (#217)
- [x] harden used IMAP and SMTP ports ([RFC 8314](https://tools.ietf.org/html/rfc8314), #219)
- [x] migrate existing services to use implicit TLS for SMTP Submission (!207)
- [ ] store alias and sender file in Ansible (encrypted in the vault)
- [ ] store keycloak UUID -> arch mail address mapping in ansible (encrypted in the vault)
- [ ] write export tool to automatically pull password hashes from keycloak (#218)
- [ ] create keycloak client with minimal permissions for the export tool (https://gitlab.archlinux.org/archlinux/mail-credential-syncer/-/issues/3)
- [x] Setup MTA-STS in testing mode and SMTP TLS Reporting ([RFC 8460](https://tools.ietf.org/html/rfc8460)) (!191, !231)
- [x] Setup monitoring (!206)
- [ ] https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/365
### Main part
- [ ] Rollout the export tool from #218
- [ ] Replace Postfix by OpenSMTPD on our relaying hosts
- [ ] Switch Dovecot to virtual users
- [ ] Replace Postfix by OpenSMTPD on our main mail server
### Aftermath
- [ ] Switch DMARC to reject
- [ ] Add archlinux.org to rspamd whitelistes: https://github.com/rspamd/maps/tree/master/rspamd (`spf_dkim` + `dmarc`)
- [ ] Setup ARC
- [ ] Cleanup OpenSMTPD and Dovecot config if possible
- [ ] remove ssh access for users on the mail host
- [x] Cleanup SPF record (#197, !229)
- [x] Deprecate STARTTLS on Port 587 (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/0ae67c4a64e2892eef58285f39e9575bdbe38268)
- [ ] use floating IPs to keep the spam reputation case we need to migrate the mail server
- [x] Remove old ip addresses from DNSWL
- [ ] Process DMARC and TLS-RPT reports automatically (#241)
- [x] Switch MTA-STS to enforce mode (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/0b87cbfd062d31814c877614a5c2e388ee5eb416)
- [ ] Setup blacklist monitoring
- [ ] Setup mails sent, received, bounced monitoring
- [x] Deprecate POP3 (https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/cf9c92fd346a6f832e5057e305759c35d59692e8)
---
**removed Tasks**
- [ ] Pull users from Keycloak ([somehow](https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/50#note_2248))
- [ ] Dovecot: Switch passdb from [pam](https://doc.dovecot.org/configuration_manual/authentication/pam/) to [passwd-file](https://doc.dovecot.org/configuration_manual/authentication/passwd_file/)
**Original description:**
Our mail server isn't ansibled and it's fairly opaque how everything is setup. Ansible the whole thing, put it on a separate box and modernize it in the process.
Some guides to follow along with a similar stack: https://prefetch.eu/blog/2020/email-server/ and https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/
There's also this: https://wiki.dovecot.org/PasswordDatabase/oauth2
@foxboron mentioned that perhaps OpenSMTPD is inadvisable due to its fairly bad security track record.Frederik SchwanKristian KlausenFrederik Schwanhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/43Set up secure-runner1.archlinux.org2020-07-17T16:21:39ZSven-Hendrik Haasesvenstaro@archlinux.orgSet up secure-runner1.archlinux.orgSven-Hendrik Haasesvenstaro@archlinux.orgLevente Polyakanthraxx@archlinux.orgSven-Hendrik Haasesvenstaro@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/39Open up account registrations to the public2021-09-04T13:21:29ZSven-Hendrik Haasesvenstaro@archlinux.orgOpen up account registrations to the publicThis issue tracks requirements for what we need in order to feel confident to open Keycloak (and therefore GitLab up to the public).
- [x] User registration security audit
- [x] Sensible password restrictions
- [x] Recaptcha for user re...This issue tracks requirements for what we need in order to feel confident to open Keycloak (and therefore GitLab up to the public).
- [x] User registration security audit
- [x] Sensible password restrictions
- [x] Recaptcha for user registration (#35)
- [x] Login flow audit
- [x] GitLab security audit
- [x] Keycloak monitoring (#23)
- [ ] GitLab monitoring (#14)
- [x] Review secure Gitlab runner (#74)
- [x] GitLab secure runner for our own projects with proper restrictions (#32)
- [x] Audit security of public runners (#8)
- [x] Add github.com as identity provider (#2)
- [ ] ~~Add gitlab.com as identity provider (#40)~~
- [x] Allow incoming mail on GitLab (#3)
- [x] Enable Keycloak event logging (#68)
- [x] Finish Arch theme of Keycloak (!28)
- [x] Research fine grade permissions for externals (#9)
- [x] Fix GitLab backups (#118)
- [x] Validate GitLab backups
- [x] Validate Keycloak backupsJelle van der WaaSven-Hendrik Haasesvenstaro@archlinux.orgLevente Polyakanthraxx@archlinux.orgJelle van der Waahttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/38Migrate the wiki to keycloak2024-03-13T15:02:24ZJelle van der WaaMigrate the wiki to keycloakIn the future we want to use keycloak for authentication mediawiki:
- https://www.mediawiki.org/wiki/Extension:OpenID_Connect
- https://stackoverflow.com/questions/16893589/prevent-users-from-changing-their-passwords-in-mediawiki
- http...In the future we want to use keycloak for authentication mediawiki:
- https://www.mediawiki.org/wiki/Extension:OpenID_Connect
- https://stackoverflow.com/questions/16893589/prevent-users-from-changing-their-passwords-in-mediawiki
- https://wiki.archlinux.org/index.php/ArchWiki:Access_levels_and_roles
/cc @pierre @archlinux/teams/wiki/adminshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/36Ensure that all Arch staff and external contributors are forced to use OTP2020-07-17T13:27:55ZSven-Hendrik Haasesvenstaro@archlinux.orgEnsure that all Arch staff and external contributors are forced to use OTPAcceptance criteria:
While logging into Keycloak...
* [x] a normal user isn't forced to set up OTP if they didn't do so before
* [x] a normal user is forced to provide exactly one OTP if they set it up before
* [x] a Staff-role user is...Acceptance criteria:
While logging into Keycloak...
* [x] a normal user isn't forced to set up OTP if they didn't do so before
* [x] a normal user is forced to provide exactly one OTP if they set it up before
* [x] a Staff-role user is forced to set up OTP if they didn't do so before
* [x] a Staff-role user is forced to provide exactly one OTP
* [x] an External Contributor-role user is forced to set up OTP if they didn't do so before
* [x] an External Contributor-role user is forced to provide exactly one OTP
* [x] upon removing an active OTP device from Keycloak, same rules are applied as before when the user didn't have OTP set up at allSven-Hendrik Haasesvenstaro@archlinux.orgSven-Hendrik Haasesvenstaro@archlinux.org