infrastructure issueshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues2023-05-25T15:50:22Zhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/506Assign keycloak packager group to devs/tus2023-05-25T15:50:22ZLevente Polyakanthraxx@archlinux.orgAssign keycloak packager group to devs/tusWe need to:
- [ ] extend the onboarding/offboarding templates to add/remove keycloak packager group
- [x] assign all devs to the "Core Package Maintainers" keycloak group
- [x] assign all TUs to the "Package Maintainers" keycloak groupWe need to:
- [ ] extend the onboarding/offboarding templates to add/remove keycloak packager group
- [x] assign all devs to the "Core Package Maintainers" keycloak group
- [x] assign all TUs to the "Package Maintainers" keycloak groupLeonidas SpyropoulosLeonidas Spyropouloshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/504Setup crates.io account for Arch Linux organization2023-10-02T04:36:51ZLevente Polyakanthraxx@archlinux.orgSetup crates.io account for Arch Linux organizationWe want to be the owner of some crates on crates.io, hence setup an account for Arch Linux organization and store access tokens in the vault.
Crates:
- [ ] alpm, alpm-sys and almp-utils
- [x] alpm-types (https://gitlab.archlinux.org/arc...We want to be the owner of some crates on crates.io, hence setup an account for Arch Linux organization and store access tokens in the vault.
Crates:
- [ ] alpm, alpm-sys and almp-utils
- [x] alpm-types (https://gitlab.archlinux.org/archlinux/alpm/alpm-types/-/merge_requests/26)
- [ ] arch-audit
- [x] arch-repro-status (https://gitlab.archlinux.org/archlinux/arch-repro-status/-/commit/b0df09edb9b660813c0e1be31bed19cdc60e407e)Levente Polyakanthraxx@archlinux.orgLevente Polyakanthraxx@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/502Spam Filter2023-03-25T21:32:59ZEric WallerSpam FilterIs there any chance to get a spam filter for the github projects? I am sick of wading through things like this
https://gitlab.archlinux.org/archlinux/service-desks/forum/-/issues/779Is there any chance to get a spam filter for the github projects? I am sick of wading through things like this
https://gitlab.archlinux.org/archlinux/service-desks/forum/-/issues/779https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/497Improve syncrepo-template.sh to allow rsync for lastupdate check2023-03-04T17:11:04ZAnton Hvornumtorxed@archlinux.orgImprove syncrepo-template.sh to allow rsync for lastupdate checkAs suggested in https://bugs.archlinux.org/task/71617.
I'm transferring the suggestion here as it's more relevant to keep track of the code changes via GitLab issues.
It also allows tagging the change to a recorded issue here.
https://...As suggested in https://bugs.archlinux.org/task/71617.
I'm transferring the suggestion here as it's more relevant to keep track of the code changes via GitLab issues.
It also allows tagging the change to a recorded issue here.
https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/master/roles/syncrepo/files/syncrepo-template.sh currently requires a http/https URL to enable lastupdate awareness.
Nikke suggests to enhance the script to allow using rsync to check lastupdate freshness.
With a reasonably modern rsync, this is IMHO easiest done with something similar to:
```bash
needupd="$(rsync -n -R -t --no-motd --out-format='%n' --timeout=60 rsync://source.site/lastupdate /destination/dir/)"
if [ $? = 0 -a -z "$needupd" ]; then
echo "Up 2 date, only rsync lastsync"
else
echo "Need update, do full rsync"
fi
```
Granted, it doesn't do the full compare of the lastupdate content, but it's good enough as a freshness check in the general use case.
With this addition, the script would work out of the box by only supplying it with an rsync URL making setup easier for mirror admins, and removes the requirement that the master site provides lastupdate via http/https. While the latter might not be stricly needed as the Arch Linux master provides it via http(s), it might help lastupdate/lastsync adoption by other projects which would be a Good Thing for us mirror admins.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/484Decommission svn2git jobs, checkouts and github repository2022-11-20T02:38:09ZLevente Polyakanthraxx@archlinux.orgDecommission svn2git jobs, checkouts and github repositoryhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/476Allow users to delete their own Keycloak account2023-02-06T22:47:35ZJelle van der WaaAllow users to delete their own Keycloak accountAs the GDPR tells us a user should be able to delete their own account, we should allow this in keycloak. See the keyclaok docs:
https://www.keycloak.org/docs/latest/server_admin/#proc-allow-user-to-delete-account_server_administration_...As the GDPR tells us a user should be able to delete their own account, we should allow this in keycloak. See the keyclaok docs:
https://www.keycloak.org/docs/latest/server_admin/#proc-allow-user-to-delete-account_server_administration_guidehttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/473Create dedicated keycloak/GitLab user for gluebuddy2022-10-24T21:58:52ZLevente Polyakanthraxx@archlinux.orgCreate dedicated keycloak/GitLab user for gluebuddyWe should follow principle of least privilege and single purpose service accounts. Which means we should create a dedciated user `gluebuddy` for keycloak/GitLab instead of reuse the GitLab token of `arch-packaging-bot`.
In case any token...We should follow principle of least privilege and single purpose service accounts. Which means we should create a dedciated user `gluebuddy` for keycloak/GitLab instead of reuse the GitLab token of `arch-packaging-bot`.
In case any token/credentials get leaked anywhere, this would make it tremendously easier for incident response analysis to understand where its coming from compared to one service account's token being used all around the places.Levente Polyakanthraxx@archlinux.orgLevente Polyakanthraxx@archlinux.orghttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/450Use Loki's recording rules to create fancy graphs for the mirros2022-04-18T20:02:27ZKristian KlausenUse Loki's recording rules to create fancy graphs for the mirroshttps://grafana.com/docs/loki/latest/rules/#recording-rules
Recording rules can be used to "parse" the nginx access logs for the mirror and create fancy graphs. Ex: number of packages downloaded split by repository (core, extra and comm...https://grafana.com/docs/loki/latest/rules/#recording-rules
Recording rules can be used to "parse" the nginx access logs for the mirror and create fancy graphs. Ex: number of packages downloaded split by repository (core, extra and community), traffic ratio for the mirrors backing our geo mirror etc..https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/417Update access to (password protected) staff-only channels via ansible2021-11-13T12:31:11ZDavid RungeUpdate access to (password protected) staff-only channels via ansibleThe current way of dealing with access to staff-only channels on libera.chat is very static (i.e. password protected channels).
It would be beneficial to add and revoke access for these channels based on an "infrastructure as code" appr...The current way of dealing with access to staff-only channels on libera.chat is very static (i.e. password protected channels).
It would be beneficial to add and revoke access for these channels based on an "infrastructure as code" approach, so that this may be updated in the regular onboarding/offboarding workflow, as well as "on demand" (e.g. additional nicks, changes in nicks, etc.).
The upside to this is to have one place where staff users may be added/ add themselves to a channel based on a simple merge request and not requiring a password protection anymore. Changes to per channel files could be auto-assigned to the specific founders or members of a given channel for acknowledgement.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/412Migrate forwards in postfix/users to Sieve2023-01-21T08:06:35ZKristian KlausenMigrate forwards in postfix/users to SieveWe have some staff acc on mail.archlinux.org using `/etc/postfix/users` for forwarding mails to their personal mail address.
Long-term we want to Ansible the `users` file so they can't stay there and staff can't change the forwarding.
...We have some staff acc on mail.archlinux.org using `/etc/postfix/users` for forwarding mails to their personal mail address.
Long-term we want to Ansible the `users` file so they can't stay there and staff can't change the forwarding.
Using Sieve would also solve the issue of probably-spam getting forwarded and affecting our mail reputation.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/404Ansible /etc/postfix/users on mail.archlinux.org2021-10-24T14:52:24ZKristian KlausenAnsible /etc/postfix/users on mail.archlinux.orgAll the mailboxes are currently managed manually on mail.al.org, we should manage them with Ansible.All the mailboxes are currently managed manually on mail.al.org, we should manage them with Ansible.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/393Implement the arch-release-promotion2021-09-09T18:36:34ZJelle van der WaaImplement the arch-release-promotion
* install arch-release-promotion
* Create directory `/srv/ftp/releases`
* Override systemd unit to adjust ReadWritePaths to `/srv/ftp/releases`
```
[[projects]]
name = "dvzrv/test"
job_name = "build"
#name = "archlinux/releng"
#job_nam...
* install arch-release-promotion
* Create directory `/srv/ftp/releases`
* Override systemd unit to adjust ReadWritePaths to `/srv/ftp/releases`
```
[[projects]]
name = "dvzrv/test"
job_name = "build"
#name = "archlinux/releng"
#job_name = "secure_build"
metrics_file = "metrics.txt"
output_dir = "output"
releases = [
]
[projects.sync_config]
#directory = "/srv/ftp/releases/"
directory = "/srv/ftp/TESTDIRECTORY/"
backlog = 4
temp_in_sync_dir = true
```
Call arch-release-sync -p dvzrv/testhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/385Add documentation for adding a new server2021-08-01T17:07:10ZKristian KlausenAdd documentation for adding a new server1. terraform
1. Add playbook
1. Add to `hosts` (add to the correct groups)
1. Add wireguard keys and address
1. Run `prometheus` role on `monitoring.archlinux.org`
1. mail?1. terraform
1. Add playbook
1. Add to `hosts` (add to the correct groups)
1. Add wireguard keys and address
1. Run `prometheus` role on `monitoring.archlinux.org`
1. mail?https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/382Restrict mail service accounts to mail2021-08-02T12:30:15ZKristian KlausenRestrict mail service accounts to mailhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/352Modernize secrets management2021-06-05T18:11:16ZKristian KlausenModernize secrets management[[_TOC_]]
### Intro
On the [2021-05-20 DevOps meeting](https://gitlab.archlinux.org/archlinux/infrastructure/-/wikis/meetings/2021-05-20#segmented-vault-access) segmenting the vault by using [Ansible's vault IDs](https://docs.ansible.co...[[_TOC_]]
### Intro
On the [2021-05-20 DevOps meeting](https://gitlab.archlinux.org/archlinux/infrastructure/-/wikis/meetings/2021-05-20#segmented-vault-access) segmenting the vault by using [Ansible's vault IDs](https://docs.ansible.com/ansible/latest/user_guide/vault.html#managing-multiple-passwords-with-vault-ids) was discussed, so access could be handed out to Junior DevOps. The idea was scratched as the tooling isn't ideal and we would end up with a lot of vaults.
### Alternatives
Instead it was decided to look at more "modern" alternatives like [Vault](https://www.vaultproject.io/) and [vaultwarden](https://github.com/dani-garcia/vaultwarden) (*Unofficial Bitwarden compatible server written in Rust*).
#### Vaultwarden
[vaultwarden](https://github.com/dani-garcia/vaultwarden) was ditched very early on as it lacks some critical features:
* No OIDC support
* Custom tooling is required for integrating with Ansible
* No Terraform provider
#### Vault
[Vault](https://www.vaultproject.io/) on the other hand, looks like a much better solution:
* Built for secrets
* [Supports OIDC](https://www.vaultproject.io/docs/auth/jwt)
* [Supported by Ansible](https://docs.ansible.com/ansible/latest/collections/community/hashi_vault/hashi_vault_lookup.html)
```yml
{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/hello:value') }}
```
* [Policy support](https://www.vaultproject.io/docs/concepts/policies)
```sh
# This section grants all access on "secret/*". Further restrictions can be
# applied to this broad policy, as shown below.
path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Even though we allowed secret/*, this line explicitly denies
# secret/super-secret. This takes precedence.
path "secret/super-secret" {
capabilities = ["deny"]
}
```
* [Group support](https://www.vaultproject.io/docs/secrets/identity#identity-groups) (ex: assign every user group X if they are part of Keycloak group Y)
* [Audit logging](https://www.vaultproject.io/docs/audit)
* [Web UI](https://learn.hashicorp.com/tutorials/vault/getting-started-ui)
* [Terraform provider](https://registry.terraform.io/providers/hashicorp/vault/latest/docs)
* [Encrypted at rest](https://www.vaultproject.io/docs/concepts/seal)
##### Vault Workflow
<details><summary>Click to expand</summary>
```sh
$ vault login
Complete the login via your OIDC provider. Launching browser to:
https://dev-2i513orw.auth0.com/authorize?client_id=FFXlsY2atr_wfNaF_hMtsE-zTAeTZnu8&nonce=73fe4c11828d3c4eb8c2c270aa1b3ab45ebda490&redirect_uri=http%3A%2F%2Flocalhost%3A8250%2Foidc%2Fcallback&response_type=code&scope=openid&state=965d81adcfad9d83a29659891cee37358c8d45cc
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.mefxZpkwUzGirhakGtQZoez0
token_accessor oGB9LqtSGxiEWp2zz4DLlIBD
token_duration 768h
token_renewable true
token_policies ["default" "reader"]
identity_policies []
policies ["default" "reader"]
token_meta_role reader
$ ansible-playbook playbooks/....
```
Example var:
```yml
vault_monitoring_grafana_client_secret: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/roles/grafana:client_secret') }}"
```
Adding/reading a secret (or use the [web UI](https://learn.hashicorp.com/tutorials/vault/getting-started-ui)):
```shell
$ vault kv put secret/my-secret my-value=s3cr3t
Key Value
--- -----
created_time 2019-06-19T17:20:22.985303Z
deletion_time n/a
destroyed false
version 1
$ vault kv get secret/my-secret
====== Metadata ======
Key Value
--- -----
created_time 2019-06-19T17:20:22.985303Z
deletion_time n/a
destroyed false
version 1
====== Data ======
Key Value
--- -----
my-value s3cr3t
```
</details>
### TODO/questions
- [ ] Is Vault the best solution?
- [ ] Package https://aur.archlinux.org/packages/python-hvac/ ~~and https://github.com/ansible-collections/community.hashi_vault~~ (**Edit**: hashi_vault is installed by the ansible package)
- [x] What do we store in Vault and how do we store it?
- **Decision:** Everything used by Ansible or Terraform (please be aware of the Keycloak credentials)
- [ ] How should we split the secrets? Per role and per host?
- Do we have any secrets not tied to a role or host? Perhaps secrets used by Terraform?
- What secrets do we have?
- Credentials (Hetzner, GitHub etc.)
- Secrets:
- database passwords
- nginx htpasswd (usually metrics)
- internet archive password ? (2FA?)
- more?
- [x] How do we handle credentials? (related #65)
- **Decision:** Postponed for now, use `ansible-vault` in the meantime
- This is not a blocker and we can handle it later
- They should be stored locally\* and we should be able to revoke access (catch-22)
- \* we can't risk losing access due to a broken server
- [x] How often should you be required to reauth with Keycloak for accessing the vault?
- **Decision**: Use a low value for now (`default_lease_ttl=1h`) but allow renewing the vault token for a bit longer (`max_lease_ttl=3h`)
- https://learn.hashicorp.com/tutorials/vault/tokens#ttl-and-max-ttl
- [x] Manual or automatic unsealing?
- **Decision:** Use a script (`misc/unseal-the-vault.sh`) initially and if it is a hassle, reevaluate
- > When a Vault server is started, it starts in a sealed state. In this state, Vault is configured to know where and how to access the physical storage, but doesn't know how to decrypt any of it.
- https://www.vaultproject.io/docs/concepts/seal
- Should we create a script which SSH to the server and unseal the vault?
- or a script which run on the server at boot and unseal the vault automatically?
- [ ] Do we make Vault publicly accessible or via a VPN / ssh tunnel?
- [ ] For vault, check if we can also use it for ssh access via Keycloak?!https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/343Allow dynamically setting the iso location with the install_arch role2021-05-18T21:18:53ZDavid RungeAllow dynamically setting the iso location with the install_arch roleIn https://gitlab.archlinux.org/archlinux/releng/-/issues/11 we are discussing the restructuring of our release artifacts.
One of the projects affected by this, is this repository, as [the install_arch role makes use of the bootstrap im...In https://gitlab.archlinux.org/archlinux/releng/-/issues/11 we are discussing the restructuring of our release artifacts.
One of the projects affected by this, is this repository, as [the install_arch role makes use of the bootstrap image](https://gitlab.archlinux.org/archlinux/infrastructure/-/blob/128edca7ea73992c3cf857366bb4b9b37a6751fa/roles/install_arch/tasks/main.yml#L47) (which will change locations).
After introducing the new directory structure, the role needs to be adapted, before the old release artifacts are phased out after three months.David RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/337Gitlab promtail logs2021-05-15T20:36:26ZJelle van der WaaGitlab promtail logsOur gitlab instance runs nginx in the gitlab container which means the logs are not in their usual place, but in `/srv/gitlab/logs/nginx`.Our gitlab instance runs nginx in the gitlab container which means the logs are not in their usual place, but in `/srv/gitlab/logs/nginx`.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/335Send blackbox exporter metrics to dashboards.al.org2021-05-13T21:58:20ZKristian KlausenSend blackbox exporter metrics to dashboards.al.orgWe need to commit the dashboard first.We need to commit the dashboard first.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/330Add watchdog2021-05-14T23:26:43ZKristian KlausenAdd watchdogWe enabled `RuntimeWatchdogSec=5min` in https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/304, but `systemd` is apparently still "pinging" the watchdog if the system is trashing heavily.
Perhaps we should switch to...We enabled `RuntimeWatchdogSec=5min` in https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/304, but `systemd` is apparently still "pinging" the watchdog if the system is trashing heavily.
Perhaps we should switch to [watchdog<sup>AUR</sup>](https://aur.archlinux.org/packages/watchdog/)? or something more configurable? or https://github.com/troglobit/watchdogd (I also created a issue for PSI support: https://github.com/troglobit/watchdogd/issues/25)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/328Add RateLimit headers for selected endpoints2021-05-11T23:17:30ZKristian KlausenAdd RateLimit headers for selected endpointshttps://datatracker.ietf.org/doc/html/draft-ietf-httpapi-ratelimit-headers-00
Ref: https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/378#note_23518https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-ratelimit-headers-00
Ref: https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/378#note_23518