infrastructure issueshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues2021-01-02T14:36:11Zhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/258Borg alertmanager notification doesn't show last backup2021-01-02T14:36:11ZJelle van der WaaBorg alertmanager notification doesn't show last backup```
description = Borg has not backuped for more than 24 hours. Last backup made on 1970-01-25 02:26:42.552 +0000 UTC
``````
description = Borg has not backuped for more than 24 hours. Last backup made on 1970-01-25 02:26:42.552 +0000 UTC
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/257FluxBB replacement2024-01-08T16:28:21ZKristian KlausenFluxBB replacementFluxBB isn't maintained so we need to migrate to something else.
https://lists.archlinux.org/pipermail/arch-devops/2019-October/000297.html
```
The below feedback has been solicited from the current Arch Linux Forum
team in October 2019...FluxBB isn't maintained so we need to migrate to something else.
https://lists.archlinux.org/pipermail/arch-devops/2019-October/000297.html
```
The below feedback has been solicited from the current Arch Linux Forum
team in October 2019, based on the premise that the DevOps team desires to
replace the current forum software.
The points have been summarized for readability, and anonymized to ensure
the points are considered based on their merit alone.
Our preference is that a demo system is set up prior to a final decision
being made so we can evaluate the moderation tools available to assess
their suitability for our community behaviors.
General Criteria
===============
* I hope that we will stick to simple and not deviate too far from the
functionality of a basic forum.
* I would opt for disabling all unnecessary features
* I am very skeptical of any system of community voting for relevance.
* The forum should remain a technical resource and not devolve into
karma farming.
* I would also strongly oppose any sort of obligatory 2-factor
authentication.
* Reasonably good functionality on limited bandwidth and/or text-mode
browsers.
* I would like better tools to help identify duplicate accounts
* Perhaps tools that can auto-detect behavior such as post blanking when
it starts.
* I'm not a fan of endless scrolling or javascript "features" either,
and I don't see any point of social media things like "likes" on a
technical support forum.
* KISS.
* One of the things that I hate most in the world is continuous scroll
down rather than pagination.
* Functionally, FluxBB does everything I expect a forum software to do.
* It shouldn't contain post-voting facilities
* I dislike sub-threading where one can reply to other posts inline. If
reddit is any indication it leads to incredibly messy and hard to follow
discussions.
* I liked myBB and misago the most from some quick browsing through the
examples.
* Based on several criteria above, Discourse seems like a particularly
poor fit for our community.
* "Oh dear god, anything but discourse."
Alternatives
===============
phpBB
* https://www.phpbb.com/
* Seems still actively developed.
* Is listed on some "top 10" forum software for 2019 sites.
* It is very similar to fluxBB.
Thredded
* https://thredded.org/
* It is actively developed
* Free & open-source
* Has a 'modern' visual style (while being readily
customizable/themeable)
Can be used with a SSO
Paginates
Demos I've found load quickly and are easy to read on a TUI.
MyBB
* https://mybb.com/
* Maintained
* Has an LDAP plugin.
Farum
* https://flarum.org/features/
* Seems to be similar to discourse; claims to be lightweight.
* Has SSO, API, and Anti-spam.
* Still beta release.
Vanilla
* https://open.vanillaforums.com/
* Maintained
* Open-core/Freemium model.
* Has SSO.
Misago
* https://misago-project.org/
* "Looks OK from a bit of playing around on their site. Not sure what
moderation tools look like."
Discourse
* https://www.discourse.org/
* There seems to be a lot of bells and whistles that increase its
complexity but I have no experience with it.
* Discourse ... design philosophy seems to be in the mindset of catering
to the majority who have high powered systems, good network connections,
and "modern" (GUI) browsers. The Arch community has always seemed to work
hard to not needlessly marginalize those who have limited bandwidth, old
hardware, or text-mode browsers; so the use of a tool like discourse
strikes me as either antithetical to our existing community ethos, or a
sign of a significant change in direction.
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/252Use terraform for managing GitLab/GitHub projects2021-01-15T00:48:17ZKristian KlausenUse terraform for managing GitLab/GitHub projectsSee the ["New Official Project"](.gitlab/issue_templates/New Official Project.md) issue template, for what needs to be configured.
https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs/resources/project
https://registry.t...See the ["New Official Project"](.gitlab/issue_templates/New Official Project.md) issue template, for what needs to be configured.
https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs/resources/project
https://registry.terraform.io/providers/hashicorp/github/latest/docs/resources/repositoryhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/250Create prometheus mailq exporter + alert2022-02-26T22:17:19ZKristian KlausenCreate prometheus mailq exporter + alertSomeone (*cough*) broke Postfix (!249) on at least aur.archlinux.org, we should setup monitoring of the mail queue length.Someone (*cough*) broke Postfix (!249) on at least aur.archlinux.org, we should setup monitoring of the mail queue length.https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/243Stop archweb services connecting over postgresql SSL2020-12-18T22:11:15ZJelle van der WaaStop archweb services connecting over postgresql SSLCurrently we have postgresql over SSL to allow archweb services on mirrors and on repos.archlinux.org for:
* mirrorcheck
* mirrorresolv
* reporead
* updating rsyncd whitelist from archweb db
To simplify things we can consider adding a ...Currently we have postgresql over SSL to allow archweb services on mirrors and on repos.archlinux.org for:
* mirrorcheck
* mirrorresolv
* reporead
* updating rsyncd whitelist from archweb db
To simplify things we can consider adding a full fledged API to archweb to be able to POST mirror results and GET allowed ips for whitelisting.
For reporead we could rsync the databases to archlinux.org, if this doesn't cause any issues with archweb reading the db while it's being rsync'd?https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/242Setup ICMP monitorings of all hosts2022-04-10T20:51:42ZKristian KlausenSetup ICMP monitorings of all hostsWe can use the [blackbox exporter](https://github.com/prometheus/blackbox_exporter).
This should help detecting misconfiguration (!199).We can use the [blackbox exporter](https://github.com/prometheus/blackbox_exporter).
This should help detecting misconfiguration (!199).https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/241Process DMARC and TLS-RPT reports automatically2020-12-17T12:33:17ZKristian KlausenProcess DMARC and TLS-RPT reports automaticallyWe should process all the DMARC and TLS-RPT reports automatically. Maybe we can use node_exporter textfile collector?
Inspiration for the DMARC logic: https://github.com/domainaware/parsedmarcWe should process all the DMARC and TLS-RPT reports automatically. Maybe we can use node_exporter textfile collector?
Inspiration for the DMARC logic: https://github.com/domainaware/parsedmarchttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/225Improve offboarding2021-01-14T19:31:48ZFrederik SchwanImprove offboardingAs mentioned [here](https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/221#note_8080),
the offboarding doesn't work yet.
* homdir on gemini is not deleted
* user on mail is not removed
* user still can login to imap
* home...As mentioned [here](https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/221#note_8080),
the offboarding doesn't work yet.
* homdir on gemini is not deleted
* user on mail is not removed
* user still can login to imap
* homedir on mail is not removed
* if the user used the IMAP server, there is still a mailbox
* it's unclear what to do with the existing data and mailbox (delete? forward?)https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/218write export tool to automatically pull password hashes from keycloak2020-12-29T21:35:01ZFrederik Schwanwrite export tool to automatically pull password hashes from keycloakMoved from: #50 and #210
Project repo: https://gitlab.archlinux.org/archlinux/mail-credential-syncer
Since many members of arch-devops work with rust and it's good security characteristics, this tool shall be written in rust (https://...Moved from: #50 and #210
Project repo: https://gitlab.archlinux.org/archlinux/mail-credential-syncer
Since many members of arch-devops work with rust and it's good security characteristics, this tool shall be written in rust (https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/210#note_6535 contains a very dirty POC in Go).
Three config parameters:
- Path to mapping file for keycloak UUID -> arch mail address on local FS
- Keycloak hostname
- Post-receive script
Implementation:
- [ ] Use inotify to receive events when the mapping file changes
- [ ] Subscribe to pw change events for the keycloak user attribute `mail_password_hash`
Whenever an event fires:
- iterate over mapping
- get pw hash from keycloak
- check if the hash is valid and contains no malicious input (probably with a regex)
- export dovecot and opensmtpd version of virtual user file
- backup old config files
- run post receive script
- when the post receive script failes, restore old config files
- report error via e-mail (or prometheus?)Kristian KlausenKristian Klausenhttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/217Store the (virtual) mail password in keycloak2023-02-28T21:46:08ZFrederik SchwanStore the (virtual) mail password in keycloakWe will store a password hash in the Keycloak user attributes. The attrs are writable <=> the user can manage his own account. This implies that we shall not save any data to the attrs that the user is not allowed to change. But his own ...We will store a password hash in the Keycloak user attributes. The attrs are writable <=> the user can manage his own account. This implies that we shall not save any data to the attrs that the user is not allowed to change. But his own mail password is fine.
- attribute to use: `mail_password_hash`
- no other attribute is saved
- preferably use the java on the server side for hashing
We also want to do some basic password validation (length > X). Keycloak already has something builtin which we should be able to use. We just need to expose it.
The custom attribute needs to be added to the Account Console. The new account console is written in React and use the REST endpoints so we will need to edit that to use our custom endpoints.
For custom provider deployment see [here](https://gitlab.archlinux.org/archlinux/infrastructure/-/commit/5ac750c909357f3cda5223b475643fc164410a1d) for reference.
TODOs:
- [ ] Modify theme to add a custom attribute "mail_password_hash" to the account management console
- [ ] Implement a domain extension to provide custom REST endpoint for bcrypt with cost 12 and 2b variant (use Bouncy Castle library - make sure to use the crypt encoded version implemented in OpenBSD)
- [ ] Implement an add "password-validate" REST endpoint which use the internal Keycloak API
- [ ] Ensure the password hash attribute in Keycloak can be modified (via the templating engine)
- [x] Discuss which pw hash algo to use (bcrypt)
- [ ] Update [mail credential syncer script](https://gitlab.archlinux.org/archlinux/mail-credential-syncer/-/blob/master/src/main.rs#L160) to use 2b variant
- [ ] write manual how to change the mail pw
- [ ] ping all arch mail users to store their passwords
References:
- https://www.keycloak.org/docs/latest/server_development/#_extensions_rest
- https://github.com/keycloak/keycloak/tree/master/examples/providers/rest
- https://github.com/keycloak/keycloak/tree/master/examples/providers/domain-extension
- https://www.keycloak.org/docs/latest/server_development/#account-management-console
- https://www.bouncycastle.org/docs/docs1.5on/org/bouncycastle/crypto/generators/OpenBSDBCrypt.htmlIra ¯\_(ツ)_/¯Ira ¯\_(ツ)_/¯https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/214Prepare virtual user setup for dovecot and OpenSMTPD2022-04-11T13:14:46ZFrederik SchwanPrepare virtual user setup for dovecot and OpenSMTPDAt the moment we use one mailbox per unix user on the mail machine. This should be changed to a virtual setup where no users login to the mail machine anymore.
- [ ] create MR for new dovecot and OpenSMTPD config
- [ ] create migration ...At the moment we use one mailbox per unix user on the mail machine. This should be changed to a virtual setup where no users login to the mail machine anymore.
- [ ] create MR for new dovecot and OpenSMTPD config
- [ ] create migration script for dovecot mailboxeshttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/208Switch to ECDSA certificate or provide dual ECDSA/RSA certificates2021-05-01T12:48:37ZKristian KlausenSwitch to ECDSA certificate or provide dual ECDSA/RSA certificatesWith the ECDSA support in Certbot slowly moving forward, I think it is time to create a issue: https://github.com/certbot/certbot/pull/8431 https://github.com/certbot/certbot/pull/8254#pullrequestreview-515564860With the ECDSA support in Certbot slowly moving forward, I think it is time to create a issue: https://github.com/certbot/certbot/pull/8431 https://github.com/certbot/certbot/pull/8254#pullrequestreview-515564860https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/206Let the world mirror our (encrypted) backup2020-12-20T16:35:39ZKristian KlausenLet the world mirror our (encrypted) backup> “Only wimps use tape backup. REAL men just upload their important stuff on ftp and let the rest of the world mirror it.”
> \- Linus Torvalds
Should protect against "rogue employee"..> “Only wimps use tape backup. REAL men just upload their important stuff on ftp and let the rest of the world mirror it.”
> \- Linus Torvalds
Should protect against "rogue employee"..https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/202postfix relayhost configuration prints warnings in log2020-10-26T19:50:59ZJelle van der Waapostfix relayhost configuration prints warnings in log```
/usr/bin/postconf: warning: /etc/postfix/master.cf: undefined parameter: post_queue_smtpd_recipient_restrictions
``````
/usr/bin/postconf: warning: /etc/postfix/master.cf: undefined parameter: post_queue_smtpd_recipient_restrictions
```https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/190Add the pacman public keyring to root user gpg.conf2021-04-28T02:04:15ZGiancarlo RazzoliniAdd the pacman public keyring to root user gpg.confSome operations require the keys of our staff and since the root user keyring is used for ansible, let's add archlinux-keyrings's file /etc/pacman.d/gnupg/pubring.gpg to /root/.gnupg/gpg.conf.Some operations require the keys of our staff and since the root user keyring is used for ansible, let's add archlinux-keyrings's file /etc/pacman.d/gnupg/pubring.gpg to /root/.gnupg/gpg.conf.Giancarlo RazzoliniGiancarlo Razzolinihttps://gitlab.archlinux.org/archlinux/infrastructure/-/issues/188Harden services with systemd sandboxing2020-10-18T13:09:03ZFrederik SchwanHarden services with systemd sandboxingWe can further harden our own systemd services with some options from man systemd.exec
For example:
```
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKern...We can further harden our own systemd services with some options from man systemd.exec
For example:
```
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
```
Later systemd will have an option to show hints about it: https://github.com/systemd/systemd/pull/10701
Migrated from https://kanboard.archlinux.org/project/1/task/103https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/187Add Secure Header for all our sites2021-08-27T19:31:48ZFrederik SchwanAdd Secure Header for all our sitesA good example of a site with correct secure headers is security.archlinux.org and we want to do the same for other sites.
Sites needing extra headers:
- bbs.archlinux.org
- wiki.archlinux.org
- aur.archlinux.org, needs patches in aurw...A good example of a site with correct secure headers is security.archlinux.org and we want to do the same for other sites.
Sites needing extra headers:
- bbs.archlinux.org
- wiki.archlinux.org
- aur.archlinux.org, needs patches in aurweb.
https://securityheaders.com/:
![image](/uploads/3820ca04ebaa3f8b40b325f41fe064ce/image.png)
![image](/uploads/262d6a35ecd72f09040135a6d752d15a/image.png)
![image](/uploads/f93c7ef25316275427b941a3ff1c086a/image.png)
Migrated from: https://kanboard.archlinux.org/project/1/task/120https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/186Create playbook for user removal2021-01-05T21:28:48ZFrederik SchwanCreate playbook for user removalCreate a playbook to deactivate/remove a user from our servers. For example when a Trusted User or Developer resigns.
- Deactive user in Archweb and move it to the corresponding fellows group and disable the user (this requires a manage...Create a playbook to deactivate/remove a user from our servers. For example when a Trusted User or Developer resigns.
- Deactive user in Archweb and move it to the corresponding fellows group and disable the user (this requires a management command from archweb)
- Remove the pubkey from the infrastructure repo
- Remove the user from all servers
- Run the relevant role to remove the users pubkey from our servers
- Optional Create a ticket for keyring
- Optional Update BBS/Bugtracker role (if this is easily scriptable)
- Optional: List packages still signed by this user in the repos (could be an archweb management command as well, combine with deactivation)
Migrated from: https://kanboard.archlinux.org/project/1/task/97https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/185Monitor postgres2021-04-27T16:48:43ZFrederik SchwanMonitor postgresselect/update/insert/delete/... per second
Migrated from: https://kanboard.archlinux.org/project/1/task/51select/update/insert/delete/... per second
Migrated from: https://kanboard.archlinux.org/project/1/task/51https://gitlab.archlinux.org/archlinux/infrastructure/-/issues/184Secure SSH setup, especially on build servers2020-11-02T10:34:55ZFrederik SchwanSecure SSH setup, especially on build servers* https://matrix.org/blog/2019/04/11/security-incident/
* https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg
* https://web.archive.org/web/20190412143901/https://github.com/matrix-org/matrix.org/issues/
* https://doi...* https://matrix.org/blog/2019/04/11/security-incident/
* https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg
* https://web.archive.org/web/20190412143901/https://github.com/matrix-org/matrix.org/issues/
* https://doi.org/10.6028/NIST.IR.7966
- determine needs of our users
- copy packages from build server to orion
- sign packages on build server
- svn?
- anything else?
- implement solutions
- disable all unneeded access; implement all useful security ideas from the NIST paper
- verify that new setup is secure
Migrated from: https://kanboard.archlinux.org/project/1/task/132