Kind of sensitive information that doesn't need to be available to all
hosts.

Change docs/ssh-known_hosts.txt to be partially managed by Ansible, so
custom entries can be added to the top of the file. Use the new format
to write down the host keys of our two borg hosts.

Using GitLab's official backup tool takes too much time and, more
importantly, space; /srv/gitlab is a bit over 430G but backing it
up nearly exhausts its 1TB volume.

As we're creating btrfs snapshots and backing those up with borg, it
seems unnecessary to also create tarballs of the same data. GitLab's
documentation mentions snapshots as a viable backup strategy, and to
the restored system it should seem like recovering from a power loss.

[1] https://docs.gitlab.com/ee/raketasks/backup_restore#alternative-backup-strategies

Collects the smart data using smartctl and outputs them in the
textcollector dir. This expects smartd to be configured to regularly
self tests on a regular interval to detect if a disk is broken.

These are already known (so no need to hide them) and are fairly static
(so variables are more of a hindrance) so it's better to use the actual
usernames in the documentation. Also, simplify the first example given.

New username; separate and longer account manager + storage passwords.

Also, have to use --remote-path=borg1 when interacting with rsync.net.

prometheus-borg-textcollector is no longer started by timer, but instead
defines a WantedBy= relationship with the borg-backup{,-offsite} service.

Jelle van der Waa authored 4 years ago and Sven-Hendrik Haase committed 3 years ago

Document how we backup our databases/gitlab instances.

Disabled in:
0ae67c4a ("postfix: Disable STARTTLS Submission (port 587)")

CPU: Intel Xeon E5-2620 -> E-2288G
Disk: 2x~1TB -> 2x~500GB

The role for the clients is named postfix_null (per [1]) and it's much
simpler and cleaner than the postfix role. I hope can cleanup the
postfix role at a later date.

[1] http://www.postfix.org/STANDARD_CONFIGURATION_README.html#null_client

Add a default rate limit for 20 req/s for the uwsgi endpoint and
automatically ban users who reach this limit. The nginx-limit-req rule
does not ban users who reach the rss limit as these are not likely DoS
attempts.

The port was removed in:
4729ba40 ("postfix: Remove special "fast-path" smtpd")

https://lists.archlinux.org/pipermail/arch-dev-public/2021-July/030471.html

Fix #86

This offers improved separation between the server backups and should
avoid bumping against the storage box 10 concurrent connection limit.

Fixes: archlinux/infrastructure#362

nginx, certbot, postfix and mailman are still missing and the DNS is
still pointing to luna.

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

[1] https://archlinux.org/news/move-of-official-irc-channels-to-liberachat/

Leonidas Spyropoulos authored 3 years ago and Jelle van der Waa committed 3 years ago

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

Document how to whitelist some metrics for the public Grafana instance.

Closes: #334

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Fix #263

The file should not be on the main domain as it adds unnecessary
complexity to the archweb role and there is a bigger chance that we
unintentionally break connectivity checking (which has happened in the
past[1][2]).

This doesn't remove the file from the main domain[3], as we need to ship
a updated NetworkManager package first.

[1] https://www.reddit.com/r/archlinux/comments/keai0g/does_anyone_know_if_this_is_normal/
[2] https://www.reddit.com/r/gnome/comments/ke9ytm/network_manager_popup/
[3] http://www.archlinux.org/check_network_status.txt

Fix #239

This adds a collaborative markdown editor as newly offered service which
is available via login for all Arch Linux Staff with an option to allow
anonymous edits by users (not default). Users are managed via keycloak
and require the Staff role to be allowed in, non staff keycloak users
currently will receive an internal server error due to an upstream
issue.