As our grafana now contains Loki logs, we don't want non devops to view
logs which potentially contain sensitive data. As Grafana does not have
a system to easily restrict data sources to roles we use Keycloak.

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Using just / works but Grafana logs four lines for every request.

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

A extra access_log entry was added with the following commands:
$ cd roles
$ grep -lr access_log | xargs -P 1 -n 1 sed -i '/access_log/ s/\(.*\)\( \)\(\(reduced\|main\);$\)/\1 \3\n\1.json json_\3/'

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Fix #263

Ensure unbound is used where we want it and removed all other places

Closes #234

See merge request !325

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

unbound is only used if dns_servers is explicit set to 127.0.0.1, which
isn't the case for any of these systems.

Fix #234

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

For spam checking it is recommend to use our own recursive resolver[1]
to avoid rate limiting by using a public resolver.

unbound is already installed but the system wasn't configured to use it.

[1] https://rspamd.com/doc/faq.html#resolver-setup

Fix nginx alias traversal

Closes #291

See merge request !334

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

[1] https://github.com/yandex/gixy/blob/641060d6355fbb5bd71695928a2bf14a9bcb8bf2/docs/en/plugins/aliastraversal.md

Fix #291

Re introduce the arch-audit rule as arch-audit no longer reports false
positives from [testing]. Lax the high cpu alert as our mediawiki
instance is perfectly fine running on 85% CPU for some time, and lax our
disk will fill within X alert as our borg backups generate enough data
in a short time to trigger the 4 hour alarm.

Remove NM connectivity check file from al.org

See merge request !312

With the ping.al.org domain added in[1] and a updated networkmanager
package shipped[2], we can now remove the file from the main domain.

[1] 498d5304 ("Merge branch 'ping' into 'master'")
    fabccd0f (""Move" NM connectivity check file to a subdomain")
[2] https://github.com/archlinux/svntogit-packages/commit/fb573170fc63c18f64eb4f155ed64a966a61b037

Setup Pages for new bugs.archlinux.org snapshot service[1]

Closes #303

See merge request !343

Fix #303

[1] https://gitlab.archlinux.org/archlinux/archlinux-bugs-snapshotter

Fix off-/onboarding inconsistent and add #archlinux-staff instructions

See merge request !341

This allows us to get proper certificates for loki which will run on logging.archlinux.org
on the same machine as monitoring.archlinux.org.

Add klausenbusk's SSH key

See merge request !340

Levente Polyak authored 4 years ago and Sven-Hendrik Haase committed 4 years ago

Strictly speaking this is not a devops duty, but I believe this is
currently the best place to ensure we won't ever forget to create a
keyring revocation ticket for an offboarded former staff.

Our qemu TCG builds generate plenty of cpu usage over time to be
excluded from warnings.

Our dbscripts role expects the default ssh key to be username + '.pub'.

Add additional ssh key for hashworks, set prefered shell

See merge request !336

To use a key as backup key to be deployed on all hosts.

dbscripts: Issue a certificate for rsync.archlinux.org

See merge request !338

Put man.archlinux.org into node_exporters group

See merge request !337

I think this is all that is missing to actually start monitoring
man.archlinux.org.