The redirects are now done by the `redirects` role.

Remove arch32 mirror role

See merge request !352

We no longer mirror arch32 on our servers and this role is currently
broken.

Now that logs are gathered centrally, the team is complaining about the
volume of logs from this server.

Group security/out of date packages alerts

Closes #191

See merge request !351

We want to get notifications of pacman/arch-audit notifications grouped
as otherwise we'll be spammed with ~ X emails for every host.

Closes: #191

prometheus: Make alertmanager.yml only readable by root and alertmanager

See merge request !350

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

It contains secrets, so it shouldn't be world readable.

promtail: Give access to the logrotated nginx access.log files

See merge request !344

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

The files are initially created by nginx as 0644/http:root, but when
logrotate rotates the files it creates the new files as 0640/http:log,
which promtail can't read.

Fix the issue by adding the log group as a supplementary group.

Resolve the users.hosts key not being defined when rolling out the
root_ssh role.

Give klausenbusk root access to {bugs,monitoring}.al.org

See merge request !342

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

klausenbusk is our new newest Junior DevOp and he needs some access:
* bugs.al.org for helping with migrating Flyspray tasks to GitLab
* monitoring.al.org for setting up centralized logging

Revert "Remove NM connectivity check file from al.org"

See merge request !346

This is causing issues for a small business, which can't reach their
"remote systems" anymore due to NM reporting "limited access".
We should be able to revert this in 1-2 weeks.

This reverts commit b909fa58.

I found it a bit short earlier.

It was somewhat broken before and even had a duplicate key.

Loki keeps logs it returns in ram, resulting in the oom killer on 2GB's
of ram.

By default the user-agent is Go-http-client/2.0 which isn't identifyable
in our loki logs. https://github.com/prometheus/blackbox_exporter/issues/555

As our grafana now contains Loki logs, we don't want non devops to view
logs which potentially contain sensitive data. As Grafana does not have
a system to easily restrict data sources to roles we use Keycloak.

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

Using just / works but Grafana logs four lines for every request.

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

A extra access_log entry was added with the following commands:
$ cd roles
$ grep -lr access_log | xargs -P 1 -n 1 sed -i '/access_log/ s/\(.*\)\( \)\(\(reduced\|main\);$\)/\1 \3\n\1.json json_\3/'

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Fix #263

Ensure unbound is used where we want it and removed all other places

Closes #234

See merge request !325

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

unbound is only used if dns_servers is explicit set to 127.0.0.1, which
isn't the case for any of these systems.

Fix #234

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

For spam checking it is recommend to use our own recursive resolver[1]
to avoid rate limiting by using a public resolver.

unbound is already installed but the system wasn't configured to use it.

[1] https://rspamd.com/doc/faq.html#resolver-setup

Fix nginx alias traversal

Closes #291

See merge request !334

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

[1] https://github.com/yandex/gixy/blob/641060d6355fbb5bd71695928a2bf14a9bcb8bf2/docs/en/plugins/aliastraversal.md

Fix #291

Re introduce the arch-audit rule as arch-audit no longer reports false
positives from [testing]. Lax the high cpu alert as our mediawiki
instance is perfectly fine running on 85% CPU for some time, and lax our
disk will fill within X alert as our borg backups generate enough data
in a short time to trigger the 4 hour alarm.