- When loki hits 5.5G RSS, aggressively reclaim its memory. This puts
  it under memory pressure.
- When loki hits 6G RSS, invoke the kernel OOM killer.
- When loki is under severe memory pressure, have systemd-oomd kill it.

- Allow killing of any cgroup if swap is about to run out. When swap is
  at least 90% full, oomd will kill a cgroup, tending to kill the cgroup
  with the highest swap usage.
- Allow killing cgroups of logged in users under severe memory pressure.
  When a cgroup is stalled waiting for memory at least 60% of the time,
  for at least 30 seconds, oomd will attempt to kill it or one of its
  descendant cgroups.

prometheus: Add monitoring and alert for orphan and foreign packages

Closes #332

See merge request archlinux/infrastructure!389

Leonidas Spyropoulos authored 3 years ago and Jelle van der Waa committed 3 years ago

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

Leonidas Spyropoulos authored 3 years ago and Jelle van der Waa committed 3 years ago

Closes: #332

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

improve onboarding documentation

Closes #226

See merge request archlinux/infrastructure!397

implements #226

make diabonas dev

Closes #346

See merge request archlinux/infrastructure!396

implements #346

Terraform uptimerobot monitors

See merge request archlinux/infrastructure!388

Add our uptimerobot to terraform so it's managed in code and we can
easily extend it. This currently only adds our to be monitored sites and
leaves the status page as is now.

Deleting resources on uptimerobot will cause terraform unable to run
see: https://github.com/louy/terraform-provider-uptimerobot/issues/82

References: #209

archmanweb: fix path to favicon.ico

Closes archmanweb#33

See merge request archlinux/infrastructure!393

Rate limit man.archlinux.org

See merge request archlinux/infrastructure!390

Limit man.archlinux.org to reduce the impact of one potential abuser on
the availability. This returns too many request when running oha with 10
connections. This does not fully negate the issue, later the abuser
should be automatically fail2ban'd.

arch_boxes_sync: Fix old releases cleanup not working due to NUL character

See merge request archlinux/infrastructure!391

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

--output-delimiter="" apparently means use the NUL character[1].

[1] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=0e46753d7c9519d7378cd3a4e0951a36ac32ffe7

Document how to whitelist some metrics for the public Grafana instance.

Closes: #334

The repository has been migrated to Gitlab as cgit will be removed in
due time.

aurweb: Don't call PHP for serving static content

Closes #318

See merge request archlinux/infrastructure!372

Leonidas Spyropoulos authored 3 years ago and Jelle van der Waa committed 3 years ago

Closes: #318

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

monitoring: add pie chart and multi suite select to rebuilderd

See merge request archlinux/infrastructure!386

Allow multiple suites to be selected as variables and respect those in
the percentage and status queries.

prometheus_exporters: Add AUR exporter

Closes #323

See merge request archlinux/infrastructure!375

Also send the metrics to dashboards.archlinux.org.

Fix #323