We are running short on storage and don't really need 3 months of log,
so reduce it to 2 months instead of expanding the disk.

Loki v2.4.0 introduced a common config section[1][2] making the config
much smaller and better defaults.

So stripped loki.yaml down, removed options where the default is "good
enough" and tweaked retention settings per the upgrade notes[3].

So the config should basically be the example loki.yaml[4] + compactor,
retention enabled and a few tweaks (ex: compress chunks with zstd).

[1] https://github.com/grafana/loki/releases/tag/v2.4.0
[2] https://grafana.com/docs/loki/latest/configuration/#common_config
[3] https://grafana.com/docs/loki/latest/upgrading/#the-single-binary-no-longer-runs-a-table-manager
[4] https://github.com/grafana/loki/blob/v2.4.0/cmd/loki/loki-local-config.yaml

Jan Alexander Steffens (heftig) authored 3 years ago and Kristian Klausen committed 3 years ago

- When loki hits 5.5G RSS, aggressively reclaim its memory. This puts
  it under memory pressure.
- When loki hits 6G RSS, invoke the kernel OOM killer.
- When loki is under severe memory pressure, have systemd-oomd kill it.

It simplifies it a bit.

Fixes: 49091634 ("loki: Bump the retention period to 3 months")

We want to be able to look at trends and past incidents.

The default is very restrictive.

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

A extra access_log entry was added with the following commands:
$ cd roles
$ grep -lr access_log | xargs -P 1 -n 1 sed -i '/access_log/ s/\(.*\)\( \)\(\(reduced\|main\);$\)/\1 \3\n\1.json json_\3/'

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Fix #263