The plan is to use this for creating metrics from the nginx log lines
(e.g. requests per second).

[1] https://grafana.com/docs/loki/latest/alert/#recording-rules

Missed this earlier for the upgrade to Loki 3.0.

Loki 3.0 requires this to be set when retention is enabled.

This is needed to upgrade to Loki 3.0.

The cloud server running Loki isn't powerful enough to do aggressive
query splitting; it makes queries execute slower and generates a lot
more queries, resulting in "too many outstanding requests" in recent
Loki versions. (Maybe a bug?)

The default log level is "info" which logs too much to the journal.

Since upgrading to Loki 2.5, the query frontend was trying to connect to
the wireguard address instead of 127.0.0.1. This appears to be caused by
upstream commit c0bec07e0de1a23cb72e8834069 ("Loki: Implement common net
interface/instance addr") which now defaults to "" instead of 127.0.0.1.

Doubt the Loki team is interested in our usage data, and it's also a
risk to the confidentiality of our logs (even though it's anonymous).

< wCPO> foutrelis: v12 fixes the "all chunks in the same directory"-issue

One month of logs is about 40G and suffices for our log retrieval needs.

We are running short on storage and don't really need 3 months of log,
so reduce it to 2 months instead of expanding the disk.

Loki v2.4.0 introduced a common config section[1][2] making the config
much smaller and better defaults.

So stripped loki.yaml down, removed options where the default is "good
enough" and tweaked retention settings per the upgrade notes[3].

So the config should basically be the example loki.yaml[4] + compactor,
retention enabled and a few tweaks (ex: compress chunks with zstd).

[1] https://github.com/grafana/loki/releases/tag/v2.4.0
[2] https://grafana.com/docs/loki/latest/configuration/#common_config
[3] https://grafana.com/docs/loki/latest/upgrading/#the-single-binary-no-longer-runs-a-table-manager
[4] https://github.com/grafana/loki/blob/v2.4.0/cmd/loki/loki-local-config.yaml

Fixes: 49091634 ("loki: Bump the retention period to 3 months")

We want to be able to look at trends and past incidents.

The default is very restrictive.

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Fix #263