This enables us to calculate the cache hit ratio, which may help
determine whether more caching would be beneficial.

Please note that this only counts requests for which caching is enabled
(e.g. {fastcgi,proxy}_cache is configured), e.g. for static served files
cache_status will be "".

[1] http://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_cache_status

Mainly because we are curious. The data may also be used to decide if we
want to drop older versions of TLS.

The plan is to use this for creating metrics from the nginx log lines
(e.g. requests per second).

[1] https://grafana.com/docs/loki/latest/alert/#recording-rules

Missed this earlier for the upgrade to Loki 3.0.

Loki 3.0 requires this to be set when retention is enabled.

This is needed to upgrade to Loki 3.0.

Fixes: 26f289b7 ("Capitalize the first letter of all task names")

ansible-lint 6.5.0 complains about:

  name: All names should start with an
        uppercase letter. (name[casing])

These are used to signal the start of the document in a stream of many
documents. As Ansible only supports one YAML document per file this is
unnecessary. About a third of our YAML documents already lacked these.

The cloud server running Loki isn't powerful enough to do aggressive
query splitting; it makes queries execute slower and generates a lot
more queries, resulting in "too many outstanding requests" in recent
Loki versions. (Maybe a bug?)

The default log level is "info" which logs too much to the journal.

Since upgrading to Loki 2.5, the query frontend was trying to connect to
the wireguard address instead of 127.0.0.1. This appears to be caused by
upstream commit c0bec07e0de1a23cb72e8834069 ("Loki: Implement common net
interface/instance addr") which now defaults to "" instead of 127.0.0.1.

Doubt the Loki team is interested in our usage data, and it's also a
risk to the confidentiality of our logs (even though it's anonymous).

< wCPO> foutrelis: v12 fixes the "all chunks in the same directory"-issue

One month of logs is about 40G and suffices for our log retrieval needs.

We are running short on storage and don't really need 3 months of log,
so reduce it to 2 months instead of expanding the disk.

Loki v2.4.0 introduced a common config section[1][2] making the config
much smaller and better defaults.

So stripped loki.yaml down, removed options where the default is "good
enough" and tweaked retention settings per the upgrade notes[3].

So the config should basically be the example loki.yaml[4] + compactor,
retention enabled and a few tweaks (ex: compress chunks with zstd).

[1] https://github.com/grafana/loki/releases/tag/v2.4.0
[2] https://grafana.com/docs/loki/latest/configuration/#common_config
[3] https://grafana.com/docs/loki/latest/upgrading/#the-single-binary-no-longer-runs-a-table-manager
[4] https://github.com/grafana/loki/blob/v2.4.0/cmd/loki/loki-local-config.yaml

Jan Alexander Steffens (heftig) authored 3 years ago and Kristian Klausen committed 3 years ago

- When loki hits 5.5G RSS, aggressively reclaim its memory. This puts
  it under memory pressure.
- When loki hits 6G RSS, invoke the kernel OOM killer.
- When loki is under severe memory pressure, have systemd-oomd kill it.

It simplifies it a bit.

Fixes: 49091634 ("loki: Bump the retention period to 3 months")

We want to be able to look at trends and past incidents.

The default is very restrictive.

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

A extra access_log entry was added with the following commands:
$ cd roles
$ grep -lr access_log | xargs -P 1 -n 1 sed -i '/access_log/ s/\(.*\)\( \)\(\(reduced\|main\);$\)/\1 \3\n\1.json json_\3/'

Kristian Klausen authored 4 years ago and Jelle van der Waa committed 3 years ago

Fix #263