This reverts commit e2346a9e since it
was supposed to go through review first.

The idea of this is to allow us to graph the number of packages per repository in grafana just like
we're already doing in the archive exporter.

We want to have the repos and the archive on different servers, to
ensure separation of concerns.

The I/O on gemini is also very sluggish, which is likely related to the
archive design (millions of files on spinning rust), and this has
resulted in rsync deleting the package database files due to a race
condtion[1][2]. The does not solve the race condition, but it should
make it less likely.

We also need to rethink the archive design, but for now we are just
isolating the problem.

There is some "brokenness" related to the archive no longer being on the
same server as the repos, which will be fixed in subsequent commits.

[1] https://lore.kernel.org/linux-btrfs/00ed09b9-d60c-4605-b3b6-f4e79bf92fca@foutras.com/
[2] https://lore.kernel.org/linux-btrfs/ZP8AWKMVYOY0mAwq@debian0.Home/

Relates to #531

Robin Candau authored 2 months ago and Robin Candau committed 2 months ago

See https://github.com/ansible/ansible/pull/77644

If the cost exceeds $0, it indicates that we have run out of credit
and/or are doing something wrong, in either case we want to be alerted.

Check the HTTPS DNS records of the following Geo domains:

- geo.mirror.pkgbuild.com
- riscv.mirror.pkgbuild.com

Ensure they return: "1 . alpn=h2,h3 ipv4hint=... ipv6hint=..."

Ref #606

This is already done for the 'sudo' role, but we also have a few more
sudoers files which currently go in unverified.

Signed-off-by: Christian Heusel <christian@heusel.eu>

Fixes: 4e5550a8 ("Decommission bugs.archlinux.org and replace it with a static copy[1]")

Related to #550
Related to #551

Fixes: 4e5550a8 ("Decommission bugs.archlinux.org and replace it with a static copy[1]")
Signed-off-by: Christian Heusel <christian@heusel.eu>

As announced[2][3] the bugtracker has been migrated to gitlab, so
bugs.a.o can be decommissioned and replaced with a static copy[1](to
avoid link rot).

[1] https://gitlab.archlinux.org/archlinux/bugs-archive/
[2] https://archlinux.org/news/bugtracker-migration-to-gitlab-completed/
[3] https://lists.archlinux.org/hyperkitty/list/arch-dev-public@lists.archlinux.org/thread/WYXDTJ3TR2DWRQCDZK44BQDH67IDVGTS/

Fix #550
Fix #551

Fixes the following error logged to the journal:

> prometheus-mysqld-exporter[2247]: ts=2023-12-24T13:16:45.938Z
> caller=exporter.go:173 level=error msg="Error from scraper"
> scraper=slave_status target=localhost:3306 err="Error 1227 (42000):
> Access denied; you need (at least one of) the SLAVE MONITOR privilege(s)
> for this operation"

[1] https://github.com/prometheus/mysqld_exporter/issues/640

The format of the MySQL user credentials has been slightly changed:
https://github.com/prometheus/mysqld_exporter/releases/tag/v0.15.0

There is no reason for this and we very rarely use the console anyway.

Ref #541

With aurweb!743 we expose those gauges directly from /metrics

Signed-off-by: moson <moson@archlinux.org>

This reverts commit 6d5a8767.

Let's try to find out the real reason first. Plus, this might not have
worked well, as the default systemd restart time limit would not catch
repeated restarts.

The gitlab-exporter service occasionally fails with errors like:

  Error: missing: `response data`
  Caused by: MissingData("response data")

Let's see if restarting it on errors automatically helps avoid this.

On asia.mirror.pkgbuild.com, 'smartctl -a --json $disk' has been exiting
with code 64. From smartctl(1) code 64 corresponds to "Bit 6: The device
error log contains records of errors". Since we're not interested in old
errors, ignore it.

This has become outdated (missing new dedicated servers) and its usage
can be replicated by checking if ansible_virtualization_role == "host".

For Ansible ad hoc commands, '!hcloud' can be used to the same effect.

WireGuard was setup to provide a internal network with confidentiality,
authenticity and integrity[1]. This migrate the remaining Prometheus
exporters to use the internal WireGuard network.

[1] 664deb67 ("WireGuard all hosts")

Fix #384

Fixes: 26f289b7 ("Capitalize the first letter of all task names")

ansible-lint 6.5.0 complains about:

  name: All names should start with an
        uppercase letter. (name[casing])

Useful if we wanted to create a Geo-based archive consisting of machines
in the archive_mirrors group (though this will likely not happen because
it'd break archlinux-repro due to the ~4 hour sync delay).

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

These are used to signal the start of the document in a stream of many
documents. As Ansible only supports one YAML document per file this is
unnecessary. About a third of our YAML documents already lacked these.

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

Ensure all the nameservers serving the geo.mirror.pkgbuild.com subzone
are working and respond with a valid resource record to A/AAAA queries.

Resolve issues when a smart self test is in progress and a disk is nvme
which has no ata smart table.

Collects the smart data using smartctl and outputs them in the
textcollector dir. This expects smartd to be configured to regularly
self tests on a regular interval to detect if a disk is broken.

After=network-online.target is likely insufficient by itself.

Fixes: 104d08a1 ("archweb: Put most services after network-online.target")
Fixes: c844d0cb ("Split storage box monitoring into new text collector")

This was previously monitored as part of the borg text collector, but
now that it only runs after each backup (instead of hourly) the stats
from monitoring.archlinux.org do not remain accurate for long. Switch
back to hourly checks of the storage box's disk usage by adding a new
text collector just for this purpose.

Instead of gathering borg statistics every hour or so, run the text
collector script only once after each borg-backup service finishes.

Also split the borg text collector script into two similar scripts,
where each one gathers borg statistics for its respective borg host.

Doing this in an attempt to be kind to our Borg hosts in cases where the
prometheus-borg-textcollector.timer is restarted on all hosts and avoids
having all machines querying the Borg hosts within the same minute. Only
downside is that the timers will trigger every 75-ish minutes instead of
exactly every hour, but this should not be a problem.

The fail2ban exporter exports the amount of bans per jail.

Thorben Günther authored 3 years ago and Kristian Klausen committed 3 years ago

Add number of pacnew/pacsave files and print non explicit installed
optdepends as orphans as well.