archivetools will add missing packages to the "packages" directory in
the archive[1], but it is only done once a day, so if a package is
updated more than once in that timeframe, the earlier version will not
be added to the archive.

For this reason, we need to sync archived packages from repos.a.o to
gemini.a.o, so we don't lose packages, but also because we want the
archived packages to show up faster on the archive server than once a
day.

The package files could already exist on gemini, created by
archivetools, in which case rsync just ignores the files
(--ignore-existing).

The archived package files are deleted on repos.a.o after three days, so
there is some leeway to stop archive-cleanup.timer if the sync breaks
for whatever reason.

[1] https://gitlab.archlinux.org/archlinux/archivetools/-/blob/d5cbd5a5a9454eecd5c46b64ebc08d5087a25f2d/archive.sh#L141-L200

Relates to #531

We want to have the repos and the archive on different servers, to
ensure separation of concerns.

The I/O on gemini is also very sluggish, which is likely related to the
archive design (millions of files on spinning rust), and this has
resulted in rsync deleting the package database files due to a race
condtion[1][2]. The does not solve the race condition, but it should
make it less likely.

We also need to rethink the archive design, but for now we are just
isolating the problem.

There is some "brokenness" related to the archive no longer being on the
same server as the repos, which will be fixed in subsequent commits.

[1] https://lore.kernel.org/linux-btrfs/00ed09b9-d60c-4605-b3b6-f4e79bf92fca@foutras.com/
[2] https://lore.kernel.org/linux-btrfs/ZP8AWKMVYOY0mAwq@debian0.Home/

Relates to #531

The last bump was to a commit signed by him.

Fixes: 28f8ba48 ("dbscripts: Upgrade to latest commit")

Relates to: 701c1d01

/srv/repos/lock is the lock file location configured via a dbscripts
config. Only accessible for users in junior-packager group, a lot better
would be if sourceballs could write a lock file into
/run/sourceballs.lck as it is unrelated to /srv/repos.

Since 257 DynamicUser sets PrivateTmp=disconnected making debuginfod
unable to read/write to /var/tmp/ properly  hampering debuginfod's
functioning.

You can't `systemctl reload debuginfod` after installing the systemd
unit we need a daemon-reload.

This was apparently hosted on the long gone "apollo" server[1], and when
archweb was migrated to a dedicated cloud VM, it was changed to a
redirect to the main site (archlinux.org)[2][3].

It may have made sense at the time, but now four years later there is no
reason for keeping this around.

I guess dev.archlinux.org was something similar to what pkgbuild.com is
today ("Public HTML server" for staff), but only for developers.

[1] f6c3af0e ("Merge branch 'apollo_decomission' into 'master'")
[2] 824fb084 ("tf-stage1/archlinux: Change DNS records for the archweb migration and also increase the machine size")
[3] 9800d023 ("roles/archweb: Create domain redirects for the domains that point to specific archweb sub urls.")

Somehow these changes were not directly applied even though the role
reloads the prometheus config.

Fixes: 10475a62 ("prometheus: Alert if a build hosts is OOM for 12h")
Signed-off-by: Christian Heusel <christian@heusel.eu>

Fixes: 4159a61f ("dbscripts: switch to Git packaging")

Signed-off-by: Christian Heusel <christian@heusel.eu>

There is not much value in knowing when one of our build hosts has no
more memory left as all of them have plenty of swap available.
Additionally these rules trigger quite often even for short spikes.

Signed-off-by: Christian Heusel <christian@heusel.eu>

Link: https://lists.archlinux.org/archives/list/staff@lists.archlinux.org/message/3JNJQJ7O3LBIXST2EFBCAKM2HCHXJZUM/


Signed-off-by: Christian Heusel <christian@heusel.eu>

Signed-off-by: Christian Heusel <christian@heusel.eu>

This was added more than 7 years ago[1] and has not been relevant for a
long time.

[1] d32ce421 ("postfix: Remove compat_maps")

Please see the reject_authenticated_sender_login_mismatch option[1] for
more details.

For now service accounts are not restricted in any way, this should be
improved in the further.

[1] https://www.postfix.org/postconf.5.html#reject_authenticated_sender_login_mismatch

Fix #365

This removes unnecessary parameters, mostly for one of these reasons:
the value is already the default value, the default value is good
enough, or the parameter is not used in our case.

A bit of reordering/"tidying" was also done.

I think it was added to improve the mail reputation (avoid being
filtered as spam), but at lot has changed since it was added (+5 years
ago), so let's remove it.

From chrony FAQ[1]:
"1.2. Should I prefer chrony over timesyncd if I do not need to run a
server?

Generally, yes.

systemd-timesyncd is a very simple NTP client included in the systemd
suite. It lacks almost all features of chrony and other advanced client
implementations listed on the comparison page. One of its main
limitations is that it cannot poll multiple servers at the same time and
detect servers having incorrect time (falsetickers in the NTP
terminology). It should be used only with trusted reliable servers,
ideally in local network.

Using timesyncd with pool.ntp.org is problematic. The pool is very
robust as a whole, but the individual servers run by volunteers cannot
be relied on. Occasionally, servers drift away or make a step to distant
past or future due to misconfiguration, problematic implementation, and
other bugs (e.g. in firmware of a GPS receiver). The pool monitoring
system detects such servers and quickly removes them from the pool DNS,
but clients like timesyncd cannot recover from that. They follow the
server as long as it claims to be synchronised. They need to be
restarted in order to get a new address from the pool DNS.

Note that the complexity of NTP and clock synchronisation is on the
client side. The amount of code in chrony specific to NTP server is very
small and it is disabled by default. If it was removed, it would not
significantly reduce the amount of memory or storage needed."

This commit fixes the issue by switching to a proper NTP client
(chrony), trustworthy time sources from Netnod and
Physikalisch-Technische Bundesanstalt which distributes the official
time for Sweden[2] and Germany[3] respectively, and finally NTS is used
to protect against MITM attacks.

Since most of our servers are in Germany or Finland (close to Sweden),
it makes sense to use these time sources as a low round-trip delay[4] is
preferred for NTP. For the few servers[5] we have outside Europe, the
root delay[4] will be higher than desired, but with the current use-case
for these servers, it should not be a problem.

[1] https://chrony-project.org/faq.html#_should_i_prefer_chrony_over_timesyncd_if_i_do_not_need_to_run_a_server
[2] https://www.netnod.se/swedish-distributed-time-service
[3] https://www.ptb.de/cms/en/ptb/fachabteilungen/abt4/fb-44/ag-442/dissemination-of-legal-time.html
[4] https://blog.meinbergglobal.com/2021/02/25/the-root-of-all-timing-understanding-root-delay-and-root-dispersion-in-ntp/
[5] {america,asia,sydney}.mirror.pkgbuild.com

Robin Candau authored 3 months ago and Robin Candau committed 3 months ago

See https://github.com/ansible/ansible/pull/77644

It has three panels showing "Cache hit ratio", "HTTP version" and "TLS
version".

The metrics are generated with Loki recording rules (see the previous
three commits).

This enables us to calculate the cache hit ratio, which may help
determine whether more caching would be beneficial.

Please note that this only counts requests for which caching is enabled
(e.g. {fastcgi,proxy}_cache is configured), e.g. for static served files
cache_status will be "".

[1] http://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_cache_status

Mainly because we are curious. The data may also be used to decide if we
want to drop older versions of TLS.

The plan is to use this for creating metrics from the nginx log lines
(e.g. requests per second).

[1] https://grafana.com/docs/loki/latest/alert/#recording-rules

Fixes: bd19c007 ("Add configuration to retain prometheus data for 1 year")

With the 11.6.2 release mariadb has made snapshot isolation a default
setting which recently caused issues on the AUR and the Forums, where
the respective database engines would fail due to the table changing
since the last time they read it:

    PHP Fatal error:  Uncaught mysqli_sql_exception: Record has changed since last read in table 'fluxbb_online' in /srv/http/fluxbb/include/dblayer/mysqli_innodb.php:79
    Stack trace:
    #0 /srv/http/fluxbb/include/dblayer/mysqli_innodb.php(79): mysqli_query()
    #1 /srv/http/fluxbb/include/functions.php(485): DBLayer->query()
    #2 /srv/http/fluxbb/include/common.php(162): update_users_online()
    #3 /srv/http/fluxbb/viewtopic.php(10): require('...')
    #4 {main}

We therefore introduce a config switch to restore the old behavior and
apply the changed setting on the two services.

Link: https://mariadb.com/kb/en/mariadb-11-6-2-release-notes/#innodb
Link: https://bbs.archlinux.org/viewtopic.php?id=301802
Link: aurweb#525


Signed-off-by: Christian Heusel <christian@heusel.eu>

The project membership must also be extended, if not the user is simply
deleted when the membership expires (defeating the purpose of extending
the access tokens).

Fixes: 639101e6 ("gitlab: Add ruby script for continuous extending of bot tokens")

If the cost exceeds $0, it indicates that we have run out of credit
and/or are doing something wrong, in either case we want to be alerted.

With the support for network.wireguard.* credentials[1] in systemd
v256[2], we can now easily avoid storing the credentials centrally in
our ansible vault, which is preferable as it makes the private keys less
exposed. It may also make fine-grained access easier in the future[3] as
there is no longer a vault file for each server.

All the keys have been rotated and the new private keys are only stored
on the servers.

[1] https://github.com/systemd/systemd/pull/30826
[2] https://github.com/systemd/systemd/releases/tag/v256
[3] #64

There is no technical reason for this at the moment, but UEFI is the de
facto firmware for x86-64, so let's be modern.