Potentially useful to quickly know which mirror we're talking to.

We had a GeoIP mirror in the past based on nginx and its GeoIP module,
but it didn't perform very well, due to the high latency (asking a
central server for the package and then redirected to the closest
mirror).

One of the reasons for offering this service, is so we can relieve
mirror.pkgbuild.com which is burning a ton of traffic (50TB/month),
likely due to it being the default mirror in our Docker image. Another
reason is so we can offer a link to our arch-boxes images in libosinfo
(used by gnome-boxes, virt-install and virt-manager), with good enough
performance for most users.

This time we take a different approach and use a DNS based solution,
which means the latency penalty is only paid once (the first DNS
request). The downside is that the mirrors must have a valid certificate
for the same domain name, which makes using third-party mirrors a
challenge. So for now, we are just using the sponsored mirorrs
controlled by the DevOps team.

Fix #101

The sponsored mirrors have a ton of storage, but mirror.pkgbuild.com
doesn't, so debug packages aren't synced to it.

[1] {america,asia,europe}.mirror.pkgbuild.com

Using a temporary directory outside of /srv/ftp was meant to protect
against incomplete files from being synced by downstream mirrors. It
does not achieve this to much effect though; each file gets uploaded
to the temporary directory but then immediately moved under a .~tmp~
directory at its target location (.~tmp~ because of --delay-updates,
otherwise the file would be renamed to its final path).

The `--delay-updates` option by itself sufficiently protects against
temp files being transferred to downstream mirrors; when used by the
receiver, it automatically adds an exclude rule for ~.tmp~, behaving
exactly like we want it to. As such, the `--temp-dir` option doesn't
provide any further benefit and can be removed.

The workaround can be skipped/removed when using rsync newer than 3.2.3.

- Replace --delete-after with more efficient --delete-delay.
- Move "-p" together with the other short options.
- Remove reference to empty ${VERBOSE} variable.

This reverts commit 75f9ca3c.

This should be fixed in rsync versions newer than 3.2.3. In Arch the fix
has been shipped in the rsync 3.2.3-4 package, which our own mirrors now
have been updated to.

[1] https://github.com/WayneD/rsync/issues/192

Leonidas Spyropoulos authored 3 years ago and Jelle van der Waa committed 3 years ago

Closes #286

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

A extra access_log entry was added with the following commands:
$ cd roles
$ grep -lr access_log | xargs -P 1 -n 1 sed -i '/access_log/ s/\(.*\)\( \)\(\(reduced\|main\);$\)/\1 \3\n\1.json json_\3/'

As reported in FS#69614 the lock should be before the delete for when
another sync job is running rsync can't move the files as they are
removed.

yaml: truthy value should be one of [false, true] (truthy)
yaml: wrong indentation: expected 4 but found 2 (indentation)
yaml: too few spaces before comment (comments)
yaml: missing starting space in comment (comments)
yaml: too many blank lines (1 > 0) (empty-lines)
yaml: too many spaces after colon (colons)
yaml: comment not indented like content (comments-indentation)
yaml: no new line character at the end of file (new-line-at-end-of-file)
load-failure: Failed to load or parse file
parser-error: couldn't resolve module/action 'hosts'. This often indicates a misspelling, missing collection, or incorrect module path.

It should make it easier to change how the certificates is issued.
Ex: If we want to switch to ECDSA certificates in the future or replace
certbot with something else.

We noticed that sometimes rsync becomes confused and leaves .~tmp~ dirs around which inhibits syncing from that point forward.

Frederik Schwan authored 4 years ago and Sven-Hendrik Haase committed 4 years ago

also use systemd instead of service module

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Allow the synchronisation of the links database used by sogrep.

Where http does not redirect to https. These are package mirrors and
the web key directory.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

The script is currently hosted on my private server, but still linked in
the wiki[1]. This way, we can update the wiki to link to this copy of
the script as an official source.

[1] https://wiki.archlinux.org/index.php/DeveloperWiki:NewMirrors



Signed-off-by: Florian Pritz <bluewind@xinu.at>

Upstream being me:
https://git.server-speed.net/users/flo/bin/tree/syncrepo.sh



Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

The discovery script now uses a regex and no longer cares where exactly
accounting is enabled. Follow systemd upstream by enabling it by
default.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

For proxy/fastcgi/uwsgi blocks, logging is still set to the old format,
but for everything else (= static data) a reduced format is used that
excludes items that no longer make sense (request_time, remote_user) and
those that are personal information all the time (remote_addr, http_x_forwarded_for).

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Disable firewall because python2 module is not avaible and the tasks
fail which makes the playbooks fail and leads to handlers not being run.

https://github.com/ansible/ansible/issues/24855



Signed-off-by: Florian Pritz <bluewind@xinu.at>