The intention of "unique" in "groups['buildservers'] | sort | unique"
was to account for combining multiple groups and passing them to the
sort and unique filters. However, with only one group it looks silly.

There is a need for build servers to never build against outdated repo
databases, even with syncrepo providing a local mirror that is updated
every minute. To that effect, we adjust mirrorlist on build servers so
the first mirror is the tier0 mirror provided by gemini.

Keep the syncrepo role on build servers in order to have a local cache
of packages and avoid concurrent build jobs downloading the same files
causing them to be corrupted.

Finally, configure gemini to use its own repos (like other mirrors do).

The sponsored mirrors have a ton of storage, but mirror.pkgbuild.com
doesn't, so debug packages aren't synced to it.

[1] {america,asia,europe}.mirror.pkgbuild.com

Morten Linderud authored 4 years ago and Kristian Klausen committed 3 years ago

Signed-off-by: Morten Linderud <morten@linderud.pw>

Signed-off-by: Morten Linderud <morten@linderud.pw>

Fix #177

The upstream branch is set by the earlier "git pull --set-upstream".

No functional change; the "restrict" key option is a shorthand for:

- no-agent-forwarding
- no-port-forwarding
- no-X11-forwarding
- no-pty
- no-user-rc

It was added in OpenSSH 7.2 (2016-02-29) as a convenient way to specify
an authorized key should have "all current and future key restrictions"
applied to it.

The default login shell for the svntogit user (/sbin/nologin) breaks
the Match Exec directives in /srv/svntogit/.ssh/config and prohibits
Git from using the correct SSH key.

While we're at it, add --set-upstream to the git pull command so the
task is more likely to accomplish its intended purpose.

- Set "proxy_cache_lock on" to avoid sending more than one auth request
  to archweb when a cache entry doesn't exist for the credentials.
- Set "proxy_cache_use_stale updating" to prevent nginx sending multiple
  auth requests to archweb while the cache is being refreshed. [1]

[1] http://mailman.nginx.org/pipermail/nginx/2016-January/049734.html

Use $uri instead of $request_uri in proxy_cache_key to avoid caching
based on the original URI which is different for each pacman request.

The cache keys are now in the following format:

  httpsarchlinux.org/devel/mirrorauth/Basic <credentials>

This implements authentication to our repos.archlinux.org tier 0 mirror
via archweb.

Kristian Klausen authored 3 years ago and Jelle van der Waa committed 3 years ago

A extra access_log entry was added with the following commands:
$ cd roles
$ grep -lr access_log | xargs -P 1 -n 1 sed -i '/access_log/ s/\(.*\)\( \)\(\(reduced\|main\);$\)/\1 \3\n\1.json json_\3/'

yaml: truthy value should be one of [false, true] (truthy)
yaml: wrong indentation: expected 4 but found 2 (indentation)
yaml: too few spaces before comment (comments)
yaml: missing starting space in comment (comments)
yaml: too many blank lines (1 > 0) (empty-lines)
yaml: too many spaces after colon (colons)
yaml: comment not indented like content (comments-indentation)
yaml: no new line character at the end of file (new-line-at-end-of-file)
load-failure: Failed to load or parse file
parser-error: couldn't resolve module/action 'hosts'. This often indicates a misspelling, missing collection, or incorrect module path.

sourceballs wasn't able to generate any new tarballs due to not being
able to sudo due to systemd hardening.

Databases used by sogrep are fetched by syncrepo from gemini, no point
in duplicating this work; consider this to be part of roles/dbscripts.

It should make it easier to change how the certificates is issued.
Ex: If we want to switch to ECDSA certificates in the future or replace
certbot with something else.

These ips are no longer controlled by Arch Linux and ftp-archlinux is
not synced by anything from us.

As we have no archive_mirrors at this time.

Remove unused devlist-mailer and apply some hardening.

The emails where not delivered to anyway nor do they continue useful
information. Log output is now viewable in journalctl and some hardening
is applied.

Archivetools syncs from kitchensink_tier1 for some reason, to create the
repository and stuff.