Jelle van der Waa authored 1 year ago and Leonidas Spyropoulos committed 1 year ago

Apply the same rate limitting and fail2ban rules for aur.archlinux.org

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Adjust bash wrapper scripts to activate our python virtualenv:
Script execution is triggered from ssh / git.
These wrapper scripts then call python scripts created by poetry.

We should make sure that this happens within our venv.

Previously this was done by using "poetry run...".
However, using poetry is costly and adds some delay.
Instead of using poetry, we can just activate our venv
and then execute our scripts.

Fixes: b15ac838 ("aurweb: Make SSH faster by avoiding slow Poetry (~2,5 sec faster)")

Signed-off-by: moson-mo <mo-son@mailbox.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

This release removes the php code and adjusts the location of .gz
artifacts.

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Liberally add "noqa no-changed-when" tags to the problematic tasks,
except for two "systemd-tmpfiles --create" calls. For these we can
simply include the creates= parameter in the command module's call.

Leonidas Spyropoulos authored 2 years ago and Evangelos Foutras committed 2 years ago

Convert the permissions to strings to avoid octal interpretation.

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

The daemon_reload is specific to systemd and produce a warning in
ansible-lint

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

https://git-scm.com/docs/git-config#Documentation/git-config.txt-receivefsckObjects
Related: aurweb#426



Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
Co-authored-by: Kristian Klausen <kristian@klausen.dk>
Co-authored-by: moson-mo <mo-son@mailbox.org>

Expose aurweb RPC using goaurrpc to reduce the load on the server.
Additionally we can now geo-serve this ro reduce load and bandwidth.

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Kevin Morris authored 2 years ago and Leonidas Spyropoulos committed 2 years ago

This commit brings in four new routes to nginx:
- /archives/metadata.git
- /archives/users.git
- /archives/pkgbases.git
- /archives/pkgnames.git

See https://gitlab.archlinux.org/archlinux/aurweb/-/blob/master/doc/git-archive.md



For now, we will be updating the repositories once every 10 minutes.

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
Co-signed by:  Kevin Morris <kevr@0cost.org>

From time to time aurweb is failing with "Too many open files"
errors[1], this could indicate a bug in aurweb or perhaps the limit is
just too low. Let's try doubling the limit and see if it helps.

[1] https://gitlab.archlinux.org/archlinux/aurweb-errors/-/issues/275

This avoids triggering a GitLab push rule which rejects files that look
like secrets.

Fixes: 92586d5b ("change(aurweb): rework ansible config for 6.0.0")

Required for poetry 1.2 until #1917 is fixed

https://github.com/python-poetry/poetry/issues/1917



Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>

Fixes: 26f289b7 ("Capitalize the first letter of all task names")

This might be a bug in ansible-lint 6.5.0, but it appears to ignore all
our 'skip_ansible_lint' tags. Fix this by replacing them with noqa tags.

ansible-lint 6.5.0 complains about:

  name: All names should start with an
        uppercase letter. (name[casing])

"poetry run" is very slow[1] and adds +1 second to the startup time.
This is made even worse by the fact that aurweb-git-serve is called
twice by sshd[2].

[1] https://github.com/python-poetry/poetry/issues/3502
[2] https://security.stackexchange.com/questions/123795/authorizedkeyscommand-of-sshd-config-getting-called-twice/123801#123801

Fixes: 68ec7871 ("aurweb: Mirror aur.git to GitHub[1]")

The burst size of 300 reportedly allows ~150 git operations. This might
not always be sufficient when installing a lot of packages from the AUR.

Specify a higher burst size to cover most legit use cases, even if this
makes us more susceptible to abuse.

This is meant as a extra "backup" and as another way for our users to
fetch PKGBUILDs from the AUR. It also allows the community to create
their own (perhaps better) "AUR" API/database as all essential data is
now available (this + [2]).

At the monent this is experimental and we aren't committing to keeping
it around.

[1] https://github.com/archlinux/aur
[2] http://aur.archlinux.org/packages-meta-ext-v1.json.gz

These are used to signal the start of the document in a stream of many
documents. As Ansible only supports one YAML document per file this is
unnecessary. About a third of our YAML documents already lacked these.

As reported by ansible-lint 6.2.1:

schema: [{'PYTHONPATH': '.'}] is not of type 'object' (schema[tasks])
roles/aurweb/tasks/main.yml:1

schema: [{'SHELL': '/bin/bash'}] is not of type 'object' (schema[tasks])
roles/dbscripts/tasks/main.yml:1

Kevin is MIA, so add my key, so we can do releases.