Verified Commit a467b184 authored by Kevin Morris's avatar Kevin Morris
Browse files

Merge branch 'pu': pre-v6.0.0

Release v6.0.0 - Python

This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.

Preface
-------

This v6.0.0 release makes the long-awaited Python port official.

Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.

Legend
------

There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:

- _self_
    - Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
    - Refers to a POST action which can be triggered via the relevent package
      page at `/{pkgbase,packages}/{name}`.

Grouped changes explained in multiple items will always be prefixed with
the same letter surround...
parents 6bb002e7 d3d4424b
[run]
disable_warnings = already-imported
[report]
include = aurweb/*
fail_under = 85
exclude_lines =
if __name__ == .__main__.:
pragma: no cover
*/*.mo
conf/config
conf/config.sqlite
conf/config.sqlite.defaults
conf/docker
conf/docker.defaults
FASTAPI_BACKEND="uvicorn"
FASTAPI_WORKERS=2
MARIADB_SOCKET_DIR="/var/run/mysqld/"
AURWEB_PHP_PREFIX=https://localhost:8443
AURWEB_FASTAPI_PREFIX=https://localhost:8444
AURWEB_SSHD_PREFIX=ssh://aur@localhost:2222
GIT_DATA_DIR="./aur.git/"
TEST_RECURSION_LIMIT=10000
COMMIT_HASH=
__pycache__/
*.py[cod]
.vim/
.pylintrc
.coverage
.idea
/cache/*
/logs/*
/build/
/dist/
/aurweb.egg-info/
/personal/
/notes/
/vendor/
/pyrightconfig.json
/taskell.md
aur.git/
aurweb.sqlite3
conf/config
conf/config.sqlite
conf/config.sqlite.defaults
conf/docker
conf/docker.defaults
data.sql
dummy-data.sql*
env/
fastapi_aw/
htmlcov/
po/*.mo
po/*.po~
po/POTFILES
web/locale/*/
aur.git/
__pycache__/
*.py[cod]
schema/aur-schema-sqlite.sql
test/test-results/
test/trash directory*
schema/aur-schema-sqlite.sql
web/locale/*/
web/html/*.gz
# Do not stage compiled asciidoc: make -C doc
doc/rpc.html
# Ignore any user-configured .envrc files at the root.
/.envrc
# Ignore .python-version file from Pyenv
.python-version
image: archlinux
image: archlinux:base-devel
cache:
key: system-v1
paths:
# For some reason Gitlab CI only supports storing cache/artifacts in a path relative to the build directory
- .pkg-cache
before_script:
- pacman -Syu --noconfirm --noprogressbar --needed --cachedir .pkg-cache
base-devel git gpgme protobuf pyalpm python-mysql-connector
python-pygit2 python-srcinfo python-bleach python-markdown
python-sqlalchemy python-alembic python-pytest python-werkzeug
python-pytest-tap python-fastapi hypercorn nginx python-authlib
python-itsdangerous python-httpx python-orjson
variables:
AUR_CONFIG: conf/config # Default MySQL config setup in before_script.
DB_HOST: localhost
TEST_RECURSION_LIMIT: 10000
CURRENT_DIR: "$(pwd)"
LOG_CONFIG: logging.test.conf
test:
stage: test
tags:
- fast-single-thread
before_script:
- export PATH="$HOME/.poetry/bin:${PATH}"
- ./docker/scripts/install-deps.sh
- ./docker/scripts/install-python-deps.sh
- useradd -U -d /aurweb -c 'AUR User' aur
- ./docker/mariadb-entrypoint.sh
- (cd '/usr' && /usr/bin/mysqld_safe --datadir='/var/lib/mysql') &
- 'until : > /dev/tcp/127.0.0.1/3306; do sleep 1s; done'
- cp -v conf/config.dev conf/config
- sed -i "s;YOUR_AUR_ROOT;$(pwd);g" conf/config
- ./docker/test-mysql-entrypoint.sh # Create mysql AUR_CONFIG.
- make -C po all install # Compile translations.
- make -C doc # Compile asciidoc.
- make -C test clean # Cleanup coverage.
script:
# Run sharness.
- make -C test sh
# Run pytest.
- pytest
- make -C test coverage # Produce coverage reports.
- flake8 --count aurweb # Assert no flake8 violations in aurweb.
- flake8 --count test # Assert no flake8 violations in test.
- flake8 --count migrations # Assert no flake8 violations in migrations.
- isort --check-only aurweb # Assert no isort violations in aurweb.
- isort --check-only test # Assert no flake8 violations in test.
- isort --check-only migrations # Assert no flake8 violations in migrations.
coverage: '/TOTAL.*\s+(\d+\%)/'
artifacts:
reports:
cobertura: coverage.xml
deploy:
stage: deploy
tags:
- secure
rules:
- if: $CI_COMMIT_BRANCH == "pu"
when: manual
variables:
FASTAPI_BACKEND: gunicorn
FASTAPI_WORKERS: 5
AURWEB_PHP_PREFIX: https://aur-dev.archlinux.org
AURWEB_FASTAPI_PREFIX: https://aur-dev.archlinux.org
AURWEB_SSHD_PREFIX: ssh://aur@aur-dev.archlinux.org:2222
COMMIT_HASH: $CI_COMMIT_SHA
GIT_DATA_DIR: git_data
script:
- make -C test
- pacman -Syu --noconfirm docker docker-compose socat openssh
- chmod 600 ${SSH_KEY}
- socat "UNIX-LISTEN:/tmp/docker.sock,reuseaddr,fork" EXEC:"ssh -o UserKnownHostsFile=${SSH_KNOWN_HOSTS} -Ti ${SSH_KEY} ${SSH_USER}@${SSH_HOST}" &
- export DOCKER_HOST="unix:///tmp/docker.sock"
# Set secure login config for aurweb.
- sed -ri "s/^(disable_http_login).*$/\1 = 1/" conf/config.dev
- docker-compose build
- docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml down --remove-orphans
- docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml up -d
- docker image prune -f
- docker container prune -f
- docker volume prune -f
environment:
name: development
url: https://aur-dev.archlinux.org
......@@ -8,3 +8,12 @@ You can add a git hook to do this by installing `python-pre-commit` and running
`pre-commit install`.
[1] https://lists.archlinux.org/listinfo/aur-dev
### Coding Guidelines
1. All source modified or added within a patchset **must** maintain equivalent
or increased coverage by providing tests that use the functionality.
2. Please keep your source within an 80 column width.
Test patches that increase coverage in the codebase are always welcome.
FROM archlinux:base-devel
VOLUME /root/.cache/pypoetry/cache
VOLUME /root/.cache/pypoetry/artifacts
ENV PATH="/root/.poetry/bin:${PATH}"
ENV PYTHONPATH=/aurweb
ENV AUR_CONFIG=conf/config
# Install system-wide dependencies.
COPY ./docker/scripts/install-deps.sh /install-deps.sh
RUN /install-deps.sh
# Copy Docker scripts
COPY ./docker /docker
COPY ./docker/scripts/* /usr/local/bin/
# Copy over all aurweb files.
COPY . /aurweb
# Working directory is aurweb root @ /aurweb.
WORKDIR /aurweb
# Copy initial config to conf/config.
RUN cp -vf conf/config.dev conf/config
RUN sed -i "s;YOUR_AUR_ROOT;/aurweb;g" conf/config
# Install Python dependencies.
RUN /docker/scripts/install-python-deps.sh
# Compile asciidocs.
RUN make -C doc
# Add our aur user.
RUN useradd -U -d /aurweb -c 'AUR User' aur
# Setup some default system stuff.
RUN ln -sf /usr/share/zoneinfo/UTC /etc/localtime
# Install translations.
RUN make -C po all install
......@@ -4,64 +4,135 @@ Setup on Arch Linux
For testing aurweb patches before submission, you can use the instructions in
TESTING for testing the web interface only.
Note that you can only do limited testing using the PHP built-in web server.
In particular, the cgit interface will be unusable as well as the ssh+git
interface. For a detailed description on how to setup a full aurweb server,
For a detailed description on how to setup a full aurweb server,
read the instructions below.
1) Clone the aurweb project:
1) Clone the aurweb project and install it (via `python-poetry`):
$ cd /srv/http/
$ git clone https://gitlab.archlinux.org/archlinux/aurweb.git
$ cd /srv/http/
$ git clone git://git.archlinux.org/aurweb.git
$ cd aurweb
$ poetry install
2) Setup a web server with PHP and MySQL. Configure the web server to redirect
all URLs to /index.php/foo/bar/. The following block can be used with nginx:
server {
listen 80;
# https is preferred and can be done easily with LetsEncrypt
# or self-CA signing. Users can still listen over 80 for plain
# http, for which the [options] disable_http_login used to toggle
# the authentication feature.
listen 443 ssl http2;
server_name aur.local aur;
# To enable SSL proxy properly, make sure gunicorn and friends
# are supporting forwarded headers over 127.0.0.1 or any if
# the asgi server is contacted by non-localhost hosts.
ssl_certificate /etc/ssl/certs/aur.cert.pem;
ssl_certificate_key /etc/ssl/private/aur.key.pem;
# Asset root. This is used to match against gzip archives.
root /srv/http/aurweb/web/html;
index index.php;
location ~ ^/[^/]+\.php($|/) {
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index index.php;
fastcgi_split_path_info ^(/[^/]+\.php)(/.*)$;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
include fastcgi_params;
# TU Bylaws redirect.
location = /trusted-user/TUbylaws.html {
return 301 https://tu-bylaws.aur.archlinux.org;
}
location ~ .* {
rewrite ^/(.*)$ /index.php/$1 last;
# smartgit location.
location ~ "^/([a-z0-9][a-z0-9.+_-]*?)(\.git)?/(git-(receive|upload)-pack|HEAD|info/refs|objects/(info/(http-)?alternates|packs)|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))$" {
include uwsgi_params;
uwsgi_pass smartgit;
uwsgi_modifier1 9;
uwsgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
uwsgi_param PATH_INFO /aur.git/$3;
uwsgi_param GIT_HTTP_EXPORT_ALL "";
uwsgi_param GIT_NAMESPACE $1;
uwsgi_param GIT_PROJECT_ROOT /srv/http/aurweb;
}
}
Ensure to enable the pdo_mysql extension in php.ini.
# cgitrc.proto should be configured and located somewhere
# of your choosing.
location ~ ^/cgit {
include uwsgi_params;
rewrite ^/cgit/([^?/]+/[^?]*)?(?:\?(.*))?$ /cgit.cgi?url=$1&$2 last;
uwsgi_modifier1 9;
uwsgi_param CGIT_CONFIG /srv/http/aurweb/conf/cgitrc.proto;
uwsgi_pass cgit;
}
# Static archive assets.
location ~ \.gz$ {
types { application/gzip text/plain }
default_type text/plain;
add_header Content-Encoding gzip;
expires 5m;
}
# For everything else, proxy the http request to (guni|uvi|hyper)corn.
# The ASGI server application should allow this request's IP to be
# forwarded via the headers used below.
# https://docs.gunicorn.org/en/stable/settings.html#forwarded-allow-ips
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol ssl;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Ssl on;
}
}
3) Optionally copy conf/config.defaults to /etc/aurweb/. Create or copy
/etc/aurweb/config (this is expected to contain all configuration settings
if the defaults file does not exist) and adjust the configuration (pay
attention to disable_http_login, enable_maintenance and aur_location).
4) Install Python modules and dependencies:
4) Install system-wide dependencies:
# pacman -S git gpgme cgit curl openssh uwsgi uwsgi-plugin-cgi \
python-poetry
# pacman -S python-mysql-connector python-pygit2 python-srcinfo python-sqlalchemy \
python-bleach python-markdown python-alembic python-jinja \
python-itsdangerous python-authlib python-httpx hypercorn \
python-orjson
# python3 setup.py install
5) Create a new user:
5) Create a new MySQL database and a user and import the aurweb SQL schema:
# useradd -U -d /srv/http/aurweb -c 'AUR user' aur
# su - aur
$ python -m aurweb.initdb
6a) Install Python dependencies via poetry:
6) Create a new user:
# Install the package and scripts as the aur user.
$ poetry install
# useradd -U -d /srv/http/aurweb -c 'AUR user' aur
6b) Setup Services
aurweb utilizes the following systemd services:
- mariadb
- redis (optional, requires [options] cache 'redis')
- `examples/aurweb.service`
7) Initialize the Git repository:
6c) Setup Cron
Using [cronie](https://archlinux.org/packages/core/x86_64/cronie/):
# su - aur
$ crontab -e
The following crontab file uses every script meant to be run on an
interval:
AUR_CONFIG='/etc/aurweb/config'
*/5 * * * * bash -c 'poetry run aurweb-mkpkglists --extended'
*/2 * * * * bash -c 'poetry run aurweb-aurblup'
*/2 * * * * bash -c 'poetry run aurweb-pkgmaint'
*/2 * * * * bash -c 'poetry run aurweb-usermaint'
*/2 * * * * bash -c 'poetry run aurweb-popupdate'
*/12 * * * * bash -c 'poetry run aurweb-tuvotereminder'
7) Create a new database and a user and import the aurweb SQL schema:
$ poetry run python -m aurweb.initdb
8) Initialize the Git repository:
# mkdir /srv/http/aurweb/aur.git/
# cd /srv/http/aurweb/aur.git/
......@@ -69,19 +140,26 @@ read the instructions below.
# git config --local transfer.hideRefs '^refs/'
# git config --local --add transfer.hideRefs '!refs/'
# git config --local --add transfer.hideRefs '!HEAD'
# ln -s /usr/local/bin/aurweb-git-update hooks/update
# chown -R aur .
Link to `aurweb-git-update` poetry wrapper provided at
`examples/aurweb-git-update.sh` which should be installed
somewhere as executable.
# ln -s /path/to/aurweb-git-update.sh hooks/update
It is recommended to read doc/git-interface.txt for more information on the
administration of the package Git repository.
8) Configure sshd(8) for the AUR. Add the following lines at the end of your
sshd_config(5) and restart the sshd. Note that OpenSSH 6.9 or newer is
needed!
9) Configure sshd(8) for the AUR. Add the following lines at the end of your
sshd_config(5) and restart the sshd.
If using a virtualenv, copy `examples/aurweb-git-auth.sh` to a location
and call it below:
Match User aur
PasswordAuthentication no
AuthorizedKeysCommand /usr/local/bin/aurweb-git-auth "%t" "%k"
AuthorizedKeysCommand /path/to/aurweb-git-auth.sh "%t" "%k"
AuthorizedKeysCommandUser aur
AcceptEnv AUR_OVERWRITE
......@@ -100,8 +178,17 @@ read the instructions below.
Sample systemd unit files for fcgiwrap can be found under conf/.
10) If you want memcache to cache MySQL data.
10) If you want Redis to cache data.
# pacman -S redis
# systemctl enable --now redis
And edit the configuration file to enabled redis caching
(`[options] cache = redis`).
# pacman -S php-memcached
11) Start `aurweb.service`.
And edit the configuration file to enabled memcache caching.
An example systemd unit has been included at `examples/aurweb.service`.
This unit can be used to manage the aurweb asgi backend. By default,
it is configured to use `poetry` as the `aur` user; this should be
configured as needed.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.