Skip to content
Snippets Groups Projects
Verified Commit a467b184 authored by Kevin Morris's avatar Kevin Morris
Browse files

Merge branch 'pu': pre-v6.0.0

Release v6.0.0 - Python

This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.

Preface
-------

This v6.0.0 release makes the long-awaited Python port official.

Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.

Legend
------

There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:

- _self_
    - Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
    - Refers to a POST action which can be triggered via the relevent package
      page at `/{pkgbase,packages}/{name}`.

Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:

- [A] Some feature that does something
- [A] The same feature where another thing has changed

Infrastructure
--------------

- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
  SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
  missing both out of the box and integrally which force us to account
  for the different database types. We now only support mysql, and should
  be able to support postgresql without much effort in the future.
  Note: Users wishing to easily spin up a database quickly can use
  `docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
  at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
  call these scripts when aurweb is installed into a poetry virtualenv.

HTML
----

- Pagers have all been modified. They still serve the same purpose, but
  they have slightly different display.
- Some markup and methods around the website has been changed for
  post requests, and some forms have been completely reworked.

Package Requests
----------------

- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
    - Deletion
        - Through `/packages` bulk delete action
        - Through `/pkgbase/{name}/delete`
    - Merge
        - Through `/pkgbase/{name}/merge`
    - Orphan
        - Through `/packages` bulk disown action
        - Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
  if no pre-existing request exists. This was done to increase tracking of
  package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
  requests pertaining to the action performed. This comes with the downside
  of multiple notifications sent out about a closure if more than one
  request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
  with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere

Package Bulk Actions: /packages
-------------------------------

- The `Merge into` field has been removed. Merges now require being
  performed via the `/pkgbase/{name}/merge` action.

Package View
------------

- Some cached metadata is no longer cached (pkginfo). Previously,
  this was defaulted to a one day cache for some package information.
  If we need to bring this back, we can.

TU Proposals
------------

- A valid username is now required for any addition or removal of a TU.

RPC
---

- `type=get-comment-form` has been removed and is now located at
  `/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
  should not heavily affect users, as a 3,000 record query is returned
  in roughly 0.20ms from a local standpoint. We will be working on this
  in aim to push it over PHP.

Archives
--------

- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
    - Enable this by passing `--extended` to `aurweb-mkpkglists`.

Performance Changes
-------------------

As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:

- `/` loads much quicker as it is now persistently cached forcibly
  for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
  various pieces of package info for `cache_pkginfo_ttl`, which is
  defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
  We now query the database for requests related to the action based on
  the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
    - We are not satisfied with this. We'll be working on pushing this
      over the edge along with the rest of the DB-intensive routes.
      However, the speed degredation is quite negligible for users'
      experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
      on my local 4-core 8-thread system.

Upcoming Work
-------------

This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.

- Update request and tu vote pagers
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
parents 6bb002e7 d3d4424b
No related branches found
No related tags found
No related merge requests found
[run]
disable_warnings = already-imported
[report]
include = aurweb/*
fail_under = 85
exclude_lines =
if __name__ == .__main__.:
pragma: no cover
*/*.mo
conf/config
conf/config.sqlite
conf/config.sqlite.defaults
conf/docker
conf/docker.defaults
.env 0 → 100644
FASTAPI_BACKEND="uvicorn"
FASTAPI_WORKERS=2
MARIADB_SOCKET_DIR="/var/run/mysqld/"
AURWEB_PHP_PREFIX=https://localhost:8443
AURWEB_FASTAPI_PREFIX=https://localhost:8444
AURWEB_SSHD_PREFIX=ssh://aur@localhost:2222
GIT_DATA_DIR="./aur.git/"
TEST_RECURSION_LIMIT=10000
COMMIT_HASH=
__pycache__/
*.py[cod]
.vim/
.pylintrc
.coverage
.idea
/cache/*
/logs/*
/build/
/dist/
/aurweb.egg-info/
/personal/
/notes/
/vendor/
/pyrightconfig.json
/taskell.md
aur.git/
aurweb.sqlite3
conf/config
conf/config.sqlite
conf/config.sqlite.defaults
conf/docker
conf/docker.defaults
data.sql
dummy-data.sql*
env/
fastapi_aw/
htmlcov/
po/*.mo
po/*.po~
po/POTFILES
web/locale/*/
aur.git/
__pycache__/
*.py[cod]
schema/aur-schema-sqlite.sql
test/test-results/
test/trash directory*
schema/aur-schema-sqlite.sql
web/locale/*/
web/html/*.gz
# Do not stage compiled asciidoc: make -C doc
doc/rpc.html
# Ignore any user-configured .envrc files at the root.
/.envrc
# Ignore .python-version file from Pyenv
.python-version
image: archlinux
image: archlinux:base-devel
cache:
key: system-v1
paths:
# For some reason Gitlab CI only supports storing cache/artifacts in a path relative to the build directory
- .pkg-cache
before_script:
- pacman -Syu --noconfirm --noprogressbar --needed --cachedir .pkg-cache
base-devel git gpgme protobuf pyalpm python-mysql-connector
python-pygit2 python-srcinfo python-bleach python-markdown
python-sqlalchemy python-alembic python-pytest python-werkzeug
python-pytest-tap python-fastapi hypercorn nginx python-authlib
python-itsdangerous python-httpx python-orjson
variables:
AUR_CONFIG: conf/config # Default MySQL config setup in before_script.
DB_HOST: localhost
TEST_RECURSION_LIMIT: 10000
CURRENT_DIR: "$(pwd)"
LOG_CONFIG: logging.test.conf
test:
stage: test
tags:
- fast-single-thread
before_script:
- export PATH="$HOME/.poetry/bin:${PATH}"
- ./docker/scripts/install-deps.sh
- ./docker/scripts/install-python-deps.sh
- useradd -U -d /aurweb -c 'AUR User' aur
- ./docker/mariadb-entrypoint.sh
- (cd '/usr' && /usr/bin/mysqld_safe --datadir='/var/lib/mysql') &
- 'until : > /dev/tcp/127.0.0.1/3306; do sleep 1s; done'
- cp -v conf/config.dev conf/config
- sed -i "s;YOUR_AUR_ROOT;$(pwd);g" conf/config
- ./docker/test-mysql-entrypoint.sh # Create mysql AUR_CONFIG.
- make -C po all install # Compile translations.
- make -C doc # Compile asciidoc.
- make -C test clean # Cleanup coverage.
script:
# Run sharness.
- make -C test sh
# Run pytest.
- pytest
- make -C test coverage # Produce coverage reports.
- flake8 --count aurweb # Assert no flake8 violations in aurweb.
- flake8 --count test # Assert no flake8 violations in test.
- flake8 --count migrations # Assert no flake8 violations in migrations.
- isort --check-only aurweb # Assert no isort violations in aurweb.
- isort --check-only test # Assert no flake8 violations in test.
- isort --check-only migrations # Assert no flake8 violations in migrations.
coverage: '/TOTAL.*\s+(\d+\%)/'
artifacts:
reports:
cobertura: coverage.xml
deploy:
stage: deploy
tags:
- secure
rules:
- if: $CI_COMMIT_BRANCH == "pu"
when: manual
variables:
FASTAPI_BACKEND: gunicorn
FASTAPI_WORKERS: 5
AURWEB_PHP_PREFIX: https://aur-dev.archlinux.org
AURWEB_FASTAPI_PREFIX: https://aur-dev.archlinux.org
AURWEB_SSHD_PREFIX: ssh://aur@aur-dev.archlinux.org:2222
COMMIT_HASH: $CI_COMMIT_SHA
GIT_DATA_DIR: git_data
script:
- make -C test
- pacman -Syu --noconfirm docker docker-compose socat openssh
- chmod 600 ${SSH_KEY}
- socat "UNIX-LISTEN:/tmp/docker.sock,reuseaddr,fork" EXEC:"ssh -o UserKnownHostsFile=${SSH_KNOWN_HOSTS} -Ti ${SSH_KEY} ${SSH_USER}@${SSH_HOST}" &
- export DOCKER_HOST="unix:///tmp/docker.sock"
# Set secure login config for aurweb.
- sed -ri "s/^(disable_http_login).*$/\1 = 1/" conf/config.dev
- docker-compose build
- docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml down --remove-orphans
- docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml up -d
- docker image prune -f
- docker container prune -f
- docker volume prune -f
environment:
name: development
url: https://aur-dev.archlinux.org
......@@ -8,3 +8,12 @@ You can add a git hook to do this by installing `python-pre-commit` and running
`pre-commit install`.
[1] https://lists.archlinux.org/listinfo/aur-dev
### Coding Guidelines
1. All source modified or added within a patchset **must** maintain equivalent
or increased coverage by providing tests that use the functionality.
2. Please keep your source within an 80 column width.
Test patches that increase coverage in the codebase are always welcome.
FROM archlinux:base-devel
VOLUME /root/.cache/pypoetry/cache
VOLUME /root/.cache/pypoetry/artifacts
ENV PATH="/root/.poetry/bin:${PATH}"
ENV PYTHONPATH=/aurweb
ENV AUR_CONFIG=conf/config
# Install system-wide dependencies.
COPY ./docker/scripts/install-deps.sh /install-deps.sh
RUN /install-deps.sh
# Copy Docker scripts
COPY ./docker /docker
COPY ./docker/scripts/* /usr/local/bin/
# Copy over all aurweb files.
COPY . /aurweb
# Working directory is aurweb root @ /aurweb.
WORKDIR /aurweb
# Copy initial config to conf/config.
RUN cp -vf conf/config.dev conf/config
RUN sed -i "s;YOUR_AUR_ROOT;/aurweb;g" conf/config
# Install Python dependencies.
RUN /docker/scripts/install-python-deps.sh
# Compile asciidocs.
RUN make -C doc
# Add our aur user.
RUN useradd -U -d /aurweb -c 'AUR User' aur
# Setup some default system stuff.
RUN ln -sf /usr/share/zoneinfo/UTC /etc/localtime
# Install translations.
RUN make -C po all install
......@@ -4,64 +4,135 @@ Setup on Arch Linux
For testing aurweb patches before submission, you can use the instructions in
TESTING for testing the web interface only.
Note that you can only do limited testing using the PHP built-in web server.
In particular, the cgit interface will be unusable as well as the ssh+git
interface. For a detailed description on how to setup a full aurweb server,
For a detailed description on how to setup a full aurweb server,
read the instructions below.
1) Clone the aurweb project:
1) Clone the aurweb project and install it (via `python-poetry`):
$ cd /srv/http/
$ git clone https://gitlab.archlinux.org/archlinux/aurweb.git
$ cd /srv/http/
$ git clone git://git.archlinux.org/aurweb.git
$ cd aurweb
$ poetry install
2) Setup a web server with PHP and MySQL. Configure the web server to redirect
all URLs to /index.php/foo/bar/. The following block can be used with nginx:
server {
listen 80;
# https is preferred and can be done easily with LetsEncrypt
# or self-CA signing. Users can still listen over 80 for plain
# http, for which the [options] disable_http_login used to toggle
# the authentication feature.
listen 443 ssl http2;
server_name aur.local aur;
# To enable SSL proxy properly, make sure gunicorn and friends
# are supporting forwarded headers over 127.0.0.1 or any if
# the asgi server is contacted by non-localhost hosts.
ssl_certificate /etc/ssl/certs/aur.cert.pem;
ssl_certificate_key /etc/ssl/private/aur.key.pem;
# Asset root. This is used to match against gzip archives.
root /srv/http/aurweb/web/html;
index index.php;
location ~ ^/[^/]+\.php($|/) {
fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
fastcgi_index index.php;
fastcgi_split_path_info ^(/[^/]+\.php)(/.*)$;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
include fastcgi_params;
# TU Bylaws redirect.
location = /trusted-user/TUbylaws.html {
return 301 https://tu-bylaws.aur.archlinux.org;
}
location ~ .* {
rewrite ^/(.*)$ /index.php/$1 last;
# smartgit location.
location ~ "^/([a-z0-9][a-z0-9.+_-]*?)(\.git)?/(git-(receive|upload)-pack|HEAD|info/refs|objects/(info/(http-)?alternates|packs)|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))$" {
include uwsgi_params;
uwsgi_pass smartgit;
uwsgi_modifier1 9;
uwsgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
uwsgi_param PATH_INFO /aur.git/$3;
uwsgi_param GIT_HTTP_EXPORT_ALL "";
uwsgi_param GIT_NAMESPACE $1;
uwsgi_param GIT_PROJECT_ROOT /srv/http/aurweb;
}
}
Ensure to enable the pdo_mysql extension in php.ini.
# cgitrc.proto should be configured and located somewhere
# of your choosing.
location ~ ^/cgit {
include uwsgi_params;
rewrite ^/cgit/([^?/]+/[^?]*)?(?:\?(.*))?$ /cgit.cgi?url=$1&$2 last;
uwsgi_modifier1 9;
uwsgi_param CGIT_CONFIG /srv/http/aurweb/conf/cgitrc.proto;
uwsgi_pass cgit;
}
# Static archive assets.
location ~ \.gz$ {
types { application/gzip text/plain }
default_type text/plain;
add_header Content-Encoding gzip;
expires 5m;
}
# For everything else, proxy the http request to (guni|uvi|hyper)corn.
# The ASGI server application should allow this request's IP to be
# forwarded via the headers used below.
# https://docs.gunicorn.org/en/stable/settings.html#forwarded-allow-ips
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol ssl;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Ssl on;
}
}
3) Optionally copy conf/config.defaults to /etc/aurweb/. Create or copy
/etc/aurweb/config (this is expected to contain all configuration settings
if the defaults file does not exist) and adjust the configuration (pay
attention to disable_http_login, enable_maintenance and aur_location).
4) Install Python modules and dependencies:
4) Install system-wide dependencies:
# pacman -S git gpgme cgit curl openssh uwsgi uwsgi-plugin-cgi \
python-poetry
# pacman -S python-mysql-connector python-pygit2 python-srcinfo python-sqlalchemy \
python-bleach python-markdown python-alembic python-jinja \
python-itsdangerous python-authlib python-httpx hypercorn \
python-orjson
# python3 setup.py install
5) Create a new user:
5) Create a new MySQL database and a user and import the aurweb SQL schema:
# useradd -U -d /srv/http/aurweb -c 'AUR user' aur
# su - aur
$ python -m aurweb.initdb
6a) Install Python dependencies via poetry:
6) Create a new user:
# Install the package and scripts as the aur user.
$ poetry install
# useradd -U -d /srv/http/aurweb -c 'AUR user' aur
6b) Setup Services
aurweb utilizes the following systemd services:
- mariadb
- redis (optional, requires [options] cache 'redis')
- `examples/aurweb.service`
7) Initialize the Git repository:
6c) Setup Cron
Using [cronie](https://archlinux.org/packages/core/x86_64/cronie/):
# su - aur
$ crontab -e
The following crontab file uses every script meant to be run on an
interval:
AUR_CONFIG='/etc/aurweb/config'
*/5 * * * * bash -c 'poetry run aurweb-mkpkglists --extended'
*/2 * * * * bash -c 'poetry run aurweb-aurblup'
*/2 * * * * bash -c 'poetry run aurweb-pkgmaint'
*/2 * * * * bash -c 'poetry run aurweb-usermaint'
*/2 * * * * bash -c 'poetry run aurweb-popupdate'
*/12 * * * * bash -c 'poetry run aurweb-tuvotereminder'
7) Create a new database and a user and import the aurweb SQL schema:
$ poetry run python -m aurweb.initdb
8) Initialize the Git repository:
# mkdir /srv/http/aurweb/aur.git/
# cd /srv/http/aurweb/aur.git/
......@@ -69,19 +140,26 @@ read the instructions below.
# git config --local transfer.hideRefs '^refs/'
# git config --local --add transfer.hideRefs '!refs/'
# git config --local --add transfer.hideRefs '!HEAD'
# ln -s /usr/local/bin/aurweb-git-update hooks/update
# chown -R aur .
Link to `aurweb-git-update` poetry wrapper provided at
`examples/aurweb-git-update.sh` which should be installed
somewhere as executable.
# ln -s /path/to/aurweb-git-update.sh hooks/update
It is recommended to read doc/git-interface.txt for more information on the
administration of the package Git repository.
8) Configure sshd(8) for the AUR. Add the following lines at the end of your
sshd_config(5) and restart the sshd. Note that OpenSSH 6.9 or newer is
needed!
9) Configure sshd(8) for the AUR. Add the following lines at the end of your
sshd_config(5) and restart the sshd.
If using a virtualenv, copy `examples/aurweb-git-auth.sh` to a location
and call it below:
Match User aur
PasswordAuthentication no
AuthorizedKeysCommand /usr/local/bin/aurweb-git-auth "%t" "%k"
AuthorizedKeysCommand /path/to/aurweb-git-auth.sh "%t" "%k"
AuthorizedKeysCommandUser aur
AcceptEnv AUR_OVERWRITE
......@@ -100,8 +178,17 @@ read the instructions below.
Sample systemd unit files for fcgiwrap can be found under conf/.
10) If you want memcache to cache MySQL data.
10) If you want Redis to cache data.
# pacman -S redis
# systemctl enable --now redis
And edit the configuration file to enabled redis caching
(`[options] cache = redis`).
# pacman -S php-memcached
11) Start `aurweb.service`.
And edit the configuration file to enabled memcache caching.
An example systemd unit has been included at `examples/aurweb.service`.
This unit can be used to manage the aurweb asgi backend. By default,
it is configured to use `poetry` as the `aur` user; this should be
configured as needed.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
......@@ -19,12 +19,14 @@ Directory Layout
* `aurweb`: aurweb Python modules, Git interface and maintenance scripts
* `conf`: configuration and configuration templates
* `static`: static resource files
* `templates`: jinja2 template collection
* `doc`: project documentation
* `po`: translation files for strings in the aurweb interface
* `schema`: schema for the SQL database
* `test`: test suite and test cases
* `upgrading`: instructions for upgrading setups from one release to another
* `web`: web interface for the AUR
* `web`: PHP-based web interface for the AUR
Links
-----
......@@ -46,3 +48,8 @@ Translations are welcome via our Transifex project at
https://www.transifex.com/lfleischer/aurweb; see `doc/i18n.txt` for details.
![Transifex](https://www.transifex.com/projects/p/aurweb/chart/image_png)
Testing
-------
See [test/README.md](test/README.md) for details on dependencies and testing.
......@@ -5,19 +5,47 @@ Note that this setup is only to test the web interface. If you need to have a
full aurweb instance with cgit, ssh interface, etc, follow the directions in
INSTALL.
docker-compose
--------------
1) Clone the aurweb project:
$ git clone https://gitlab.archlinux.org/archlinux/aurweb.git
2) Install the necessary packages:
# pacman -S --needed php php-sqlite sqlite words fortune-mod \
python python-sqlalchemy python-alembic \
python-fastapi uvicorn nginx \
python-authlib python-itsdangerous python-httpx \
words fortune-mod
# pacman -S docker-compose
2) Build the aurweb:latest image:
$ cd /path/to/aurweb/
$ docker-compose build
3) Run local Docker development instance:
$ cd /path/to/aurweb/
$ docker-compose up -d nginx
4) Browse to local aurweb development server.
Python: https://localhost:8444/
PHP: https://localhost:8443/
Bare Metal
----------
1) Clone the aurweb project:
$ git clone git://git.archlinux.org/aurweb.git
2) Install the necessary packages:
# pacman -S python-poetry
4) Install the package/dependencies via `poetry`:
Ensure to enable the pdo_sqlite extension in php.ini.
$ cd /path/to/aurweb/
$ poetry install
3) Copy conf/config.dev to conf/config and replace YOUR_AUR_ROOT by the absolute
path to the root of your aurweb clone. sed can do both tasks for you:
......@@ -27,15 +55,23 @@ INSTALL.
Note that when the upstream config.dev is updated, you should compare it to
your conf/config, or regenerate your configuration with the command above.
4) Prepare the testing database:
4) Prepare a database:
$ cd /path/to/aurweb/
$ AUR_CONFIG=conf/config python -m aurweb.initdb
$ AUR_CONFIG=conf/config poetry run python -m aurweb.initdb
$ schema/gendummydata.py data.sql
$ sqlite3 aurweb.sqlite3 < data.sql
$ poetry run schema/gendummydata.py dummy_data.sql
$ mysql -uaur -paur aurweb < dummy_data.sql
5) Run the test server:
$ AUR_CONFIG=conf/config python -m aurweb.spawn
## set AUR_CONFIG to our locally created config
$ export AUR_CONFIG=conf/config
## with aurweb.spawn
$ poetry run python -m aurweb.spawn
## with systemd service
$ sudo install -m644 examples/aurweb.service /etc/systemd/system/
$ systemctl enable --now aurweb.service
import hashlib
import http
import io
import os
import re
import sys
import traceback
import typing
from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse
from urllib.parse import quote_plus
from fastapi import FastAPI, HTTPException, Request, Response
from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles
from jinja2 import TemplateNotFound
from prometheus_client import multiprocess
from sqlalchemy import and_, or_
from starlette.exceptions import HTTPException as StarletteHTTPException
from starlette.middleware.authentication import AuthenticationMiddleware
from starlette.middleware.sessions import SessionMiddleware
import aurweb.captcha # noqa: F401
import aurweb.config
import aurweb.filters # noqa: F401
import aurweb.logging
import aurweb.pkgbase.util as pkgbaseutil
from aurweb import logging, prometheus, util
from aurweb.auth import BasicAuthBackend
from aurweb.db import get_engine, query
from aurweb.models import AcceptedTerm, Term
from aurweb.packages.util import get_pkg_or_base
from aurweb.prometheus import instrumentator
from aurweb.redis import redis_connection
from aurweb.routers import APP_ROUTES
from aurweb.scripts import notify
from aurweb.templates import make_context, render_template
from aurweb.routers import sso
logger = logging.get_logger(__name__)
# Setup the FastAPI app.
app = FastAPI()
session_secret = aurweb.config.get("fastapi", "session_secret")
if not session_secret:
raise Exception("[fastapi] session_secret must not be empty")
# Instrument routes with the prometheus-fastapi-instrumentator
# library with custom collectors and expose /metrics.
instrumentator().add(prometheus.http_api_requests_total())
instrumentator().add(prometheus.http_requests_total())
instrumentator().instrument(app)
@app.on_event("startup")
async def app_startup():
# https://stackoverflow.com/questions/67054759/about-the-maximum-recursion-error-in-fastapi
# Test failures have been observed by internal starlette code when
# using starlette.testclient.TestClient. Looking around in regards
# to the recursion error has really not recommended a course of action
# other than increasing the recursion limit. For now, that is how
# we handle the issue: an optional TEST_RECURSION_LIMIT env var
# provided by the user. Docker uses .env's TEST_RECURSION_LIMIT
# when running test suites.
# TODO: Find a proper fix to this issue.
recursion_limit = int(os.environ.get(
"TEST_RECURSION_LIMIT", sys.getrecursionlimit() + 1000))
sys.setrecursionlimit(recursion_limit)
backend = aurweb.config.get("database", "backend")
if backend not in aurweb.db.DRIVERS:
raise ValueError(
f"The configured database backend ({backend}) is unsupported. "
f"Supported backends: {str(aurweb.db.DRIVERS.keys())}")
session_secret = aurweb.config.get("fastapi", "session_secret")
if not session_secret:
raise Exception("[fastapi] session_secret must not be empty")
app.mount("/static/css",
StaticFiles(directory="web/html/css"),
name="static_css")
app.mount("/static/js",
StaticFiles(directory="web/html/js"),
name="static_js")
app.mount("/static/images",
StaticFiles(directory="web/html/images"),
name="static_images")
# Add application middlewares.
app.add_middleware(AuthenticationMiddleware, backend=BasicAuthBackend())
app.add_middleware(SessionMiddleware, secret_key=session_secret)
# Add application routes.
def add_router(module):
app.include_router(module.router)
util.apply_all(APP_ROUTES, add_router)
# Initialize the database engine and ORM.
get_engine()
app.add_middleware(SessionMiddleware, secret_key=session_secret)
app.include_router(sso.router)
def child_exit(server, worker): # pragma: no cover
""" This function is required for gunicorn customization
of prometheus multiprocessing. """
multiprocess.mark_process_dead(worker.pid)
@app.exception_handler(HTTPException)
async def http_exception_handler(request, exc):
async def internal_server_error(request: Request, exc: Exception) -> Response:
"""
Dirty HTML error page to replace the default JSON error responses.
In the future this should use a proper Arch-themed HTML template.
Catch all uncaught Exceptions thrown in a route.
:param request: FastAPI Request
:return: Rendered 500.html template with status_code 500
"""
context = make_context(request, "Internal Server Error")
# Print out the exception via `traceback` and store the value
# into the `traceback` context variable.
tb_io = io.StringIO()
traceback.print_exc(file=tb_io)
tb = tb_io.getvalue()
context["traceback"] = tb
# Produce a SHA1 hash of the traceback string.
tb_hash = hashlib.sha1(tb.encode()).hexdigest()
# Use the first 7 characters of the sha1 for the traceback id.
# We will use this to log and include in the notification.
tb_id = tb_hash[:7]
redis = redis_connection()
pipe = redis.pipeline()
key = f"tb:{tb_hash}"
pipe.get(key)
retval, = pipe.execute()
if not retval:
# Expire in one hour; this is just done to make sure we
# don't infinitely store these values, but reduce the number
# of automated reports (notification below). At this time of
# writing, unexpected exceptions are not common, thus this
# will not produce a large memory footprint in redis.
pipe.set(key, tb)
pipe.expire(key, 3600)
pipe.execute()
# Send out notification about it.
notif = notify.ServerErrorNotification(
tb_id, context.get("version"), context.get("utcnow"))
notif.send()
retval = tb
else:
retval = retval.decode()
# Log details about the exception traceback.
logger.error(f"FATAL[{tb_id}]: An unexpected exception has occurred.")
logger.error(retval)
return render_template(request, "errors/500.html", context,
status_code=http.HTTPStatus.INTERNAL_SERVER_ERROR)
@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc: HTTPException) \
-> Response:
""" Handle an HTTPException thrown in a route. """
phrase = http.HTTPStatus(exc.status_code).phrase
return HTMLResponse(f"<h1>{exc.status_code} {phrase}</h1><p>{exc.detail}</p>",
status_code=exc.status_code)
context = make_context(request, phrase)
context["exc"] = exc
context["phrase"] = phrase
# Additional context for some exceptions.
if exc.status_code == http.HTTPStatus.NOT_FOUND:
tokens = request.url.path.split("/")
matches = re.match("^([a-z0-9][a-z0-9.+_-]*?)(\\.git)?$", tokens[1])
if matches:
try:
pkgbase = get_pkg_or_base(matches.group(1))
context = pkgbaseutil.make_context(request, pkgbase)
except HTTPException:
pass
try:
return render_template(request, f"errors/{exc.status_code}.html",
context, exc.status_code)
except TemplateNotFound:
return render_template(request, "errors/detail.html",
context, exc.status_code)
@app.middleware("http")
async def add_security_headers(request: Request, call_next: typing.Callable):
""" This middleware adds the CSP, XCTO, XFO and RP security
headers to the HTTP response associated with request.
CSP: Content-Security-Policy
XCTO: X-Content-Type-Options
RP: Referrer-Policy
XFO: X-Frame-Options
"""
try:
response = await util.error_or_result(call_next, request)
except Exception as exc:
return await internal_server_error(request, exc)
# Add CSP header.
nonce = request.user.nonce
csp = "default-src 'self'; "
script_hosts = []
csp += f"script-src 'self' 'nonce-{nonce}' " + ' '.join(script_hosts)
# It's fine if css is inlined.
csp += "; style-src 'self' 'unsafe-inline'"
response.headers["Content-Security-Policy"] = csp
# Add XTCO header.
xcto = "nosniff"
response.headers["X-Content-Type-Options"] = xcto
# Add Referrer Policy header.
rp = "same-origin"
response.headers["Referrer-Policy"] = rp
# Add X-Frame-Options header.
xfo = "SAMEORIGIN"
response.headers["X-Frame-Options"] = xfo
return response
@app.middleware("http")
async def check_terms_of_service(request: Request, call_next: typing.Callable):
""" This middleware function redirects authenticated users if they
have any outstanding Terms to agree to. """
if request.user.is_authenticated() and request.url.path != "/tos":
unaccepted = query(Term).join(AcceptedTerm).filter(
or_(AcceptedTerm.UsersID != request.user.ID,
and_(AcceptedTerm.UsersID == request.user.ID,
AcceptedTerm.TermsID == Term.ID,
AcceptedTerm.Revision < Term.Revision)))
if query(Term).count() > unaccepted.count():
return RedirectResponse(
"/tos", status_code=int(http.HTTPStatus.SEE_OTHER))
return await util.error_or_result(call_next, request)
@app.middleware("http")
async def id_redirect_middleware(request: Request, call_next: typing.Callable):
id = request.query_params.get("id")
if id is not None:
# Preserve query string.
qs = []
for k, v in request.query_params.items():
if k != "id":
qs.append(f"{k}={quote_plus(str(v))}")
qs = str() if not qs else '?' + '&'.join(qs)
path = request.url.path.rstrip('/')
return RedirectResponse(f"{path}/{id}{qs}")
return await util.error_or_result(call_next, request)
import functools
from http import HTTPStatus
from typing import Callable
import fastapi
from fastapi import HTTPException
from fastapi.responses import RedirectResponse
from starlette.authentication import AuthCredentials, AuthenticationBackend
from starlette.requests import HTTPConnection
import aurweb.config
from aurweb import db, filters, l10n, time, util
from aurweb.models import Session, User
from aurweb.models.account_type import ACCOUNT_TYPE_ID
class StubQuery:
""" Acts as a stubbed version of an orm.Query. Typically used
to masquerade fake records for an AnonymousUser. """
def filter(self, *args):
return StubQuery()
def scalar(self):
return 0
class AnonymousUser:
""" A stubbed User class used when an unauthenticated User
makes a request against FastAPI. """
# Stub attributes used to mimic a real user.
ID = 0
class AccountType:
""" A stubbed AccountType static class. In here, we use an ID
and AccountType which do not exist in our constant records.
All records primary keys (AccountType.ID) should be non-zero,
so using a zero here means that we'll never match against a
real AccountType. """
ID = 0
AccountType = "Anonymous"
# AccountTypeID == AccountType.ID; assign a stubbed column.
AccountTypeID = AccountType.ID
LangPreference = aurweb.config.get("options", "default_lang")
Timezone = aurweb.config.get("options", "default_timezone")
Suspended = 0
InactivityTS = 0
# A stub ssh_pub_key relationship.
ssh_pub_key = None
# Add stubbed relationship backrefs.
notifications = StubQuery()
package_votes = StubQuery()
# A nonce attribute, needed for all browser sessions; set in __init__.
nonce = None
def __init__(self):
self.nonce = util.make_nonce()
@staticmethod
def is_authenticated():
return False
@staticmethod
def is_trusted_user():
return False
@staticmethod
def is_developer():
return False
@staticmethod
def is_elevated():
return False
@staticmethod
def has_credential(credential, **kwargs):
return False
@staticmethod
def voted_for(package):
return False
@staticmethod
def notified(package):
return False
class BasicAuthBackend(AuthenticationBackend):
async def authenticate(self, conn: HTTPConnection):
unauthenticated = (None, AnonymousUser())
sid = conn.cookies.get("AURSID")
if not sid:
return unauthenticated
timeout = aurweb.config.getint("options", "login_timeout")
remembered = ("AURREMEMBER" in conn.cookies
and bool(conn.cookies.get("AURREMEMBER")))
if remembered:
timeout = aurweb.config.getint("options",
"persistent_cookie_timeout")
# If no session with sid and a LastUpdateTS now or later exists.
now_ts = time.utcnow()
record = db.query(Session).filter(Session.SessionID == sid).first()
if not record:
return unauthenticated
elif record.LastUpdateTS < (now_ts - timeout):
with db.begin():
db.delete_all([record])
return unauthenticated
# At this point, we cannot have an invalid user if the record
# exists, due to ForeignKey constraints in the schema upheld
# by mysqlclient.
with db.begin():
user = db.query(User).filter(User.ID == record.UsersID).first()
user.nonce = util.make_nonce()
user.authenticated = True
return (AuthCredentials(["authenticated"]), user)
def _auth_required(auth_goal: bool = True):
"""
Enforce a user's authentication status, bringing them to the login page
or homepage if their authentication status does not match the goal.
NOTE: This function should not need to be used in downstream code.
See `requires_auth` and `requires_guest` for decorators meant to be
used on routes (they're a bit more implicitly understandable).
:param auth_goal: Whether authentication is required or entirely disallowed
for a user to perform this request.
:return: Return the FastAPI function this decorator wraps.
"""
def decorator(func):
@functools.wraps(func)
async def wrapper(request, *args, **kwargs):
if request.user.is_authenticated() == auth_goal:
return await func(request, *args, **kwargs)
url = "/"
if auth_goal is False:
return RedirectResponse(url, status_code=int(HTTPStatus.SEE_OTHER))
# Use the request path when the user can visit a page directly but
# is not authenticated and use the Referer header if visiting the
# page itself is not directly possible (e.g. submitting a form).
if request.method in ("GET", "HEAD"):
url = request.url.path
elif (referer := request.headers.get("Referer")):
aur = aurweb.config.get("options", "aur_location") + "/"
if not referer.startswith(aur):
_ = l10n.get_translator_for_request(request)
raise HTTPException(status_code=HTTPStatus.BAD_REQUEST,
detail=_("Bad Referer header."))
url = referer[len(aur) - 1:]
url = "/login?" + filters.urlencode({"next": url})
return RedirectResponse(url, status_code=int(HTTPStatus.SEE_OTHER))
return wrapper
return decorator
def requires_auth(func: Callable) -> Callable:
""" Require an authenticated session for a particular route. """
@functools.wraps(func)
async def wrapper(*args, **kwargs):
return await _auth_required(True)(func)(*args, **kwargs)
return wrapper
def requires_guest(func: Callable) -> Callable:
""" Require a guest (unauthenticated) session for a particular route. """
@functools.wraps(func)
async def wrapper(*args, **kwargs):
return await _auth_required(False)(func)(*args, **kwargs)
return wrapper
def account_type_required(one_of: set):
""" A decorator that can be used on FastAPI routes to dictate
that a user belongs to one of the types defined in one_of.
This decorator should be run after an @auth_required(True) is
dictated.
- Example code:
@router.get('/some_route')
@auth_required(True)
@account_type_required({"Trusted User", "Trusted User & Developer"})
async def some_route(request: fastapi.Request):
return Response()
:param one_of: A set consisting of strings to match against AccountType.
:return: Return the FastAPI function this decorator wraps.
"""
# Convert any account type string constants to their integer IDs.
one_of = {
ACCOUNT_TYPE_ID[atype]
for atype in one_of
if isinstance(atype, str)
}
def decorator(func):
@functools.wraps(func)
async def wrapper(request: fastapi.Request, *args, **kwargs):
if request.user.AccountTypeID not in one_of:
return RedirectResponse("/",
status_code=int(HTTPStatus.SEE_OTHER))
return await func(request, *args, **kwargs)
return wrapper
return decorator
from aurweb.models.account_type import DEVELOPER_ID, TRUSTED_USER_AND_DEV_ID, TRUSTED_USER_ID, USER_ID
from aurweb.models.user import User
ACCOUNT_CHANGE_TYPE = 1
ACCOUNT_EDIT = 2
ACCOUNT_EDIT_DEV = 3
ACCOUNT_LAST_LOGIN = 4
ACCOUNT_SEARCH = 5
ACCOUNT_LIST_COMMENTS = 28
COMMENT_DELETE = 6
COMMENT_UNDELETE = 27
COMMENT_VIEW_DELETED = 22
COMMENT_EDIT = 25
COMMENT_PIN = 26
PKGBASE_ADOPT = 7
PKGBASE_SET_KEYWORDS = 8
PKGBASE_DELETE = 9
PKGBASE_DISOWN = 10
PKGBASE_EDIT_COMAINTAINERS = 24
PKGBASE_FLAG = 11
PKGBASE_LIST_VOTERS = 12
PKGBASE_NOTIFY = 13
PKGBASE_UNFLAG = 15
PKGBASE_VOTE = 16
PKGREQ_FILE = 23
PKGREQ_CLOSE = 17
PKGREQ_LIST = 18
TU_ADD_VOTE = 19
TU_LIST_VOTES = 20
TU_VOTE = 21
PKGBASE_MERGE = 29
user_developer_or_trusted_user = set([USER_ID, TRUSTED_USER_ID, DEVELOPER_ID, TRUSTED_USER_AND_DEV_ID])
trusted_user_or_dev = set([TRUSTED_USER_ID, DEVELOPER_ID, TRUSTED_USER_AND_DEV_ID])
developer = set([DEVELOPER_ID, TRUSTED_USER_AND_DEV_ID])
trusted_user = set([TRUSTED_USER_ID, TRUSTED_USER_AND_DEV_ID])
cred_filters = {
PKGBASE_FLAG: user_developer_or_trusted_user,
PKGBASE_NOTIFY: user_developer_or_trusted_user,
PKGBASE_VOTE: user_developer_or_trusted_user,
PKGREQ_FILE: user_developer_or_trusted_user,
ACCOUNT_CHANGE_TYPE: trusted_user_or_dev,
ACCOUNT_EDIT: trusted_user_or_dev,
ACCOUNT_LAST_LOGIN: trusted_user_or_dev,
ACCOUNT_LIST_COMMENTS: trusted_user_or_dev,
ACCOUNT_SEARCH: trusted_user_or_dev,
COMMENT_DELETE: trusted_user_or_dev,
COMMENT_UNDELETE: trusted_user_or_dev,
COMMENT_VIEW_DELETED: trusted_user_or_dev,
COMMENT_EDIT: trusted_user_or_dev,
COMMENT_PIN: trusted_user_or_dev,
PKGBASE_ADOPT: trusted_user_or_dev,
PKGBASE_SET_KEYWORDS: trusted_user_or_dev,
PKGBASE_DELETE: trusted_user_or_dev,
PKGBASE_EDIT_COMAINTAINERS: trusted_user_or_dev,
PKGBASE_DISOWN: trusted_user_or_dev,
PKGBASE_LIST_VOTERS: trusted_user_or_dev,
PKGBASE_UNFLAG: trusted_user_or_dev,
PKGREQ_CLOSE: trusted_user_or_dev,
PKGREQ_LIST: trusted_user_or_dev,
TU_ADD_VOTE: trusted_user,
TU_LIST_VOTES: trusted_user_or_dev,
TU_VOTE: trusted_user,
ACCOUNT_EDIT_DEV: developer,
PKGBASE_MERGE: trusted_user_or_dev,
}
def has_credential(user: User,
credential: int,
approved_users: list = tuple()):
if user in approved_users:
return True
return user.AccountTypeID in cred_filters[credential]
from datetime import datetime
class Benchmark:
def __init__(self):
self.start()
def _timestamp(self) -> float:
""" Generate a timestamp. """
return float(datetime.utcnow().timestamp())
def start(self) -> int:
""" Start a benchmark. """
self.current = self._timestamp()
return self.current
def end(self):
""" Return the diff between now - start(). """
n = self._timestamp() - self.current
self.current = float(0)
return n
from redis import Redis
from sqlalchemy import orm
async def db_count_cache(redis: Redis, key: str, query: orm.Query,
expire: int = None) -> int:
""" Store and retrieve a query.count() via redis cache.
:param redis: Redis handle
:param key: Redis key
:param query: SQLAlchemy ORM query
:param expire: Optional expiration in seconds
:return: query.count()
"""
result = redis.get(key)
if result is None:
redis.set(key, (result := int(query.count())))
if expire:
redis.expire(key, expire)
return int(result)
""" This module consists of aurweb's CAPTCHA utility functions and filters. """
import hashlib
from jinja2 import pass_context
from aurweb.db import query
from aurweb.models import User
from aurweb.templates import register_filter
def get_captcha_salts():
""" Produce salts based on the current user count. """
count = query(User).count()
salts = []
for i in range(0, 6):
salts.append(f"aurweb-{count - i}")
return salts
def get_captcha_token(salt):
""" Produce a token for the CAPTCHA salt. """
return hashlib.md5(salt.encode()).hexdigest()[:3]
def get_captcha_challenge(salt):
""" Get a CAPTCHA challenge string (shell command) for a salt. """
token = get_captcha_token(salt)
return f"LC_ALL=C pacman -V|sed -r 's#[0-9]+#{token}#g'|md5sum|cut -c1-6"
def get_captcha_answer(token):
""" Compute the answer via md5 of the real template text, return the
first six digits of the hexadecimal hash. """
text = r"""
.--. Pacman v%s.%s.%s - libalpm v%s.%s.%s
/ _.-' .-. .-. .-. Copyright (C) %s-%s Pacman Development Team
\ '-. '-' '-' '-' Copyright (C) %s-%s Judd Vinet
'--'
This program may be freely redistributed under
the terms of the GNU General Public License.
""" % tuple([token] * 10)
return hashlib.md5((text + "\n").encode()).hexdigest()[:6]
@register_filter("captcha_salt")
@pass_context
def captcha_salt_filter(context):
""" Returns the most recent CAPTCHA salt in the list of salts. """
salts = get_captcha_salts()
return salts[0]
@register_filter("captcha_cmdline")
@pass_context
def captcha_cmdline_filter(context, salt):
""" Returns a CAPTCHA challenge for a given salt. """
return get_captcha_challenge(salt)
import configparser
import os
from typing import Any
# Publicly visible version of aurweb. This is used to display
# aurweb versioning in the footer and must be maintained.
# Todo: Make this dynamic/automated.
AURWEB_VERSION = "v5.0.0"
_parser = None
......@@ -12,6 +19,7 @@ def _get_parser():
defaults = os.environ.get('AUR_CONFIG_DEFAULTS', path + '.defaults')
_parser = configparser.RawConfigParser()
_parser.optionxform = lambda option: option
if os.path.isfile(defaults):
with open(defaults) as f:
_parser.read_file(f)
......@@ -20,6 +28,17 @@ def _get_parser():
return _parser
def rehash():
""" Globally rehash the configuration parser. """
global _parser
_parser = None
_get_parser()
def get_with_fallback(section, option, fallback):
return _get_parser().get(section, option, fallback=fallback)
def get(section, option):
return _get_parser().get(section, option)
......@@ -28,5 +47,25 @@ def getboolean(section, option):
return _get_parser().getboolean(section, option)
def getint(section, option):
return _get_parser().getint(section, option)
def getint(section, option, fallback=None):
return _get_parser().getint(section, option, fallback=fallback)
def get_section(section):
if section in _get_parser().sections():
return _get_parser()[section]
def unset_option(section: str, option: str) -> None:
_get_parser().remove_option(section, option)
def set_option(section: str, option: str, value: Any) -> None:
_get_parser().set(section, option, value)
return value
def save() -> None:
aur_config = os.environ.get("AUR_CONFIG", "/etc/aurweb/config")
with open(aur_config, "w") as fp:
_get_parser().write(fp)
from fastapi import Request
from fastapi.responses import Response
from aurweb import config
def samesite() -> str:
""" Produce cookie SameSite value based on options.disable_http_login.
When options.disable_http_login is True, "strict" is returned. Otherwise,
"lax" is returned.
:returns "strict" if options.disable_http_login else "lax"
"""
secure = config.getboolean("options", "disable_http_login")
return "strict" if secure else "lax"
def timeout(extended: bool) -> int:
""" Produce a session timeout based on `remember_me`.
This method returns one of AUR_CONFIG's options.persistent_cookie_timeout
and options.login_timeout based on the `extended` argument.
The `extended` argument is typically the value of the AURREMEMBER
cookie, defaulted to False.
If `extended` is False, options.login_timeout is returned. Otherwise,
if `extended` is True, options.persistent_cookie_timeout is returned.
:param extended: Flag which generates an extended timeout when True
:returns: Cookie timeout based on configuration options
"""
timeout = config.getint("options", "login_timeout")
if bool(extended):
timeout = config.getint("options", "persistent_cookie_timeout")
return timeout
def update_response_cookies(request: Request, response: Response,
aurtz: str = None, aurlang: str = None,
aursid: str = None) -> Response:
""" Update session cookies. This method is particularly useful
when updating a cookie which was already set.
The AURSID cookie's expiration is based on the AURREMEMBER cookie,
which is retrieved from `request`.
:param request: FastAPI request
:param response: FastAPI response
:param aurtz: Optional AURTZ cookie value
:param aurlang: Optional AURLANG cookie value
:param aursid: Optional AURSID cookie value
:returns: Updated response
"""
secure = config.getboolean("options", "disable_http_login")
if aurtz:
response.set_cookie("AURTZ", aurtz, secure=secure, httponly=secure,
samesite=samesite())
if aurlang:
response.set_cookie("AURLANG", aurlang, secure=secure, httponly=secure,
samesite=samesite())
if aursid:
remember_me = bool(request.cookies.get("AURREMEMBER", False))
response.set_cookie("AURSID", aursid, secure=secure, httponly=secure,
max_age=timeout(remember_me),
samesite=samesite())
return response
try:
import mysql.connector
except ImportError:
pass
import functools
import hashlib
import math
import os
import re
try:
import sqlite3
except ImportError:
pass
from typing import Iterable, NewType
import sqlalchemy
from sqlalchemy import create_engine, event
from sqlalchemy.engine.base import Engine
from sqlalchemy.engine.url import URL
from sqlalchemy.orm import Query, Session, SessionTransaction, scoped_session, sessionmaker
import aurweb.config
import aurweb.util
DRIVERS = {
"mysql": "mysql+mysqldb"
}
# Some types we don't get access to in this module.
Base = NewType("Base", "aurweb.models.declarative_base.Base")
def make_random_value(table: str, column: str, length: int):
""" Generate a unique, random value for a string column in a table.
:return: A unique string that is not in the database
"""
string = aurweb.util.make_random_string(length)
while query(table).filter(column == string).first():
string = aurweb.util.make_random_string(length)
return string
def test_name() -> str:
"""
Return the unhashed database name.
The unhashed database name is determined (lower = higher priority) by:
-------------------------------------------
1. {test_suite} portion of PYTEST_CURRENT_TEST
2. aurweb.config.get("database", "name")
During `pytest` runs, the PYTEST_CURRENT_TEST environment variable
is set to the current test in the format `{test_suite}::{test_func}`.
This allows tests to use a suite-specific database for its runs,
which decouples database state from test suites.
:return: Unhashed database name
"""
db = os.environ.get("PYTEST_CURRENT_TEST",
aurweb.config.get("database", "name"))
return db.split(":")[0]
engine = None # See get_engine
def name() -> str:
"""
Return sanitized database name that can be used for tests or production.
If test_name() starts with "test/", the database name is SHA-1 hashed,
prefixed with 'db', and returned. Otherwise, test_name() is passed
through and not hashed at all.
:return: SHA1-hashed database name prefixed with 'db'
"""
dbname = test_name()
if not dbname.startswith("test/"):
return dbname
sha1 = hashlib.sha1(dbname.encode()).hexdigest()
return "db" + sha1
# Module-private global memo used to store SQLAlchemy sessions.
_sessions = dict()
def get_session(engine: Engine = None) -> Session:
""" Return aurweb.db's global session. """
dbname = name()
def get_sqlalchemy_url():
global _sessions
if dbname not in _sessions:
if not engine: # pragma: no cover
engine = get_engine()
Session = scoped_session(
sessionmaker(autocommit=True, autoflush=False, bind=engine))
_sessions[dbname] = Session()
return _sessions.get(dbname)
def pop_session(dbname: str) -> None:
"""
Build an SQLAlchemy for use with create_engine based on the aurweb configuration.
Pop a Session out of the private _sessions memo.
:param dbname: Database name
:raises KeyError: When `dbname` does not exist in the memo
"""
import sqlalchemy
global _sessions
_sessions.pop(dbname)
def refresh(model: Base) -> Base:
""" Refresh the session's knowledge of `model`. """
get_session().refresh(model)
return model
def query(Model: Base, *args, **kwargs) -> Query:
"""
Perform an ORM query against the database session.
This method also runs Query.filter on the resulting model
query with *args and **kwargs.
:param Model: Declarative ORM class
"""
return get_session().query(Model).filter(*args, **kwargs)
def create(Model: Base, *args, **kwargs) -> Base:
"""
Create a record and add() it to the database session.
:param Model: Declarative ORM class
:return: Model instance
"""
instance = Model(*args, **kwargs)
return add(instance)
def delete(model: Base) -> None:
"""
Delete a set of records found by Query.filter(*args, **kwargs).
:param Model: Declarative ORM class
"""
get_session().delete(model)
def delete_all(iterable: Iterable) -> None:
""" Delete each instance found in `iterable`. """
session_ = get_session()
aurweb.util.apply_all(iterable, session_.delete)
def rollback() -> None:
""" Rollback the database session. """
get_session().rollback()
def add(model: Base) -> Base:
""" Add `model` to the database session. """
get_session().add(model)
return model
def begin() -> SessionTransaction:
""" Begin an SQLAlchemy SessionTransaction. """
return get_session().begin()
def get_sqlalchemy_url() -> URL:
"""
Build an SQLAlchemy URL for use with create_engine.
:return: sqlalchemy.engine.url.URL
"""
constructor = URL
parts = sqlalchemy.__version__.split('.')
major = int(parts[0])
minor = int(parts[1])
if major == 1 and minor >= 4: # pragma: no cover
constructor = URL.create
aur_db_backend = aurweb.config.get('database', 'backend')
if aur_db_backend == 'mysql':
return sqlalchemy.engine.url.URL(
'mysql+mysqlconnector',
param_query = {}
port = aurweb.config.get_with_fallback("database", "port", None)
if not port:
param_query["unix_socket"] = aurweb.config.get(
"database", "socket")
return constructor(
DRIVERS.get(aur_db_backend),
username=aurweb.config.get('database', 'user'),
password=aurweb.config.get('database', 'password'),
password=aurweb.config.get_with_fallback('database', 'password',
fallback=None),
host=aurweb.config.get('database', 'host'),
database=aurweb.config.get('database', 'name'),
query={
'unix_socket': aurweb.config.get('database', 'socket'),
},
database=name(),
port=port,
query=param_query
)
elif aur_db_backend == 'sqlite':
return sqlalchemy.engine.url.URL(
return constructor(
'sqlite',
database=aurweb.config.get('database', 'name'),
)
......@@ -39,26 +207,83 @@ def get_sqlalchemy_url():
raise ValueError('unsupported database backend')
def get_engine():
def sqlite_regexp(regex, item) -> bool: # pragma: no cover
""" Method which mimics SQL's REGEXP for SQLite. """
return bool(re.search(regex, str(item)))
def setup_sqlite(engine: Engine) -> None: # pragma: no cover
""" Perform setup for an SQLite engine. """
@event.listens_for(engine, "connect")
def do_begin(conn, record):
create_deterministic_function = functools.partial(
conn.create_function,
deterministic=True
)
create_deterministic_function("REGEXP", 2, sqlite_regexp)
# Module-private global memo used to store SQLAlchemy engines.
_engines = dict()
def get_engine(dbname: str = None, echo: bool = False) -> Engine:
"""
Return the global SQLAlchemy engine.
Return the SQLAlchemy engine for `dbname`.
The engine is created on the first call to get_engine and then stored in the
`engine` global variable for the next calls.
:param dbname: Database name (default: aurweb.db.name())
:param echo: Flag passed through to sqlalchemy.create_engine
:return: SQLAlchemy Engine instance
"""
from sqlalchemy import create_engine
global engine
if engine is None:
if not dbname:
dbname = name()
global _engines
if dbname not in _engines:
db_backend = aurweb.config.get("database", "backend")
connect_args = dict()
if aurweb.config.get("database", "backend") == "sqlite":
# check_same_thread is for a SQLite technicality
# https://fastapi.tiangolo.com/tutorial/sql-databases/#note
is_sqlite = bool(db_backend == "sqlite")
if is_sqlite: # pragma: no cover
connect_args["check_same_thread"] = False
engine = create_engine(get_sqlalchemy_url(), connect_args=connect_args)
Session = sessionmaker(autocommit=False, autoflush=False, bind=engine)
session = Session()
return engine
kwargs = {
"echo": echo,
"connect_args": connect_args
}
_engines[dbname] = create_engine(get_sqlalchemy_url(), **kwargs)
if is_sqlite: # pragma: no cover
setup_sqlite(_engines.get(dbname))
return _engines.get(dbname)
def pop_engine(dbname: str) -> None:
"""
Pop an Engine out of the private _engines memo.
:param dbname: Database name
:raises KeyError: When `dbname` does not exist in the memo
"""
global _engines
_engines.pop(dbname)
def kill_engine() -> None:
""" Close the current session and dispose of the engine. """
dbname = name()
session = get_session()
session.close()
pop_session(dbname)
engine = get_engine()
engine.dispose()
pop_engine(dbname)
def connect():
......@@ -72,34 +297,24 @@ def connect():
return get_engine().connect()
class Connection:
class ConnectionExecutor:
_conn = None
_paramstyle = None
def __init__(self):
aur_db_backend = aurweb.config.get('database', 'backend')
if aur_db_backend == 'mysql':
aur_db_host = aurweb.config.get('database', 'host')
aur_db_name = aurweb.config.get('database', 'name')
aur_db_user = aurweb.config.get('database', 'user')
aur_db_pass = aurweb.config.get('database', 'password')
aur_db_socket = aurweb.config.get('database', 'socket')
self._conn = mysql.connector.connect(host=aur_db_host,
user=aur_db_user,
passwd=aur_db_pass,
db=aur_db_name,
unix_socket=aur_db_socket,
buffered=True)
self._paramstyle = mysql.connector.paramstyle
elif aur_db_backend == 'sqlite':
aur_db_name = aurweb.config.get('database', 'name')
self._conn = sqlite3.connect(aur_db_name)
def __init__(self, conn, backend=aurweb.config.get("database", "backend")):
self._conn = conn
if backend == "mysql":
self._paramstyle = "format"
elif backend == "sqlite":
import sqlite3
self._paramstyle = sqlite3.paramstyle
else:
raise ValueError('unsupported database backend')
def execute(self, query, params=()):
def paramstyle(self):
return self._paramstyle
def execute(self, query, params=()): # pragma: no cover
# TODO: SQLite support has been removed in FastAPI. It remains
# here to fund its support for PHP until it is removed.
if self._paramstyle in ('format', 'pyformat'):
query = query.replace('%', '%%').replace('?', '%s')
elif self._paramstyle == 'qmark':
......@@ -117,3 +332,45 @@ class Connection:
def close(self):
self._conn.close()
class Connection:
_executor = None
_conn = None
def __init__(self):
aur_db_backend = aurweb.config.get('database', 'backend')
if aur_db_backend == 'mysql':
import MySQLdb
aur_db_host = aurweb.config.get('database', 'host')
aur_db_name = name()
aur_db_user = aurweb.config.get('database', 'user')
aur_db_pass = aurweb.config.get_with_fallback(
'database', 'password', str())
aur_db_socket = aurweb.config.get('database', 'socket')
self._conn = MySQLdb.connect(host=aur_db_host,
user=aur_db_user,
passwd=aur_db_pass,
db=aur_db_name,
unix_socket=aur_db_socket)
elif aur_db_backend == 'sqlite': # pragma: no cover
# TODO: SQLite support has been removed in FastAPI. It remains
# here to fund its support for PHP until it is removed.
import sqlite3
aur_db_name = aurweb.config.get('database', 'name')
self._conn = sqlite3.connect(aur_db_name)
self._conn.create_function("POWER", 2, math.pow)
else:
raise ValueError('unsupported database backend')
self._conn = ConnectionExecutor(self._conn, aur_db_backend)
def execute(self, query, params=()):
return self._conn.execute(query, params)
def commit(self):
self._conn.commit()
def close(self):
self._conn.close()
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment