See doc/git-archive.md for general Git archive specifications
See doc/repos/metadata-repo.md for info and direction related to the new Git metadata archive

A new option has been added for configuration of SMTP timeout:
- notifications.smtp-timeout

During tests, we can change this timeout to be small, so we aren't
depending on hardware-based RNG to pass the timeout.

Without a timeout, users can run into a long-running test for no
particular reason.

Signed-off-by: Kevin Morris <kevr@0cost.org>

As repeats of these traceback notifications were annoying some of
the devops staff, and it took coordination to share tracebacks with
developers, this commit removes that responsibility off of devops
by reporting tracebacks to Gitlab repositories in the form of issues.

- removed ServerErrorNotification
- removed notifications.postmaster configuration option
- added notifications.gitlab-instance option
- added notifications.error-project option
- added notifications.error-token option
- added aurweb.exceptions.handle_form_exceptions, a POST route decorator

Issues are filed confidentially. This change will need updates
in infrastructure's ansible configuration before this can be
applied to aur.archlinux.org.

Signed-off-by: Kevin Morris <kevr@0cost.org>

This commit does quite a bit:
- Catches unhandled exceptions raised in the route handler and
  produces a 500 Internal Server Error Arch-themed response.
- Each unhandled exception causes a notification to be sent to new
  `notifications.postmaster` email with a "Traceback ID."
- Traceback ID is logged to the server along with the traceback which
  caused the 500: `docker-compose logs fastapi | grep '<traceback_id>'`
- If `options.traceback` is set to `1`, traceback is displayed in
  the new 500.html template.

Signed-off-by: Kevin Morris <kevr@0cost.org>

- Removed tuvotereminder sharness test.
- Added [tuvotereminder] section to config.defaults.
- Added `range_start` option to config.defaults [tuvotereminder].
- Added `range_end` option to config.defaults [tuvotereminder].

Signed-off-by: Kevin Morris <kevr@0cost.org>

Signed-off-by: Kevin Morris <kevr@0cost.org>

Signed-off-by: Kevin Morris <kevr@0cost.org>

A user that can create databases is now required for tests,
we use the 'root' user in Docker.

Added docker services:
---------------------
- mariadb_test - host localhost:13307

Signed-off-by: Kevin Morris <kevr@0cost.org>

- added config option [mkpkglists] archivedir
    - created by mkpkglists

Signed-off-by: Kevin Morris <kevr@0cost.org>

Better defaults for Docker here.

Signed-off-by: Kevin Morris <kevr@0cost.org>

The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.

Two command line arguments are available:

    --extended | Include License, Keywords, Groups, relations
                 and dependencies.

When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.

Archive JSON is in the following format: line-separated package objects
enclosed in a list:

    [
    {...},
    {...},
    {...}
    ]

Signed-off-by: Kevin Morris <kevr@0cost.org>

Signed-off-by: Kevin Morris <kevr@0cost.org>

Here, we default to using root as the storage directory. Primarily
because it makes sense in Docker; config.dev can always be fixed up
by developers to reflect local system changes.

Signed-off-by: Kevin Morris <kevr@0cost.org>

It seems the ftp mirror from kernel.org cannot be used anymore,
but the https mirror can. So, the default config has been updated
to reflect this; otherwise, aurblup bugs out.

Signed-off-by: Kevin Morris <kevr@0cost.org>

The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.

Two command line arguments are available:

    --extended | Include License, Keywords, Groups, relations
                 and dependencies.

When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.

Archive JSON is in the following format: line-separated package objects
enclosed in a list:

    [
    {...},
    {...},
    {...}
    ]

Signed-off-by: Kevin Morris <kevr@0cost.org>

Two new options have been added:

- [devel] commit_url
    - URL including an %s format specifier that can be used to link
      to a webpage for the commit.
- [devel] commit_hash
    - HEAD's commit hash (produced via `git rev-parse HEAD`)

If a `[devel] commit_hash` is configured, a link to the commit based on
`[devel] commit_url` will be displayed in the aurweb footer in
the form: `HEAD@<commit_hash>`. If no `[devel] commit_url` is
configured, a non-linked hash will be displayed.

Signed-off-by: Kevin Morris <kevr@0cost.org>

New configuration options:

- `[ratelimit] cache`
    - A boolean indicating whether we should use configured cache (1)
      or database (0) for ratelimiting.

Signed-off-by: Kevin Morris <kevr@0cost.org>

Signed-off-by: Kevin Morris <kevr@0cost.org>

For the dev environment, we use a no-op address. We don't want
to be spamming aur-requests with development notifications.

Signed-off-by: Kevin Morris <kevr@0cost.org>

In terms of performance, most queries on this page win over
PHP in query times, with the exception of sorting by Voted or
Notify (archlinux/aurweb#102

).
Otherwise, there are a few modifications: described below.

* Pagination
    * The `paginate` Python module has been used in the FastAPI
      project
      here to implement paging on the packages search page. This
      changes how pagination is displayed, however it serves the
      same purpose. We'll take advantage of this module in other
      places as well.
* Form action
    * The form action for actions now use `POST /packages` to
      perform. This is currently implemented and will be
      addressed in a follow-up commit.
* Input names and values
    * Input names and values have been modified to satisfy the
      snake_case naming convention we'd like to use as much as
      possible.
    * Some input names and values were modified to comply with
      FastAPI Forms: (IDs[<id>]) -> (IDs, <id>).

Signed-off-by: Kevin Morris <kevr@0cost.org>

As the new-age Python package manager, Poetry brings a lot
of good additions to the table. It allows us to more easily
deal with virtualenvs for the project and resolve dependencies.

As of this commit, `requirements.txt` is replaced by Poetry,
configured at `pyproject.toml`.

In Docker and GitLab, we currently use Poetry in a root fashion.
We should work toward purely using virtualenvs in Docker, but,
for now we'd like to move forward with other things. The project
can still be installed to a virtualenv and used on a user's system
through Poetry; it is just not yet doing so in Docker.

Modifications:

* docker/scripts/install-deps.sh
    * Remove python dependencies.
* conf/config.defaults
    * Script paths have been updated to use '/usr/bin'.
* docker/git-entrypoint.sh
    * Use '/usr/bin/aurweb-git-auth' instead of
      '/usr/local/bin/aurweb-git-auth'.

Additions:

* docker/scripts/install-python-deps.sh
    * A script used purely to install Python dependencies with Poetry.
      This has to be used within the aurweb project directory and
      requires system-wide dependencies are installed beforehand.
    * Also upgrades system-wide pip.

Signed-off-by: Kevin Morris <kevr@0cost.org>

This includes the addition of the python-fakeredis package,
used for stubbing python-redis when a user does not have a
configured cache.

Signed-off-by: Kevin Morris <kevr@0cost.org>

Additionally, setup memcached for php-fpm.

Signed-off-by: Kevin Morris <kevr@0cost.org>

During development, the lower this value is (must be >= 4)
equals faster User generation. This is particularly useful
for running tests.

In production, a higher value (like 12 which is used by various
popular frameworks) should be used.

Signed-off-by: Kevin Morris <kevr@0cost.org>

This also updates `test/README.md` to be a bit more specific
and precise with our current state of testing.

Signed-off-by: Kevin Morris <kevr@0cost.org>

All the arch-x@archlinux.org -> arch-x@lists.archlinux.org aliases will
be dropped soon[1].

[1] https://lists.archlinux.org/pipermail/arch-dev-public/2021-June/030462.html

Steven Guikal authored 3 years ago and Eli Schwartz committed 3 years ago

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>

First off: This commit changes the default development database
backend to mysql. sqlite, however, is still completely supported
with the caveat that a user must now modify config.dev to use
the sqlite backend.

While looking into this, it was discovered that our SQLAlchemy
backend for mysql (mysql-connector) completely broke model
attributes when we switched to utf8mb4_bin (binary) -- it does
not correct the correct conversion to and from binary utf8mb4.

The new, replacement dependency mysqlclient does. mysqlclient
is also recommended in SQLAlchemy documentation as the "best"
one available.

The mysqlclient backend uses a different exception flow then
sqlite, and so tests expecting IntegrityError has to be modified
to expect OperationalError from sqlalchemy.exc.

So, for each model that we define, check keys that can't be
NULL and raise sqlalchemy.exc.IntegrityError if we have to.
This way we keep our exceptions uniform.

Signed-off-by: Kevin Morris <kevr@0cost.org>

Introduced `get|post` `/passreset` routes. These routes mimic the
behavior of the existing PHP implementation, with the exception of
HTTP status code returns.

Routes added:
    GET /passreset
    POST /passreset

Routers added:
    aurweb.routers.accounts

* On an unknown user or mismatched resetkey (where resetkey must ==
  user.resetkey), return HTTP status NOT_FOUND (404).
* On another error in the request, return HTTP status BAD_REQUEST (400).

Both `get|post` routes requires that the current user is **not**
authenticated, hence `@auth_required(False, redirect="/")`.

+ Added auth_required decorator to aurweb.auth.
+ Added some more utility to aurweb.models.user.User.
+ Added `partials/error.html` template.
+ Added `passreset.html` template.
+ Added aurweb.db.ConnectionExecutor functor for paramstyle logic.
  Decoupling the executor logic from the database connection logic
  is needed for us to easily use the same logic with a fastapi
  database session, when we need to use aurweb.scripts modules.

At this point, notification configuration is now required to complete
tests involved with notifications properly, like passreset.
`conf/config.dev` has been modified to include [notifications] sendmail,
sender and reply-to overrides. Dockerfile and .gitlab-ci.yml have been
updated to setup /etc/hosts and start postfix before running tests.

* setup.cfg: ignore E741, C901 in aurweb.routers.accounts

These two warnings (shown in the commit) are not dangerous and a bi-product
of maintaining compatibility with our current code flow.

Signed-off-by: Kevin Morris <kevr@0cost.org>

Leonidas Spyropoulos authored 3 years ago and Kevin Morris committed 3 years ago

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>

+ Mounted static files (at web/html) to /static.
+ Added AURWEB_VERSION to aurweb.config (this is used around HTML
  to refer back to aurweb's release on git.archlinux.org), so we
  need it easily accessible in the Python codebase.
+ Implemented basic Jinja2 partials to put together whole aurweb
  pages. This may be missing some things currently and is a WIP
  until this set is ready to be merged.
+ Added config [options] aurwebdir = YOUR_AUR_ROOT; this configuration
  option should specify the root directory of the aurweb project.
  It is used by various parts of the FastAPI codebase to target
  project directories.

Added routes via aurweb.routers.html:
    * POST /language: Set your session language.
    * GET /favicon.ico: Redirect to /static/images/favicon.ico.
        * Some browsers always look for $ROOT/favicon.ico to get an icon
          for the page being loaded, regardless of a specified "shortcut
          icon" given in a <link> directive.
    * GET /: Home page; WIP.

* Updated aurweb.routers.html.language passes query parameters to
  its next redirection.

When calling aurweb.templates.render_template, the context passed should
be formed via the aurweb.templates.make_context. See
aurweb.routers.html.index for an example of this.

Signed-off-by: Kevin Morris <kevr@0cost.org>

+ Added SUPPORTED_LANGUAGES, a global constant dictionary of
  language => display pairs for languages we support.
+ Add Translator.get_translator, a function used to retrieve a
  translator after initializing it (if needed). Use `fallback=True`
  while creating languages, in case we setup a language that we
  don't have a translation for, it will noop the translation.
  This is particularly useful for "en," since we do not translate
  it, but doing this will allow us to go through our normal translation
  flow in any case.
+ Added typing.
+ Added get_request_language, a function that grabs the language for
  a request session, defaulting to aurweb.config [options] default_lang.
+ Added get_raw_translator_for_request, a function that retrieves
  the concrete translation object for a given language.
+ Added tr, a jinja2 contextfilter that can be used to inline translate
  strings in jinja2 templates.
+ Added `python-jinja` dep to .gitlab-ci.yml. This needs to be
  included in documentation before this set is merged in.
+ Introduce pytest units (test_l10n.py) in `test` along with
  __init__.py, which marks `test` as a test package.
+ Additionally, fix up notify.py to use the global translator. Also
  reduce its source width to <= 80 by newlining some code.
+ Additionally, prepare locale in .gitlab-ci.yml and add
  aurweb.config [options] localedir to config.dev with YOUR_AUR_ROOT
  like others.

Signed-off-by: Kevin Morris <kevr@0cost.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Frédéric Mangano-Tarumi authored 4 years ago and Lukas Fleischer committed 4 years ago

This feature was originally introduced by
f961ffd9 as a fix for FS#12898
<https://bugs.archlinux.org/task/12898

>.

As of today, it is broken because of the `q.SessionID IS NULL` condition
in the WHERE clause, which can’t be true because SessionID is not
nullable. As a consequence, the session limit was not applied.

The fact the absence of the session limit hasn’t caused any issue so
far, and hadn’t even been noticed, suggests the feature is unneeded.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Frédéric Mangano-Tarumi authored 4 years ago and Lukas Fleischer committed 4 years ago

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Frédéric Mangano-Tarumi authored 4 years ago and Lukas Fleischer committed 4 years ago

Developers can go to /sso/login to get redirected to the SSO. On
successful login, the ID token is displayed.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Frédéric Mangano-Tarumi authored 4 years ago and Lukas Fleischer committed 4 years ago

conf/config.dev’s purpose is to provide a lighter configuration template
for developers, and split development-specific options off the default
configuration file.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Frédéric Mangano-Tarumi authored 4 years ago and Lukas Fleischer committed 4 years ago

aurweb.spawn used to launch only PHP’s built-in server. Now it spawns a
dummy FastAPI application too. Since both stacks spawn their own HTTP
server, aurweb.spawn also spawns nginx as a reverse proxy to mount them
under the same base URL, defined by aur_location in the configuration.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>