The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.

Two command line arguments are available:

    --extended | Include License, Keywords, Groups, relations
                 and dependencies.

When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.

Archive JSON is in the following format: line-separated package objects
enclosed in a list:

    [
    {...},
    {...},
    {...}
    ]

Signed-off-by: Kevin Morris <kevr@0cost.org>

This reverts commit 986fa9ee.

Along with this initial requests metric implementation,
we also now serve the `/metrics` route, which grabs request
metrics out of cache and renders them properly for Prometheus.

**NOTE** Metrics are only enabled when the aurweb system admin
has enabled caching by configuring `options.cache` correctly
in `$AUR_CONFIG`. Otherwise, an error is logged about no cache
being configured.

New dependencies have been added which require the use of
`composer`. See `INSTALL` for the dependency section in regards
to composer dependencies and how to install them properly for
aurweb.

Metrics are in the following forms:

    aurweb_http_requests_count(method="GET",route="/some_route")
    aurweb_api_requests_count(method="GET",route="/rpc",type="search")

This should allow us to search through the requests for specific routes
and queries.

Signed-off-by: Kevin Morris <kevr@0cost.org>

turret-dev authored 3 years ago and Eli Schwartz committed 3 years ago

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>

Kevin Morris authored 4 years ago and Eli Schwartz committed 3 years ago

uvicorn is subjectively nicer to play with for local dev work, but
hypercorn is required in order to do HTTP/2 which is fairly
performance-important.

Signed-off-by: Kevin Morris <kevr@0cost.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Co-authored-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>

Frédéric Mangano-Tarumi authored 5 years ago and Lukas Fleischer committed 5 years ago

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Frédéric Mangano-Tarumi authored 5 years ago and Lukas Fleischer committed 5 years ago

The new schema was generated with sqlacodegen and then manually adjusted
to fit schema/aur-schema.sql faithfully, both in the organisation of the
code and in the SQL generated by SQLAlchemy.

Initializing the database now requires the new tool aurweb.initdb.
References to aur-schema.sql have been updated and the old schema
dropped.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Eli Schwartz authored 6 years ago and Lukas Fleischer committed 6 years ago

In the process, rename config.proto to config.defaults (because that is
what it is now).

Also use dict.get('key', default_value) when querying os.environ, rather
than an if block, as it is more pythonic/readable/concise, and reduces
the number of dict lookups.

This change allows aurweb configuration to be done via either:
- copying config.defaults to config and modifying values
- creating a new config only containing modified values, next to a
  config.defaults containing unmodified values

The motivation for this change is to enable ansible configuration in our
flagship deployment by storing only changed values, and deferring to
config.defaults otherwise.

A side benefit is, it is easier to see what has changed by inspecting
only the site configuration file.

If a config.defaults file does not exist next to $AUR_CONFIG or in
$AUR_CONFIG_DEFAULTS, it is ignored and *all* values are expected to
live in the modified config file.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Jelle van der Waa authored 7 years ago and Lukas Fleischer committed 7 years ago

Change the defines to config_get and add one cache option and one option
to define memcache_servers. Mention the required dependency to get
memcached working in the INSTALL file.

Signed-off-by: Jelle van der Waa <jelle@vdwaa.nl>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Remy Marquis authored 7 years ago and Lukas Fleischer committed 7 years ago

To people unfamiliar with the code, it is not obvious that
the pdo_* PHP extensions must be enabled.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Add installation instructions for python-bleach and python-markdown
which are required for the rendercomment script.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Eli Schwartz authored 7 years ago and Lukas Fleischer committed 7 years ago

AUR_PRIVILEGED allows people with privileged AUR accounts to evade the
block on non-fast-forward commits. While valid in this case, we should
not do so by default, since in at least one case a TU did this without
realizing there was an existing package.
( https://aur.archlinux.org/packages/rtmidi/

 )

Switch to using allow_overwrite to check for destructive actions.
Use .ssh/config "SendEnv" on the TU's side and and sshd_config
"AcceptEnv" in the AUR server to specifically request overwrite access.
TUs should use: `AUR_OVERWRITE=1 git push --force`

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Mark Weiman authored 7 years ago and Lukas Fleischer committed 7 years ago

Add instructions to test aurweb's web interface via the PHP built-in web
server.

Signed-off-by: Mark Weiman <mark.weiman@markzz.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Add a note to the Git/SSH interface documentation that we recommend to
disable automatic garbage collection and use a maintenance script to
cleanup and optimize the Git repository instead.

Also, add a reference to the Git/SSH interface documentation to the Git
repository setup instructions in INSTALL.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Since d4fe77ac (Reorganize Git interface scripts, 2016-10-08), the key
components of the aurweb SSH interface are installed system-wide. Update
the default configuration path to point to a central location.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Move the Git interface scripts from git-interface/ to aurweb/git/. Use
setuptools to automatically create wrappers which can be installed using
`python3 setup.py install`. Update the configuration files, the test
suite as well as the INSTALL and README files to reflect these changes.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

The projects.archlinux.org subdomain was moved to git.archlinux.org.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

In 002d348d (Describe how to omit "have" lines, 2015-11-14), we added
instructions on how to omit "have" lines originating from other package
repositories. Fix those instructions such that the HEAD ref of the
repository is transferred properly.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Johannes Löthberg authored 9 years ago and Lukas Fleischer committed 9 years ago

python-srcinfo is a more transparent and simpler library for parsing
SRCINFO files.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Mark Weiman authored 9 years ago and Lukas Fleischer committed 9 years ago

Signed-off-by: Mark Weiman <mark.weiman@markzz.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

A new feature in Git allows for omitting "have" lines corresponding to
refs outside the current Git namespace. Explain how to enable this
feature in the INSTALL instructions and in the Git interface
documentation.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Using uWSGI for the Smart HTTP protocol caused some issues, see e.g.
FS#45428. Suggest using fcgiwrap instead which is more lightweight, has
better documentation and is easier to debug.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Leonidas Spyropoulos authored 9 years ago and Lukas Fleischer committed 9 years ago

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Specifically check for URIs git-http-backend(1) can handle. This also
allows us to make the ".git" suffix optional.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

To make sure we never lose any history, non-fast-forwards are forbidden.
Instead of relying on receive.denyNonFastForwards, add a simple check to
the update hook. This has the added benefit of more flexibility.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Avoid manual editing of the wrapper script when a non-standard path is
used.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

* Move scripts/git-integration/ to git-interface/.
* Move scripts/aurblup/aurblup.py to scripts/aurblup.py.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Instead of using one Git repository per package, use a single large
object storage for space efficiency. The refs of the individual package
bases are divided using gitnamespaces(7) which allows for exposing each
namespace as an independent repository easily. Also, git-serve is
modified to create a branch for each package, allowing to browse the
large repository with cgit.

Helped-by: Florian Pritz <bluewind@xinu.at>
Helped-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Extended AuthorizedKeysCommand parameters are now officially supported
by OpenSSH.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Use the latest version of Damien Miller's patch to extend the parameters
to the AuthorizedKeysCommand.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>

Rename the project to help differentiate between the software providing
access to the Arch User Repository and the collection of source packages
itself.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>

Do not waste disk space by copying dozens of unneeded sample hooks. Use
a custom template directory that only includes the git-update hook.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>

* Add more details on how to configure the web server.
* Clarify that the OpenSSH patch is build against OpenSSH 6.7p1.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>

There aren't any other subdirectories in support/. Reduce the nesting
depth by moving schema/ to the top-level source directory.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>

Signed-off-by: canyonknight <canyonknight@gmail.com>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>