This gives a more fluent migration phase which does not forcefully
require a coordinated release with devtools.

Fixes #45

Use GIT_TERMINAL_PROMPT=0 to enforce Git not asking for any prompts or
credentials. GitLab does not return none existing repositories but
instead prompts for credentials to avoid side channel information leaks.
However, we explicitly do not want to get any such prompts, instead we
fail gracefully.

This allows to easily override the disabled networking, like passing the
host network to have a simple development environment.

Use nproc to determine the default job counter.

This effectively also fixes multiple issues with combinations of any
packages and their native subset removal. The sacrifice is a different
CLI interface that caller need to adapt to.

For specific use cases a new --arch options has been introduced to just
remove a specific architecture.

For removing only partial packages from a split pkgbase, the option
--partial must be passed. By default only allow removing complete
pkgbase. This makes both operation mutual and explicit, either remove
partial packages or remove complete pkgbases.

Fixes #21
Fixes #39

Ensure that staged packages in outer stability layers must be newer and
not staged multiple times in the same layer.

Iterate through all staged packages and check that all stability layers
above only provide newer variants.

Also check that each stability layer only contains one variant of the
same pkgname and arch.

Fixes #30

These helper functions can be used to convert repos to and from
stability layers. The repos are layered as depths, and outer layers
above the lower layers override packages.

This will be used later for checks to ensure outer layers can only
contain higher versioned variants.

This also aligns with upstream recommendations and simplifies
readability.

Previously fulfilling only one of the found repositories was enough to
pass this check, which is a severe shortcoming leading to potentially
still bypassed unstable package rebuilds. One scenario was to stage a
stable and testing package while currently both, the testing and staging
repo contained rebuilds. This would leave a none updated staging package
behind, rendering this check insufficient.

Now, after collecting all required repositories, ensure we have staged
candidates to every single one of them, otherwise fail.

Also operate purely on the staging transaction for improved atomicity
instead of accessing the staging directory multiple times during
execution.

Check for reproducible builds dependent packages in a batch for all
staged packages. Greatly improve the performance by building a lookup
table for staged packages and only check all required dependents
collected from all stages packages once.

Furthermore, add a fast path to directly check for the existence of
dependent dependencies with the currently default PKGEXT and only
fall back to glob search in case of lookup miss.

Also operate purely on the staging transaction for improved atomicity
instead of accessing the staging directory multiple times during
execution.

Let's just append a newline at the end of the file, which makes it a
readable and valid tar file, but should fail when verifying the
integrity and authentication.

Fixes #40

This setup isn't strictly required for the unit test setup, however the
additional configuration is very handy when the container runtime is
used to get an interactive development environment where dbscripts
operations can be executed on the CLI.

The execution used to be way too fragile, as the script simply executed
even during errors if not each call was explicitly checked and exited.
For the cleanup routine we preserve the previous error status in case
cleanup was not explicitly called with a status code.

Avoid using globs and wildcards on the staging directory multiple times
during the execution. Previously this lead to data races and
inconsistencies which also allowed to bypass checks due to the nature of
this TOCTOU problem.

Now as first step we collect all files of this transaction into an
associative array and only operate on this snapshot, avoiding any side
effects if further packages are uploaded during execution. On top all
checks are done before anything is released.

Release operations and state repo changes are collected into actions
before execution, which also avoids unnecessary NOP actions.

Additionally, execute checks in different batches, reducing the window
where the repository needs to be locked, which subsequently will block
release processes for a shorter period of time during large rebuilds.

Fixes #22
Fixes #35
Fixes #42

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

The `any` arch in the context of dbscripts arguments was inconsistently
handled. From a pacman binary repo perspective, there is no `any` repo,
just binary repositories that contain `any` packages that can be used
across multiple arches.
Add proper handling of `any` as an argument to db-remove by collecting
all possible targets from all native arches. This fixes the issue of
having left over state entries that lead to inconsistencies.

Fixes #44

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Fixes #37

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

This fixes the issue that a Git operation may already be running on the
state repo leading to an error which subsequently aborts db operations.

Fixes #32

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Felix Yan authored 1 year ago and Levente Polyak committed 1 year ago

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Jelle van der Waa authored 1 year ago and Levente Polyak committed 1 year ago

This automatically gives access to multilib without the need to have an
extra unix multilib group for the ACL. The multilib repo follows the
same principles as extra for junior packager access.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

This avoids most fetch scenarios when moving packages between repos as
the released tags should already be available from a previous db-update.

Now the existence of the expected tag is checked prior to any Git fetch
operations, massively speeding up big rebuild moves.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

When local users are using git config values or environment variables
then may unexpectedly propagate into the state repo commit. To avoid
this declare and unset the environment variables and set options
accordingly.

This is required for a lot of cross repo checks like the "leapfrog"
check.

The only viable way to properly check for packager identity is when
doing it on the backend as client side packaging utility can
deliberately circumvent expectations. Hence we stop blindly trusting
this side and check the official identity at package release, as it was
supposed to work.

https://docs.gitlab.com/ee/user/reserved_names.html

This avoids potentially unreproducible packages when they were created
using unofficial tooling. In the future we could also check for the
actual tooling version.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Automatic path conversion is limited to GitLab API v4 and will be
removed in the future. It's expected that the caller does the path
conversion on caller side and only passes a valid path to the API within
its limitations.

Hence convert project names to valid paths:
  1. replace single '+' between word boundaries with '-'
  2. replace any other '+' with literal 'plus'
  3. replace any special chars other than '_', '-' and '.' with '-'

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>