Fixes #48

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Also lock critical sections to each user has an own lock on their
staging directory to avoid race conditions during multiple processes
handling the same package releases.

Fixes #49
Fixes #38

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

An initial git clone and especially a git fetch operation can occur
concurrently through multiple packagers and processes. Guard these git
operations with a fetch lock inside the actual git repository as well as
a special lock stored in the shared global lock directory before
starting an initial clone.

Fixes #49

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Using read does not work as the previous implementer expected the
feature to work, as a delimiter of d'\n' will not operate like expected.

Use proper line based output parsing by using mapfile to read the output
line by line into an array, and then use array range to reference the
content into variables. This way we can actually check if the subshell
had any errors and abort in case anything went wrong.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

This paradigm seems to be in dbscripts since a decade, which effectively
is void. Using a subshell with a local variable declaration basically
always leads to success as the local declaration masks the return code
of the subshell, which means an early exit using `|| return 1` will
never work.

Fix this by splitting local variable declaration from assignment.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Maybe this doesn't matter because the change isn't staged yet anyway,
but it makes me feel safer.

This broke db-update for more than one repo at a time. Most importantly,
because the `pkgkeys` array kept accumulating keys, we created bogus
copies of state repo entries, "releasing" packages into multiple repos.

This gives a more fluent migration phase which does not forcefully
require a coordinated release with devtools.

Fixes #45

Use GIT_TERMINAL_PROMPT=0 to enforce Git not asking for any prompts or
credentials. GitLab does not return none existing repositories but
instead prompts for credentials to avoid side channel information leaks.
However, we explicitly do not want to get any such prompts, instead we
fail gracefully.

This allows to easily override the disabled networking, like passing the
host network to have a simple development environment.

Use nproc to determine the default job counter.

This effectively also fixes multiple issues with combinations of any
packages and their native subset removal. The sacrifice is a different
CLI interface that caller need to adapt to.

For specific use cases a new --arch options has been introduced to just
remove a specific architecture.

For removing only partial packages from a split pkgbase, the option
--partial must be passed. By default only allow removing complete
pkgbase. This makes both operation mutual and explicit, either remove
partial packages or remove complete pkgbases.

Fixes #21
Fixes #39

Ensure that staged packages in outer stability layers must be newer and
not staged multiple times in the same layer.

Iterate through all staged packages and check that all stability layers
above only provide newer variants.

Also check that each stability layer only contains one variant of the
same pkgname and arch.

Fixes #30

These helper functions can be used to convert repos to and from
stability layers. The repos are layered as depths, and outer layers
above the lower layers override packages.

This will be used later for checks to ensure outer layers can only
contain higher versioned variants.

This also aligns with upstream recommendations and simplifies
readability.

Previously fulfilling only one of the found repositories was enough to
pass this check, which is a severe shortcoming leading to potentially
still bypassed unstable package rebuilds. One scenario was to stage a
stable and testing package while currently both, the testing and staging
repo contained rebuilds. This would leave a none updated staging package
behind, rendering this check insufficient.

Now, after collecting all required repositories, ensure we have staged
candidates to every single one of them, otherwise fail.

Also operate purely on the staging transaction for improved atomicity
instead of accessing the staging directory multiple times during
execution.

Check for reproducible builds dependent packages in a batch for all
staged packages. Greatly improve the performance by building a lookup
table for staged packages and only check all required dependents
collected from all stages packages once.

Furthermore, add a fast path to directly check for the existence of
dependent dependencies with the currently default PKGEXT and only
fall back to glob search in case of lookup miss.

Also operate purely on the staging transaction for improved atomicity
instead of accessing the staging directory multiple times during
execution.

Let's just append a newline at the end of the file, which makes it a
readable and valid tar file, but should fail when verifying the
integrity and authentication.

Fixes #40

This setup isn't strictly required for the unit test setup, however the
additional configuration is very handy when the container runtime is
used to get an interactive development environment where dbscripts
operations can be executed on the CLI.

The execution used to be way too fragile, as the script simply executed
even during errors if not each call was explicitly checked and exited.
For the cleanup routine we preserve the previous error status in case
cleanup was not explicitly called with a status code.

Avoid using globs and wildcards on the staging directory multiple times
during the execution. Previously this lead to data races and
inconsistencies which also allowed to bypass checks due to the nature of
this TOCTOU problem.

Now as first step we collect all files of this transaction into an
associative array and only operate on this snapshot, avoiding any side
effects if further packages are uploaded during execution. On top all
checks are done before anything is released.

Release operations and state repo changes are collected into actions
before execution, which also avoids unnecessary NOP actions.

Additionally, execute checks in different batches, reducing the window
where the repository needs to be locked, which subsequently will block
release processes for a shorter period of time during large rebuilds.

Fixes #22
Fixes #35
Fixes #42

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

The `any` arch in the context of dbscripts arguments was inconsistently
handled. From a pacman binary repo perspective, there is no `any` repo,
just binary repositories that contain `any` packages that can be used
across multiple arches.
Add proper handling of `any` as an argument to db-remove by collecting
all possible targets from all native arches. This fixes the issue of
having left over state entries that lead to inconsistencies.

Fixes #44

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Fixes #37

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

This fixes the issue that a Git operation may already be running on the
state repo leading to an error which subsequently aborts db operations.

Fixes #32

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Felix Yan authored 1 year ago and Levente Polyak committed 1 year ago

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Jelle van der Waa authored 1 year ago and Levente Polyak committed 1 year ago

This automatically gives access to multilib without the need to have an
extra unix multilib group for the ACL. The multilib repo follows the
same principles as extra for junior packager access.

Signed-off-by: Levente Polyak <anthraxx@archlinux.org>