Jakub Klinkovský authored 1 year ago and Christian Heusel committed 4 months ago

Instead of passing the command as one complex string to ssh, we create
an SSH master connection and use its control socket in multiple simpler
commands. The same socket is passed also to rsync to transfer the srcpkg
to the remote and to download the build artifacts.

Previously, the srcpkg was passed via stdin to ssh, which prevented
`pkgctl build --offload --inspect` from working. This change frees stdin
for proper remote ptty allocation.

However, it seems that ssh commands with and without the `-t` flag
cannot be multiplexed on a single connection, so there are technically
two SSH connections active for the offload-build execution.

Setup a blank config for packages without remote sources. This is
helpful so other commands like `pkgctl version check` operate gracefully
as well as we have easy way to find packages that miss nvchecker
config.

This must only be used for cases without an upstream, please reach out
to the developer team for guidance regarding upstream sources that are
hard to configure.

Component: pkgctl version setup
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Andreas Schleifer authored 4 months ago and Christian Heusel committed 4 months ago

This allows for tools and data visualization to interface against pkgctl
with a machine readable output.

Fixes #237

Component: pkgctl version check
Signed-off-by: Andreas Schleifer <segaja@archlinux.org>
Co-authored-by: Levente Polyak <anthraxx@archlinux.org>

From systemd-nspawn(1),

	--timezone=
		Configures how /etc/localtime inside of the container (i.e. local timezone synchronization from host to container) shall be handled. Takes one of "off", "copy", "bind", "symlink", "delete" or "auto". If set to "off" the /etc/localtime file in the container
		is left as it is included in the image, and neither modified nor bind mounted over. If set to "copy" the /etc/localtime file of the host is copied into the container. Similarly, if "bind" is used, the file is bind mounted from the host into the container. If
		set to "symlink", a symlink is created pointing from /etc/localtime in the container to the timezone file in the container that matches the timezone setting on the host. If set to "delete", the file in the container is deleted, should it exist. If set to
		"auto" and the /etc/localtime file of the host is a symlink, then "symlink" mode is used, and "copy" otherwise, except if the image is read-only in which case "bind" is used instead. Defaults to "auto".

		Added in version 239.

After this commit, we need to recreate all build environment to clean up
pollution already existed.

resolve #250

Signed-off-by: Celeste Liu <CoelacanthusHex@gmail.com>

Since commit 1d433f60 ("feat(db): confirm list of all packages that will be removed") packages need confirmation by default when being dropped from the db. If we make it to the DB drop phase the package already is pushed to the AUR, so it is safe to remove from the database and not removing it would create a somewhat broken state, so we assume that the package should unconditionally be removed from the db.

Component: pkgctl aur drop-from-repo
Signed-off-by: Christian Heusel <christian@heusel.eu>

Orhun Parmaksız authored 6 months ago and Christian Heusel committed 6 months ago

PAT settings are now under `user_settings` instead of `profile`

Component: pkgctl auth login

Jaroslav Lichtblau authored 6 months ago and Christian Heusel committed 6 months ago

Without the "--release" flag the example command fails with
==> ERROR: cannot use --message without --release.

Component: pkgctl build

It seems like nvchecker emits two log entries for errors:

    $ nvchecker --logger json -c .nvchecker.toml
    {"logger_name": "nvchecker.core", "name": "curl", "event": "token not given but it is required", "level": "error"}
    {"logger_name": "nvchecker.core", "name": "curl", "error": "token not given but it is required", "event": "no-result", "level": "error"}

This leads to a double error message as described in the related issue,
which we fix by narrowing the selector to filter for the error entry.

Fixes #235

Component: pkgctl version check
Signed-off-by: Christian Heusel <christian@heusel.eu>

If the chroot was created in a way where it resides on a BTRFS file
system but "$copydir/root" is not a snapshot an error like the following
would be emitted:

  $ makechrootpkg -r ~/chroot
  ==> Synchronizing chroot copy [/home/chris/chroot/root] -> [chris]...ERROR: Not a Btrfs subvolume: Invalid argument
  ==> ERROR: Unable to create subvolume /home/chris/chroot/chris

Fix this by adding an additional check, which detects if the folder is
actually the root of a BTRFS snapshot before attempting to clone it.

Related to !259



Signed-off-by: Christian Heusel <christian@heusel.eu>

Jaroslav Lichtblau authored 9 months ago and Christian Heusel committed 9 months ago

the '--pkgver' argument is not space-separated but instead specified
with an equals sign.

Component: pkgctl build

Replace `sdiff` with `diff` (also from `diffutils`) with the following
parameters:

- `--side-by-side` for the `sdiff`-like output
- `--suppress-common-lines` for the `sdiff -s` behavior
- `--width="$COLUMNS"` to use the full terminal width (long lines are
  still truncated but it is definitely better than the default width of
  130 chars)
- `--color=auto` just because 

Component: pkgctl repo configure

This avoids the unwanted removal of the folder if someone has already
pre-defined the variable.

Fixes #219

Suggested-by: Levente Polyak <anthraxx@archlinux.org>
Signed-off-by: Christian Heusel <christian@heusel.eu>

Signed-off-by: Christian Heusel <christian@heusel.eu>

David Runge authored 9 months ago and Levente Polyak committed 9 months ago

Since version 2.15.1 nvchecker emits a warning for version strings that
it consideres invalid (e.g. in the case of PyPI).
These warning messages get in the way (the first version emitted via a
warning is used as version to compare against) of retrieving the latest
version of an upstream and therefore we ignore them.

Component: pkgctl version check
Signed-off-by: David Runge <dvzrv@archlinux.org>

The expected output for these tests has now changed since git smash
(which we test against) has changed their version scheme.

Component: pkgctl version setup
Signed-off-by: Christian Heusel <christian@heusel.eu>

Systemd 256 introduces functionality which colors the terminal
background on systemd-nspawn invocations which makes the pkgctl output
look weird.

Disable this bevaviour for pkgctl, so it stays active for arch-nspawn
(for now).

Component: pkgctl
Signed-off-by: Christian Heusel <christian@heusel.eu>

Maharshi Basu authored 10 months ago and Christian Heusel committed 10 months ago

Changes made:
- add description in doc/man/pkgctl.1.asciidoc
- add description in doc/man/pkgctl-db(1).asciidoc
- add manpage for pkgctl-db(1) in doc/man/pkgctl-db(1).asciidoc

Fixes #231

Signed-off-by: Maharshi Basu <basumaharshi10@gmail.com>

lilydjwg lilydjwg authored 2 years ago and Christian Heusel committed 10 months ago

A .cache can be bind mounted into the container to save the caches for
reuse, but sometimes we want to exclude specific caches (because they
don't work well, e.g. bazel's).

Component: makechrootpkg

Frédéric Pierret authored 1 year ago and Christian Heusel committed 10 months ago

Co-authored-by: Christian Heusel <christian@heusel.eu>

Jan Alexander Steffens (heftig) authored 10 months ago and Christian Heusel committed 10 months ago

Git 2.45.1 expanded its security checks to deny cloning even local repos
that are owned by another user. Previously, this just affected network
filesystems.

On our buildserver, this prevents makepkg from cloning repos from our
shared srcdest into the srcdir, if these repos were created by another
packager.

To disable this check, set `safe.directory` to `*`. This looks like a
glob, but is really just a special value. The only other option would be
to add each Git repository in srcdest to the configuration.

Component: makechrootpkg

This fixes issues with packages containing plus signs, that need to be
escaped in toml as well as the extended grep regex.

Component: pkgctl version check

Jakub Klinkovský authored 1 year ago and Christian Heusel committed 11 months ago

Since logs from offloaded builds are collected in a temporary directory
on the remote server, it is rather difficult/error-prone to get to them,
because the path changes in each rebuild. Fetching logs from the server
into $LOGDEST makes it easier to investigate them and also brings the
behavior of offload-build closer to archbuild.

Log files are always downloaded, even for failed builds.

Component: offload-build
Signed-off-by: Jakub Klinkovský <lahwaacz@archlinux.org>
Co-authored-by: Levente Polyak <anthraxx@archlinux.org>

So far the commands would stop execution if one of the target
directories did not contain a PKGBUILD instead of just reporting failure
for that directory. Fix this by replacing the 'die' calls with setting
the error for the spinner facility.

Component: pkgctl version check
Component: pkgctl version upgrade
Signed-off-by: Christian Heusel <christian@heusel.eu>

Robin Candau authored 11 months ago and Levente Polyak committed 11 months ago

This introduces the `pkgctl repo clean` command which removes every
untracked files from local package repositories (via `git clean`).

The usage is as simple as `pkgctl repo clean [OPTION] [PATH]` (where
"[PATH]" can be equal to a wildcard "*").

Component: pkgctl repo clean

Felix Yan authored 1 year ago and Christian Heusel committed 11 months ago

pacman 6.1 added support for CacheServer which allows us to download
packages faster and reduce the burden on tier0 server.

They won't be used to fetch databases so it should be pretty safe in the
repository consistency context.

Component: arch-nspawn

Return an error and abort operation when an invalid target repo is
passed.

Component: pkgctl db remove
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Sometimes it isn't obvious which set of packages are removed from a
split package when the pkgbase matches also a subset of a pkgbase. This
can happen for example with bootstrapping packages, when the intention
is to just remove a partial part of the bootstrap pkgbase.

To make the intention more explicit, list all to be removed packages and
await for confirmation.

Component: pkgctl db remove
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

By default passing a pkgbase removes all split packages, debug packages
as well as entries from the state repo for all existing architectures.

When using the `--partial` option it may most likely lead to undesired
effects by leaving debug packages behind as well as dangling entries in
the state repository. However, for specific use cases its required to
get rid of old split package parts.

Fixes #218

Component: pkgctl db remove
Signed-off-by: Levente Polyak <anthraxx@archlinux.org>

Stefan 　 authored 2 years ago and Levente Polyak committed 11 months ago

systemd-nspawn sets up a pts in the container by default which build
scripts might use to ask for interactive input.  This is only okay if
the host side is also a TTY.  The 'autopipe' mode deals with this
situation appropriately.

Christian Heusel authored 11 months ago and Levente Polyak committed 11 months ago

The architecture definition of the variable was using invalid bash
syntax and was previously unused:

    $ _binary_arch=${DEVTOOLS_VALID_ARCHES[*]:0:-1}
    bash: -1: substring expression < 0

We therefore fix the definition of the variable and use it for the
autocompletion of the offload-build command.

Furthermore fix wrongly named architecture variables that have been
missed from previous refactoring.

Fixes #222

Component: completion
Fixes: f961e2e9 ("completion: implemented structured declarative bash completions")
Fixes: 4173e0a ("chore: refactor variable names in valid-{tags,repos}.sh")
Signed-off-by: Christian Heusel <christian@heusel.eu>
Co-authored-by: Levente Polyak <anthraxx@archlinux.org>

Use a central location in common.sh to define the default rsync options.
Switch to use batched uploads by targeting a partial directory which is
not taken into account by db update. Instead, once all packages that
belong to a split package group are uploaded into the .partial
directory, all artifacts are moved in one batch into the staging
directory of the repo server. This reduced the window of opportunity for
a partial release significantly to a tiny window.

Component: pkgctl release

This allows us to have more control over the output and status logs.
Using this method we are able to avoid cluttering the version upgrade
subcommand while downloading sources for updating the checksums.

Having this internally will also allow us in the future to have smart
checksums updating by only trying to change the checksums of sources
that have actually changed, for example when adjusting a patch file we
should avoid trying to overwrite the archive checksums unintentionally.

Component: pkgctl version upgrade