repod issueshttps://gitlab.archlinux.org/archlinux/repod/-/issues2023-07-17T19:54:04Zhttps://gitlab.archlinux.org/archlinux/repod/-/issues/155Evaluate Git/ PGP integration for signing and signature checking2023-07-17T19:54:04ZDavid RungeEvaluate Git/ PGP integration for signing and signature checkingFor checking signatures against a list of allowed PGP key IDs (e.g. from pacman-key) and the local signing of repository sync databases, we need integration with tooling that can deal with PGP.
When looking at other avenues for signatur...For checking signatures against a list of allowed PGP key IDs (e.g. from pacman-key) and the local signing of repository sync databases, we need integration with tooling that can deal with PGP.
When looking at other avenues for signature validation in git repositories (e.g. #151 ), one could also look at openssh based signature integration (this would not be something that pacman can handle for repository sync databases though, so it falls flat in the use-case of repository sync database signing - #32).
## Requirements
| repod mode | git | smartcard (db signing) | git tag validation | WOT integration (git tag/ db validation) |
| --- | --- | --- | --- | --- |
| user | Yes | Yes | Yes | Yes [2] |
| system | Yes | Yes [1] | Yes | Yes [2] |
[1] If we want to allow a dedicated machine to be able to sign sync databases itself, it would probably be good to also have smartcard support here.
[2] Depending on support in keyringctl (see https://gitlab.archlinux.org/archlinux/keyringctl/-/issues/3), we can also evaulate the use of a flat file here (i.e. a keyring file with all "allowed keys") instead of hooking into an existing web-of-trust.
## PGP providers
### gpgme
* [pypi.org package](https://pypi.org/project/gpg/) is extremely out-of-date (last release 2018, 12(!) releases behind current gpgme)
* package does not build in venv (as the sdist is so outdated and uses some super ancient integration), only system integration can be used until upstream fixes this
* [patches have been provided to upstream](https://lists.gnupg.org/pipermail/gnupg-devel/2022-November/035158.html) for fixing sdist tarball generation and PEP517 based builds (so far no response)
### sequoia
* [no sdist tarballs or wheels on pypi.org](https://gitlab.com/sequoia-pgp/sequoia-ffi/-/issues/23), only system integration can be used until upstream fixes this
* unclear feature set
* upstream proposes using pyo3 to build something custom or [johnnycanencrypt](https://github.com/kushaldas/johnnycanencrypt) instead
### johnnycanencrypt
* provides sdist tarballs and wheels on [pypi.org](https://pypi.org/project/johnnycanencrypt/)
* written in rust, based on sequoia
* basic, does not integrate with keyring, requires a local "keystore" directory
* seems to have smartcard integration
* does not yet use sequoia's internal keystore implementation/specification
### python-gnupg
* has sdist tarballs and wheels on pypi.org
* deveveloped at https://github.com/vsajip/python-gnupg
* shells out to gpg :cry:
### pgpy
* has sdist tarballs and wheels on pypi.org
* developed at: https://github.com/SecurityInnovation/PGPy
* PGP implementation in Python
* feature set needs evaluation
## Git providers
### dulwich
* somehow [integrates with gpg](https://github.com/jelmer/dulwich/blob/82476465134520952ccb2f3b56a10bf431629725/setup.cfg#L37) for signatures (unclear if they ever test this, given that everything is terribly out-of-date and broken)
### pygit2
* pygit2 can get to signature strings, so PGP integration is likely open/ can be implemented whichever way
### gitpython
* long track record of botched releases (faith in upstream quite low also given that the project seems to be in maintenance-only mode over developing gitoxide)
* relies on system git and gnupg executablesworkflowshttps://gitlab.archlinux.org/archlinux/repod/-/issues/113Transactional Tasks and Checks for atomic actions2022-09-14T21:43:39ZDavid RungeTransactional Tasks and Checks for atomic actionsTo be able to have atomic actions (e.g. add a set of packages to a repository), which can be reverted if it fails halfway, all of its steps (aka. Tasks) must be revertible.
We need a framework in which repod is able to work on its actio...To be able to have atomic actions (e.g. add a set of packages to a repository), which can be reverted if it fails halfway, all of its steps (aka. Tasks) must be revertible.
We need a framework in which repod is able to work on its actions, while allowing us to retain a before and after state, to easily revert faulty Tasks.
In this context Checks are supposed to help Tasks to evaluate whether they should be run at all (pre checks) or whether they were successful (post checks).0.3.0 - workflows and checksDavid RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/159Create a parser for .SRCINFO files2022-12-10T08:36:12ZDavid RungeCreate a parser for .SRCINFO filesTo enable us to validate data of an upstream package source repository, we require a parser for `.SRCINFO` files.To enable us to validate data of an upstream package source repository, we require a parser for `.SRCINFO` files.0.3.0 - workflows and checksDavid RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/39Track URL of source repository in management repository2022-11-01T13:06:50ZDavid RungeTrack URL of source repository in management repositoryTo be able to track the repository with the package sources that a particular pkgbase has been built from, we need to be able to track the URL of the source repository per pkgbase in the management repository.
This field is not written ...To be able to track the repository with the package sources that a particular pkgbase has been built from, we need to be able to track the URL of the source repository per pkgbase in the management repository.
This field is not written to the sync db (e.g. json2db).
The field should be able to be guarded with a configurable URL match (e.g. https://gitlab.archlinux.org/) to ensure, that no packages from outside sources end up in a target binary repository).0.3.0 - workflows and checksDavid RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/149Define repository groups2022-12-03T18:47:49ZDavid RungeDefine repository groupsIn relation to #145 we require the notion of "repositories belonging together".
This allows us to check whether e.g. packages must be unique in a set of repositories (e.g. core, extra, community must not contain the same pkgbase/pkgnames).In relation to #145 we require the notion of "repositories belonging together".
This allows us to check whether e.g. packages must be unique in a set of repositories (e.g. core, extra, community must not contain the same pkgbase/pkgnames).0.3.0 - workflows and checksDavid RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/96Implement archiving functionality2022-11-15T17:49:01ZDavid RungeImplement archiving functionalityAt the moment the Arch Linux setup relies on https://gitlab.archlinux.org/archlinux/archivetools to archive packages added to a repository below a given directory structure.
As repod is handling the movement of packages from an input al...At the moment the Arch Linux setup relies on https://gitlab.archlinux.org/archlinux/archivetools to archive packages added to a repository below a given directory structure.
As repod is handling the movement of packages from an input already, we can handle the (configurable) archiving of packages as well.0.3.0 - workflows and checksDavid RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/14Add handling of file actions2022-08-14T17:22:59ZDavid RungeAdd handling of file actionsFile actions are atomic actions used to e.g.
- [x] add new package files and their signatures to the package pool
- [x] Handle symlinking of package files and their signatures from a package pool to a target repository directory
- [x] ...File actions are atomic actions used to e.g.
- [x] add new package files and their signatures to the package pool
- [x] Handle symlinking of package files and their signatures from a package pool to a target repository directory
- [x] add
- [ ] ~~move~~
- [x] remove0.2.0David RungeAnton Hvornumtorxed@archlinux.orgDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/126Evaluate debug repository setup2022-10-31T21:39:28ZDavid RungeEvaluate debug repository setupCurrently debug repository locations are per repository instance. Each repository has its staging and testing locations as attributes and debug locations work the same way (but there is only one debug location per repository instance).
...Currently debug repository locations are per repository instance. Each repository has its staging and testing locations as attributes and debug locations work the same way (but there is only one debug location per repository instance).
This setup is problematic, as it means that all debug packages (those from stable, staging and testing locations) end up in them and pacman would install the latest version of a given debug package.0.3.0 - workflows and checksDavid RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/120Check PKGBUILD exists and matches sha256sum2023-07-17T19:54:02ZDavid RungeCheck PKGBUILD exists and matches sha256sumWhen adding or updating packages we need to ensure that the package's PKGBUILD sha256sum (in its .BUILDINFO file) matches the one from the upstream repository at the given tag.When adding or updating packages we need to ensure that the package's PKGBUILD sha256sum (in its .BUILDINFO file) matches the one from the upstream repository at the given tag.workflowshttps://gitlab.archlinux.org/archlinux/repod/-/issues/118Verify package version is consistent with source repository tag2023-07-17T19:54:02ZDavid RungeVerify package version is consistent with source repository tagWhen consuming packages (and when the feature for validating the upstream source repository is activated) we want make sure that the package's `$epoch:$pkgver-$pkgrel` is consistent with its upstream tag.When consuming packages (and when the feature for validating the upstream source repository is activated) we want make sure that the package's `$epoch:$pkgver-$pkgrel` is consistent with its upstream tag.workflowshttps://gitlab.archlinux.org/archlinux/repod/-/issues/106management repo symlinks from pkgname.json to pkgbase.json for easy reverse s...2022-09-30T08:59:46ZLevente Polyakanthraxx@archlinux.orgmanagement repo symlinks from pkgname.json to pkgbase.json for easy reverse searchThere may be different use cases where we want to quickly and directly search for the current `pkgbase` of a `pkgname` inside the management repo without the need to iterate over all `pkgbase.json` files to check for patching `pkgname`. ...There may be different use cases where we want to quickly and directly search for the current `pkgbase` of a `pkgname` inside the management repo without the need to iterate over all `pkgbase.json` files to check for patching `pkgname`. This could be the case for checks if a to be released package `pkgname` currently belongs to a different `pkgbase` and hence avoid edge case problems.
To aid such use cases and have an easier reverse lookup, we can create symlinks from `pkgname.json` to `pkgbase.json`.
- [ ] Check that we need to clean up `pkgname.json` symlinks in case a split package part is removed from a `pkgbase` to avoid having dangling symlinks pointing to a `pkgbase` that does not provide that `pkgname` anymore.0.3.0 - workflows and checksDavid RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/65Create unified CLI for file actions2022-06-30T22:00:05ZDavid RungeCreate unified CLI for file actionsFor common file operations the existing scripts should be merged into a unified one (e.g. `repod-file`).
It should provide:
* db2json (functionality)
* json2db (functionality)
* introspection into package metadata by returning all or se...For common file operations the existing scripts should be merged into a unified one (e.g. `repod-file`).
It should provide:
* db2json (functionality)
* json2db (functionality)
* introspection into package metadata by returning all or selected components of a consumed package (e.g. Package, .MTREE, .BUILDINFO, .PKGINFO representation in JSON)
* transformation of package data into management repo representation
* create model for package consumption and representation !57
* add creation from Package to OutputPackageBase0.1.0David RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/53Add schema and parser for .PKGINFO2022-06-15T20:03:30ZDavid RungeAdd schema and parser for .PKGINFOWe need a JSON schema and a parser for the .PKGINFO format (when consuming packages).We need a JSON schema and a parser for the .PKGINFO format (when consuming packages).0.1.0https://gitlab.archlinux.org/archlinux/repod/-/issues/52Add schema and parser for .BUILDINFO2022-06-15T20:03:29ZDavid RungeAdd schema and parser for .BUILDINFOWe need a JSON schema and parser for the .BUILDINFO format (when consuming packages).We need a JSON schema and parser for the .BUILDINFO format (when consuming packages).0.1.0David RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/51Add schema and parser for .MTREE2022-06-15T20:03:29ZDavid RungeAdd schema and parser for .MTREEWe need a schema and a parser for the .MTREE file format (when consuming packages).We need a schema and a parser for the .MTREE file format (when consuming packages).0.1.0https://gitlab.archlinux.org/archlinux/repod/-/issues/49Establish workflow scenarios2022-10-22T16:27:12ZDavid RungeEstablish workflow scenariosThis ticket is supposed to document the already established workflows that have been elaborated on. Unfortunately the tooling is not ready for them yet, which makes their place in user facing documentation a problem. Hence I'm moving the...This ticket is supposed to document the already established workflows that have been elaborated on. Unfortunately the tooling is not ready for them yet, which makes their place in user facing documentation a problem. Hence I'm moving them here, so they can be worked on again once the tooling has gained more features to even support these workflows:
Packager Workflows
__________________
In this section the different workflows are listed, to give an overview, what
they would mean in the different git repository layouts.
Adding a Package
================
**Developer machine/ build server**:
1. Create repository
1. Update, build *package* and commit changes in *package*'s `PKGBUILD`_
1. Tag release
.. note::
Force pushing tags is disallowed.
1. Sign *package*
1. Upload built *package* and signature
1. Call application on repository/ package server to add *package*
**Repository server/ package server**:
.. important::
The following steps need to be atomic (reversible).
1. Verify user permissions https://gitlab.archlinux.org/archlinux/repod/-/issues/26
1. Verify that an upstream URL is provided for any new `pkgbase` https://gitlab.archlinux.org/archlinux/repod/-/issues/39
1. Lock repository sync databases and management repository https://gitlab.archlinux.org/archlinux/repod/-/issues/117
1. ~~Inspect built files of *package*~~
1. Lock tags (by storing them in *package*'s bare repository) @heftig @alad @maximbaz (please clarify what this means)
1. ~~Modify management repo to reflect changes~~
1. Verify *package* file versioning and tag is consistent https://gitlab.archlinux.org/archlinux/repod/-/issues/118
1. ~~Copy built *package* and signature to pool and create symlink to them in
target repository~~
1. ~~Add *package* to the package database~~
1. Unlock repository sync databases and management repository https://gitlab.archlinux.org/archlinux/repod/-/issues/117
Updating a Package
==================
All steps, but the first, of **Developer machine/ build server** in `Adding a
Package`_ apply.
All steps of **Repository server/ package server** in `Adding a Package`_ apply.
1. Verify user permissions https://gitlab.archlinux.org/archlinux/repod/-/issues/26
1. Lock repository sync databases and management repository https://gitlab.archlinux.org/archlinux/repod/-/issues/117
1. ~~Inspect built files of *package*~~
1. Lock tags (by storing them in *package*'s bare repository) @heftig @alad @maximbaz (please clarify what this means)
1. ~~Modify management repo to reflect changes~~
1. Verify *package* file versioning and tag is consistent https://gitlab.archlinux.org/archlinux/repod/-/issues/118
1. ~~Copy built *package* and signature to pool and create symlink to them in
target repository~~
1. ~~Add *package* to the package database~~
1. Remove symlinks and files of previous versions of package(s) https://gitlab.archlinux.org/archlinux/repod/-/issues/107
1. Unlock repository sync databases and management repository https://gitlab.archlinux.org/archlinux/repod/-/issues/117
Removing a Package
==================
**Developer machine/ build server**:
1. Call application on repository/ package server to remove *package*
**Repository server/ package server**:
.. important::
The following steps need to be transactional (reversible).
.. note::
The remove command should be able to remove stale packages (e.g. leftover
packages, when removing a member of a split package)
1. Verify user permissions https://gitlab.archlinux.org/archlinux/repod/-/issues/26
1. Lock repository sync databases and management repository https://gitlab.archlinux.org/archlinux/repod/-/issues/117
1. ~~Modify monorepo to reflect changes~~
1. ~~Remove *package* from the package database~~
1. Remove built *package* and signature from pool and remove symlink to them in
target repository https://gitlab.archlinux.org/archlinux/repod/-/issues/107
1. Unlock package database and monorepo https://gitlab.archlinux.org/archlinux/repod/-/issues/117
Moving a Package
================
**Developer machine/ build server**:
1. Call application on repository/ package server to move *package*
**Repository server/ package server**:
.. important::
The following steps need to be transactional (reversible).
1. Verify user permissions https://gitlab.archlinux.org/archlinux/repod/-/issues/26
1. Lock repository sync databases and management repository https://gitlab.archlinux.org/archlinux/repod/-/issues/117
1. ~~Modify management repo(s) to reflect changes~~
1. Remove *package* from the source package database (write to tmp location) https://gitlab.archlinux.org/archlinux/repod/-/issues/105
1. Add *package* to the destination package database (write to tmp location) https://gitlab.archlinux.org/archlinux/repod/-/issues/105
1. Remove symlinks to package and signature files from source repository https://gitlab.archlinux.org/archlinux/repod/-/issues/107
1. ~~Add symlinks to package and signature them to the target repository~~
1. Unlock source and target package databases and management repo https://gitlab.archlinux.org/archlinux/repod/-/issues/117
.. _PKGBUILD: https://man.archlinux.org/man/pacman/PKGBUILD.5.en0.3.0 - workflows and checkshttps://gitlab.archlinux.org/archlinux/repod/-/issues/25Define relationship with source repositories for deriving a package's repository2022-11-01T13:06:49ZDavid RungeDefine relationship with source repositories for deriving a package's repositoryIn a setup where the sources for each pkgbase are kept in a git repository each, we need to figure out, what the relationship between this application (or dbscripts for that matter) will be.
Having repository based grouping for pkgbases...In a setup where the sources for each pkgbase are kept in a git repository each, we need to figure out, what the relationship between this application (or dbscripts for that matter) will be.
Having repository based grouping for pkgbases (e.g. `/core/pacman`) in a VCS forge such as gitlab can have the upside for us to derive the target location of a given set of packages in the binary package repository.
When thinking of a potential bootstrap scenario, where we initialize a given package source repository in a group (e.g. `/foo/bar`) we can upon trying to push packages - built from that particular pkgbase - towards the binary repository management software (such as arch-repo-management or dbscripts) derive the target from its remote link (e.g. `https://gitlab.archlinux.org/packages/foo/bar` -> repository => foo, pkgbase => bar).
This type of scenario would allow easy detection of target location (which is paired with an ACL of the application that checks whether the given user is allowed to do this operation). However, at this point in time the workflow around this is yet to be defined and needs discussion.David RungeMorten Linderudfoxboron@archlinux.orgLevente Polyakanthraxx@archlinux.orgDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/13Add configuration handling2022-06-15T20:03:43ZDavid RungeAdd configuration handlingThe eventual application/ set of applications require validated configuration, that e.g. allows defining locations for git repositories, location for repository sync dbs and location for per-repository binary package locations.The eventual application/ set of applications require validated configuration, that e.g. allows defining locations for git repositories, location for repository sync dbs and location for per-repository binary package locations.0.1.0David RungeDavid Rungehttps://gitlab.archlinux.org/archlinux/repod/-/issues/12Add handling for git repository per repository sync db2022-08-07T15:26:25ZDavid RungeAdd handling for git repository per repository sync dbThe handling (i.e. setup, adding commits, reverting commits) of git repositories per repository sync db needs to be added.The handling (i.e. setup, adding commits, reverting commits) of git repositories per repository sync db needs to be added.0.4.0 - git backend for management repohttps://gitlab.archlinux.org/archlinux/repod/-/issues/158Add configurable check to dissallow unsafe checksum algorithms in source info...2023-07-17T19:54:04ZDavid RungeAdd configurable check to dissallow unsafe checksum algorithms in source informationWe want to ensure that we can disallow unsafe checksum algorithms (e.g. SHA-1, MD5) in the package sources.
This can be ensured by checking out the package source repo and parsing its .SRCINFO file.We want to ensure that we can disallow unsafe checksum algorithms (e.g. SHA-1, MD5) in the package sources.
This can be ensured by checking out the package source repo and parsing its .SRCINFO file.workflows