Verified Commit 270fda4e authored by David Runge's avatar David Runge
Browse files

Merge branch 'issues/12'

* issues/12:
  pyproject.toml: Update project version to 0.2.0
  README: Extend with information on arch-release-sync
  example.toml: Add synchronization related documentation
  Add script entrypoint for arch-release-sync
  cli: Add entrypoint for arch-release-sync
  files: Add ProjectFiles class to handle synchronization
  gitlab: More explicit print when downloading build artifacts
  gitlab: Implement download of promotion artifact
  argparse: Add parser for arch-release-sync
  gitlab: Change signature of constructor
  gitlab: More generically retrieve types of releases
  config: Make PRIVATE_TOKEN optional
  config: Add integration to configure synchronization
parents 086fd12e aa865808
Pipeline #10967 passed with stages
in 1 minute and 33 seconds
......@@ -2,8 +2,8 @@
arch-release-promotion
======================
This project allows for promoting existing releases of a project in Arch
Linux's Gitlab instance.
This project allows for promotion and synchronization of existing releases of a
project in Arch Linux's Gitlab instance.
Releases of a project (e.g. ``project``) may consist of several release types
(e.g. ``image_a`` and ``image_b``), which are addressed separately.
......@@ -13,21 +13,30 @@ artifacts (optional), a torrent file (optional) and a JSON payload which can be
used by `archweb <https://github.com/archlinux/archweb>`_ to display
information about each release type.
Synchronization with a local directory can be achieved for a configurable
maximum amount of release versions (each consisting of their respective
configured release types) of a project.
Requirements
============
The arch-release-promotion tool is Python based. All requirements are specified
in its `pyproject.toml <pyproject.toml>`_.
Arch-release-promotion is Python based. All language specific requirements are
specified in its `pyproject.toml <pyproject.toml>`_.
Additionally, ``arch-release-promotion`` requires `gnupg <https://gnupg.org/>`_
to handle detached PGP signatures.
Use
===
After installation, refer to the output of ``arch-release-promotion -h``.
After installation, refer to the output of ``arch-release-promotion -h`` and
``arch-release-sync -h``.
Configuration
=============
The command-line tool ``arch-release-promotion`` makes use of two sources of configuration:
The command-line tools ``arch-release-promotion`` and ``arch-release-sync``
make use of two sources of configuration:
* `makepkg.conf <https://man.archlinux.org/man/makepkg.conf.5>`_ is read from
any of its locations in the same priority as `makepkg
......@@ -38,11 +47,11 @@ The command-line tool ``arch-release-promotion`` makes use of two sources of con
* ``PACKAGER`` is recognized for establishing who is doing the signature and
is important for `WKD
<https://wiki.archlinux.org/title/GnuPG#Web_Key_Directory>`_ lookup
* ``MIRRORLIST_URL`` (not used by makepkg) is used during the generation of torrent files to add
webseeds (defaults to
* ``MIRRORLIST_URL`` (not used by makepkg) is used during the generation of
torrent files to add webseeds (defaults to
``"https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https"``)
* ``GITLAB_URL`` (not used by makepkg) is used to connect to a GitLab instance to select, download
and promote releases of a project (defaults to
* ``GITLAB_URL`` (not used by makepkg) is used to connect to a GitLab
instance to select, download and promote releases of a project (defaults to
``"https://gitlab.archlinux.org"``)
* ``PRIVATE_TOKEN`` (not used by makepkg) is used to authenticate against the
GitLab instance. The `personal access token
......@@ -239,7 +248,54 @@ each release type in the release.
release.
* ``version``: The version of the release type.
Synchronization
===============
The synchronization of releases works by retrieving the list of promoted
releases of the project from the remote. For each promoted release version, the
promotion artifact is downloaded and used to establish whether all of the
configured release types are fully synchronized.
Location and cleanup
--------------------
All release types for each release version are synchronized to a local
directory. The directory and and the maximum amount of synchronized release
versions are configurable globally or per project.
.. code::
sync_dir
├── example_a
│   ├── example_a-0.1.0
│   │   ├── foo.txt
│   │   └── foo.txt.sig
│   ├── example_a-0.1.0.json
│   ├── example_a-0.1.0.torrent
│   └── latest -> example_a-0.1.0
└── example_b
├── example_b-0.1.0
│   ├── bar.txt
│   └── bar.txt.sig
├── example_b-0.1.0.json
├── example_b-0.1.0.torrent
└── latest -> example_b-0.1.0
A ``latest`` symlink is created to point at the currently latest version of
each release type.
Any files and directories that are not owned by versions of release types of
the currently synchronized release versions are removed from the
synchronization directory.
If changes are introduced to files in the target directory (due to a
synchronization action), it is possible to write a Unix timestamp to a file
that is configurable globally or per project (the directory in which the file
resides in has to exist).
License
=======
Arch-release-promotion is licensed under the terms of the **GPL-3.0-or-later** (see `LICENSE <LICENSE>`_).
Arch-release-promotion is licensed under the terms of the **GPL-3.0-or-later**
(see `LICENSE <LICENSE>`_).
......@@ -67,6 +67,37 @@ class ArgParseFactory:
return instance.parser
@classmethod
def synchronize(self) -> argparse.ArgumentParser:
"""A class method to create an ArgumentParser for synchronization
Returns
-------
argparse.ArgumentParser
An ArgumentParser instance specific for synchronization
"""
instance = self(
prog="arch-release-sync",
description="Synchronize promoted releases of a project with a local directory",
)
instance.parser.add_argument(
"-p",
"--project",
type=self.non_zero_string,
help=(
"the project to synchronize from a remote (e.g. 'group/project'). "
f"By default {instance.parser.prog} attempts to synchronize releases for "
"all projects specified in its config"
),
)
if instance.parser.parse_args().version:
print(f"{instance.parser.prog} {metadata.version('arch_release_promotion')}")
exit(0)
return instance.parser
@classmethod
def non_zero_string(self, input_: str) -> str:
"""Validate a string to be of non_zero length
......
......@@ -117,3 +117,27 @@ def main() -> None:
else:
for project in config.Projects().projects:
promote_project_release(project=project)
def arch_release_sync() -> None:
"""Synchronize releases
If the argument parser contains a specific project to synchronize, only synchronize that, else all configured
projects.
"""
args = argparse.ArgParseFactory.synchronize().parse_args()
projects = config.Projects()
settings = config.Settings()
if args.project:
files.ProjectFiles.sync(
project_config=projects.get_project(name=args.project),
settings=settings,
)
else:
for project in projects.projects:
files.ProjectFiles.sync(
project_config=project,
settings=settings,
)
from collections import Counter
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
import toml
from dotenv import dotenv_values
from email_validator import EmailNotValidError, validate_email
from pydantic import BaseModel, BaseSettings, Extra, validator
from pydantic import BaseModel, BaseSettings, Extra, root_validator, validator
from pydantic.env_settings import SettingsSourceCallable
from xdg.BaseDirectory import xdg_config_home
......@@ -19,6 +20,9 @@ PROJECTS_CONFIGS = [
Path(f"{xdg_config_home}/arch-release-promotion/projects.toml"),
]
PROJECTS_SYNC_DIR = Path("/var/lib/arch-release-promotion/")
PROJECTS_SYNC_BACKLOG = 3
class ReleaseConfig(BaseModel):
"""A pydantic model describing the configuration of a project's release
......@@ -50,6 +54,31 @@ class ReleaseConfig(BaseModel):
create_torrent: bool = False
class SyncConfig(BaseModel):
"""A pydantic model describing configuration for synchronization
Attributes
----------
backlog: int
The backlog of releases to retain at a maximum (defaults to PROJECTS_SYNC_BACKLOG when a SysConfig instance is
used in a Projects instance or a ProjectConfig instance).
directory: Path
A directory into which to sync project release types and their respective releases (defaults to
PROJECTS_SYNC_DIR when a SysConfig instance is used in a Projects instance or a ProjectConfig instance).
last_updated_file: Optional[Path]
The optional path to a file, that is used to write a timestamp to, if the synchronization of a project leads to
the changing of data on disk (defaults to None).
temp_in_sync_dir: bool
A bool specifying whether to download temporary data to temporary directories below sync_dir (defaults to True).
If False is specified the temporary data is downloaded to the respective user's temporary directory.
"""
backlog: Optional[int]
directory: Optional[Path]
last_updated_file: Optional[Path] = None
temp_in_sync_dir: bool = True
class ProjectConfig(BaseModel):
"""A pydantic model describing the configuration of a project
......@@ -65,6 +94,8 @@ class ProjectConfig(BaseModel):
The project's metrics file
releases: List[ReleaseConfig]
The project's list of releases
sync_config: Optional[SyncConfig]
An optional SyncConfig instance, which is used to override any global defaults.
"""
name: str
......@@ -72,6 +103,7 @@ class ProjectConfig(BaseModel):
output_dir: Path
metrics_file: Path
releases: List[ReleaseConfig]
sync_config: Optional[SyncConfig]
class Projects(BaseSettings):
......@@ -81,9 +113,13 @@ class Projects(BaseSettings):
----------
projects: List[ProjectConfig]
A list of project configurations
sync_config: Optional[SyncConfig]
An optional SyncConfig instance, which is used to override any implicit defaults and sets defaults for all
ProjectConfig instances in projects.
"""
projects: List[ProjectConfig]
sync_config: Optional[SyncConfig]
class Config:
......@@ -98,6 +134,97 @@ class Projects(BaseSettings):
) -> Tuple[SettingsSourceCallable, ...]:
return (read_projects_conf,)
@validator("projects")
def validate_project_releases_unique(cls, projects: List[ProjectConfig]) -> List[ProjectConfig]:
"""Validate the list of ProjectConfig instances to only contain uniquely named ReleaseConfig instances
Parameters
----------
projects: List[ProjectConfig]
A list of ProjectConfig instances
Raises
------
ValueError
If a non-unique name is encountered among any ReleaseConfig instance
Returns
-------
List[ProjectConfig]
The unmodified list of ProjectConfig instances
"""
release_types: List[ReleaseConfig] = []
for project in projects:
release_types += project.releases
names = [release_type.name for release_type in release_types]
if len(set(names)) < len(names):
duplicates = [name for name, count in Counter(names).items() if count > 1]
raise ValueError(
f"The following release type {'name' if len(duplicates) == 1 else 'names'} "
f"{'is' if len(duplicates) == 1 else 'are'} not unique: "
f"{duplicates[0] if len(duplicates) == 1 else duplicates}"
)
return projects
@root_validator
def validate_projects(cls, values: Dict[str, Any]) -> Dict[str, Any]:
"""Validate the list of ProjectConfig instances and override defaults
If a ProjectConfig does not specify a SysConfig, override it with the global SysConfig. If a global SysConfig
does not exist, override with an implicit default where directory defaults to PROJECTS_SYNC_DIR and backlog to
PROJECTS_SYNC_BACKLOG.
Parameters
----------
values: Dict[str, Any]
A dict with all values of the Projects instance
Returns
-------
values: Dict[str, Any]
The (potentially modified) dict with all values of the Projects instance
"""
default_sync_config = SyncConfig(
directory=PROJECTS_SYNC_DIR,
backlog=PROJECTS_SYNC_BACKLOG,
last_updated_file=None,
)
projects: List[ProjectConfig] = values.get("projects") # type: ignore
sync_config: Optional[SyncConfig] = values.get("sync_config")
# merge global sync_config with defaults
if sync_config:
global_sync_config = SyncConfig(
directory=sync_config.directory or default_sync_config.directory,
backlog=sync_config.backlog or default_sync_config.backlog,
last_updated_file=sync_config.last_updated_file or default_sync_config.last_updated_file,
temp_in_sync_dir=False if not sync_config.temp_in_sync_dir else default_sync_config.temp_in_sync_dir,
)
else:
global_sync_config = default_sync_config
for project in projects:
if not project.sync_config:
project.sync_config = global_sync_config
else:
project.sync_config = SyncConfig(
directory=project.sync_config.directory or global_sync_config.directory,
backlog=project.sync_config.backlog or global_sync_config.backlog,
last_updated_file=project.sync_config.last_updated_file or global_sync_config.last_updated_file,
temp_in_sync_dir=False
if not project.sync_config.temp_in_sync_dir
else global_sync_config.temp_in_sync_dir,
)
values["projects"] = projects
return values
def get_project(self, name: str) -> ProjectConfig:
"""Return a ProjectConfig by name
......@@ -156,17 +283,24 @@ class Settings(BaseSettings):
Attributes
----------
gpgkey: str
GITLAB_URL: str
A URL for a GitLab upstream (defaults to "https://gitlab.archlinux.org")
GPGKEY: str
The PGP key id to use for artifact signatures
packager: str
The packager name and mail address to use for artifact signatures
MIRRORLIST_URL: str
A URL to derive a mirrorlist from (defaults to
"https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https")
PACKAGER: str
The packager name and mail address (UID) to use for artifact signatures
PRIVATE_TOKEN: Optional[str]
An optional private token to use for authenticating against an upstream
"""
MIRRORLIST_URL: str = "https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https"
GITLAB_URL: str = "https://gitlab.archlinux.org"
GPGKEY: str
MIRRORLIST_URL: str = "https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https"
PACKAGER: str
PRIVATE_TOKEN: str
PRIVATE_TOKEN: Optional[str]
class Config:
......@@ -244,7 +378,7 @@ class Settings(BaseSettings):
return gpgkey
@validator("PRIVATE_TOKEN")
def validate_private_token(cls, private_token: str) -> str:
def validate_private_token(cls, private_token: Optional[str]) -> Optional[str]:
"""A validator for the PRIVATE_TOKEN attribute
Parameters
......@@ -263,6 +397,9 @@ class Settings(BaseSettings):
A gpgkey string in long-format
"""
if private_token is None:
return None
if len(private_token) < 20:
raise ValueError("The PRIVATE_TOKEN string has to represent a valid private token (20 chars).")
......
import shutil
import tempfile
import time
import zipfile
from pathlib import Path
from typing import List, Optional, Tuple
import orjson
from prometheus_client.parser import text_fd_to_metric_families
from pydantic import BaseModel
from arch_release_promotion.config import ProjectConfig, Settings
from arch_release_promotion.gitlab import Upstream
from arch_release_promotion.release import (
AmountMetric,
Release,
......@@ -176,6 +180,24 @@ def write_release_info_to_file(release: Release, path: Path) -> None:
)
def load_release_from_json_payload(path: Path) -> Release:
"""Read a JSON payload and return it as a Release instance
Parameters
----------
path: Path
The path to a file containing a JSON payload
Returns
-------
Release
A Release instance reflecting the data from the JSON payload
"""
with open(path, "r") as file:
return Release(**orjson.loads(file.read()))
def write_zip_file_to_parent_dir(path: Path, name: str = "promotion", format: str = "zip") -> None:
"""Create ZIP file of all contents in a directory and write it to the directory's parent
......@@ -277,3 +299,441 @@ def read_metrics_file(
)
]
return (amount_metrics, size_metrics, version_metrics)
def create_dir(path: Path) -> Path:
"""Create a directory
Parameters
----------
path: Path
The path for which to create a directory
Raises
------
RuntimeError
If the path exists but is not a directory
Returns
-------
Path
The path representing the existing, absolute directory
"""
path = path.resolve(strict=False)
if path.exists() and not path.is_dir():
raise RuntimeError(f"The provided path is not a directory: {path}")
else:
path.mkdir(parents=True, exist_ok=True)
return path
class ProjectFiles(BaseModel):
"""A pydantic model to operate on a project's files and releases
Attributes
----------
project_config: ProjectConfig
A ProjectConfig instance describing the project
upstream: Upstream
An Upstream instance used for queries to releases of a project
promoted_releases: List[str]
A list of version strings
"""
project_config: ProjectConfig
upstream: Upstream
promoted_releases: List[str]
class Config:
arbitrary_types_allowed = True
def __init__(self, project_config: ProjectConfig, settings: Settings) -> None:
"""A custom constructor to initialize an instance of ProjectFiles
The names of the configured maximum number of promoted releases is retrieved using an Upstream instance.
If the project's sync_dir does not exist, it will be created.
Parameters
----------
project_config: ProjectConfig
A ProjectConfig instance describing the project
settings: Settings
A Settings instance used to initialize an Upstream instance
"""
upstream = Upstream(
url=settings.GITLAB_URL,
private_token=None,
name=project_config.name,
)
promoted_releases = upstream.get_releases(
max_releases=project_config.sync_config.backlog, # type: ignore
promoted=True,
)
print(f"Synchronizing release versions for {project_config.name}: " f"{', '.join(promoted_releases)}")
create_dir(path=project_config.sync_config.directory) # type: ignore
super().__init__(
project_config=project_config,
upstream=upstream,
promoted_releases=promoted_releases,
)
@classmethod
def sync(self, project_config: ProjectConfig, settings: Settings) -> None:
"""A factory method to initialize an instance of ProjectFiles and synchronize all of its promoted_releases
Parameters
----------
project_config: ProjectConfig
A ProjectConfig instance describing the project
settings: Settings
A Settings instance used to initialize an Upstream instance
"""
change_state: List[bool] = []
project_files = self(
project_config=project_config,
settings=settings,
)
for child in project_files.project_config.sync_config.directory.iterdir(): # type: ignore
if child.name.startswith(".tmp-"):
print(f"Removing pre-existing temporary directory: {child}")
shutil.rmtree(child)
with tempfile.TemporaryDirectory(
prefix=".tmp-",
dir=project_files.project_config.sync_config.directory # type: ignore
if project_files.project_config.sync_config.temp_in_sync_dir # type: ignore
else None,
) as temp_dir_base_name:
for promoted_release in project_files.promoted_releases:
change_state += [
project_files._sync_version(temp_dir_base=Path(temp_dir_base_name), version=promoted_release)
]
change_state += [project_files._set_latest_version_symlink()]
change_state += [project_files._remove_obsolete_releases()]
if any(change_state):
project_files._set_last_update_file_timestamp()
def _set_last_update_file_timestamp(self) -> None:
"""Write the current seconds since the epoch to a "last update file" if it is configured"""
if self.project_config.sync_config.last_updated_file: # type: ignore
print(f"Updating timestamp in {self.project_config.sync_config.last_updated_file}...") # type: ignore
with open(self.project_config.sync_config.last_updated_file, "w") as file: # type: ignore
file.write(f"{int(time.time())}")
print("Done!")
def _remove_obsolete_releases(self) -> bool:
"""Remove obsolete releases of a project from its sync_dir"""
state: List[bool] = []
expected_dirs: List[Path] = []
expected_files: List[Path] = []
for release_type in self.project_config.releases:
release_type_dir = self.project_config.sync_config.directory / Path(release_type.name) # type: ignore
print(f"Removing obsolete release files from '{release_type_dir}'...")
expected_dirs += [release_type_dir / Path("latest")]
expected_dirs += [
release_type_dir / Path(f"{release_type.name}-{version}") for version in self.promoted_releases
]
expected_files += [
release_type_dir / Path(f"{release_type.name}-{version}.json") for version in self.promoted_releases
]
expected_files += [
release_type_dir / Path(f"{release_type.name}-{version}.torrent") for version in self.promoted_releases
]
for file in release_type_dir.iterdir():
if file.is_dir() and file not in expected_dirs:
print(f"Removing directory '{file}'")
shutil.rmtree(path=file)
state += [True]
if file.is_file() and file not in expected_files:
print(f"Removing file '{file}'")
file.unlink()
state += [True]