Merge branch 'issues/12'

* issues/12:
  pyproject.toml: Update project version to 0.2.0
  README: Extend with information on arch-release-sync
  example.toml: Add synchronization related documentation
  Add script entrypoint for arch-release-sync
  cli: Add entrypoint for arch-release-sync
  files: Add ProjectFiles class to handle synchronization
  gitlab: More explicit print when downloading build artifacts
  gitlab: Implement download of promotion artifact
  argparse: Add parser for arch-release-sync
  gitlab: Change signature of constructor
  gitlab: More generically retrieve types of releases
  config: Make PRIVATE_TOKEN optional
  config: Add integration to configure synchronization

README.rst

@@ -2,8 +2,8 @@
 arch-release-promotion
 ======================
 
-This project allows for promoting existing releases of a project in Arch
-Linux's Gitlab instance.
+This project allows for promotion and synchronization of existing releases of a
+project in Arch Linux's Gitlab instance.
 
 Releases of a project (e.g. ``project``) may consist of several release types
 (e.g. ``image_a`` and ``image_b``), which are addressed separately.
@@ -13,21 +13,30 @@ artifacts (optional), a torrent file (optional) and a JSON payload which can be
 used by `archweb <https://github.com/archlinux/archweb>`_ to display
 information about each release type.
 
+Synchronization with a local directory can be achieved for a configurable
+maximum amount of release versions (each consisting of their respective
+configured release types) of a project.
+
 Requirements
 ============
 
-The arch-release-promotion tool is Python based. All requirements are specified
-in its `pyproject.toml <pyproject.toml>`_.
+Arch-release-promotion is Python based. All language specific requirements are
+specified in its `pyproject.toml <pyproject.toml>`_.
+
+Additionally, ``arch-release-promotion`` requires `gnupg <https://gnupg.org/>`_
+to handle detached PGP signatures.
 
 Use
 ===
 
-After installation, refer to the output of ``arch-release-promotion -h``.
+After installation, refer to the output of ``arch-release-promotion -h`` and
+``arch-release-sync -h``.
 
 Configuration
 =============
 
-The command-line tool ``arch-release-promotion`` makes use of two sources of configuration:
+The command-line tools ``arch-release-promotion`` and ``arch-release-sync``
+make use of two sources of configuration:
 
 * `makepkg.conf <https://man.archlinux.org/man/makepkg.conf.5>`_ is read from
   any of its locations in the same priority as `makepkg
@@ -38,11 +47,11 @@ The command-line tool ``arch-release-promotion`` makes use of two sources of con
   * ``PACKAGER`` is recognized for establishing who is doing the signature and
     is important for `WKD
     <https://wiki.archlinux.org/title/GnuPG#Web_Key_Directory>`_ lookup
-  * ``MIRRORLIST_URL`` (not used by makepkg) is used during the generation of torrent files to add
-    webseeds (defaults to
+  * ``MIRRORLIST_URL`` (not used by makepkg) is used during the generation of
+    torrent files to add webseeds (defaults to
     ``"https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https"``)
-  * ``GITLAB_URL`` (not used by makepkg) is used to connect to a GitLab instance to select, download
-    and promote releases of a project (defaults to
+  * ``GITLAB_URL`` (not used by makepkg) is used to connect to a GitLab
+    instance to select, download and promote releases of a project (defaults to
     ``"https://gitlab.archlinux.org"``)
   * ``PRIVATE_TOKEN`` (not used by makepkg) is used to authenticate against the
     GitLab instance. The `personal access token
@@ -239,7 +248,54 @@ each release type in the release.
   release.
 * ``version``: The version of the release type.
 
+Synchronization
+===============
+
+The synchronization of releases works by retrieving the list of promoted
+releases of the project from the remote. For each promoted release version, the
+promotion artifact is downloaded and used to establish whether all of the
+configured release types are fully synchronized.
+
+Location and cleanup
+--------------------
+
+All release types for each release version are synchronized to a local
+directory. The directory and and the maximum amount of synchronized release
+versions are configurable globally or per project.
+
+.. code::
+
+   sync_dir
+   ├── example_a
+   │   ├── example_a-0.1.0
+   │   │   ├── foo.txt
+   │   │   └── foo.txt.sig
+   │   ├── example_a-0.1.0.json
+   │   ├── example_a-0.1.0.torrent
+   │   └── latest -> example_a-0.1.0
+   └── example_b
+       ├── example_b-0.1.0
+       │   ├── bar.txt
+       │   └── bar.txt.sig
+       ├── example_b-0.1.0.json
+       ├── example_b-0.1.0.torrent
+       └── latest -> example_b-0.1.0
+
+A ``latest`` symlink is created to point at the currently latest version of
+each release type.
+
+Any files and directories that are not owned by versions of release types of
+the currently synchronized release versions are removed from the
+synchronization directory.
+
+If changes are introduced to files in the target directory (due to a
+synchronization action), it is possible to write a Unix timestamp to a file
+that is configurable globally or per project (the directory in which the file
+resides in has to exist).
+
 License
 =======
 
-Arch-release-promotion is licensed under the terms of the **GPL-3.0-or-later** (see `LICENSE <LICENSE>`_).
+Arch-release-promotion is licensed under the terms of the **GPL-3.0-or-later**
+(see `LICENSE <LICENSE>`_).
+

arch_release_promotion/argparse.py

@@ -67,6 +67,37 @@ class ArgParseFactory:
 
         return instance.parser
 
+    @classmethod
+    def synchronize(self) -> argparse.ArgumentParser:
+        """A class method to create an ArgumentParser for synchronization
+
+        Returns
+        -------
+        argparse.ArgumentParser
+            An ArgumentParser instance specific for synchronization
+        """
+
+        instance = self(
+            prog="arch-release-sync",
+            description="Synchronize promoted releases of a project with a local directory",
+        )
+        instance.parser.add_argument(
+            "-p",
+            "--project",
+            type=self.non_zero_string,
+            help=(
+                "the project to synchronize from a remote (e.g. 'group/project'). "
+                f"By default {instance.parser.prog} attempts to synchronize releases for "
+                "all projects specified in its config"
+            ),
+        )
+
+        if instance.parser.parse_args().version:
+            print(f"{instance.parser.prog} {metadata.version('arch_release_promotion')}")
+            exit(0)
+
+        return instance.parser
+
     @classmethod
     def non_zero_string(self, input_: str) -> str:
         """Validate a string to be of non_zero length

arch_release_promotion/cli.py

@@ -117,3 +117,27 @@ def main() -> None:
     else:
         for project in config.Projects().projects:
             promote_project_release(project=project)
+
+
+def arch_release_sync() -> None:
+    """Synchronize releases
+
+    If the argument parser contains a specific project to synchronize, only synchronize that, else all configured
+    projects.
+    """
+
+    args = argparse.ArgParseFactory.synchronize().parse_args()
+    projects = config.Projects()
+    settings = config.Settings()
+
+    if args.project:
+        files.ProjectFiles.sync(
+            project_config=projects.get_project(name=args.project),
+            settings=settings,
+        )
+    else:
+        for project in projects.projects:
+            files.ProjectFiles.sync(
+                project_config=project,
+                settings=settings,
+            )

arch_release_promotion/config.py

+from collections import Counter
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple
 
 import toml
 from dotenv import dotenv_values
 from email_validator import EmailNotValidError, validate_email
-from pydantic import BaseModel, BaseSettings, Extra, validator
+from pydantic import BaseModel, BaseSettings, Extra, root_validator, validator
 from pydantic.env_settings import SettingsSourceCallable
 from xdg.BaseDirectory import xdg_config_home
 
@@ -19,6 +20,9 @@ PROJECTS_CONFIGS = [
     Path(f"{xdg_config_home}/arch-release-promotion/projects.toml"),
 ]
 
+PROJECTS_SYNC_DIR = Path("/var/lib/arch-release-promotion/")
+PROJECTS_SYNC_BACKLOG = 3
+
 
 class ReleaseConfig(BaseModel):
     """A pydantic model describing the configuration of a project's release
@@ -50,6 +54,31 @@ class ReleaseConfig(BaseModel):
     create_torrent: bool = False
 
 
+class SyncConfig(BaseModel):
+    """A pydantic model describing configuration for synchronization
+
+    Attributes
+    ----------
+    backlog: int
+        The backlog of releases to retain at a maximum (defaults to PROJECTS_SYNC_BACKLOG when a SysConfig instance is
+        used in a Projects instance or a ProjectConfig instance).
+    directory: Path
+        A directory into which to sync project release types and their respective releases (defaults to
+        PROJECTS_SYNC_DIR when a SysConfig instance is used in a Projects instance or a ProjectConfig instance).
+    last_updated_file: Optional[Path]
+        The optional path to a file, that is used to write a timestamp to, if the synchronization of a project leads to
+        the changing of data on disk (defaults to None).
+    temp_in_sync_dir: bool
+        A bool specifying whether to download temporary data to temporary directories below sync_dir (defaults to True).
+        If False is specified the temporary data is downloaded to the respective user's temporary directory.
+    """
+
+    backlog: Optional[int]
+    directory: Optional[Path]
+    last_updated_file: Optional[Path] = None
+    temp_in_sync_dir: bool = True
+
+
 class ProjectConfig(BaseModel):
     """A pydantic model describing the configuration of a project
 
@@ -65,6 +94,8 @@ class ProjectConfig(BaseModel):
         The project's metrics file
     releases: List[ReleaseConfig]
         The project's list of releases
+    sync_config: Optional[SyncConfig]
+        An optional SyncConfig instance, which is used to override any global defaults.
     """
 
     name: str
@@ -72,6 +103,7 @@ class ProjectConfig(BaseModel):
     output_dir: Path
     metrics_file: Path
     releases: List[ReleaseConfig]
+    sync_config: Optional[SyncConfig]
 
 
 class Projects(BaseSettings):
@@ -81,9 +113,13 @@ class Projects(BaseSettings):
     ----------
     projects: List[ProjectConfig]
         A list of project configurations
+    sync_config: Optional[SyncConfig]
+        An optional SyncConfig instance, which is used to override any implicit defaults and sets defaults for all
+        ProjectConfig instances in projects.
     """
 
     projects: List[ProjectConfig]
+    sync_config: Optional[SyncConfig]
 
     class Config:
 
@@ -98,6 +134,97 @@ class Projects(BaseSettings):
         ) -> Tuple[SettingsSourceCallable, ...]:
             return (read_projects_conf,)
 
+    @validator("projects")
+    def validate_project_releases_unique(cls, projects: List[ProjectConfig]) -> List[ProjectConfig]:
+        """Validate the list of ProjectConfig instances to only contain uniquely named ReleaseConfig instances
+
+        Parameters
+        ----------
+        projects: List[ProjectConfig]
+            A list of ProjectConfig instances
+
+        Raises
+        ------
+        ValueError
+            If a non-unique name is encountered among any ReleaseConfig instance
+
+        Returns
+        -------
+        List[ProjectConfig]
+            The unmodified list of ProjectConfig instances
+        """
+
+        release_types: List[ReleaseConfig] = []
+        for project in projects:
+            release_types += project.releases
+        names = [release_type.name for release_type in release_types]
+
+        if len(set(names)) < len(names):
+            duplicates = [name for name, count in Counter(names).items() if count > 1]
+            raise ValueError(
+                f"The following release type {'name' if len(duplicates) == 1 else 'names'} "
+                f"{'is' if len(duplicates) == 1 else 'are'} not unique: "
+                f"{duplicates[0] if len(duplicates) == 1 else duplicates}"
+            )
+
+        return projects
+
+    @root_validator
+    def validate_projects(cls, values: Dict[str, Any]) -> Dict[str, Any]:
+        """Validate the list of ProjectConfig instances and override defaults
+
+        If a ProjectConfig does not specify a SysConfig, override it with the global SysConfig. If a global SysConfig
+        does not exist, override with an implicit default where directory defaults to PROJECTS_SYNC_DIR and backlog to
+        PROJECTS_SYNC_BACKLOG.
+
+        Parameters
+        ----------
+        values: Dict[str, Any]
+            A dict with all values of the Projects instance
+
+        Returns
+        -------
+        values: Dict[str, Any]
+            The (potentially modified) dict with all values of the Projects instance
+        """
+
+        default_sync_config = SyncConfig(
+            directory=PROJECTS_SYNC_DIR,
+            backlog=PROJECTS_SYNC_BACKLOG,
+            last_updated_file=None,
+        )
+
+        projects: List[ProjectConfig] = values.get("projects")  # type: ignore
+        sync_config: Optional[SyncConfig] = values.get("sync_config")
+
+        # merge global sync_config with defaults
+        if sync_config:
+            global_sync_config = SyncConfig(
+                directory=sync_config.directory or default_sync_config.directory,
+                backlog=sync_config.backlog or default_sync_config.backlog,
+                last_updated_file=sync_config.last_updated_file or default_sync_config.last_updated_file,
+                temp_in_sync_dir=False if not sync_config.temp_in_sync_dir else default_sync_config.temp_in_sync_dir,
+            )
+        else:
+            global_sync_config = default_sync_config
+
+        for project in projects:
+            if not project.sync_config:
+                project.sync_config = global_sync_config
+            else:
+                project.sync_config = SyncConfig(
+                    directory=project.sync_config.directory or global_sync_config.directory,
+                    backlog=project.sync_config.backlog or global_sync_config.backlog,
+                    last_updated_file=project.sync_config.last_updated_file or global_sync_config.last_updated_file,
+                    temp_in_sync_dir=False
+                    if not project.sync_config.temp_in_sync_dir
+                    else global_sync_config.temp_in_sync_dir,
+                )
+
+        values["projects"] = projects
+
+        return values
+
     def get_project(self, name: str) -> ProjectConfig:
         """Return a ProjectConfig by name
 
@@ -156,17 +283,24 @@ class Settings(BaseSettings):
 
     Attributes
     ----------
-    gpgkey: str
+    GITLAB_URL: str
+        A URL for a GitLab upstream (defaults to "https://gitlab.archlinux.org")
+    GPGKEY: str
         The PGP key id to use for artifact signatures
-    packager: str
-        The packager name and mail address to use for artifact signatures
+    MIRRORLIST_URL: str
+        A URL to derive a mirrorlist from (defaults to
+        "https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https")
+    PACKAGER: str
+        The packager name and mail address (UID) to use for artifact signatures
+    PRIVATE_TOKEN: Optional[str]
+        An optional private token to use for authenticating against an upstream
     """
 
-    MIRRORLIST_URL: str = "https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https"
     GITLAB_URL: str = "https://gitlab.archlinux.org"
     GPGKEY: str
+    MIRRORLIST_URL: str = "https://archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https"
     PACKAGER: str
-    PRIVATE_TOKEN: str
+    PRIVATE_TOKEN: Optional[str]
 
     class Config:
 
@@ -244,7 +378,7 @@ class Settings(BaseSettings):
         return gpgkey
 
     @validator("PRIVATE_TOKEN")
-    def validate_private_token(cls, private_token: str) -> str:
+    def validate_private_token(cls, private_token: Optional[str]) -> Optional[str]:
         """A validator for the PRIVATE_TOKEN attribute
 
         Parameters
@@ -263,6 +397,9 @@ class Settings(BaseSettings):
             A gpgkey string in long-format
         """
 
+        if private_token is None:
+            return None
+
         if len(private_token) < 20:
             raise ValueError("The PRIVATE_TOKEN string has to represent a valid private token (20 chars).")
 

arch_release_promotion/files.py

 import shutil
 import tempfile
+import time
 import zipfile
 from pathlib import Path
 from typing import List, Optional, Tuple
 
 import orjson
 from prometheus_client.parser import text_fd_to_metric_families
+from pydantic import BaseModel
 
+from arch_release_promotion.config import ProjectConfig, Settings
+from arch_release_promotion.gitlab import Upstream
 from arch_release_promotion.release import (
     AmountMetric,
     Release,
@@ -176,6 +180,24 @@ def write_release_info_to_file(release: Release, path: Path) -> None:
         )
 
 
+def load_release_from_json_payload(path: Path) -> Release:
+    """Read a JSON payload and return it as a Release instance
+
+    Parameters
+    ----------
+    path: Path
+        The path to a file containing a JSON payload
+
+    Returns
+    -------
+    Release
+        A Release instance reflecting the data from the JSON payload
+    """
+
+    with open(path, "r") as file:
+        return Release(**orjson.loads(file.read()))
+
+
 def write_zip_file_to_parent_dir(path: Path, name: str = "promotion", format: str = "zip") -> None:
     """Create ZIP file of all contents in a directory and write it to the directory's parent
 
@@ -277,3 +299,441 @@ def read_metrics_file(
                             )
                         ]
     return (amount_metrics, size_metrics, version_metrics)
+
+
+def create_dir(path: Path) -> Path:
+    """Create a directory
+
+    Parameters
+    ----------
+    path: Path
+        The path for which to create a directory
+
+    Raises
+    ------
+    RuntimeError
+        If the path exists but is not a directory
+
+    Returns
+    -------
+    Path
+        The path representing the existing, absolute directory
+    """
+
+    path = path.resolve(strict=False)
+
+    if path.exists() and not path.is_dir():
+        raise RuntimeError(f"The provided path is not a directory: {path}")
+    else:
+        path.mkdir(parents=True, exist_ok=True)
+
+    return path
+
+
+class ProjectFiles(BaseModel):
+    """A pydantic model to operate on a project's files and releases
+
+    Attributes
+    ----------
+    project_config: ProjectConfig
+        A ProjectConfig instance describing the project
+    upstream: Upstream
+        An Upstream instance used for queries to releases of a project
+    promoted_releases: List[str]
+        A list of version strings
+    """
+
+    project_config: ProjectConfig
+    upstream: Upstream
+    promoted_releases: List[str]
+
+    class Config:
+        arbitrary_types_allowed = True
+
+    def __init__(self, project_config: ProjectConfig, settings: Settings) -> None:
+        """A custom constructor to initialize an instance of ProjectFiles
+
+        The names of the configured maximum number of promoted releases is retrieved using an Upstream instance.
+        If the project's sync_dir does not exist, it will be created.
+
+        Parameters
+        ----------
+        project_config: ProjectConfig
+            A ProjectConfig instance describing the project
+        settings: Settings
+            A Settings instance used to initialize an Upstream instance
+        """
+
+        upstream = Upstream(
+            url=settings.GITLAB_URL,
+            private_token=None,
+            name=project_config.name,
+        )
+        promoted_releases = upstream.get_releases(
+            max_releases=project_config.sync_config.backlog,  # type: ignore
+            promoted=True,
+        )
+        print(f"Synchronizing release versions for {project_config.name}: " f"{', '.join(promoted_releases)}")
+
+        create_dir(path=project_config.sync_config.directory)  # type: ignore
+
+        super().__init__(
+            project_config=project_config,
+            upstream=upstream,
+            promoted_releases=promoted_releases,
+        )
+
+    @classmethod
+    def sync(self, project_config: ProjectConfig, settings: Settings) -> None:
+        """A factory method to initialize an instance of ProjectFiles and synchronize all of its promoted_releases
+
+        Parameters
+        ----------
+        project_config: ProjectConfig
+            A ProjectConfig instance describing the project
+        settings: Settings
+            A Settings instance used to initialize an Upstream instance
+        """
+
+        change_state: List[bool] = []
+
+        project_files = self(
+            project_config=project_config,
+            settings=settings,
+        )
+
+        for child in project_files.project_config.sync_config.directory.iterdir():  # type: ignore
+            if child.name.startswith(".tmp-"):
+                print(f"Removing pre-existing temporary directory: {child}")
+                shutil.rmtree(child)
+
+        with tempfile.TemporaryDirectory(
+            prefix=".tmp-",
+            dir=project_files.project_config.sync_config.directory  # type: ignore
+            if project_files.project_config.sync_config.temp_in_sync_dir  # type: ignore
+            else None,
+        ) as temp_dir_base_name:
+            for promoted_release in project_files.promoted_releases:
+                change_state += [
+                    project_files._sync_version(temp_dir_base=Path(temp_dir_base_name), version=promoted_release)
+                ]
+
+        change_state += [project_files._set_latest_version_symlink()]
+        change_state += [project_files._remove_obsolete_releases()]
+
+        if any(change_state):
+            project_files._set_last_update_file_timestamp()
+
+    def _set_last_update_file_timestamp(self) -> None:
+        """Write the current seconds since the epoch to a "last update file" if it is configured"""
+
+        if self.project_config.sync_config.last_updated_file:  # type: ignore
+            print(f"Updating timestamp in {self.project_config.sync_config.last_updated_file}...")  # type: ignore
+            with open(self.project_config.sync_config.last_updated_file, "w") as file:  # type: ignore
+                file.write(f"{int(time.time())}")
+
+            print("Done!")
+
+    def _remove_obsolete_releases(self) -> bool:
+        """Remove obsolete releases of a project from its sync_dir"""
+
+        state: List[bool] = []
+        expected_dirs: List[Path] = []
+        expected_files: List[Path] = []
+
+        for release_type in self.project_config.releases:
+            release_type_dir = self.project_config.sync_config.directory / Path(release_type.name)  # type: ignore
+            print(f"Removing obsolete release files from '{release_type_dir}'...")
+
+            expected_dirs += [release_type_dir / Path("latest")]
+            expected_dirs += [
+                release_type_dir / Path(f"{release_type.name}-{version}") for version in self.promoted_releases
+            ]
+            expected_files += [
+                release_type_dir / Path(f"{release_type.name}-{version}.json") for version in self.promoted_releases
+            ]
+            expected_files += [
+                release_type_dir / Path(f"{release_type.name}-{version}.torrent") for version in self.promoted_releases
+            ]
+
+            for file in release_type_dir.iterdir():
+                if file.is_dir() and file not in expected_dirs:
+                    print(f"Removing directory '{file}'")
+                    shutil.rmtree(path=file)
+                    state += [True]
+                if file.is_file() and file not in expected_files:
+                    print(f"Removing file '{file}'")
+                    file.unlink()
+                    state += [True]