1. 18 Mar, 2021 1 commit
    • David Runge's avatar
      Correct and extend models and defaults · 5c491242
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnum `RepoDbType` to be able to identify/address the
      different types of binary database files (e.g. .db vs. .files).
      Change the %LICENSE% entry in `DESC_JSON` to not track the name in
      plural (use "license" instead of "licenses") to stay in line with the
      naming of the original variable.
      Add `DB_USER`, `DB_GROUP`, `DB_FILE_MODE`, `DB_DIR_MODE`, to provide
      defaults for which user and group to create files and directories with
      which file mode for, when generating a database file.
      
      repo_management/models.py:
      Change the `License` model to track an optional list of license
      identifiers using the singular term ("license" instead of "licenses") to
      have more predictable and coherent naming of variables and attributes.
      Add `Base` to `OutputPackageBase` as this way the model is more complete
      and it becomes much easier to use it, without having to also pass the
      pkgbase alongside it.
      Add the convenience method `get_packages_as_models()` to
      `OutputPackageBase` which returns the list of packages as tuples of
      `PackageDesc` and `Files`.
      Extend `PackageDesc` by the convenience method `get_output_package()` to
      easily convert an instance of `PackageDesc` and an optional instance of
      `Files` to an instance of `OutputPackage`.
      Move the declaration of `OutputPackage` above that of `PackageDesc` so
      that `PackageDesc.get_output_package()` can declare typing.
      
      tests/test_models.py
      Add tests for `PackageDesc.get_output_package()` and
      `OutputPackageBase.get_packages_as_models()`.
      5c491242
  2. 17 Mar, 2021 3 commits
    • David Runge's avatar
      Add a set of error classes · 002b821e
      David Runge authored
      repo_management/errors.py:
      Add the Exception class `RepoManagementError` serving as a base, from
      which `RepoManagementFileError` is derived, which serves as a general
      Error class to signal problems with file interaction.
      Add the Exception class `RepoManagementFileNotFoundError`, which is
      derived from `RepoManagementFileError`.
      Add `RepoManagementValidationError` which derives from
      `RepoManagementError` and is raised on validation issues of file
      data.
      002b821e
    • David Runge's avatar
      Add jinja2 to lock file · 1dfe6f80
      David Runge authored
      poetry.lock:
      Add the current version of jinja2 to the lockfile.
      1dfe6f80
    • David Runge's avatar
      Add jinja2 as project dependency · 6ed416e1
      David Runge authored
      pyproject.toml:
      Add jinja2 as repository dependency.
      6ed416e1
  3. 15 Mar, 2021 1 commit
  4. 09 Mar, 2021 5 commits
    • David Runge's avatar
      Remove obsolete db2json script · cf332096
      David Runge authored
      db2json/*:
      Remove obsolete db2json script and README.
      It has been superseded by the script definition in pyproject.toml.
      cf332096
    • David Runge's avatar
      Add a project script setup for db2json · 35a2d7dc
      David Runge authored
      pyproject.toml:
      Add a script setup for dbj2son pointing to `repo_management.cli:db2json`.
      This can be used via `poetry run db2json`.
      35a2d7dc
    • David Runge's avatar
      Add CLI entrypoint for db2json · 8836d5c3
      David Runge authored
      repo_management/cli.py:
      Add `db2json()` which may serve as an entrypoint for a db2json script,
      as it makes use of the argument parser defined in
      `argparse.ArgParseFactory.db2json()` and calls
      `operations.dump_db_to_json_files()` using the provided inputs.
      
      tests/test_cli.py:
      Add a unittest for `db2json()`, which patches all called methods.
      The methods are tested extensively elsewhere. However, the function
      should see proper integration testing via this endpoint in the future.
      8836d5c3
    • David Runge's avatar
      Remove Path checking from operations · 6099e271
      David Runge authored
      repo_management/operations.py:
      Remove the explicit Path checks from `dump_db_to_json_files()` as they
      are now generically implemented in argparse.ArgParseFactory.
      
      tests/test_operations.py
      Remove tests for Path checks in `dump_db_to_json_files()`.
      6099e271
    • David Runge's avatar
      Add ArgumentParser factory · 9bb539dd
      David Runge authored
      repo_management/argparse.py:
      Add `ArgParseFactory`, which serves as an argparse.ArgumentParser
      factory (new variants are implemented as classmethods).
      The class provides a generic constructor, that offers a default parser
      with a verbose argument and a description, alongside a specific one for
      the db2json script.
      Additionally, helper methods are provided to transform strings to Path
      instances, while ensuring that they are either an existing file or
      directory.
      
      tests/test_argparse.py:
      Add tests for all of `ArgParseFactory`.
      9bb539dd
  5. 08 Mar, 2021 13 commits
    • David Runge's avatar
      Add method to dump repo dbs to JSON files · 25b0c7ca
      David Runge authored
      repo_management/operations.py:
      Add `dump_db_to_json_files()` which allows for a repository database
      file to be read and all of its members to be serialized as JSON and
      written to files based upon the packages respective pkgbases.
      For the JSON serialization orjson is used, as it provides the highest
      speeds available and seems the most correct implementation.
      
      tests/test_operations.py:
      Add a fixture to create and destroy a temporary directory (used to write
      JSON files to).
      Add tests for `dump_db_to_json_files()`.
      25b0c7ca
    • David Runge's avatar
      Update lock file for orjson and mock · 7e3a448d
      David Runge authored
      poetry.lock:
      Update lock file for current versions of orjson and mock.
      7e3a448d
    • David Runge's avatar
      Add orjson and mock · fe9f7c1d
      David Runge authored
      pyproject.toml:
      Add orjson to the list of dependencies.
      Add mock to the list of development dependencies.
      fe9f7c1d
    • David Runge's avatar
      Make use of common helper for fixtures · e04209ae
      David Runge authored
      tests/test_files.py:
      Make use of common helper in `tests/fixtures.py` for creating repository
      database fixtures.
      e04209ae
    • David Runge's avatar
      Add iterator over repository database · 4e4d5bbe
      David Runge authored
      repo_management/operations.py:
      Add the public method `db_file_as_models()`, which yields all members of
      a repository database as a Tuple of the pkgbase name and the pkgbase
      data (represented as an instance of models.OutputPackageBase).
      
      tests/test_operations.py:
      Add a test for `db_file_as_models()`.
      4e4d5bbe
    • David Runge's avatar
      Add a fixture helper for tests · 53f6c29d
      David Runge authored
      tests/fixtures.py:
      Add a fixture helper for tests to easily create a valid repository
      database with a few dummy entries.
      53f6c29d
    • David Runge's avatar
      Add resources to create a dummy repository database · 538b11f3
      David Runge authored
      tests/resources/repo_db/*:
      Add a few dummy resources, representing actual package data to create a
      dummy repository database from for testing purposes.
      The resources include default and split packages.
      Note: These resources will (hopefully) become obsolete with a switch to
      python-pytest-pacman.
      538b11f3
    • David Runge's avatar
      Rename tests for files · 2ec219d6
      David Runge authored
      Rename the tests file used to test files.
      2ec219d6
    • David Runge's avatar
      Change repository member iterator to yield model · dc8d4b80
      David Runge authored
      repo_management/files.py:
      Add the private method `_extract_db_member_package_name()` which can be
      used to extract the package name from the name of a member of repository
      database.
      Rename `_read_db_file_member()` to `_db_file_member_as_model()` and
      change the method to yield a `models.RepoDbMemberData` instead of a
      `tarfile.TarInfo` to provide actual data (and required context) to
      callers of the method.
      
      tests/test_file.py:
      Add parametrized test for `_extract_db_member_package_name()`.
      Change test for `_db_file_member_as_model()` to assert the correct
      return type.
      dc8d4b80
    • David Runge's avatar
      Add conversion method for PackageDesc to OutputPackage · 45e96813
      David Runge authored
      repo_management/convert.py:
      Add private method `_transform_package_desc_to_output_package()` which
      can be used to construct a `models.OutputPackage` from a
      `models.PackageDesc` (and optionally a `models.PackageFiles`).
      The method drops attributes from a dict representation of the
      `models.PackageDesc` instance, that are not relevant for the
      `models.OutputPackage`.
      
      tests/test_convert.py:
      Add parametrized test for `_transform_package_desc_to_output_package()`.
      45e96813
    • David Runge's avatar
      Add models for describing output facing package data · f549d51a
      David Runge authored
      repo_management/models.py:
      Add model `PackageFiles` to track the list of files for a given package.
      Add model `RepoDbMemberType` to track the type of a repository database
      member (distinguished by the IntEnum defaults.RepoDbMemberType).
      Add model `RepoDbMemberData` to track the data (represented by an
      io.StringIO instance) of a repository database member.
      Add model `OutputPackage` to track the attributes required for a single
      package in the context of a representation within its pkgbase.
      Add model `OutputPackageBase` to track the attributes required for a
      pkgbase and its list of packages.
      f549d51a
    • David Runge's avatar
      Add IntEnum to distinguish members of a repository db · 98c227d1
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnum `RepoDbMemberType` to distinguish different members
      (files) in a repository database (e.g. 'desc' or 'files' files).
      98c227d1
    • David Runge's avatar
      Add iterator for TarFile members · 2660ca6c
      David Runge authored
      repo_management/files.py:
      Add iterator method `_read_db_file_member()` to yield the members of a
      provided tarfile.TarFile.
      The members are filtered by name using a regular expression, which can
      be overridden.
      
      tests/test_file.py:
      Change `create_db_file()` to also create 'desc' and 'files' files below
      the created temporary directories.
      Assert, that all tested methods return a value.
      Add test for `_read_db_file_member()`.
      2660ca6c
  6. 07 Mar, 2021 5 commits
    • David Runge's avatar
      Add method to read db file · 70bdbd76
      David Runge authored
      repo_management/files.py:
      Add `_read_db_file()` to open a db file and return it as
      tarfile.Tarfile. The method assumes gzip-compressed databases by
      default, but can be used with other compression algorithms.
      
      tests/test_file.py:
      Add fixtures and tests for `_read_db_file()`.
      70bdbd76
    • David Runge's avatar
      Add conversion of 'desc' files · 84e32326
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnmu `FieldType` which tracks the required type for an
      attribute when converting from 'desc' or 'files' files to pydantic
      models.
      Extend the DESC_JSON and FILES_JSON globals by introducing a dict per
      key, that describes the name and the required type, when converting to
      pydantic models.
      
      repo_management/convert.py:
      Rename `_files_data_to_dict()` to `_read_files_data_to_model()` and
      return a pydantic model (`models.Files`) instead of a custom dict.
      Add `_desc_data_to_model()` which allows for conversion of a 'desc' data
      file (represented as io.StringIO) to a pydantic model
      (`models.PackageDesc`).
      
      tests/test_convert.py:
      Add another test for raising RuntimeError in
      `_read_files_data_to_model()`.
      Add a parametrized test for `_desc_data_to_model()`.
      84e32326
    • David Runge's avatar
      Add pydantic models for 'desc' and 'files' files · 72cfd707
      David Runge authored
      repo_management/models.py:
      Add pydantic models for all headers in 'desc' and 'files' files (found
      in a repository database per pkgbase).
      The models describe the attribute names in use when reading data from
      the respective files and whether they are required or not.
      72cfd707
    • David Runge's avatar
      Update lock file to include pydantic · 59dfe2e8
      David Runge authored
      poetry.lock:
      Update the lock file to include the latest version of pydantic.
      59dfe2e8
    • David Runge's avatar
      Add pydantic to dependencies · 505ddd15
      David Runge authored
      pyproject.toml:
      Add pydantic to dependencies, as it will be used to validate
      configuration and internal typing.
      505ddd15
  7. 06 Mar, 2021 10 commits
    • David Runge's avatar
      Add gitlab-ci integration · c7227396
      David Runge authored
      .gitlab-ci.yml:
      Add Gitlab CI integration, that runs tox's linter and coverage targets
      in a lint stage.
      c7227396
    • David Runge's avatar
      Add initial tests for convert · d9105841
      David Runge authored
      tests/test_convert.py:
      Add initial tests for the convert module by adding a parametrized test
      for `_files_data_to_dict()`.
      d9105841
    • David Runge's avatar
      Add repo_management package · 02936e68
      David Runge authored
      repo_management/defaults.py:
      Add the basic defaults module, providing data that translates keywords
      found in the 'desc' and 'files' files of a binary repository database to
      those used in JSON files, representing the same data.
      
      repo_management/convert.py:
      Add the convert module, and introduce the simple private method
      (reimplementing a subset of the features from db2json)
      `_files_data_to_dict()`, which translates the contents of a `files` file
      to a typed dict.
      02936e68
    • David Runge's avatar
      Add tox integration · 190fff47
      David Runge authored
      tox.ini:
      Add tox integration for testing (linting and coverage) in virtual
      environments using poetry.
      190fff47
    • David Runge's avatar
      Add poetry lock file · 355cb562
      David Runge authored
      poetry.lock:
      Add lock file with all dependencies currently used to test the project.
      355cb562
    • David Runge's avatar
      Add pyproject.toml for project · 6a28acf0
      David Runge authored
      pyproject.toml:
      Add a pyproject.toml for describing the project, configuring all
      currently required dependencies and development dependencies.
      The configuration adds entries for the code formatter black, the import
      sorting tool isort and for the coverage report generation tool coverage.
      6a28acf0
    • David Runge's avatar
      Add a gitignore file · 78bc4945
      David Runge authored
      .gitignore:
      Add a ignore file for git, with many default entries for python projects
      related files, that should never end up in version control.
      78bc4945
    • David Runge's avatar
      Add mypy configuration file · 38e95d2b
      David Runge authored
      .mypy.ini:
      As mypy can not yet be configured through pyproject.toml, add a specific
      configuration file for it.
      38e95d2b
    • David Runge's avatar
      Add flake8 configuration · 1370fc60
      David Runge authored
      .flake8:
      As flake8 can not yet be configured in a pyproject.toml, use a specific
      file for it.
      Set the maximum complexity to 10 (can even be lower) and add default
      exclusions for project unrelated files.
      1370fc60
    • David Runge's avatar
      Add editorconfig integration · f9926c15
      David Runge authored
      .editorconfig:
      Add editorconfig integration, that covers .py, .rst and .yml files.
      f9926c15
  8. 28 Dec, 2019 2 commits