Skip to content
Snippets Groups Projects

Refactor db2json into repo_management

Merged David Runge requested to merge issues/2 into master
  1. Mar 09, 2021
    • David Runge's avatar
      Remove obsolete db2json script · cf332096
      David Runge authored
      db2json/*:
      Remove obsolete db2json script and README.
      It has been superseded by the script definition in pyproject.toml.
      Verified
      cf332096
    • David Runge's avatar
      Add a project script setup for db2json · 35a2d7dc
      David Runge authored
      pyproject.toml:
      Add a script setup for dbj2son pointing to `repo_management.cli:db2json`.
      This can be used via `poetry run db2json`.
      Verified
      35a2d7dc
    • David Runge's avatar
      Add CLI entrypoint for db2json · 8836d5c3
      David Runge authored
      repo_management/cli.py:
      Add `db2json()` which may serve as an entrypoint for a db2json script,
      as it makes use of the argument parser defined in
      `argparse.ArgParseFactory.db2json()` and calls
      `operations.dump_db_to_json_files()` using the provided inputs.
      
      tests/test_cli.py:
      Add a unittest for `db2json()`, which patches all called methods.
      The methods are tested extensively elsewhere. However, the function
      should see proper integration testing via this endpoint in the future.
      Verified
      8836d5c3
    • David Runge's avatar
      Remove Path checking from operations · 6099e271
      David Runge authored
      repo_management/operations.py:
      Remove the explicit Path checks from `dump_db_to_json_files()` as they
      are now generically implemented in argparse.ArgParseFactory.
      
      tests/test_operations.py
      Remove tests for Path checks in `dump_db_to_json_files()`.
      Verified
      6099e271
    • David Runge's avatar
      Add ArgumentParser factory · 9bb539dd
      David Runge authored
      repo_management/argparse.py:
      Add `ArgParseFactory`, which serves as an argparse.ArgumentParser
      factory (new variants are implemented as classmethods).
      The class provides a generic constructor, that offers a default parser
      with a verbose argument and a description, alongside a specific one for
      the db2json script.
      Additionally, helper methods are provided to transform strings to Path
      instances, while ensuring that they are either an existing file or
      directory.
      
      tests/test_argparse.py:
      Add tests for all of `ArgParseFactory`.
      Verified
      9bb539dd
  2. Mar 08, 2021
    • David Runge's avatar
      Add method to dump repo dbs to JSON files · 25b0c7ca
      David Runge authored
      repo_management/operations.py:
      Add `dump_db_to_json_files()` which allows for a repository database
      file to be read and all of its members to be serialized as JSON and
      written to files based upon the packages respective pkgbases.
      For the JSON serialization orjson is used, as it provides the highest
      speeds available and seems the most correct implementation.
      
      tests/test_operations.py:
      Add a fixture to create and destroy a temporary directory (used to write
      JSON files to).
      Add tests for `dump_db_to_json_files()`.
      Verified
      25b0c7ca
    • David Runge's avatar
      Update lock file for orjson and mock · 7e3a448d
      David Runge authored
      poetry.lock:
      Update lock file for current versions of orjson and mock.
      Verified
      7e3a448d
    • David Runge's avatar
      Add orjson and mock · fe9f7c1d
      David Runge authored
      pyproject.toml:
      Add orjson to the list of dependencies.
      Add mock to the list of development dependencies.
      Verified
      fe9f7c1d
    • David Runge's avatar
      Make use of common helper for fixtures · e04209ae
      David Runge authored
      tests/test_files.py:
      Make use of common helper in `tests/fixtures.py` for creating repository
      database fixtures.
      Verified
      e04209ae
    • David Runge's avatar
      Add iterator over repository database · 4e4d5bbe
      David Runge authored
      repo_management/operations.py:
      Add the public method `db_file_as_models()`, which yields all members of
      a repository database as a Tuple of the pkgbase name and the pkgbase
      data (represented as an instance of models.OutputPackageBase).
      
      tests/test_operations.py:
      Add a test for `db_file_as_models()`.
      Verified
      4e4d5bbe
    • David Runge's avatar
      Add a fixture helper for tests · 53f6c29d
      David Runge authored
      tests/fixtures.py:
      Add a fixture helper for tests to easily create a valid repository
      database with a few dummy entries.
      Verified
      53f6c29d
    • David Runge's avatar
      Add resources to create a dummy repository database · 538b11f3
      David Runge authored
      tests/resources/repo_db/*:
      Add a few dummy resources, representing actual package data to create a
      dummy repository database from for testing purposes.
      The resources include default and split packages.
      Note: These resources will (hopefully) become obsolete with a switch to
      python-pytest-pacman.
      Verified
      538b11f3
    • David Runge's avatar
      Rename tests for files · 2ec219d6
      David Runge authored
      Rename the tests file used to test files.
      Verified
      2ec219d6
    • David Runge's avatar
      Change repository member iterator to yield model · dc8d4b80
      David Runge authored
      repo_management/files.py:
      Add the private method `_extract_db_member_package_name()` which can be
      used to extract the package name from the name of a member of repository
      database.
      Rename `_read_db_file_member()` to `_db_file_member_as_model()` and
      change the method to yield a `models.RepoDbMemberData` instead of a
      `tarfile.TarInfo` to provide actual data (and required context) to
      callers of the method.
      
      tests/test_file.py:
      Add parametrized test for `_extract_db_member_package_name()`.
      Change test for `_db_file_member_as_model()` to assert the correct
      return type.
      Verified
      dc8d4b80
    • David Runge's avatar
      Add conversion method for PackageDesc to OutputPackage · 45e96813
      David Runge authored
      repo_management/convert.py:
      Add private method `_transform_package_desc_to_output_package()` which
      can be used to construct a `models.OutputPackage` from a
      `models.PackageDesc` (and optionally a `models.PackageFiles`).
      The method drops attributes from a dict representation of the
      `models.PackageDesc` instance, that are not relevant for the
      `models.OutputPackage`.
      
      tests/test_convert.py:
      Add parametrized test for `_transform_package_desc_to_output_package()`.
      Verified
      45e96813
    • David Runge's avatar
      Add models for describing output facing package data · f549d51a
      David Runge authored
      repo_management/models.py:
      Add model `PackageFiles` to track the list of files for a given package.
      Add model `RepoDbMemberType` to track the type of a repository database
      member (distinguished by the IntEnum defaults.RepoDbMemberType).
      Add model `RepoDbMemberData` to track the data (represented by an
      io.StringIO instance) of a repository database member.
      Add model `OutputPackage` to track the attributes required for a single
      package in the context of a representation within its pkgbase.
      Add model `OutputPackageBase` to track the attributes required for a
      pkgbase and its list of packages.
      Verified
      f549d51a
    • David Runge's avatar
      Add IntEnum to distinguish members of a repository db · 98c227d1
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnum `RepoDbMemberType` to distinguish different members
      (files) in a repository database (e.g. 'desc' or 'files' files).
      Verified
      98c227d1
    • David Runge's avatar
      Add iterator for TarFile members · 2660ca6c
      David Runge authored
      repo_management/files.py:
      Add iterator method `_read_db_file_member()` to yield the members of a
      provided tarfile.TarFile.
      The members are filtered by name using a regular expression, which can
      be overridden.
      
      tests/test_file.py:
      Change `create_db_file()` to also create 'desc' and 'files' files below
      the created temporary directories.
      Assert, that all tested methods return a value.
      Add test for `_read_db_file_member()`.
      Verified
      2660ca6c
  3. Mar 07, 2021
    • David Runge's avatar
      Add method to read db file · 70bdbd76
      David Runge authored
      repo_management/files.py:
      Add `_read_db_file()` to open a db file and return it as
      tarfile.Tarfile. The method assumes gzip-compressed databases by
      default, but can be used with other compression algorithms.
      
      tests/test_file.py:
      Add fixtures and tests for `_read_db_file()`.
      Verified
      70bdbd76
    • David Runge's avatar
      Add conversion of 'desc' files · 84e32326
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnmu `FieldType` which tracks the required type for an
      attribute when converting from 'desc' or 'files' files to pydantic
      models.
      Extend the DESC_JSON and FILES_JSON globals by introducing a dict per
      key, that describes the name and the required type, when converting to
      pydantic models.
      
      repo_management/convert.py:
      Rename `_files_data_to_dict()` to `_read_files_data_to_model()` and
      return a pydantic model (`models.Files`) instead of a custom dict.
      Add `_desc_data_to_model()` which allows for conversion of a 'desc' data
      file (represented as io.StringIO) to a pydantic model
      (`models.PackageDesc`).
      
      tests/test_convert.py:
      Add another test for raising RuntimeError in
      `_read_files_data_to_model()`.
      Add a parametrized test for `_desc_data_to_model()`.
      Verified
      84e32326
    • David Runge's avatar
      Add pydantic models for 'desc' and 'files' files · 72cfd707
      David Runge authored
      repo_management/models.py:
      Add pydantic models for all headers in 'desc' and 'files' files (found
      in a repository database per pkgbase).
      The models describe the attribute names in use when reading data from
      the respective files and whether they are required or not.
      Verified
      72cfd707
    • David Runge's avatar
      Update lock file to include pydantic · 59dfe2e8
      David Runge authored
      poetry.lock:
      Update the lock file to include the latest version of pydantic.
      Verified
      59dfe2e8
    • David Runge's avatar
      Add pydantic to dependencies · 505ddd15
      David Runge authored
      pyproject.toml:
      Add pydantic to dependencies, as it will be used to validate
      configuration and internal typing.
      Verified
      505ddd15
Loading