1. 20 Mar, 2021 2 commits
  2. 18 Mar, 2021 9 commits
    • David Runge's avatar
      Merge branch 'issues/4' into 'master' · bb999eab
      David Runge authored
      Add functionality to write a database file from a set of JSON files
      
      Closes #4
      
      See merge request !4
      bb999eab
    • David Runge's avatar
      Add script entrypoint for json2db · c51b92b9
      David Runge authored
      pyproject.toml:
      Add script entrypoint for `json2db`.
      c51b92b9
    • David Runge's avatar
      Add entrypoint for json2db · d542d1f4
      David Runge authored
      repo_management/cli.py:
      Add `json2db()` as entrypoint for the `json2db` script.
      
      tests/test_cli.py:
      Add a test for `json2db()`.
      d542d1f4
    • David Runge's avatar
      Add argparse for json2db · 0f02fe86
      David Runge authored
      repo_management/argparse.py:
      Add `ArgParseFactory.json2db()` as an ArgumentParser for the `json2db`
      script.
      Add `ArgParseFactory.string_to_writable_file_path()` for returning a
      Path object, that is guaranteed to be a writable file.
      
      tests/test_argparse.py:
      Add tests for
      `ArgParseFactory.{json2db,string_to_writable_file_path}()`.
      0f02fe86
    • David Runge's avatar
      Add operation to create a db from JSON files · d1fa043d
      David Runge authored
      repo_management/operations.py:
      Add `create_db_from_json_files()` to create a repository database from
      JSON files in a directory.
      Change `db_file_as_models()` to rely on
      `models.PackageDesc.get_output_packages()` instead of
      `convert._transform_package_desc_to_output_package()` and add a missing
      parameter (base) to the instantiation of `models.OutputPackageBase()`.
      
      tests/test_operations.py:
      Add the two fixtures `dummy_json_files_in_dir()` and `empty_file()` to
      create a few (valid) dummy input JSON files in a temporary directory and
      an empty file (respectively).
      Add a test for `create_db_from_json_files()`.
      d1fa043d
    • David Runge's avatar
      Add methods for reading and streaming JSON files · 3103d0f7
      David Runge authored
      repo_management/files.py:
      Add `_json_files_in_directory()` yield a list of files with a .json
      suffix in a directory.
      Add `_read_pkgbase_json_file()` to read a JSON file representing a
      pkgbase and return an `OutputPackageBase` model.
      Add `_write_db_file()` to open a tar file for writing.
      Add `_stream_package_base_to_db()` to stream a `models.OutputPackage()`
      to a tar file (which represents a .db or a .files repository database).
      
      tests/test_files.py:
      Add tests for `_json_files_in_directory()`, `_read_pkgbase_json_file()`,
      `_write_db_file()` and `_stream_package_base_to_db()`.
      3103d0f7
    • David Runge's avatar
      Add fixture helper for json files · 3b9c7b5a
      David Runge authored
      tests/fixtures.py:
      Add `create_empty_json_files()` which creates a set of empty .json files
      in a temporary directory and can be used by fixtures (which remove the
      temporary directory afterwards).
      Add `create_json_files()` which creates a set of valid JSON files in a
      temporary directory using the `models.OutputPackageBase` model.
      3b9c7b5a
    • David Runge's avatar
      Simplify convert and add class for template rendering · 9cff77d7
      David Runge authored
      repo_management/convert.py:
      Add the class `RepoDbFile` which is used to initialize a jinja
      environment and allows to render the 'desc.j2' and 'files.j2' templates.
      Add `_desc_data_line_to_dicts()` which allows for assigning the data
      from a line in a 'desc' file to be assigned to a dict, that corresponds
      to the line's designated data type.
      Simplify `_desc_data_to_model()` by making use of
      `_desc_data_line_to_dicts()`.
      Change `_desc_data_to_model()` by guarding against pydantic
      ValidationErrors and raising a RepoManagementValidationError instead.
      
      tests/test_convert.py:
      Add tests for `RepoDbFile`.
      9cff77d7
    • David Runge's avatar
      Correct and extend models and defaults · 5c491242
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnum `RepoDbType` to be able to identify/address the
      different types of binary database files (e.g. .db vs. .files).
      Change the %LICENSE% entry in `DESC_JSON` to not track the name in
      plural (use "license" instead of "licenses") to stay in line with the
      naming of the original variable.
      Add `DB_USER`, `DB_GROUP`, `DB_FILE_MODE`, `DB_DIR_MODE`, to provide
      defaults for which user and group to create files and directories with
      which file mode for, when generating a database file.
      
      repo_management/models.py:
      Change the `License` model to track an optional list of license
      identifiers using the singular term ("license" instead of "licenses") to
      have more predictable and coherent naming of variables and attributes.
      Add `Base` to `OutputPackageBase` as this way the model is more complete
      and it becomes much easier to use it, without having to also pass the
      pkgbase alongside it.
      Add the convenience method `get_packages_as_models()` to
      `OutputPackageBase` which returns the list of packages as tuples of
      `PackageDesc` and `Files`.
      Extend `PackageDesc` by the convenience method `get_output_package()` to
      easily convert an instance of `PackageDesc` and an optional instance of
      `Files` to an instance of `OutputPackage`.
      Move the declaration of `OutputPackage` above that of `PackageDesc` so
      that `PackageDesc.get_output_package()` can declare typing.
      
      tests/test_models.py
      Add tests for `PackageDesc.get_output_package()` and
      `OutputPackageBase.get_packages_as_models()`.
      5c491242
  3. 17 Mar, 2021 3 commits
    • David Runge's avatar
      Add a set of error classes · 002b821e
      David Runge authored
      repo_management/errors.py:
      Add the Exception class `RepoManagementError` serving as a base, from
      which `RepoManagementFileError` is derived, which serves as a general
      Error class to signal problems with file interaction.
      Add the Exception class `RepoManagementFileNotFoundError`, which is
      derived from `RepoManagementFileError`.
      Add `RepoManagementValidationError` which derives from
      `RepoManagementError` and is raised on validation issues of file
      data.
      002b821e
    • David Runge's avatar
      Add jinja2 to lock file · 1dfe6f80
      David Runge authored
      poetry.lock:
      Add the current version of jinja2 to the lockfile.
      1dfe6f80
    • David Runge's avatar
      Add jinja2 as project dependency · 6ed416e1
      David Runge authored
      pyproject.toml:
      Add jinja2 as repository dependency.
      6ed416e1
  4. 15 Mar, 2021 1 commit
  5. 09 Mar, 2021 5 commits
    • David Runge's avatar
      Remove obsolete db2json script · cf332096
      David Runge authored
      db2json/*:
      Remove obsolete db2json script and README.
      It has been superseded by the script definition in pyproject.toml.
      cf332096
    • David Runge's avatar
      Add a project script setup for db2json · 35a2d7dc
      David Runge authored
      pyproject.toml:
      Add a script setup for dbj2son pointing to `repo_management.cli:db2json`.
      This can be used via `poetry run db2json`.
      35a2d7dc
    • David Runge's avatar
      Add CLI entrypoint for db2json · 8836d5c3
      David Runge authored
      repo_management/cli.py:
      Add `db2json()` which may serve as an entrypoint for a db2json script,
      as it makes use of the argument parser defined in
      `argparse.ArgParseFactory.db2json()` and calls
      `operations.dump_db_to_json_files()` using the provided inputs.
      
      tests/test_cli.py:
      Add a unittest for `db2json()`, which patches all called methods.
      The methods are tested extensively elsewhere. However, the function
      should see proper integration testing via this endpoint in the future.
      8836d5c3
    • David Runge's avatar
      Remove Path checking from operations · 6099e271
      David Runge authored
      repo_management/operations.py:
      Remove the explicit Path checks from `dump_db_to_json_files()` as they
      are now generically implemented in argparse.ArgParseFactory.
      
      tests/test_operations.py
      Remove tests for Path checks in `dump_db_to_json_files()`.
      6099e271
    • David Runge's avatar
      Add ArgumentParser factory · 9bb539dd
      David Runge authored
      repo_management/argparse.py:
      Add `ArgParseFactory`, which serves as an argparse.ArgumentParser
      factory (new variants are implemented as classmethods).
      The class provides a generic constructor, that offers a default parser
      with a verbose argument and a description, alongside a specific one for
      the db2json script.
      Additionally, helper methods are provided to transform strings to Path
      instances, while ensuring that they are either an existing file or
      directory.
      
      tests/test_argparse.py:
      Add tests for all of `ArgParseFactory`.
      9bb539dd
  6. 08 Mar, 2021 13 commits
    • David Runge's avatar
      Add method to dump repo dbs to JSON files · 25b0c7ca
      David Runge authored
      repo_management/operations.py:
      Add `dump_db_to_json_files()` which allows for a repository database
      file to be read and all of its members to be serialized as JSON and
      written to files based upon the packages respective pkgbases.
      For the JSON serialization orjson is used, as it provides the highest
      speeds available and seems the most correct implementation.
      
      tests/test_operations.py:
      Add a fixture to create and destroy a temporary directory (used to write
      JSON files to).
      Add tests for `dump_db_to_json_files()`.
      25b0c7ca
    • David Runge's avatar
      Update lock file for orjson and mock · 7e3a448d
      David Runge authored
      poetry.lock:
      Update lock file for current versions of orjson and mock.
      7e3a448d
    • David Runge's avatar
      Add orjson and mock · fe9f7c1d
      David Runge authored
      pyproject.toml:
      Add orjson to the list of dependencies.
      Add mock to the list of development dependencies.
      fe9f7c1d
    • David Runge's avatar
      Make use of common helper for fixtures · e04209ae
      David Runge authored
      tests/test_files.py:
      Make use of common helper in `tests/fixtures.py` for creating repository
      database fixtures.
      e04209ae
    • David Runge's avatar
      Add iterator over repository database · 4e4d5bbe
      David Runge authored
      repo_management/operations.py:
      Add the public method `db_file_as_models()`, which yields all members of
      a repository database as a Tuple of the pkgbase name and the pkgbase
      data (represented as an instance of models.OutputPackageBase).
      
      tests/test_operations.py:
      Add a test for `db_file_as_models()`.
      4e4d5bbe
    • David Runge's avatar
      Add a fixture helper for tests · 53f6c29d
      David Runge authored
      tests/fixtures.py:
      Add a fixture helper for tests to easily create a valid repository
      database with a few dummy entries.
      53f6c29d
    • David Runge's avatar
      Add resources to create a dummy repository database · 538b11f3
      David Runge authored
      tests/resources/repo_db/*:
      Add a few dummy resources, representing actual package data to create a
      dummy repository database from for testing purposes.
      The resources include default and split packages.
      Note: These resources will (hopefully) become obsolete with a switch to
      python-pytest-pacman.
      538b11f3
    • David Runge's avatar
      Rename tests for files · 2ec219d6
      David Runge authored
      Rename the tests file used to test files.
      2ec219d6
    • David Runge's avatar
      Change repository member iterator to yield model · dc8d4b80
      David Runge authored
      repo_management/files.py:
      Add the private method `_extract_db_member_package_name()` which can be
      used to extract the package name from the name of a member of repository
      database.
      Rename `_read_db_file_member()` to `_db_file_member_as_model()` and
      change the method to yield a `models.RepoDbMemberData` instead of a
      `tarfile.TarInfo` to provide actual data (and required context) to
      callers of the method.
      
      tests/test_file.py:
      Add parametrized test for `_extract_db_member_package_name()`.
      Change test for `_db_file_member_as_model()` to assert the correct
      return type.
      dc8d4b80
    • David Runge's avatar
      Add conversion method for PackageDesc to OutputPackage · 45e96813
      David Runge authored
      repo_management/convert.py:
      Add private method `_transform_package_desc_to_output_package()` which
      can be used to construct a `models.OutputPackage` from a
      `models.PackageDesc` (and optionally a `models.PackageFiles`).
      The method drops attributes from a dict representation of the
      `models.PackageDesc` instance, that are not relevant for the
      `models.OutputPackage`.
      
      tests/test_convert.py:
      Add parametrized test for `_transform_package_desc_to_output_package()`.
      45e96813
    • David Runge's avatar
      Add models for describing output facing package data · f549d51a
      David Runge authored
      repo_management/models.py:
      Add model `PackageFiles` to track the list of files for a given package.
      Add model `RepoDbMemberType` to track the type of a repository database
      member (distinguished by the IntEnum defaults.RepoDbMemberType).
      Add model `RepoDbMemberData` to track the data (represented by an
      io.StringIO instance) of a repository database member.
      Add model `OutputPackage` to track the attributes required for a single
      package in the context of a representation within its pkgbase.
      Add model `OutputPackageBase` to track the attributes required for a
      pkgbase and its list of packages.
      f549d51a
    • David Runge's avatar
      Add IntEnum to distinguish members of a repository db · 98c227d1
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnum `RepoDbMemberType` to distinguish different members
      (files) in a repository database (e.g. 'desc' or 'files' files).
      98c227d1
    • David Runge's avatar
      Add iterator for TarFile members · 2660ca6c
      David Runge authored
      repo_management/files.py:
      Add iterator method `_read_db_file_member()` to yield the members of a
      provided tarfile.TarFile.
      The members are filtered by name using a regular expression, which can
      be overridden.
      
      tests/test_file.py:
      Change `create_db_file()` to also create 'desc' and 'files' files below
      the created temporary directories.
      Assert, that all tested methods return a value.
      Add test for `_read_db_file_member()`.
      2660ca6c
  7. 07 Mar, 2021 5 commits
    • David Runge's avatar
      Add method to read db file · 70bdbd76
      David Runge authored
      repo_management/files.py:
      Add `_read_db_file()` to open a db file and return it as
      tarfile.Tarfile. The method assumes gzip-compressed databases by
      default, but can be used with other compression algorithms.
      
      tests/test_file.py:
      Add fixtures and tests for `_read_db_file()`.
      70bdbd76
    • David Runge's avatar
      Add conversion of 'desc' files · 84e32326
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnmu `FieldType` which tracks the required type for an
      attribute when converting from 'desc' or 'files' files to pydantic
      models.
      Extend the DESC_JSON and FILES_JSON globals by introducing a dict per
      key, that describes the name and the required type, when converting to
      pydantic models.
      
      repo_management/convert.py:
      Rename `_files_data_to_dict()` to `_read_files_data_to_model()` and
      return a pydantic model (`models.Files`) instead of a custom dict.
      Add `_desc_data_to_model()` which allows for conversion of a 'desc' data
      file (represented as io.StringIO) to a pydantic model
      (`models.PackageDesc`).
      
      tests/test_convert.py:
      Add another test for raising RuntimeError in
      `_read_files_data_to_model()`.
      Add a parametrized test for `_desc_data_to_model()`.
      84e32326
    • David Runge's avatar
      Add pydantic models for 'desc' and 'files' files · 72cfd707
      David Runge authored
      repo_management/models.py:
      Add pydantic models for all headers in 'desc' and 'files' files (found
      in a repository database per pkgbase).
      The models describe the attribute names in use when reading data from
      the respective files and whether they are required or not.
      72cfd707
    • David Runge's avatar
      Update lock file to include pydantic · 59dfe2e8
      David Runge authored
      poetry.lock:
      Update the lock file to include the latest version of pydantic.
      59dfe2e8
    • David Runge's avatar
      Add pydantic to dependencies · 505ddd15
      David Runge authored
      pyproject.toml:
      Add pydantic to dependencies, as it will be used to validate
      configuration and internal typing.
      505ddd15
  8. 06 Mar, 2021 2 commits
    • David Runge's avatar
      Add gitlab-ci integration · c7227396
      David Runge authored
      .gitlab-ci.yml:
      Add Gitlab CI integration, that runs tox's linter and coverage targets
      in a lint stage.
      c7227396
    • David Runge's avatar
      Add initial tests for convert · d9105841
      David Runge authored
      tests/test_convert.py:
      Add initial tests for the convert module by adding a parametrized test
      for `_files_data_to_dict()`.
      d9105841