1. 24 Mar, 2021 1 commit
    • David Runge's avatar
      Add first validators to models · 1a61c3d1
      David Runge authored
      repo_management/models.py:
      Add validators to the `BuildDate`, `CSize`, `ISize`, `Name` and
      `Version` models to ensure positive integers, correct names and
      versions.
      Add the comparator instance methods `Version.is_older_than()` and
      `Version.is_newer_than()` which rely on pyalpm's `vercmp()` to compare
      two version strings and return whether they are older or newer
      (respectively).
      Update the documentation for `ISize` to reflect, that it represents size
      of an installed package.
      
      tests/test_models.py:
      Add tests for added validators and comparator methods.
      1a61c3d1
  2. 22 Mar, 2021 3 commits
    • David Runge's avatar
      Use required license attribute in tests · 64bd6e25
      David Runge authored
      tests/test_{convert,files}.py:
      Ensure that instantiation of models using the `License` model are done
      correctly.
      64bd6e25
    • David Runge's avatar
      Add documentation to all models · 9fa1d0e0
      David Runge authored
      repo_management/models.py:
      Document all models and sort the single attribute models alphabetically
      for easier readibility.
      Change the `License` model to describe a required attribute instead of
      an optional (all packages need a license).
      Remove the unused model `PackageFiles`.
      9fa1d0e0
    • David Runge's avatar
      Merge branch 'issues/9' into 'master' · 3c443e22
      David Runge authored
      Add integration tests for db2json and json2db
      
      Closes #9
      
      See merge request !5
      3c443e22
  3. 20 Mar, 2021 8 commits
    • David Runge's avatar
      Add integration tests to gitlab CI · 9ba649d1
      David Runge authored
      .gitlab-ci.yml:
      Rename "lint" stage to "test" stage.
      Change basic target to also update the pacman files databases, as they
      are required for integration tests.
      Add integration target to run `tox -e integration`.
      9ba649d1
    • David Runge's avatar
      Add integration test target for tox · 441d7c00
      David Runge authored
      tox.ini:
      Add test target that only runs tests marked "integration".
      441d7c00
    • David Runge's avatar
      Extend tests for CLI and add integration tests · aaf40e02
      David Runge authored
      repo_management/cli.py:
      Change `db2json` and `json2db` to try/except
      `errors.RepoManagementError` and `argpase.ArgumentTypeError` and exit
      with a non-zero return code in that case.
      
      tests/test_cli.py:
      Change the tests for `db2json` and `json2db` to cover potentially
      raising `errors.RepoManagementError` and `argpase.ArgumentTypeError` and
      checking whether exiting with a non-zero return code in that case.
      Add parametrized integration test `test_transform_databases()` that
      transforms existing pacman .files sync databases to JSON files and back
      to .files repository databases again.
      Add parametrized integration
      `test_transform_databases_and_use_with_pacman()` to transform existing
      pacman .files sync databases to JSON files and create .files and .db
      repository databases from them. Afterwards pacman uses the created
      repository databases to search for packages with the "linux" keyword in
      them and list the files owned by the "linux" package using the created
      .files sync databases.
      aaf40e02
    • David Runge's avatar
      Add pytest configuration for markers · e47f74bd
      David Runge authored
      pyproject.toml:
      Add pytest configuration to setup the custom marker "integration".
      e47f74bd
    • David Runge's avatar
      Improve Exception handling in convert · 6e821be6
      David Runge authored
      repo_management/convert.py:
      Change the docuemntation of `_desc_data_line_to_dicts()` to reflect,
      that it may raise ValueError under certain conditions.
      Change `_desc_data_to_model()` to try/except ValueError raised by
      `_desc_data_line_to_dicts()` and raise a
      `errors.REpomanagementValidationError` instead.
      
      tests/test_convert.py:
      Change tests expecting to raise `ValueError` to expect
      `errors.RepoManagementValidationError` instead.
      6e821be6
    • David Runge's avatar
      Add commands module to run external commands · ac42d90a
      David Runge authored
      repo_management/commands.py:
      Add `run_command()` which allows for running an external command with
      the help of subprocess-tee.
      Add the private method `_print_env()` which is used to print the
      optional environment variables given to `run_command()`.
      
      tests/test_commands.py:
      Add tests for `_print_env()` and `run_command()`.
      ac42d90a
    • David Runge's avatar
      Update lock file for subprocess-tee · f588beec
      David Runge authored
      poetry.lock:
      Update lock file for subprocess-tee 0.2.0.
      f588beec
    • David Runge's avatar
      Add subprocess-tee to dependencies · 899cc204
      David Runge authored
      pyproject.toml:
      Add subprocess-tee to the list of dependencies for the project.
      899cc204
  4. 18 Mar, 2021 9 commits
    • David Runge's avatar
      Merge branch 'issues/4' into 'master' · bb999eab
      David Runge authored
      Add functionality to write a database file from a set of JSON files
      
      Closes #4
      
      See merge request !4
      bb999eab
    • David Runge's avatar
      Add script entrypoint for json2db · c51b92b9
      David Runge authored
      pyproject.toml:
      Add script entrypoint for `json2db`.
      c51b92b9
    • David Runge's avatar
      Add entrypoint for json2db · d542d1f4
      David Runge authored
      repo_management/cli.py:
      Add `json2db()` as entrypoint for the `json2db` script.
      
      tests/test_cli.py:
      Add a test for `json2db()`.
      d542d1f4
    • David Runge's avatar
      Add argparse for json2db · 0f02fe86
      David Runge authored
      repo_management/argparse.py:
      Add `ArgParseFactory.json2db()` as an ArgumentParser for the `json2db`
      script.
      Add `ArgParseFactory.string_to_writable_file_path()` for returning a
      Path object, that is guaranteed to be a writable file.
      
      tests/test_argparse.py:
      Add tests for
      `ArgParseFactory.{json2db,string_to_writable_file_path}()`.
      0f02fe86
    • David Runge's avatar
      Add operation to create a db from JSON files · d1fa043d
      David Runge authored
      repo_management/operations.py:
      Add `create_db_from_json_files()` to create a repository database from
      JSON files in a directory.
      Change `db_file_as_models()` to rely on
      `models.PackageDesc.get_output_packages()` instead of
      `convert._transform_package_desc_to_output_package()` and add a missing
      parameter (base) to the instantiation of `models.OutputPackageBase()`.
      
      tests/test_operations.py:
      Add the two fixtures `dummy_json_files_in_dir()` and `empty_file()` to
      create a few (valid) dummy input JSON files in a temporary directory and
      an empty file (respectively).
      Add a test for `create_db_from_json_files()`.
      d1fa043d
    • David Runge's avatar
      Add methods for reading and streaming JSON files · 3103d0f7
      David Runge authored
      repo_management/files.py:
      Add `_json_files_in_directory()` yield a list of files with a .json
      suffix in a directory.
      Add `_read_pkgbase_json_file()` to read a JSON file representing a
      pkgbase and return an `OutputPackageBase` model.
      Add `_write_db_file()` to open a tar file for writing.
      Add `_stream_package_base_to_db()` to stream a `models.OutputPackage()`
      to a tar file (which represents a .db or a .files repository database).
      
      tests/test_files.py:
      Add tests for `_json_files_in_directory()`, `_read_pkgbase_json_file()`,
      `_write_db_file()` and `_stream_package_base_to_db()`.
      3103d0f7
    • David Runge's avatar
      Add fixture helper for json files · 3b9c7b5a
      David Runge authored
      tests/fixtures.py:
      Add `create_empty_json_files()` which creates a set of empty .json files
      in a temporary directory and can be used by fixtures (which remove the
      temporary directory afterwards).
      Add `create_json_files()` which creates a set of valid JSON files in a
      temporary directory using the `models.OutputPackageBase` model.
      3b9c7b5a
    • David Runge's avatar
      Simplify convert and add class for template rendering · 9cff77d7
      David Runge authored
      repo_management/convert.py:
      Add the class `RepoDbFile` which is used to initialize a jinja
      environment and allows to render the 'desc.j2' and 'files.j2' templates.
      Add `_desc_data_line_to_dicts()` which allows for assigning the data
      from a line in a 'desc' file to be assigned to a dict, that corresponds
      to the line's designated data type.
      Simplify `_desc_data_to_model()` by making use of
      `_desc_data_line_to_dicts()`.
      Change `_desc_data_to_model()` by guarding against pydantic
      ValidationErrors and raising a RepoManagementValidationError instead.
      
      tests/test_convert.py:
      Add tests for `RepoDbFile`.
      9cff77d7
    • David Runge's avatar
      Correct and extend models and defaults · 5c491242
      David Runge authored
      repo_management/defaults.py:
      Add the IntEnum `RepoDbType` to be able to identify/address the
      different types of binary database files (e.g. .db vs. .files).
      Change the %LICENSE% entry in `DESC_JSON` to not track the name in
      plural (use "license" instead of "licenses") to stay in line with the
      naming of the original variable.
      Add `DB_USER`, `DB_GROUP`, `DB_FILE_MODE`, `DB_DIR_MODE`, to provide
      defaults for which user and group to create files and directories with
      which file mode for, when generating a database file.
      
      repo_management/models.py:
      Change the `License` model to track an optional list of license
      identifiers using the singular term ("license" instead of "licenses") to
      have more predictable and coherent naming of variables and attributes.
      Add `Base` to `OutputPackageBase` as this way the model is more complete
      and it becomes much easier to use it, without having to also pass the
      pkgbase alongside it.
      Add the convenience method `get_packages_as_models()` to
      `OutputPackageBase` which returns the list of packages as tuples of
      `PackageDesc` and `Files`.
      Extend `PackageDesc` by the convenience method `get_output_package()` to
      easily convert an instance of `PackageDesc` and an optional instance of
      `Files` to an instance of `OutputPackage`.
      Move the declaration of `OutputPackage` above that of `PackageDesc` so
      that `PackageDesc.get_output_package()` can declare typing.
      
      tests/test_models.py
      Add tests for `PackageDesc.get_output_package()` and
      `OutputPackageBase.get_packages_as_models()`.
      5c491242
  5. 17 Mar, 2021 3 commits
    • David Runge's avatar
      Add a set of error classes · 002b821e
      David Runge authored
      repo_management/errors.py:
      Add the Exception class `RepoManagementError` serving as a base, from
      which `RepoManagementFileError` is derived, which serves as a general
      Error class to signal problems with file interaction.
      Add the Exception class `RepoManagementFileNotFoundError`, which is
      derived from `RepoManagementFileError`.
      Add `RepoManagementValidationError` which derives from
      `RepoManagementError` and is raised on validation issues of file
      data.
      002b821e
    • David Runge's avatar
      Add jinja2 to lock file · 1dfe6f80
      David Runge authored
      poetry.lock:
      Add the current version of jinja2 to the lockfile.
      1dfe6f80
    • David Runge's avatar
      Add jinja2 as project dependency · 6ed416e1
      David Runge authored
      pyproject.toml:
      Add jinja2 as repository dependency.
      6ed416e1
  6. 15 Mar, 2021 1 commit
  7. 09 Mar, 2021 5 commits
    • David Runge's avatar
      Remove obsolete db2json script · cf332096
      David Runge authored
      db2json/*:
      Remove obsolete db2json script and README.
      It has been superseded by the script definition in pyproject.toml.
      cf332096
    • David Runge's avatar
      Add a project script setup for db2json · 35a2d7dc
      David Runge authored
      pyproject.toml:
      Add a script setup for dbj2son pointing to `repo_management.cli:db2json`.
      This can be used via `poetry run db2json`.
      35a2d7dc
    • David Runge's avatar
      Add CLI entrypoint for db2json · 8836d5c3
      David Runge authored
      repo_management/cli.py:
      Add `db2json()` which may serve as an entrypoint for a db2json script,
      as it makes use of the argument parser defined in
      `argparse.ArgParseFactory.db2json()` and calls
      `operations.dump_db_to_json_files()` using the provided inputs.
      
      tests/test_cli.py:
      Add a unittest for `db2json()`, which patches all called methods.
      The methods are tested extensively elsewhere. However, the function
      should see proper integration testing via this endpoint in the future.
      8836d5c3
    • David Runge's avatar
      Remove Path checking from operations · 6099e271
      David Runge authored
      repo_management/operations.py:
      Remove the explicit Path checks from `dump_db_to_json_files()` as they
      are now generically implemented in argparse.ArgParseFactory.
      
      tests/test_operations.py
      Remove tests for Path checks in `dump_db_to_json_files()`.
      6099e271
    • David Runge's avatar
      Add ArgumentParser factory · 9bb539dd
      David Runge authored
      repo_management/argparse.py:
      Add `ArgParseFactory`, which serves as an argparse.ArgumentParser
      factory (new variants are implemented as classmethods).
      The class provides a generic constructor, that offers a default parser
      with a verbose argument and a description, alongside a specific one for
      the db2json script.
      Additionally, helper methods are provided to transform strings to Path
      instances, while ensuring that they are either an existing file or
      directory.
      
      tests/test_argparse.py:
      Add tests for all of `ArgParseFactory`.
      9bb539dd
  8. 08 Mar, 2021 10 commits
    • David Runge's avatar
      Add method to dump repo dbs to JSON files · 25b0c7ca
      David Runge authored
      repo_management/operations.py:
      Add `dump_db_to_json_files()` which allows for a repository database
      file to be read and all of its members to be serialized as JSON and
      written to files based upon the packages respective pkgbases.
      For the JSON serialization orjson is used, as it provides the highest
      speeds available and seems the most correct implementation.
      
      tests/test_operations.py:
      Add a fixture to create and destroy a temporary directory (used to write
      JSON files to).
      Add tests for `dump_db_to_json_files()`.
      25b0c7ca
    • David Runge's avatar
      Update lock file for orjson and mock · 7e3a448d
      David Runge authored
      poetry.lock:
      Update lock file for current versions of orjson and mock.
      7e3a448d
    • David Runge's avatar
      Add orjson and mock · fe9f7c1d
      David Runge authored
      pyproject.toml:
      Add orjson to the list of dependencies.
      Add mock to the list of development dependencies.
      fe9f7c1d
    • David Runge's avatar
      Make use of common helper for fixtures · e04209ae
      David Runge authored
      tests/test_files.py:
      Make use of common helper in `tests/fixtures.py` for creating repository
      database fixtures.
      e04209ae
    • David Runge's avatar
      Add iterator over repository database · 4e4d5bbe
      David Runge authored
      repo_management/operations.py:
      Add the public method `db_file_as_models()`, which yields all members of
      a repository database as a Tuple of the pkgbase name and the pkgbase
      data (represented as an instance of models.OutputPackageBase).
      
      tests/test_operations.py:
      Add a test for `db_file_as_models()`.
      4e4d5bbe
    • David Runge's avatar
      Add a fixture helper for tests · 53f6c29d
      David Runge authored
      tests/fixtures.py:
      Add a fixture helper for tests to easily create a valid repository
      database with a few dummy entries.
      53f6c29d
    • David Runge's avatar
      Add resources to create a dummy repository database · 538b11f3
      David Runge authored
      tests/resources/repo_db/*:
      Add a few dummy resources, representing actual package data to create a
      dummy repository database from for testing purposes.
      The resources include default and split packages.
      Note: These resources will (hopefully) become obsolete with a switch to
      python-pytest-pacman.
      538b11f3
    • David Runge's avatar
      Rename tests for files · 2ec219d6
      David Runge authored
      Rename the tests file used to test files.
      2ec219d6
    • David Runge's avatar
      Change repository member iterator to yield model · dc8d4b80
      David Runge authored
      repo_management/files.py:
      Add the private method `_extract_db_member_package_name()` which can be
      used to extract the package name from the name of a member of repository
      database.
      Rename `_read_db_file_member()` to `_db_file_member_as_model()` and
      change the method to yield a `models.RepoDbMemberData` instead of a
      `tarfile.TarInfo` to provide actual data (and required context) to
      callers of the method.
      
      tests/test_file.py:
      Add parametrized test for `_extract_db_member_package_name()`.
      Change test for `_db_file_member_as_model()` to assert the correct
      return type.
      dc8d4b80
    • David Runge's avatar
      Add conversion method for PackageDesc to OutputPackage · 45e96813
      David Runge authored
      repo_management/convert.py:
      Add private method `_transform_package_desc_to_output_package()` which
      can be used to construct a `models.OutputPackage` from a
      `models.PackageDesc` (and optionally a `models.PackageFiles`).
      The method drops attributes from a dict representation of the
      `models.PackageDesc` instance, that are not relevant for the
      `models.OutputPackage`.
      
      tests/test_convert.py:
      Add parametrized test for `_transform_package_desc_to_output_package()`.
      45e96813