1. 09 Apr, 2021 2 commits
  2. 07 Apr, 2021 3 commits
    • David Runge's avatar
      Change repo_management to be async · 6f8cf758
      David Runge authored
      Remove the obsolete
      `convert._transform_package_desc_to_output_package()` (the functionality
      is now covered by pydantic models directly).
      Change all relevant methods to be async.
      Change file open commands to make use of aiofiles.
      Remove tests for the obsolete
      Change all tests for async methods to test using pytest-asyncio.
    • David Runge's avatar
      Update lock for aiofiles and pytest-asyncio · ec78c4b3
      David Runge authored
      Update lock for aiofiles and pytest-asyncio.
    • David Runge's avatar
      Add aiofiles and pytest-asyncio · 7f7daf5e
      David Runge authored
      Add aiofiles to the project's dependencies.
      Add pytest-asyncio to the project's development dependencies.
  3. 28 Mar, 2021 2 commits
  4. 24 Mar, 2021 2 commits
    • David Runge's avatar
      Merge branch 'issues/6' into 'master' · d5cdad27
      David Runge authored
      Add documentation and validators to models
      Closes #6
      See merge request archlinux/arch-repo-management!6
    • David Runge's avatar
      Add first validators to models · 1a61c3d1
      David Runge authored
      Add validators to the `BuildDate`, `CSize`, `ISize`, `Name` and
      `Version` models to ensure positive integers, correct names and
      Add the comparator instance methods `Version.is_older_than()` and
      `Version.is_newer_than()` which rely on pyalpm's `vercmp()` to compare
      two version strings and return whether they are older or newer
      Update the documentation for `ISize` to reflect, that it represents size
      of an installed package.
      Add tests for added validators and comparator methods.
  5. 22 Mar, 2021 3 commits
    • David Runge's avatar
      Use required license attribute in tests · 64bd6e25
      David Runge authored
      Ensure that instantiation of models using the `License` model are done
    • David Runge's avatar
      Add documentation to all models · 9fa1d0e0
      David Runge authored
      Document all models and sort the single attribute models alphabetically
      for easier readibility.
      Change the `License` model to describe a required attribute instead of
      an optional (all packages need a license).
      Remove the unused model `PackageFiles`.
    • David Runge's avatar
      Merge branch 'issues/9' into 'master' · 3c443e22
      David Runge authored
      Add integration tests for db2json and json2db
      Closes #9
      See merge request archlinux/arch-repo-management!5
  6. 20 Mar, 2021 8 commits
    • David Runge's avatar
      Add integration tests to gitlab CI · 9ba649d1
      David Runge authored
      Rename "lint" stage to "test" stage.
      Change basic target to also update the pacman files databases, as they
      are required for integration tests.
      Add integration target to run `tox -e integration`.
    • David Runge's avatar
      Add integration test target for tox · 441d7c00
      David Runge authored
      Add test target that only runs tests marked "integration".
    • David Runge's avatar
      Extend tests for CLI and add integration tests · aaf40e02
      David Runge authored
      Change `db2json` and `json2db` to try/except
      `errors.RepoManagementError` and `argpase.ArgumentTypeError` and exit
      with a non-zero return code in that case.
      Change the tests for `db2json` and `json2db` to cover potentially
      raising `errors.RepoManagementError` and `argpase.ArgumentTypeError` and
      checking whether exiting with a non-zero return code in that case.
      Add parametrized integration test `test_transform_databases()` that
      transforms existing pacman .files sync databases to JSON files and back
      to .files repository databases again.
      Add parametrized integration
      `test_transform_databases_and_use_with_pacman()` to transform existing
      pacman .files sync databases to JSON files and create .files and .db
      repository databases from them. Afterwards pacman uses the created
      repository databases to search for packages with the "linux" keyword in
      them and list the files owned by the "linux" package using the created
      .files sync databases.
    • David Runge's avatar
      Add pytest configuration for markers · e47f74bd
      David Runge authored
      Add pytest configuration to setup the custom marker "integration".
    • David Runge's avatar
      Improve Exception handling in convert · 6e821be6
      David Runge authored
      Change the docuemntation of `_desc_data_line_to_dicts()` to reflect,
      that it may raise ValueError under certain conditions.
      Change `_desc_data_to_model()` to try/except ValueError raised by
      `_desc_data_line_to_dicts()` and raise a
      `errors.REpomanagementValidationError` instead.
      Change tests expecting to raise `ValueError` to expect
      `errors.RepoManagementValidationError` instead.
    • David Runge's avatar
      Add commands module to run external commands · ac42d90a
      David Runge authored
      Add `run_command()` which allows for running an external command with
      the help of subprocess-tee.
      Add the private method `_print_env()` which is used to print the
      optional environment variables given to `run_command()`.
      Add tests for `_print_env()` and `run_command()`.
    • David Runge's avatar
      Update lock file for subprocess-tee · f588beec
      David Runge authored
      Update lock file for subprocess-tee 0.2.0.
    • David Runge's avatar
      Add subprocess-tee to dependencies · 899cc204
      David Runge authored
      Add subprocess-tee to the list of dependencies for the project.
  7. 18 Mar, 2021 9 commits
    • David Runge's avatar
      Merge branch 'issues/4' into 'master' · bb999eab
      David Runge authored
      Add functionality to write a database file from a set of JSON files
      Closes #4
      See merge request archlinux/arch-repo-management!4
    • David Runge's avatar
      Add script entrypoint for json2db · c51b92b9
      David Runge authored
      Add script entrypoint for `json2db`.
    • David Runge's avatar
      Add entrypoint for json2db · d542d1f4
      David Runge authored
      Add `json2db()` as entrypoint for the `json2db` script.
      Add a test for `json2db()`.
    • David Runge's avatar
      Add argparse for json2db · 0f02fe86
      David Runge authored
      Add `ArgParseFactory.json2db()` as an ArgumentParser for the `json2db`
      Add `ArgParseFactory.string_to_writable_file_path()` for returning a
      Path object, that is guaranteed to be a writable file.
      Add tests for
    • David Runge's avatar
      Add operation to create a db from JSON files · d1fa043d
      David Runge authored
      Add `create_db_from_json_files()` to create a repository database from
      JSON files in a directory.
      Change `db_file_as_models()` to rely on
      `models.PackageDesc.get_output_packages()` instead of
      `convert._transform_package_desc_to_output_package()` and add a missing
      parameter (base) to the instantiation of `models.OutputPackageBase()`.
      Add the two fixtures `dummy_json_files_in_dir()` and `empty_file()` to
      create a few (valid) dummy input JSON files in a temporary directory and
      an empty file (respectively).
      Add a test for `create_db_from_json_files()`.
    • David Runge's avatar
      Add methods for reading and streaming JSON files · 3103d0f7
      David Runge authored
      Add `_json_files_in_directory()` yield a list of files with a .json
      suffix in a directory.
      Add `_read_pkgbase_json_file()` to read a JSON file representing a
      pkgbase and return an `OutputPackageBase` model.
      Add `_write_db_file()` to open a tar file for writing.
      Add `_stream_package_base_to_db()` to stream a `models.OutputPackage()`
      to a tar file (which represents a .db or a .files repository database).
      Add tests for `_json_files_in_directory()`, `_read_pkgbase_json_file()`,
      `_write_db_file()` and `_stream_package_base_to_db()`.
    • David Runge's avatar
      Add fixture helper for json files · 3b9c7b5a
      David Runge authored
      Add `create_empty_json_files()` which creates a set of empty .json files
      in a temporary directory and can be used by fixtures (which remove the
      temporary directory afterwards).
      Add `create_json_files()` which creates a set of valid JSON files in a
      temporary directory using the `models.OutputPackageBase` model.
    • David Runge's avatar
      Simplify convert and add class for template rendering · 9cff77d7
      David Runge authored
      Add the class `RepoDbFile` which is used to initialize a jinja
      environment and allows to render the 'desc.j2' and 'files.j2' templates.
      Add `_desc_data_line_to_dicts()` which allows for assigning the data
      from a line in a 'desc' file to be assigned to a dict, that corresponds
      to the line's designated data type.
      Simplify `_desc_data_to_model()` by making use of
      Change `_desc_data_to_model()` by guarding against pydantic
      ValidationErrors and raising a RepoManagementValidationError instead.
      Add tests for `RepoDbFile`.
    • David Runge's avatar
      Correct and extend models and defaults · 5c491242
      David Runge authored
      Add the IntEnum `RepoDbType` to be able to identify/address the
      different types of binary database files (e.g. .db vs. .files).
      Change the %LICENSE% entry in `DESC_JSON` to not track the name in
      plural (use "license" instead of "licenses") to stay in line with the
      naming of the original variable.
      Add `DB_USER`, `DB_GROUP`, `DB_FILE_MODE`, `DB_DIR_MODE`, to provide
      defaults for which user and group to create files and directories with
      which file mode for, when generating a database file.
      Change the `License` model to track an optional list of license
      identifiers using the singular term ("license" instead of "licenses") to
      have more predictable and coherent naming of variables and attributes.
      Add `Base` to `OutputPackageBase` as this way the model is more complete
      and it becomes much easier to use it, without having to also pass the
      pkgbase alongside it.
      Add the convenience method `get_packages_as_models()` to
      `OutputPackageBase` which returns the list of packages as tuples of
      `PackageDesc` and `Files`.
      Extend `PackageDesc` by the convenience method `get_output_package()` to
      easily convert an instance of `PackageDesc` and an optional instance of
      `Files` to an instance of `OutputPackage`.
      Move the declaration of `OutputPackage` above that of `PackageDesc` so
      that `PackageDesc.get_output_package()` can declare typing.
      Add tests for `PackageDesc.get_output_package()` and
  8. 17 Mar, 2021 3 commits
    • David Runge's avatar
      Add a set of error classes · 002b821e
      David Runge authored
      Add the Exception class `RepoManagementError` serving as a base, from
      which `RepoManagementFileError` is derived, which serves as a general
      Error class to signal problems with file interaction.
      Add the Exception class `RepoManagementFileNotFoundError`, which is
      derived from `RepoManagementFileError`.
      Add `RepoManagementValidationError` which derives from
      `RepoManagementError` and is raised on validation issues of file
    • David Runge's avatar
      Add jinja2 to lock file · 1dfe6f80
      David Runge authored
      Add the current version of jinja2 to the lockfile.
    • David Runge's avatar
      Add jinja2 as project dependency · 6ed416e1
      David Runge authored
      Add jinja2 as repository dependency.
  9. 15 Mar, 2021 1 commit
  10. 09 Mar, 2021 5 commits
    • David Runge's avatar
      Remove obsolete db2json script · cf332096
      David Runge authored
      Remove obsolete db2json script and README.
      It has been superseded by the script definition in pyproject.toml.
    • David Runge's avatar
      Add a project script setup for db2json · 35a2d7dc
      David Runge authored
      Add a script setup for dbj2son pointing to `repo_management.cli:db2json`.
      This can be used via `poetry run db2json`.
    • David Runge's avatar
      Add CLI entrypoint for db2json · 8836d5c3
      David Runge authored
      Add `db2json()` which may serve as an entrypoint for a db2json script,
      as it makes use of the argument parser defined in
      `argparse.ArgParseFactory.db2json()` and calls
      `operations.dump_db_to_json_files()` using the provided inputs.
      Add a unittest for `db2json()`, which patches all called methods.
      The methods are tested extensively elsewhere. However, the function
      should see proper integration testing via this endpoint in the future.
    • David Runge's avatar
      Remove Path checking from operations · 6099e271
      David Runge authored
      Remove the explicit Path checks from `dump_db_to_json_files()` as they
      are now generically implemented in argparse.ArgParseFactory.
      Remove tests for Path checks in `dump_db_to_json_files()`.
    • David Runge's avatar
      Add ArgumentParser factory · 9bb539dd
      David Runge authored
      Add `ArgParseFactory`, which serves as an argparse.ArgumentParser
      factory (new variants are implemented as classmethods).
      The class provides a generic constructor, that offers a default parser
      with a verbose argument and a description, alongside a specific one for
      the db2json script.
      Additionally, helper methods are provided to transform strings to Path
      instances, while ensuring that they are either an existing file or
      Add tests for all of `ArgParseFactory`.
  11. 08 Mar, 2021 2 commits
    • David Runge's avatar
      Add method to dump repo dbs to JSON files · 25b0c7ca
      David Runge authored
      Add `dump_db_to_json_files()` which allows for a repository database
      file to be read and all of its members to be serialized as JSON and
      written to files based upon the packages respective pkgbases.
      For the JSON serialization orjson is used, as it provides the highest
      speeds available and seems the most correct implementation.
      Add a fixture to create and destroy a temporary directory (used to write
      JSON files to).
      Add tests for `dump_db_to_json_files()`.
    • David Runge's avatar
      Update lock file for orjson and mock · 7e3a448d
      David Runge authored
      Update lock file for current versions of orjson and mock.