- 18 Mar, 2021 5 commits
-
-
David Runge authored
repo_management/operations.py: Add `create_db_from_json_files()` to create a repository database from JSON files in a directory. Change `db_file_as_models()` to rely on `models.PackageDesc.get_output_packages()` instead of `convert._transform_package_desc_to_output_package()` and add a missing parameter (base) to the instantiation of `models.OutputPackageBase()`. tests/test_operations.py: Add the two fixtures `dummy_json_files_in_dir()` and `empty_file()` to create a few (valid) dummy input JSON files in a temporary directory and an empty file (respectively). Add a test for `create_db_from_json_files()`.
-
David Runge authored
repo_management/files.py: Add `_json_files_in_directory()` yield a list of files with a .json suffix in a directory. Add `_read_pkgbase_json_file()` to read a JSON file representing a pkgbase and return an `OutputPackageBase` model. Add `_write_db_file()` to open a tar file for writing. Add `_stream_package_base_to_db()` to stream a `models.OutputPackage()` to a tar file (which represents a .db or a .files repository database). tests/test_files.py: Add tests for `_json_files_in_directory()`, `_read_pkgbase_json_file()`, `_write_db_file()` and `_stream_package_base_to_db()`.
-
David Runge authored
tests/fixtures.py: Add `create_empty_json_files()` which creates a set of empty .json files in a temporary directory and can be used by fixtures (which remove the temporary directory afterwards). Add `create_json_files()` which creates a set of valid JSON files in a temporary directory using the `models.OutputPackageBase` model.
-
David Runge authored
repo_management/convert.py: Add the class `RepoDbFile` which is used to initialize a jinja environment and allows to render the 'desc.j2' and 'files.j2' templates. Add `_desc_data_line_to_dicts()` which allows for assigning the data from a line in a 'desc' file to be assigned to a dict, that corresponds to the line's designated data type. Simplify `_desc_data_to_model()` by making use of `_desc_data_line_to_dicts()`. Change `_desc_data_to_model()` by guarding against pydantic ValidationErrors and raising a RepoManagementValidationError instead. tests/test_convert.py: Add tests for `RepoDbFile`.
-
David Runge authored
repo_management/defaults.py: Add the IntEnum `RepoDbType` to be able to identify/address the different types of binary database files (e.g. .db vs. .files). Change the %LICENSE% entry in `DESC_JSON` to not track the name in plural (use "license" instead of "licenses") to stay in line with the naming of the original variable. Add `DB_USER`, `DB_GROUP`, `DB_FILE_MODE`, `DB_DIR_MODE`, to provide defaults for which user and group to create files and directories with which file mode for, when generating a database file. repo_management/models.py: Change the `License` model to track an optional list of license identifiers using the singular term ("license" instead of "licenses") to have more predictable and coherent naming of variables and attributes. Add `Base` to `OutputPackageBase` as this way the model is more complete and it becomes much easier to use it, without having to also pass the pkgbase alongside it. Add the convenience method `get_packages_as_models()` to `OutputPackageBase` which returns the list of packages as tuples of `PackageDesc` and `Files`. Extend `PackageDesc` by the convenience method `get_output_package()` to easily convert an instance of `PackageDesc` and an optional instance of `Files` to an instance of `OutputPackage`. Move the declaration of `OutputPackage` above that of `PackageDesc` so that `PackageDesc.get_output_package()` can declare typing. tests/test_models.py Add tests for `PackageDesc.get_output_package()` and `OutputPackageBase.get_packages_as_models()`.
-
- 17 Mar, 2021 3 commits
-
-
David Runge authored
repo_management/errors.py: Add the Exception class `RepoManagementError` serving as a base, from which `RepoManagementFileError` is derived, which serves as a general Error class to signal problems with file interaction. Add the Exception class `RepoManagementFileNotFoundError`, which is derived from `RepoManagementFileError`. Add `RepoManagementValidationError` which derives from `RepoManagementError` and is raised on validation issues of file data.
-
David Runge authored
poetry.lock: Add the current version of jinja2 to the lockfile.
-
David Runge authored
pyproject.toml: Add jinja2 as repository dependency.
-
- 15 Mar, 2021 1 commit
-
-
David Runge authored
repo_management/templates/{desc,files}.j2: Add jinja2 templates for the 'desc' and 'files' files in a repository database (standard sync db or files sync db).
-
- 09 Mar, 2021 5 commits
-
-
David Runge authored
db2json/*: Remove obsolete db2json script and README. It has been superseded by the script definition in pyproject.toml.
-
David Runge authored
pyproject.toml: Add a script setup for dbj2son pointing to `repo_management.cli:db2json`. This can be used via `poetry run db2json`.
-
David Runge authored
repo_management/cli.py: Add `db2json()` which may serve as an entrypoint for a db2json script, as it makes use of the argument parser defined in `argparse.ArgParseFactory.db2json()` and calls `operations.dump_db_to_json_files()` using the provided inputs. tests/test_cli.py: Add a unittest for `db2json()`, which patches all called methods. The methods are tested extensively elsewhere. However, the function should see proper integration testing via this endpoint in the future.
-
David Runge authored
repo_management/operations.py: Remove the explicit Path checks from `dump_db_to_json_files()` as they are now generically implemented in argparse.ArgParseFactory. tests/test_operations.py Remove tests for Path checks in `dump_db_to_json_files()`.
-
David Runge authored
repo_management/argparse.py: Add `ArgParseFactory`, which serves as an argparse.ArgumentParser factory (new variants are implemented as classmethods). The class provides a generic constructor, that offers a default parser with a verbose argument and a description, alongside a specific one for the db2json script. Additionally, helper methods are provided to transform strings to Path instances, while ensuring that they are either an existing file or directory. tests/test_argparse.py: Add tests for all of `ArgParseFactory`.
-
- 08 Mar, 2021 13 commits
-
-
David Runge authored
repo_management/operations.py: Add `dump_db_to_json_files()` which allows for a repository database file to be read and all of its members to be serialized as JSON and written to files based upon the packages respective pkgbases. For the JSON serialization orjson is used, as it provides the highest speeds available and seems the most correct implementation. tests/test_operations.py: Add a fixture to create and destroy a temporary directory (used to write JSON files to). Add tests for `dump_db_to_json_files()`.
-
David Runge authored
poetry.lock: Update lock file for current versions of orjson and mock.
-
David Runge authored
pyproject.toml: Add orjson to the list of dependencies. Add mock to the list of development dependencies.
-
David Runge authored
tests/test_files.py: Make use of common helper in `tests/fixtures.py` for creating repository database fixtures.
-
David Runge authored
repo_management/operations.py: Add the public method `db_file_as_models()`, which yields all members of a repository database as a Tuple of the pkgbase name and the pkgbase data (represented as an instance of models.OutputPackageBase). tests/test_operations.py: Add a test for `db_file_as_models()`.
-
David Runge authored
tests/fixtures.py: Add a fixture helper for tests to easily create a valid repository database with a few dummy entries.
-
David Runge authored
tests/resources/repo_db/*: Add a few dummy resources, representing actual package data to create a dummy repository database from for testing purposes. The resources include default and split packages. Note: These resources will (hopefully) become obsolete with a switch to python-pytest-pacman.
-
David Runge authored
Rename the tests file used to test files.
-
David Runge authored
repo_management/files.py: Add the private method `_extract_db_member_package_name()` which can be used to extract the package name from the name of a member of repository database. Rename `_read_db_file_member()` to `_db_file_member_as_model()` and change the method to yield a `models.RepoDbMemberData` instead of a `tarfile.TarInfo` to provide actual data (and required context) to callers of the method. tests/test_file.py: Add parametrized test for `_extract_db_member_package_name()`. Change test for `_db_file_member_as_model()` to assert the correct return type.
-
David Runge authored
repo_management/convert.py: Add private method `_transform_package_desc_to_output_package()` which can be used to construct a `models.OutputPackage` from a `models.PackageDesc` (and optionally a `models.PackageFiles`). The method drops attributes from a dict representation of the `models.PackageDesc` instance, that are not relevant for the `models.OutputPackage`. tests/test_convert.py: Add parametrized test for `_transform_package_desc_to_output_package()`.
-
David Runge authored
repo_management/models.py: Add model `PackageFiles` to track the list of files for a given package. Add model `RepoDbMemberType` to track the type of a repository database member (distinguished by the IntEnum defaults.RepoDbMemberType). Add model `RepoDbMemberData` to track the data (represented by an io.StringIO instance) of a repository database member. Add model `OutputPackage` to track the attributes required for a single package in the context of a representation within its pkgbase. Add model `OutputPackageBase` to track the attributes required for a pkgbase and its list of packages.
-
David Runge authored
repo_management/defaults.py: Add the IntEnum `RepoDbMemberType` to distinguish different members (files) in a repository database (e.g. 'desc' or 'files' files).
-
David Runge authored
repo_management/files.py: Add iterator method `_read_db_file_member()` to yield the members of a provided tarfile.TarFile. The members are filtered by name using a regular expression, which can be overridden. tests/test_file.py: Change `create_db_file()` to also create 'desc' and 'files' files below the created temporary directories. Assert, that all tested methods return a value. Add test for `_read_db_file_member()`.
-
- 07 Mar, 2021 5 commits
-
-
David Runge authored
repo_management/files.py: Add `_read_db_file()` to open a db file and return it as tarfile.Tarfile. The method assumes gzip-compressed databases by default, but can be used with other compression algorithms. tests/test_file.py: Add fixtures and tests for `_read_db_file()`.
-
David Runge authored
repo_management/defaults.py: Add the IntEnmu `FieldType` which tracks the required type for an attribute when converting from 'desc' or 'files' files to pydantic models. Extend the DESC_JSON and FILES_JSON globals by introducing a dict per key, that describes the name and the required type, when converting to pydantic models. repo_management/convert.py: Rename `_files_data_to_dict()` to `_read_files_data_to_model()` and return a pydantic model (`models.Files`) instead of a custom dict. Add `_desc_data_to_model()` which allows for conversion of a 'desc' data file (represented as io.StringIO) to a pydantic model (`models.PackageDesc`). tests/test_convert.py: Add another test for raising RuntimeError in `_read_files_data_to_model()`. Add a parametrized test for `_desc_data_to_model()`.
-
David Runge authored
repo_management/models.py: Add pydantic models for all headers in 'desc' and 'files' files (found in a repository database per pkgbase). The models describe the attribute names in use when reading data from the respective files and whether they are required or not.
-
David Runge authored
poetry.lock: Update the lock file to include the latest version of pydantic.
-
David Runge authored
pyproject.toml: Add pydantic to dependencies, as it will be used to validate configuration and internal typing.
-
- 06 Mar, 2021 8 commits
-
-
David Runge authored
.gitlab-ci.yml: Add Gitlab CI integration, that runs tox's linter and coverage targets in a lint stage.
-
David Runge authored
tests/test_convert.py: Add initial tests for the convert module by adding a parametrized test for `_files_data_to_dict()`.
-
David Runge authored
repo_management/defaults.py: Add the basic defaults module, providing data that translates keywords found in the 'desc' and 'files' files of a binary repository database to those used in JSON files, representing the same data. repo_management/convert.py: Add the convert module, and introduce the simple private method (reimplementing a subset of the features from db2json) `_files_data_to_dict()`, which translates the contents of a `files` file to a typed dict.
-
David Runge authored
tox.ini: Add tox integration for testing (linting and coverage) in virtual environments using poetry.
-
David Runge authored
poetry.lock: Add lock file with all dependencies currently used to test the project.
-
David Runge authored
pyproject.toml: Add a pyproject.toml for describing the project, configuring all currently required dependencies and development dependencies. The configuration adds entries for the code formatter black, the import sorting tool isort and for the coverage report generation tool coverage.
-
David Runge authored
.gitignore: Add a ignore file for git, with many default entries for python projects related files, that should never end up in version control.
-
David Runge authored
.mypy.ini: As mypy can not yet be configured through pyproject.toml, add a specific configuration file for it.
-