1. 30 Jun, 2022 17 commits
    • David Runge's avatar
      Update documentation for repod-file · 9e10c31b
      David Runge authored
      docs/repod/installation.rst:
      Mention `repod-file` instead of `db2json`/ `json2db` as the latter are
      removed.
      
      docs/repod/tooling.rst:
      Replace the documentation for `db2json`/ `json2db` with new
      documentation on the various subcommands of `repod-file`.
      9e10c31b
    • David Runge's avatar
      Remove db2json and json2db scripts · 80fae6eb
      David Runge authored
      Remove the `db2json` and `json2db` scripts as they have been superseded
      by `repod-file`.
      80fae6eb
    • David Runge's avatar
      Replace db2json and json2db with repod-file in integration tests · 3531ef3b
      David Runge authored
      tests/cli/test_cli.py:
      Call repod-file instead of db2json and json2db in integration tests.
      3531ef3b
    • David Runge's avatar
      Export schema for docs using repod-file · 1599da20
      David Runge authored
      tox.ini:
      Use a call to `repod-file schema export` to export the JSON schemas in
      one command.
      1599da20
    • David Runge's avatar
      Add repod-file script · fb10b174
      David Runge authored
      repod/cli/argparse.py:
      Add `ArgParseFactory.repod_file()` to provide an ArgumentParser for the
      `repod-file` script which provides various functionalities (package,
      management repository, repository sync database and schemas).
      Add default argument for indicating the need for debug output.
      
      repod/cli/cli.py:
      Add the `repod_file()`, `repod_file_management()`,
      `repod_file_package()`, `repod_file_syncdb()` and `repod_file_schema()`
      functions to provide the calls for functionalities exposed by the
      `repod-file` script.
      
      tests/cli/test_argparse.py:
      Add tests for `ArgParseFactory.repod_file()`.
      
      tests/cli/test_cli.py:
      Add tests for `repod_file()`, `repod_file_management()`,
      `repod_file_package()`, `repod_file_syncdb()` and `repod_file_schema()`.
      
      repod/cli/__init__.py:
      Expose `repod.cli.cli.repod_file`.
      
      pyproject.toml:
      Add script repod-file pointing at `repod.cli.cli.repod_file`.
      fb10b174
    • David Runge's avatar
      Add default for repository sync database compression · 8d46f574
      David Runge authored
      repod/config/defaults.py:
      Add `DEFAULT_DATABASE_COMPRESSION` to provide a default
      `CompressionTypeEnum` member.
      8d46f574
    • David Runge's avatar
      Lower info to debug message for Package · ee4093c4
      David Runge authored
      repod/files/package.py:
      Lower info call to debug call when logging for checksum creation.
      ee4093c4
    • David Runge's avatar
      Expose exporting of JSON schema in all packages · d9b3a7f5
      David Runge authored
      repod/__init__.py:
      Add a top-level `export_schemas()` function to export the JSON schemas
      of all repod packages.
      
      repod/files/__init__.py:
      Add a top-level `export_schemas()` function to export the JSON schemas
      of `repod.files`.
      
      repod/repo/__init__.py:
      Add a top-level `export_schemas()` function to export the JSON schemas
      of `repod.repo`.
      
      repod/repo/management/__init__.py:
      Expose `repo.repo.management.export_schemas`.
      
      repod/repo/package/__init__.py:
      Expose `repo.repo.package.export_schemas`.
      
      tests/*:
      Add tests for all `export_schemas()` calls.
      d9b3a7f5
    • David Runge's avatar
      Convenience classmethods for CompressionTypeEnum · cf9cdb06
      David Runge authored
      repod/common/enums.py:
      Add `CompressionTypeEnum.from_string()` to return a CompressionTypeEnum
      member by providing a string.
      Add `CompressionTypeEnum.as_db_file_suffixes()` to return a list of
      strings representing valid default repository sync database suffixes.
      Add `CompressionTypeEnum.as_files_file_suffixes()` to return a list of
      strings representing valid files repository sync database suffixes.
      
      tests/common/test_enums.py:
      Add tests for `CompressionTypeEnum.from_string()`,
      `CompressionTypeEnum.as_db_file_suffixes()` and
      `CompressionTypeEnum.as_files_file_suffixes()`.
      cf9cdb06
    • David Runge's avatar
      Return List instead of AsyncIterator with SyncDatabase.outputpackagebases() · 08e430e6
      David Runge authored
      repod/repo/package/syncdb.py:
      Return a `List` instead of an `AsyncIterator` over the names and
      OutputPackageBase tuples, as that is easier to use with `asyncio.run()`.
      
      repod/operations.py:
      Adapt calls to `SyncDatabase.outputpackagebases()` to expect a List and
      not an AsyncIterator.
      
      tests/repo/package/test_syncdb.py:
      Adapt tests for `SyncDatabase.outputpackagebases()` to reflect the
      changed return type.
      08e430e6
    • David Runge's avatar
      Set correct imports for repod.cli files · faaa5e5d
      David Runge authored
      repod/cli/__init__.py:
      Export repod.cli.cli.db2json and repod.cli.cli.json2db.
      
      repod/cli/argparse.py:
      Change the argparse imports for ArgumentParser and ArgumentTypeError to
      be top-level, so that they may be reused more easily when importing from
      repod.cli.argparse.
      
      repod/cli/cli.py:
      Import ArgumentTypeError from repod.cli.argparse instead of argparse to
      reduce direct imports.
      
      tests/cli/test_argparse.py:
      Use ArgumentTypeError imports from repod.cli.argparse instead of
      argparse to reduce direct imports.
      Adapt patch calls to use the correct repod module locations.
      
      tests/cli/test_cli.py:
      Adapt patch calls to use the correct repod module locations.
      faaa5e5d
    • David Runge's avatar
      Move CLI related files to repod.cli module · d5341714
      David Runge authored
      Move repod/{argparse,cli}.py to repod.cli.
      Move tests/test_{argparse,cli}.py to tests.cli.
      d5341714
    • David Runge's avatar
      Update documentation link in pyproject.toml · d681435b
      David Runge authored
      pyproject.toml:
      Change the project's documentation link to point at
      https://archlinux.page.
      d681435b
    • David Runge's avatar
      Fix documentation link and make it more visible · 3a9d46e9
      David Runge authored
      README.md:
      Fix the markdown link for the published documentation.
      Make note about the project being alpha grade software more clear and
      visible.
      3a9d46e9
    • David Runge's avatar
      Add note about changelog to contribution guidelines · f294f397
      David Runge authored
      CONTRIBUTING.md:
      Add information on how and where to add changelog entries.
      f294f397
    • David Runge's avatar
      Add symlink for changelog file · 4dcc7631
      David Runge authored
      Expose docs/repod/changelog.rst as CHANGELOG.rst.
      4dcc7631
    • David Runge's avatar
      Add changelog for unreleased version · 7e55a6e3
      David Runge authored
      docs/repod/changelog.rst:
      Add a changelog based on https://keepachangelog.com which tracks the
      added features of the first released (currently unreleased).
      
      docs/repod/changelog.rst:
      Integrate changelog in the "Using repod" section.
      7e55a6e3
  2. 26 Jun, 2022 7 commits
    • David Runge's avatar
      Extend SyncDatabase to extract OutputPackageBases · 35753401
      David Runge authored
      repod/repo/package/syncdb.py:
      Remove the obsolete `RepoDbMemberType` and `RepoDbMemberData`.
      Add instance method `SyncDatabase.outputpackagebases()` to be able to
      return the names of pkgbases and their accompanying OutputPackageBase
      instances.
      
      tests/repo/package/test_syncdb.py:
      Add test for `SyncDatabase.outputpackagebases()`.
      
      repod/files/__init__.py:
      Remove `_extract_db_member_package_name()` and
      `_db_file_member_as_model()` as they are superseded by
      `SyncDatabase.outputpackagebases()`.
      Remove obsolete imports.
      Add imports for `BuildInfo`, `MTree`, `MTreeEntry` and `PkgInfo`.
      
      repod/operations.py:
      Remove `db_file_as_models()` as it is superseded by
      `SyncDatabase.outputpackagebases()`.
      Replace the use of `db_file_as_models()` with
      `SyncDatabase.outputpackagebases()`.
      
      repod/repo/__init__.py, repod/repo/package/__init__.py:
      Remove import of removed classes.
      
      tests/test_files.py:
      Remove tests for removed file.
      
      tests/test_operations.py:
      Remove test for removed function.
      35753401
    • David Runge's avatar
      Add SyncDatabase model · 77503692
      David Runge authored
      repod/repo/package/syncdb.py:
      Add `DB_USER`, `DB_GROUP`, `DB_FILE_MODE` and `DB_DIR_MODE` as they are
      too specific for repod.config.defaults.
      Add `SyncDatabase` model that exposes instance and class methods for
      writing JSON files and `OutputPackageBase` instances to repository sync
      databases.
      
      repod/repo/package/__init__.py:
      Expose `SyncDatabase`.
      
      repod/config/defaults.py:
      Remove obsolete `DB_USER`, `DB_GROUP`, `DB_FILE_MODE` and `DB_DIR_MODE`
      defaults.
      
      repod/files/__init__.py:
      Remove the obsolete `_json_files_in_directory()`, `_write_db_file()` and
      `_stream_package_base_to_db()` functions.
      
      repod/operations.py:
      Replace use of `_json_files_in_directory()`, `_write_db_file()`,
      `_stream_package_base_to_db()` and `open_tarfile()` with `SyncDatabase`
      and its instance methods.
      
      tests/conftest.py:
      Replace use of `_json_files_in_directory()`, `_write_db_file()` and
      `_stream_package_base_to_db()` with `SyncDatabase` and its instance
      methods.
      
      tests/repo/package/test_syncdb.py:
      Add tests for `SyncDatabase.outputpackagebase_to_tarfile()`,
      `SyncDatabase.add()` and `SyncDatabase.stream_management_repo()`.
      
      tests/test_files.py:
      Remove tests for removed functions.
      77503692
    • David Runge's avatar
      Initialize OutputPackageBase from JSON file · 61c582c9
      David Runge authored
      repod/repo/management/outputpackage.py:
      Add the `OutputPackageBase.from_file()` class method to initialize an
      instance from a JSON file.
      
      tests/repo/management/test_outputpackage.py:
      Add tests for `OutputPackageBase.from_file()`.
      
      repod/operations.py:
      Replace the use of `_read_pkgbase_json_file()` with
      `OutputPackageBase.from_file()`.
      
      repod/files/__init__.py:
      Remove the obsolete `_read_pkgbase_json_file()`.
      
      tests/test_files.py:
      Remove tests for obsolete `_read_pkgbase_json_file()`.
      61c582c9
    • David Runge's avatar
      PackageDesc and Files initialization from stream · 7d4077b9
      David Runge authored
      repod/repo/package/syncdb.py:
      Add the `Files.from_stream()` and `PackageDesc.from_stream()` class
      methods to allow creating Files and PackageDesc instances from an I/O
      stream.
      
      tests/repo/package/test_syncdb.py:
      Add tests for `Files.from_stream()` and `PackageDesc.from_stream()`.
      
      repod/operations.py:
      Change `db_file_as_models()` to use `Files.from_stream()` and
      `PackageDesc.from_stream()` instead of `convert.file_data_to_model()`.
      
      repod/convert.py, tests/test_convert.py:
      Remove obsolete files.
      7d4077b9
    • David Runge's avatar
      Move desc and files rendering to repod.repo.package.syncdb · 1872540e
      David Runge authored
      repod/repo/package/syncdb.py:
      Add `render()` methods to `PackageDesc` and `Files` to allow instances
      of them to render their contents to an output stream using a jinja
      template.
      
      tests/repo/package/test_syncdb.py:
      Add tests for `PackageDesc.render()` and `Files.render()`.
      
      repod/convert.py:
      Remove obsolete `RepoDbFile` class.
      
      tests/test_convert.py:
      Remove `RepoDbFile` related tests.
      
      repod/files/__init__.py, tests/conftest.py, tests/test_files.py:
      Remove the use of `RepoDbFile`.
      
      repod/operations.py:
      Replace use of `RepoDbFile` rendering methods with `PackageDesc` and
      `Files` instance `render()` methods.
      1872540e
    • David Runge's avatar
      Merge remote-tracking branch 'klausenbusk/pages' · 646397af
      David Runge authored
      * klausenbusk/pages:
        Publish documentation to our own documentation domain
      646397af
    • Kristian Klausen's avatar
      Publish documentation to our own documentation domain · ceb1310d
      Kristian Klausen authored
      We finally got our own domain (archlinux.page)[1] for project
      documentation, so let's publish the documentation there.
      
      [1] infrastructure!594
      ceb1310d
  3. 24 Jun, 2022 1 commit
    • David Runge's avatar
      Add conversion from Package to OutputPackageBase and OutputPackage · c92733ce
      David Runge authored
      repod/files/__init__.py:
      Do not import top-level from repod.repo.management and repod.convert to
      circumvent circular import.
      
      repod/files/package.py:
      Change `PackageV1` to also derive from the common models `CSize`,
      `FileName`, `Md5Sum`, `PgpSig` and `Sha256Sum`, as these fields are
      required in the context of the management repository.
      Change `Package.from_file()` to accept a signature file as second
      argument and to also populate the `csize`, `filename`, `md5sum`,
      `pgpsig` and `sha256sum` fields.
      
      repod/repo/management/outputpackage.py:
      Add `OUTPUT_PACKAGE_VERSIONS` to track required and optional fields for
      the derivates of OutputPackage.
      Add the `OutputPackage.from_package()` classmethod to create a derivate
      of OutputPackage from a `Package`.
      Add the `OutputPackageBase.from_package()` classmethod to create a
      derivate of OutputPackage from a list of `Package` instances.
      
      tests/conftest.py:
      Add session scoped `BuildinfoV1`, `BuildInfoV2`, `PkgInfoV1`,
      `PkgInfoV2` and `PackageV1` fixtures.
      Change `default_package_file` fixture to also return a dummy signature
      file alongside the package file.
      
      tests/files/test_common.py:
      Adapt tests to changed `default_package_file` fixture.
      
      tests/files/test_package.py:
      Adapt tests to changes in `PackageV1` and `Package.from_file()`
      behavior.
      
      tests/repo/management/test_outputpackage.py:
      Add tests for `OutputPackage.from_package()` and
      `OutputPackageBase.from_package()`.
      c92733ce
  4. 21 Jun, 2022 1 commit
    • David Runge's avatar
      Make PGP signatures optional in desc files · 131ea02c
      David Runge authored
      repod/common/models.py:
      Change pgpgsig to Optional[str], as desc files may be created without
      the %PGPSIG% field.
      
      repod/repo/package/syncdb.py:
      Move pgpsig in PACKAGE_DESC_VERSIONS from required to optional.
      
      repod/templates/desc_v1.j2:
      Change template to only create and populate the %PGPSIG% field if pgpsig
      is not None.
      
      tests/repo/package/test_syncdb.py:
      Add test case to test_package_desc_from_dict_derive_file_versions() to
      test the scenario in which no pgpsig is provided.
      
      tests/test_convert.py:
      Change `test_file_data_to_model()` to include a test case in which only
      the required fields are present (and no pgpsig).
      Change `test_repodbfile_render_desc_template()` to also include a case
      in which pgpsig is not used.
      131ea02c
  5. 19 Jun, 2022 2 commits
    • David Runge's avatar
      Split repository models by functionality · 4463e7fb
      David Runge authored
      repod/repo/package/syncdb.py:
      Add `FILES_VERSIONS`, `DEFAULT_FILES_VERSION`,
      `DEFAULT_PACKAGE_DESC_VERSION`, `PACKAGE_DESC_VERSIONS`, `PackageDesc`,
      `PackageDescV1`, `Files` and `FilesV1`.
      Import from `repod.repo.management.outputpackage` instead of importing
      its models top-level to circumenvent circular import.
      
      repod/repo/management/outputpackage.py:
      Remove `FILES_VERSIONS`, `DEFAULT_FILES_VERSION`,
      `DEFAULT_PACKAGE_DESC_VERSION`, `PACKAGE_DESC_VERSIONS`, `PackageDesc`,
      `PackageDescV1`, `Files` and `FilesV1`.
      
      repod/cli.py:
      Import RepoDbTypeEnum top-level.
      
      repod/convert.py:
      Import functions and classes from `repod.repo.package` top-level.
      
      repod/files/__init__.py:
      Import functions and classes from `repod.repo.package` and
      `repod.repo.management` top-level.
      
      repod/operations.py:
      Import functions and classes from `repod.convert`, `repod.repo.package`
      and `repod.repo.management` top-level.
      
      repod/repo/__init__.py:
      Import from `repod.repo.package` and `repod.repo.management` to expose
      them top level in the `repod.repo` package.
      
      repod/repo/management/__init__.py:
      Import from `repod.repo.management.outputpackage` to expose them top
      level in the `repod.repo.management` package.
      
      repod/repo/package/__init__.py:
      Import from `repod.repo.management.syncdb` to expose them top
      level in the `repod.repo.package` package.
      
      tests/common/test_models.py:
      Add tests for `BuildDate`, `CSize`, `FileList`, `Name` and `Packager`
      from the removed tests/test_models.py.
      
      tests/conftest.py:
      Import repo related classes top-level from their respective new
      locations.
      
      tests/test_cli.py:
      Import `RepoDbTypeEnum` top-level from its new location.
      
      tests/test_convert.py:
      Import `repod.repo.package` and `repod.repo.package.syncdb` classes
      top-level.
      
      tests/test_files.py:
      Import `repod.repo.management.outputpackage` and `repod.repo.package`
      classes top-level.
      
      tests/test_models.py:
      Remove obsolete tests. They have been distributed in the tests for the
      split repo functionality and the common models.
      
      tests/test_operations.py:
      Import `OutputPackageBaseV1` top-level from its new location.
      
      tests/repo/management/test_outputpackage.py:
      Add tests for all classes and functions in
      `repod.repo.management.outputpackage`.
      
      tests/repo/package/test_syncdb.py;
      Add tests for all classes and functions in
      `repod.repo.package.syncdb`.
      
      tox.ini:
      Export the JSON schema for repod.repo subpackages for the docs.
      4463e7fb
    • David Runge's avatar
      Move repository related models to repo package · edb2e79e
      David Runge authored
      repod/models/package.py -> repod/repo/management/outputpackage.py:
      Move all repository contents related models to a management repository
      specific location. This still encludes models used by package
      repositories, which are split out in a follow-up commit.
      
      repod/models/repo.py -> repod/repo/package/syncdb.py:
      Move package repository related models and functions to a package
      repository specific location.
      edb2e79e
  6. 18 Jun, 2022 12 commits
    • David Runge's avatar
      Add documentation about Package · fc714bee
      David Runge authored
      docs/packages/contents.rst:
      Add documentation about Package and include PackageV1 JSON schema.
      fc714bee
    • David Runge's avatar
      Export JSON schema for PackageV1 in docs target · 0495b27c
      David Runge authored
      tox.ini:
      Export JSON schema for PackageV1 in docs target.
      0495b27c
    • David Runge's avatar
      Remove redundant file read functions · a0b786a0
      David Runge authored
      repod/files/buildinfo.py:
      Remove `read_buildinfo()` as it is superseded by
      `repod.files.common.read_file_from_tarfile`.
      
      repod/files/mtree.py:
      Remove `read_mtree()` as it is superseded by
      `repod.files.common.read_file_from_tarfile`.
      
      repod/files/pkginfo.py:
      Remove `read_pkginfo()` as it is superseded by
      `repod.files.common.read_file_from_tarfile`.
      
      tests/files/test_{buildinfo,mtree,pkginfo}.py:
      Remove tests for removed functions.
      Adapt integration tests to make use of
      `repod.files.common.read_file_from_tarfile`.
      a0b786a0
    • David Runge's avatar
      Add package model for reading · 3bf0a029
      David Runge authored
      repod/files/package.py:
      Add integration for reading and representing package data in a
      versioned `Package` model.
      
      tests/files/test_package.py:
      Add tests for `Package.from_file()`.
      
      repod/files/__init__.py:
      Expose repod.files.package.Package.
      3bf0a029
    • David Runge's avatar
      Add features to function for reading file from tarfile · b1a0441c
      David Runge authored
      repod/files/common.py:
      Extend `extract_file_from_tarfile()` to optionally return the extracted
      file contents in a StringIO and to deal with extracting from gzip
      compressed files in the tarfile.
      Add `names_in_tarfile()` to check whether all strings in a list or set
      are found in the list of filenames in a tarfile.
      
      tests/files/test_common.py:
      Parametrize the test for `extract_file_from_tarfile()`.
      Add parametrized tests for `names_in_tarfile()`.
      b1a0441c
    • David Runge's avatar
      Ensure that default pkginfo fixtures are valid · 3fb6b144
      David Runge authored
      tests/conftest.py:
      Ensure to not add empty lines to StringIO representations of .PKGINFO
      files.
      3fb6b144
    • David Runge's avatar
      Raise project specific error on invalid .PKGINFO line · a2774346
      David Runge authored
      repod/files/pkginfo.py:
      Try/except ValueError when evaluating .PKGINFO lines in
      `PkgInfo.from_file()` and raise a RepoManagementFileError instead. Add
      debug message about which line is being processed.
      
      tests/files/test_pkginfo.py:
      Raise on more invalid lines when testing `PkgInfo.from_file()` and also
      show debug log output.
      a2774346
    • David Runge's avatar
      Add fixture for default package · a82141b3
      David Runge authored
      tests/conftest.py:
      Add fixture for a default package file, containing .BUILDINFO, .MTREE
      and .PKGINFO file.
      a82141b3
    • David Runge's avatar
      Add project metadata to pyproject.toml · b3ffb932
      David Runge authored
      pyproject.toml:
      Add metadata which is needed for releases on pypi.org.
      b3ffb932
    • David Runge's avatar
      Extend CI integration to system tests and publish · c3e1c667
      David Runge authored
      .gitlab-ci.yml:
      Move linting to a check stage, which runs first.
      Only download files databases for integration tests.
      Run regex tests in a separate test job.
      Add variables to track required packages for various use-cases (e.g.
      tests in a locked poetry context, tests using system packages).
      Add build stage to run a job that uses poetry to build the sdist and
      wheel of the package.
      Add publish stage which runs a publish job if the commit is a tagged
      commit (relying on API token for pypi.org).
      Add jobs to test the project using system dependencies.
      c3e1c667
    • David Runge's avatar
      Add tox target for regex tests · 551b6710
      David Runge authored
      tox.ini:
      Add tox testenv "regex" for regular expression tests.
      551b6710
    • David Runge's avatar
      Mark all regular expression tests · 87bc7adc
      David Runge authored
      tests/common/test_regex.py:
      Mark all regular expression tests with "regex".
      87bc7adc