Skip to content
Snippets Groups Projects

fix(rules): detect unrequired Python packages

Open Jelle van der Waa requested to merge jelle/namcap:teach-pydepends-metadata into master
3 unresolved threads

When using PEP517 to build a Python package a RECORD file is included containing all the dependencies of a package. For backports (like importlib_metadata) projects usually specify until which Python version it is required combined with an optional import usually by checking sys.version_info and comparing the Python version.

As using the AST to figure out if a package is required is tricky and a lot of work and finding it out via shadowed imports of builtins for example with ExceptionGroup can work as shown below, but does not hold up for importlib.metadata as the backport is called importlib_metadata.

{ 'builtins.ExceptionGroup': {'usr/bin/optional.py'}, 'exceptiongroup.ExceptionGroup': {'usr/bin/optional.py'} }

Instead of these methods, rely on the project to provide a marker for example python_version < 3.11 for required packages.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • David Runge
  • David Runge
  • Nice, this looks like a good improvement! :clap:

  • David Runge approved this merge request

    approved this merge request

  • added 1 commit

    • a93742bd - fix(rules): detect unrequired Python packages

    Compare with previous version

  • Jelle van der Waa reset approvals from @dvzrv by pushing to the branch

    reset approvals from @dvzrv by pushing to the branch

  • 123 def get_unrequired_deps(metadata):
    124 try:
    125 raw, _ = parse_email(metadata.read())
    126 parsed = Metadata.from_raw(raw)
    127 except Exception:
    128 return []
    129
    130 pyversion = f"{sys.version_info.major}.{sys.version_info.minor}"
    131 environment = {"python_version": pyversion}
    132 unrequired_dep = []
    133 for dep in parsed.requires_dist:
    134 if dep.marker is None:
    135 continue
    136 if "python_version" in str(dep.marker) and not dep.marker.evaluate(environment):
    137 # Some modules on PyPi are typing-extensions while upstream calls them typing_extensions
    138 unrequired_dep.append(f"python-{dep.name.replace('-', '_')}")
    • The conversion does not work for all kinds of backport packages. For example, importlib-metadata is from package python-importlib-metadata.

    • Instead of simple replace, how do you think about a lookup table? The table can be maintained in namcap for now. In the future, it can be auto generated by scanning all official packages, like sogrep does.

    • Please register or sign in to reply
  • 127 148 modules = defaultdict(set)
    128 149 gir_modules = defaultdict(set)
    129 150 gir_versions = defaultdict(str)
    151 unrequired_packages = []
    130 152
    131 153 for entry in tar:
    132 154 if not entry.isfile():
    133 155 continue
    134 156 f = tar.extractfile(entry)
    157 if entry.name.endswith("METADATA"):
  • Thanks, I love this change!

  • By the way, I assum this will fix #76 ? not really, see #76 (comment 188847)

    Edited by Chih-Hsuan Yen
  • added 15 commits

    Compare with previous version

  • Jelle van der Waa requested review from @dvzrv

    requested review from @dvzrv

  • Chih-Hsuan Yen mentioned in issue #76

    mentioned in issue #76

  • Chih-Hsuan Yen mentioned in issue #97

    mentioned in issue #97

  • Please register or sign in to reply
    Loading