Fails on a very large package

Task Info (Flyspray)
Opened By	Anadon Aeroheart (anadon)
Task ID	58029
Type	Bug Report
Project	Arch Linux
Category	Arch Projects
Version	None
OS	All
Opened	2018-03-29 21:04:19 UTC
Status	Assigned
Assignee	Kyle Keen (keenerd)

Details

Description: When packaging unusually large files (240GB), namcap is unable to check the resulting package.

Additional info:

• namcap 3.2.7
• package takes at least 1.1TB to build, and I can't verify that the built package installs due to disk limits and no alternative testing machines.
• Largest file is about 240GB, with about 880GB is total.
• The disk uses BTRFS with transparent compression enabled, with ~1TB physical space left so the disk is not actually running out of space.

Steps to reproduce: I've uploaded my example packages here:

• https://aur.archlinux.org/packages/pfam/
• https://aur.archlinux.org/packages/superfamily/ standard build procedure, but I highly suggest you modify your makepkg.conf to add '-T0' to the XZCOMPRESS variable. It takes over 110 CPU hours to compress.

The following output is generated:

[anadon@doge pfam]$ namcap pfam-31.0-1-any.pkg.tar.xz 
xz: (stdout): Write error: No space left on device
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/site-packages/namcap.py", line 250, in <module>
    process_realpackage(package, active_modules)
  File "/usr/lib/python3.6/site-packages/namcap.py", line 87, in process_realpackage
    pkgtar = open_package(package)
  File "/usr/lib/python3.6/site-packages/namcap.py", line 60, in open_package
    if '.PKGINFO' not in tar.getnames():
  File "/usr/lib/python3.6/tarfile.py", line 1766, in getnames
    return [tarinfo.name for tarinfo in self.getmembers()]
  File "/usr/lib/python3.6/tarfile.py", line 1758, in getmembers
    self._load()        # all members, we first have to
  File "/usr/lib/python3.6/tarfile.py", line 2355, in _load
    tarinfo = self.next()
  File "/usr/lib/python3.6/tarfile.py", line 2288, in next
    raise ReadError("unexpected end of data")
tarfile.ReadError: unexpected end of data

Given the nature of this problem, it is very hard for be to test more. The PKGBUILD script has been modified to reduce space used, and similar variants build, check, and install without issue but are all smaller.

Added 2023-05-19 21:17:55 UTC - Commented by Neko-san (Neko-san)

I also have this issue with an archive that's about 15GBs (compressed) and about 143GBs uncompressed (the Unreal Engine AUR package, namely)

Added 2023-08-08 19:11:45 UTC - Commented by Buggy McBugFace (bugbot)

This is an automated comment as this bug is open for more then 2 years. Please reply if you still experience this bug otherwise this issue will be closed after 1 month.

Happy to accept contributions that address this (MRs are open) but I don't have the bandwidth to chance it down myself.

changed the description

changed title from [namcap] fails on a very large package to Fails on a very large package

mentioned in issue archlinux/packaging/packages/cuda#3 (closed)