Switch from xtarfile to libarchive

See merge request !1

The former was used for .zst support. It does it by decompressing the
whole tarball into a temporary location. With intel-oneapi-basekit
exceeding 16G uncompressed size, we're running out of space on gemini.

Switch to libarchive which should be able to stop after reading .PKGINFO
without decompressing and/or reading the rest of the .zst package file.

Support .zst files by using xtarfile

xtarfile can handle opening .zst files which sadly Python does not
support yet. The xtarfile support however does not support the '|*' flag
nor an encoding flag which seem to be unrequired for parsing the
PKGFINO contents.

https://bugs.python.org/issue37095

Some (meta) packages like kdeaccessibility-meta do not have a license
set and right now this triggers an exception. Fix it by just setting an
empty value like archweb does.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

For a KeyError, print(e) just prints the key name that wasn't found.
That's not even remotely helpful/obvious. Now we get a stacktrace and
the usual "KeyError: foo" message, which is much better and actually
what I wanted when I changed this.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Apparently some package doesn't have the pkginfo['license'] value set so
we get an exception, but that exception isn't expected and thus we don't
get the name of the broken package. Also, we really want that the
uploader tries to upload all the other files instead of aborting early.
We still want it to exit non-zero though.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Having the default parameter in the parameter list always initializes
the class. That's not what we want here since it actually create a real
database when running the tests.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Oops. There should have been an index there... Having it should improve
performance somewhat.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

We sometimes get various types of exception (timeouts, SSL errors,
broken remove servers). Running the script again will error at the same
point. Instead of terminating the current run, we now try to complete as
much as possible by pushing on.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Some pacakges have very many files (stack has ~1376) that are currently
uploaded all at once. If there is an exception during this run or if the
program is killed (happened due to 'open files' limit), the sucessfully
uploaded files are not added to the sqlite db. Next time they are
uploaded again and the upload may be interrupted again, leading to huge
time loss.

To fix this, we now upload in chunks of 20 files at a time and add them
to the db after this. Thus, as long as we manage to upload 20 files
without crashing, the next time, there will be fewer files to upload.

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>

Signed-off-by: Florian Pritz <bluewind@xinu.at>