Upload files in chunks of 20 files at a time
Some pacakges have very many files (stack has ~1376) that are currently
uploaded all at once. If there is an exception during this run or if the
program is killed (happened due to 'open files' limit), the sucessfully
uploaded files are not added to the sqlite db. Next time they are
uploaded again and the upload may be interrupted again, leading to huge
time loss.
To fix this, we now upload in chunks of 20 files at a time and add them
to the db after this. Thus, as long as we manage to upload 20 files
without crashing, the next time, there will be fewer files to upload.
Signed-off-by:
Florian Pritz <bluewind@xinu.at>
Please register or sign in to comment