mkpkglists improvements
This overlaid MR changes a couple things, namely the format of the meta package archive and only updating archives when needed.
Update When Needed:
We now use the information_schema.tables AUTO_INCREMENT column to keep track of a "serial number" that can be used to tell us whether we need to update or not.
Metadata Format:
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords and Groups fields
When one of these arguments are passed, the script will create
a packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
This allows users to stream the archive and process packages
line-by-line, during the local storage process.
Signed-off-by: Kevin Morris <kevr@0cost.org>Merge request reports
Activity
added database label
assigned to @klausenbusk and @kevr
mentioned in merge request !181 (closed)
added 1 commit
- 2b17dfa1 - fix(mkpkglists): only check against cache if we're using mysql
added 1 commit
- 5af86a9a - fix(php): route packages-meta-v1.json.gz for packages meta archive
added 1 commit
- c2dc6083 - fix(mkpkglists): revert to iterating over cur.fetchall() once
Status: Not really sure why sharness tests are now failing. I'll look into this and resolve it, but otherwise, the script is functional.
Edit: Fixed.
Edited by Kevin Morris-
Question: Should I just remove the
print(...)s out of this MR? I figure that logging for cache state would be nice, but that means if a persistent log is kept, we can't let it get too large.Edited by Kevin Morris added 1 commit
- 89cdabda - fix(mkpkglists): revert to iterating over cur.fetchall() once
added 1 commit
- d33845e8 - fix(mkpkglists): fix tuple return for non-mysql
-
We probably want to provide the same exact format that RPC [multi]info does.
Tasks:
- Use [multi]info JSON format.
Edited by Kevin Morris added 1 commit
- 3d31e860 - fix(mkpkglists): add all info fields to packages-meta-v1
added 1 commit
- 33d58386 - fix(mkpkglists): add all info fields to packages-meta-v1
added 1 commit
- 2949e39d - fix(mkpkglists): add all RPC type=search fields to packages-meta-v1
added 1 commit
- 491e1018 - fix(mkpkglists): add all RPC type=search fields to packages-meta-v1
added 1 commit
- 001e16a9 - fix(mkpkglists): add all RPC type=search fields to packages-meta-v1
added 1 commit
- 1229c464 - fix(mkpkglists): add all RPC type=search fields to packages-meta-v1
added 1 commit
- ce635a1d - fix(mkpkglists): add all RPC type=search fields to packages-meta-v1
added 1 commit
- e5cd3e1e - fix(mkpkglists): add all RPC type=search fields to packages-meta-v1
added 1 commit
- 334f73af - cleanup(mkpkglists): Last minute sanitization
added 13 commits
-
334f73af...64ba18e4 - 4 commits from branch
archlinux:master - 69773a5b - feat(PHP): Add packages dump file with more metadata
- 928ef295 - fix(mkpkglists): improve package meta archive
- 5b4931d9 - feat(mkpkglists): cache update states of mkpkglists
- adee633d - fix(mkpkglists): only check against cache if we're using mysql
- 197b04d6 - fix(php): route packages-meta-v1.json.gz for packages meta archive
- d8c0921e - fix(mkpkglists): revert to iterating over cur.fetchall() once
- c53041f9 - fix(mkpkglists): fix tuple return for non-mysql
- 37e7c201 - fix(mkpkglists): add all RPC type=search fields to packages-meta-v1
- 758c755f - cleanup(mkpkglists): Last minute sanitization
Toggle commit list-
334f73af...64ba18e4 - 4 commits from branch
added 1 commit
- 51fb24ab - fix(mkpkglists): improve package meta archive
mentioned in commit infrastructure@a4a8a640
