mkpkglists improvements

This overlaid MR changes a couple things, namely the format of the meta package archive and only updating archives when needed.

Update When Needed:

We now use the information_schema.tables AUTO_INCREMENT column to keep track of a "serial number" that can be used to tell us whether we need to update or not.

Metadata Format:

    The SQL logic in this file for package metadata now exactly
    reflects RPC's search logic, without searching for specific
    packages.

    Two command line arguments are available:

        --extended | Include License, Keywords and Groups fields

    When one of these arguments are passed, the script will create
    a packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.

    Archive JSON is in the following format: line-separated package objects
    enclosed in a list:

        [
        {...},
        {...},
        {...}
        ]

    This allows users to stream the archive and process packages
    line-by-line, during the local storage process.

    Signed-off-by: Kevin Morris <kevr@0cost.org>

aurweb/scripts/mkpkglists.py

@@ -4,6 +4,7 @@ import datetime
 import gzip
 import json
 import os
+import sys
 
 from collections import defaultdict
 from decimal import Decimal
@@ -151,39 +152,32 @@ def main():
         output = dict()
         with gzip.open(packagesmetafile, "wt") as f:
             """ The output "data" json key points to a list of dictionaries,
-            each representing a single result, filled with column names as
-            keys and column values as values.
+            each representing a single result. All items are produced in
+            the same format that RPC type=search uses.
 
             Example:
                 {
-                    "data": [
-                        {
-                            "ID": 123,
-                            "Name": "package_name",
-                            "PackageBaseID": 234,
-                            "Version": "0.1.1",
-                            "Description": "Some description...",
-                            "URL": "https://some.url"
-                        },
-                        ...
-                    ]
+                    "warning": "...",
+                    "data": [{...}, {...}, ...]
                 }
             """
 
             def is_decimal(column):
+                """ Check if an SQL column is of decimal.Decimal type. """
                 if isinstance(column, Decimal):
                     return float(column)
                 return column
 
             output = []
             for result in cur.fetchall():
-                extended_fields = get_extended_fields(result[0], result[2])
 
                 item = {
                     column[0]: is_decimal(result[i])
                     for i, column in enumerate(cur.description)
                 }
-                item.update(extended_fields)
+                if sys.argv[1] == "--extended":
+                    extended_fields = get_extended_fields(result[0], result[2])
+                    item.update(extended_fields)
                 output.append(item)
 
             json.dump({