mkpkglists improvements

This overlaid MR changes a couple things, namely the format of the meta package archive and only updating archives when needed.

Update When Needed:

We now use the information_schema.tables AUTO_INCREMENT column to keep track of a "serial number" that can be used to tell us whether we need to update or not.

Metadata Format:

    The SQL logic in this file for package metadata now exactly
    reflects RPC's search logic, without searching for specific
    packages.

    Two command line arguments are available:

        --extended | Include License, Keywords and Groups fields

    When one of these arguments are passed, the script will create
    a packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.

    Archive JSON is in the following format: line-separated package objects
    enclosed in a list:

        [
        {...},
        {...},
        {...}
        ]

    This allows users to stream the archive and process packages
    line-by-line, during the local storage process.

    Signed-off-by: Kevin Morris <kevr@0cost.org>

aurweb/scripts/mkpkglists.py

@@ -2,11 +2,13 @@
 
 import datetime
 import gzip
+import json
 
 import aurweb.config
 import aurweb.db
 
 packagesfile = aurweb.config.get('mkpkglists', 'packagesfile')
+packagesmetafile = aurweb.config.get('mkpkglists', 'packagesmetafile')
 pkgbasefile = aurweb.config.get('mkpkglists', 'pkgbasefile')
 userfile = aurweb.config.get('mkpkglists', 'userfile')
 
@@ -27,6 +29,14 @@ def main():
                            "WHERE PackageBases.PackagerUID IS NOT NULL")
         f.writelines([bytes(x[0] + "\n", "UTF-8") for x in cur.fetchall()])
 
+    with gzip.open(packagesmetafile, "wt") as f:
+        cur = conn.execute("SELECT * FROM Packages")
+        json.dump({
+            "warning": "This is a experimental! It can be removed or modified without warning!",
+            "columns": [d[0] for d in cur.description],
+            "data": cur.fetchall()
+        }, f)
+
     with gzip.open(pkgbasefile, "w") as f:
         f.write(bytes(pkgbaselist_header + "\n", "UTF-8"))
         cur = conn.execute("SELECT Name FROM PackageBases " +