initial import

README.txt

+This is the finalized draft of the project requirements for the
+new Arch package submittal process.  AUR (Arch User-community Repo).
+The sub-directories contain more specific implementation notes
+for each component of the project.
+
+
+Requirements:
+-------------
+1) User accounts (users, TUs)
+ - Create account. (email address required)
+ - Update account (change password/email address)
+ - Log in/out
+
+2) Search for packages (public)
+ - needs knowledge of ALL pkgs (OfficalRepos/AUR/Unsupported).  This
+   should be easy enough if this site lives on the same machine as
+   the current package database (dragon?), or is allowed to query
+   the db.
+ - Display official repo (current/extra) a package lives in.
+
+3) Manual voting (requires user acct)
+ - reset/clear all votes (for starting over, this can be added later
+   if there is any demand for it)
+
+4) Package Management
+ - A package can be submitted by anyone (as long as they create
+   an account with a valid email address on the web site).  From
+   there, a TU inspects the package and works with the submitter
+   to resolve any bugs/issues.  Once approved, the TU can place the
+   package in the AUR.  From there, if the package is popular enough
+   (or deemed necessary), a developer can move the package from the
+   AUR to extra/current/etc.  A developer can also downgrade a
+   package from extra/current/etc to the AUR.
+ - The person that uploaded the new package can make changes to
+   it before it has been added to the AUR.
+ - TUs need to be able to purge packages in "Unsupported" if the
+   package is no longer maintained and there is no interest in
+   keeping it.
+ - Packages in the AUR/Unsupported need some sort of 'flag out of
+   date' support.
+ - Interested users can download the package from "Unsupported"
+   and build/install it by hand.
+ - Provide a separate installation of flyspray for tracking bugs
+   for packages living in the AUR.  All bugs should be resolved
+   in either flyspray (AUR/official) prior to a package being
+   promoted/demoted.
+
+5) Reports
+ - package popularity by number of votes
+
+6) Wiki Docs (UID/GID db, provides db, irc nicks/names TUs/devs)
+ - Move the appropriate dev wiki pages to the new system's
+   wiki area.  The devs will just need to consult the UID/GID
+   list from the new system site rather than our own wiki.
+
+7) Submitting 'new' packages by users.  Initially start with
+   a simple web upload page where users submit a tgz containing
+   the PKGBUILD, scriptlets, patches, etc.  The script will
+   unpack the tgz in an appropriate location and update the
+   backend database to 'register' the pacakge.
+
+8) TU package management
+ - A TU adopts a package from "Unsupported" and that shows users
+   and other TUs that the package is being reviewed.
+ - When the TU is ready to move the package to the AUR, they
+   use a custom utility/script that allows them to upload the
+   pkg.tar.gz (web uploads are inadequate for this process).
+   The upload utility/script has a server counterpart that
+   performs TU authentication and updates the database.
+ - A cronjob on the server can handle the new AUR package,
+   update the database, and rebuild the AUR sync db, and send
+   email notices to the TU about errors if any occur.
+ - The TUs should also be able to demote a package from the
+   AUR via the web interface.
+ - TUs will use cvs/svn interface (just like devs) to pull
+   down the PKGBUILD tree.  This tree should reflect the same
+   layout as extra for easier package migration.  They make
+   changes to their local copy, test, and then commit.  They
+   use the xfer utility to upload the binary package to the
+   server.  No shell access is required.
+
+
+Automated Voting Tool (similar to ArchStats client)
+=====================
+
+Requirements:
+-------------
+  1) Name of tool is 'pkgvote'
+
+  2) Requires registered account on web - email address not required
+
+  3) Casts 'yes' votes for all installed packages (except itself?)
+
+Implementation:
+---------------
+  A statically compiled C program that gathers the list of installed
+  packages and casts the vote to the web site.  Very similar to the
+  way that ArchStats works now.  When making the HTTP Post, it adds
+  a custom HEADER that the PHP script(s) can detect to know that it
+  is receiving a vote from a 'pkgvote' client.  If the PHP script
+  does not see the special HEADER, it assumes it is a web browser
+  and gives the user HTML output.
+
+  Once installed, the user edits the config file and supplies their
+  username/password.  If no username/password exists in the config
+  file when it starts, it spits out an error message and exits.
+

support/README.txt

+The schema dir contains the AUR database schema file.
+
+The scripts dir will contain any supporting scripts for the AUR such
+as the script that handles moving newly uploaded packages via tupkg
+to the AUR.
+

support/schema/aur-schema.sql

+-- The MySQL database layout for the AUR.  Certain data
+-- is also included such as AccountTypes, PackageLocations, etc.
+--
+
+-- Define the Account Types for the AUR.
+--
+CREATE TABLE AccountTypes (
+	ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
+	AccountType char(32) NOT NULL DEFAULT '',
+	PRIMARY KEY (ID)
+);
+INSERT INTO TABLE (ID, AccountType) VALUES (1, 'User');
+INSERT INTO TABLE (ID, AccountType) VALUES (2, 'Trusted User');
+INSERT INTO TABLE (ID, AccountType) VALUES (3, 'Developer');
+
+
+-- User information for each user regardless of type.
+--
+CREATE TABLE Users (
+	ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT,
+  AccountTypeID UNSIGNED TINYINT NOT NULL DEFAULT 1,
+	Suspended UNSIGNED TINYINT NOT NULL DEFAULT 0,
+	Email CHAR(64) NOT NULL,
+	Passwd CHAR(32) NOT NULL,
+	RealName CHAR(64) NOT NULL DEFAULT '',
+	IRCNick CHAR(32) NOT NULL DEFAULT '',
+	LastVoted UNSIGNED BIGINT NOT NULL DEFAULT 0,
+	NewPkgNotify UNSIGNED TINYINT NOT NULL DEFAULT 0,
+	PRIMARY KEY (ID),
+	UNIQUE INDEX Emailx (Email),
+	INDEX AccountTypeIDx (AccountTypeID),
+	INDEX NewPkgNotifyx (NewPkgNotify),
+	FOREIGN KEY AccountTypeIDr REFERENCES AccountTypes (ID)
+);
+-- A default developer account for testing purposes
+INSERT INTO Users (ID, AccountTypeID, Email, Passwd) VALUES (
+	1, 3, 'root@localhost', 'changeme');
+
+
+-- Track Users logging in/out of AUR web site.
+--
+CREATE TABLE Sessions (
+	UsersID UNSIGNED INTEGER NOT NULL,
+	SessionID CHAR(32) NOT NULL,
+	LastUpdateTS UNSIGNED BIGINT NOT NULL,
+	FOREIGN KEY UsersIDr REFERENCES Users (ID)
+);
+
+
+-- Categories for grouping packages when they reside in
+-- Unsupported or the AUR - based on the categories defined
+-- in 'extra'.
+--
+CREATE TABLE PackageCategories (
+	ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
+	Category CHAR(32) NOT NULL,
+	PRIMARY KEY (ID)
+);
+INSERT INTO PackageCategories (Category) VALUES ('daemons');
+INSERT INTO PackageCategories (Category) VALUES ('devel');
+INSERT INTO PackageCategories (Category) VALUES ('editors');
+INSERT INTO PackageCategories (Category) VALUES ('emulators');
+INSERT INTO PackageCategories (Category) VALUES ('games');
+INSERT INTO PackageCategories (Category) VALUES ('gnome');
+INSERT INTO PackageCategories (Category) VALUES ('i18n');
+INSERT INTO PackageCategories (Category) VALUES ('kde');
+INSERT INTO PackageCategories (Category) VALUES ('lib');
+INSERT INTO PackageCategories (Category) VALUES ('modules');
+INSERT INTO PackageCategories (Category) VALUES ('multimedia');
+INSERT INTO PackageCategories (Category) VALUES ('network');
+INSERT INTO PackageCategories (Category) VALUES ('office');
+INSERT INTO PackageCategories (Category) VALUES ('science');
+INSERT INTO PackageCategories (Category) VALUES ('system');
+INSERT INTO PackageCategories (Category) VALUES ('x11');
+INSERT INTO PackageCategories (Category) VALUES ('xfce');
+
+
+-- The various repositories that a package could live in.
+--
+CREATE TABLE PackageLocations (
+	ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
+	Location CHAR(32) NOT NULL,
+	PRIMARY KEY (ID)
+);
+INSERT INTO PackageLocations (ID, Location) VALUES (1, 'Unsupported');
+INSERT INTO PackageLocations (ID, Location) VALUES (2, 'AUR');
+INSERT INTO PackageLocations (ID, Location) VALUES (3, 'Current');
+INSERT INTO PackageLocations (ID, Location) VALUES (4, 'Extra');
+INSERT INTO PackageLocations (ID, Location) VALUES (5, 'Unstable');
+
+
+-- Information about the actual packages
+--
+CREATE TABLE Packages (
+	ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT,
+	Name CHAR(32) NOT NULL,
+	Version CHAR(32) NOT NULL DEFAULT '',
+	CategoryID UNSIGNED TINYINT NOT NULL,
+	Description CHAR(128) NOT NULL DEFAULT "An Arch Package",
+	URL CHAR(256) NOT NULL DEFAULT "http://www.archlinux.org",
+	Source CHAR(256) NOT NULL DEFAULT "/dev/null",
+	LocationID UNSIGNED TINYINT NOT NULL,
+	OutOfDate UNSIGNED TINYINT DEFAULT 0,
+	SubmittedTS UNSIGNED BIGINT NOT NULL,
+	SubmitterUID UNSIGNED INTEGER NOT NULL DEFAULT 0,
+	MaintainerUID UNSIGNED INTEGER NOT NULL DEFAULT 0,
+	PRIMARY KEY (ID),
+	UNIQUE INDEX Namex (Name),
+	INDEX CategoryIDx (CategoryID),
+	INDEX LocationIDx (LocationID),
+	INDEX OutOfDatex (OutOfDate),
+	INDEX SubmitterUIDx (SubmitterUID),
+	INDEX MaintainerUIDx (MaintainerUID),
+	FOREIGN KEY CategoryIDr REFERENCES PackageCategories (ID),
+	FOREIGN KEY LocationIDr REFERENCES PackageLocations (ID)
+	FOREIGN KEY SubmitterUIDr REFERENCES Users (ID)
+	FOREIGN KEY MaintainerUIDr REFERENCES Users (ID)
+);
+
+
+-- Track votes for packages
+--
+CREATE TABLE PackageVotes (
+	UsersID UNSIGNED INTEGER NOT NULL,
+	PackageID UNSIGNED INTEGER NOT NULL,
+	PRIMARY KEY (ID),
+	FOREIGN KEY UsersIDx REFERENCES Users (ID),
+	FOREIGN KEY PackageIDx REFERENCES Packages (ID)
+);
+
+
+-- The individual files and their file system location.
+--
+CREATE TABLE PackageContents (
+	PackageID UNSIGNED INTEGER NOT NULL,
+	Path CHAR(256) NOT NULL,
+	INDEX PackageIDx (PackageID)
+);
+

tupkg/README.txt

+TU Packaging Tools (tupkg)
+--------------------------
+- client side (python for proof of concept, later re-write to C?)
+  The main purpose of this tool is to upload the compiled
+  pkg.tar.gz to the server.  It can (should?) do some verification
+  on the package prior to uploading to the server.  It will have
+  a config file to store run-time information such as username
+  (email), password, and server name.
+
+- server side (python for proof of concept, later re-write to C?)
+  The server side will handle incoming connections from its client
+  side counterpart.  The server should bind to port 80 (maybe a
+  vhost such as tupkg.archlinux.org?) so that firewalls won't be
+  an issue.  The server verifies the client authentication data,
+  and then accepts the package(s).  If port 80 is not available,
+  perhaps 443, or are there other 'standard' ports that usually
+  do not get filtered?
+
+  I think the server should be multithreaded to handle simultaneous
+  uploads rather than queue up requests.  The download should be
+  stored in a temp directory based on the username to prevent
+  directory, filename clashes.
+
+  Once the package(s) is uploaded, the server can either kick off
+  a gensync, or we can write a separate script to call gensync once
+  or twice a day.  My preference would be a separate script to call
+  gensync (like the *NIX philosophy of one tool per task).
+
+- protocol (c: => client, s: => server)
+  Whenever the client/server exchange a message, it is always
+  preceeded by two-bytes representing the following message's
+  length.  For example, when the client connects, it will send:
+
+    0x0028username=bfinch@example.net&password=B0b
+
+  0x0028 is the 40 byte strlen of the message in two-bytes.  The
+  client and server always read two-bytes from the socket, and
+  then know how much data is coming and can read that amount of
+  bytes from the socket.
+
+  ==> authentication
+  c: username=emailaddy&password=mypassword
+  s: result=PASS|FAIL
+
+     NOTE:  We can add encryption easily enough with the python
+            version using the socket.ssl method.
+
+  ==> uploading package data
+  if PASS:
+
+    c: numpkgs=2&name1=p1.pkg.tar.gz&size1=123&md5sum1=abcd\
+        name2=p2.pkg.tar.gz&size2=3&md5sum2=def1
+    s: numpkgs=2&name1=p1.pkg.tar.gz&size1=119&\
+        name2=p2.pkg.tar.gz&size2=0 (*)
+
+    (*) NOTE: The server will reply back to the client how many
+        packages it has already received and its local file size.
+        This way, the client can resume an upload.  In the example
+        above, the server still needs the last four (123-119) bytes
+        for the first package, and that it has no part of the
+        second package.  The client would then begin sending the
+        last four bytes from the first package (p1.pkg.tar.gz) and
+        then follow it with the full second package (p2.pkg.tar.gz).
+        The data would be sent as a continuous chunk of data.  The
+        server will then need to track which bytes belong to which
+        package.
+
+  else FAIL:
+    c: -spits out error message on stderr to user-
+
+
+  ==> after upload completes
+  The server should verify the integrity of the uploaded packages
+  by doing an md5sum on each and sending the info back to the client
+  for comparison.  After sending the message, the server waits for
+  the 'ack' message from the client and then closes the connection.
+
+  s: np=2&m1=PASS&m2=FAIL
+  c: ack
+
+  The client replies with the 'ack' and then closes its connection
+  to the server.  It then reports the PASS/FAIL status of each
+  package's upload attempt.
+
+  NOTE: If the upload fails (client connection dies), the server
+  keeps any data it has received in order to support resuming an
+  upload.  However, if the client uploads all data, and the server
+  successully reads all data and the final MD5 fails, the server
+  deletes the failed package.
+
+
+Terms/definitions:
+======================
+TU - No change (trusted by the community, if anyone asks what trust
+     means)
+TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for
+      versions)
+Incoming - renamed to "Unsupported"
+

tupkg/client/tupkg

+#!/usr/bin/python -O
+#
+# Description:
+# ------------
+# This is the client-side portion of the Trusted User package
+# manager.  The TUs will use this program to upload packages into
+# the AUR.  For more information, see the ../README.txt file.
+#
+# Python Indentation:
+# -------------------
+# Use tabs not spaces.  If you use vim, the following comment will
+# configure it to use tabs.
+# vim: ts=2 sw=2 noet ft=python
+#
+
+
+
+# TODO write the code
+#

tupkg/server/tupkgs

+#!/usr/bin/python -O
+#
+# Description:
+# ------------
+# This is the server-side portion of the Trusted User package
+# manager.  This program will receive uploads from its client-side
+# couterpart, tupkg.  Once a package is received and verified, it
+# is placed in a specified temporary incoming directory where
+# a separate script will handle migrating it to the AUR.  For
+# more information, see the ../README.txt file.
+#
+# Python Indentation:
+# -------------------
+# Use tabs not spaces.  If you use vim, the following comment will
+# configure it to use tabs.
+# vim: ts=2 sw=2 noet ft=python
+#
+
+
+
+# TODO write the code
+#

web/README.txt

+Web Interface:
+==============
+
+Directory Layout:
+-----------------
+./html        - DocumentRoot for AUR, where the PHP scripts live.
+./html/css    - CSS stylesheets
+./html/images - Any AUR images live here.
+./lib         - Supporting PHP include files.  Access denied to Apache.
+
+
+Scripts:
+--------
+- lib/funcs.inc
+  This is where we can stick functions that can be shared
+  between the various scripts.  Also a good place to put the
+  MySQL authentication variables since it should live outside
+  the DocumentRoot.
+
+- html/login.php (probably index.php)
+  PHP script to handle logging users into the AUR web site.  It
+  authenticates using the email address and a password against
+  the Users table.  Once authenticated, a session id is generated
+  and stored in the Sessions table and sent as a cookie to the
+  user's browser.
+
+- html/logout.php
+  PHP script to logout.  It clears the session id from the
+  Sessions table and unsets the cookie.
+
+- html/account.php
+  PHP script to handle registering for a new account.  It prompts
+  the visitor for account information: Email, password, real name,
+  irc nick.  The info is recorded in the Users table.  Perhaps later,
+  we can add a preferences field that allows the user to request to
+  be notified when new packages are submitted so that they can cast
+  votes for them?
+
+  If a TU is logged into the system, they can edit accounts and set
+  the account type (regular user or TU).  If a Dev is logged in, they
+  can also set the account type to Dev.  TUs and Devs are able to
+  delete accounts.  If an account is deleted, all "Unsupported"
+  packages are orphaned (the Users.ID field in the UnsupportedPackages
+  table is set to Null).
+
+- html/pkgsearch.php
+  PHP script to search the package database.  It should support
+  searching by location ("unsupported", "AUR", "extra"), name,
+  category, maintainer, popularity, etc.  It should resemble the
+  packages.php script on archlinux.org.  A checkbox should be
+  included next to each package to allow users to flag a package
+  out of date.
+
+- html/pkgvote.php
+  The PHP script that handles voting for packages.  It works like
+  the search script above to provide a list of packages (maybe by
+  location only?) with a checkbox for the user to register their
+  'yes' vote.  It should probably only list 50-ish packages per page
+  and allow the user to vote a page at a time.  Each page contains a
+  'Continue' button to advance to the next list of packages.  At the
+  final page, a summary is presented with a 'Cast Vote' button.  Once
+  the vote is cast, the PackageVotes table is first cleared for that
+  User and then all new entries are added for the new vote (this will
+  be easier than trying to figure out if the vote has changed for a
+  particular package).
+
+- html/pkgmgmnt.php
+  The PHP script for managing packages.  It provides a list of
+  packages under the management of the logged in user (normal or
+  TU).  The user can edit the package information stored in the
+  database such as the description, url, category, and location
+  (a TU can move it to/from "unsupported" and the "AUR").  This
+  is where TUs can adopt packages that are in the "unsupported"
+  area.
+
+- html/pkgsubmit.php
+  This is the PHP script that allows users to upload a new package.
+  The package format will be a tgz containing the PKGBUILD,
+  scriptlets, and patches necessary to build the package from
+  source.  Initially, the user submitting the package can select
+  its category (network, devel, etc) but that can be modified
+  later by the adopting TU.  The script makes appropriate entries
+  into the database (and perhaps notifies interested users of the
+  new package - see question above).
+
+
+New terms/definitions:
+======================
+TU - No change (trusted by the community, if anyone asks what trust
+     means)
+TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for
+      versions)
+Incoming - renamed to "Unsupported"
+