Commit 569216db authored by eric's avatar eric
Browse files

initial import

parent 36d0b810
This is the finalized draft of the project requirements for the
new Arch package submittal process. AUR (Arch User-community Repo).
The sub-directories contain more specific implementation notes
for each component of the project.
Requirements:
-------------
1) User accounts (users, TUs)
- Create account. (email address required)
- Update account (change password/email address)
- Log in/out
2) Search for packages (public)
- needs knowledge of ALL pkgs (OfficalRepos/AUR/Unsupported). This
should be easy enough if this site lives on the same machine as
the current package database (dragon?), or is allowed to query
the db.
- Display official repo (current/extra) a package lives in.
3) Manual voting (requires user acct)
- reset/clear all votes (for starting over, this can be added later
if there is any demand for it)
4) Package Management
- A package can be submitted by anyone (as long as they create
an account with a valid email address on the web site). From
there, a TU inspects the package and works with the submitter
to resolve any bugs/issues. Once approved, the TU can place the
package in the AUR. From there, if the package is popular enough
(or deemed necessary), a developer can move the package from the
AUR to extra/current/etc. A developer can also downgrade a
package from extra/current/etc to the AUR.
- The person that uploaded the new package can make changes to
it before it has been added to the AUR.
- TUs need to be able to purge packages in "Unsupported" if the
package is no longer maintained and there is no interest in
keeping it.
- Packages in the AUR/Unsupported need some sort of 'flag out of
date' support.
- Interested users can download the package from "Unsupported"
and build/install it by hand.
- Provide a separate installation of flyspray for tracking bugs
for packages living in the AUR. All bugs should be resolved
in either flyspray (AUR/official) prior to a package being
promoted/demoted.
5) Reports
- package popularity by number of votes
6) Wiki Docs (UID/GID db, provides db, irc nicks/names TUs/devs)
- Move the appropriate dev wiki pages to the new system's
wiki area. The devs will just need to consult the UID/GID
list from the new system site rather than our own wiki.
7) Submitting 'new' packages by users. Initially start with
a simple web upload page where users submit a tgz containing
the PKGBUILD, scriptlets, patches, etc. The script will
unpack the tgz in an appropriate location and update the
backend database to 'register' the pacakge.
8) TU package management
- A TU adopts a package from "Unsupported" and that shows users
and other TUs that the package is being reviewed.
- When the TU is ready to move the package to the AUR, they
use a custom utility/script that allows them to upload the
pkg.tar.gz (web uploads are inadequate for this process).
The upload utility/script has a server counterpart that
performs TU authentication and updates the database.
- A cronjob on the server can handle the new AUR package,
update the database, and rebuild the AUR sync db, and send
email notices to the TU about errors if any occur.
- The TUs should also be able to demote a package from the
AUR via the web interface.
- TUs will use cvs/svn interface (just like devs) to pull
down the PKGBUILD tree. This tree should reflect the same
layout as extra for easier package migration. They make
changes to their local copy, test, and then commit. They
use the xfer utility to upload the binary package to the
server. No shell access is required.
Automated Voting Tool (similar to ArchStats client)
=====================
Requirements:
-------------
1) Name of tool is 'pkgvote'
2) Requires registered account on web - email address not required
3) Casts 'yes' votes for all installed packages (except itself?)
Implementation:
---------------
A statically compiled C program that gathers the list of installed
packages and casts the vote to the web site. Very similar to the
way that ArchStats works now. When making the HTTP Post, it adds
a custom HEADER that the PHP script(s) can detect to know that it
is receiving a vote from a 'pkgvote' client. If the PHP script
does not see the special HEADER, it assumes it is a web browser
and gives the user HTML output.
Once installed, the user edits the config file and supplies their
username/password. If no username/password exists in the config
file when it starts, it spits out an error message and exits.
The schema dir contains the AUR database schema file.
The scripts dir will contain any supporting scripts for the AUR such
as the script that handles moving newly uploaded packages via tupkg
to the AUR.
-- The MySQL database layout for the AUR. Certain data
-- is also included such as AccountTypes, PackageLocations, etc.
--
-- Define the Account Types for the AUR.
--
CREATE TABLE AccountTypes (
ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
AccountType char(32) NOT NULL DEFAULT '',
PRIMARY KEY (ID)
);
INSERT INTO TABLE (ID, AccountType) VALUES (1, 'User');
INSERT INTO TABLE (ID, AccountType) VALUES (2, 'Trusted User');
INSERT INTO TABLE (ID, AccountType) VALUES (3, 'Developer');
-- User information for each user regardless of type.
--
CREATE TABLE Users (
ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT,
AccountTypeID UNSIGNED TINYINT NOT NULL DEFAULT 1,
Suspended UNSIGNED TINYINT NOT NULL DEFAULT 0,
Email CHAR(64) NOT NULL,
Passwd CHAR(32) NOT NULL,
RealName CHAR(64) NOT NULL DEFAULT '',
IRCNick CHAR(32) NOT NULL DEFAULT '',
LastVoted UNSIGNED BIGINT NOT NULL DEFAULT 0,
NewPkgNotify UNSIGNED TINYINT NOT NULL DEFAULT 0,
PRIMARY KEY (ID),
UNIQUE INDEX Emailx (Email),
INDEX AccountTypeIDx (AccountTypeID),
INDEX NewPkgNotifyx (NewPkgNotify),
FOREIGN KEY AccountTypeIDr REFERENCES AccountTypes (ID)
);
-- A default developer account for testing purposes
INSERT INTO Users (ID, AccountTypeID, Email, Passwd) VALUES (
1, 3, 'root@localhost', 'changeme');
-- Track Users logging in/out of AUR web site.
--
CREATE TABLE Sessions (
UsersID UNSIGNED INTEGER NOT NULL,
SessionID CHAR(32) NOT NULL,
LastUpdateTS UNSIGNED BIGINT NOT NULL,
FOREIGN KEY UsersIDr REFERENCES Users (ID)
);
-- Categories for grouping packages when they reside in
-- Unsupported or the AUR - based on the categories defined
-- in 'extra'.
--
CREATE TABLE PackageCategories (
ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
Category CHAR(32) NOT NULL,
PRIMARY KEY (ID)
);
INSERT INTO PackageCategories (Category) VALUES ('daemons');
INSERT INTO PackageCategories (Category) VALUES ('devel');
INSERT INTO PackageCategories (Category) VALUES ('editors');
INSERT INTO PackageCategories (Category) VALUES ('emulators');
INSERT INTO PackageCategories (Category) VALUES ('games');
INSERT INTO PackageCategories (Category) VALUES ('gnome');
INSERT INTO PackageCategories (Category) VALUES ('i18n');
INSERT INTO PackageCategories (Category) VALUES ('kde');
INSERT INTO PackageCategories (Category) VALUES ('lib');
INSERT INTO PackageCategories (Category) VALUES ('modules');
INSERT INTO PackageCategories (Category) VALUES ('multimedia');
INSERT INTO PackageCategories (Category) VALUES ('network');
INSERT INTO PackageCategories (Category) VALUES ('office');
INSERT INTO PackageCategories (Category) VALUES ('science');
INSERT INTO PackageCategories (Category) VALUES ('system');
INSERT INTO PackageCategories (Category) VALUES ('x11');
INSERT INTO PackageCategories (Category) VALUES ('xfce');
-- The various repositories that a package could live in.
--
CREATE TABLE PackageLocations (
ID UNSIGNED TINYINT NOT NULL AUTO_INCREMENT,
Location CHAR(32) NOT NULL,
PRIMARY KEY (ID)
);
INSERT INTO PackageLocations (ID, Location) VALUES (1, 'Unsupported');
INSERT INTO PackageLocations (ID, Location) VALUES (2, 'AUR');
INSERT INTO PackageLocations (ID, Location) VALUES (3, 'Current');
INSERT INTO PackageLocations (ID, Location) VALUES (4, 'Extra');
INSERT INTO PackageLocations (ID, Location) VALUES (5, 'Unstable');
-- Information about the actual packages
--
CREATE TABLE Packages (
ID UNSIGNED INTEGER NOT NULL AUTO_INCREMENT,
Name CHAR(32) NOT NULL,
Version CHAR(32) NOT NULL DEFAULT '',
CategoryID UNSIGNED TINYINT NOT NULL,
Description CHAR(128) NOT NULL DEFAULT "An Arch Package",
URL CHAR(256) NOT NULL DEFAULT "http://www.archlinux.org",
Source CHAR(256) NOT NULL DEFAULT "/dev/null",
LocationID UNSIGNED TINYINT NOT NULL,
OutOfDate UNSIGNED TINYINT DEFAULT 0,
SubmittedTS UNSIGNED BIGINT NOT NULL,
SubmitterUID UNSIGNED INTEGER NOT NULL DEFAULT 0,
MaintainerUID UNSIGNED INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (ID),
UNIQUE INDEX Namex (Name),
INDEX CategoryIDx (CategoryID),
INDEX LocationIDx (LocationID),
INDEX OutOfDatex (OutOfDate),
INDEX SubmitterUIDx (SubmitterUID),
INDEX MaintainerUIDx (MaintainerUID),
FOREIGN KEY CategoryIDr REFERENCES PackageCategories (ID),
FOREIGN KEY LocationIDr REFERENCES PackageLocations (ID)
FOREIGN KEY SubmitterUIDr REFERENCES Users (ID)
FOREIGN KEY MaintainerUIDr REFERENCES Users (ID)
);
-- Track votes for packages
--
CREATE TABLE PackageVotes (
UsersID UNSIGNED INTEGER NOT NULL,
PackageID UNSIGNED INTEGER NOT NULL,
PRIMARY KEY (ID),
FOREIGN KEY UsersIDx REFERENCES Users (ID),
FOREIGN KEY PackageIDx REFERENCES Packages (ID)
);
-- The individual files and their file system location.
--
CREATE TABLE PackageContents (
PackageID UNSIGNED INTEGER NOT NULL,
Path CHAR(256) NOT NULL,
INDEX PackageIDx (PackageID)
);
TU Packaging Tools (tupkg)
--------------------------
- client side (python for proof of concept, later re-write to C?)
The main purpose of this tool is to upload the compiled
pkg.tar.gz to the server. It can (should?) do some verification
on the package prior to uploading to the server. It will have
a config file to store run-time information such as username
(email), password, and server name.
- server side (python for proof of concept, later re-write to C?)
The server side will handle incoming connections from its client
side counterpart. The server should bind to port 80 (maybe a
vhost such as tupkg.archlinux.org?) so that firewalls won't be
an issue. The server verifies the client authentication data,
and then accepts the package(s). If port 80 is not available,
perhaps 443, or are there other 'standard' ports that usually
do not get filtered?
I think the server should be multithreaded to handle simultaneous
uploads rather than queue up requests. The download should be
stored in a temp directory based on the username to prevent
directory, filename clashes.
Once the package(s) is uploaded, the server can either kick off
a gensync, or we can write a separate script to call gensync once
or twice a day. My preference would be a separate script to call
gensync (like the *NIX philosophy of one tool per task).
- protocol (c: => client, s: => server)
Whenever the client/server exchange a message, it is always
preceeded by two-bytes representing the following message's
length. For example, when the client connects, it will send:
0x0028username=bfinch@example.net&password=B0b
0x0028 is the 40 byte strlen of the message in two-bytes. The
client and server always read two-bytes from the socket, and
then know how much data is coming and can read that amount of
bytes from the socket.
==> authentication
c: username=emailaddy&password=mypassword
s: result=PASS|FAIL
NOTE: We can add encryption easily enough with the python
version using the socket.ssl method.
==> uploading package data
if PASS:
c: numpkgs=2&name1=p1.pkg.tar.gz&size1=123&md5sum1=abcd\
name2=p2.pkg.tar.gz&size2=3&md5sum2=def1
s: numpkgs=2&name1=p1.pkg.tar.gz&size1=119&\
name2=p2.pkg.tar.gz&size2=0 (*)
(*) NOTE: The server will reply back to the client how many
packages it has already received and its local file size.
This way, the client can resume an upload. In the example
above, the server still needs the last four (123-119) bytes
for the first package, and that it has no part of the
second package. The client would then begin sending the
last four bytes from the first package (p1.pkg.tar.gz) and
then follow it with the full second package (p2.pkg.tar.gz).
The data would be sent as a continuous chunk of data. The
server will then need to track which bytes belong to which
package.
else FAIL:
c: -spits out error message on stderr to user-
==> after upload completes
The server should verify the integrity of the uploaded packages
by doing an md5sum on each and sending the info back to the client
for comparison. After sending the message, the server waits for
the 'ack' message from the client and then closes the connection.
s: np=2&m1=PASS&m2=FAIL
c: ack
The client replies with the 'ack' and then closes its connection
to the server. It then reports the PASS/FAIL status of each
package's upload attempt.
NOTE: If the upload fails (client connection dies), the server
keeps any data it has received in order to support resuming an
upload. However, if the client uploads all data, and the server
successully reads all data and the final MD5 fails, the server
deletes the failed package.
Terms/definitions:
======================
TU - No change (trusted by the community, if anyone asks what trust
means)
TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for
versions)
Incoming - renamed to "Unsupported"
#!/usr/bin/python -O
#
# Description:
# ------------
# This is the client-side portion of the Trusted User package
# manager. The TUs will use this program to upload packages into
# the AUR. For more information, see the ../README.txt file.
#
# Python Indentation:
# -------------------
# Use tabs not spaces. If you use vim, the following comment will
# configure it to use tabs.
# vim: ts=2 sw=2 noet ft=python
#
# TODO write the code
#
#!/usr/bin/python -O
#
# Description:
# ------------
# This is the server-side portion of the Trusted User package
# manager. This program will receive uploads from its client-side
# couterpart, tupkg. Once a package is received and verified, it
# is placed in a specified temporary incoming directory where
# a separate script will handle migrating it to the AUR. For
# more information, see the ../README.txt file.
#
# Python Indentation:
# -------------------
# Use tabs not spaces. If you use vim, the following comment will
# configure it to use tabs.
# vim: ts=2 sw=2 noet ft=python
#
# TODO write the code
#
Web Interface:
==============
Directory Layout:
-----------------
./html - DocumentRoot for AUR, where the PHP scripts live.
./html/css - CSS stylesheets
./html/images - Any AUR images live here.
./lib - Supporting PHP include files. Access denied to Apache.
Scripts:
--------
- lib/funcs.inc
This is where we can stick functions that can be shared
between the various scripts. Also a good place to put the
MySQL authentication variables since it should live outside
the DocumentRoot.
- html/login.php (probably index.php)
PHP script to handle logging users into the AUR web site. It
authenticates using the email address and a password against
the Users table. Once authenticated, a session id is generated
and stored in the Sessions table and sent as a cookie to the
user's browser.
- html/logout.php
PHP script to logout. It clears the session id from the
Sessions table and unsets the cookie.
- html/account.php
PHP script to handle registering for a new account. It prompts
the visitor for account information: Email, password, real name,
irc nick. The info is recorded in the Users table. Perhaps later,
we can add a preferences field that allows the user to request to
be notified when new packages are submitted so that they can cast
votes for them?
If a TU is logged into the system, they can edit accounts and set
the account type (regular user or TU). If a Dev is logged in, they
can also set the account type to Dev. TUs and Devs are able to
delete accounts. If an account is deleted, all "Unsupported"
packages are orphaned (the Users.ID field in the UnsupportedPackages
table is set to Null).
- html/pkgsearch.php
PHP script to search the package database. It should support
searching by location ("unsupported", "AUR", "extra"), name,
category, maintainer, popularity, etc. It should resemble the
packages.php script on archlinux.org. A checkbox should be
included next to each package to allow users to flag a package
out of date.
- html/pkgvote.php
The PHP script that handles voting for packages. It works like
the search script above to provide a list of packages (maybe by
location only?) with a checkbox for the user to register their
'yes' vote. It should probably only list 50-ish packages per page
and allow the user to vote a page at a time. Each page contains a
'Continue' button to advance to the next list of packages. At the
final page, a summary is presented with a 'Cast Vote' button. Once
the vote is cast, the PackageVotes table is first cleared for that
User and then all new entries are added for the new vote (this will
be easier than trying to figure out if the vote has changed for a
particular package).
- html/pkgmgmnt.php
The PHP script for managing packages. It provides a list of
packages under the management of the logged in user (normal or
TU). The user can edit the package information stored in the
database such as the description, url, category, and location
(a TU can move it to/from "unsupported" and the "AUR"). This
is where TUs can adopt packages that are in the "unsupported"
area.
- html/pkgsubmit.php
This is the PHP script that allows users to upload a new package.
The package format will be a tgz containing the PKGBUILD,
scriptlets, and patches necessary to build the package from
source. Initially, the user submitting the package can select
its category (network, devel, etc) but that can be modified
later by the adopting TU. The script makes appropriate entries
into the database (and perhaps notifies interested users of the
new package - see question above).
New terms/definitions:
======================
TU - No change (trusted by the community, if anyone asks what trust
means)
TUR - renamed to Arch User-community Repo (AUR) (so we can use -u for
versions)
Incoming - renamed to "Unsupported"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment