qdb - maintain and query CPAN's update history
Usage: qdb [-v] [-q] [-d] [-T] [-h] [-t] [-i] [-c conf] option v : be verbose option q : be quiet option d : show debug info option T : show trace info option h : show help ; exit option t : test the config, using port+1 ; exit option i : interactive ; don't fork the daemon option c : use config $conf ; default [ qdb.conf, /etc/qdb/conf]
At the moment, CPAN's instant mirroring uses RECENT-files,
which contain CPAN's update history.
The idea is to maintain the update history in an online database,
which instant mirroring clients can query for new events.
This would obsolete the RECENT-files and simplify rrr-server
and instant mirroring clients.
Program qdb stops/starts a server daemon thats maintains a database containing CPAN's update history.
The update history is a list of tuples (events)
⟦ id, type, path, time ⟧
where
id is some (auto-increment) unique number,
type is either new or delete,
path is a file-path in CPAN,
time is some meaningless timestamp.
Server qdb listens on port 22722, and accepts simple databse queries :
q first_id # the id of the first (oldest) event q last_id # the id of the last (most recent) event q from $n # the oldest 1000 events, with id >= $n q from $n limit $x # same ; max $x results
This information would enable an instant mirroring client to track CPAN updates ;
it would obsolete CPAN's RECENT-files.
At the moment, qdb can add new events to the database is two ways :
The server can accept new ⟦type,path,time⟧ events from an authorised client ;
for instance rrr-server, rrr-client, or iim.
The server can periodicly extract new events from the RECENT-files in the local CPAN tree.
This is quite flexible. We could, for instance, have the situation where :
nothing changes on Pause,
CPAN-master runs the root qdb, and gleans new events
from the RECENT-files it gets from Pause,
instant mirroring clients can use CPAN-master's qdb
instead of RECENT-files.
be quiet ; only errors
be verbose ; show all actions
show debug info ; internals
trace ; show all debug info
show help ; exit
test config, using $port + 1 ; exit
interactive ; don't fork the daemon
use config file ; default : qdb.conf, /etc/qdb/conf
The default locations of the config file are :
A config file looks like this :
+-------------------------------------------------- |# lines that start with '#' are comment |# blank lines are ignored too |# tabs are replaced by a space | |# the config entries are 'key' and 'value' pairs |# a 'key' begins in column 1 |# the 'value' is the rest of the line |somekey part1 part2 part3 ... |otherkey part1 part2 part3 ... | |# indented lines are glued |# the next three lines mean 'somekey part1 part2 part3' |somekey part1 | part2 | part3 +--------------------------------------------------
In the config, an interval-spec specifies an interval (or time-span) ; it can be given in seconds (as in 22 or 22s), minutes [m], hours [h], days [d] and/or weeks [w].
The interval-specs can be combined in any order :
dw # a day and a week 7d+24h # same thing w-0.5h # a week minus half an hour hm6 # 3666 seconds
Specify the (absolute) path of your CPAN directory.
Specify a number ; the default is :
port 22722
The server listens for connections on port $port.
Option permit defines which IPs may access (connect to) the qdb server.
allow/deny-list :: ❪ allow-list ∥ deny-list ❫ ... allow-list :: 'allow' ❪ hostname ∥ netblock ❫ ... deny-list :: 'deny' ❪ hostname ∥ netblock ❫ ... netblock :: IP ∥ CIDR
The first allow or deny rule that matches a given IP determines the result. If no rules match then access is denied.
Examples :
# permit all hosts permit allow 0.0.0.0/32 # permit the 131.211-network permit allow 131.211.0.0/16 # permit all hosts except the 131.211-network permit deny 131.211.0.0/16 allow 0.0.0.0/32
Specify the interval after which the server drops an idle connection. The default is five minutes :
max_idle 5m
As an option, the server can periodicly glean new events from
CPAN's RECENT-files. The default is 0, meaning :
don't look at the RECENT-files.
ival_next_rrr 0
If/when ival_next_rrr is non-zero, qdb inits $epoch
from $database.meta.epoch ; a fatal error occurs if
$database.meta.epoch is not set (zero or undefined).
$database.meta.epoch is set by program qdb-fsck when
it initialises the database.
Specify the server's secret. There is no default.
The secret is shared between the server and some client. The client uses $secret to authenticate itself. Only authenticated clients may send history-update commands to the server.
Applications qdb-send and qdb-fsck (with -s) send new events
to a qdb-server.
The default is
server localhost
Specify how the database must be initialised. The default is :
db_init 0
For a root server, specify :
db_init root
This option is only used by qdb-fsck.
Specify a log-level ; the default is
loglvl Terse
Specify parameters for log rotation ; the default is
rotate 8 1d
On start-up, and after interval-spec seconds, qdb will
rotate its logfile (/var/log/qdb/qdb.log),
saving num files.
Specify the directory where the logfiles live. The default is :
logdir /var/log/qdb
Specify a qdb keeps it files. The default is :
vardir /var/qdb
Specify a run-dir. The default is :
rundir /var/run/qdb
This is where the daemon keeps its files.
Specify a lock-dir. The default is :
lckdir /var/lock/subsys
By default, the server listens on port 22722, accepting connections from permitted hosts. It expects commands from the list below.
By default, each command returns a (json) hash like { cmd ⇒ .., err ⇒ .., res ⇒ ... } If $err is not empty, $res is undefined. In PRETTY-mode, the result is human-readable, and not specified.
To send a command to the server, use qdb-send :
% qdb-send -i
... or use netcat (nc) :
% echo <command> | nc localhost 22722 % echo <command> | nc <server-hostname> 22722
PONG from <hostname> <qdb-version>
QUIT
qdb help
-- version qdb-0.02-p6 -- logfile /var/log/qdb/qdb.log -- loglevel Trace -- database /var/qdb/qdb.lite -- listening on port 22722 as a Server -- Server is processing a command-session << 131.211.32.146 port 22722 -- hostname science-vs14.science.uu.nl
PRETTY 1 # if $arg or no arg PRETTY 0 # if ! $arg
PRETTY 1 # if in pretty-mode PRETTY 0 # if not in pretty-mode
For queries and results, see below.
The server response to AUTH1 is a challenge
like 6469b16e5256cf7713c94c409a8692de.
The client is supposed to respond with :
AUTH2 checksum("$challenge $secret\n"),
where $secret is shared between client and server.
The server expects a new event ; default $time is the current time. The server pushes the event on a (connection-private) PUSH-list and responds with
PUSH $x
where $x is the number of items in the PUSH-list.
A push fails if the specified $path does not exist in the local CPAN tree.
The server disconnects if the connection is not authenticated.
PUSH $x
where $x is the number of items in the PUSH-list.
With no arguments, the server adds the events in the PUSH-list to the database, and empties the PUSH-list.
With arguments, the server adds the specified event to the database ; it doesn't check if the specified $path exists in the local CPAN tree.
The server disconnects if the connection is not authenticated.
The result is the $id of the first (oldest) event.
The result is the $id of the last (most recent) event.
The result is a (ordered) list of the oldest events with $id >= $n. At most 1000 (or $x) events are returned.
The result is the last event pertaining to $path.
The result is [ stat "/path/to/cpan/$path" ] ;
likewise for lstat.
The result is readlink "/path/to/cpan/$path".
Daemon qdb can/should be run by some unpriviledged user.
The config-defaults are, however, geared towards root :
config in /etc/qdb, logs in /var/log/%PROG%/ etc.
The server listens on port 22722 for connections ; that port must most likely be world-reachable.
Program qdb requires a bunch of perl modules, most of which
are CORE (come with perl) ; the others are widely available
from platform-repo's.
Create some directory ; for instance /local/qdb/ ;
then get qdb from
archive.cs.uu.nl :
% rsync -avz archive.science.uu.nl::qdb-dev/ /local/qdb/
You will get
Either install qdb-0.04.tar.gz (for instance with program cpanm) ;
or install qdb, qdb-fsck, qdb-send
% cp qdb qdb-fsck qdb-send /usr/bin/ % ( cd /usr/bin/ ; chmod 755 qdb qdb-fsck qdb-send )
Run qdb -v -t ; this tests the config :
% qdb -v -t
Perl may complain with something like :
Can't locate xxx.pm ...
... where xxx is a missing module.
Install the module with your favorite tool (yum, apt-get),
or use program cpanm.
Installing modules with cpan(1) is usually horrible ;
use cpanm instead ; also CPAN's
How to install CPAN modules.
If cpanm fails, view the cpanm-build-log ;
some modules require gcc(1).
Install missing perl modules until qdb complains about
a missing config file.
/etc/qdb/conf
Supply the path to your local CPAN tree.
cpan /path/to/cpan loglvl Verbose
Start the daemon ; for now, don't fork (use -i) :
% qdb start -i # stop it with ^C
If all is well, you will see qdb start up as server.
qdb as a service
copy init.d :
% cp init.d /etc/init.d/qdb
Make sure qdb is started after a reboot ;
use chkconfig :
% chkconfig --add qdb
or (on Ubuntu) :
% update-rc.d qdb defaults 85 15
HUP : qdb reloads ; qdb re-reads the config ; re-inits client and server
USR1 : qdb re-execs ; qdb stops with END { exec $PROGRAM_NAME, @ARGV }
/etc/qdb/conf qdb-configuration /etc/init.d/qdb qdb start/stop script /var/lock/subsys/qdb touched on qdb-startup ; removed on stop /var/log/qdb logfiles /var/qdb/data.lite sqlite database with event-history and meta-data /var/run/qdb/qdb.lck lock-file /var/run/qdb/qdb.pid pid-file /var/run/qdb/qdb.stp qdb stop-secret ; used in STOP-command
qdb-fsck(1), qdb-send(1), qdb-cgi(1)
You may distribute under the terms of either the GNU General Public License or the Artistic License, as specified in the Perl 5.10.0 README file.