=pod

=head1 NAME

  qdb - maintain and query CPAN's update history

=head1 SYNOPSIS

  Usage: qdb [-v] [-q] [-d] [-T] [-h] [-t] [-i] [-c conf]
  option v : be verbose
  option q : be quiet
  option d : show debug info
  option T : show trace info
  option h : show help ; exit
  option t : test the config, using port+1 ; exit
  option i : interactive ; don't fork the daemon
  option c : use config $conf ; default [ qdb.conf, /etc/qdb/conf]
  

=head1 DESCRIPTION

At the moment, CPAN's I<instant mirroring> uses C<RECENT>-files,
which contain CPAN's update history.
The idea is to maintain the update history in an online database,
which I<instant mirroring> clients can query for new events.
This would obsolete the C<RECENT>-files and simplify C<rrr-server>
and I<instant mirroring> clients.

Program B<qdb> stops/starts a server I<daemon> thats maintains
a database containing CPAN's update history.

The update history is a list of tuples (events)

  ⟦ id, type, path, time ⟧

where

=over 4

=item *

I<id> is some (auto-increment) unique number,

=item *

I<type> is either C<new> or C<delete>,

=item *

I<path> is a file-path in CPAN,

=item *

I<time> is some meaningless timestamp.

=back

Server B<qdb> listens on port 22722, and accepts simple databse B<q>ueries :

  q first_id          # the id of the first (oldest) event
  q last_id           # the id of the last (most recent) event
  q from $n           # the oldest 1000 events, with id >= $n
  q from $n limit $x  # same ; max $x results

This information would enable an I<instant mirroring> client to track CPAN updates ;
it would obsolete CPAN's C<RECENT>-files.

At the moment, B<qdb> can add new events to the database is two ways :

=over 4

=item 1

The server can accept new ⟦I<type>,I<path>,I<time>⟧ events from an authorised client ;
for instance C<rrr-server>, C<rrr-client>, or C<iim>.

=item 2

The server can periodicly extract new events from the C<RECENT>-files in the local CPAN tree.

=back

This is quite flexible. We could, for instance, have the situation where :

=over 4

=item *

nothing changes on I<Pause>,

=item *

I<CPAN-master> runs the I<root> B<qdb>, and gleans new events
from the C<RECENT>-files it gets from Pause,

=item *

I<instant mirroring> clients can use CPAN-master's B<qdb>
instead of C<RECENT>-files.

=back

=head1 OPTIONS

=over 4

=item -q

be quiet ; only errors

=item -v

be verbose ; show all actions

=item -d

show debug info ; internals

=item -T

trace ; show all debug info

=item -h

show help ; exit

=item -t

test config, using $port + 1 ; exit

=item -i

interactive ; don't fork the daemon

=item -c

use config I<file> ; default : C<qdb.conf>, C</etc/qdb/conf>

=back

=head1 CONFIG FILE

=head2 location

The default locations of the config file are :

=over 4

=item * B<./qdb.conf>

=item * B</etc/qdb/conf>

=back

=head2 syntax

A config file looks like this :

  +--------------------------------------------------
  |# lines that start with '#' are comment
  |# blank lines are ignored too
  |# tabs are replaced by a space
  |
  |# the config entries are 'key' and 'value' pairs
  |# a 'key' begins in column 1
  |# the 'value' is the rest of the line
  |somekey  part1 part2 part3 ...
  |otherkey part1 part2 part3 ...
  |
  |# indented lines are glued
  |# the next three lines mean 'somekey part1 part2 part3'
  |somekey part1
  |  part2
  |  part3
  +--------------------------------------------------

=head2 config entries

In the config, an B<interval-spec> specifies an interval (or time-span) ;
it can be given in seconds (as in B<22> or B<22s>),
minutes [B<m>], hours [B<h>], days [B<d>] and/or weeks [B<w>].

The I<interval-specs> can be combined in any order :

  dw      # a day and a week
  7d+24h  # same thing
  w-0.5h  # a week minus half an hour
  hm6     # 3666 seconds

=over 4

=item cpan I</path/to/cpan>

Specify the (absolute) path of your CPAN directory.

=item port I<port>

Specify a number ; the default is :

  port 22722

The server listens for connections on port I<$port>.

=item permit I<allow/deny-list>

Option I<permit> defines which IPs may access (connect to) the B<qdb> server.

  allow/deny-list :: ❪ allow-list ∥ deny-list ❫ ...
  allow-list :: 'allow' ❪ hostname ∥ netblock ❫ ...
  deny-list  :: 'deny'  ❪ hostname ∥ netblock ❫ ...
  netblock   :: IP ∥ CIDR

The first I<allow> or I<deny> rule that matches a given IP
determines the result.
If no rules match then access is denied.

Examples :

  # permit all hosts
  permit allow 0.0.0.0/32
  # permit the 131.211-network
  permit allow 131.211.0.0/16
  # permit all hosts except the 131.211-network
  permit deny 131.211.0.0/16 allow 0.0.0.0/32

=item max_idle I<interval-spec>

Specify the interval after which the server drops an idle connection.
The default is five minutes :

  max_idle 5m

=item ival_next_rrr I<interval-spec>

As an option, the server can periodicly glean new events from
CPAN's C<RECENT>-files. The default is I<0>, meaning :
I<don't look at the C<RECENT>-files>.

  ival_next_rrr 0

If/when I<ival_next_rrr> is non-zero, B<qdb> inits I<$epoch>
from C<$database.meta.epoch> ; a fatal error occurs if
C<$database.meta.epoch> is not set (zero or undefined).

C<$database.meta.epoch> is set by program C<qdb-fsck> when
it initialises the database.

=item secret I<$secret>

Specify the server's I<secret>. There is no default.

The secret is shared between the server and some client.
The client uses $secret to authenticate itself.
Only authenticated clients may send history-update commands
to the server.

=item server I<$server>

Applications L<qdb-send|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-send.html> and L<qdb-fsck|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-fsck.html> (with C<-s>) send new events
to a B<qdb>-server.
The default is

  server localhost

=item db_init $bool

Specify how the database must be initialised. The default is :

  db_init 0

For a I<root> server, specify :

  db_init root

This option is only used by L<qdb-fsck|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-fsck.html>.

=item loglvl Silent | Quiet | Terse | Verbose | Debug | Trace

Specify a I<log-level> ; the default is

  loglvl Terse

=item rotate I<num> I<interval-spec>

Specify parameters for I<log rotation> ; the default is

  rotate 8 1d

On start-up, and after I<interval-spec> seconds, B<qdb> will
rotate its logfile (C</var/log/qdb/qdb.log>),
saving I<num> files.

=item logdir I<dir>

Specify the directory where the logfiles live. The default is :

  logdir /var/log/qdb

=item vardir I<dir>

Specify a B<qdb> keeps it files. The default is :

  vardir /var/qdb

=item rundir I<dir>

Specify a I<run-dir>. The default is :

  rundir /var/run/qdb

This is where the daemon keeps its files.

=item lckdir I<dir>

Specify a I<lock-dir>. The default is :

  lckdir /var/lock/subsys

=back

=head1 Server

By default, the server listens on port 22722,
accepting connections from permitted hosts.
It expects I<commands> from the list below.

By default, each command returns a (json) hash like
  { cmd ⇒ .., err ⇒ .., res ⇒ ... }
If $err is not empty, $res is undefined.
In PRETTY-mode, the result is human-readable, and not specified.

To send a command to the server, use C<qdb-send> :

  % qdb-send -i

... or use I<netcat> (C<nc>) :

  % echo <command> | nc localhost 22722
  % echo <command> | nc <server-hostname> 22722


=head2 Server commands & results

=over 4

=item * PING

  PONG from <hostname> <qdb-version>

=item * QUIT

  QUIT

=item * HELP

  qdb help

=item * STATE

  -- version  qdb-0.02-p6
  -- logfile  /var/log/qdb/qdb.log
  -- loglevel Trace
  -- database /var/qdb/qdb.lite
  -- listening on port 22722 as a Server
  -- Server is processing a command-session << 131.211.32.146 port 22722
  -- hostname science-vs14.science.uu.nl

=item * PRETTY [$arg]

  PRETTY 1 # if $arg or no arg
  PRETTY 0 # if ! $arg

=item * PRETTY?

  PRETTY 1 # if in pretty-mode
  PRETTY 0 # if not in pretty-mode

=item * Q query

For queries and results, see below.

=item * AUTH1

=item * AUTH2 I<response>

The server response to C<AUTH1> is a I<challenge>
like C<6469b16e5256cf7713c94c409a8692de>.

The client is supposed to respond with :
C<AUTH2 checksum("$challenge $secret\n")>,
where $secret is shared between client and server.

=item * PUSH ❪ new ∥ delete ❫ $path [$time]

The server expects a new I<event> ; default I<$time> is the current time.
The server pushes the event on a (connection-private) PUSH-list
and responds with

  PUSH $x

where I<$x> is the number of items in the PUSH-list.

A push fails if the specified I<$path> does not exist in the local CPAN tree.

The server disconnects if the connection is not authenticated.

=item * PUSH?

  PUSH $x

where I<$x> is the number of items in the PUSH-list.

=item * COMMIT

=item * COMMIT ❪ new ∥ delete ❫ $path [$time]

With no arguments, the server adds the events in the PUSH-list to the database,
and empties the PUSH-list.

With arguments, the server adds the specified event to the database ;
it doesn't check if the specified I<$path> exists in the local CPAN tree.

The server disconnects if the connection is not authenticated.

=back

=head2 Server queries & results

=over 4

=item q first_id

The result is the I<$id> of the first (oldest) event.

=item q last_id

The result is the I<$id> of the last (most recent) event.

=item q from I<$n> ⟦ limit I<$x> ⟧

The result is a (ordered) list of the oldest events with I<$id> >= I<$n>.
At most 1000 (or I<$x>) events are returned.

=item q path $path

The result is the last event pertaining to I<$path>.

=item q stat $path

=item q lstat $path

The result is C<[ stat "/path/to/cpan/$path" ]> ;
likewise for I<lstat>.

=item q readlink $path

The result is C<readlink "/path/to/cpan/$path">.

=back

=head1 INSTALL

=head2 Requirements

=over 4

=item *

Daemon C<qdb> can/should be run by some unpriviledged user.
The config-defaults are, however, geared towards C<root> :
config in C</etc/qdb>, logs in C</var/log/%PROG%/> etc.

=item *

The server listens on port B<22722> for connections ;
that port must most likely be world-reachable.

Program C<qdb> requires a bunch of perl modules, most of which
are I<CORE> (come with C<perl>) ; the others are widely available
from platform-repo's.

=back

=head2 Installation

=over 4

=item * Fetch the software

Create some directory ; for instance C</local/qdb/> ;
then get C<qdb> from
L<archive.cs.uu.nl|http://archive.cs.uu.nl/pub/PERL/qdb-dev/> :

  % rsync -avz archive.science.uu.nl::qdb-dev/ /local/qdb/

You will get

=over 4

=item *

module file C<qdb-0.04.tar.gz>

=item *

utilities L<qdb|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb.html>, L<qdb-fsck|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-fsck.html>, L<qdb-send|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-send.html>, documentation etc ;
the utilities are I<stand-alone> (contain C<Qdb.pm> if necessary)
and can be installed I<by hand> anywhere.

=back

=item * Install the software

Either install C<qdb-0.04.tar.gz> (for instance with program C<cpanm>) ;
or install L<qdb|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb.html>, L<qdb-fsck|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-fsck.html>, L<qdb-send|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-send.html>

  % cp qdb qdb-fsck qdb-send /usr/bin/
  % ( cd /usr/bin/ ; chmod 755 qdb qdb-fsck qdb-send )

=item * Install perl modules :

Run C<qdb -v -t> ; this tests the config :

  % qdb -v -t

Perl may complain with something like :

  Can't locate xxx.pm ...

... where I<xxx> is a missing module.

Install the module with your favorite tool (yum, apt-get),
or use program C<cpanm>.

Installing modules with C<cpan(1)> is usually horrible ;
use C<cpanm> instead ; also CPAN's
L<How to install CPAN modules|http://www.cpan.org/modules/INSTALL.html>.

If C<cpanm> fails, view the C<cpanm>-build-log ;
some modules require C<gcc(1)>.

=item * Install perl modules (repeat)

Install missing perl modules until C<qdb> complains about
a missing I<config file>.

=item * create a qdb-config file in C</etc/qdb/conf>

Supply the path to your local CPAN tree.

  cpan /path/to/cpan
  loglvl Verbose

Start the daemon ; for now, don't fork (use C<-i>) :

  % qdb start -i
  # stop it with ^C

If all is well, you will see C<qdb> start up as server.

=item * install C<qdb> as a I<service>

=over 4

=item *

copy C<init.d> :

  % cp init.d /etc/init.d/qdb

=item *

Make sure C<qdb> is started after a reboot ;
use C<chkconfig> :

  % chkconfig --add qdb

or (on Ubuntu) :

  % update-rc.d qdb defaults 85 15

=back

=back

=head1 SIGNALS

  HUP  : qdb reloads  ; qdb re-reads the config ; re-inits client and server
  USR1 : qdb re-execs ; qdb stops with END { exec $PROGRAM_NAME, @ARGV }

=head1 FILES

  /etc/qdb/conf        qdb-configuration
  /etc/init.d/qdb      qdb start/stop script
  /var/lock/subsys/qdb touched on qdb-startup ; removed on stop
  /var/log/qdb         logfiles
  /var/qdb/data.lite   sqlite database with event-history and meta-data
  /var/run/qdb/qdb.lck lock-file
  /var/run/qdb/qdb.pid pid-file
  /var/run/qdb/qdb.stp qdb stop-secret ; used in STOP-command

=head1 SEE ALSO

L<qdb-fsck|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-fsck.html>(1), L<qdb-send|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-send.html>(1), L<qdb-cgi|http://archive.science.uu.nl/pub/PERL/qdb-dev/qdb-cgi.html>(1)


=head1 LICENSE

You may distribute under the terms of either the GNU General Public
License or the Artistic License, as specified in the Perl 5.10.0 README
file.

=head1 AUTHOR

=for html qdb &copy; 2015-2017
<a href="http://www.staff.science.uu.nl/~penni101">Henk P. Penning</a>
- All rights reserved
; qdb-0.04-p12 - Sat Jan 21 10:42:19 2017

=for man Henk P. Penning, <http://www.staff.science.uu.nl/~penni101> ;
qdb-0.04-p12 - Sat Jan 21 10:42:19 2017


=cut