chaqum —
[ˈkeɪkjuːm], the queue manager
for chaotic job queues
chaqum |
[ -hv
]
[-e
ENTRY
]
[-l
LOG
]
DIRECTORY
[ARGUMENT ... ] |
Start a job manager with
DIRECTORY as its job
tree passing
ARGUMENT ... as commandline
parameters to the initial job script
ENTRY.
Then run the job manager as long as there are jobs running or repeats
registered or it receives
SIGINT,
SIGHUP or
SIGTERM.
The options are as follows:
-
-
-d,
--daemonize
PIDFILE[:USER:GROUP]- Daemonize job manager. Use PIDFILE as lockfile to ensure only one instance
is running. If USER and GROUP are specified run job manager under these
credentials (this usually requires you to be root).
-
-
-e,
--entry
ENTRY- Set the entry script to run as the first in the job tree. Defaults to
'entry'.
-
-
-h- Show a help message and exit.
-
-
-l,
--log
LOG- Configure logging. Can be one of
'console' for logging to stderr or
'syslog[:[APPNAME][:FACILITY][:SOCKET_PATH]]'
for sane defaults for logging to local Syslog. Otherwise the will be
interpreted as the path to a JSON file containing the
logging.config.dictConfig() format.
Defaults to 'syslog' if running as a
daemon 'console' otherwise.
-
-
-v- Turn on verbose logging. Can be repeated up to two times for even more
verbosity.
Job trees are simply directory trees with at least one executable somewhere
inside. The only requirement for these executables is, that they either use
the
chaqum protocol to communicate with the
job manager (see
PROTOCOL if
you're interested in the inner workings) or completly ignore file descriptors
3 and 4.
Having exclusivly executables of only the later type is unusual since such job
trees can't make use of
chaqum's job
management, process spawning and inter-process messaging.
Most commonly job excutables will be written in an interpreted language and use
one of the client libraries. References to those are in the
SEE ALSO section.
The layout of the job tree directory as well as what exactly consitutes a single
job (and thus executable within that job tree) is completly up to the user.
There are simple examples demonstrating different features, available in the
projects
Git
Repository.
chaqum captures everything the job processes
write to stdout and stderr and redirects it to its interal logging mechanism
(console, default local Syslog configuration or custom Python
logging
configuration; see
-l option). It generates
one log message per line of output.
By default messages are logged with log level
INFO,
but jobs can select the level by prepending the lines with one of
C<US>
E<US>
W<US>
I<US>
D<US>
to log as level CRITICAL, ERROR, WARNING, INFO and DEBUG respectivly (in Syslog
terminology these are
LOG_CRIT,
LOG_ERR,
LOG_WARNING,
LOG_INFO and
LOG_DEBUG respectivly).
<US> refers to the ASCII unit
seperator character a.k.a. decimal 31, hex 0x1F or octal 037.
The client libraries supply functions that make logging using this protocol
easy.
To allow jobs to communicate with the job manager
chaqum implements a simple line based
protocol. Communication is always initiated by the job; never by the server.
Commands are handled one after the other and always follow the flow of: Job
writes command to file descriptor 3, job reads managers reply by reading from
file descriptor 4.
> enqueue [-F] [-g GROUP] [-m MAXPROC] [-c MAXCPU] -- SCRIPT [ARGUMENT ...]<LF>
< { S JOBIDENT<LF>,
E<LF> }
> repeat [-c CRON] [-i INTERVAL] -- SCRIPT [ARGUMENT ...]<LF>
< { S<LF>,
E<LF> }
> waitjobs [-t TIMEOUT] -- JOBIDENT [...]<LF>
< { S JOBIDENT {T,N,EXITCODE} [...]<LF>,
E<LF> }
> killjobs [-t TIMEOUT] -- JOBIDENT [...]<LF>
< { S JOBIDENT {T,N,EXITCODE} [...]<LF>,
E<LF> }
> sendmsg -- JOBIDENT LENGTH<LF>
BYTES<LF>
< { S MSGIDENT<LF>,
E<LF> }
> waitrecv [-t TIMEOUT] -- MSGIDENT [...]<LF>
< { S MSGIDENT {T,R} [...]<LF>
E<LF> }
> recvmsg [-t TIMEOUT]<LF>
< { S LENGTH<LF>
BYTES<LF>,
E<LF> }
chaqum.lib(3) —
Python job library.
Written by Florian Wagner.