no longer does Queen::synchronize_hive as part of autonomous loop
-sync option allows user to manually trigger a hard sync.
also removed default display of full hive status and addded option -status
which will print this full status.
Also removed adjusting needed worker count for 'pending' workers. lsf will
sometimes leave jobs in pending state for no apparent reason (new bsubed job
will run yet older pending job stays pending). Current 'pending' count also didn't
differentiate between lsf_beekeeper submited jobs and manually submitted
jobs. This pend adjustment isn't a critcal subsystem so I've removed it for
now.  If a runWorker starts (after a long pend) and there is no work left
it will die immeadiately.  I may rewrite a smarter 'pending' adjustment
in the future.

and distributed manner as it interacts with the workers over the course of its life.
When a runWorker.pl script starts and asks a queen to create a worker the queen has
a list of known analyses which are 'above the surface' where full hive analysis has
been done and the number of needed workers has been calculated. Full synch requires
joining data between the analysis, analysis_job, analysis_stats, and hive tables.
When this reached 10e7 jobs, 10e4 analyses, 10e3 workers a full hard sync took minutes
and it was clear this bit of the system wasn't scaling and wasn't going to make it
to the next order of magnitude. This occurred in the compara blastz pipeline between
mouse and rat.
Now there are some analyses 'below the surface' that have partial synchronization.
These analyses have been flagged as having 'x' new jobs (AnalysisJobAdaptor updating
analysis_stats on job insert).  If no analysis is found to asign to the newly
created worker, the queen will dip below the surface and start checking
the analyses with the highest probablity of needing the most workers.
This incremental sync is also done in Queen::get_num_needed_workers
When calculating ahead a total worker count, this routine will also dip below
the surface until the hive reaches it's current defined worker saturation.
A beekeeper is no longer a required component for the system to function.
If workers can get onto cpus the hive will run.  The beekeeper is now mainly a
user display program showing the status of the hive.  There is no longer any
central process doing work and one hive can potentially scale
beyond 10e9 jobs in graphs of 10e6 analysis nodes and 10e6 running workers.

the most time since last update are at the top of the returned list

When jobs are inserted into the analysis_job table, the analysis_stats table
for the given analysis is updated by incrementing the total_job_count,
and unclaimed_job_count and setting the status to 'LOADING'.
If the analysis is 'BLOCKED' this incremental update does not happen.
When an analysis_stats is 'BLOCKED' and then unblocked this automatically
will trigger a resync so this progress partial update is not needed.

sync unless some jobs have been loaded.

also changed default to 'LOADING' so that it can trigger a sync

than one status

AnalysisStats object now hold the new status.

stored truncated batch_size (when job_limit < batch_size).  Fixed with a
'smart' worker->batch_size method which returns the lesser of either the
analysis->stats->batch_size or worker->job_limit

failed job to the failed worker.

added 'SYNCHING' and 'LOADING' to analysis_stat.status

added disconnect_if_idle before the sleep

The synchronization of the analysis_stat summary statistics was done by
the beekeeper at the top of it's loop.  For graphs with 40,000+ analyses
this centralized syncing became a bottle neck.  This new system allows
the Queen attached to each worker process to synchronize it's analysis.
Syncing happens when a worker 'checks in' and when it dies.  The sync on
'check in' only updates if the stats are >60secs out of date to prevent
over syncing.
The beekeeper still needs to do whole system syncs when a subsection has
finished and the next section needs to be 'unblocked'.  For homology this
will happen 2 times in a 16 hour run.

to turn this on like this anymore

which takes a hash ref as a parameter and returns a string which can
be evaled back to the hash.  $hash_ref = eval(encode_hash($hash_ref));

allows simple $stats->update_status('DONE');

so that branch_code is set explicitly rather than replying on the return
value of the write_output method.  Switched Worker.pm to use this value.

changed to varchar(255) (but dropped joining to analysis_data table)
If modules need more than 255 characters of input_id
they can pass the anaysis_data_id via the varchar(255) : example {adid=>365902}

workers can change batch_size as they run.

1) input_id is the command
2) input_id is formated like '{did=>123}'
  where did is short hand for analysis_data_id and the real command
  is stored in the analysis_data table

All STDOUT and STDERR from the command are autoamtically captured and redirected
to files (locations stored in analysis_job_file table).
Very simple idea, but might prove useful.

all parts of the Hive system.  Allows one to have a single use/include
use Bio::EnsEMBL::Hive;

Use instead DataflowRule and DataflowRuleAdaptor

on all anlyses, not just the ones with entries in the analysis_job table.
New logic is also faster.

logic as -loop option, but returns right away). Also modified checkk_for_dead
to take into account jobs with 'EXIT' status.

table.  Doing join on analysis_job.input_analysis_data_id=analysis_data.analysis_data_id
gives same performance as having analysis_job.input_id in table rather than second query