Estentially a mini filesystem so that data that would normally be stored in
NFS files and referenced via a path, can now be stored in the database and
referenced via a dbID. Data is a LONGTEXT.
Can be used to store configuration data, paramater strings,
BLOSSUM matrix data, uuencode of binary data .....

Added get_available_adaptors to get the pairs of name / adaptor modules. This should make it easier to read etc

an analysis if one of it's conditions are not fullfilled.  Needed for case
when system is done, and new data is flowed through the system (progressive runs).

each cycle of loop.  This is to even out the start/stop waves to make it
easier for others to get jobs started on LSF and to reduce
the startup mysql load that can happen when 700 workers all birth at once.
Defaults to 50 (every 5 minutes), but can be altered with -wlimit option

worker request by number of jobs PENDing to prevent excessive queuing of workers.

If not a clean exit, it will record it as a FATALITY and reset it's jobs right away.

should also specify logic_name or analysis_id so that the input_id is run on the
correct analysis.  Doesn't insert job into database. Designed for testing RunnableDB's
but may prove useful in other contexts.

added methods reset_job and global_cleanup to Bio::EnsEMBL::Pipeline::RunnableDB via category extension.
Worker calls global_cleanup on it's runnableDB after all jobs are done.

turn disconnect ON, when there will be lots of them and they have
moments when there will be little DB activity.  The new disconnect system
disconnects so much, that it's slower than before, so must use sparingly.

output jobs) is it needs a fast database, so don't disconnect_when_inactive

created new() methods where needed, replaced throw, rearrange as needed

If type is defined in url string, that will take precedence.
Bio::EnsEMBL::Hive::URLFactory->fetch($url, 'compara'); #defaults to compara DBA
Bio::EnsEMBL::Hive::URLFactory->fetch($url); #defaults to hive DBA

inherit from them (as in $dba->dbc->)

eg: mysql://ensro@ecs4:3352/ensembl_compara_22_1;type=compara
current types are (core, hive, compara, pipeline) (default is hive)

Also changed to a global class instance so that cleanup can be
automatic via  DESTROY method. Deprecated cleanup method.

as DONE now with LSF, and EXIT will be reserved for true failure cases

blocking can occurr both at the job level and the analysis level.
To block and unblock at the job level will require specific analyses to
determine logic, and will not be implemented in a generic way within the
hive system.

on failure and retry_count>=5. Also changed Queen analysis summary to
classify an analysis as 'DONE' when all jobs are either DONE or FAILED
and hence allow the processing to proceed forward.

it syncs and displays a full summary of the state of the hive including
what workers were overdue, how many are needed, and what workers are running.
Also changed the check_for_dead to use the LSF bjobs command since I now
store an LSF job_id and array_index in the process_id for LSF workers.
Also changed the overdue time limit to 75minutes since the expected lifetime
is 60minutes.

as well also set the beekeeper to LSF.  Commented this.

to prevent excessive workers from saturating the system.

when a bsub is issued without a job array and bjobs does not like 234234[0] as a job_id

added to a users path and the programs run from any directory

which causes a more even distribution of workers types as they are created by the Queen.

lsf job arrays (e.g. 72344[3]).
Also changed analysis_job index (analysis_id, status) so that the
analysis_id is indexed first which provides better indexing when only
the analysis_id is specified.

also changed time intervals (overdue workers now to 75 minutes and
poll interval to 5 minutes).  The polling load to check hive status is
essentially zero.