jobs that were created outside the database.  Also added ability to not
write (doesn't execute write_output or dataflow jobs).  Improved logic so
that if a job is specified, but something is wrong that the worker fails
rather than reverting to autonomous behaviour mode.

runWorker.pl now provides much better debugging capability through the
use of the -logic_name, -input_id and -no_write combination.

Fully tested by running a complete homology production test.

jobs that were created outside the database.  Also added ability to not
write (doesn't execute write_output or dataflow jobs).  Improved logic so
that if a job is specified, but something is wrong that the worker fails
rather than reverting to autonomous behaviour mode.
Fully tested by running a complete homology production test.

there is no longer the possibility that a worker might accidentally
claim the job it just failed on so there is no longer a need to check
the hive_id of the job when claiming. Removed check for hive_id

for use within the Hive workflow system.  Includes sections on the design
principles that went into the Hive including listing the emergent behaviour
rules that the Workers were designed around that allows this
controller-less system to function.
  http://en.wikipedia.org/wiki/Emergence
  http://en.wikipedia.org/wiki/Autonomous_Agent
  http://en.wikipedia.org/wiki/Artificial_life

jobs are created outside of the database (ie for debug runs).

level while not killing worker caused catastrophic hammering of analysis_job
table if there was a problem in the code where by all jobs fail.

analysis have failed.

control structure where a process/program has been made aware of the job(s)
they are responsible for controlling.  This is facilited via a job url:
   mysql://ia64e:3306/jessica_compara32b_tree/analysis_job?dbID=6065355
AnalysisJobAdptor::CreateNewJob now returns this url on job creation.
When a job is datflowed, an array of these urls is returned (one for each rule).
Jobs can now be dataflowed from a Process subclass with blocking enabled.
A job can be fetched directly with one of these URLs.
A commandline utility ehive_unblock.pl has been added to unblock a url job.
To unblock a job do:
   Bio::EnsEMBL::Hive::URLFactory->fetch($url)->update_status('READY');
This is primarily useful in asynchronous split process/parsing situations.

I needed to add in a check to prevent the worker from grabbing the same
job back and trying to run it again.  The retry works best when the job is
run on a different machine at a different moment in time (ie a different
hive_id).  This randomizes the run environment.

yet allow the worker to continue working.  Previous design killed the worker
when one of it's jobs fails.  This new design should be more efficient yet
still provide some level of inter-job-mangle-proof-ness.

but on a specific job.  For new system which catches job exceptions and
fails that job, but allows the worker to continue working.

to        eval "require Bio::EnsEMBL::Compara::DBSQL::DBAdaptor";

(taken from %self->input_job->dbID).

have not been run before (< retry_count)

don't have any ensembl-hive extensions.  To get access to extended hive functionality
one must inherit from Hive::Process.  A multiple inheritance is allowed like:
  our @ISA = qw( Bio::EnsEMBL::Hive::Process Bio::EnsEMBL::Pipeline::RunnableDB );

  into the Worker object (and out of the Process)
- added Process::worker method so that running processes can talk to the
  worker that is currently running itself.
- modified system so that if a process subclass uses Process::dataflow_output_id
  on branch_code 1, it will turn off the automatic of flowing of the input_job
  out on branch_code 1.  This will make coding much cleaner so that processes
  no longer need to modifiy the input_id of the input_job
- added method Process::autoflow_inputjob which toggles this autoflow behaviour
  if a subclass would like to modify this directly
- auto_dataflow now happens right after the Process::write_output stage

thus allowing one to do hybrid URL/Registry code like...
  if($url) {
    $dbc = Bio::EnsEMBL::Hive::URLFactory->fetch($url, 'compara')->dbc;
    $dbname = $dbc->dbname();
  }
  else { $dbc = Bio::EnsEMBL::Registry->get_DBAdaptor($dbname,'compara')->dbc; }
  $fa = Bio::EnsEMBL::Registry->get_adaptor($dbname,'compara','Family');

which is a simple paired file like.  If file is missing it does nothing
ia64e:3306      127.0.0.1:3371

-lsf_options <string>
which passes whatever is specified in the string to the bsub command
thus allowing the user complete flexibility

program has evolved into being the primary portal for user interaction
with the Hive, so 95% of functions are compute resource agnostic so it
makes more sense to have internal switches for which compute resource to
submit/check.  Expanded local logic to allow multiple local cpus, submit
into background(fork), check with 'ps', kill with 'kill -9'.

added columns hive_id and retry.  Allows user to join to failed workers
in the hive table, and to see which retry level the job was at when the
STDOUT/STDERR files were generated.  Sets at beginning of job run, and
deletes those for 'empty' files at job end.

in a more filesystem friendly manner (creates at 256 layer hash which distributes
the directories evenly and reduces concurrent directory modification.
Also reordered how the job output files are saved (done at the beginning right
after redirection starts, and at the end right before it's closed).

any problems related to setting undef or '0' values.

made the worse (Tim Cutts).  This will do until we figure this out....
I like the '>/dev/null + rerun failed jobs manually with debug' option personally :)

each digit becomes a directory with a final directory created with the full hive_id
hive_id=1234 => <base_dir>/1/2/3/4/hive_id_1234/
hive_id=12   => <base_dir>/1/2/hive_id_12/
this should distribute the output directories

is calculated.  If batch_size>0 use batch_size, else use avg_msec_per_job equation.

to RunnableDB to allow full benefit of dataflow graph capabilities.
- Removed from Extension.pm branch_code, analysis_job_id, reset_job extensions to
  RunnableDB (no longer trying to shoe-horn hive 'extra' functions into them)
- Bio::EnsEMBL::Hive::Process mirrors some of the RunnableDB interface
  (new, analysis, fetch_input, run, write_output)
  but uses a new job interface (input_job, dataflow_output_id) instead of
  input_id (but provides convenience method $self->input_id which redirects to
  $self->input_job->input_id to simplify porting)
- Changed Worker to only use hive 'extended' function if the processing module
  isa(Bio::EnsEMBL::Hive::Process).  Also allows all RunnableDB modules to
  still be used (or any object which implements a minimal 'RunnableDB interface')
  (new, input_id, db, fetch_input, run, write_output)