the deletion of the method is done in the right place now - still, be careful about using this method - Albert Vilella and Michael Schuster

adding a remove_analysis_id method that will DELETE FROM analysis, analysis_stats, and analysis_job table WHERE analysis_id equals the number given

there is no longer the possibility that a worker might accidentally
claim the job it just failed on so there is no longer a need to check
the hive_id of the job when claiming. Removed check for hive_id

control structure where a process/program has been made aware of the job(s)
they are responsible for controlling.  This is facilited via a job url:
   mysql://ia64e:3306/jessica_compara32b_tree/analysis_job?dbID=6065355
AnalysisJobAdptor::CreateNewJob now returns this url on job creation.
When a job is datflowed, an array of these urls is returned (one for each rule).
Jobs can now be dataflowed from a Process subclass with blocking enabled.
A job can be fetched directly with one of these URLs.
A commandline utility ehive_unblock.pl has been added to unblock a url job.
To unblock a job do:
   Bio::EnsEMBL::Hive::URLFactory->fetch($url)->update_status('READY');
This is primarily useful in asynchronous split process/parsing situations.

I needed to add in a check to prevent the worker from grabbing the same
job back and trying to run it again.  The retry works best when the job is
run on a different machine at a different moment in time (ie a different
hive_id).  This randomizes the run environment.

but on a specific job.  For new system which catches job exceptions and
fails that job, but allows the worker to continue working.

have not been run before (< retry_count)

added columns hive_id and retry.  Allows user to join to failed workers
in the hive table, and to see which retry level the job was at when the
STDOUT/STDERR files were generated.  Sets at beginning of job run, and
deletes those for 'empty' files at job end.

when debugging an analysis which fails and would increment the retry_count.

needed to better manage the hive system's load on the database housing all
the hive related tables (in case the database is overloaded by multiple users).
Added analysis_stats.sync_lock column (and correspondly in Object and Adaptor)
Added Queen::safe_synchronize_AnalysisStats method which wraps over the
  synchronize_AnalysisStats method and does various checks and locks to ensure
  that only one worker is trying to do a 'synchronize' on a given analysis at
  any given moment.
Cleaned up API between Queen/Worker so that worker only talks directly to the
  Queen, rather than getting the underlying database adaptor.
Added analysis_job columns runtime_msec, query_count to provide more data on
  how the jobs hammer a database (queries/sec).

is asked to 're-run' a specific job.  By reclaiming, this job is properly
processed so when it finishes it looks like it was run normally by the system.

keep analysis_job.input_id as varchar(255) to allow UNIQUE(analysis_id,input_id)
but in adaptor added logic so that if input_id in AnalysisJob object exceeds
the 255 char limit to store/fetch from the analysis_data table.  The input_id
in the analysis_job table becomes '_ext_input_analysis_data_id ##' which is a unique
internal variable to trigger the fetch routine to know to get the 'real' input_id
from the analysis_data table.
NO MORE 255 char limit on input_id and completely transparent to API user.

to 'LOADING' to trigger sync so system knows that something changed

added method reset_all_jobs_for_analysis_id to facilitate re-flowing data through new dataflow rules.
extended perldoc
changed the 'retry count' to 7 (so runs 1 + 7 retrys)

Initially used to manually re-run a job with runWorker.pl -job_id

changed INSERT syntax to be more SQL compliant

When jobs are inserted into the analysis_job table, the analysis_stats table
for the given analysis is updated by incrementing the total_job_count,
and unclaimed_job_count and setting the status to 'LOADING'.
If the analysis is 'BLOCKED' this incremental update does not happen.
When an analysis_stats is 'BLOCKED' and then unblocked this automatically
will trigger a resync so this progress partial update is not needed.

failed job to the failed worker.

changed to varchar(255) (but dropped joining to analysis_data table)
If modules need more than 255 characters of input_id
they can pass the anaysis_data_id via the varchar(255) : example {adid=>365902}

table.  Doing join on analysis_job.input_analysis_data_id=analysis_data.analysis_data_id
gives same performance as having analysis_job.input_id in table rather than second query

Removed select before store (made new method store_if_needed if that functionality is required by users)
and added option in AnalysisJobAdaptor::CreateNewJob to pass input_analysis_data_id
so if already know the CreateNewJob will be as fast as before.  Plus there are no limits on the
size of the input_id string.

input_analysis_data_id int(10) which joins to analysis_data table.
added output_analysis_data_id int(10) for storing output_id
External analysis_data.data is LongText which will allow much longer
parameter sets to be passed around than was previously possible.
AnalysisData will also allow processes to manually store 'other' data and
pass it around via ID reference now.

created new() methods where needed, replaced throw, rearrange as needed

blocking can occurr both at the job level and the analysis level.
To block and unblock at the job level will require specific analyses to
determine logic, and will not be implemented in a generic way within the
hive system.

on failure and retry_count>=5. Also changed Queen analysis summary to
classify an analysis as 'DONE' when all jobs are either DONE or FAILED
and hence allow the processing to proceed forward.

job reseting.  This allowed direct UPDATE..WHERE.. sql to be used.
Also changed the retry_count system: retry_count is only incremented
for jobs that failed (status in ('GET_INPUT','RUN','WRITE_OUTPUT')).
Job that were CLAIMED by the dead worker are just reset without
incrementing the retry_count since they were never attempted to run.
Also the fetching of claimed jobs now has an 'ORDER BY retry_count'
so that jobs that have failed are at the bottom of the list of jobs
to process.  This allows the 'bad' jobs to filter themselves out.