GitLab

Change in the monitor table: analyses -> analysis to improve auto-completion in mysql client for all the analysis_* tables.

gets large

added columns hive_id and retry.  Allows user to join to failed workers
in the hive table, and to see which retry level the job was at when the
STDOUT/STDERR files were generated.  Sets at beginning of job run, and
deletes those for 'empty' files at job end.

needed to better manage the hive system's load on the database housing all
the hive related tables (in case the database is overloaded by multiple users).
Added analysis_stats.sync_lock column (and correspondly in Object and Adaptor)
Added Queen::safe_synchronize_AnalysisStats method which wraps over the
  synchronize_AnalysisStats method and does various checks and locks to ensure
  that only one worker is trying to do a 'synchronize' on a given analysis at
  any given moment.
Cleaned up API between Queen/Worker so that worker only talks directly to the
  Queen, rather than getting the underlying database adaptor.
Added analysis_job columns runtime_msec, query_count to provide more data on
  how the jobs hammer a database (queries/sec).

  analysis_stats.avg_msec_per_job int
modified
  analysis_stats.last_update to a datetime so that it is only changed when the API tells it to.

sync unless some jobs have been loaded.

also changed default to 'LOADING' so that it can trigger a sync

added 'SYNCHING' and 'LOADING' to analysis_stat.status

changed to varchar(255) (but dropped joining to analysis_data table)
If modules need more than 255 characters of input_id
they can pass the anaysis_data_id via the varchar(255) : example {adid=>365902}

input_analysis_data_id int(10) which joins to analysis_data table.
added output_analysis_data_id int(10) for storing output_id
External analysis_data.data is LongText which will allow much longer
parameter sets to be passed around than was previously possible.
AnalysisData will also allow processes to manually store 'other' data and
pass it around via ID reference now.

Estentially a mini filesystem so that data that would normally be stored in
NFS files and referenced via a path, can now be stored in the database and
referenced via a dbID. Data is a LONGTEXT.
Can be used to store configuration data, paramater strings,
BLOSSUM matrix data, uuencode of binary data .....

lsf job arrays (e.g. 72344[3]).
Also changed analysis_job index (analysis_id, status) so that the
analysis_id is indexed first which provides better indexing when only
the analysis_id is specified.

job reseting.  This allowed direct UPDATE..WHERE.. sql to be used.
Also changed the retry_count system: retry_count is only incremented
for jobs that failed (status in ('GET_INPUT','RUN','WRITE_OUTPUT')).
Job that were CLAIMED by the dead worker are just reset without
incrementing the retry_count since they were never attempted to run.
Also the fetching of claimed jobs now has an 'ORDER BY retry_count'
so that jobs that have failed are at the bottom of the list of jobs
to process.  This allows the 'bad' jobs to filter themselves out.

added default values of 1 to batch_size and hive_capacity

the analysis table/object.  New object class and tables includes a good
guess of what kind of stats I'll need for the Queen to monitor a hive
Also cleaned up some bugs caused by the extraction from compara.