GitLab

adding a silent option (do not try to synchronize too much when the hive load is too high) and maximise_concurrency option (try to run as many concurrent analyses as possible instead of lots of jobs of one single analysis)

Added an optional Bio::EnsEMBL::Analysis argument to get_num_needed_workers subroutine, to only get the number of workers needed for this analysis

jobs that were created outside the database.  Also added ability to not
write (doesn't execute write_output or dataflow jobs).  Improved logic so
that if a job is specified, but something is wrong that the worker fails
rather than reverting to autonomous behaviour mode.
Fully tested by running a complete homology production test.

control structure where a process/program has been made aware of the job(s)
they are responsible for controlling.  This is facilited via a job url:
   mysql://ia64e:3306/jessica_compara32b_tree/analysis_job?dbID=6065355
AnalysisJobAdptor::CreateNewJob now returns this url on job creation.
When a job is datflowed, an array of these urls is returned (one for each rule).
Jobs can now be dataflowed from a Process subclass with blocking enabled.
A job can be fetched directly with one of these URLs.
A commandline utility ehive_unblock.pl has been added to unblock a url job.
To unblock a job do:
   Bio::EnsEMBL::Hive::URLFactory->fetch($url)->update_status('READY');
This is primarily useful in asynchronous split process/parsing situations.

  into the Worker object (and out of the Process)
- added Process::worker method so that running processes can talk to the
  worker that is currently running itself.
- modified system so that if a process subclass uses Process::dataflow_output_id
  on branch_code 1, it will turn off the automatic of flowing of the input_job
  out on branch_code 1.  This will make coding much cleaner so that processes
  no longer need to modifiy the input_id of the input_job
- added method Process::autoflow_inputjob which toggles this autoflow behaviour
  if a subclass would like to modify this directly
- auto_dataflow now happens right after the Process::write_output stage

is calculated.  If batch_size>0 use batch_size, else use avg_msec_per_job equation.

reordered where the blocking checks are done (added, deleted, moved).

a job that has been flowed into an analysis/process

needed to better manage the hive system's load on the database housing all
the hive related tables (in case the database is overloaded by multiple users).
Added analysis_stats.sync_lock column (and correspondly in Object and Adaptor)
Added Queen::safe_synchronize_AnalysisStats method which wraps over the
  synchronize_AnalysisStats method and does various checks and locks to ensure
  that only one worker is trying to do a 'synchronize' on a given analysis at
  any given moment.
Cleaned up API between Queen/Worker so that worker only talks directly to the
  Queen, rather than getting the underlying database adaptor.
Added analysis_job columns runtime_msec, query_count to provide more data on
  how the jobs hammer a database (queries/sec).

called when worker dies to replace itself in the needed_workers count since
it's decremented when it's born, and it's counted as living (and subtracted)
as long as it's running.  This gunarantees that another worker will quickly
be created after this one dies (and not need to wait for a synch to happen)

is when there are lots of workers 'WORKING' so as to avoid them falling over
each other.  The 'WORKING' state only exists in the middle of a large run.
When the last worker dies, the state is 'ALL_CLAIMED' so the sync on death
will happen properly.  As the last pile of workers die they will all do
a synch, but that's OK since the system needs to be properly synched when
the last one dies since there won't be anybody left to do it.
Also added 10 minute check for if already 'SYNCHING' to deal with case if
worker dies in the middle of 'SYNCHING'.

so to reduce the sychronization frequency.

added check/set of status to 'SYNCHING' right before the synch procedure
so as to prevent multiple workers from simultaneously trying to synch at the same time.

complete an analysis.  If no job has been run (0 msec) it will assume 1 job
per worker up to the hive_capacity (maximum parallization).
Also changed worker->process_id to be the pid of the process not the ppid.

In synchronize_AnalysisStats method, added a POSIX::ceil when setting the num_required_workers for an AnalysisStats object

'total jobs' count so one can calculate a 'progress bar'

and print_running_worker_status.

used by lsf_beekeeper to decide when it needs to do a hard resync.

and distributed manner as it interacts with the workers over the course of its life.
When a runWorker.pl script starts and asks a queen to create a worker the queen has
a list of known analyses which are 'above the surface' where full hive analysis has
been done and the number of needed workers has been calculated. Full synch requires
joining data between the analysis, analysis_job, analysis_stats, and hive tables.
When this reached 10e7 jobs, 10e4 analyses, 10e3 workers a full hard sync took minutes
and it was clear this bit of the system wasn't scaling and wasn't going to make it
to the next order of magnitude. This occurred in the compara blastz pipeline between
mouse and rat.
Now there are some analyses 'below the surface' that have partial synchronization.
These analyses have been flagged as having 'x' new jobs (AnalysisJobAdaptor updating
analysis_stats on job insert).  If no analysis is found to asign to the newly
created worker, the queen will dip below the surface and start checking
the analyses with the highest probablity of needing the most workers.
This incremental sync is also done in Queen::get_num_needed_workers
When calculating ahead a total worker count, this routine will also dip below
the surface until the hive reaches it's current defined worker saturation.
A beekeeper is no longer a required component for the system to function.
If workers can get onto cpus the hive will run.  The beekeeper is now mainly a
user display program showing the status of the hive.  There is no longer any
central process doing work and one hive can potentially scale
beyond 10e9 jobs in graphs of 10e6 analysis nodes and 10e6 running workers.

The synchronization of the analysis_stat summary statistics was done by
the beekeeper at the top of it's loop.  For graphs with 40,000+ analyses
this centralized syncing became a bottle neck.  This new system allows
the Queen attached to each worker process to synchronize it's analysis.
Syncing happens when a worker 'checks in' and when it dies.  The sync on
'check in' only updates if the stats are >60secs out of date to prevent
over syncing.
The beekeeper still needs to do whole system syncs when a subsection has
finished and the next section needs to be 'unblocked'.  For homology this
will happen 2 times in a 16 hour run.

workers can change batch_size as they run.

on all anlyses, not just the ones with entries in the analysis_job table.
New logic is also faster.