Experimental/misc master
Created by: muffato
Here is a list of minor changes I've made whilst working on the scheduler. Users will notice that beekeeper's output looks different, e.g.
mattxps:~ $ beekeeper.pl -url mysql://ensadmin:ensembl@localhost:3306/matthieu_long_mult --debug 1 -analyses_pattern 2,3,1
Pipeline name: long_mult
Default meadow: LOCAL/mattxps
Beekeeper : the following Analyses matched your -analyses_pattern '2,3,1' : take_b_apart(1), part_multiply(2), add_together(3)
Beekeeper : 0 Analyses are not shown
take_b_apart(1) DONE, 2002d jobs, avg: 33.2 sec, 0 workers ( 0 required), h.cap:- a.cap:- (sync'd 40119 sec ago)
part_multiply(2) EMPTY, =0 jobs, avg: N/A, 0 workers ( 0 required), h.cap:- a.cap:- (sync'd 40116 sec ago)
add_together(3) READY, 95r+1555d=1650 jobs, avg: N/A, 0 workers (95 required), h.cap:- a.cap:- (sync'd 747 sec ago)
total over 3 analyses : 97.40% complete (< 0.00 CPU_hrs) (95 to_do + 3557 done + 0 failed + 0 excluded = 3652 total)
===== Stats of active Roles as recorded in the pipeline database: ======
======= TOTAL ======= : 0 active Roles
Scheduler : Discarded 2 analyses because they do not need any Workers.
Scheduler : Analysis 'add_together' is READY, safe-synching it... succeeded
Scheduler : add_together(3) READY, 95r+1555d=1650 jobs, avg: N/A, 0 workers (95 required), h.cap:- a.cap:- (sync'd 0 sec ago)
Scheduler : Hit the limit of *** Max number of Workers scheduled this time ***, settling for 50 Workers.
Scheduler : Hit the limit of *** Number of workers in 'LOCAL/mattxps' meadow ***, settling for 4 Workers.
Scheduler : The Scheduler allocated 4 x LOCAL:default extra workers for 'add_together' [1.0000 hive_load remaining]
Scheduler : I recommend submitting 4 x LOCAL:default workersinstead of
mattxps:~ $ beekeeper.pl -url mysql://ensadmin:ensembl@localhost:3306/matthieu_long_mult --debug 1 -analyses_pattern 2,3,1
Pipeline name: long_mult
Default meadow: LOCAL/mattxps
Beekeeper : the following Analyses matched your -analyses_pattern '2,3,1' : take_b_apart(1), part_multiply(2), add_together(3)
Beekeeper : 0 Analyses are not shown
take_b_apart ( 1) DONE, jobs( 2002d ), avg: 33.2 sec, workers(Running:0, Est.Required:0) h.cap:- a.cap:- (sync'd 40125 sec ago)
part_multiply( 2) EMPTY, jobs( =0 ), avg: 0.0 ms , workers(Running:0, Est.Required:0) h.cap:- a.cap:- (sync'd 40122 sec ago)
add_together ( 3) READY, jobs( 95r+1555d=1650 ), avg: 0.0 ms , workers(Running:0, Est.Required:95) h.cap:- a.cap:- (sync'd 6 sec ago)
total over 3 analyses : 97.40% complete (< 0.00 CPU_hrs) (95 to_do + 3557 done + 0 failed + 0 excluded = 3652 total)
===== Stats of active Roles as recorded in the pipeline database: ======
======= TOTAL ======= : 0 active Roles
Scheduler : Discarded 2 analyses because they do not need any Workers.
Scheduler : Analysis 'add_together' is READY, safe-synching it...
Scheduler : Safe-sync of Analysis 'add_together' succeeded.
Scheduler : Hit the limit of *** Max number of Workers scheduled this time ***, settling for 50 Workers.
Scheduler : Hit the limit of *** Number of workers in 'LOCAL/mattxps' meadow ***, settling for 4 Workers.
Scheduler : add_together( 3) READY, jobs( 95r+1555d=1650 ), avg: 0.0 ms , workers(Running:0, Est.Required:95) h.cap:- a.cap:- (sync'd 0 sec ago)
Scheduler : Before checking the Valley for pending jobs, the Scheduler allocated 4 x LOCAL:default extra workers for 'add_together' [1.0000 hive_load remaining]
Scheduler : I recommend submitting 4 x LOCAL:default workersIt's a bit more compact, guaranteed to be vertically aligned, and I've collapsed some of the messages that come from the scheduler.
Having this diff as a separate PR makes the other PR #46 cleaner