This project is mirrored from https://:*****@github.com/Ensembl/ensembl-hive.git. Pull mirroring updated .
  1. 09 Nov, 2004 1 commit
    • Jessica Severin's avatar
      refactored synchronization logic to allow for worker distributed syncing. · e6fb56d1
      Jessica Severin authored
      The synchronization of the analysis_stat summary statistics was done by
      the beekeeper at the top of it's loop.  For graphs with 40,000+ analyses
      this centralized syncing became a bottle neck.  This new system allows
      the Queen attached to each worker process to synchronize it's analysis.
      Syncing happens when a worker 'checks in' and when it dies.  The sync on
      'check in' only updates if the stats are >60secs out of date to prevent
      over syncing.
      The beekeeper still needs to do whole system syncs when a subsection has
      finished and the next section needs to be 'unblocked'.  For homology this
      will happen 2 times in a 16 hour run.
      e6fb56d1
  2. 20 Oct, 2004 1 commit
  3. 12 Oct, 2004 1 commit
  4. 11 Aug, 2004 2 commits
  5. 06 Aug, 2004 1 commit
  6. 03 Aug, 2004 1 commit
  7. 16 Jul, 2004 2 commits
  8. 15 Jul, 2004 1 commit
  9. 14 Jul, 2004 1 commit
  10. 13 Jul, 2004 3 commits
  11. 09 Jul, 2004 1 commit
    • Jessica Severin's avatar
      changed Queen->create_new_worker method to use rearrange formating. · 54927cb4
      Jessica Severin authored
      Also added functionality so that runWorker can be run without
      specification of an analysis.  The create_new_worker method now will
      query for a 'needed worker' analysis from the AnalysisStats adaptor when
      the analysis_id is undef.  This simplifies the API interface between the
      Queen and the beekeepers.  Now the beekeeper only needs to receive a count
      of workers.  The workers can still be run with explicit analyses for
      testing or situations where one wants to manually control the processing.
      Now one can simply do
      bsub -JW[1-100] runWorker -url mysql://ensadmin:<pass>@ecs2:3361/compara_hive_jess_23
      to create 100 workers which will become whatever analysis that needs to be done.
      54927cb4
  12. 17 Jun, 2004 1 commit
  13. 16 Jun, 2004 1 commit
  14. 14 Jun, 2004 5 commits
  15. 08 Jun, 2004 3 commits
  16. 07 Jun, 2004 1 commit
    • Jessica Severin's avatar
      complete switch over to new DataflowRule design. Dataflow rules use · e45d4761
      Jessica Severin authored
      URL's to specify analysis objects from mysql databases distributed
      across a network.  AnalysisJobAdaptor was switched to create jobs with
      a cless method that gets the db connection from the analysis object that
      is passed.  Thus the system now exists in a distributed state.
      The dataflow rule also implements branching via the branch_code.
      SimpleRule will be deprecated.
      e45d4761
  17. 04 Jun, 2004 1 commit
  18. 27 May, 2004 1 commit
  19. 25 May, 2004 1 commit