Skip to content
  • Jessica Severin's avatar
    refactored synchronization logic to allow for worker distributed syncing. · e6fb56d1
    Jessica Severin authored
    The synchronization of the analysis_stat summary statistics was done by
    the beekeeper at the top of it's loop.  For graphs with 40,000+ analyses
    this centralized syncing became a bottle neck.  This new system allows
    the Queen attached to each worker process to synchronize it's analysis.
    Syncing happens when a worker 'checks in' and when it dies.  The sync on
    'check in' only updates if the stats are >60secs out of date to prevent
    over syncing.
    The beekeeper still needs to do whole system syncs when a subsection has
    finished and the next section needs to be 'unblocked'.  For homology this
    will happen 2 times in a 16 hour run.