Skip to content

Fix for DS slow registry

Marek Szuba requested to merge github/fork/muffato/slow_workers_as_pending into master

Created by: muffato

This fixes @danstaines's issue: [Feb 11th at 09:49] I've got a problem with hive where I supply a registry, and the registry takes a couple of minutes to load (this is normal with 40k bacteria unfortunately).

The registry takes a lot of time to load, so the worker is running from the LSF point-of-view but is not registered in the database yet (because the DBAdaptor object is not ready yet). Because of the discrepancy, beekeeper thinks that nothing is running and keep on submitting new workers (beyond the analysis capacity)

My fix is to basically count those workers as pending. Because the Valley/Meadow had a method to get the running workers and another one to get the pending workers, I had to unify them into a single method that takes the list of the workers that have registered and return the correct counts. This super-hash is then carried around in various places so that all the decisions are consistent to each other. Overall the Valley now does more things and the Meadows are a bit shorter

Merge request reports