Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
ensembl-gh-mirror
ensembl-hive
Commits
ae614ad5
Commit
ae614ad5
authored
Jan 23, 2015
by
Leo Gordon
Browse files
optimization: avoid calling count_running_workers twice per Scheduler iteration
parent
4abd6675
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
12 additions
and
23 deletions
+12
-23
modules/Bio/EnsEMBL/Hive/Scheduler.pm
modules/Bio/EnsEMBL/Hive/Scheduler.pm
+3
-2
modules/Bio/EnsEMBL/Hive/Valley.pm
modules/Bio/EnsEMBL/Hive/Valley.pm
+9
-21
No files found.
modules/Bio/EnsEMBL/Hive/Scheduler.pm
View file @
ae614ad5
...
...
@@ -66,7 +66,8 @@ sub schedule_workers_resync_if_necessary {
my
$submit_capacity
=
$valley
->
config_get
('
SubmitWorkersMax
');
my
$default_meadow_type
=
$valley
->
get_default_meadow
()
->
type
;
my
$meadow_capacity_limiter_hashed_by_type
=
$valley
->
get_meadow_capacity_hash_by_meadow_type
(
$meadow_users_of_interest
);
my
(
$valley_running_worker_count
,
$meadow_capacity_limiter_hashed_by_type
)
=
$valley
->
count_running_workers_and_generate_limiters
(
$meadow_users_of_interest
);
my
(
$workers_to_submit_by_analysis
,
$workers_to_submit_by_meadow_type_rc_name
,
$total_extra_workers_required
,
$log_buffer
)
=
schedule_workers
(
$queen
,
$submit_capacity
,
$default_meadow_type
,
$list_of_analyses
,
$meadow_capacity_limiter_hashed_by_type
,
$analysis_id2rc_name
);
...
...
@@ -77,7 +78,7 @@ sub schedule_workers_resync_if_necessary {
scheduler_say
(
"
According to analysis_stats no workers are required... let's see if anything went out of sync.
"
);
# FIXME: here is an (optimistic) assumption all Workers the DB knows about are reachable from the Valley:
if
(
$queen
->
db
->
get_RoleAdaptor
->
count_active_roles
()
!=
$valley
->
aggregated_count_running_workers
(
$meadow_users_of_interest
)
)
{
if
(
$queen
->
db
->
get_RoleAdaptor
->
count_active_roles
()
!=
$valley
_running_worker_count
)
{
scheduler_say
(
"
Mismatch between DB's active Roles and Valley's running Workers detected, checking for dead workers...
"
);
$queen
->
check_for_dead_workers
(
$valley
,
1
);
}
...
...
modules/Bio/EnsEMBL/Hive/Valley.pm
View file @
ae614ad5
...
...
@@ -170,38 +170,26 @@ sub get_pending_worker_counts_by_meadow_type_rc_name {
}
sub
get_meadow_capacity_hash_by_meadow_type
{
sub
count_running_workers_and_generate_limiters
{
my
(
$self
,
$meadow_users_of_interest
)
=
@_
;
my
%meadow_capacity_hash
=
();
my
$valley_running_worker_count
=
0
;
my
%meadow_capacity_limiter_hashed_by_type
=
();
foreach
my
$meadow
(
@
{
$self
->
get_available_meadow_list
})
{
my
$this_worker_count
=
$meadow
->
count_running_workers
(
$meadow_users_of_interest
);
$valley_running_worker_count
+=
$this_worker_count
;
my
$available_worker_slots
=
defined
(
$meadow
->
config_get
('
TotalRunningWorkersMax
'))
?
$meadow
->
config_get
('
TotalRunningWorkersMax
')
-
$
meadow
->
count_running_workers
(
$meadow_users_of_interest
)
?
$meadow
->
config_get
('
TotalRunningWorkersMax
')
-
$
this_worker_count
:
undef
;
# so the hash will contain limiters for every meadow_type, but not all of them active:
$meadow_capacity_hash
{
$meadow
->
type
}
=
Bio::EnsEMBL::Hive::
Limiter
->
new
(
"
Number of workers in '
"
.
$meadow
->
signature
.
"
' meadow
",
$available_worker_slots
);
}
return
\
%meadow_capacity_hash
;
}
sub
aggregated_count_running_workers
{
# just an aggregator
my
(
$self
,
$meadow_users_of_interest
)
=
@_
;
my
$valley_running_workers
=
0
;
foreach
my
$meadow
(
@
{
$self
->
get_available_meadow_list
})
{
$valley_running_workers
+=
$meadow
->
count_running_workers
(
$meadow_users_of_interest
);
$meadow_capacity_limiter_hashed_by_type
{
$meadow
->
type
}
=
Bio::EnsEMBL::Hive::
Limiter
->
new
(
"
Number of workers in '
"
.
$meadow
->
signature
.
"
' meadow
",
$available_worker_slots
);
}
return
$valley_running_worker
s
;
return
(
$valley_running_worker
_count
,
\
%meadow_capacity_limiter_hashed_by_type
)
;
}
1
;
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment