Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
ensembl-gh-mirror
ensembl-hive
Commits
d69de348
Commit
d69de348
authored
Sep 11, 2013
by
Matthieu Muffato
Browse files
Added documentation
parent
06ab0afc
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
22 additions
and
23 deletions
+22
-23
scripts/generate_timeline.pl
scripts/generate_timeline.pl
+22
-23
No files found.
scripts/generate_timeline.pl
View file @
d69de348
#!/usr/bin/env perl
# Gets the activity of each analysis along time, in a CSV file
# Gets the activity of each analysis along time, in a CSV file
or in an image (see list of formats supported by GNUplot)
use
strict
;
use
warnings
;
...
...
@@ -275,47 +275,46 @@ __DATA__
=head1 NAME
lsf_report
.pl
generate_profile
.pl
=head1 DESCRIPTION
This script is used for offline examination of resources used by a Hive pipeline running on LSF
(the script is [Pp]latform-dependent).
This script is used for offline examination of the allocation of workers.
Based on the command-line parameters 'start_date' and 'end_date', or on the start time of the first
worker and end time of the last worker (as recorded in pipeline DB), it pulls the relevant data out
of LSF's 'bacct' database, parses it and stores in 'lsf_report' table.
You can join this table to 'worker' table USING(process_id) in the usual MySQL way
to filter by analysis_id, do various stats, etc.
of the 'worker' table for accurate timing.
By default, the output is in CSV format, to allow extra analaysis to be carried.
You can optionally ask the script to dump the 'bacct' database in a dump file,
or fill in the 'lsf_report' table from an existing dump file (most time is taken by querying bacct).
You can optionally ask the script to generate an image with Gnuplot.
Please note the script
may additionally pull information about LSF processes that you ran simultaneously
with
running
the
pipeline
. It is easy to ignore them by joining into 'worker' table
.
Please note the script
runs a query for each interval (default: 5 minutes), which can take some time
for long-
running pipeline
s
.
=head1 USAGE EXAMPLES
# Just run it the usual way:
query 'bacct' and load the relevant data into 'lsf_report' table:
lsf_report
.pl -url mysql://username:secret@hostname:port/
long_mult_test
# Just run it the usual way:
only the top 19 analysis will be reported in CSV format
generate_profile
.pl -url mysql://username:secret@hostname:port/
database > profile.csv
# The same, but
assuming LSF user someone_else ran the pipeline:
lsf_report
.pl -url mysql://username:secret@hostname:port/
long_mult_test -lsf_user someone_else
# The same, but
getting the analysis that fill 99.5% of the global activity in a PNG file
generate_profile
.pl -url mysql://username:secret@hostname:port/
database -top .995 -output profile.png
# Assuming
the dump file existed. Load the dumped bacct data into 'lsf_report' tab
le
:
lsf_report
.pl -url mysql://username:secret@hostname:port/
long_mult_test -dump long_mult.bacct
# Assuming
you are only interested in a precise interval (in a PNG fi
le
)
generate_profile
.pl -url mysql://username:secret@hostname:port/
database -start_date 2013-06-15T10:34 -end_date 2013-06-15T16:58 -granularity 1 -output profile.png
# Assuming th
e dump file did not exist. Query 'bacct', dump the data into a file and load it into 'lsf_report':
lsf_report
.pl -url mysql://username:secret@hostname:port/
long_mult_test -dump long_mult_again.bacct
# Assuming th
at the pipeline has large periods of inactivity
generate_profile
.pl -url mysql://username:secret@hostname:port/
database -granularity 10 -skip_no_activity 1 > profile.csv
=head1 OPTIONS
-help : print this help
-url <url string> : url defining where hive database is located
-dump <filename> : a filename for bacct dump. It will be read from if the file exists, and written to otherwise.
-lsf_user <username> : if it wasn't you who ran the pipeline, LSF user name of that user can be provided
-start_date <date> : minimal start date of a job (the format is '2012/01/25/13:46')
-end_date <date> : maximal end date of a job (the format is '2012/01/25/13:46')
-start_date <date> : minimal start date of a worker (the format is ISO8601, e.g. '2012-01-25T13:46')
-end_date <date> : maximal end date of a worker (the format is ISO8601, e.g. '2012-01-25T13:46')
-granularity <int> : size of the intervals on which the activity is computed (minutes) (default: 5)
-skip_no_activity <int> : only for CSV output: shrink the periods of inactivity which are longer than "skip_no_activity" hours (default: 2)
-top <float> : maximum number (> 1) or fraction (< 1) of analysis to report (default: 19)
-output <string> : output file: its extension must match one of the Gnuplot terminals. Otherwise, the CSV output is produced on stdout
=head1 CONTACT
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment