Commit 2bcbfcce authored by Matthieu Muffato's avatar Matthieu Muffato
Browse files

Updated the scripts documentation

parent cfb521f7
ensembl-hive/scripts $ for i in *pl; do pod2html --noindex --title=$i $i > x.html; pandoc -o ../docs/user_manual/scripts/`echo $i | sed 's/pl$/rst/'` x.html; done
# By default pandoc maps =head1 to <h1>, =head2 to <h2>, etc. I wanted to shift the headings to place the
# script name as <h1>. I'm using --base-header-level=2 to shift the levels, and --title and --standalone
# together to aadd a title
for i in *.pl
do
pod2html --noindex --title=$i $i | pandoc --standalone --base-header-level=2 -f html -t rst -o ../docs/user_manual/appendix/scripts/`echo $i | sed 's/pl$/rst/'`
done
......@@ -2,24 +2,48 @@
Scripts options quick reference
-------------------------------
Execution
~~~~~~~~~
.. toctree::
:titlesonly:
scripts/beekeeper.rst
scripts/init_pipeline.rst
scripts/seed_pipeline.rst
Debugging
~~~~~~~~~
.. toctree::
:titlesonly:
scripts/db_cmd.rst
scripts/hoover_pipeline.rst
scripts/runWorker.rst
scripts/standaloneJob.rst
scripts/tweak_pipeline.rst
Reporting
~~~~~~~~~
.. toctree::
:titlesonly:
scripts/generate_graph.rst
scripts/generate_timeline.rst
scripts/load_resource_usage.rst
scripts/visualize_jobs.rst
Internal scripts
~~~~~~~~~~~~~~~~
.. toctree::
:numbered:
:maxdepth: 0
:glob:
scripts/*
Execution:
- init_pipeline.pl
- seed_pipeline.pl
- beekeeper.pl
Debugging:
- runWorker.pl
- db_cmd.pl
- standaloneJob.pl
- hoover_pipeline.pl
Reporting:
- generate_graph.pl
- load_resource_usage.pl
- generate_timeline.pl
:titlesonly:
scripts/create_sql_patches.rst
scripts/make_branch_glossary.rst
scripts/make_docs.rst
============
beekeeper.pl
============
NAME
====
----
::
beekeeper.pl [options]
DESCRIPTION
===========
-----------
::
......@@ -18,7 +22,7 @@ DESCRIPTION
unexpectedly so that she can free the dead workers and reclaim unfinished jobs.
USAGE EXAMPLES
==============
--------------
::
......@@ -44,10 +48,10 @@ USAGE EXAMPLES
beekeeper.pl -url mysql://username:secret@hostname:port/long_mult_test -dead
OPTIONS
=======
-------
Connection parameters
---------------------
~~~~~~~~~~~~~~~~~~~~~
::
......@@ -58,14 +62,14 @@ Connection parameters
-nosqlvc <0|1> : skip sql version check if 1
Configs overriding
------------------
~~~~~~~~~~~~~~~~~~
::
-config_file <string> : json file (with absolute path) to override the default configurations (could be multiple)
Looping control
---------------
~~~~~~~~~~~~~~~
::
......@@ -84,7 +88,7 @@ Looping control
-sleep <num> : when looping, sleep <num> minutes (default 1 min)
Current Meadow control
----------------------
~~~~~~~~~~~~~~~~~~~~~~
::
......@@ -95,7 +99,7 @@ Current Meadow control
-submit_log_dir <dir> : record submission output+error streams into files under the given directory (to see why some workers fail after submission)
Worker control
--------------
~~~~~~~~~~~~~~
::
......@@ -110,7 +114,7 @@ Worker control
-debug <debug_level> : set debug level of the workers
Other commands/options
----------------------
~~~~~~~~~~~~~~~~~~~~~~
::
......@@ -133,12 +137,12 @@ Other commands/options
-unblock_semaphored_jobs : set SEMAPHORED jobs of -analyses_filter'ed ones to READY so they can start
LICENSE
=======
-------
::
Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Copyright [2016] EMBL-European Bioinformatics Institute
Copyright [2016-2017] EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at
......@@ -150,9 +154,8 @@ LICENSE
See the License for the specific language governing permissions and limitations under the License.
CONTACT
=======
-------
::
Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
=======================
create\_sql\_patches.pl
=======================
==========
db\_cmd.pl
==========
NAME
====
----
::
db_cmd.pl
SYNOPSIS
========
--------
::
db_cmd.pl {-url <url> | [-reg_conf <reg_conf>] -reg_alias <reg_alias> [-reg_type <reg_type>] } [ -exec <alt_executable> ] [ -prepend <prepend_params> ] [ -sql <sql_command> ] [ -verbose ] [other arguments to append to the command line]
DESCRIPTION
===========
-----------
::
......@@ -22,7 +26,7 @@ DESCRIPTION
If the arguments that have to be appended contain options (i.e. start with dashes), first use a double-dash to indicate the end of db_cmd.pl's options and the start of the arguments that have to be passed as-is (see the example below with --html)
USAGE EXAMPLES
==============
--------------
::
......@@ -36,12 +40,12 @@ USAGE EXAMPLES
db_cmd.pl -reg_conf ${ENSEMBL_CVS_ROOT_DIR}/ensembl-compara/scripts/pipeline/production_reg_conf.pl -reg_alias squirrel -reg_type core -sql 'SELECT * FROM coord_system'
LICENSE
=======
-------
::
Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Copyright [2016] EMBL-European Bioinformatics Institute
Copyright [2016-2017] EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at
......@@ -53,9 +57,8 @@ LICENSE
See the License for the specific language governing permissions and limitations under the License.
CONTACT
=======
-------
::
Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
==================
generate\_graph.pl
==================
NAME
====
----
::
generate_graph.pl
SYNOPSIS
========
--------
::
......@@ -15,7 +19,7 @@ SYNOPSIS
./generate_graph.pl [ -url mysql://user:pass@server:port/dbname | -reg_conf <reg_conf_file> -reg_alias <reg_alias> ] [-pipeconfig TopUp_conf.pm]* -output OUTPUT_LOC
DESCRIPTION
===========
-----------
::
......@@ -26,7 +30,7 @@ DESCRIPTION
hive_config.json configuration file.
OPTIONS
=======
-------
**--url**
......@@ -86,19 +90,19 @@ OPTIONS
Print this help message
EXTERNAL DEPENDENCIES
=====================
---------------------
::
GraphViz
LICENSE
=======
-------
::
Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Copyright [2016] EMBL-European Bioinformatics Institute
Copyright [2016-2017] EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at
......@@ -110,9 +114,8 @@ LICENSE
See the License for the specific language governing permissions and limitations under the License.
CONTACT
=======
-------
::
Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
=====================
generate\_timeline.pl
=====================
NAME
====
----
generate\_timeline.pl
SYNOPSIS
========
--------
::
......@@ -16,7 +20,7 @@ SYNOPSIS
[-n_core <int>] [-mem <int>]
DESCRIPTION
===========
-----------
::
......@@ -30,7 +34,7 @@ DESCRIPTION
You can optionally ask the script to generate an image with Gnuplot.
USAGE EXAMPLES
==============
--------------
::
......@@ -47,7 +51,7 @@ USAGE EXAMPLES
generate_timeline.pl -url mysql://username:secret@hostname:port/database -mode memory -output timeline_memory.png
OPTIONS
=======
-------
::
......@@ -71,18 +75,18 @@ OPTIONS
-mem <int> : the default memory allocated to a worker (default: 100Mb)
EXTERNAL DEPENDENCIES
=====================
---------------------
::
Chart::Gnuplot
LICENSE
=======
-------
Copyright [1999-2015] Wellcome Trust Sanger Institute and the
EMBL-European Bioinformatics Institute Copyright [2016] EMBL-European
Bioinformatics Institute
EMBL-European Bioinformatics Institute Copyright [2016-2017]
EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
......@@ -99,7 +103,7 @@ See the License for the specific language governing permissions and
limitations under the License.
CONTACT
=======
-------
Please subscribe to the Hive mailing list:
http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss
......
===================
hoover\_pipeline.pl
===================
NAME
====
----
::
hoover_pipeline.pl
SYNOPSIS
========
--------
::
hoover_pipeline.pl {-url <url> | -reg_conf <reg_conf> -reg_alias <reg_alias>} [ { -before_datetime <datetime> | -days_ago <days_ago> } ]
DESCRIPTION
===========
-----------
::
hoover_pipeline.pl is a script used to remove old 'DONE' jobs from a continuously running pipeline database
USAGE EXAMPLES
==============
--------------
::
......@@ -39,7 +43,7 @@ USAGE EXAMPLES
hoover_pipeline.pl -url "mysql://ensadmin:${ENSADMIN_PSW}@localhost:3306/lg4_long_mult" -before_datetime "2013-02-14 15:42:50"
OPTIONS
=======
-------
::
......@@ -53,12 +57,12 @@ OPTIONS
-h | -help : show this help message
LICENSE
=======
-------
::
Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Copyright [2016] EMBL-European Bioinformatics Institute
Copyright [2016-2017] EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at
......@@ -70,9 +74,8 @@ LICENSE
See the License for the specific language governing permissions and limitations under the License.
CONTACT
=======
-------
::
Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
=================
init\_pipeline.pl
=================
NAME
====
----
::
init_pipeline.pl
SYNOPSIS
========
--------
::
init_pipeline.pl <config_module_or_filename> [<options_for_this_particular_pipeline>]
DESCRIPTION
===========
-----------
::
init_pipeline.pl is a generic script that is used to create+setup=initialize eHive pipelines from PipeConfig configuration modules.
USAGE EXAMPLES
==============
--------------
::
......@@ -35,7 +39,7 @@ USAGE EXAMPLES
init_pipeline.pl LongMult_conf -password <yourpassword> -first_mult 375857335 -second_mult 1111333355556666
OPTIONS
=======
-------
::
......@@ -46,12 +50,12 @@ OPTIONS
-h | --help : Show this help message
LICENSE
=======
-------
::
Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Copyright [2016] EMBL-European Bioinformatics Institute
Copyright [2016-2017] EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at
......@@ -63,9 +67,8 @@ LICENSE
See the License for the specific language governing permissions and limitations under the License.
CONTACT
=======
-------
::
Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
========================
load\_resource\_usage.pl
========================
NAME
====
----
::
load_resource_usage.pl
DESCRIPTION
===========
-----------
::
......@@ -22,7 +26,7 @@ DESCRIPTION
and then the data will be taken from your source and parsed from there.
USAGE EXAMPLES
==============
--------------
::
......@@ -39,7 +43,7 @@ USAGE EXAMPLES
load_resource_usage.pl -url mysql://username:secret@hostname:port/long_mult_test -source "bacct -l -C 2012/01/25/13:33,2012/01/25/14:44 |" -meadow_type LSF
OPTIONS
=======
-------
::
......@@ -50,12 +54,12 @@ OPTIONS
-meadow_type <type> : only used when -source is given. Tells which meadow type the source filename relates to. Defaults to the first available meadow (LOCAL being considered as the last available)
LICENSE
=======
-------
::
Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Copyright [2016] EMBL-European Bioinformatics Institute
Copyright [2016-2017] EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at
......@@ -67,9 +71,8 @@ LICENSE
See the License for the specific language governing permissions and limitations under the License.
CONTACT
=======
-------
::
Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
=========================
make\_branch\_glossary.pl
=========================
NAME
====
----
scripts/make\_branch\_glossary.pl
DESCRIPTION
===========
-----------
::
An internal eHive script for regenerating the document that lists all (most ?) of the dataflow patterns.
LICENSE
=======
-------
::
Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Copyright [2016] EMBL-European Bioinformatics Institute
Copyright [2016-2017] EMBL-European Bioinformatics Institute
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at
......@@ -28,9 +32,8 @@ LICENSE
See the License for the specific language governing permissions and limitations under the License.
CONTACT
=======
-------
::
Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates