Commit 51f6c220 authored by Brandon Walts's avatar Brandon Walts Committed by Matthieu Muffato
Browse files

tweaking doc

parent 9164c000
......@@ -41,6 +41,7 @@ User documentation
running_pipelines/running
running_pipelines/monitoring
running_pipelines/management
running_pipelines/tweaking
.. toctree::
......
===================================================================
Changing and Viewing pipeline configuration using tweak_pipeline.pl
===================================================================
eHive provides a facility to modify many aspects of a pipeline that is loaded into a hive database. This is the tweak_pipeline.pl script. Using this script, users can change the values of analysis or pipeline-wide parameters. This script can also change resource classes for analyses, and it can even be used to alter the dataflow structure of a pipeline.
Basic operation
===============
Typically, tweak_pipeline.pl is invoked with two sets of parameters: the pipeline database (passed as a url (``-url``) or as part of a registry configuration (``-reg_conf``)), and a statement written in the tweak language (for details, see the :ref:`tweak-language-reference`). The tweak language is designed to be intuitive, consisting of a verb (``-SET``, ``-SHOW``, ``-DELETE``, or ``-tweak``) followed by the name of the attribute and the attribute's new value (if appropriate). Some examples:
Setting or changing attributes
------------------------------
- Set or change the value of a pipeline-wide parameter:
``tweak_pipeline.pl -url sqlite:///my_hive_db -SET 'pipeline.param[take_time]=20'``
- Set or change the value of a hive meta-attribute:
``tweak_pipeline.pl -url sqlite:///my_hive_db -SET 'pipeline.hive_pipeline_name=new_name'``
- Set or change the resource class for a group of analyses, using pattern matching to match multiple analysis names:
``tweak_pipeline.pl -url sqlite:///my_hive_db -SET 'analysis[blast%].resource_class=himem'``
- Set or change dataflow for an analysis:
``tweak_pipeline.pl -url sqlite:///my_hive_db -SET 'analysis[some_analysis].flow_into={1=>"another_analysis"}'``
Viewing attributes
------------------
- View a hive meta-attribute:
``tweak_pipeline.pl -url sqlite:///my_hive_db -SHOW 'pipeline.hive_pipeline_name'``
- View the description for a resource class:
``tweak_pipeline.pl -url sqlite:///my_hive_db -SHOW 'resource_class[urgent].LSF'``
Deleting attributes
-------------------
- Delete an analysis-wide parameter:
``tweak_pipeline.pl -url sqlite:///my_hive_db -DELETE 'analysis[add_together].param[bar]'``
Remember that tweak_pipeline.pl only affects values in the hive database. In order to make tweaks permanent, the changes need to also be made in the corresponding PipeConfig file.
.. _tweak-language-reference:
Tweak language reference
========================
The tweak_pipeline.pl script uses a flexible language to specify which pipeline attributes to set, change, or show. Each tweak consists of a verb, followed by a description of the attribute. Finally, the new value for that attribute is given if required.
Verbs
-----
The verb can be one of four values:
- ``-SET``: changes the current value for the given attribute.
- ``-SHOW``: returns the current value for the given attribute.
- ``-DELETE``: deletes any value for the given attribute.
- ``-tweak``: generic invocation of a tweak.
- If ``-tweak`` is followed by an attribute name and a ``?``, then the current value of that attribute is returned - similar to ``-SHOW``. Example:
``tweak_pipeline.pl -url sqlite:///my_hive_db -tweak 'pipeline.hive_pipeline_name?'``
- If an attribute name is provided, along with a new value separated by ``=``, then the attribute's value is updated to the new value. This is the same as ``-SET``. Example:
``tweak_pipeline.pl -url sqlite:///my_hive_db -tweak 'pipeline.param[take_time]=20'``
- If tweak is followed by an attribute name and a ``#``, then the attribute is deleted. This is the same as ``-DELETE``. Example:
```tweak_pipeline.pl -url sqlite:///my_hive_db -tweak 'pipeline.param[take_time]#``
Attribute description
---------------------
The attribute being tweaked is identified using a two-part name, with the two parts separated by a dot.
- The first part identifies the "domain" of the attribute. This can be one of:
- pipeline
- analysis
- resource_class
- In the case of analysis or resource_class, the particular analysis or resource class is identified by placing the logic name in brackets like this:
``analysis[logic_name]`` e.g. ``analysis[dump_sequence]``
- The second part identifies the particular attribute within that domain to view, modify, or delete. Allowable values for this part depend on the domain:
+----------------+--------------------------------+-----------------------------------------+
| Domain | Possible attributes | Notes |
+================+================================+=========================================+
| pipeline | hive_auto_rebalance_semaphores | |
+ +--------------------------------+-----------------------------------------+
| | hive_pipeline_name | |
+ +--------------------------------+-----------------------------------------+
| | hive_sql_schema_version | display only |
+ +--------------------------------+-----------------------------------------+
| | hive_use_param_stack | |
+ +--------------------------------+-----------------------------------------+
| | param | Requires a parameter name in [brackets] |
+----------------+--------------------------------+-----------------------------------------+
| analysis | analysis_capacity | |
+ +--------------------------------+-----------------------------------------+
| | batch_size | |
+ +--------------------------------+-----------------------------------------+
| | can_be_empty | |
+ +--------------------------------+-----------------------------------------+
| | comment | |
+ +--------------------------------+-----------------------------------------+
| | dbID | display only |
+ +--------------------------------+-----------------------------------------+
| | failed_job_tolerance | |
+ +--------------------------------+-----------------------------------------+
| | flow_into | |
+ +--------------------------------+-----------------------------------------+
| | hive_capacity | |
+ +--------------------------------+-----------------------------------------+
| | max_retry_count | |
+ +--------------------------------+-----------------------------------------+
| | meadow_type | |
+ +--------------------------------+-----------------------------------------+
| | param | requires a parameter name in [brackets] |
+ +--------------------------------+-----------------------------------------+
| | priority | |
+ +--------------------------------+-----------------------------------------+
| | resource_class | |
+ +--------------------------------+-----------------------------------------+
| | tags | |
+ +--------------------------------+-----------------------------------------+
| | wait_for | |
+----------------+--------------------------------+-----------------------------------------+
| resource_class | meadow name (e.g. LSF) | |
+----------------+--------------------------------+-----------------------------------------+
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment