beekeeper.html 7.38 KB
Newer Older
Matthieu Muffato's avatar
Matthieu Muffato committed
1 2 3 4 5 6
<?xml version="1.0" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>beekeeper.pl</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
7
<link rev="made" href="mailto:vxd@glow.apple.com" />
Matthieu Muffato's avatar
Matthieu Muffato committed
8 9 10 11 12
</head>

<body style="background-color: white">


13 14 15 16 17 18 19 20

<h1 id="NAME">NAME</h1>

<pre><code>    beekeeper.pl [options]</code></pre>

<h1 id="DESCRIPTION">DESCRIPTION</h1>

<pre><code>    The Beekeeper is in charge of interfacing between the Queen and a compute resource or &#39;compute farm&#39;.
Matthieu Muffato's avatar
Matthieu Muffato committed
21
    Its job is to initialize/sync the eHive database (via the Queen), query the Queen if it needs any workers
22 23
    and to send the requested number of workers to open machines via the runWorker.pl script.

Matthieu Muffato's avatar
Matthieu Muffato committed
24
    It is also responsible for interfacing with the Queen to identify workers which died
25 26 27 28 29 30 31
    unexpectedly so that she can free the dead workers and reclaim unfinished jobs.</code></pre>

<h1 id="USAGE-EXAMPLES">USAGE EXAMPLES</h1>

<pre><code>        # Usually run after the pipeline has been created to calculate the internal statistics necessary for eHive functioning
    beekeeper.pl -url mysql://username:secret@hostname:port/ehive_dbname -sync

Matthieu Muffato's avatar
Matthieu Muffato committed
32
        # Do not run any additional Workers, just check for the current status of the pipeline:
33 34
    beekeeper.pl -url mysql://username:secret@hostname:port/ehive_dbname

Matthieu Muffato's avatar
Matthieu Muffato committed
35
        # Run the pipeline in automatic mode (-loop), run all the workers locally (-meadow_type LOCAL) and allow for 3 parallel workers (-total_running_workers_max 3)
36 37
    beekeeper.pl -url mysql://username:secret@hostname:port/long_mult_test -meadow_type LOCAL -total_running_workers_max 3 -loop

38
        # Run in automatic mode, but only restrict to running blast-related analyses with the exception of analyses 4..6
39 40
    beekeeper.pl -url mysql://username:secret@hostname:port/long_mult_test -analyses_pattern &#39;blast%-4..6&#39; -loop

Matthieu Muffato's avatar
Matthieu Muffato committed
41
        # Restrict the normal execution to one iteration only - can be used for testing a newly set up pipeline
42 43 44 45 46
    beekeeper.pl -url mysql://username:secret@hostname:port/long_mult_test -run

        # Reset failed &#39;buggy_analysis&#39; jobs to &#39;READY&#39; state, so that they can be run again
    beekeeper.pl -url mysql://username:secret@hostname:port/long_mult_test -analyses_pattern buggy_analysis -reset_failed_jobs

Matthieu Muffato's avatar
Matthieu Muffato committed
47
        # Do a cleanup: find and bury dead workers, reclaim their jobs
48 49 50 51 52 53 54 55
    beekeeper.pl -url mysql://username:secret@hostname:port/long_mult_test -dead</code></pre>

<h1 id="OPTIONS">OPTIONS</h1>

<h2 id="Connection-parameters">Connection parameters</h2>

<pre><code>    -reg_conf &lt;path&gt;       : path to a Registry configuration file
    -reg_type &lt;string&gt;     : type of the registry entry (&#39;hive&#39;, &#39;core&#39;, &#39;compara&#39;, etc - defaults to &#39;hive&#39;)
Matthieu Muffato's avatar
Matthieu Muffato committed
56
    -reg_alias &lt;string&gt;    : species/alias name for the Hive DBAdaptor
57 58 59 60 61 62 63 64 65
    -url &lt;url string&gt;      : url defining where hive database is located</code></pre>

<h2 id="Configs-overriding">Configs overriding</h2>

<pre><code>    -config_file &lt;string&gt;  : json file (with absolute path) to override the default configurations (could be multiple)</code></pre>

<h2 id="Looping-control">Looping control</h2>

<pre><code>    -loop                  : run autonomously, loops and sleeps
Matthieu Muffato's avatar
Matthieu Muffato committed
66 67 68 69
    -max_loops &lt;num&gt;       : perform max this # of loops in autonomous mode
    -keep_alive            : do not stop when there are no more jobs to do - carry on looping
    -job_id &lt;job_id&gt;       : run 1 iteration for this job_id
    -run                   : run 1 iteration of automation loop
70 71 72 73 74
    -sleep &lt;num&gt;           : when looping, sleep &lt;num&gt; minutes (default 1 min)</code></pre>

<h2 id="Current-Meadow-control">Current Meadow control</h2>

<pre><code>    -meadow_type &lt;string&gt;               : the desired Meadow class name, such as &#39;LSF&#39; or &#39;LOCAL&#39;
Matthieu Muffato's avatar
Matthieu Muffato committed
75 76 77
    -total_running_workers_max &lt;num&gt;    : max # workers to be running in parallel
    -submit_workers_max &lt;num&gt;           : max # workers to create per loop iteration
    -submission_options &lt;string&gt;        : passes &lt;string&gt; to the Meadow submission command as &lt;options&gt; (formerly lsf_options)
78 79 80 81 82
    -submit_log_dir &lt;dir&gt;               : record submission output+error streams into files under the given directory (to see why some workers fail after submission)</code></pre>

<h2 id="Worker-control">Worker control</h2>

<pre><code>    -analyses_pattern &lt;string&gt;  : restrict the sync operation, printing of stats or looping of the beekeeper to the specified subset of analyses
83
    -can_respecialize &lt;0|1&gt;     : allow workers to re-specialize into another analysis (within resource_class) after their previous analysis was exhausted
Matthieu Muffato's avatar
Matthieu Muffato committed
84
    -life_span &lt;num&gt;            : life_span limit for each worker
85
    -job_limit &lt;num&gt;            : #jobs to run before worker can die naturally
Matthieu Muffato's avatar
Matthieu Muffato committed
86 87
    -retry_throwing_jobs 0|1    : if a job dies *knowingly*, should we retry it by default?
    -hive_log_dir &lt;path&gt;        : directory where stdout/stderr of the hive is redirected
88 89 90 91 92
    -debug &lt;debug_level&gt;        : set debug level of the workers</code></pre>

<h2 id="Other-commands-options">Other commands/options</h2>

<pre><code>    -help                  : print this help
Matthieu Muffato's avatar
Matthieu Muffato committed
93 94
    -versions              : report both Hive code version and Hive database schema version
    -dead                  : detect all unaccounted dead workers and reset their jobs for resubmission
95
    -unkwn                 : detect all workers in UNKWN state and reset their jobs for resubmission (careful, they *may* reincarnate!)
Matthieu Muffato's avatar
Matthieu Muffato committed
96 97
    -alldead               : tell the database all workers are dead (no checks are performed in this mode, so be very careful!)
    -balance_semaphores    : set all semaphore_counts to the numbers of unDONE fan jobs (emergency use only)
98
    -no_analysis_stats     : don&#39;t show status of each analysis
Matthieu Muffato's avatar
Matthieu Muffato committed
99 100 101
    -worker_stats          : show status of each running worker
    -failed_jobs           : show all failed jobs
    -reset_job_id &lt;num&gt;    : reset a job back to READY so it can be rerun
102 103 104 105 106
    -reset_failed_jobs     : reset FAILED jobs of -analyses_filter&#39;ed ones back to READY so they can be rerun
    -reset_all_jobs        : reset ALL jobs of -analyses_filter&#39;ed ones back to READY so they can be rerun</code></pre>

<h1 id="LICENSE">LICENSE</h1>

107
<pre><code>    Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
Matthieu Muffato's avatar
Matthieu Muffato committed
108
<pre><code>    Copyright [2016-2018] EMBL-European Bioinformatics Institute
109

Matthieu Muffato's avatar
Matthieu Muffato committed
110
    Licensed under the Apache License, Version 2.0 (the &quot;License&quot;); you may not use this file except in compliance with the License.
111 112 113 114
    You may obtain a copy of the License at

         http://www.apache.org/licenses/LICENSE-2.0

Matthieu Muffato's avatar
Matthieu Muffato committed
115 116
    Unless required by applicable law or agreed to in writing, software distributed under the License
    is distributed on an &quot;AS IS&quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
117 118 119 120 121 122
    See the License for the specific language governing permissions and limitations under the License.</code></pre>

<h1 id="CONTACT">CONTACT</h1>

<pre><code>    Please subscribe to the Hive mailing list:  http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users  to discuss Hive-related questions or to be notified of our updates</code></pre>

Matthieu Muffato's avatar
Matthieu Muffato committed
123 124 125 126

</body>

</html>
127 128