runnable_api.rst 6.45 KB
Newer Older
1 2 3 4 5 6 7

Runnable API
============

eHive exposes an interface for Runnables (jobs) to interact with the
system:

8 9 10 11
  - query their own parameters (see :ref:`parameters-in-jobs`),
  - control its own execution and report issues,
  - run system commands,
  - trigger some *dataflow* events (e.g. create new jobs).
12 13 14 15 16


Reporting and logging
---------------------

17 18 19 20
Jobs can log messages to the standard output with the
``$self->say_with_header($message, $important)`` method. However they are only printed
when the *debug* mode is enabled (see below) or when the ``$important`` flag is switched on.
They will also be prefixed with a standard prefix consisting of the
21
runtime context (Worker, Role, Job).
22 23 24 25 26 27 28 29 30 31

The debug mode is controlled by the ``--debug X`` option of
:ref:`script-beekeeper` and :ref:`script-runWorker`. *X* is an integer,
allowing multiple levels of debug, although most of the modules will only
check whether it is 0 or not.

``$self->warning($message)`` calls ``$self->say_with_header($message, 1)``
(so that the messages are printed on the standard output) but also stores
them in the database (in the ``log_message`` table).

32
To indicate that a Job has to be terminated earlier (i.e. before reaching
33 34
the end of ``write_output``), you can call:

35
- ``$self->complete_early($message)`` to mark the Job as *DONE*
36 37
  (successful run) and record the message in the database. Beware that this
  will trigger the *autoflow*.
38 39
- ``$self->complete_early($message, $branch_code)`` is a variation of the
  above that will replace the autoflow (branch 1) with a dataflow on the
40 41
  branch given.
- ``$self->throw($message)`` to log a failed attempt. The Job may be given
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61
  additional retries following the analysis' *max_retry_count* parameter,
  or is marked as *FAILED* in the database.

System interactions
-------------------

All Runnables have access to the ``$self->run_system_command`` method to run
arbitrary system commands (the ``SystemCmd`` Runnable is merely a wrapper
around this method).

``run_system_command`` takes two arguments:

#. The command to run, given as a single string or an arrayref. Arrayrefs
   are the preferred way as they simplify the handling of whitespace and
   quotes in the command-line arguments. Arrayrefs that correspond to
   straightforward commands, e.g. ``['find', '-type', 'd']``, are passed to
   the underlying ``system`` function as lists. Arrayrefs can contain shell
   meta-characters and delimiters such as ``>`` (to redirect the output to a
   file), ``;`` (to separate two commands that have to be run sequentially)
   or ``|`` (a pipe) and will be quoted and joined and passed to ``system``
62
   as a single string.
63 64 65 66 67 68 69
#. An hashref of options. Accepted options are:

   - ``use_bash_pipefail``: Normally, the exit status of a pipeline (e.g.
     ``cmd1 | cmd2`` is the exit status of the last command, meaning that
     errors in the first command are not captured. With the option turned
     on, the exit status of the pipeline will capture errors in any command
     of the pipeline, and will only be 0 if *all* the commands exit
70
     successfully.
71 72
   - ``use_bash_errexit``: Exit immediately if a command fails. This is
     mostly useful for cases like ``cmd1; cmd2`` where by default, ``cmd2``
73
     would always be executed, regardless of the exit status of ``cmd1``.
74 75
   - ``timeout``: the maximum number of seconds the command is allowed to
     run for. The exit status will be set to -2 if the command had to be
76
     aborted.
77 78 79

During their execution, jobs may certainly have to use temporary files.
eHive provides a directory that will exist throughout the lifespan of the
80 81
Worker with the ``$self->worker_temp_directory`` method. The directory is created
the first time the method is called, and deleted when the Worker ends. It is the Runnable's
82
responsibility to leave the directory in a clean-enough state for the next
83
Job (by removing some files, for instance), or to clean it up completely
84 85 86
with ``$self->cleanup_worker_temp_directory``.

By default, this directory will be put under /tmp, but it can be overriden
87 88
by setting the ``-worker_base_tmp_dir`` option for workers through their
resource classes. This can
89 90
be used to:

91
- use a faster filesystem (although /tmp is usually local to the machine),
92 93
- use a network filesystem (needed for distributed applications, e.g. over
  MPI). See :ref:`worker_temp_directory_name-mpi` in the :ref:`howto-mpi` section.
94

95
.. _runnable_api_dataflows:
96 97 98 99 100

Dataflows
---------

eHive is an *event-driven* system whereby agents trigger events that
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121
are immediately reacted upon. The main event is called "dataflow" (see
:ref:`dataflows` for more information). A dataflow event is made up of
two parts: An event, which is identified by a "branch number", with an
attached data payload, consisting of parameters. A Runnable can create
as many events as desired, whenever desired. The branch number can be
any integer, but note that "-2", "-1", "0", and "1" have special meaning
within eHive. -2, -1, and 0 are special branches for
:ref:`error handling <resource-limit-dataflow>`, and 1 is the autoflow branch.

.. warning::

    If a Runnable explicitly generates a dataflow event on branch 1, then
    no autoflow event will be generated when the Job finishes. This is
    unusual behaviour -- many pipelines expect and depend on autoflow
    coinciding with Job completion. Therefore, you should avoid explicitly
    creating dataflow on branch 1, unless no alternative exists to produce
    the correct logic in the Runnable. If you do override the autoflow by
    creating an event on branch 1, be sure to clearly indicate this in the
    Runnable's documentation.

Within a Runnable, dataflow events are performed via the ``$self->dataflow_output_id($data,
122 123 124 125
$branch_number)`` method.

The payload ``$data`` must be of one of these types:

126 127 128
- A hash-reference that maps parameter names (strings) to their values,
- An array-reference of hash-references of the above type, or
- ``undef`` to propagate the Job's input_id.
129

130
If no branch number is provided, it defaults to 1.
131

132 133
Runnables can also use ``dataflow_output_ids_from_json($filename, $default_branch)``.
This method simply wraps ``dataflow_output_id``, allowing external programs
134 135 136 137 138
to easily generate events. The method takes two arguments:

#. The path to a file containing one JSON object per line. Each line can be
   prefixed with a branch number (and some whitespace), which will override
   the default branch number.
139
#. The default branch number (defaults to 1).
140

141
Use of this is demonstrated in the Runnable :doxehive:`Bio::EnsEMBL::Hive::RunnableDB::SystemCmd`