eHive_install_usage.txt 5.18 KB
Newer Older
1 2 3 4
	eHive installation, setup and usage.

1. Download and install the necessary external software:

5
1.1. Perl 5.10 or higher, since eHive code is written in Perl
6 7 8 9 10 11 12 13 14 15 16 17

	# see http://www.perl.com/download.csp

1.2. MySQL 5.1 or higher

	eHive keeps its state in a MySQL database, so you will need
	(1) a MySQL server installed on the machine where you want to maintain the state and
	(2) MySQL clients installed on the machines where the jobs are to be executed.
	MySQL version 5.1 or higher is recommended to maintain compatibility with Compara pipelines.

	see http://dev.mysql.com/downloads/

18
1.3. Perl DBI API version 1.6 or higher
19 20 21 22 23 24 25 26
	Perl database interface that includes API to MySQL

	# see http://dbi.perl.org/


2. Download and install essential and optional packages from BioPerl and EnsEMBL CVS

2.1. It is advised to have a dedicated directory where EnsEMBL-related packages will be deployed.
27
Unlike DBI modules that can be installed system-wide by the system administrator,
28 29 30 31 32 33 34 35
you will benefit from full (read+write) access to the EnsEMBL files/directories,
so it is best to install them under your home directory. For example,
	
	$ mkdir $HOME/ensembl_main

It will be convenient to set a variable pointing at this directory for future use:

# using bash syntax:
36
	$ export ENSEMBL_CVS_ROOT_DIR="$HOME/ensembl_main"
37 38 39 40
		#
		# (for best results, append this line to your ~/.bashrc configuration file)

# using [t]csh syntax:
41
	$ setenv ENSEMBL_CVS_ROOT_DIR "$HOME/ensembl_main"
42 43 44 45 46
		#
		# (for best results, append this line to your ~/.cshrc or ~/.tcshrc configuration file)

2.2. Change into your ensembl codebase directory:

47
	$ cd $ENSEMBL_CVS_ROOT_DIR
48

49
2.3. Log into the EnsEMBL CVS server at Sanger (using "CVSUSER" for password):
50 51 52 53 54

	$ cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/ensembl login
	Logging in to :pserver:cvsuser@cvs.sanger.ac.uk:2401/cvsroot/ensembl
	CVS password: CVSUSER

55
2.4. Export ensembl and ensembl-hive CVS modules:
56

57 58
	$ cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/ensembl checkout ensembl
	$ cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/ensembl checkout ensembl-hive
59

60
2.5. In the likely case you are going to use eHive in the context of Compara pipelines,
61 62
	you will also need to install ensembl-compara:

63
	$ cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/ensembl checkout ensembl-compara
64

65
2.6. Add new packages to the PERL5LIB variable:
66 67

# using bash syntax:
68
	$ export PERL5LIB=${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/bioperl-live              # optional for eHive, but may be needed for EnsEMBL
69 70
	$ export PERL5LIB=${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/ensembl/modules
	$ export PERL5LIB=${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/ensembl-hive/modules
71
	$ export PERL5LIB=${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/ensembl-compara/modules   # only needed if you are planning to run Compara pipelines, see 2.5
72 73 74 75
		#
		# (for best results, append these lines to your ~/.bashrc configuration file)

# using [t]csh syntax:
76
	$ setenv PERL5LIB  ${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/bioperl-live             # optional for eHive, but may be needed for EnsEMBL
77 78
	$ setenv PERL5LIB  ${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/ensembl/modules
	$ setenv PERL5LIB  ${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/ensembl-hive/modules
79
	$ setenv PERL5LIB  ${PERL5LIB}:${ENSEMBL_CVS_ROOT_DIR}/ensembl-compara/modules  # only needed if you are planning to run Compara pipelines, see 2.5
80 81 82 83
		#
		# (for best results, append these lines to your ~/.cshrc or ~/.tcshrc configuration file)


84
3. Useful files and directories of the eHive repository.
85

86 87
3.1 In ensembl-hive/scripts we keep perl scripts used for controlling the pipelines.
    Adding this directory to your $PATH may make your life easier.
88

89 90
    * init_pipeline.pl is used to create hive databases, populate hive-specific and pipeline-specific tables and load data

91 92
    * seed_pipeline.pl is used to add new jobs to the pipeline.

93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113
    * beekeeper.pl is used to run the pipeline; send 'Workers' to the 'Meadow' to run the jobs of the pipeline

3.2 In ensembl-hive/modules/Bio/EnsEMBL/Hive/PipeConfig we keep example pipeline configuration modules that can be used by init_pipeline.pl .
    A PipeConfig is a parametric module that defines the structure of the pipeline.
    That is, which analyses with what parameters will have to be run and in which order.
    The code for each analysis is contained in a RunnableDB module.
    For some tasks bespoke RunnableDB have to be written, whereas some other problems can be solved by only using 'universal buliding blocks'.
    A typical pipeline is a mixture of both.

3.3 In ensembl-hive/modules/Bio/EnsEMBL/Hive/RunnableDB we keep 'universal building block' RunnableDBs:

    * SystemCmd.pm  is a parameter substitution wrapper for any command line executed by the current shell

    * SqlCmd.pm     is a parameter substitution wrapper for running any MySQL query or a session of linked queries
                    against a particular database (eHive pipeline database by default, but not necessarily)

    * JobFactory.pm is a universal module for dynamically creating batches of same analysis jobs (with different parameters)
                    to be run within the current pipeline

3.4 In ensembl-hive/modules/Bio/EnsEMBL/Hive/RunnableDB/LongMult we keep bespoke RunnableDBs for long multiplication example pipeline.