Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
E
ensembl-hive
Manage
Activity
Members
Labels
Plan
Issues
0
Issue boards
Milestones
Iterations
Wiki
Requirements
Jira
Code
Merge requests
7
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Test cases
Artifacts
Deploy
Releases
Package Registry
Container Registry
Operate
Environments
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Insights
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ensembl-gh-mirror
ensembl-hive
Commits
a2b75d4a
Commit
a2b75d4a
authored
16 years ago
by
Javier Herrero
Browse files
Options
Downloads
Patches
Plain Diff
Update doc. Explain how to use eHive as a batch job throttlingmanager
parent
db6c9553
No related branches found
No related tags found
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
docs/Setup.txt
+44
-15
44 additions, 15 deletions
docs/Setup.txt
with
44 additions
and
15 deletions
docs/Setup.txt
+
44
−
15
View file @
a2b75d4a
...
...
@@ -4,6 +4,8 @@
perl DBI
Data::UUID (from CPAN.org)
ensembl
ensembl-analysis
ensembl-compara
ensembl-pipeline
1.2 Code checkout
...
...
@@ -16,9 +18,9 @@
cvs -d :ext:cvs.sanger.ac.uk:/nfs/ensembl/cvsroot co ensembl
ensembl-
pipeline code (for
Runnables)
ensembl-
analysis, ensembl-pipeline, ensembl-compara code (OPTIONAL, for using e!
Runnables)
cvs -d :ext:cvs.sanger.ac.uk:/nfs/ensembl/cvsroot co ensembl-pipeline
cvs -d :ext:cvs.sanger.ac.uk:/nfs/ensembl/cvsroot co ensembl-pipeline
ensembl-compara ensembl-analysis
ensembl-hive code
...
...
@@ -26,32 +28,45 @@
in tcsh
setenv BASEDIR /some/path/to/modules
setenv PERL5LIB ${BASEDIR}/ensembl/modules:${BASEDIR}/ensembl-pipeline/modules:${BASEDIR}/ensembl-genepair/modules:${BASEDIR}/bioperl-live
setenv PERL5LIB ${PERL5LIB}:${BASEDIR}/ensembl/modules
setenv PERL5LIB ${PERL5LIB}:${BASEDIR}/ensembl-hive/modules
setenv PERL5LIB ${PERL5LIB}:${BASEDIR}/ensembl-analysis/modules (OPTIONAL)
setenv PERL5LIB ${PERL5LIB}:${BASEDIR}/ensembl-compara/modules (OPTIONAL)
setenv PERL5LIB ${PERL5LIB}:${BASEDIR}/ensembl-pipeline/modules (OPTIONAL)
in bash
BASEDIR=/some/path/to/modules
PERL5LIB=${BASEDIR}/ensembl/modules:${BASEDIR}/ensembl-pipeline/modules:${BASEDIR}/ensembl-genepair/modules:${BASEDIR}/bioperl-live
PERL5LIB=${PERL5LIB}:${BASEDIR}/ensembl/modules
PERL5LIB=${PERL5LIB}:${BASEDIR}/ensembl-hive/modules
PERL5LIB=${PERL5LIB}:${BASEDIR}/ensembl-compara/modules (OPTIONAL)
PERL5LIB=${PERL5LIB}:${BASEDIR}/ensembl-analysis/modules (OPTIONAL)
PERL5LIB=${PERL5LIB}:${BASEDIR}/ensembl-pipeline/modules (OPTIONAL)
export PERL5LIB
2-
Configur
e database
2-
Setup a eHiv
e database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pick a mysql instance and create a database
mysql
admin -h ecs2 -P3361 -uensadmin -pxxxx
-e "create database hive
-
test1"
mysql
-h HOST -u USER -pSECRET
-e "create database hive
_
test1"
cd ~/src/ensembl_main
/ensembl-hive/sql
mysql -h
ecs2 -P3361 -uensadmin -pxxxx jessica_
hive_test1 < tables.sql
cd ${BASEDIR}
/ensembl-hive/sql
mysql -h
HOST -u USER -pSECRET
hive_test1 < tables.sql
3- Create location where worker and job STDOUT/STDERR is redirected to
3- (OPTIONAL) Create location where worker and job STDOUT/STDERR is redirected to
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a) create a working directory with enough disk space to hold hive worker output
mkdir /
nfs/ecs4/work2/ensembl/jessica/data/hive_output/jessica_
hive_test1/
mkdir /
scratch/
hive_test1/
b) insert into meta table
$outdir = '/nfs/ecs4/work2/ensembl/jessica/data/hive_output/jessica_hive_test1/'
$dba->get_MetaContainer->store_key_value('hive_output_dir', $outdir);
4- Create pipeline graph
$outdir = '/scratch/hive_test1/'
mysql -h HOST -u USER -pSECRET hive_test1 \
-e "INSERT INTO meta(meta_key, meta_value) VALUES ('hive_output_dir', '$outdir')"
4a- Create pipeline graph
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a) write RunnableDB modules to process data
...
...
@@ -61,7 +76,21 @@ mysql -h ecs2 -P3361 -uensadmin -pxxxx jessica_hive_test1 < tables.sql
done before another part of pipeline needs to 'unblock'
e) insert starting job(s) into analysis_job table to kick off pipeline
4b- To use the eHive as a simple batch job throttlingmanager
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a) Create one analysis for the SystemCmd module
mysql -h HOST -u USER -pSECRET hive_test1 \
-e "INSERT INTO analysis(logic_name, module) VALUES ('SysmtemCmd', 'Bio::EnsEMBL::Hive::RunnableDB::SystemCmd')"
b) Add as many jobs as needed
mysql -h HOST -u USER -pSECRET hive_test1 \
-e "INSERT INTO analysis_job (analysis_id, input_id) VALUES ('1', 'echo 1')"
5) Run hive (queen and workers) through a beekeeper
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
eg: lsf_beekeeper.pl -url mysql://ensadmin:xxxx@ecs2:3361/jessica_hive_test1 -loop
eg: lsf_beekeeper.pl -url mysql://USER:SECRET@HOST/hive_test1 -loop
This diff is collapsed.
Click to expand it.
Preview
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment