index.rst 8.27 KB
Newer Older
1
2
3
4
.. tsi-cc documentation master file, created by
   sphinx-quickstart on Mon Nov 12 16:26:08 2018.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.
5
 :orphan:
6
7
8
9

Welcome to Cloud Consultancy Team
=================================

C.D. Tiwari's avatar
C.D. Tiwari committed
10
This site is maintained by `the Cloud Consultancy Team at EBI <https://www.ebi.ac.uk/seqdb/confluence/display/TSI/Cloud+Consultancy+Team>`__
Tony Wildish's avatar
Tony Wildish committed
11

12
13
14
Table of contents
^^^^^^^^^^^^^^^^^

David Yu Yuan's avatar
David Yu Yuan committed
15
* `ResOps Training`_ - ResOps Training
Tony Wildish's avatar
Tony Wildish committed
16
* `Recommended Tools`_ - a list of recommended tools, and to get the most out of them
17
* `EBI Cloud Portal`_ - the EBI multi-cloud way of deploying applications
Tony Wildish's avatar
Tony Wildish committed
18
19
20
* `Architectural Design Patterns`_ - how to build cloud-native applications
* `About this documentation`_
* `Contact us`_
21
22
* `Site map`_

David Yu Yuan's avatar
David Yu Yuan committed
23
24
25
ResOps Training
^^^^^^^^^^^^^^^

26
* Notice
David Yu Yuan's avatar
David Yu Yuan committed
27

28
  * Please check the course pre-requisite on the agenda page.
29

David Yu Yuan's avatar
David Yu Yuan committed
30
* `Agenda for ResOps 2019 <ResOps/2019/Agenda-2019.html>`_
31
* `Agenda for ResOps 2020 <ResOps/2019/Agenda-2020-HD.html>`_
David Yu Yuan's avatar
David Yu Yuan committed
32

Tony Wildish's avatar
Tony Wildish committed
33
34
Recommended Tools
^^^^^^^^^^^^^^^^^
35
36
37

* Ansible:

38
..  * `Ansible best practice <bestpractices/Ansible/playbooks_best_practices.html>`_
39

David Yu Yuan's avatar
David Yu Yuan committed
40
* Cloud Consultancy Team toolbox:
41
42
43

  * `How to install tools <Tech-tips/Cloud-Consulting-Team-toolbox.html>`_

David Yu Yuan's avatar
David Yu Yuan committed
44
45
46
47
* Common Workflow Language

  * `CWL self-contained training course <https://www.melbournebioinformatics.org.au/tutorials/tutorials/cwl/cwl/>`_

48
49
50
51
52
53
54
* Docker:

  * `Docker best practice <Tech-tips/Tips-and-tricks-with-docker.html>`_

* GitLab:

  * Overview
55
56
57
58
59
60

    * `How to create docker image <Tech-tips/DevOps-toolchain-docker.html>`_
    * `How to deploy ECP instance <Tech-tips/DevOps-toolchain-ecp.html>`_
    * `How to publish documentation <Tech-tips/DevOps-toolchain-readthedocs.html>`_
    * `GitLab best practice <bestpractices/GitLab/README.html>`_

61
  * Useful links
David Yu Yuan's avatar
David Yu Yuan committed
62

David Yu Yuan's avatar
David Yu Yuan committed
63
    * `GitLab CI with Maven <https://www.ebi.ac.uk/seqdb/confluence/display/EXT/GitLab+CI+with+Maven>`_
64

65
.. * Grafana:
66

67
68
69
* Google Cloud Platform:

  * `Accessing GCP node from CLI <Tech-tips/Accessing-GCP-node-from-CLI.html>`_
70

71
72
73
* HPC in the cloud:

  * `HPC with Slurm on GCP <Tech-tips/HPC-with-Slurm-on-GCP.html>`_
74
75
  * Azure CycleCloud

76
    * `HPC with CycleCloud on Azure <Tech-tips/HPC-with-Azure-CycleCloud.html>`_
77
78
79
80
81
    * Useful links:

      * `CycleCloud Overview <https://docs.microsoft.com/en-us/azure/cyclecloud/>`_
      * `Setting up CycleCloud as a container <https://docs.microsoft.com/en-us/azure/cyclecloud/container-distribution>`_

82
83
* Kubernetes:

David Yu Yuan's avatar
David Yu Yuan committed
84
  * `Deployment of Kubernetes cluster on various clouds <Tech-tips/Deployment-of-Kubernetes-onto-clouds.html>`_
85
  * `Kubeflow for Machine Learning <Tech-tips/Kubeflow.html>`_
86
87
  * Useful links:

88
..  * `How to set up Container-as-a-service <Tech-tips/Installing-Docker-Kubernetes-for-CaaS.html>`_
89
..    * `CaaS Service Description <https://www.ebi.ac.uk/seqdb/confluence/display/CAAS/CaaS+Service+Description>`_
90

91
92
.. * Logstash (ELK):

93
94
95
96
97
98
99
100
* OneData:

  * `Overview <https://onedata.org/#/home/documentation/doc/getting_started/what_is_onedata.html>`_
  * `Oneclient <https://onedata.org/#/home/documentation/doc/using_onedata/oneclient.html>`_
  * `User Guide <https://onedata.org/#/home/documentation/doc/user_guide.html>`_
  * `Onedata API <https://onedata.org/#/home/api/latest/onezone>`_
  * EBI setup details
  * `EBI onedata portal <https://embl.tk>`_
101

102
.. * Prometheus:
103
104
105
106
107

* Terraform:

  * `Best Practice <Tech-tips/Tips-and-tricks-with-Terraform.html>`_

C.D. Tiwari's avatar
C.D. Tiwari committed
108
* Supervisor:
C.D. Tiwari's avatar
C.D. Tiwari committed
109

C.D. Tiwari's avatar
C.D. Tiwari committed
110
111
112
  * Description: Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.
  * `Reference documentation <http://supervisord.org/>`_

113
114
115
116
117
118
119
120
121
122
EBI Cloud Portal
^^^^^^^^^^^^^^^^

* `EMBL-EBI Cloud Portal documentation <https://ebi-cloud-portal.readthedocs.io/en/latest/>`_

  * `Using the EMBL-EBI Cloud Portal <https://ebi-cloud-portal.readthedocs.io/en/latest/using_the_portal.html>`_
  * `Packaging Applications for the EMBL-EBI Cloud Portal <https://ebi-cloud-portal.readthedocs.io/en/latest/applications_packaging.html>`_
  * `Avoid security credentials on git public repository <https://ebi-cloud-portal.readthedocs.io/en/latest/securing_credentials.html>`_
  * `API Endpoint documentation <https://ebi-cloud-portal.readthedocs.io/en/latest/api_endpoints.html>`_

Tony Wildish's avatar
Tony Wildish committed
123
124
Architectural Design Patterns
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
125

C.D. Tiwari's avatar
C.D. Tiwari committed
126
* `How to convert monolithic design to microservices <architecture/monolithic_to_microservice/create_microservices.html>`_
127
128
129

.. Coming soon...
.. * Best practices
130
131
132
.. * Types of architecture approach at EBI
.. * Better way of doing
.. * Deployment design/architecture
133

Tony Wildish's avatar
Tony Wildish committed
134
135
About this documentation
^^^^^^^^^^^^^^^^^^^^^^^^
136
This documentation is `maintained at in a gitlab repository at EBI <https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc>`_. Feel free to clone a local copy. You would need to run a local build to generate a local HTML site from the source files of `*.md` and `*.rst`. Install Sphinx with the instructions of `Getting started with sphinx <https://docs.readthedocs.io/en/latest/intro/getting-started-with-sphinx.html>`_. Follow the instructions of `Using markdown with sphinx <Tech-tips/DevOps-toolchain-readthedocs.html#using-markdown-with-sphinx>`_ and `Local build with Sphinx <Tech-tips/DevOps-toolchain-readthedocs.html#local-build-with-sphinx>`_ to generate your local site.
Tony Wildish's avatar
Tony Wildish committed
137
138
139

The **master** branch will always be the latest, most up-to-date version. Other versions may be available through the selection drop-down on the bottom-left corner of this page.

Tony Wildish's avatar
Tony Wildish committed
140
If you find any mistakes, have suggestions for improvement, have questions, or wish to talk to us about your project, please `contact us <mailto:cloud-consultants@ebi.ac.uk>`_. Pull requests or `issues filed against the git repository <https://gitlab.ebi.ac.uk/TSI/tsi-ccdoc/issues>`_ are also welcome.
141

Tony Wildish's avatar
Tony Wildish committed
142
143
144
Contact us
^^^^^^^^^^
You can `contact us by email <mailto:cloud-consultants@ebi.ac.uk>`_ or come find us in pod 14 of the South Building.
145
146
147
148
149
150
151
152

Site map
^^^^^^^^

.. toctree::
   :maxdepth: 2

   Presentation/Presentation.rst
153

154
   ResOps/2019/Agenda-2019.rst
David Yu Yuan's avatar
David Yu Yuan committed
155
   ResOps/2019/Demo-Creating-VMs-with-Terraform.rst
Tony Wildish's avatar
Tony Wildish committed
156
157
   ResOps/2019/Docker.md
   ResOps/2019/docker/01_running-containers.md
158
159
160
161
   ResOps/2019/docker/02_running-a-biobox-container.md
   ResOps/2019/docker/03_creating-a-docker-image.md
   ResOps/2019/docker/04_optimising-docker-images.md
   ResOps/2019/docker/05_using-the-builder-pattern.md
162
163
164
   ResOps/2019/docker/06_running-as-root.md
   ResOps/2019/docker/07_using-metadata.md
   ResOps/2019/docker/08_cleaning-up.md
165
166
167
168
169
170
171
   ResOps/2019/Gitlab.md
   ResOps/2019/gitlab/01_gitlab-account-setup.md
   ResOps/2019/gitlab/02_create-sample-project.md
   ResOps/2019/gitlab/03_run-docker-image.md
   ResOps/2019/gitlab/04_using-git-tags.md
   ResOps/2019/gitlab/05_add-further-steps.md
   ResOps/2019/gitlab/06_change-order-of-steps.md
Tony Wildish's avatar
Tony Wildish committed
172
   ResOps/2019/gitlab/07_pass-build-secrets.md
Soumyadip De's avatar
Soumyadip De committed
173
174
175
176
   ResOps/2019/gitlab/Error-tracking.rst
   ResOps/2019/gitlab/Github-pipeline.rst
   ResOps/2019/gitlab/Gitlab-DevOps.rst
   ResOps/2019/gitlab/Tracing.rst
Soumyadip De's avatar
Soumyadip De committed
177
   ResOps/2019/gitlab/auto-devops-tutorial.md
178
   ResOps/2019/Important-considerations-for-research-pipelines.rst
179
   ResOps/2019/Kubernetes-Demo-2019.rst
180
   ResOps/2019/Minikube-and-NGINX-Practical-2019.rst
David Yu Yuan's avatar
David Yu Yuan committed
181
   ResOps/2019/Scaling-up-Kubernetes.rst
182
   Tech-tips/Accessing-GCP-node-from-CLI.rst
David Yu Yuan's avatar
David Yu Yuan committed
183
   Tech-tips/Arvados-on-kubernetes.rst
184
185
186
187
   Tech-tips/Cloud-Consulting-Team-toolbox.md
   Tech-tips/DevOps-toolchain-docker.rst
   Tech-tips/DevOps-toolchain-ecp.rst
   Tech-tips/DevOps-toolchain-readthedocs.rst
188
   Tech-tips/HPC-with-Azure-CycleCloud.rst
189
   Tech-tips/HPC-with-Slurm-on-GCP.rst
David Yu Yuan's avatar
David Yu Yuan committed
190
   Tech-tips/Installing-openstack-cli-on-mac-os-x.md
191
   Tech-tips/Kubeflow.rst
192
   Tech-tips/Tips-and-tricks-with-docker.md
David Yu Yuan's avatar
David Yu Yuan committed
193
   Tech-tips/Tips-and-tricks-with-Terraform.md
194
.. Tech-tips/Deployment-of-Kubernetes-onto-clouds.rst
195
..   Tech-tips/Installing-Docker-Kubernetes-for-CaaS.rst
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
..   Rfam-cloud/Requirements-and-status-2018.11.08.md
..   Rfam-cloud/Requirements-and-status-2018.11.02.md
..   Rfam-cloud/Requirements-and-status-2018.10.26.md
..   Rfam-cloud/Requirements-and-status-2018.10.22.md
..   Rfam-cloud/Requirements-and-status-2018.10.09.md
..   Rfam-cloud/Prototype-of-nfs-and-k8s-cluster.md
..   Rfam-cloud/Quick-reference-for-rfam-cloud.md
..   Rfam-cloud/Rfam-tmp-results.md


.. Indices and tables
.. ==================

.. * :ref:`genindex`
.. * :ref:`modindex`
.. * :ref:`search`