Skip to content

GitLab

Commit ace4e6a0 authored by Chandra Deep Tiwari's avatar Chandra Deep Tiwari
Browse files

Merge branch 'master-external' into 'master' 

Master external merge

See merge request !137
parents 31bd0261 54443ddc
Pipeline #95593 passed with stages
in 11 minutes and 57 seconds
Hide whitespace changes
Inline Side-by-side
Showing with 61 additions and 94 deletions
source/ResOps/2019/Agenda-2019.rst
View file @ ace4e6a0
Agenda for ResOps
======================
ResOps Training & Workshop
==========================
.. note:: Trainee who wanted to try self-pace mode of the ResOps Training please click `here <https://bit.ly/resops-berlin>`_
..
The dates reserved for 2019 are:
+----------------+-----------------+---------+----------------------------------------------------------+---------------------+
| Date | Location | #places | Registration link | Registration Close |
+================+=================+=========+==========================================================+=====================+
| 21 Aug 2019 | Garden Room | 30 | `21 Aug Register <http://bit.ly/cc-resops-aug19>`_ | closed |
+----------------+-----------------+---------+----------------------------------------------------------+---------------------+
| 04 Sep 2019 | Training room 1 | 30 | `04 Sep Register <http://bit.ly/cc-resops-190904>`_ | closed |
+----------------+-----------------+---------+----------------------------------------------------------+---------------------+
| 13 Sep 2019 | Garden Room | 30 | `13 Sep Register <http://bit.ly/cc-resops-190913>`_ | closed |
+----------------+-----------------+---------+----------------------------------------------------------+---------------------+
| 02 Oct 2019 | Garden Room | 30 | `02 Oct Register <http://bit.ly/cc-resops-191002>`_ | closed |
+----------------+-----------------+---------+----------------------------------------------------------+---------------------+
| 18 Oct 2019 | Garden Room | 30 | `18 Oct Register <http://bit.ly/cc-resops-191018>`_ | closed |
+----------------+-----------------+---------+----------------------------------------------------------+---------------------+
Bookings are managed via eventbrite. If you're interested in a particular date and it's already full, please add your name to the waiting list, we'll notify you if a place becomes available.
If we don't get enough bookings for a particular date we may cancel it and ask people to re-subscribe for a different session.
If all the courses fill up we will schedule more. We may also increase the number of places available on the course as we gain experience with it, so again, make use of the waiting list.
Announcement
Introduction
------------
The ResOps course will not be delivered in-person for the foreseeable future, but is available as a self-paced course by viewing the `'external' <https://tsi-ccdoc.readthedocs.io/en/external/ResOps/2019/Agenda-2019.html>`_ branch.
ResOps training is mainly focus to research community who are willing to develop their cloud native skills. The ResOps course will not be delivered in-person for the foreseeable future, This course can be taken standalone, self-paced and for **FREE**.
Course objectives
-----------------
This workshop will provide some background to cloud computing and practical experience in building, deploying and running applications in cloud platforms - OpenStack, Google, Amazon and Azure. Using examples drawn from EMBL-EBI’s experiences in the life-sciences, but using tools and technologies that are cross-platform and cross-domain, attendees will come away with a knowledge as to the user-centric hybrid cloud strategy as well as practical aspects of deploying across different clouds and the architectural considerations around porting applications to a cloud.
This workshop will provide some background to cloud computing and practical experience in building, deploying and running applications in cloud platforms - OpenStack, Google, Amazon and Azure.
Using examples drawn from EMBL-EBI’s experiences in the life-sciences, but using tools and technologies that are cross-platform and cross-domain, attendees will come away with a knowledge as to the user-centric hybrid cloud strategy as well as practical aspects of deploying across different clouds and the architectural considerations around porting applications to a cloud.
Prerequisite
------------
* Must bring your laptops, we can’t provide machines.
* Your laptop should have an SSH client(for window user Putty or any choice e.g. WDL) and a valid public SSH key.
* You should be comfortable with basic Linux command-line usage and have a working knowledge of GIT.
* Account should be created on `Public GitLab <https://gitlab.com/users/sign_in>`_ Gitlab practical.
* We are providing sandbox over OpenStack cloud to play exercises. Workshop participants can use Nano editor by default. Other CLI editors are also available in the sandbox.
* Installation of Minikube and GIT, on your own laptop OR your VM* where you want to do your hand-on exercise.
* If you are choosing VM as your choice then please use at least 1vCPU, RAM 4GB and storage as per your choice.
* Understanding of Linux commands and GIT repository commands
* An account should be created on `Public GitLab <https://gitlab.com/users/sign_in>`_ CI/CD practical with GitLab.
* We are providing sandbox over OpenStack cloud to play exercises.
* Workshop participants can use Nano editor by default. Other CLI editors are also available in the sandbox.
Schedule and links
Course guidelines
------------------
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| Time | Duration | Topic |
+==========+==========+====================================================================================================================+
| 10.00 | 10 min | Setting up, signing in. |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 10.10 | 10 min | `Introduction to ResOps, Day schedule <../../_static/pdf/resops2019/Introduction.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 10.20 | 15 min | `Cloud 101 <../../_static/pdf/resops2019/Cloud-101_V1.6.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 10:35 | 20 min | `Porting apps into clouds <../../_static/pdf/resops2019/PortingResearchPipelinesintoClouds.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 10.55 | 15 min | Break |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 11:10 | 35 min | `Creating Containers with Docker <../../_static/pdf/resops2019/Creating-Containers-with-Docker.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 11.45 | 35 min | `Introduction to Gitlab <../../_static/pdf/resops2019/Introduction-to-Gitlab.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 12.20 | 60 min | Lunch - plus Docker and Gitlab practicals |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 13.20 | 45 min | `Kubernetes 101 <../../_static/pdf/resops2019/Kubernetes-101.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 14.05 | 30 min | `Kubernetes (Demo) <Kubernetes-Demo-2019.html>`_ |
| | | / `Overview <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 14.35 | 50 min | `Kubernetes Practical <Minikube-and-NGINX-Practical-2019.html>`_ |
| | | / `Overview <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 15.25 | 15 min | Break |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 15.40 | 70 min | `Advanced Kubernetes Practical <Scaling-up-Kubernetes.html>`_ |
| | | / `Overview <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| 16:50 | 10 min | Summary & Feedback |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| Homework | | `Advanced Kubernetes Reading <Important-considerations-for-research-pipelines.html>`_ |
| | | / `Overview <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
| Homework | | Continue with the exercises |
+----------+----------+--------------------------------------------------------------------------------------------------------------------+
.. +-------+----------+--------------------------------------------------------------------------------------------------------------------+
.. | 11.30 | 30 min | `Case study of porting Rfam into K8S <../../_static/pdf/resops2019/CaseStudyofPortingRfamintoCloud.pdf>`_ |
These resources are being developed as part of the EOSC-Life project. EOSC-Life has received funding from the European Union’s Horizon 2020 programme under grant agreement number 824087.
+----------+------------------------------------------------------------------------------------------------------------------+
| Duration | Topic |
+==========+==================================================================================================================+
| 30 min | `Cloud 101 <../../_static/pdf/resops2019/Cloud-101_V1.7.pdf>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 30 min | `Cloud 101 [EBI Only] <../../_static/pdf/resops2019/Cloud-101_V1.6.pdf>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 40 min | `Porting apps into clouds <../../_static/pdf/resops2019/PortingResearchPipelinesintoClouds.pdf>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 60 min | `Creating Containers with Docker <../../_static/pdf/resops2019/Creating-Containers-with-Docker.pdf>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 90 min | `Docker Practicals <Docker.html>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 60 min | `Introduction to Gitlab <../../_static/pdf/resops2019/Introduction-to-Gitlab.pdf>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 90 min | `GitLab Practicals <Gitlab.html>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 60 min | `Kubernetes 101 <../../_static/pdf/resops2019/Kubernetes-101.pdf>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 120 min | `Kubernetes (Demo) <Kubernetes-Demo-2019.html>`_ |
| | / `Kubernetes Practicals <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 45 min | `Overview <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
| | / `Kubernetes Practical <Minikube-and-NGINX-Practical-2019.html>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 180 min | `Overview <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
| | / `Advanced Kubernetes Practical <Scaling-up-Kubernetes.html>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
| 150 min | `Overview <../../_static/pdf/resops2019/KubernetesPracticals.pdf>`_ |
| | / `Advanced Kubernetes Reading <Important-considerations-for-research-pipelines.html>`_ |
+----------+------------------------------------------------------------------------------------------------------------------+
.. footer:: These resources are being developed as part of the EOSC-Life project. EOSC-Life has received funding from the European Union’s Horizon 2020 programme under grant agreement number 824087.
source/ResOps/2019/docker/01_running-containers.md
View file @ ace4e6a0
## Exercise 1: Running images ##
## Exercise 1: Running images
### Objective ###
### Objective
Learn how to run images, either one-off or as a running service
### Terminology ###
### Terminology
In docker-speak, an **image** is a file (or set of files) that contain the application you want to run. You can copy images around, upload them, download them etc.
A **container** is a process that is started from an image. You can use the same image to run multiple containers, in the same way that you can use the same executable with many different arguments.
So, an image corresponds to files, a container corresponds to processes.
### Running a simple command in a container ###
### Running a simple command in a container
Docker images have a _name_ and a _tag_. The default for the tag is 'latest', and can be omitted. If you ask docker to run an image that is not present on your system, it will download it from [hub.docker.com](https://hub.docker.com/) first, then run it.
......@@ -40,7 +40,7 @@ If we run the same command again, docker will find the image cached on our local
hello world
```
### Running an interactive command in an image ###
### Running an interactive command in an image
To do something more exciting, you might want an interactive shell, so you can poke around and do stuff. That's easy:
```
......@@ -61,7 +61,7 @@ This is a useful way of practicing and debugging the build process for an applic
When the container environment gets messy, perhaps because you've installed stuff you didn't really want, you quit the container, build your image with what you've learned so far, and start again, from _that_ image. This way, you progress step by step to a working image. Then, you optimise it!
### Making changes to a container ###
### Making changes to a container
In an interactive shell in a container, you can change the container contents. But the changes do not persist once you exit the container. If you re-run the image, you get a _new_ container, not a re-run of the one you modified. Let's create a file in /tmp, then exit and restart the image, and look for the file:
```
......@@ -116,7 +116,7 @@ If you know you don't want to re-start a container afer you've run it, you can t
> docker run --rm ubuntu echo 'hello world'
```
### Starting a long-running service in a container ###
### Starting a long-running service in a container
Containers are useful for running services, like web-servers etc. Many come packaged from the developers, so you can start one easily, but first you need to find the one you want to run. You can either search on [hub.docker.com](https://hub.docker.com/), or you can use the **docker search** command. Nginx is a popular web-server, let's look for that:
```
......@@ -231,10 +231,10 @@ pensive_booth
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
```
### Conclusion ###
You now know how to find a container that you want to run. You can start it, re-start it, run it as a detached service, attach a terminal to it for debugging, view the logs externally and even map ports between it and your host machine.
### Conclusion
You now know how to find a container that you want to run. You can start it, re-start it, run it as a detached service, attach a terminal to it for debugging, view the logs externally and even map ports between it and your host machine.
### Best practices ###
### Best practices
- avoid making too many interactive changes to containers, see later exercises for how to modify and build your own containers
- prefer official images over those built by third-parties. Docker runs with privileges, so you have to be a bit careful what you run.
tsi-cc/ResOps/scripts/kubespray/ansipoor.sh
View file @ ace4e6a0
......@@ -2,7 +2,7 @@
# Adding Minikube to the new VMs
# https://computingforgeeks.com/how-to-install-vnc-server-on-ubuntu-18-04-lts/
# xfce4-goodies xfce4
# xfce4 xfce4-goodies
cmd_minikube='sudo sed -i.bak -e "s#PasswordAuthentication no#PasswordAuthentication yes#g" /etc/ssh/sshd_config && sudo service sshd restart && grp=$(id -ng) && i=$(hostname -I | cut -d "." -f 4 | cut -d " " -f 1) && sudo useradd -m -g ${grp} -s /bin/bash -p $(openssl passwd -1 passw${i}rd) resops${i} && sudo apt-get update && sudo apt-get install -y socat nfs-common docker.io firefox vnc4server && sudo apt-get -y autoremove && sudo snap install kubectl --classic && curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo cp minikube /usr/local/bin && rm minikube && sudo adduser resops${i} sudo && sudo usermod -G docker -a ubuntu && sudo usermod -G docker -a resops${i} && git config --global credential.helper "cache --timeout=3600" && git config --global user.name "resops${i}" && git config --global user.email "resops${i}@localhost" && sudo mv /home/ubuntu/.gitconfig /home/resops${i}'
# Set environment to connect to certain project with the openrc script generated by Horizon
......@@ -38,14 +38,14 @@ done
wait
# enable VNC
#cmd_vnc='vncserver -kill :1 && echo "exec /usr/bin/startxfce4 &" >> ~/.vnc/xstartup && vncserver :1 -geometry 1024x768 -depth 24'
cmd_vnc='i=$(hostname -I | cut -d "." -f 4 | cut -d " " -f 1) && vncserver -kill :1 && echo "exec /usr/bin/startxfce4 &" >> ~/.vnc/xstartup && sudo cp -R ~/.vnc /home/resops${i}/ && sudo chown -R resops${i} /home/resops${i}/.vnc/'
#cmd_vnc='vnc4server -kill :1 && echo "exec /usr/bin/startxfce4 &" >> ~/.vnc/xstartup && vnc4server :1 -geometry 1024x768 -depth 24'
cmd_vnc='i=$(hostname -I | cut -d "." -f 4 | cut -d " " -f 1) && vnc4server -kill :1 && echo "exec /usr/bin/startxfce4 &" >> ~/.vnc/xstartup && sudo cp -R ~/.vnc /home/resops${i}/ && sudo chown -R resops${i} /home/resops${i}/.vnc/'
for worker in "${workers[@]}"; do
ORG1_IFS=${IFS}; IFS=', ' tokens=( ${worker} ) && worker=${tokens[${#tokens[@]}-2]}; IFS=${ORG1_IFS} && echo "Tokens: ${tokens[*]}." && echo "Worker: ${worker}." && echo
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=No -o UserKnownHostsFile=/dev/null -o ProxyCommand="ssh -W %h:%p -i ~/.ssh/id_rsa ubuntu@${bastion}" ubuntu@${worker} "echo 'warm up bastion server.'"
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=No -o UserKnownHostsFile=/dev/null -o ProxyCommand="ssh -W %h:%p -i ~/.ssh/id_rsa ubuntu@${bastion}" ubuntu@${worker} 'i=$(hostname -I | cut -d "." -f 4 | cut -d " " -f 1) && vncserver :1 -geometry 1024x768 -depth 24 <<EOF
ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=No -o UserKnownHostsFile=/dev/null -o ProxyCommand="ssh -W %h:%p -i ~/.ssh/id_rsa ubuntu@${bastion}" ubuntu@${worker} 'i=$(hostname -I | cut -d "." -f 4 | cut -d " " -f 1) && vnc4server :1 -geometry 1024x768 -depth 24 <<EOF
passw${i}rd
passw${i}rd
EOF'
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment