Merge branch 'latest' into 'master'

Latest

See merge request !72

source/ResOps/2019/Agenda-2019.rst

@@ -9,17 +9,15 @@ The dates reserved for 2019 are:
 +----------------+-----------------+---------+----------------------------------------------------------+---------------------+
 |  Date          | Location        | #places | Registration link                                        | Registration Close  |
 +================+=================+=========+==========================================================+=====================+
-| 31 Jul 2019    | Garden Room     |    30   |    `31 July Register <http://bit.ly/cc-resops>`_         | 21 Jul 2019         |
-+----------------+-----------------+---------+----------------------------------------------------------+---------------------+
 | 21 Aug 2019    | Garden Room     |    30   |    `21 Aug Register <http://bit.ly/cc-resops-aug19>`_    | 11 Aug 2019         |
 +----------------+-----------------+---------+----------------------------------------------------------+---------------------+
-| 04 Sep 2019    | Training room 1 |    30   |    `04 Sep Register <http://bit.ly/cc-resops-sep19>`_    | 25 Aug 2019         |
+| 04 Sep 2019    | Training room 1 |    30   |    `04 Sep Register <http://bit.ly/cc-resops-190904>`_   | 25 Aug 2019         |
 +----------------+-----------------+---------+----------------------------------------------------------+---------------------+
-| 13 Sep 2019    | Garden Room     |    30   |    `13 Sep Register <http://bit.ly/cc-resops-sep19>`_    | 03 Sep 2019         |
+| 13 Sep 2019    | Garden Room     |    30   |    `13 Sep Register <http://bit.ly/cc-resops-190913>`_   | 03 Sep 2019         |
 +----------------+-----------------+---------+----------------------------------------------------------+---------------------+
-| 02 Oct 2019    | Garden Room     |    30   |    `02 Oct Register <http://bit.ly/cc-resops-oct19>`_    | 22 Sep 2019         |
+| 02 Oct 2019    | Garden Room     |    30   |    `02 Oct Register <http://bit.ly/cc-resops-191002>`_   | 22 Sep 2019         |
 +----------------+-----------------+---------+----------------------------------------------------------+---------------------+
-| 18 Oct 2019    | Garden Room     |    30   |    `18 Oct Register <http://bit.ly/cc-resops-oct19>`_    | 08 Oct 2019         |
+| 18 Oct 2019    | Garden Room     |    30   |    `18 Oct Register <http://bit.ly/cc-resops-191018>`_   | 08 Oct 2019         |
 +----------------+-----------------+---------+----------------------------------------------------------+---------------------+
 
 Bookings are managed via eventbrite. If you're interested in a particular date and it's already full, please add your name to the waiting list, we'll notify you if a place becomes available.

source/ResOps/2019/Gitlab.md

@@ -12,8 +12,7 @@ The easy way to get access to the example files used in the exercises is to clon
 ```
 
 The exercises are:
- - [Log into gitlab, set up your SSH key](gitlab/01_gitlab-account-setup.md)
- - [(Optional) Configure email notifications from gitlab](gitlab/01a_configure-email-notification.md)
+ - [Log into gitlab, set up your account](gitlab/01_gitlab-account-setup.md)
  - [Create a new project and import the example code](gitlab/02_create-sample-project.md)
  - [Download and run the docker image from your gitlab project](gitlab/03_run-docker-image.md)
  - [Use git tags to create a named version of a docker image](gitlab/04_using-git-tags.md)

source/ResOps/2019/gitlab/01_gitlab-account-setup.md

@@ -6,22 +6,72 @@ Get your account ready for you to use it.
 
 ### Introduction ###
 
-Before you can make use of your gitlab account, you need to upload your SSH key to it. Here's how:
+Before you can make use of your gitlab account, there are a few things you should do to make it more useful:
 
-- Click on your username or avatar in the top right-hand corner, and select **Settings**
+- create a Personal Access Token (mandatory if you want to do the exercises)
+- upload your SSH key (optional)
+- configure email notifications (optional)
+
+### Create a Personal Access Token ###
+
+This is a way to access gitlab.ebi.ac.uk if you have two-factor authentication turned on, which you should have by default. With 2FA, you can't use your normal password to authenticate a `git push`, instead you need to use a **Personal Access Token**, which you create in the gitlab website.
+
+Click on your username or avatar in the top right-hand corner, and select **Settings**
 
 ![](/static/images/resops2019/gitlab-01-settings.png)
 
-- Click on **SSH Keys** on the navigation bar on the left
+Click on **Access Tokens** on the navigation bar on the left, then fill in the form
+- give your token a name, e.g. **my-pat**
+- set an expiry date. One year in the future is a good value to use
+- select **api**, **write_repository** and **read_registry** for the **scope**
+- Then click the **Create personal access token** button
+
+![](/static/images/resops2019/gitlab-02-access-token.png)
+
+Make a note of the access token you see on the next screen, keep it somewhere safe. This is almost as powerful as your password, and should be treated with the same level of caution. If you lose it, you won't be able to recover it from the web interface, but you can create another one easily enough.
+
+![](/static/images/resops2019/gitlab-03-access-token-created.png)
+
+N.B. It's good hygiene to create separate tokens for separate purposes. That way you can revoke the access token for one use-case, if needed, without affecting others. Don't share tokens either, the token belongs to _you_, not to individual projects, so giving it to someone gives them access to all your projects, not just to any one project.
+
+### Upload your SSH key (optional!) ###
+
+This is useful to help you avoid having to type in your password every time you commit code to your projects. However, SSH access to gitlab.ebi.ac.uk is only available via the EBI network, not from outside, so its usefulness is a bit limited. Also, you will need to know how to use an SSH agent to cache your password. Google has a number of tutorials on that.
+
+Click on your username or avatar in the top right-hand corner, and select **Settings**
+
+Click on **SSH Keys** on the navigation bar on the left
 
 ![](/static/images/resops2019/gitlab-01-ssh-keys.png)
 
-- Follow the instructions on the right to upload your public SSH key. Not your private key, only the public one!
+Follow the instructions on the right to upload your public SSH key. Not your private key, only the public one!
+
+### Configure email notifications (optional!) ###
+
+You can control the email notifications you get from gitlab in great detail. Here's how to customise them:
+
+Click on your username or avatar in the top right-hand corner, and select **Settings**
+
+Click on **Notifications** on the navigation bar on the left, then on the button under **Global notification level**
+
+![](/static/images/resops2019/gitlab-05-email-notifications.png)
+
+Check any/all boxes that might interest you. I've checked all mine, and will uncheck as/when I decide I don't want to get notifications anymore.
+
+![](/static/images/resops2019/gitlab-05-email-option-panel.png)
+
+Elsewhere on the page you'll see that you can choose at the Group or Project level which set of notification presets you want for that group or project.
+
+That's it, you don't need to save the options, they're saved automatically.
 
 ### Conclusion ###
 
 You should now be able to push to your gitlab repositories without having to log in with every commit, exactly as with github and other hosting services.
 
+You will get notification by email for any interesting events in your repositories, and can tune that as needed to control the noise.
+
 ### Best Practices ###
 
 Ideally you should use a different SSH key for every service (gitlab, github, bitbucket, whatever) that requires you to register with a key. This limits the damage if ever one of those keys is compromised.
+
+Likewise, use different access tokens for different purposes, with only the level of access required.
\ No newline at end of file

source/ResOps/2019/gitlab/01a_configure-email-notification.md

-## Exercise 1a: (Optional) Configure email notification ##
-
-### Objective ###
-
-Customise your email notification settings
-
-### Introduction ###
-
-You can control the email notifications you get from gitlab in great detail. Here's how to customise them:
-
-- Click on your username or avatar in the top right-hand corner, and select **Settings**
-- Click on **Notifications** on the navigation bar on the left, then on the button under **Global notification level**
-
-![](/static/images/resops2019/gitlab-05-email-notifications.png)
-
-- Check any/all boxes that might interest you. I've checked all mine, and will uncheck as/when I decide I don't want to get notifications anymore.
-
-![](/static/images/resops2019/gitlab-05-email-option-panel.png)
-
-Elsewhere on the page you'll see that you can choose at the Group or Project level which set of notification presets you want for that group or project.
-
-That's it, you don't need to save the options, they're saved automatically.
\ No newline at end of file

source/ResOps/2019/gitlab/02_create-sample-project.md

@@ -13,8 +13,10 @@ Create a new project and import the sample code, see the build process it trigge
   - set the project visibility to 'internal' or 'public' if you want to share it with others
   - **do not* click the 'Initialize repository with README' button
 - Click 'Create Project'
+  - that's all you need to do in the browser, you don't need to follow the instructions you see on-screen next, instead do what it says below.
 
-- Do the **git config** steps:
+- Log into your virtual machine, or, if you're using your laptop, open a terminal window
+- Do the **git config** steps, this stops git from prompting you if you haven't set these fields before. You only need to do it once per machine:
 ```
 git config --global user.name "YOUR NAME"
 git config --global user.email "YOUR_USERNAME@ebi.ac.uk"
@@ -28,13 +30,19 @@ cd ~/gitlab-test
 git init
 ```
 
-- Add the code to your repository, commit it, and push it to the server:
+- Add the code to your repository and commit it:
 
 ```
 tar xvf ~/tsi-ccdoc/tsi-cc/ResOps/scripts/gitlab/tiny-test.tar
-git remote add origin git@gitlab.ebi.ac.uk:YOUR_GITLAB_USERNAME/YOUR_PROJECT_NAME.git
+git remote add origin https://gitlab.ebi.ac.uk/YOUR_GITLAB_USERNAME/YOUR_PROJECT_NAME.git
 git add .
 git commit -m "Initial commit"
+```
+
+- Tell git to cache your password for one hour. This saves you typing it in all the time whenever you make a push to the server. This is only needed when you use the **https** URL, with the SSH URL you can use an ssh-agent to cache the connection information instead. However, the SSH URL isn't accessible outside the EBI network, so we use the https URL for portability here:
+
+```
+git config --global credential.helper "cache --timeout=3600"
 git push -u origin master
 ```
 
@@ -44,7 +52,7 @@ Then go to the **CI/CD** -> **Pipelines** tab and watch the progress of your pip
 
 The build fails because the code which is uploaded points to my personal project, which only I have access to.
 
-To fix this, edit the **.gitlab-ci.yml** file, change the **APPLICATION** name to the name of your project, and the **REGISTRY_USER** to your gitlab username. Save the file, commit it to git, and push it to the server again.
+To fix this, edit the **.gitlab-ci.yml** file, change the **APPLICATION** name to the name of your project, and the **REGISTRY_USER** to your gitlab username. Save the file, commit it to git, and push it to the server again. This time, you only need a ```git push```, with no extra arguments.
 
 Go to the **CI/CD** -> **Pipelines** tab again, this time it should succeed.
 

source/ResOps/2019/gitlab/03_run-docker-image.md

@@ -38,7 +38,9 @@ Now that you're logged into the registry you can even push images there by hand,
 
 ### Deploy tokens ###
 
-If you want to run this docker image in a batch script, or a web service, you won't want to have to log into the registry on every machine you use, or every time it gets rebooted. Instead, if you only want to _use_ the images, but not to _upload_ them from the coomand line, you want a way to have read-only access to the registry. Gitlab provides **deploy-tokens** for that purpose, follow the link on the Registry page to learn more about them.
+If you want to run this docker image in a batch script, or a web service, you won't want to have to log into the registry on every machine you use, or every time it gets rebooted. Instead, if you only want to _use_ the images, but not to _upload_ them from the coomand line, you want a way to have read-only access to the registry. Gitlab provides **deploy-tokens** for that purpose, which are basically the same as the access token you've already created in the first exercise. For running batch jobs, you should create another, giving it only **read_registry** access, nothing else. You can then safely publish this anywhere you like, the worst people can do with it is to download your images.
+
+You'll need to make a note of the expiry date of your deploy tokens, and set a reminder to create a new one well in advance if you're running in a production environment. Alternatively, if your containers are fit for public release, you can push them to a public service like dockerhub instead.
 
 ### Conclusion ###
 

source/ResOps/2019/gitlab/04_using-git-tags.md

@@ -49,6 +49,14 @@ Hello World
 Compiled on Tue Jul 30 10:58:09 UTC 2019
 ```
 
+### Bonus exercise: git branches ###
+
+What happens if you use a git branch, instead of a tag? The CI/CD pipeline will still fire, and will know the branch name, so it uses that in the docker image tag. However, since your **.gitlab-ci.yml** file only runs the **test** stage for git _tags_ and not for git _branches_, you will only get a two-stage pipeline.
+
+Can you modify the YAML file so it runs for branches too? [Check the documentation](https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic) for help.
+
+If you're on a branch, and you use a tag too, what do you think gitlab will do? Try it and find out.
+
 ### Conclusion ###
 
 You now know how to control the activities in a build when using git tags, and how to use those tags to automatically tag the docker image with the same name.

source/index.rst

@@ -156,7 +156,6 @@ Site map
    ResOps/2019/docker/06_cleaning-up.md
    ResOps/2019/Gitlab.md
    ResOps/2019/gitlab/01_gitlab-account-setup.md
-   ResOps/2019/gitlab/01a_configure-email-notification.md
    ResOps/2019/gitlab/02_create-sample-project.md
    ResOps/2019/gitlab/03_run-docker-image.md
    ResOps/2019/gitlab/04_using-git-tags.md

tsi-cc/ResOps/scripts/docker/bioboxes/output_data/.gitignore

+*