Skip to content

Add an inactive proof-of-concept GitLab-CI config file

Marek Szuba requested to merge feature/gitlab-ci_poc into master

Created by: mkszuba

Description

Add an inactive (in case it isn't obvious, it has been deactivated by inserting '.inactive' into the correct file name) proof-of-concept GitLab-CI config file.

Unlike Ubuntu-based Travis builds, on GitLab-CI we use official Perl images from Docker Hub. These are Debian-based so no major changes have been required, support both perl-5.14 and perl-5.30 without any fuss (and even if they eventually removed images for some version we could always grab their relevant Dockerfiles and rebuild them ourselves), and since they only contain core Perl contain only a few most common non-core modules in default images (which is what this uses now) and only core Perl in case of slim ones, they give us better control over our dependency trees (i.e. much less risk of forgetting to add a non-core module to cpanfile because it was already there as distro package on the developer's machine).

What works:

  • the test suite runs and passes for both perl-5.14/sqlite and perl-5.30/mysql;
  • enable Coveralls reporting should work for 5.30/mysql jobs if an appropriate secret variable in GitLab CI configuration is set to the correct token from coveralls.io;
  • it is possible to trigger dependent jobs for downstream projects configured to run GitLab-CI.

What still needs to be done to achieve full feature parity with Travis builds:

  • tweak the condition of triggering dependent builds so that they appropriately cover pull requests.

Does not have to be implemented in GitLab-CI:

  • configuration of e-mail and Slack notifications - in GitLab this is done in projects, not at CI level.

What could be done even though it isn't done in Travis:

  • consider supporting dependent jobs in repositories which do not support GitLab-CI;
  • create our own build images with all the common dependencies already installed. GitLab repositories can function as Docker-compatible container registries so it is quite simple to create a new repo with this feature enabled, shove a couple of Dockerfiles into it which install the necessary common dependencies onto base Perl images, and set up CI for that repo so that the base images are periodically rebuilt in order to account for possible updates. Result: simpler API-level CI configuration, faster runs, less bandwidth wasted;
  • process cpanfiles in ensembl-io and ensembl-variation. These are problematic because they both depend on Bio::DB::BigFile, which requires messing with Kent libraries and linker flags in order to install properly, At present we get away with not processing them because unlike ensembl-compara (which luckily does not demand any difficult modules), Core unit tests depending on -io and -variation code only import modules which do not use either of the above. This of course is a lucky co-incidence and cannot be relied upon to last. We have already got scripts for dealing with both Bio::DB::BigFile and GD, in Otter for instance, so we could in principle address this right away. Then again, in my opinion it would be better to delegate this to the aforementioned custom build images and keep repository-specific CI scripting as lean as possible;
  • setting passwords for MySQL users. Trivial to do but in order to avoid having those passwords in plain text in .gitlab-ci.yml, we should either set them as GitLab project variables or encrypt them somehow.

Use case

Our Travis space has really been struggling to keep up with the load we impose on it, especially in ensembl-rest. Moreover, with Ubuntu 14.04 having reached end of life in April 2019 the "trusty" build environment is now deprecated, and perl-5.14 is not officially supported by Travis for Xenial and newer. Finally, with many features of Travis being reserved for paying customers we are forced do things not entirely efficiently: having to install all common dependencies every build, not running Travis on private repositories etc.

At the same time, we have at EBI an instance of Gitlab Enterprise Edition whose CI system (which has already been paid for as part of the Enterprise licence) offers more features than Travis CE, which integrates better with Jira than GitHub (to the best of my knowledge, GitHub Jira integration only works with the cloud edition of the latter) and which can either fully mirror GitHub repositories (there is also a possibility of mirroring whole projects - issues, PRs, you name it - but WebProd have told me that so far they have had problems enabling it without simultaneously letting people log in with GitHub account and therefore have decided to keep it off for now) or simply run CI jobs for repositories still located on GitHub, meaning we could use this without actually migrating off GH.

Benefits

Demonstrates that it is not only feasible but in fact quite straightforward to run Ensembl CI jobs on EBI GitLab.

Possible Drawbacks

Care should be taken to keep this file in sync with .travis.yml in the event of changes to the latter. For this reason, GitLab-CI configuration has been made as similar to Travis configuration as possible.

Testing

Have you added/modified unit tests to test the changes?

No.

If so, do the tests pass/fail?

N/A

Have you run the entire test suite and no regression was detected?

Neither the test suite itself nor Travis builds are affected by this. GitLab-CI builds succeed.

Merge request reports