Patch test databases

Source: master branches of all relevant Ensembl repositories as of 10:00
today. This brings all test databases at hand to schema version 100.

Enscoresw 3267

Ameya Chaubal authored 5 years ago and Marek Szuba committed 5 years ago

Includes recent Variation schema updates.

Registry: add support for versioned ncbi_taxonomy DBs + improve support for versioned metadata DBs

Registry::get_species_and_object_type(): filter out versioned ensembl_metadata and ncbi_taxonomy DBs

Again, for ensembl_metadata this should have been part of ENSCORESW-3119.

For ensembl_metadata this should have been part of ENSCORESW-3119 but
alas, I missed it.

Not sure if ncbi_taxonomy is actually processed by this function but
let's play it safe, will have to confirm.

See ENSCORESW-3253. If both a versioned and an unversioned database are
present, use the versioned one if the version number matches that of the
software.

Add next-value buffering to Bio::EnsEMBL::Utils::Iterator

Fix fetch_all_by_outward_search

This is useful when working with coderefs that defer to
`$sth->fetchrow_arrayref`, which throws an error if called again after
having returned undef (which signifies the end of the data stream)

```
DBD::mysql::st fetchrow_arrayref failed: fetch() without execute()
```

Enable VGNC xrefs for callithrix_jacchus and papio_anubis [master]

E100 version bump

Travis: reactivate warnings about different API and schema versions

Deprecate Bio::EnsEMBL::DBSQL::GenomeContainer::get_total_length()

As per the Ensembl coding guidelines, i.e.: added a deprecation warning
to the method, tagged it as deprecated in the POD block, and listed it
as scheduled for removal in Ensembl 102.

Issue: ENSINT-283

Add an inactive proof-of-concept GitLab-CI config file

Patch non-core test databases to schema version 99

Since the Operations meeting on 2019-08-27 the bumping of Ensembl API
and DB-schema version numbers has been an Ensembl-wide policy.
Therefore, have Travis take notice of possible discrepancies in this
respect.

Note that simply removing the before_script line creating the file
disabling these checks is in general NOT sufficient - it might have
been left behind by a previous build in the event of runner re-use. It
is safer to make sure it does not exist regardless of initial conditions
of the runner.

cpanfile: require Try::Tiny

bugfix: should be sprintf here, as printf returns 1

(in case it isn't obvious, it has been deactivated by inserting
'.inactive' into the correct file name)

Unlike Ubuntu-based Travis builds, on GitLab-CI we use official Perl
images from Docker Hub. These are Debian-based so no major changes have
been required, support both perl-5.14 and perl-5.26 without any fuss
(and even if they eventually removed images for some version we could
always grab their relevant Dockerfiles and rebuild them ourselves), and
since they only contain core Perl they give us better control over our
dependency trees (i.e. much less risk of forgetting to add a non-core
module to cpanfile because it was already there as distro package on the
developer's machine).

What works:
 - the test suite runs and passes for both perl-5.14/sqlite and
   perl-5.26/mysql.

What still needs to be done to achieve full feature parity with Travis
builds:
 - enable Coveralls checks for 5.26/mysql jobs. Should work in theory but
   would require adjusting coveralls.io configuration to accept reports
   from EBI GitLab;
 - trigger dependent builds. According to GitLab-CI documentation this
   is quite simple to do but unsurprisingly, the relevant GitLab API
   differs from its Travis counterpart so we cannot simply re-use the
   existing script. Besides, for the time being we do not mirror any of
   the dependent repos on EBI GitLab;
 - disable API version checks in non-release builds. Left out on purpose
   because we might have the whole of Ensembl officially adopt the
   early-version-bump policy by the time this is put into use.

Does not have to be implemented in GitLab-CI:
 - configuration of e-mail and Slack notifications - in GitLab this is
   done in projects, not at CI level.

What could be done even though it isn't done in Travis:
 - create our own build images with all the common dependencies already
   installed. GitLab repositories can function as Docker-compatible
   container registries so it is quite simple to create a new repo with
   this feature enabled, shove a couple of Dockerfiles into it which
   install the necessary common dependencies onto base Perl images, and
   set up CI for that repo so that the base images are periodically
   rebuilt in order to account for possible updates. Result: simpler
   API-level CI configuration, faster runs, less bandwidth wasted;
 - process cpanfiles in ensembl-io and ensembl-variation. These are
   problematic for two reasons:
    * both depend on Bio::DB::BigFile, which requires messing
      with Kent libraries and linker flags in order to work properly,
    * ensembl-variation depends on Bio::DB::HTS, which pulls in BioPerl,
      which unless explicitly instrumented to install in minimal mode
      depends on known-to-be-broken GD module.
   At present we get away with not processing them because unlike
   ensembl-compara (which luckily does not demand any difficult
   modules), Core unit tests depending on -io and -variation code only
   import modules which do not use either of the above. This of course
   is a lucky co-incidence and cannot be relied upon to last.
   We have already got scripts for dealing with both Bio::DB::BigFile
   and GD, in Otter for instance, so we could in principle address this
   right away. Then again, in my opinion it would be better to delegate
   this to the aforementioned custom build images and keep
   repository-specific CI scripting as lean as possible;
 - setting passwords for MySQL users. Trivial to do but in order to
   avoid having those passwords in plain text in .gitlab-ci.yml, we
   should either set them as GitLab project variables or encrypt them
   somehow.

1. I must have set a branch wrong or something like that for the
   previous patching because it didn't update the version of the ontology
   schema. Fixed;
2. There is now an official variation schema-version patch;
3. For consistency between compara test DBs (officially still at 98) and
   all the rest (officially all at 99 now), apply the same unofficial
   compara schema-version patch as in ensembl-rest.

It is used by Bio::EnsEMBL::Feature and it is not a core module.
The reason Travis has not caught it seems to be that it comes with the
Perl installations they provide.