Merging and Building the ZMap version of the FooCanvas

[The merge/build system was completely rewritten/restructured in March 2010 so previous code/canvas versions have been archived.]

add stuff about version numbers as well...perhaps should add version number to code but also needs adding to probably the pkgconfig and maybe other files...

Contents

Overview

To use Gnomes foocanvas widget for ZMap we have to make a number of modifications and additions. You will need to install this altered version of foocanvas in order to build and use zmap. This directory contains the source files, diff files and scripts used to construct our version of the foocanvas. This is not straight forward because:

Philosophy

Current Implementation

The unix utilities diff and patch are used to manage the merge of our code into the standard foocanvas.

The philosophy of the approach is that the CVS ZMap/foocanvas directory will contain a copy of all changed or new files (i.e. modified original foocanvas files or completely new zmap files) that are required to make the zmap version of the foocanvas. This directory also contains the diff files that were used to to merge these files into the gnome foocanvas to create the zmap foocanvas.

In fact only the diff files are really required and this was what was done previously. The problem with this approach is that if the merge goes wrong the user then has to do work to recover the full zmap version of the file so that it can be compared with the latest gnome foocanvas file to see what went wrong. This might seem straight forward but it is not and presents a real barrier to fixing patches that fail or assessing whether patches have produced valid code. Access to full versions of both gnome and zmap files makes it easy to see why a patch went wrong and to check the validity of patches.

All diffing and merging is done in temporary directories and the diff files reference those directories and so will only work if the gnome and zmap foocanvas files are in those directories. This is deliberate, the diff/patch cycle is easy to mess up, it should be done with the supplied scripts. These scripts ensure the diffing and later copying back into the CVS directory is done correctly. It is not a good idea to work from the CVS directory because then the diff files must reference the CVS directory and also you have no reference set of files to check against as you are making changes.

Current Limitations

Naming: The code currently builds into various library files with the prefix libzmapfoocanvas to avoid clashes with the gnome foocanvas which uses libfoocanvas. Really all the include files should be installed into an include directory called XXXX/include/libzmapfoocanvas but this has not been done yet BUT SHOULD BE !!

The build scripts are not bullet proof, it is possible to mess things up so that diff files become empty. On the whole though the committing to CVS is solid so that these files can be reconstructed.

makedist needs more thought and should not be used currently.

Scripts and Directories

There are two separate processes, the whole thing has to be "boot-strapped", i.e. you can't make a diff between the zmap version and the original until you have a zmap version, from then on it's an "updating" process of downloading the latest foocanvas from gnome and merging our changes into it.

There are a set of scripts that download, diff, patch and package files to produce the zmap version of foocanvas.

The scripts create/require a certain directory structure and should be run from the cvs "foocanvas" directory where they are found (i.e. ZMap/foocanvas with the ZMap CVS repository):

              ----- gnome_foocanvas     Created/populated by makedirs, contains copy
             |                          of latest gnome foocanvas for reference when
             |                          the merge goes wrong.
             |
foocanvas ---|----- merged_foocanvas    Created/populated by makedirs, initialised to
             |                          be same as gnome_canvas but after processing by
             |                          makepatch contains the merge of gnome_foocanvas
             |                          and zmap_foocanvas.
             |
             |----- zmap_foocanvas      Created/populated by makedirs, initialised to
             |                          be the same as zmap_source. You do all your edits here
             |                          and then use makediff and makepatch to apply your edits
             |                          to merged_foocanvas. makecvs may copy files back
             |                          into here from merged_foocanvas prior to these files
             |                          being used to make the final cvs update of zmap_source.
             |
              ----- zmap_source         The CVS copy of files (complete source and associated
                                        diff files) that are changed or added to create the
                                        zmap foocanvas. Not updated until makecvs is run
                                        to copy changed code from zmap_foocanvas into here
                                        and store the new/changed files in CVS.

Note:

The scripts are:

makecvs
compares source files in zmap_foocanvas with those in merged_foocanvas, any that have changed are put back into cvs. puts any that have back into zmap cvs.checks which source have changedand diff files have changed in zmap_foocanvas following a merge and updat
makediff
does a diff on gnome_foocanvas and zmap_foocanvas to create diff files for use by makepatch
makedist
copies, tars and gzips the merged zmap foocanvas for distribution, uses a date stamp on the tar file and extracted directory to avoid version clashes.
makedirs
retrieves the latest version of the gnome foocanvas into the directory gnome_foocanvas and makes a copy of that into merged_foocanvas in preparation for merging.
makepatch
takes the diff files made by makediff and applies them to the relevant files in gnome_canvas to make an up to date zmap version of the foo canva

Bootstrapping

This needed to be done initially but shouldn't need to be done again unless either zmap or foocanvas change radically in which case a remerge by hand may be appropriate.

  1. cd ZMap/foocanvas
  2. run makedirs to download the latest foocanvas from gnome into gnome_foocanvas directory and make a copy in merged_foocanvas.
  3. in a separate directory called zmap_foocanvas make a copy of any foocanvas files that need to be changed and add any new ones.
  4. run the script makediff to do a diff for changes required to the code in merged_foocanvas to merge in the zmap_foocanvas files. This step leaves a series of *.diff files in zmap_foocanvas.
  5. run makepatch to check that the diff files will patch the zmap foocanvas files into the new ones correctly, this should produce a full set of merged files in merged_foocanvas from which a distribution can be made.
  6. run makecvs to update the copies of source and diff files in zmap_foocanvas and check them back into cvs.
  7. run makedist to create a gzip'd tar file that can be installed on different systems to build the zmap foocanvas.

Updating the Version of Gnome FooCanvas

This is the standard procedure for updating ZMapFooCanvas with a new version of gnome foocanvas.

  1. cd ZMap/foocanvas
  2. run makedirs to download the latest foocanvas from gnome into gnome_foocanvas directory and make a copy in merged_foocanvas.
  3. run makepatch to check that the diff files will patch the zmap foocanvas files into the new ones correctly, this should produce a full set of merged files in merged_foocanvas from which a distribution can be made.
  4. if makepatch fails it will report all merges that failed and the patches that failed will be listed in *.rej files. For files that failed:

  5. run makecvs to update the copies of source and diff files in zmap_foocanvas and check them back into cvs.
  6. run makedist to create a gzip'd tar file that can be installed on different systems to build the zmap foocanvas.

Updating the Version of ZMap FooCanvas

This is the standard procedure for updating ZMapFooCanvas with new zmap code.

  1. cd ZMap/foocanvas
  2. Make changes to the files in zmap_source.
  3. run makedirs to download the latest foocanvas from gnome into gnome_foocanvas directory and make a copy in merged_foocanvas and copy zmap_source files into zmap_foocanvas.
  4. for each file that you changed:

  5. now run makepatch to check that all the diff files will patch the zmap foocanvas files into the new ones correctly, this should produce a full set of merged files in merged_foocanvas from which a distribution can be made.
  6. if makepatch fails it will report all merges that failed and the patches that failed will be listed in *.rej files. For files that failed:

  7. run makecvs to update the copies of source and diff files in zmap_foocanvas and check them back into cvs.

Building/Distributing ZMmap FooCanvas

The preceding sections will have left you with a patched foocanvas in tmp_merged_foocanvas that you can build, this section describes how to do the build.

Overview

There are two types of build:

In theory building from scratch should only need to be done once and then that build can be used to make a "distribution" build which can then be used to build foocanvas on on other platforms in a simpler more straight forward way.

The following sections describe these two types of build on various platforms.

Building from scratch using autogen.sh

The build requires that the system has the autoconf tools installed and will not work without them. It also requires that the user has the right permissions to install the zmap foocanvas into the chosen install directories (e.g. for /usr the user may need to able to use sudo).

The build install steps are:

  1. cd tmp_merged_foocanvas
  2. If your autoconf is not installed in /usr or /usr/local then:
          export ACLOCAL_FLAGS=" -I <path to autoconf tools>"  # NOTE the spaces !!!!!
    
          e.g.
          
          export ACLOCAL_FLAGS=" -I /opt/local/share/aclocal"  # NOTE the spaces !!!!!
          
  3. Run autogen.sh will run the autoconf tools to make the ./configure script and then run ./configure:
          ./autogen.sh --prefix=<directory where you will want to install foocanvas>
    
          e.g.
          
          ./autogen.sh --prefix=/opt/local
          
  4. Build the code:
          make
          
  5. Make a distribution for building on other platforms:
          make dist
          
  6. Install the code on this platform:
          make install
          

Sanger Linux Build

Sanger has a system of allowing developers to install later versions of standard software in the NFS mounted /software directory, for acedb and ZMap our versions are in /software/acedb and we can install new versions in this as we like. The foocanvas build system needs to use versions of automake and other tools from this directory, to do this follow these steps:


> cd tmp_merged_foocanvas

> export PATH="/software/acedb/bin:$PATH"

> export ACLOCAL_FLAGS=' -I /software/acedb/share/aclocal -I /software/acedb/share/aclocal-1.9 '  # NOTE the spaces !!!!!

> ./autogen.sh --prefix=/software/acedb

> make

> make dist

> make install

Building using ./configure from a ready made distribution

Use the tar.gz file produced by the "make dist" build target from the preceding section and do the following:


> mv libzmapfoocanvas-0.1.tar.gz <temporary build directory>

> cd <temporary build directory>

> gunzip libfoocanvas-0.1.tar.gz

> tar -xvf libzmapfoocanvas-0.1.tar.gz

> cd libzmapfoocanvas-0.1

On a Sanger Linux system (32 or 64 bit):

> ./configure --prefix=/software/acedb

On a Mac with GTK in the default MacPorts directories in /opt/local

> ./configure --prefix=/opt/local CPPFLAGS="-I/opt/local/include -I./" LDFLAGS="-L/opt/local/lib"

> make

> make install

To build a debug version of the canvas and install it locally:


> ./configure --prefix=<your local directory> CFLAGS=-g

> make

> make install

If you need the source for debugging then you should copy the untar'd libzmapfoocanvas-0.1 directory into some convenient directory. For /software/acedb/ this is:


/software/acedb/src/libzmapfoocanvas-0.1

Future Work

Future tidying up/work:

Need to rationalise and automate the version number we use for our version of foocanvas. I think we need to use the version from the package config file but this needs some investigation.

Appendix

Foocanvas Versions

Originally FooCanvas was held in the GNome CVS repository but this became defunct and it was moved to the GNome SVN repository, now this has become defunct and it's in the GNome GIT repository.

So currently there are two versions available:

They are currently identical but it looks like the git version is the active one.

Patch versioning

We use a combination of diff and patch to manage the merge of zmap foocanvas code into gnome foocanvas code. It is possible to add versioning to diff files to stop the wrong diff's being applied. The Prereq flag can be added to the start of diff files for this purpose:

Prereq: 1.1.1
--- old.txt	2010-03-05 09:43:46.000000000 +0000
+++ new.txt	2010-03-05 09:43:05.000000000 +0000
@@ -1,5 +1,5 @@
 
-line 1 of target
+first line of target
 
 second line of target

All source files in zmap_source would need to have the version number that follows "Prereq:" embedded in them and then if the patch and diff files had different versions patch would not do the merge unless the user forced it.

Currently this is not implemented because it would be tedious.

Notes from previous build system

Notes from the previous system which may be useful if there are problems.



Creating the tar.gz
-------------------

$ gcvs co -d libfoocanvas-VERSION foocanvas

*** additional files fix ***

now we have additional files this step is a little more difficult and makes me
think we should just distribute a fully patched tar.gz.

$ cd libfoocanvas-VERSION/libfoocanvas/
$ patch -p0 -u < ../path/to/additional_files/Makefile.am.patch
$ cp ../path/to/additional_files/*.{c,h} .
****************************

now make all the autoconf stuff.  I had to do this first.
NOT $ export AUTOMAKE=~/bin/automake
try this...
$ export AUTOMAKE=/nfs/team71/acedb/zmap/prefix/LINUX/bin/automake

aclocal needs to be told where it's m4 files are.
~zmap/prefix/LINUX/share/aclocal
~zmap/prefix/LINUX/share/aclocal-1.9
are good bets at the moment. glib-gettext.m4 seems to be a dependancy 
which breaks everything silently....

NOT $ export ACLOCAL_FLAGS="-I ~/share/aclocal -I ~/share/aclocal-1.9"
try this...
$ export ACLOCAL_FLAGS=' -I /nfs/team71/acedb/zmap/prefix/LINUX/share/aclocal -I /nfs/team71/acedb/zmap/prefix/LINUX/share/aclocal-1.9 '
autogen.sh also requires gnome-common 

$ gcvs co gnome-common; ./configure; make; make install

For  the  last  build  (20060326)  I  had  to  modify  autogen.sh  and
configure.in as per  build.patch. Unsure as to why,  possibly a broken
autotools setup. 

$ ./autogen.sh

$ make dist-gzip
  this uses VERSION from configure.in (currently 0.1), I was using YYYYMMDD, who knows what's best.
  to change this edit the Makefile
-distdir = $(PACKAGE)-$(VERSION)
+distdir = $(PACKAGE)-YYYYMMDD


$ mv libfoocanvas-VERSION.tar.gz ZMap/foocanvas/support/
$ cvs add -ko -m'message' libfoocanvas-VERSION.tar.gz
$ cvs commit -m'message' libfoocanvas-VERSION.tar.gz

Should alter untar_patch.sh to set the version variable to VERSION and
tag the files as libfoocanvas-VERSION.
Test by running ./untar_patch.sh and seeing no errors in patch and a complete build