Skip to content
Snippets Groups Projects
Commit ef60c083 authored by mh17's avatar mh17
Browse files

moving docs

parent 46d63a8a
No related branches found
No related tags found
No related merge requests found
ZMap Meeting
Date: 09.03.2005
Present: Richard, Tim, Roy, Ed
Aim:
We want to get annotators trying out zmap in the lace system as early as
possible. zmap will stand or fall by whether it is better to use than
xace/fmap, its important to design a user interface that implements features
that annotators will find useful.
Overview:
The lace system in essence merges various data sources into an acedb database,
runs xace on the database allowing the annotators to use fmap to display the
sequence. The lace system then provides its own GUI for the annotators to edit
data, this GUI drives xace via the xremote mechanism to update its display and
the local database as annotations are made.
Plan:
From this several requirements emerge for the intial integration of zmap:
- replace xace with the sgifaceserver
- run zmap off the sgifaceserver
- add xremote commands to zmap: zmap already supports some commands but
requires several more to replicate the existing interface
- add support to call blixem/dotter, this will be using the same mechanism as
xace does when calling them externally.
This in essence would allow the annotators to use zmap for annotation. In
practice it is certain that there will be missing functions but it will give
us a system that we can develop in tandem with them.
Future:
The merging of data into an acedb database is a weak link in the lace system
and the longer term aim is to remove any need for it by having zmap extract
data directly from the mysql, ensembl etc. databases. Once the intial system
is working these individual data sources can be picked off one by one.
Decisions:
It was agreed that being able to display multiple alignments would make zmap
significantly better than the current fmap and that this should be a priority.
The display of alignments should not be limited in the number of alignments
that can be displayed. It was also agreed that display should be of "blocks
of features" of one or more alignments against a master sequence rather than
the more familiar but confusing "crossing line displays".
Compara/ensembl can provide data for Dog-Human already and this should be the
data source for our first experiments with multiple alignment display.
The most straight forward approach to getting multiple alignment data is to
code a perl DAS server to use the ensembl interface to retrieve the sets of
features already mapped. The ZMap will interface to this server to make
requests for the alignment data. In this way we will not have to add mapping
code or understand the Compara alignment scheme in detail. This would mean
however that zmap could not swop the reference/alignment sequences around
without fetching all the data again.
It was agreed that if we do add mapping code to ZMap, then this should be a
subset of the smap code to do more restricted mapping translations. In
addition a new function would be needed to map multiple alignments as the
current code while dealing mapping over gaps expects the mapping as a whole to
be colinear.
Blixem and dotter will be supported although it would be good to remove the
need to use blixem simply to verfiy splice site boundaries in relation to
homologies.
We should look at changing the lace xremote interface so that when edits are
made, zmap can simply update the single feature updated rather than completely
refetching the data.
Also that zmap should initially not do any editing, unlike current xace, where
at least remarks/comments are set using the xace display. This seems like a
trivial addition to the otterlace interface, that I imagine would be
relatively easy for anacode to set up. We need to look at how the editting
cycle works in the current lace/xace setup.
Zmap does not currently display a number of datatypes that will be needed, e.g.
Stops, the frame translations, the actual sequence.
We will provide web pages that document:
- the xremote interface between lace and xace
- data types (e.g. feature, transcript) and styles/methods
- GUI interactions in the context of data types/methods
ZMap Meeting
Date: 23.03.2005
Present: Richard, Ed, Roy
Decisions:
It was agreed that we need to work more on the integration of zmap into
the otterlace system. To this end the launching of both xace and zmap from
otterlace should be possible. Whilst we have a prototype working,
which has replaced xace with zmap and a sgifaceserver, it is by no means
production quality and needs work.
Currently using Compara and a DAS server to dynamically serve align
features appears to be off the cards at the moment. It was decided a
static solution, most probably for the NOD/normal mouse comparisons that
Charlie Steward is working on would be a good start to a) provide the
data to prototype the display and b) create a visualisation tool for
Charlie's annotation efforts.
There was some discusion about two way communication between zmap and the
Perl/Tk layer of otterlace to facilitate a richer user experience. James
has already requested a feature of the new interface should be to request
information from zmap and the concept of zmap being able to send otterlace
information when something changes in zmap sounds like a reasonable
possibly in terms of implementation.
The demonstration of zmap highlighted a number of issues with respect to
the GUI interaction.
- A selected feature should be highlighted in all views
- A selected feature should have it's "friends" highlighted
- Zooming and scrolling should be "locked" by default and user
"unlockable".
- A v-split should by default lock the vertical scrolling between a pair
of views
- A h-split should by default lock the horizontal scrolling between a
pair of views
- scrolling should by default be locked between a pair of views
- the display should be centred on click
- Rich visualisation should be possible from context menus. e.g. the
spliting of an intron to display the whole gene, 5' end of the intron
and 3' end. (see attachment split_intron.png)
- the currently focused view should be more obviously the currently
focused view
It was agreed we should provide a release/demonstration version of zmap
for each meeting.
It was agreed the website needs further organisation/work and that a
zmap@sanger.ac.uk mail alias/list is a good idea.
Our next meeting will be in three weeks at 1.00pm on the 19th April.
ZMap Meeting
Date: 23.04.2005
Present: Richard, Ed, Roy, Rob
ZMap/otterlace:
Roy has a prototype system with lace starting simultaneous xace/zmap processes
which is good for comparison. He has been working on the atom/property
communication system as the existing xremote code cannot give us all that we
want.
Roy has added code to Perl/Tk to do atoms/properties...amazingly this is
missing from the Tk interface. We have a working system so that James can
write perl to directly control the atoms/properties to communicate with
zmap. A substantial improvement over the old system.
Compara:
We have not worked on this more.
zmap GUI:
Ed has been working on locking of windows and this is working, highlighting is
on the way.
The currently focussed view is more obvious but its hard work with gtk.
The code is now in-place to more quickly do things like the intelligent viewing
of transcripts.
ZMap demo code/website/mail:
There is now a ~acedb/ZMap, the demo code for alphas is:
~acedb/ZMap/ZMap/src/zmap
We now have a "zmap" mail alias set to Ed, Roy and Rob.
Rob will be doing some work to set up web pages for the web site.
Decisions:
We need to produce a demonstration of zmap working alongside of xace in the
lace system that is good enough that an annotator can start to try it out. In
particular this needs to show how zmap will display multiple alignments as
this is seen as a key factor in annotators wanting to use zmap at all.
Ed will:
- this week send out an email to James, Jen, Tim (cc'ing us) saying
that we will produce the demonstration zmap, giving an idea of what
it will do and suggesting/soliciting opinions about who would be
a good person to test it.
- send Richard an email on 9th May detailing more precisely what the
demo. zmap will do, how it can be used etc.
We will produce the demo. zmap for the 23rd May to show to Richard in our next
zmap team meeting on that day. The demo. will need to include facilities that
the annotators commonly use e.g. blixem.
An obvious annotator would be Charlie as he is working the NOD and Black6
mouse variants and this would provide a good test of the multiple alignment
display.
We also discussed:
- we need to decide on the semantics of single/double clicking on features
- the feature list needs some enhancements/fixes (e.g. it should centre on the
feature the user clicked on).
- we could allow the user to supply a list of their own features (like the
"keyset" concept of acedb), this would not necessarily be features of the same
type. This would be useful but is not seen as a priority.
Our next meeting will be in three weeks at 1.00pm on the 23rd May.
ZMap Meeting
Date: 06.01.2006
Present: Ed, Roy
lace system:
status - needs testing for robustness and speed... **EG to install on his account and try it.
problems -
Currently
Actions - **Roy to arrange a meeting with James et al. to dicuss following issues:
1) Does the lace system read any data from the acedb database at the end of the
session or is the database effectively READONLY ?
2) Unique ids...can James supply these in the data from the database so we can read
them directly. We already have a slot in our feature data struct for them. This unique
id may be good to use in a separte hash of unique_ids -> canvas features which could
unambiguously go from one to the other.
3) Genefinder - can this be run from the perl code and then the results sent across
to us as a Style + the data ? It makes for this to be done in the lace layer because
gene finder is principally about interactive annotation.
4) we have a performance problem with the g_dataset, to circumvent this can we reduce
the number of homols being displayed ?
5) Can we migrate quickly from the dual xace/zmap system to a zmap only system. **RS to check.
6) discuss adding whole columns with James, e.g. genefinder data.
XML-xremote mechanism:
Currently we do not have the basic functions to add/modify/remove features to support
this. We need this, and we may also need a function to get the server to parse in
data. Can be tested via the feature editor. **EG to make sure this works.
We need to review the operations in the XML-xremote interface, **RS to do this.
We will need the xremote mechanism to dynanically add whole columns. This will involve
operations to do at least:
- adding the column name into our column list so it appears in the correct position
in the columns.
- adding the style for the column to our set of styles _before_ we are given the column
itself.
- adding a whole column with data.
ZMap Display:
- transcripts now display the CDS but not start/end not found. This data should be
dumped in GFF and available. **RS to check this out.
- separate scale window...not done but not a priority.
- 3-frame translation, not done **RS to code this.
- reverse complement, not done **EG to investigate, will users be prepared to put up
with the hit of refetching the data or should we fudge it locally. Check how acedb does
this now, I think Simon changed it not to cache data so much.
- DNA highlighting - EG thinks he may have broken it in moving the code for
the DNA into ordinary feature group. **RS to check.
- mulitple feature types in a single column needs checking to see if it is still
working. **EG to do this.
- blixem needs to be checked to see it still works. **EG to do this.
- methods and columns and things: we had a useful dicussion about these and came
up with:
- we need to publish a list of "reservered" style/method names for
columns that are either derived dynamically or special in some way,
e.g. dna, genefinder output, 3-frame translation. The user can create
styles with these names and their style will be applied to these specific
datasets (if available, e.g. the dna may not be available). The user
cannot use these names for their own columns/data, they are reserved.
In addition the rules concerning display will be tightened to be:
- if any column is not in the columns list it will not be displayed.
- if a column is in the columns list but there is no style for it, it will not be
displayed.
i.e. columns without styles will be removed from the columns list and will
not be displayed.
g_dataset problem:
To recap:
the glib g_dataset code grinds almost to a halt while we are processing the GFF
from the server. We've not been able to profile this well within our threaded
environment sadly. What we do know is that it seems to spend a lot of time in
a free routine so it maybe a problem with glibs chunk routines for allocating
blocks of memory. It may also be, as Roy suggests, that its having to search
longer and longer lists of blocks...
solution is to write a small program that uses the GFF parser to read in a file,
this should be simple, in a single threaded program and see where the problem is.
A temporary solution is to try and reduce the enormous number of homols being
displayed.
Styles, Features and canvas item drawing....:
this is an old chestnut...what decides how a feature is drawn ? Is it only
the style ? Is it the feature type (e.g. allele etc.) ? In acedb the feature
type was the thing that decide it. We wanted to generalise this but there
is a problem in that if the style is the only thing its hard to make pop-up
menus and other things that are specific to certain feature types unless
you add a feature type field to the method/style which means that now the
style is the feature...and in any case the feature just contain exons etc
so it is the feature as well...it just doesn't really work...
So this is our sytem which is a blend...
Feature drawn shape Style
BASIC => read shape from style read shape as basic box,
splice markers (for genefinder)
constant shapes (for alleles)
TRANSCRIPT => usual transcript shape read for colours
ALIGNMENT => usual align shape read for colours
This works because there three basic feature types should be enough to allow popup menus etc
to work well but then we can avoid having endless types fields for alleles etc by having a
drawing type in the style which will sort all this out.
So in otherwords, the feature type decides what type of main shape the feature will be
drawn as, but the style allows tailoring of the basic feature types.
ZMap Meeting
Date: 07.02.2007
Present: James, Leo, Roy, Ed
Overview:
There are currently a few missing links in the interactions between
zmap and otterlace, specifically, only editing is possible at the
moment. We should be doing more to support things such as
highlighting objects in both programs after selection in one or the
other. This includes multiple selection. In addition there are
further operations we need to be able to do like opening a new
sequence in zmap.
Discussion:
Currently the xremote interactions are only at edit time and generally
all follow the same style of xml, namely:
<zmap action="__action__">
... data to complete the action ...
</zmap>
This is fine for editing at the moment, but I can see it all getting a
little confusing. What action does what? and what information do they
all require? We need a document or two in cvs that describes what and
who can each action applies to. The xsd document I initially provided
is all well and good, but it's a) out of date b) in the zmap cvs (can
you guys see it) c) incomplete when it comes to finding out what an
action does. and d) xml :(
So what should go in this file? I've prepared a list of what I know
each application can provide/do.
Action Program Details
---------------------------------------------------------------------
open zmap (m) opens a zmap feature display window. Requires
<segment sequence="str" start="int" end="int"/>
close zmap (m) closes zmap. The complete application
shutdown zmap (m) alias for close.
zoom_in zmap zoom the currently focused window by 2x
zoom_out zmap zoom the currently focused window by 0.5x
create_client zmap register's a client internally with zmap. Req.
<client id="id" request="str" response="str"/>
create_feature zmap create feature(s) on the zmap display from the
featureset supplied.
delete_feature zmap deletes feature(s) that already exist on the
zmap display.
new_view zmap opens a new sequence using the supplied
information to fetch the data. Requires
<segment sequence="str" start="int" end="int">
source{
url = "url"
featuresets = "feature sets"
}
</segment>
edit otterlace Edit an object in otterlace.
Requires a featureset.
ZMap expects a response <response handled="boolean">
register_client otterlace register's a client internally with zmap. Req.
<client id="id" request="str" response="str"/>
Why is this not create_client???
Additions/Changes are up for discussion.
For the multiple species display is new_view enough?
Here's some RT tickets...
RT ticket #20283
----------------
Liz,
> When I click on a bit of cDNA evidence and bring up the Zmap Feature
> Editor Window, it would be really useful if I could highlight the name
> of the cDNA and so that we can copy and paste into somewhere else (e.g
> into search box of NCBI EntrezGene website).
>
> I suppose I could do this from blixem, but, just a thought...
This presents a bit of a quandry......our "feature list" and "feature
edit" windows are both set up to scroll to and highlight the name that
you click on, this also results in a brief description of the feature
being put into the common "edit" buffer in this form:
"Em:BT020903.1" 40645 40742 (98)
This is a hangover from acedb and perahps we should just put the feature
name in the edit buffer. I'll have a think about it. In the meantime I
guess you could use the above and delete the bits you dno't want...not
ideal though, I can see that....
Ed
----------------
I can't remember why this string is like this. Does it get used by
xace? Or it just for otterlace to get hold of the data via the
clipboard. Could we subvert this with xml so that we only stick the
name in the clipboard?
RT ticket #20304
----------------
I would like single click events that highlight objects in Zmap to be
passed to otterlace. This will enable us to do things like highlight the
exon coordinates in a transcript editing window when an exon is clicked
on, or highlight the name of the transcript in the XaceSeqChooser.
Similarly, I would like to be able to send requests to Zmap to highlight
(and scroll to reveal, if necessary) objects from otterlace.
James
RT ticket #20255
----------------
One thing that came out of yesterdays zmap demo was the requirement to
build objects from supporting evidence. This is not a new request, but
now zmap can do multiple select [ticket #10152] we should work out a way
to enable this. Firstly I need to do some work to remember what
currently happens [ticket #20173]. Is there a preferred solution?
RT ticket #8588
---------------
Hi,
At the moment it's possible to create a new object based on the exon of
the evidence highlighted on the fmap. Would it be possible to do the
same with zmap.
Also, it was mentioned yesterday that exons of ESTs or cDNAs may be
linked by lines similar to the intron lines in objects. If this is the
case, would it be possible to build a new object based on all these
linked exons instead of just the highlighted one?
thanks
Sarah
Actions:
--------
1) Investigate checking out from ~zmap/CVS for anacode users. EG
2) Create a synopsis document to accompany the xsd document and keep
them up to date. RDS
3) Add zoom_to to the xremote protocol. This should accpet either a
simple range or a feature to which zmap should add a border. RDS
4) otterlace needs to be notified when a zmap view is closed. RDS
5) Although zmap understands the action new_view, this is not enough
for all the alternatives that exist. It should also be possible to
close each individual view. Also the open command is very similar
if not identical so we need to merge these. RDS
6) Views need to be addressable by otterlace. The reply to an open
command should contain the addressable id. RDS
7) register_client and create_client should be merged to only be
register_client. RDS, JGRG
8) Single and multiple selects are to be transferred a click at a time
and will be distinguishable by an attribute. The cut buffer will
be additively extended for multiple selects. This will be
bi-directional, so that selections will be mirrored in the two
progams. RDS, JGRG
9) Check copy / paste from zmap info boxes. EG
10) build a new zmap RDS, EG
ZMap - items for lace integration
---------------------------------
The plan is to put zmap into test_otterlace straight away even though we know
not everything that is required is implemented yet. Liz/Charlie will be the
guinea pigs for the test initially. We will meet again in a month to review
how it's gone and what we need to do next.
Priorities:
-----------
*** Do right now
** Do this month
* Think about
Configuration
-------------
*
- need to be able to do "per team" configurations of different parts of zmap
(e.g. of what goes in the cut buffer and in what format).
Transcript operations
---------------------
** EG doing this. <DONE>
- contextual menu on gene object:
+ show CDS translation in zmap
+ show CDS translation (including *stopcodon) in separate (copy-pastable)
window
+ export CDS translation (fasta seq saved to file)
+ show CDS sequence (nucleotide seq of CDS incl stop codon) in separate
(copy-pastable) window
+ export CDS sequence (fasta of nucleotide seq of CDS incl stop codon
saved to a file)
+ export entire mRNA sequence (fasta seq saved to file)
+ show entire mRNA sequence in separate (copy-pastable) window
+ export genomic sequence of transcript (fasta seq saved to file)
(we could colour the genomic span when displayed in a separate
window to make it obvious which are exons and which introns.
EG acedb has translation code which handles different translation tables, I
will pinch the code....
I have a comment about showing the size of a protein translation, I think this
might be in the status line if you click on the translation ?
<DONE>
**
- objects should carry http links so that web pages (e.g. ensemble/pfam) that
carry more information about them can be displayed.
EG Sequence class has a web_location tag which points to a url object, this is
quite complex and poorly documented so I will need to look in the code....so
it maybe very simple for us to add this.
*
- pattern or word search (peptide and nucleotide) of genomic seq as currently
under "restriction enzyme digest" and highlight results in fmap (currently
when DNA is shown, highlighted with colored background (different colors for
forward and reverse match) and also the matches are indicated by printing out
the search term in a column close to the central yellow divider)
EG This is probably best implemented by lace doing a patten search and then
sending us the results as a new column + matches that we need to draw as
"features" in the column.
Display stuff
-------------
***
- copy-paste from pfetch text window
EG I think this is done already via gtk text widget..
<DONE>
**
- display ORF's
EG I will hack ace server to allow ORFs to be dumped.
**
- need 3 frame translation in the window.
- show ORFs in all three frames (i.e. show stopcodons)
- show 3-frame translation of genomic seq (sequence selectable and copyable
like dna seq above)
**
- clicking on a transcript or exon should highlight that exons dna if the dna
is displayed (agreed that dna could be highlighted by colouring the
background).
- if dna seq is shown, highlight the extent of any box/feature user clicks on.
*** EG THIS IS DONE APART FROM THE RESTORE DEFAULT BUTTON, THEY DIDN'T HAVE THIS
IN ACEDB SO ITS NOT THAT GREAT A PRIORITY...
- column configuration should be a main button in the zmap window, should be
able to reset to defaults with single click....perhaps reset should reset the
whole interface.
- "configure all columns" button in button bar at top (currently a submenu in
column contextual menu)
- "restore global default" button in button bar at top (i.e. restore default
columns etc.)
<DONE>
- double clicking on an object should display the "properties" window.
- double click (proper double click !!!) on a feature and get a "property
inspector" window (like tree display in fmap). Possibly super-user (but not
user!) configurable
<DONE>
*** RDS_DOING_THIS
- put the scale by the yellow strand separator
*** RDS_DOING_THIS
- cut displayed DNA direct into the cut buffer for access from other
applications.
- when dna seq is shown in zmap, select a stretch of sequence -> contextual
menu item "copy" or it could happen automatically
**
- allow printing of the current zmap to a file
- postscript printing the zmap
EG acedb does this by steam and we could pinch the code.
*** RDS_DOING_THIS
- clicking on a column will turn the background of that column to light grey
to emphasise that it has been selected.
- trial of subtly highlighting entire column background when left-clicking on
the column background
<DONE>
***
- each sub window of a zmap should contain more information in the title which
should include at least the species.
*
- the current left hand scroll bar should be replaced with a locator which
shows clone position/direction/overlaps/names in a similar "stepped" display
as fmap does currently. We could also put a scale in this window.
- canvas with clone tiling path (with names) on the left and proprtional
scroll thumb for navigational purposes.
**
- users should be able to lock windows together in positions they choose.
Blixem
------
*
- there should be more communication between blixem and zmap so that things
clicked on in zmap get highlighted in zmap and vice versa. Probably we need to
fork blixem so that we can leave behind the old acedb version.
Genefinder
----------
*
we can get the genefinder information from gifaceserver but it requires a
change to the server to the ORFs output and we need to decide whether to get
it all at the start or to get some later...
- genefinder features (start codons (with Kozak score), coding potential,
splice sites, etc.)
--
------------------------------------------------------------------------
| Ed Griffiths, Acedb development, Informatics Group, |
| The Morgan Building, Sanger Institute, Wellcome Trust Genome Campus |
| Hinxton, Cambridge CB10 1HH |
| |
| email: edgrif@sanger.ac.uk Tel: +44-1223-496844 Fax: +44-1223-494919 |
------------------------------------------------------------------------
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment