zmapView is quite a large module! When adding a topic to this file please include a link here.
Historically, when ZMap creates a view it goes through a process of 'data loading' during which it requests all the data implied in its source stanzas. This has typically been ACEDB and this connection is maintained open for the life of the view to allow further requests by the user. Till this initial phase has completed the user may not interact with the view. With the advent of pipeServaers this model is no longer enough to cope with incremental/ optimised loading of data and we need to be able to specify sources as not active on startup, and to be able to start previously unknown sources on request from Otterlace.
A source stanza in ZMap's configuration file ZMap specifies the URL of the data source and some other options including featuresets supported. A new option delayed=true/false will set whether or not a data source will be activated automatically on startup. A pipeServer source will default to delayed=true and other types false.
Any request for data after the initial 'data loading' phase will be activated immediately regardles of the setting of 'delayed'.
The existing X-Remote protocol is suffcient for our purposes and will be used unchanged, this means that data will be requested by featureset name; a sample data request looks like this:
(coming RSN)
On receipt of a request ZMap will re-read its config file and scan the list of source stanzas for the requested featuresets - this will define the data sources to request the data from (each source stanza defines the supported featuresets). A new connection will be started for each request to allow multiple requests to be processed by servers without imposing delays implied by queuing.
A View has a set of data structures that implement a StepList (see zmapView_P.h), which allows a sequence or actions to be created and operated in turn, and also to specify actions to take in case of error. Currently (10 Mar 2010) the View has one of these lists and this is used to control the list of data servers together. To handle concurrent requests from different servers this will be changed to be a list of Step Lists, each operating independantly.
The request process is modelled on the ACEDB interface, consisting of a number of steps defined as ZMapServerReqType in include/ZMap/zmapServerProtocol.h. Old code used the first connection in the view's list and assumes that this is already active and for request to pipeSevers this is not correct - the connection must be started fresh each time and closed when finished.
Code that handles this is found in zmapView.c/zmapViewLoadFeatures(), zmapViewUtils.c/loadFeatures(), and zmapView.c/commandCB() (to request a DNA sequence). The latter uses View->sequenece_server to fidn the right connection.
The function zmapView.c/zmapViewConnect() is used for the initial 'data loadng' phase on startup and uses similar code.
This will involve the following steps, (any legacy code left over by mistake can be removed)
Note that a separate request will be allocated for each featureset, even if several are included in the same external request and as supported by the same data server. This is to allow maximum concurrency in the hope that data will be loaded faster. For example EST_human and Saturated_EST_Human need to be requested together and form a logical category together. This will also apply to ACEDB requests if more than one request is active.
Currently Otterlace shows a progress bar for data loading and to retain this while usihng pipeServers X-Remote messages will be added to drive this. Initially this will consist of reporting 'completed', but if GZip chunks are implemented this can be broken down somewhat. The X-remote message format will be like:
hello