Introduction

Datapools distribute updates from the Server to the Clients. Datapools are only required if your application will distribute updates of data (this includes both updates from the back-end application and from other clients) - if all clients will continue to use the initial data then you do not need Datapools.

Please note: datapools are read-only; and do not update data on the back-end application.

Each Datapool definition maps to a pre-configured Service Function to create a single Datapool. This is instantiated when the first Client subscribes. Datapools can support client-specific data in the same manner as Service Functions. To update or insert a new XML element on the XML Desktop, the Presentation Server distributes the change from the back-end application through the Datapool mechanism.

Datapools supply new data by default and do not remember which data was in use previously. This means you always need to send replacement XML elements in the update. These overwrite any existing element in the Datapool with a matching Datakey. You can set a Datapool to be non-transient, which means that its data will be stored in the Presentation Server while the application is active. The main reason to do this is to have the data used as a filter for other Datapools.

Datapools track changes to the Initial Data sent to the Client and received from the Application using timestamp information in the Datapool and in the Initial Data.

In a single Presentation Server configuration, the updates are received in the Datapools of the Presentation Server from the Application. Datapools can use the same communication methods that are available for Service Functions. The Client(s) either polls for or receives streamed updates from the Datapools. The Slave section of the Datapool is included in the definition, but is ignored in practice.

In a Master-Slave configuration, on receipt of an update, the Master Presentation Server adds the update to the queue of updates to be sent to the Slave Presentation Server. The Slave Presentation Server polls for the Update at specified interval. On receipt of an update, the Slave Presentation Server adds the update to the queue of updates to be sent to the Client(s). The Client(s) polls for or is streamed to the updates in the Datapools of the Slave Presentation Server.

It is better to have small Datapools for specific data rather than an all-embracing large Datapool. This allows more control over refresh rates and avoids conflicts over XML attribute names. It is also more efficient to filter data on the Presentation Server than on the Client.

Datapools and Service Functions

Datapools get data from an existing Service Function they are pointing to. The Service Function contains all of the configuration information.

Normally you will use two separate Service Functions. The first one gets initial data, and has no connection with the datapool. This one is selected as the Initial Data Service (on the General tab of the datapool). The second one is used for updates, and includes timestamp information to enable the Datapool to sequentially order updates. This one is selected as the Service (on the Master tab of the datapool).

Datapool Feeding methods

Five mechanisms are available to provide new data to the Datapools:

POLLING: used for HTTP, JDBC and SOAP Datapools. The Presentation Server periodically polls the application for updates. When using Polling, the first poll occurs immediately. The Presentation Server is inactive until the first user connects.
HTTPSTREAM: the Presentation Server opens an HTTP Request and the Application sends updates as they occur.
JMS: provides updates via Java Messaging Service.
SOCKET: as per HTTPSTREAM but using the underlying socket mechanism.
CONNECTOR: user defined connector for adding updates to the Datapool. The connector has to implement DataUpdateConnector interface. The Datapool is then registered as listener to the connector. When new update is available, it is sent to the Datapool via listener's call-back function, newUpdate(). For more information see Connector Fed Datapool.

Of these mechanisms, Polling is normally the easiest to implement against an existing Application (as it requires least change to the Application architecture). Streaming and Socket are the best for rapidly changing data.

Please note: None of these three mechanisms accept compressed format data (they should only be handling small data sets)

If the Socket mechanism is used then the Presentation Server will automatically add an additional parameter in the HTTP request (SOCKET="nn") to specify the socket address for the Application to use in responding with data.

Datapool using a Timestamp

Each Datapool can optionally use a Timestamp. These are used to ensure that data arrives sequentially at the client and that old data on the client is replaced when required. We recommend using timestamps. This is a parameter which uses Date/Time format, or a long integer value. The Timestamp is initially set when the first Client subscribes to the Datapool and receives its Initial Data. In this case there must be a TIMESTAMP=nn attribute in the initial tag.

Example:

Service: ...
Parameter:
Name: TIMESTAMP
Value: ${datapool.timestamp}

Thereafter, the Timestamp is updated whenever the Presentation Server receives updated data from the Application whether by Polling (in which case it passes its current value in the request) or through Streaming. The new Timestamp is specified as an attribute in the root element of the response.

The interpretation of the Timestamp is determined by the application and is specific to each Datapool (i.e. AltioLive does not specify whether the actual number used is milliseconds, seconds, next sequence number etc). However, the Timestamp must get numerically larger with each update.

The Timestamp is sent in integer or date/time format. If using an integer, the number should increase with each communication. The Timestamp should support the expected lifetime of an Application. In the unlikely event that the Timestamp has to be reset, then the Presentation Server(s) would need to be reset.

A Timestamp parameter accompanies the Data Update request sent by the Presentation Server to the Application. In some cases, while integrating an existing Application with AltioLive, it would be necessary to exclude the Timestamp parameter from the Data Update request.

By default, the Application returns the Timestamp value as an attribute of the parent node of the data update posted by the Application. However, the Timestamp can be retrieved from a specified element attribute from the data update received from the Application in a specified format.

If an update is received with a duplicate of an earlier Timestamp then the Presentation Server will display a warning message on the client.

The timestamp will be returned as an empty string if it is used in an initial data service function.

For more information: see Timestamp integer and date formats.

Data routing

Data routing allows selectively distribution of data updates from a datapool to clients. For a given subscriber, a datapool may be set to forward all, none or some of each data update received.

For more information: see Data Routing.

Update Filters

In its simplest form a filter acts upon only the data provided in the update itself. For example to filter the personalized watch list in the "chem" demo use the following:

Datapool:
     Name: WATCH_POOL
     ...
     Filter: /WATCH/WATCH_ITEM[@USER=${connection.user}]

This filter would apply to a data update such as:

For each update of the WATCH_POOL datapool, only elements which match the filter are sent to a given user. In this case the user's logon name, as specified by ${connection.user}, is compared against the USER attribute of WATCH_ITEM elements that pass through the Datapool. Only updates that match are sent to the user. In this case we are specifying a particular user to receive updates, so only a user connected as 'Joe' would receive the update.

If no filter for a datapool is specified, updates for that datapool are sent to all users.

Filter Parameters

Curly brackets are used to indicate a substitutable parameter in the filter XPath. Parameters are substituted from a subscriber's session. These can be either Altio specific Property References, or may be set by a custom login servlet.

Filter parameters are evaluated once for each subscriber when they first request data from the filtered datapool. For example, changing the value of a session parameter during the session will not alter the filter.

Non-transient Datapools

To perform anything other than basic filtering, XPath queries must access a portion of the application data to be resolved.

Marking Datapools as being non-transient means that a copy of this application data is held within the Presentation Server. The XPath filter can then operate on both the data update node being processed and the data from all the Datapools designated as non-transient.

An example of non-transient Datapool:

Name: USERS_POOL
TRANSIENT='N'
INITDATASERVICE = "USER_LIST"

In this case the Datapool maintains the following data:

A non-transient datapool has the same functionality as a normal transient Datapool. Service Functions allow the non-transient data to be updated so as to modify the effect of filters dynamically.

For example, with the above data, users can be added and removed, and can have stock symbols added and removed from their data. These changes will be held by the Presentation Server and then can be used when the next data update to those users is filtered.

The Initial Data Service parameter is specific to non-transient Datapools. It is an optional Service Function that can be used to fill a non-transient Datapool with initial data.

Please note: this is not the same as the initial data that is sent to a new subscriber when they connect, as it is only used once to initialize the internal data held by the Presentation Server.

An example of a Dtapool that filters based on the contents of a non-transient Datapool.

Name: SECURITIES_POOL
Filter: /SECURITIES/SECURITY[@SYMBOL=/USERS/USER[@USER_ID=${connection.user}]/USERSYMBOL/@NM]

This would apply to a data update such as this:

The Filter is interpreted as: 'get any SECURITY that has a SYMBOL matching one listed under my USER element in the non-transient data'. The Datapool would only send to a user a SECURITY update if there is an entry in the USERS data (as in the example above) which matches them to that filter. In this way a user may be sent none, some or all the security updates received by the Presentation Server, and likewise each update may go to none, some or all the current subscribers to the SECURITIES_POOL datapool.

In this example the update, only users with a USERSYMBOL of MSFT would get this update. So user Joe would receive the new stock price, but user Jane wouldn't.

When the contents of the USERS data changes, so will the updates that get sent out from this datapool. So, in this example, a Service Function can be set up to add a symbol to the list of symbols for a user, and they would immediately begin receiving updates for that symbol.

An important thing to note is that the initial context of the XPath in a datapool filter is the current update node being processed. The context within the filter XPath's predicates, however, is resolved using non-transient data. In the above example /SECURITIES/SECURITY relates to the update being processed and /USERS/USER[@USER_ID=${connection.user}]/USERSYMBOL/@NM to the non-transient data.

Datapool Longevity

The Datapool effectively has an entry per timestamp. The size of the Datapool is determined through the CACHESIZE attribute.

The required size of the Datapool depends on the frequency of update to Clients and the frequency of update from the Application. If the Client were to poll at 3-second intervals and the Application creates updates at a rate of 2 per second, then it should be set to at least four times the rate (for example: 4*3*2=24) to provide contingency.

The last number of updates equal to the quarter of the cache size per Datapool are cached in memory. Earlier updates are archived to the disc.

Actions

It is also possible to send actions along with a data update. This could include, for example, the ability to open a new window to display the updated data in.

Datapool Types

There are two types of datapool:

JMS, to get updates which are "pushed" from Java Messaging Service,
and another type of datapool to support service function which "pull" data from a back-end application ( HTTP, JDBC or SOAP).

Client Update

Datapool updates are either polled from the Presentation Server by the Client, or streamed to the Client:

In the case of Polling, the Client polls the Presentation Server using its current Timestamp(s) (either the Initial Data or last successful poll). In response to the poll, the Presentation Server will send all updates (for all Datapools subscribed by the Client) subsequent to the Timestamp.
In the case of Streaming, the Presentation Server maintains a queue of Clients that are subscribed for streamed updates. As the Datapool is updated, the relevant updates are added to a queue for each Client.

The preferred mechanism is HTTP streaming as this minimizes update latency. However, as this uses a connection per client, HTTP polling can support a larger number of Clients where there is a less immediate update requirement. If polling is used and no connection can be established (to request new data, save user preferences, update data on the Application), then the Client retries after a number of seconds and the users gets an error message.

The Client Queue is fixed length (16 entries). In the event of a queue being full (only likely if the Application generates updates at a high rate and/or the Client is subscribed to many Datapools), then the Datapool updating will suspend for up to 30 seconds to allow the queue to clear. If the queue is still full after 30 seconds, the Client session is terminated. In the event of Client Queue being full or disconnected, warning messages are output to the Presentation Server log.

Alerts

As well as sending data, you can send messages to clients using Alerts.