Metrici derivation

By Andrew Clifford

Metrici's data derivation makes it easy to build complicated processing and increases performance by orders of magnitude.

We have covered Metrici concepts, web pages, and building websites using the Metrici MiniSite app. Now its time for something more technical.

Within Metrici, the term "derivation" is used to mean the automatic recalculation of data. It is a fundamental concept used by all Metrici components and solutions.

When we were designing Metrici, we knew that a triple store would be very versatile. But we knew that versatility would come at a cost. A triple store can involve a large number of data accesses and be horribly slow.

To overcome this problem, we came up with a simple solution: have two triple stores.

The first store contains the data originally entered by the user. The second store, the derived data, contains the original data plus calculated data, and is used for read operations. In our final design, we merged these two stores into one store, called "node members", with a flag on each member to indicate whether it is original or derived.

Calculating derived data is simple. The starting point is to copy all the original data for a node to the derived data. After this, other processes perform additional calculations.

The calculations are defined on the node type's member types (see How Metrici is structured), and are written in JavaScript that runs on the server (see Metrici scripting). The simple example below shows the calculation of one number based on the value of two others. If you want a complicated example, see how quick links are resolved in the page type.

Timing of derivation is important. Nodes are marked as out of date whenever they are changed. When a node is subsequently accessed, derivation is performed if it is out of date. This means that update processing is fast (there is no need to recalculate data between updates), and repeated read processing is fast (calculated data is stored between reads).

Derivation may access linked nodes. Before a node is derived, all out-of-date nodes to which it links are derived. Conversely, when a node is changed, not only is it set out of date, but all nodes which link to it are also set out of date. By managing these dependencies, Metrici always returns a completely up-to-date view of data.

Metrici contains further optimisations. There is a process that performs derivation in the background so that data can be recalculated in advance of being needed. There are settings to control which nodes are set out of date when a node changes, and whether nodes can be accessed in their out-of-date or "stale" state.

Inheritance is a special case of derivation. Inheritance copies members from another node as part of the derivation process. When applied to node types, this provides something similar to class-based inheritance in object-oriented system. It can also be applied in a less structured way. For example, you could use it to the styles across all the pages in a website.

Derivation in Metrici is very powerful and efficient. It allows complicated processing, such as calculating and formatting a report, to be broken down to a series of steps which are automatically rerun when data changes. It allows Metrici to exploit the versatility of triple stores without a performance penalty.

Subscription