Minimal integration 6: standardise data transfer

By Andrew Clifford

"We like standards - we've got lots of them" is a good description of most businesses' approach to transferring data between systems. If you want any-to-any integration, you need to reduce the number of different data transfer technologies.

There are many technologies for transferring data between systems, such as file transfer, shared databases, messaging middleware, web services and remote procedure calls. How do you pick a standard data transfer technology that is compatible, simple and effective?

There are three requirements for a standard data transfer technology.

• It must transfer the data you want. Assuming you use XML, you simply need to be able to transfer a string of characters that contains data coded as XML.
• It must support the patterns of transfer that you want. Most integration is "asynchronous", a one-way transfer of data in which the sender does not wait for a reply. (This allows systems to run at different times, which helps keep them flexible.) Some integration is "synchronous", a two-way transfer of data in which the sender must wait for a reply before continuing.
• It must support all the different operating systems and business systems that you use. There's little point in a standard that can only work on a fraction of your systems.

Ideally you would pick a data transfer technology that meets all these requirements directly.

If a data transfer technology does not meet these requirements directly, you can usually bend it so that it meets them somehow. You can implement simple string passing using database connectivity by defining tables that contain long strings. You can implement asynchronous transfer by building a store-and-forward component on top of synchronous transfer, or implement synchronous transfer by building a poll-for-reply component on top of asynchronous. You can use gateways to bridge to systems that are not supported by your chosen data transfer technology. All of these methods add complexity. Avoid them if you can.

So, what data transfer technologies should you use? In approximate descending order of preference, here is my list.

• For a large organisation, use message-oriented middleware. IBM's WebSphere MQ (WMQ, formerly MQSeries) is the market leader. It meets all of the requirements amply.
• In a Java environment, Java messaging services (JMS) provides a consistent messaging interface. JMS can be configured to use other underlying messaging services, so you can use it with something like WMQ to integrate with non-Java systems on other platforms.
• Web services provide a cross-platform XML-based transfer technology. Most current implementations are synchronous, so you will need to build an additional store-and-forward mechanism to implement asynchronous transfer.
• Simple files and file transfer can provide good integration solutions. It is hard to build effective synchronous solutions with files.
• Where databases have been standardised, a central database to store messages can provide good asynchronous data transfer.

It would be unrealistic to suppose that you could ever have just one standard technology. But minimising the technologies you use will pay dividends.

Remember that no standard lasts for ever. Always keep data transfer separate from the processing of the business-relevant content of the data, so that you can change the data transfer with minimum impact to your systems.

Subscription