8 November 2005

Minimal integration 6: standardise data transfer

By Andrew Clifford

"We like standards - we've got lots of them" is a good description of most businesses' approach to transferring data between systems. If you want any-to-any integration, you need to reduce the number of different data transfer technologies.

There are many technologies for transferring data between systems, such as file transfer, shared databases, messaging middleware, web services and remote procedure calls. How do you pick a standard data transfer technology that is compatible, simple and effective?

There are three requirements for a standard data transfer technology.

Ideally you would pick a data transfer technology that meets all these requirements directly.

If a data transfer technology does not meet these requirements directly, you can usually bend it so that it meets them somehow. You can implement simple string passing using database connectivity by defining tables that contain long strings. You can implement asynchronous transfer by building a store-and-forward component on top of synchronous transfer, or implement synchronous transfer by building a poll-for-reply component on top of asynchronous. You can use gateways to bridge to systems that are not supported by your chosen data transfer technology. All of these methods add complexity. Avoid them if you can.

So, what data transfer technologies should you use? In approximate descending order of preference, here is my list.

It would be unrealistic to suppose that you could ever have just one standard technology. But minimising the technologies you use will pay dividends.

Remember that no standard lasts for ever. Always keep data transfer separate from the processing of the business-relevant content of the data, so that you can change the data transfer with minimum impact to your systems.