3 November 2009

Necessary complexity

By Andrew Clifford

Necessary complexity is a unifying principle that should underpin our IT management and design.

As you go about your work, what do you think about?

If you are like me, a lot of the time you think about the immediate task. Planning, proposals, design, programming.

But stepping back from the immediate, what sort of concepts do you use to help you think?

This is important for us in IT. IT is pure though stuff. It is an extension of our thinking selves. Like a builder uses concrete, bricks, mortar and wood, what are the mental materials we use when we put together IT solutions for our organisations and clients?

A lot of the mental constructs we use are not specific to IT. We have ways of thinking about time and money and political relationships. These are important, but they would be much the same if we were working in any field.

We also think about the technology detail. We think about how much RAM our servers need, what operating system to install, how to configure firewalls, and the syntax of the substring command. These are specific to IT, but they are not really concepts. They are just the syntax for representing the ideas we have about IT. But where do we get those ideas?

To crystallise the question, what are the mental constructs we use that are particularly related to IT and are not just the detail of the technology?

I can think of a few. Normalisation, partitioning, componentisation. These have analogies in other fields, but their meaning in IT is very specific.

But there is one concept that underpins a large part of this: necessary complexity. It is something we should always think about, something that permeates all the aspects of our IT work.

Necessary complexity can be defined as, "as complicated as it has to be, but no more". Necessary complexity is important generally, but particularly in IT because IT has both the power to simplify and the ability to confuse.

Many of the concepts we use in IT boil down to the pursuit of necessary complexity. For example, we use normalisation techniques to fully represent the semantics of data in a form that can be stored as efficiently as possible. Normalisation is a way of making data as complicated as it needs to be, but no more. Normalisation is the pursuit of necessary complexity in data design.

The same applies to IT architecture and methods. Huge frameworks have been defined for IT architecture and methods to cope with the complexity of the IT task. But these have in many cases gone too far, and there has been a counter-movement to simpler designs and more agile methods. As an industry, we are searching for architecture and methods that are as complicated as they need to be, but no more.

When we are considering what IT response is appropriate for our organisations and clients, we need to think of necessary complexity too. We need to define sophisticated approaches that meet the complexities of the requirements. But often we get carried away with the designs and technology, increasing complexity but not value. We need to take a step back, and make sure that all the complexity is necessary.