14 November 2006

StupidClever methods

By Andrew Clifford

StupidClever is something that starts off clever, but ends up stupid. IT design methods and working methods become stupidclever when they are applied on too large a scale.

If you have one or two small computer programs, design methods are not that important. But for larger systems, design methods are critical. We use all sorts of design methods: structured programming, object orientation, component based development, data normalisation, and so on. These help us build larger and more complicated systems.

Some clever people try to scale the methods up even further, and apply them to design problems across multiple systems. But this is stupidclever. On a large scale, IT is driven by commerce and politics, not design elegance. Even with the budget and the backing, IT is too big to follow a single design.

You can see this most clearly with database design.

In a single system, it is clever to have a single, normalised database which all parts of the system use. But on a larger scale, projects and systems come and go for all sorts of short-term tactical and political reasons. It is unrealistic to force these into the same database. And it is certainly unrealistic to rework existing systems to meet new database requirements. Attempts to build a single, central database inevitably fail, with just a handful of systems using the database before it is labelled "legacy".

Something similar happens with working methods.

In a small project, or a small support group, everyone mucks in. As the project or group grows larger, it is clever to introduce specialism, a division of labour, and some formal processes.

Some clever people try to take this further. As the work grows, they subdivide it into ever more finely cut specialisms, supported by ever more detailed processes. But in the end, the overhead of subdivision exceeds the benefit of specialisation. The work is paralysed by bureaucracy.

Design methods and working methods share the same stupidclever principle.

When systems, projects or work groups are tiny, there is little or no need for methods of any type. As they grow in size, you need to introduce methods to subdivide and control the work.

But as systems, projects and work groups grow even larger, you need a different approach. You can not just keep adding more of the same methods, with more formalised design and more standardised processes. You need to split the work, and manage it as two or more systems, projects or work groups.

Some would see this as an admission of failure, that we can not manage large-scale IT. I do not. IT is a human activity. Our methods need to reflect this. In the real, non-IT world, we cope with growing scale by subdividing responsibility. We do this in business, government, public bodies, research, military, and every other human endeavour. We should do it in IT too.

IT is complicated, and it needs clever design methods and working methods. But you have to know where to stop being clever and start being human. Stretching the method too far, and forgetting how we, as people, need to manage IT, is definitely stupidclever.