22 January 2013

Multidisciplinary performance improvement

By Andrew Clifford

There are many ways to improve system performance, but the best performance improvements come from a multidisciplinary approach.

My experience with performance improvements go back to the very first project I worked on. We were writing a program to format a file of stolen credit cards to send to point of sale systems. The program was taking about 12 hours to run every night. We changed the program, from using an indexed file to a non-indexed file and a sort. The program then took less than 1 second to run.

This experience made me suspicious of performance improvement techniques. A lot of performance improvement tools and techniques involve analysing code, and tuning hardware and system software. But in this case that would not have made any difference. The problem was more fundamental, and no amount of code analysis or system tuning would have made any difference.

Recently we have needed to do some work to improve the performance of the Metrici platform. Historically we have had very few performance problems, but more recently, as the service has grown, we noticed things were slowing down and we needed to do something about it.

I was not sure where to start. We have plenty of hardware resources, and, thanks to a very skilled colleague of mine, the database has always been well tuned.

To help understand the performance problems, we used the VisualVM analysis tool, which is a free tool that ships with Java. This confirmed our suspicions about the main cause of the slow down. Whenever data is changed, Metrici automatically recalculates derived data. This was the process that was taking all the time.

Understanding where the problems lie is only the start. Working out what to do about it requires a knowledge of the system. If something is taking a long time, you need to decide whether you can stop doing it, do less of it, do it more quickly, or change when it's done so it has less impact.

In the end, we improved performance by changing both the design and coding. We sped up the calculation process by caching a compiled form of calculation scripts and by replacing some scripts with built-in functions. More significantly, we changed the design of the derivation processing, reducing the number of things that have to be recalculated, allowing access to out-of-date data, and performing some recalculation in the background.

We have had to make other changes to support the performance improvements. We needed to re-tune the database and change how some parts of solutions are configured. The changes have been successful, and the performance issues have gone away.

This recent experience has given me a better insight into performance improvement. You can make improvements in performance by looking at just one aspect, such as hardware or coding. However, the best performance improvement is multidisciplinary. It requires a good knowledge of hardware and tuning, and a good grasp of coding. Most importantly, it requires a good understanding of the design of the system, which is at the root of the most significant performance problems and solutions.