Minimal IT logo and link to home page
Research, training, consultancy and software to reduce IT costs
Home | About | Newsletter | Contact
Previous | Next Printer friendly
27 November 2012

Why programming takes so long

By Andrew Clifford

Some things that look simple can take a very long time.

Over the past week I have been working on something that is much harder than it looks, and I though it might be interesting to share the experience.

The requirement was to paste data in from Excel. This seems like a simple requirement: it is something we do every day, and lots of programs support it.

At a more detailed level, I needed to meet lots of technical requirements.

  • It has to support both the format used when you paste directly out of Excel (where values are separated by tab characters), or when you export a comma separated values (CSV) file.
  • CSV data contains lines of values separated by commas, with quotes around values that contain special characters. But there are lots of variations. For example, in continental Europe where a comma is used as a decimal point, CSV files can use semicolons instead of commas.
  • The program usually has to interpret the format of the incoming data automatically, but also needs options to specify what separator and quote characters are used.
  • The program needs options to ignore blank lines, or lines that are intended as comments.
  • Different technologies have different rules for new lines. Windows uses a carriage return character followed by a line feed character; Unix uses just line feed; and some Apple systems use just carriage return.
  • Pasted data and CSV files may have column names as the first row. The program has to cope sensibly if there are more data than columns.
  • The program needs to output data to programs which have different formatting needs. Some need cleaned-up CSV. JavaScript programs need the data in JavaScript Object Notation (JSON) format. Other programs need Extensible Markup Language (XML), optionally using column headings as the names of the XML elements.

What looked like a simple requirement has ended up complicated, and is a few days work even under ideal conditions. To develop the new component I used existing components as much as possible, particularly handling the XML and JSON output. The only thing I had to program from scratch was the logic to interpret the data. I have a lot of experience of this type of code. I am completely familiar with the development and test environment. But even under these near-perfect conditions, the component required 750 lines of code, 450 lines of test code, and took me 20 hours to develop. If it was an area I was less familiar with, or where I had fewer existing components, or where the development and test environment was unfamiliar, it would have taken me many times longer, perhaps around 100 hours.

Now that we have this new component, we can now use it to meet other seemingly more complicated requirements really quickly. It only took me about four hours to develop a new bulk emailer component with it (for sending out and following-up on surveys). And I can add that component to new solutions in a matter of minutes.

The time taken to add a new feature depends hugely on the components available and the developer's experience of the situation. Sometimes things that look simple take ages, and things that look hard take no time at all.

Next: Video game training

Subscription

Subscribe to RSS feed

Latest newsletter:
Magical metadata

We use the term "metadata-driven" to describe IT solutions in which functionality is defined in data. Taking this to the extreme can provide unparalleled levels of speed, simplicity and versatility.
Read full newsletter

System governance

System governance helps you implement high-quality systems, manage existing systems proactively, and improve failing systems.

Find out more