Facts Not Opinions
The need for Experimentation
As aspiring Software Craftsmen we are always looking for way to raise the bar for professional software development. Practicing the crafting of high quality code that keeps adding value is essential. Working together in a community of professionals enables us to teach and learn from our shared experience.
However, if we really want to raise the bar of software craftsmanship, I believe we also need to perform experiments. Do you agree with me? If you don’t, perhaps it is because we have a different idea about what an experiment is.
If we look back in history we see the industrial revolution changing the world when a groups of craftsman came together to raise the bar for their various industries. Back then craftsmanship and engineering were the same thing. It is a time we have much to learn from them. Let’s consider one man, testing pioneer David Kirkaldy.
If you visit Kirkaldy’s Testing museum on Southwark High Street, just behind the Tate Modern, you’ll see David Kirkaldy’s motto carved above the door: “facts not options.” I think we have a lot to learn from David Kirkaldy. He performed experiments that replaced conflicting opinions with clear facts. He was made famous for his work related to the first Tay Bridge.
Bridge Building and Software Development
It is often said that writing software is not like building bridges, as Stack Overflow founders Jeff Atwood puts it: (original emphasis)
I find these discussions extremely frustrating, because I don’t think bridge building has anything in common with software development.* It’s a specious comparison. Software development is only like bridge building if you’re building a bridge on the planet Jupiter, out of newly invented materials, using construction equipment that didn’t exist five years ago.
We don’t have to go very far to find a time when bridge building was just like software development. It was just one hundred and fifty years ago that bridges were being built out of newly invented materials using construction equipment that didn’t exist a few years before.
When we think of timeless engineering we might envisage the Fowler and Baker’s Forth Bridge. Many might be surprised to discover that the famous design was not the first to be chosen. An earlier design by Thomas Bouch had been commissioned and the foundation had already been laid before the design was found to be inadequate.
Let us consider Thomas Bouch for a moment. He had an impressive reputation, having helped with the invention of train ferries and the construction of rail lines. His Tay Bridge had successfully passed three days of inspection to be declared safe for public traffic. In June 1879 Queen Victoria herself travelled across the bridge and knighted Bouch for his achievement. He appeared to have an impressive track record of successful projects despite some mishaps caused by poor engineering practices and sloppy shortcuts.
The general opinion was that Bouch was an excellent engineer. The facts, however, were quite different. In December 1879, the Tay Bridge collapsed with 60 lives lost. The official enquiry concluded that the bridge had been “badly designed, badly built and badly maintained, and that its downfall was due to inherent defects in the structure, which must sooner or later have brought it down.”
A badly designed bridge was poorly constructed and yet it managed to pass what appeared to be rigorous acceptance testing. Then it was poorly maintained before finally collapsing disastrously. Is it possible to imagine that bridge building back then was anything like software development is today?
So how did bridge building change? Is there anything that we can learn from the pioneers who brought about those changes? Can we follow in their footsteps so that the crafting of code might one day be held in the same regard as the building of bridges?
As Software Craftsmen we may be able to relate with surprising ease to David Kirkaldy and the way the Scottish engineer revolutionised testing with the invention of the Tensometer,
From Opinions to Facts
Alistair Cockburn describes the difference between bridge building and software development:
Civil engineers who design bridges are not supposed to invent new structures. Given a river and a projected traffic load, they are supposed to take soil samples and use code books to look for the simplest structure that handles the required load over the given distance, building on the soil at hand. They base their work on centuries of tabulation of known solutions.
Chapter 1 of Agile software development: the cooperative game
“Centuries” may be over stating the length of time these tabulations have been recorded for. For wrought-iron and steel it started with David Kirkaldy. When he started investigating the matter in 1862 he was surprised to discover very little was to be found:
It seems remarkable that whilst we have the results of many important and reliable experiments on Cast-iron, extremely few have been made, or at least published, on Wrought-iron, and almost none on Steel.
While very little hard experimental data was to be found on the subject, there was no shortage of opinions:
Although much has been written on the subject of wrought-iron and steel, yet, such is the great diversity of opinions held and stated by different individuals.
Kirkaldy’s solution to the problem was to provide experimental data.
It is hoped the results of these experiments, intended simply to elicit the truth, will be considered worthy of examination by those interested, and also at the same time prove of practical utility.
The same lack of facts and diversity of opinions has been observed in software In his book “Software Conflict” Robert L Glass observes:
In the processional literature we tend to see opinions presented as truth and advocacy presented as fact, with nothing acknowledging the tentative nature of some of these facts and much of this truth. Even noted computer scientist David Parnas has labeled much of our computer science truth “folklore,” because it has not been experimentally verified.
How can we progress past the diverse opinions of advocacy to a better understanding of the materials we work with? Is it only possible with large universities and generous grants? Is it within the practical reach of regular practitioners like ourselves?
Kirkaldy was a regular practitioner just like us. He built the tool needed and carried out his experiments as a personal side project:
At the time it was only intended to test a few specimens of each, but the investigation proved so interesting in itself and so likely to conduce to important practical results, that I was induced… to extend the experiments, as leisure and opportunity offered, very considerably beyond what had been originally contemplated.
His invention, the Tensometer, was the jUnit of its day, it wasn’t overly clever but it did it’s job well:
The apparatus employed was of the simplest construction, and proved during the experiment to work most satisfactory.
You can go and see it today, in the Kirkaldy’s Testing museum in London on Southwark High Street, just behind the Tate Modern. To our eyes it may look like a monster of a machine, but working at this scale was all in a day’s work for the Victorian engineer.
Is experimentation applicable to Software Craftsmanship? Certainly we do not work with iron and steel like the Victorian engineers, but that does not mean that experimentation is not applicable. These engineers were learning from the work of chemists who worked with liquid and gases. The materials were very different, but the principles of truth and rigour remained the same.
Practical vs Academic Experimentation
As craftsmen we are practical people. Do we really have time to experiment? Is it really a productive way to add value?
Today we think of experiments as the exclusive realm of the academic given to the pursuit of abstract goals, not the hard working practitioner. This was not always the case. For David Kirkaldy experimentation was essential if the conflicting complex theories and opinions were to be replaced with straightforward, simple facts.
The academics were there, too. They were usually wealthy members of the Royal Society. Being of independent means they were not bound by the need to earn a living. Sometimes there would be conflict between the two groups of experimenters. Consider, for example, the Safety Lamp, an important invention that was made, independently through careful experimentation, by two different people during the same year: 1815.
Sir Humphrey Davey invented the Davey Lamp. He was a knight of the realm, first baronet and a fellow of the royal society. He was already famous for his work on gases, such as the discovery of laughing gas (nitrous oxide). His lectures were well attended in fashionable London. No only did he invent a safety lamp, but he also progressed the scientific understanding of hydrogen.
It would be a couple decades before George Stephenson would become historically famous for building the first railways. He was an unknown engine-wright in the north of England responsible for maintaining and repairing the steam engines used at the collieries of Killingworth.
Stephenson was largely self educated and his experiments did not make him popular. However he made them with a practical purpose in mind. Samual Smiles relates:
For several years he had engaged, in his own rude way, in making experiments with the fire-damp in the Killingworth mine… One of the sinkers, observing him holding up lighted candles to the windward “blower” or fissure from which the inflammable gas escaped, entreated him to desist; but Stephenson’s answer was, that “he was busy with a plan which he hoped to make his experiments useful for preserving men’s lives.” On these occasions the miners usually got out of the way before he lit the gas.
Stephenson’s work did prove useful. His theory was flawed and his risk management was poor: The experiments with his prototype lamps involved carrying it into a pit known to be full of explosive gasses. He did not advance scientific theory, but he did create a working safety lamp. Some controversy followed regarding who should take credit for the invention.
While Davey was clearly the better scientist, the question remains regarding whose experiments yielded the best results. In normal conditions both lamps performed equally well. However, in some exceptional circumstances there was a very important difference. Davey’s would burn red hot and potentially cause an explosion while Stephenson’s would safely go out:
A sudden outburst of gas took place… Upon this occasion, the whole of the Stephenson’s lamps, over a space of five hundred yards, were extinguished almost instantaneously; whereas the Davy lamps filled with fire and became red-hot… Had a strong current of air been blowing through the gallery at the time, an explosion would most probably taken place.
The lamps were put to rigorous testing by Dr Pereira for the Committee on Accidents in Mines. While both lamps had their faults, the conclusion was that “when exposed to a current of explosive gas the Davy lamp is ‘decidedly unsafe,’ and that the experiments by which its safety had been “demonstrated” in the lecture-room had proved entire ‘fallacious.'” On the ground practical experimentation results in better products, not inferior science. I know which lamp I would have preferred.
Having established that the experiments are not only carried out by serious scientists in white lab coats but also sober mutton chopped engineers in frock coats, how about software developers in t-shirts and trainers? Does experimentation have any place when writing software?
Experimentation was once common practice among practitioners. In his book “Software Conflict” Robert L Glass relates the finding of an area of research called “protocol analysis” where observers would sit quietly and observe practitioners at work. They filmed them and then scrutinised the tapes to see how the design process worked. The process is familiar to all of us:
- Understanding the problem
- decomposing the problem into goals and objects
- selecting and composing plans to solve the problem
- implementing the plans
- reflecting on the product and the process
It is the way in which these software engineers pursued the second step, the decomposition of the problem into goals and objects, that we might find surprising.
The designers, mentally and at lightening speed, were doing the following things:
- The constructed a mental model of a proposed solution to the problem.
- They mentally executed the model – in essence, running a simulation of the model – to see if it solved the problem.
- When they found that it didn’t (usually because it was too simple), they played the inadequate model back against those parts of the problem to see where it failed, and enhanced the model in those areas.
- They repeated steps 1-3 until they had a model that appeared to solve the problem.
Are you taking an Agile approach to your coding? If you are, then good for you. Now, are you following the approach described above? Are you constructing models, either in your head or as tests, and then testing them to destruction so that you can find where the model fails and improvement is required?
Or does your process more closely resemble the approach taken by those teams that were observed to fail:
These same researchers have explored the problems of people who are not very good at design. Those people tend to build representations of a design rather than models; they are then unable to perform simulation runs; and the result is they invent and are stuck with inadequate design solutions.
In Scrum I have observed the process where architects create complex, epic stories that span many months of work. These stories are a representation of the final system. They are broken down into smaller stories that are then fed to the developers in sprint size chunks. The developers are simply order takers, being fed their orders just in time. They have no opportunity to build and refine their own models through practical experimentation.
A recent thread on the Lean seems to indicate that I am not alone. John Herbert describes a similar situation.
Relegating developers to ‘order takers’ is exactly what I am describing. This is basically what the developers are saying, they have no room/time for creative thinking. That is at the root of my question. How do we achieve this balance? The dev team is on pretty strict 2 week sprint schedule, but it seems the 2 weeks has no time built in for anything other than getting the specific requirement developed. Where is the ‘alternate’ solutioning, or outside the box thinking achieved?? …
So where do we assign time for innovation without excluding any members of the team?
Are you working on a tight sprint schedule working hard to deliver the backlog items promised? Are you in a position to say “Stop!” Can you declare the current approach a failure, and go back a few steps to pursue an alternative approach? Are such ideas unrealistic? Is is simply impractical? Then how come that was how successful programmers were able to work?
Our Software Craft
If we are Software Craftsmen then what is our craft? Is it the creation of code? No! If typing out lines of code is what we aspire to do how can we ever follow this principle of the Agile Manifesto?
Simplicity–the art of maximizing the amount of work not done–is essential.
If the best code we will ever write is the code we avoid writing then how can writing code be our craft?
As Software Craftsmen we solve problems using code. To build those solutions we may use code, but first we must solve the problem.
Ask yourself: are you a Software Craftsman? If you are just taking orders rather than solving problems then you are not a craftsman, you are a factory worker. If you are learning, then is your mentor showing you how they solve the problem? Are they sharing their craft or simply giving orders?