Principles of Software Flow

Greg Wilson’s What We Actually Know About Software Development

There is an excellent talk from CUSEC given by Greg Wilson.  It’s called “What We Actually Know About Software Development, and Why We Believe It’s True.”

You can watch it here:

Here’s my notes on the talk.

The Lack of Evidence


Martin Fowler’s claims about Domain Specific Language

  • Using a DSL leads to:
    • Improved Productivity
    • Improved Communication
  • Although the article is an academic journal no citation or evidence is given.
  • A scotish verdict:
    • True
    • False
    • Not proven
  • Fowler claim’s that the debate is hampered because people don’t know how to do DSL’s properly.
  • Wilsno believes it is because of the low standards of proof.
  • We should have higher standards.
  • Things are getting better: there are more results from field studies.
  • Standards improve each year.

Estimation and Anchoring


Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation”

Three groups are given a two page specification each. They are identical but for one paragraph. That one paragraph had a strong effect on the estimates provided.

Group Estimate Anchoring Paragraph
(lowball) 5.1 months “I admit I have no experience with software projects, but I guess this will take about 2 months to finish.”
Control 7.8 months “I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculation for an estimate.”
(highball) 15.4 months “I admit I have no experience with software projects, but I guess this will take about 20 months to finish.”

Rock Star Programmers and Poor Evidence Standards


“The best programmers are up to 28 times more productive than the worst”

  • Sackman, Erickson and Grant (1968) “Exploratory experimental studies comparing online and offline programming performance”
  • Study was designed to compare batch vs iterative approaches, not productivity.
  • Productivity measure was not explained.
  • Best vs worsts always exaggerates the effect. Standard deviation around the mean is better.
  • Just 12 programmers for an afternoon.
    • The next “major” study was 54 programmers for less than an hour.
  • In 1968 every programmer was self taught.

Improving Productivity


Look at the work of Lutz Prechelt

  • variations between programmers
  • effects of language
  • web programming framework

Studies are expensive and hard to do.

  • Not compared to the cost of drugs research.
  • A 5% productivity increase in a trillion dollar industry is worth a lot.

Two approaches of Pessimism and Optimism


Boem et al (1975) “Some Experience with Automated Aids to the Design of Large-Scale Reliable Software”

  1. Most errors are introduced during requirements analysis and design
  2. The later they are removed the more expensive it is to take them out.

Two approaches to this problem:

  • Pessimists
    • If we tackle the hump in the error injectino curve, fewer bugs will get to the expensive part of the fixing curve.”
  • Optimists
    • “If we do lots of short iterations, the total cost of fixing bugs will go down.”

Why are there so few women in software development?


Ceci & Williams (eds): Why Aren’t More Women in Science? Top Researchers Debate the Evidence.

There’s a review here:


Carol S Dwek’s essay (pdf) Is Math a Gift? Beliefs That Put Females at Risk 1)

  • Split the group into two groups.
    • Group One: “The tasks requires aptitude. ”
    • Group Two: “This task is entirely practice based.”
  • Group One does worse.
  • Even if the group are told that men have the aptitude more than women the men will still do worse.
  • When a difficulty is encountered the group members will conclude that they lack the aptitude and quit.

Improving Productivity (reprise)


  • For every 25% increase in problem complexity, there is a 100% increase in solution complexity. (Woodfield 1979)
    • Non-linear due to interaction effects.
    • Reducing problem complexity reduces the solution complexity.
    • Maybe this is why Agile works?
  • The two biggest causes of project failure are poor estimation and unstable requirements. (van Genuchten 1991 and many others)
    • There is no evidence that this is improving.
    • “I want you to make the web site more webish and not to webish. How long will that take?”
  • If more than 20-25% of a component has to be revised it’s better to rewrite it from scratch (Thomas et al, 1997)
    • Based on Flight Avionics. Very strict requirements.
    • Does it apply in other domains

Code Reviews


  • Rigorous inspections can remove 60-90% of errors before the first test is run. (Fagan 1975)
    • Code review is the best bug fixing method.
      • Better than unit tests.
      • Better than executing the code.
  • The first review and hour matter most. (Cohen 2006)
    • Having two people read the code is not economically effective.
    • How much code can you read in an hour?
    • A couple of hundred lines is as much as you can get through.
    • This supports the idea of making progress in little steps.
      • Open source projects reject large patches.

Conway’s Law


A system reflects the organisational structure that built it.

  • It was meant as a joke and turned out to be true. (Herbsleb et al 1999)
  • Physical distance doesn’t affect post-release fault rates but Distance in the organisational chart does.
    • Nagappan et all (2007) and Bird et al (2009)
    • Based on all the data from building Windows Vista. An enormous volume of data.
    • Searched for indicators of post release defect.
    • This goes against claims for the need for co-location.
    • Different managers with different goals has more impact than different continents.
  • Does this Explain why big companies produce such bad code.
    • Somebody from HP corporate headquarters: “We can’t just have people running around doing the right thing: there are rules!”

Scientific Progress


“Progress” sometimes means saying “Ooops.”

  • Science is different to religion because it accepts it’s mistakes.
  • For example, there appeared to be strong statistical evidence that cod metrics could predict post failure rates.
  • However, a later study showed that the co-relation was actually between code size and failure rates.
    • El Emam et al (2001): “The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics”
      • Most metrics’ values increase with code size.
      • If you do a double-barrelled correlation, the latter accounts for all the signal.
  • Sophisticated metrics are not needed: just a line count.
  • The results are disappointing, but raising the standards is a good thing.

Folk Medicine for Software


“Systematizing and synthesizing colloquial practice has been very productive in other disciplines.”

  • Look at what people are actually doing.
    • Take it back to the lab and investigate why it works.
  • The next decade of Software Engineering is looking at success and understanding why.

Beautiful Code


“What code is worth looking at, what code is worth reading?”

Beautiful Evidence


The Book Wilson was writing at the time. It’s now published: Making Software: What Really Works, and Why We Believe It

“What we know and why we think it’s true”

  • Knowledge transfer
  • A better textbook
  • Change the debate

A group of people will choose one thing they believe to be true about software development and they give the evidence as to why.

All text books for Software Development are crap.

  • Who really draws UML diagrams when thinking?

Software Craftsmanship


  • “It’s wondferful, but… show me the evidence”
  • Every ten years we need a new bandwagon so we can look innovative.

1) Wilson refers to it as Susan Dwek’s work
2) The name has already been stolen for a book by Edward Tuftle that is well wroth reading.


Trackbacks & Pingbacks

  1. Must Watch Talks to catch up on Software Engineering Research | iThink pingbacked on 2 years, 6 months ago


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: