Principles of Software Flow


My Book Reviews on the LJC Book Club

I’m running the Book Club for the LJC and I’ve posted a couple of book reviews there:

I have a few more reviews in the pipeline for Cassandra, OSGi and Flow Based Programming.


Greg Wilson’s What We Actually Know About Software Development

There is an excellent talk from CUSEC given by Greg Wilson.  It’s called “What We Actually Know About Software Development, and Why We Believe It’s True.”

You can watch it here: http://vimeo.com/9270320

Here’s my notes on the talk.

The Lack of Evidence

11:10

Martin Fowler’s claims about Domain Specific Language

  • Using a DSL leads to:
    • Improved Productivity
    • Improved Communication
  • Although the article is an academic journal no citation or evidence is given.
  • A scotish verdict:
    • True
    • False
    • Not proven
  • Fowler claim’s that the debate is hampered because people don’t know how to do DSL’s properly.
  • Wilsno believes it is because of the low standards of proof.
  • We should have higher standards.
  • Things are getting better: there are more results from field studies.
  • Standards improve each year.

Estimation and Anchoring

14:50

Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation”

Three groups are given a two page specification each. They are identical but for one paragraph. That one paragraph had a strong effect on the estimates provided.

Group Estimate Anchoring Paragraph
(lowball) 5.1 months “I admit I have no experience with software projects, but I guess this will take about 2 months to finish.”
Control 7.8 months “I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculation for an estimate.”
(highball) 15.4 months “I admit I have no experience with software projects, but I guess this will take about 20 months to finish.”

Rock Star Programmers and Poor Evidence Standards

18:18

“The best programmers are up to 28 times more productive than the worst”

  • Sackman, Erickson and Grant (1968) “Exploratory experimental studies comparing online and offline programming performance”
  • Study was designed to compare batch vs iterative approaches, not productivity.
  • Productivity measure was not explained.
  • Best vs worsts always exaggerates the effect. Standard deviation around the mean is better.
  • Just 12 programmers for an afternoon.
    • The next “major” study was 54 programmers for less than an hour.
  • In 1968 every programmer was self taught.

Improving Productivity

21:05

Look at the work of Lutz Prechelt

  • variations between programmers
  • effects of language
  • web programming framework

Studies are expensive and hard to do.

  • Not compared to the cost of drugs research.
  • A 5% productivity increase in a trillion dollar industry is worth a lot.

Two approaches of Pessimism and Optimism

23:40

Boem et al (1975) “Some Experience with Automated Aids to the Design of Large-Scale Reliable Software”

  1. Most errors are introduced during requirements analysis and design
  2. The later they are removed the more expensive it is to take them out.

Two approaches to this problem:

  • Pessimists
    • If we tackle the hump in the error injectino curve, fewer bugs will get to the expensive part of the fixing curve.”
  • Optimists
    • “If we do lots of short iterations, the total cost of fixing bugs will go down.”

Why are there so few women in software development?

25:55

Ceci & Williams (eds): Why Aren’t More Women in Science? Top Researchers Debate the Evidence.

There’s a review here: http://www.americanscientist.org/bookshelf/pub/changing-assumptions

28:22

Carol S Dwek’s essay (pdf) Is Math a Gift? Beliefs That Put Females at Risk 1)

  • Split the group into two groups.
    • Group One: “The tasks requires aptitude. ”
    • Group Two: “This task is entirely practice based.”
  • Group One does worse.
  • Even if the group are told that men have the aptitude more than women the men will still do worse.
  • When a difficulty is encountered the group members will conclude that they lack the aptitude and quit.

Improving Productivity (reprise)

30:58

  • For every 25% increase in problem complexity, there is a 100% increase in solution complexity. (Woodfield 1979)
    • Non-linear due to interaction effects.
    • Reducing problem complexity reduces the solution complexity.
    • Maybe this is why Agile works?
  • The two biggest causes of project failure are poor estimation and unstable requirements. (van Genuchten 1991 and many others)
    • There is no evidence that this is improving.
    • “I want you to make the web site more webish and not to webish. How long will that take?”
  • If more than 20-25% of a component has to be revised it’s better to rewrite it from scratch (Thomas et al, 1997)
    • Based on Flight Avionics. Very strict requirements.
    • Does it apply in other domains

Code Reviews

33:11

  • Rigorous inspections can remove 60-90% of errors before the first test is run. (Fagan 1975)
    • Code review is the best bug fixing method.
      • Better than unit tests.
      • Better than executing the code.
  • The first review and hour matter most. (Cohen 2006)
    • Having two people read the code is not economically effective.
    • How much code can you read in an hour?
    • A couple of hundred lines is as much as you can get through.
    • This supports the idea of making progress in little steps.
      • Open source projects reject large patches.

Conway’s Law

35:45

A system reflects the organisational structure that built it.

  • It was meant as a joke and turned out to be true. (Herbsleb et al 1999)
  • Physical distance doesn’t affect post-release fault rates but Distance in the organisational chart does.
    • Nagappan et all (2007) and Bird et al (2009)
    • Based on all the data from building Windows Vista. An enormous volume of data.
    • Searched for indicators of post release defect.
    • This goes against claims for the need for co-location.
    • Different managers with different goals has more impact than different continents.
  • Does this Explain why big companies produce such bad code.
    • Somebody from HP corporate headquarters: “We can’t just have people running around doing the right thing: there are rules!”

Scientific Progress

39:15

“Progress” sometimes means saying “Ooops.”

  • Science is different to religion because it accepts it’s mistakes.
  • For example, there appeared to be strong statistical evidence that cod metrics could predict post failure rates.
  • However, a later study showed that the co-relation was actually between code size and failure rates.
    • El Emam et al (2001): “The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics”
      • Most metrics’ values increase with code size.
      • If you do a double-barrelled correlation, the latter accounts for all the signal.
  • Sophisticated metrics are not needed: just a line count.
  • The results are disappointing, but raising the standards is a good thing.

Folk Medicine for Software

41:36

“Systematizing and synthesizing colloquial practice has been very productive in other disciplines.”

  • Look at what people are actually doing.
    • Take it back to the lab and investigate why it works.
  • The next decade of Software Engineering is looking at success and understanding why.

Beautiful Code

43:40

“What code is worth looking at, what code is worth reading?”

Beautiful Evidence

44:58

The Book Wilson was writing at the time. It’s now published: Making Software: What Really Works, and Why We Believe It

“What we know and why we think it’s true”

  • Knowledge transfer
  • A better textbook
  • Change the debate

A group of people will choose one thing they believe to be true about software development and they give the evidence as to why.

All text books for Software Development are crap.

  • Who really draws UML diagrams when thinking?

Software Craftsmanship

54:55

  • “It’s wondferful, but… show me the evidence”
  • Every ten years we need a new bandwagon so we can look innovative.

1) Wilson refers to it as Susan Dwek’s work
2) The name has already been stolen for a book by Edward Tuftle that is well wroth reading.


Analysts Guess Badly and Apple Is Blamed

There are two groups of people: those that do stuff and those that make guesses.

If the people who do stuff make a mistake, it is their fault.

If the people who guess are wrong, it the the fault of the people who do stuff.

Apple said Tuesday it sold more than 17 million iPhones in its fiscal fourth quarter ended Sept. 24, up from more than 14 million a year ago but lower than the 20 million or more that analysts had been expecting.

http://online.wsj.com/article/SB100...

Are the city analysts able to conclude that their estimates were wrong?   Of course not:

Some analysts said the results shouldn’t be seen as too negative. “We don’t think this is a slowdown in [market] share gains, it’s a pause,” said Brian Marshall, an analyst ISI Group.

ibid

What bump in the road?  You’re estimates were wrong.  Do a better job next time!

My guess is that Wall Street ]will continue to be blind to their own mistakes, no matter how many people try to point them out.


Woollard and the origins of Flow Production

Professor John Seddon has published an excellent article challenging the conventional wisdom around the need to achieve economies of scale.  I whole heartedly agree with the point being made, and the conclusion that is reached: “Economy of scale is a myth. Economy comes from flow.”

I would, however, like to challenge one assertion that is made: that the benefits of flow were first identified by Taiichi Ohno at Toyota.

Ohno minimised stock throughout the process, his ideal batch size being one. Whereas most manufacturers still focus on unit costs (and employ accountants for whom it is central to their doctrine: for example, inventory counts as ‘value’ on the balance sheet), Ohno focused on the flow of the work, confident that better flow would lead to lower overall costs. And so it did. His system would tolerate higher unit costs; it was not dependent on low costs per unit. What was critical was the availability of the part, not the cost – an affront to convention. Ohno was the first to demonstrate that greater economy comes from flow rather than scale.

His second and more profound challenge to convention was to put variety into the line, making different models in the same production line.

Why do we believe in economies of scale?

These principles were identified by Frank G. Woollard, who introduced flow at Morris and Austin.  In his 1954 book “The Principles of Mass and Flow Production” he set these principles out in detail.  On the subject of minimising stock and work in progress he writes on page 69:

One of the many advantages of flow production is that it reduces the inventory, that is the stock in stores and the work in progress…  The ideal objective… is an inventory kept to the lowest figure consistent with the maintenance of the flow of supplies to the production lines.  The reduction of working capital or the improvement of the liquid position due to a smaller inventory is not the only advantage; the lesser bin capacity absorbed, the smaller stores area required and the saving of double handling and of multiple paper-work are all very real economies.  Incidentally with flow production methods the reduction of inventory may be a as much as 75 per cent and even more.

Page  69 of Principles of Mass and Flow Production

In his article Professor Seddon presents a clear progression from Smith through Ford to Ohno, with manufacturing laboring under Smith’s flawed theory until Ohno developed the Toyota Production System.  Woollard’s writing suggests a more tragic story, where these principles were well understood in the factories of Britain’s industrial heartland, only to be lost in the decades that followed.  It seems likely that the principles were preserved or possibly rediscovered by Toyota.

In his comments on page 84 regarding the production of multiple products on a single production line Woollard shows that this knowledge was well dispersed and not limited to Birmingham, England.

At one period, for instance, the Ford Motor Co. handled, in sequence, all current types of cars and trucks – without pause or intermission – on one assembly line.  To-day, in their assembly plant, the Austin Motor Co.handle three body types and right- and left- hand steering on the same assembly track.  One American concern assembles 500 different sizes and types of air cleaners on one conveyor line.  They do not, in this instance, come in sequence but by a changeover limited to a 24 hour run for any one model.  These mutations can be matched on the machine lines provided sufficient care and attention is given to jig and fixture design, and the method of changing tools is carefully studied.

Page  84 of Principles of Mass and Flow Production

Seddon observes that the true cost in the public-sector factories is about the human factors and not just the economic figures.

One cost that is apparent is sickness, absenteeism and staff turnover. Being treated as a ‘resource’ to be ‘optimised’ is not motivating. Nor is the realisation that it is impossible to help people solve their problems because of the need to work to the internal arbitrary measures. In some respects life in modern public-sector factories is little different to the conditions that created Ford’s ‘five-day man’[4]. Both HMRC and NHS Direct currently report low levels of staff morale.

Why do we believe in economies of scale?

The fact that human engagement is more important than machine utilisation was also observed by Woollard and his contemporaries.  He relates this on page 45.

The Scani-Vabis Company of Sweden… claim that on group production there is a reduction of the cycle time fo some 40 percent; that the morale of these lines has been improved; that the workmanship is of higher quality; that the training time has been reduced, and that a greater degree of expertness is acquired.  They say that this latter is largely due to the greater interest engendered by several jobs flowing through the line.  It is true that certain machines are working at 10 per cent to 15 per cent less efficiency than would be achieved on the batch method : but, that be all, it is a small price to pay for the general improvement due to the group production system which they are hoping to extend.

Page  45 of Principles of Mass and Flow Production

Woollard saw the potential for Flow production as another “complete turn in the industrial revolution” (page 15) but he was also aware of the dangers and limitations.  Regarding human matters the warning given by Woollard in his closing comments is poignant  considering the tragedy of the decades that were to follow.

On the human side it must also be watched, for – like all tools of management – it can be misused.  Flow production, with its obvious sequences and accurate timing, could be the instrument of a slave-driving tyranny, whereas properly employed it will promote discipline in an equitable and gentle, if irresistible, manner, making the daily task lighter for all.

Flow production is, in fact, a logical development that has tremendous advantages and when properly applied is of benefit to the whole community.  These methods may not promote any individual art but they can provide a common basis for a comfortable existence , and, when they relieve mankind of the more arduous labours – as ultimately they will – those who labour can, if they desire, follow their bent as individual craftsman in their extended leisure hours

Page  187 of Principles of Mass and Flow Production

Growing up as a child in the 70s Birmingham was dominated by the motor industry and strikes and conflict.  Now very little of that motor industry remains.

The 1970’s and its associated strikes and management problems decimated the industry. Japanese imports made matters worse and the car and motor cycle industry went through many mergers and closures. The great names such as BSA and Triumph lost ground against the Suzuki’s and Yamaha’s from Japan and the Datsun and Honda’s looked set to finish off what remained of the British Motor Industry.

http://www.birminghamuk.com/motorindustry.htm

In Britain it was the mindset of slave-driving tyranny tyranny that triumphed.  Since Woollard’s time we have traveled further and further away from the goal of equity.


Lean’s 98+% Failure Rate

Bob Marshall finds Lean’s 98% failure rate scary.  So he should!  Why would anybody want to attempt something when success is so unlikely?

Just as I was about to throw all my Lean books into the bin, the wise words of Vic Reeve’s came to mind.

“88.2% of Statistics are made up on the spot”

Less than 20% of Statistics are actually based on any facts!   It is possible, even probable, that this figure of 98+% is just a work of pure fiction.  I need to find out where this number comes from.

Some people point to an Evolving Excellence blog post as a source, where we find Bill Waddell doing the hansei:

I keep using the Clifford Ransom numbers – 98%+ lean failure rate – which most folks seem to think jives with our feel for the situation.

Here we see the figure attributed to Clifford Ransom, a man with fine Lean credentials.  It took some effort to find the original source.

Bill Waddell quotes the figure with full attribution in a Super Factory article, but the link provided there is now dead.  Fortunately, there is a title given: “Lean Manufacturing: Fat Cash Flow“.

The original interview with Clifford Ransom by Dr. Robert Hall, AME Target Editor, can be found in full on the BMA Inc website.  Here we find some context and the actual definition of failure.

Q: Do you track many lean manufacturers?

A: No. Very few companies have advanced with lean manufacturing until you can see the results financially — perhaps one or two percent at best. Another two-three percent are “getting there” —OK but not outstanding. Another 10-15 percent mostly “just talk lean.” The majority, 80 percent or so, don’t even have the buzz words straight. Unless I see three pieces of evidence, I do not consider a management to be serious about lean manufacturing. 1) They must proclaim that they are becoming lean. They can call it whatever they want, but intentions must be boldly stated in a vision that everyone can understand. 2) They must tie compensation to lean systems. You are not becoming lean if you reward people for doing unlean things. 3) They have to drive the company with lean metrics — time and inventory measures. You have to persist to see results. You won’t see much change in the financials for 12 to 18 months, sometimes longer. Clearly, confirming the sustainability of superior performance takes much longer — years. Most managements waffle around, make only a half-hearted attempt, and never get rid of the inconsistencies in their own leadership.

The figure of 98+% and the word “fail” do not occur here.  What is says is that only one or two percent, at best, advance with lean  manufacturing to a point where the results can be seen financially.

The inversion of 1-2% at best to 98+% is made by Bill Waddell in his article, where he paraphrases the interview:

But there is, according to Ransom, a 98%+ probability that whatever looks so lean on the shop floor makes no difference to the bottom line of the company.

So it turns out that this frightening statistic was not made up on the spot after all.  However, now we have the context there is clearly nothing to be afraid of.

98%+ lean failure rate

lean = “what looks like lean”, including attempts that “waffle arround” and make only a “half-hearted attempt”.

fail = “No difference to the bottom line of the company” significant enough to attract the interest of a Wall Street investor.

webinar with Clifford Ransom explains why so many Lean initiatives will fail in this way:

Lean is a terribly fragile thing. It is not robust, it can fail, it needs constant feeding and watering and reinforcing and scrutiny. And quite frankly I think it probably fails much more often than it succeeds. It’s counterintuitive, it’s innovative, it forces new ways of thinking. I think that empowering employees can be scary for both bosses and employees in some instances. There — I talk about the failure rate of Lean and I guess this slide would be better why companies fail at lean, or fail to even start at Lean. Change is threatening.

In the same webinar the 98+% figure is revised a little more favourably

I think there’s really only 5%who practice the art skilfully in a world class master practitioner kind ofway. I’m actually mellowing in my old age. I used to say only 2 to 3% of companies did it.

Perhaps Clifford Ransom’s third criteria for true Lean success explains why so many find it hard to attain:

3) They have to drive the company with lean metrics

Successful Lean is driven by the numbers, and it seems that people struggle to understand the numbers and their implications.  Why else would somebody use the following metrics to support a case for Lean’s failure?

Supplemental Evidence

a) “Only 2% of the companies reported achieving World Class manufacturing status.”

b) The 2007 IndustryWeek/Manufacturing Performance Institute Census of Manufacturers is a study of manufacturing metrics, management practices and financial results at the plant level.

17.8% say continuous improvement programs led to a major increase in productivity:

67.2% report some increase

12.4% report no change

2.2% report some decrease

0.5% report a major decrease

c) 10-20% of leaders in a typical organization are unable or unwilling to make the lean conversion.

Any approach that may lead to world class performance must be worth a try.  Larry Rubrich’s figure of 2% of companies achieving a “world class” manufacturing status is in line with Clifford Ransom’s observations.  He credits an Industry Week census as the source.

The results from another Industry Week census in 2007 provides a clear story for the success of continuous improvement, a pillar of Lean.  85% of companies succeeded in improving their productivity and nearly 20% achieved a dramatic improvement.  On the failing side we see 15% who reported no change or decline, and the 10-20% who are unwilling to even give it a try.

Implementing lean is like taking regular exercise.  It isn’t easy but done right it can benefit anybody.  My own abysmal failure to maintain an exercise regime does not change this.

Exceptional athletes have the dedication to take their exercise and training all the way to Olympic gold.  Their achievements do not make the rest of us failures, their achievements inspire us all to try harder.


The Merging of Lean and Agile

It has been nearly 25 years since Fred Brooks assured us that there would be no silver bullets to help us slay the werewolf of software developement as it transformed apparently mild mannered projects into an insatiable monsters baying for our blood.

Brooks seminal essay was written in 1986, an nearly a quarter of a century has passed since then. Has nothing changed, are things still exactly as they were? No, they are not.

The expectation for so long was that building software would become more and more like manufacturing. The reality is that manufacturing has actually become more and more like software development. For example, Brooks argued that software development was always going to be more complex than manufacturing because in software development we never have to repeat ourselves:

Software entities are more complex for their size than perhaps any other human construct because no two parts are alike (at least above the statement level). If they are, we make the two similar parts into a subroutine–open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound.

Brooks, 1986

For decades the global economy has abounded with races like the Red Queen’s, with companies having to innovate faster and faster just to hold there position. Anything that can be repeated will be repeated by the competition. To stay ahead companies are forced to embrace ever increasing levels of complexity. Building products just isn’t enough, they must build capabilities that cannot be easily replicated. Just as programmers move anything repeatable into a subroutine or method, organisations now off-shore or out-source everything but their core competencies.

As manufacturing and software development become more alike, the approaches also begin to merge. Lean and Agile and merging, as Dan Woods observes:

What surprises me is that most advanced practitioners of Agile now use more vocabulary from Lean manufacturing than from Agile. In essence, as a practical matter, good ideas from Agile are being absorbed into a new approach to software development that is more Lean than anything else. Someone else can name this phenomenon, but Lean and Agile are merging.

Dan Woods at Forbes.com

Lean and Agile are merging. This is a two way process: not only is Agile benefiting from Lean’s inspiration, but Lean also has a lot to learn from Agile. Software Developers have been grappling with these levels of complexity for decades and the Agile methods they have developed can be usefully applied in other areas. Scrum, for example, has been applied outside the iT world. The DSDM consortium has taken this even further with DSDM Atern being a generally applicable Agile method adapted for use beyond software:

Atern is the culmination of practitioners’ experiences drawn from a wide range of public and private sector projects over more that a decade. This unparalleled expertise has resulted in a robust, but agile framework, which can be successfully applied across a variety or organisation and project contexts.

http://www.dsdm.org/atern/

The essential nature of software demands constant innovation and practitioners have been developed strategies, methods and process that allow this innovation in a way that is controllable and predictable. The global market now demands constant innovation from every organisation, and those organisations are able to look to the experience of software developers and the Agile methods they have developed for inspiration.


A Calculus for Software

Uncle Bob is looking for the Software Calculus, the new level of abstraction that will change the way we think about software.

Uncle Bob introduces the need for Calculus with Zeno’s Paradox regarding Achilles and the tortoise:

While it was intuitively clear that Achilles would pass the Tortoise quickly, the algebra and logic of the day seemed to suggest that the Tortoise would win every race given a head start. Every time Achilles got to where the tortoise was, the tortoise would have moved on.

The paradox highlights a limitation in algebra, and inability to deal with infinity.  Calculus provided a way of dealing with infinity that overcame this limitation and provided a means for solving a whole new set of problems.

There is an equivalent paradox in software development, and the calculus for software development must be able to resolve it.

The best code you will ever write is… code you never write.

He is right, we cannot keep piling complexity upon complexity, and sometimes that means more than simply writing better code, it means writing no code at all.  This means that we cannot look to programming languages or architectures for the next step because they exist to serve the purpose of writing code or building systems.  Sometime we want to do neither, sometimes it is better to do nothing.  Another discovery in mathematics even more fundamental than calculus was the discovery of zero, a way to represent nothing.  In software development we need a method that enables us to produce nothing.

This means that Model Driven Architecture, 4GL Database Languages and Quantum Computing cannot be the answer.  These are all advances that seek to provide a new way to write code, or have it generated or executed.  They are still, fundamentally, about producing code.  Calculus did not provide  a new way of doing algebra, if didn’t even make possible things that had been impossible using algebra.  While Issac Newton used his own idiosyncratic form of calculus to solve problems, he would translate the resulting ideas into a geometric form that his readers would more readily accept.  It provided a new way of thinking about problems.

The monster to by slain is complexity, but there is no silver bullet left that will allow us to put the beast to rest.  There are two types of complexity: essential and accidental.  Essential complexity is unavoidable, as Brooks explains:

The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence. For three centuries, mathematics and the physical sciences made great strides by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.

If the complexity of software is all essential, then is there no hope for improvement?  Back in 1987, when the essay was written, Brooks concluded by looking forward to what was to become Agile:

Therefore, one of the most promising of the current technological efforts, and one that attacks the essence, not the accidents, of the software problem, is the development of approaches and tools for rapid prototyping of systems as prototyping is part of the iterative specification of requirements.

It was the development of rapid prototyping and iterative development, in RAD and DSDM, that laid the foundations for the Agile movement.  Principles like Test Driven Development and YAGNI help us to avoid writing code that did not need to be written.

The question now is what next?  What is the next method for attacking the essential software problem while ensuring that the accidental problems do not return?  Personally I look to the principle of Lean for inspiration, not for new ways to write code but for new ways to see the problems that the code is attempting to solve.


Achieving Flow

This is a file from the Wikimedia Commons.To achieve flow is to be happy. Some developers call it ‘the zone‘, others ‘hack mode,’ but all agree that it is the best place to be when writing code. It isn’t unique to Software Development. Getting things Done author David Allen borrows a simile from karate to describe it: “Mind Like Water“.

In his classic book ‘Flow – The psychology of Optimal ExperienceMihály Csíkszentmihályi describes the state of flow in great detail.

It is what the sailor holding a tight course feels when the wind whips through her hair, when the boat lunges through the waves like a colt – sails, hull, wind, and sea humming a harmony that vibrates in the sailor’s veins. It is what a painter feels when the colors on the canvas begin to set up a magnetic tension with each other., and a new thing, a living form, takes shape in front of the astonished creator.”

Flow

As a programmer I can relate to the painter’s experience. When I spend the day in my text editor writing code: moving from green bar to green bar, filling files with text and re-factoring complex structures into something clean and simple. The time flows by and I’m shocked to discover when I’ve run out of day.

Flow can be achieved by teams working together as well as individuals.  Surgeon Atul Gwande recounts an experience of a team achieving flow as they raced to save a patients life.  The clock was ticking and the patient’s chances of survival were reduced with every passing the second.  The assembled team had never worked together before and the patient’s life depended upon them coming together as a team.

They were swift, methodical, and in sync.  The case was far from easy, but for whatever reason, it seemed like nothing could thwart us….  Steve, thinking ahead, asked Jay to grab a retractor we needed.  Joaquim nudged me to make the abdominal incision bigger, and stayed with me at every step….  Because we worked as a single unit, not as separate technicians, the man survived….  The operation had been symphonic, a thing of orchestral beauty….  From the moment we six were all dropped together into this particular case, things clicked.  It had been almost criminally enjoyable.

The Checklist Manifesto

It may be possible for small teams of highly trained medical professionals to achieve flow in the well resourced operating theatre, but what about more realistic jobs. Can the factory worker achieve flow?   A recent episode of This American Life shows that they can.

The experience of General Motors and Toyota at the NUMMI plant in Freemont, California suggests that they can.  Episode 403 of This American Life tells the story.  General Motor’s  workers enjoyed a level of pay and benefits they could not achieve working anywhere else.    The job for life it provided was a gilded cage because the work itself gave no satisfaction.

The way the plant was organised made it hard for them to achieve flow, so they sought happiness elsewhere, even in drinking, gambling, prostitution and drugs.  Management just wanted them to do there job, and yet they could fail to do even that. Absenteeism was high and one worker deliberately sabotaged the cars.  With the power of the union they thought there jobs were safe no matter how badly they behaved.  They were wrong.  The plant was closed in 1982.

In 1984 the plant was reopened as part of a joint venture between General Motors and Toyota.  Most of the same workforce were sent to Japan to learn the Toyota Production System.  The Toyota Production System introduces flow to the production line based on it’s two fundamental principles: Continuous Improvement and Respect for People.  Following these principles allows not only supplies and materials to flow, it also allows the people on the production line to achieve a state of flow where they are challenged but not overburdened.

The workers were proud of the Chevy Nova they were building. One worker would leave postcards with his name and address under the wiper of Nova’s he saw around, asking the owners to drop him a line. Another worker would go to the Chevy dealership to stare at the Nova’s just sitting there on the lot.

It may be nice that the employees were happy in there work, but surely car factories exist to generate profit by making cars, not keep the workforce happy.  What were the benefits for General Motors?  Plenty: within three months the same plant with the same workers achieved near perfect quality at a much reduced cost. It was estimated that it would have taken 50% more workers under the old system to produce the same car.

Nummi may have closed and Toyata may be experiencing problems right now, but in Nummi they were able to prove that a large, industrialised workforce can achieve flow.  General Motors also discovered the huge commercial benefits of allowing the workforce to achieve flow.

NUMMI plant in Fremont with Mission Peak behind it. (Joint venture between General Motors and Toyota.

I work in Software Development and I wan’t to achieve flow. I want the end users of the applications that I help to create to achieve flow.  I want the business analysts and the testers and the help desk and the project managers to all achieve flow. In ten years time I want the maintenance programmers extending the code base to be achieving flow as they implement their change requests.

Is this an unrealistic goal?   I hope not.   Is there some amazing secret that has to be uncovered before it is possible? No, because none of these principles are new. They really just seem to be common sense once they are being followed.  The unfathomable thing is why everybody isn’t doing this already.  The second half of the This American Life episode provies a clue when it talks about General Motor’s failure to replicate what had happened at Nummi.   They would take pictures of every inch of the Nummi plant, and then ensure that another plant looked exactly the same. In every aspect they did things exactly the same way, and yet they failed to replicate the success. Why didn’t it work?

Csikszentmihalyi  explains the problem:

It would be erroneous to expect that if all jobs were constructed like games, everyone would enjoy them. Even the most favorable external conditions do not guarantee that a person will be in flow. Because optimal experience depends on a subjective evaluation of what the possibilities for action are, and of one’s own capabilities, it happens quite often than an individual will be discontented even with a potentially great job.

Flow

If Flow comes from within and the external conditions needed to achieve it are unique to each individual then copying the external environment will not copy the flow state. In each place the worker must be able to find what they need to achieve flow.  However, the Toyota Production System and the experience of Atul Gawande seem to contradict this idea. At Toyota work is standardised, with every activity scripted to the finest point of detail. Gawande found the use of a standard checklist helped even the finest surgical teams to find flow and avoid mistakes.

In this blog I will make my own personal attempt to understand these contradictions, evaluating the conflicting claims and attempting to untangle the apparent paradoxes. My goal is to produce the best software I possibly can, and I believe learning how to achieve flow in software development is the best way to achieve that goal.


Artistic Principles and Apple’s Success

Back in 1997 every body was talking about Apple.  Steve Jobs had just returned and many doubted his ability to turn the failing company around.

Michael Dell was outspoken in his opinion:

“What would I do? I’d shut it down and give the money back to the shareholders.”

At the end of this week the iPad is set for release here in the UK and Apple have overtaken Microsoft to become the biggest software technology company in the world.  They passed Dell years ago.

How did Job’s turn Apple around?  What did he do to achieve such great success?  During the launch of the iPad he suggested that it was by bringing together technology and the liberal arts.

Apple has been guided by artistic principles.  While strategy and tactics may change over time, principles do not.  They provide the constancy of purpose needed for long term success built on continuous improvement.

The artistic principles that guided the restoration of Apple are exactly the same as when the original Mac was being developed back in 1983, as Steven Levy explains:

“Macintosh does indeed have a distinctive demeanor, but this is a result of human effort and creativity – just as the traits of a character in a novel or film stem from the imagination of its author. It is essential to recognize that Macintosh’s creators viewed themselves as artists. Those who conceive of that term in the traditional manner – painters in smocks, poets in garrets, auteurs in film school – have to stretch a bit to snare this concept. The Mac creators are emblematic of a new kind of artist spawned by the protean nature of the computer.”

Over time Apple’s circumstances have changed dramatically.  They have been a rapidly growing start up and on the brink of collapse.  Now they are a mature company defining new markets.  During all of these periods they have been able to move forward and succeed guided by the same, consistent, unfaltering  artistic principles.

Are these artists a breed apart from the rest of us?  Can only the lucky few enjoy the luxury of artistic principles?  Can we only look on in envy at those who pursue their art and create something special?

I don’t believe so.  Personally I agree with Seth Godin when he says that anybody can be an artist:

“Art isn’t only a painting. Art is anything that’s creative, passionate, and personal. And great art resonates with the viewer, not only with the creator.

What makes someone an artist? I don’t think it has anything to do with a paintbrush. There are painters who follow the numbers, or paint billboards, or work in a small village in China, painting reproductions. These folks, while swell people, aren’t artists. On the other hand, Charlie Chaplin was an artist, beyond a doubt. So is Jonathan Ive, who designed the iPod. You can be an artist who works with oil paint or marble, sure. But there are artists who worked with numbers, business models, and customer conversations. Art is about intent and communication, not substances.”

I work with code and business models.  When I pursue my art I get the best results.  It isn’t alway easy, circumstance don’t always allow for great success, but as long as my intentions are pure and my communication is clear I can achieve something substantial.