Smashing the Myth of Predictable Software Development Part 2: Living with Unpredictability

 

In a previous post I argue that software development is inherently unpredictable.  I believe the best metaphor for software is a growing human relationship from first meeting to marriage.  Relationships like software grow along a predictable path–but at a random, unpredictable rate.  Thus they perfectly illustrate the nature of a software “project” with too many unknowns to allow for a predictable schedule.

Effectively creating software requires a process that can flex with this unpredictability and still produce useful results.

Fundamentally there are 3 levers that control software development.

1)       Risk

2)       Scope

3)       Schedule

 Risk

To have any hope of consistently delivering software developers must have control over risk.  This includes things like: leaving room in the budget for beefier hardware, developing and testing key architectural components before investing too heavily in the rest of the code, identifying scalability bottlenecks and testing them with code fragments that simulate full system activity, etc.   

Unfortunately, risk is consistently underestimated and often costs are cut in this area with catastrophic results.  In particular the concept of risk can be hard to get across to non-technical users.  Many people outside of IT misinterpret open and honest discussions of risk as a lack of confidence.

Personally, I’ve found the best way to approach risk with non-technical people is to (over) explain the reason for discussing risk in detail—namely that is how we avoid problems to produce “SUCCESS”!  We programmers (and other IT people) take it for granted that risk management is yellow brick road that leads to Oz, but non-technical users do not have knowledge hard wired in.  As the technical expert we programmers are often expected to double as teachers for those without the benefit of our technical training.  For those still in school—analyze your favorite teachers and take some notes of their methods.  Teaching skills can be a real asset to a programming career.

Scope

Control over scope is rare.  There is the infamous “scope creep” that runs havoc, but even more insidious is hidden scope.  I can’t count how many times I’ve had a request that seemed fairly straight-forward.  Then somewhere between two days and two weeks later I learn some funky little quirk in the business process that totally trips up the design.  The requirements remain unchanged while the project balloons to 4 times its original size.   

A few software houses have managed to control scope.  Intuit makers of popular financial software releases new products on a yearly cycle.  There are some hard-fast scope requirements—tax law for example—but most of their scope is flexible.  Features that aren’t stable in time for the 2008 version end up in the 2009 edition.

Starting with Windows 95 Microsoft seemed to be flirting with that model, but their markets (or maybe the company itself) seem to prefer well defined improvements in scope over precise release schedules so they never have been able to deliver consistently timed releases.  Since 2000 they seem to be steadily drifting away from dated releases.  (Ironically this excluding IT oriented products–SQL Server and Visual Studio—we developers like our naming schemes simple and mathematical.  Just the word “Vista” is enough to make me physically ill. J)

Schedule

While scope is somewhat flexible it is typically not flexible enough to account for all the unexpected roadblocks.  The only other alterative is to vary the schedule enough to allow for the work to be completed.  Management (particularly non-IT management) finds this unsettling.  Many of the software horror stories circulating today have their origin in management that is unwilling to sacrifice either scope or schedule and end up sacrificing staff instead. 

In order to work around the scope/schedule problem some have resorted to saying—“We can meet the schedule, but must cut quality.”  Personally, I see this as a cop out.  Quality is an attribute of scope.  A buggy or hard to use solution is an unfinished solution.  Phrasing scope in terms of “quality” might be easier for non-technical people to understand.  However, it gives the idea that quality is optional.  The results are rarely pretty.

Personally, my approach to schedule management is to keep a close relationship with my user/customer community updating them regularly on the status and challenges of the current project.  Initially this approach often results in a burst of initial scope when people realize that someone in IT is finally listening!  However, it’s not long before a balance emerges.  I’ve found that (most) customers will voluntarily cut/modify scope in order to achieve their top priorities within a reasonable schedule.

However, a note of warning is due here.  A key part of a good relationship with a customer is to emphasize the infrastructural investments that must be made.  The customer must “own” not only their features, but the entire system that supports those features.  Otherwise they won’t value/appreciate the huge investment that must be made.  Software is a lot like an ice-berg.  That 20% that is visible—the UI—is supported by 80% that is under the water.

Summary

Far too often programmers are left without control of scope or schedule and are forced to absorb the randomness of software development by working extra hours.  If the software industry recognized the true nature of software we’d abandon our project-centric approach and turn to a more dynamic model.  (I see signs of this in the current wave of web 2.0 companies)  One of evolving systems (or sets of interacting systems) developed over the long-term with an eye towards process and integration.

Additional Reading:In this article Joel Spolsky comments that the construction phase of software can be estimated with decent accuracy, but the design and bug fixing phases of software are unpredictable.  An interesting idea with a number of practical applications.