Tag Archives: software estimation

Granular Software Estimation is a Myth

Leave a reply

Software estimation has got to be one of the best examples of humanity as a whole not learning from its mistakes. Time and time again we attempt to estimate how long our software is going to take and it’s always always wrong; And usually it’s wrong by a lot. But we keep doing it.

Why Estimate at All? Can you?

icebergWhen time and money are finite it’s important to weigh alternatives. Usually the way this happens is we try weigh software X vs software Y in terms of what will have the highest impact per time input. But complexity determines time input. And you can’t understand the complexities of software and the variables involved because they do not become clear until you are in the middle of developing the software and uncover that complexity. At the start of a project our level of understanding about the software is the lowest it will ever be, which makes it the worst possible time to estimate.

So what do you do? How do we estimate then?

Software estimation is a lot like estimating nature:

  • You can’t accurately estimate whether a volcano is going to explode unless you are estimating within a matter of weeks.
  • You can’t accurately estimate the weather unless the estimate is within a matter of weeks.
  • You can’t accurately estimate where a hurricane is going to landfall unless you are estimating within a week.

What’s the common theme? Complexity. Nature is a very complex thing and as such the only way to estimate is to reduce complexity. The only way to reduce complexity is to reduce the amount of time that complexity can take place. Software is like that. Software is complex. If anyone tells you that anything in software development is simple they either haven’t been bit by this or don’t understand the complexity involved.

The only way to estimate accurately is by doing one of the following:

  1. Provide very low confidence very coarse-grained estimates.
  2. Provide estimates for things with low complexity.
  3. Provide estimates for things that are going to happen in the next week or two.

Items 2 and 3 are really just the same thing, stated in different terms. If you attempt to granularly estimate anything farther than a couple weeks out you are lying to yourself and everyone involved in your project. Item 1 is only valuable when it helps you make a significant decision about finite resources like time and money.

It’s been my experience that item 1 estimates are too tempting and can be abused, leaving item 2 and 3 as more attractive options. Don’t plan and estimate big software. Plan and estimate small incremental software.

Just Throw More Bodies at the Problem

2 Replies

Fred Brooks authored a book in 1975 called ‘The Mythical Man Month‘. In it, he argued that adding manpower to a late software project makes it later. The core of this belief has been captured in Brooks’s Law:

It takes some time for the people added to a project to become productive. This is known as ‘ramp up’ time. Software projects are complex engineering endeavors, and new workers on the project must first become educated about the work that has preceded them; this education requires diverting resources already working on the project, temporarily diminishing their productivity while the new workers are not yet contributing meaningfully.

Communication overheads increase as the number of people increases. The number of different communication channels increases along with the square of the number of people; doubling the number of people results in four times as many different conversations. Everyone working on the same task needs to keep in sync, so as more people are added they spend more time trying to find out what everyone else is doing.
(Source)

Why do these death marches happen in the first place? These are some common reasons:

  1. Unrealistic expectations or timelines.
  2. Poor communication between developers and stakeholders.
  3. Overly optomistic feature scope.
  4. Lack of understanding about the problem being solved for.
  5. Inappropriate resources for the job.

The effects of this on the developers can lead to burnout, and a higher rate of bugs introduced into the system. Particularly challenging for the developers are when new ones are added to the project, because they’ve got to split their time between standard development and helping newcomers ramp-up.

There are some tactics that can be implemented to mitigate the impacts, but preventing the problem from being introduced in the first place is a management challenge. Some tactics that can be taken:

Iterative Development

Developing features iteratively and hitting smaller milestones works better than trying to hit a deadline far in the distance. The cone of uncertainty grows the farther out the deadline is. Shorter bodies of work are easier to estimate, and demonstrate progress towards a goal. In some places, this model of not necessarily having a hard deadline, and just developing iteratively/continuously can be a hard culture change to make.

Separation of Concerns

Separation of concerns is important to developing maintainable loosely-coupled software, but it’s also a valuable management technique. Having small teams with a focused ownership area can reduce the ramp up time and reduce the overhead involved with many developers. It’s important to remember however that the segmentation should not impeded cross communication between teams. The segmentation is only intended as an abstraction tool to limit the amount of information any one developer needs to hold on to. All developers should feel free to deal with any area of the code base.

Don’t Rely on Hero Programming

Sometimes an organization will find itself with a few developers who carry most of the weight, and become almost a sort of single-point-of-failure. Fostering this behavior will result in a dangerous situation in which you depend on a small number of individuals, as well as cause chafing between them and the ordinary body of developers. It’s not as important to have superstar programmers as it is a large pool of developers who can work together efficiently.

Summary

Even though this has been proven time and time again, death marches are still common in software development. They often involve a ‘home stretch’ of unsustainable work, in which poor management practices lead to the belief that throwing more bodies at the problem will allow the software to be shipped sooner. Obviously, if we assume Brooks Law is valid, this is false.