Less Estimation, More Trust

22 Mar 2015. comments

A good deal of this article is based off of thoughts from Ernie Miller’s talk at Ruby on Ales 2015.

We’re in the business of producing software. If you produce software for products/services then you know that these businesses tend to value predictability, measureability, and control. The way most businesses try to obtain this predictability/measurability is by abstracting software developers to ‘resources’ and then using math derived from software estimation. This fails utterly, every time.

Software developers are not ‘resources’ in the same way that software managers are not ‘overhead’ and using those terms are offensive. Software developers and those who manage them are humans. And it turns out humans are messy. Software is messy too, because it’s abstract and built by humans.

Software != Construction

There is a long standing tradition in this field of comparing software development to construction, like constructing a building. This was a reasonable place to start when software development was a new field since humans operate best with analogy and comparison to things they already know. But we quickly found out that building software is nothing like building buildings. Buildings are static and we’ve built a lot of them over hundreds of years. Software, on the other hand, is an organic, creative, moving target. Software is malleable. There are very few bits of software that are identical because the people and businesses for whom the software is being built are rarely identical in nature.

The time you know the least about the software you are building is at the start.

Dan Milstein writes:

“Writing software involves figuring out something in such incredibly precise detail that you can tell a computer how to do it.”

“If you were to write a specification in such detail that it would capture [all] those issues, you’d be writing the software itself.”

Estimates are Lies


Given this organic nature of software, I believe long term estimates are lies. Giving these estimates means that you are saying:

  1. I understand everything about the problem.
  2. I know that my customer will absoloutely love the software I give them and will not want any changes.
  3. I know that the software constructed according to the plan will be the exact software we end up with at the end.
  4. I have built this exact thing before and therefor know the time it takes.

All of those are ludicrous and we shouldn’t kid ourselves by thinking otherwise. At best these estimates are useless; at worst they are malicious and destructive to the business who is planning based on them.


Did I mention that software also never lives on its own? Every piece of software has dependencies on other software. That other software is just as organic, malleable, and creative as your software. The effect then is multiplicative. The uncertainty isn’t even contained to just your software.

The issues can be wide ranging:

  • The service you are integrating with doesn’t actually meet your needs but this isn’t discovered until down the road.
  • The library you are building on top of has a big bug. That bug is going to be addressed, but you don’t control the timeline of that fix.
  • There are unforseen network issues between you and a service. They arose during development.
  • The service you depend on decides to close shop.
  • The service you depend on decides to start charging you money instead of being free.
  • The customer realizes something about the software that couldn’t be realized until part of the software was written and a demo was given, revealing the issue. New dependencies are needed.
  • Etc, etc, etc.

Be a Professional

So providing these estimates is lying. Estimates have a funny way of turning into deadlines. And because the organic nature of software development those deadlines quickly become out of touch and just as useless as the deadlines they were produced from. Deadlines emphasize the dates as being the important part and de-emphasize building the right software that will please the business and its customers.

So are all estimates hopeless? Maybe. Large software tasks have been proven to be uncertain/organic/malleable with uncertainty multipled by every dependency you integrate with. But what about small development tasks? Tasks that fit into less than 4 hours (a half day) are usually estimated reasonably well or at least well enough to where it doesn’t matter if they are off by a bit. Tasks of this size also tend to be much more ‘regular’ than large scale development. They also have the advantage of having a tight feedback loop that allows you to course-correct if they fell prey to the same issues outlined before with other estimation.

But even if you estimate in this way you’ve really just brushed the real problem under the rug by making the failures of estimation less costly. What if instead we ignore estimation in service of some finish line and instead just worry about the next feature we need to deliver? Why not focus on delivering the Next Most Important Thing continuously at a sustainable pace? Maybe you do it weekly; demo working software to your customer at the end of every week? Working at these time scales allow the organic nature of software to be dealt with. Customer changes can be incorporated, dependency issues can be dealt with, course corrections can be made.

If you think this is unrealistic then you might have stumbled across The Divide…

The Divide Between Business and Developers is a Lie.

A lot of the disconnect between business and software development stems from:

  • Software developers continuing to provide estimates.
  • Business continuing to pressure developers to provide those estimates.
  • Both believing that those estimates are meaningful.

It’s no wonder there is a divide between business and software development. Software developers regularly provide estimates and those estimates are very often off by an order of magnitude. This leads to a lack of trust. Why would you trust software developers when their estimates are so bad? We need to stop estimating at this scale. Stop committing to months-long deadlines. Start talking openly about the true nature of software development and commit to delivering value weekly.

The uncertainty about software development timelines needs to be put right out in the open. The factors that cause the inability to estimate with any kind of accuracy need to be put out in the open. These are issues that need to be on the table for the entire business to grapple with, not just software developers. Businesses need information to be successful and if that information is tied up in the Mysterious Land of Software Development then the business won’t know about the impending failure until it arrives and it’s too late. Developers who all-too-frequently crunch to meet these estimates/deadlines are doing a disservice to the business.

A true professional doesn’t change how he/she operates under pressure. Be honest about estimation and why it fails. Be honest about all the uncertainty factors that make up estimation woes. Work together to deliver valuable working software on a more frequent, less estimation focused basis.



Tagged: estimation agile software engineering software craftsmanship

There's a Bug In Your Blind Spot

25 Jan 2015. comments

Recently Robert ‘Uncle Bob’ Martin went to twitter to make a bold observation:

The initial gut reaction for some is “Yeah, we should just write less bugs!” which is of course what he’s implying here. But is it that simple?

When does a bug become a bug?


What’s implied when questioning the value of a bug tracking system is that if we would only just not create the bugs in the first place that we’d be able to dump the tracking software and have happier users. What that implies is that all these bugs are programmer slip-ups that could be prevented at the time of writing the software.

The above line of thinking ignores the reality of many bugs whose lives did not begin at code construction time but rather evolved into a bug based on humans discussing the software with each other. Was Feature X supposed to behave in that way? Was Feature Y supposed to only be available to users A and B? Do product and design agree on what Feature Z should do and is that how it was implemented?

There’s no way around it, you need to have human conversations.

Questions like these are not solvable by well-tested software or software engineering rigor. So long as it is possible for humans to misunderstand each other there will be bugs filed based on expected behavior not matching what is implemented by engineers. The only way to solve questions like these are by having conversations with your users, your team, and your business. These conversations must also happen often and continuously because many times a feature is not well understood until it’s actually prototyped or built, and then these ‘bugs’ are filed. The sooner you have these conversations the shorter the lifetime of these sorts of bugs; but they will have a lifetime.

Maybe we’re splitting hairs here about the definition of a ‘bug’ but that’s not really important when you’re trying to debate what constitues a ‘bug’ to your non-technical colleagues. The software isn’t doing what they expected and it’s your job to make the necessary adjustments.

Subtle arrogance

Making a bold claim like ‘we shouldn’t have a bug tracker’ is also a bit arrogant if you think about it. Let’s assume for a moment that you are the best engineer in the world and use the best software development practices. You practice TDD, you work closely with your customer, you release early-and-often, the whole nine yards.

I hate to break it to you but you’re not perfect. You’re still going to introduce bugs. Because you’re a human and humans make mistakes. Human’s misunderstand each other.

And if you agree that you’re not perfect and that it’s possible you won’t have 100% understanding with other humans, isn’t it responsible to track your mistakes? If for no other reason than for you and the reporter to not let the bugs slip through the cracks?

I understand the higher level sentiment

Uncle Bob is no dummy. He’s been around a lot longer than most of us in software. I fully expect that he’d agree with much of what I’ve written here because there’s more nuance than 140 characters on Twitter will allow for. We do write too many bugs and we could use more rigor, more testing, more professionalism. But it isn’t a zero-sum game and we can’t “win” the battle against bugs. Your job isn’t to write zero bugs but rather its to minimize the ones that are mistakes and manage the ones that stem from misunderstanding.


Tagged: bugs software engineering soft skills software craftsmanship

AMD, CommonJS, and ES6

11 Oct 2014. comments

As mentioned in previous posts I’ve been digging into javascript things more in recent days. I’ve been out of the javascript game for several years and the landscape has obviously changed quite a lot since then. Most of the dramatic change that I needed to brush up on was module specs. I knew there was a pile of vocabulary I ought to look into based upon seeing/hearing them in other developers work:

I apologize in advance if some of this is old-hat, but if you are a new like me then read on.

JavaScript Modules and Package Management Is a Mess

Something I started to discover along the way is that the current state of modules in JavaScript is an unholy disaster. There are competing module specs and an ever changing venn-diagram of libraries available/shimmed for each. The 2 main competing module specs are RequireJS and CommonJS, but ES6 is on the horizon which complicates things further because it doesn’t really correspond to either of those current leading specs.

The Old Way

The old way of making modules looks like this:

 1 // mymodule.js
 3 var mymodule = (function(dep1, dep2) {
 5   var mymodule = {};
 7   mymodule.foo = function() {
 8     return 'hello world';
 9   };
11   return mymodule;
13 })(dependency1, dependency2);
15 mymodule.foo();

The downside to this approach is that you must loaded the dependencies in order, usually via a plethora of script tags.

RequireJS and AMD

RequireJS is a javascript module loader for the browser. The main selling point here is that it can load modules on-demand and cache them for future calls. It uses the AMD module spec which looks like this:

1 //mymodule.js
3 define(["./dependency1", "./dependency2"], function(dependency1, dependency2) {
4   return {
5     foo: function() {
6       return 'hello world';
7     }
8   };
9 });

You then use such a module like this:

1 require(["mymodule"], function(mymodule) {
2   mymodule.foo()
3 });

Browserify and CommonJS

CommonJS is more or less the same module loading that you see in Node.js. Using Browserify you can take NPM modules and use them in the browser. This approach is far simpler than RequireJS but rather than using on-demand loading it instead parses the AST and concatenates everything into isolated modules that can be cached. CommonJS looks like this:

1 // mymodule.js
3 var dependency1 = require('dependency1');
4 var dependency2 = require('dependency1');
6 exports.foo = function() {
7   return 'hello world';
8 };

And its usage:

1 var mymodule = require('./mymodule.js');
2 mymodule.foo();


If the previous 2 module specs weren’t enough, ES6 Harmony is on the horizon and its spec is entirely different. Here’s what ES6 modules look like:

1 // mymodule.js
3 import dependency1 from 'dependency1';
4 import dependency2 from 'dependency2';
6 export default function foo() {
7   return 'hello world';
8 }

And its usage:

1 import foo from 'mymodule';
2 foo();

This is due to hit late-2014/early-2015. It will be interesting to see what becomes of RequireJS and CommonJS.


Tagged: javascript amd commonjs es6 harmony ecmascript

Angular and Ember

15 Sep 2014. comments

I’ve been diving deeper on some javascript MV* frameworks recently to try and stay sharp on the growing trends in web development. In the past I’ve used Backbone.js but I wanted to try out something new this time. I started doing my homework on Ember.js and Angular.js since those are the two large players in the space.


I first started looking at Ember.js because it seems to have a lot of traction on the ruby/rails conference circuit and because I’ve been following the development of Jeff Atwood’s latest project Discourse.

Convention Over Configuration

Ember’s take on things is a lot like Rails; It’s convention over configuration and super opinionated about the way you should do things. While its true I do a lot of Rails development these days I wouldn’t say I love Rails. My take is that Rails opinionated nature tends to go too far in a lot of cases and ends up dictating how your app is written; often times involving lots of global data/functions. The tradeoff there is of course that you can usually take a Rails Developer A and drop them into a Rails Project B and they will generally get along just fine. The story is no different with Ember which is unsurprising given the core team’s history from the Rails community.

ember I started building a prototype in Ember with Rails as a JSON backend. I followed the Getting Started guide on the official site and built Todo MVC. The documentation for Ember is fine but the guide tutorial doesn’t really look like an actual application that you’d end up building and so it required a lot of digging around to see how other people were building real apps.

Funny Taste

One of the first things that weirded me out a little bit was that a significant portion of Ember as a framework is that it essentially redefines the javascript language by introducing its own object model (yes you read that right). It tries to mask javascript’s prototypal inheritance by making it seem like you’re doing classical OO that many programmers are familiar with rather than embrace the language’s true nature:

1 App.Person = Ember.Object.extend({
2   say: function(thing) {
3     alert(thing);
4   }
5 });

Why do I need an entire object model when all I want is a framework for managing my client-side apps? Why not just use plain javascript? This seems like over-engineering.

Dictating Your REST Resources

The next major roadblock I ran into was a real show stopper for me, and quite frankly I was dumbfounded that it’s even still an issue: The Ember RESTful Resource adapter does not allow singular resources. For example if I had a resource in rails declared thusly:

1 resource :user, only: [:show]
2 # ends up as: GET /user

And lets say it returned JSON like this:

1 {
2   "id": 123,
3   "first_name": "Ben"
4 }

There is no way by default to consume such a route from Ember’s REST adapter. It’s opinion is that resources are always plural and always return arrays of data in a specific JSON structure, and camelized casing. There are some discussions about it on stackoverflow and open issues that appear to have been largely ignored, which is unfortunate:

So that combined with the general difficulty of figuring out how to do stuff with Ember made me table it for now and move on to something else.

Update: I’ve since picked it back up after finding some things about Angular distasteful. (Read on)


Honestly the last time I looked into Angular was back when it wasn’t really a thing yet and Backbone vs. Ember seemed to be the only real contenders. This is why I didn’t look at it first but after my experience with Ember I decided to do some research on usage.

Wow. I guess I didn’t realize how much it’s taken off because the numbers are pretty staggering at the time of this writing:

Angular Ember
GitHub stars 29k 11k
StackOverflow questions 54k 11k
StackOverflow questions this week 1200 150

So that was pretty encouraging. I’d also heard good things from the community:

ember My brother in law also uses it and has said good things about it.

So I started to dig in to Angular.

… And then in a matter of an hour or so I’d finished the tutorial and prototyped the same app I tried to develop with Ember in another hour.

Get Things Done

With Ember there was quite a steep learning curve that just wasn’t present with Angular. I was able to get stuff done right away and the architecture just Made Sense. When I read the documentation I found myself virtually skimming sections of it because it just seemed like the obvious/correct/boring way that it should be.

In fact it’s so simple that I can show you the bulk of it in a few simple code snippets without any explanation and I bet you can see what’s going on:


 1 <!doctype html>
 2 <html lang="en" ng-app="fooApp">
 3 <head>
 4   <meta charset="utf-8">
 5   <title>Foo App</title>
 6   <script src="angular.js"></script>
 7   <script src="angular-route.js"></script>
 8   <script src="angular-resource.js"></script>
 9   <script src="foo.js"></script>
10 </head>
11 <body>
12   <div ng-view class="view-frame"></div>
13   <script type="text/ng-template" id="user.html">
14     <p>Hello {{user.name}}!</p>
15   </script>
16 </body>
17 </html>


1 angular.module('foo', ['ngRoute']).
2   config(function($routeProvider) {
3     $routeProvider.  when('/user', { templateUrl: 'user.html', controller: 'user' });
4   }).
5   controller('user', function($scope, User) {
6     $scope.user = User.query();
7   });


Here’s a few reasons why Angular is good:

  • It’s mostly just plain old javascript and embraces the prototypal inheritance model rather than masking it with classical OO.
  • There are no models dictated by the framework. There’s not even a concept of it beyond being able to pass things from controller to view.


While Angular is very nice and intuitive there are some drawbacks:

1 <ul class="phones">
2   <li ng-repeat="phone in phones | filter:query">
3     {{phone.name}}
4   </li>
5 </ul>

Yep that’s logic in your templates. This is technically a violation of Separation of Concerns but it can make markup read a little cleaner. YMMV.


Tagged: ruby rails angular

Introducing broken_windows

01 Aug 2014. comments

I’ve been writing code in ruby professionally for about 8 months now but until now I hadn’t ever published a gem (well, sort of… I’ve published gems on our internal geminabox.)

So today I published broken_windows. It’s a gem that helps you find broken links in markup, either by pointing it at a url from the command line or by including it in your code. You can read more about the motivations and how to use it in the README.

How I Got Here

Broken Windows is a gem I originally created to both learn ruby idioms and how to create gems. Early on in the process I had some mentoring from Pete Higgins and Ryan Davis. I was transitioning from writing Java day-to-day to becoming a rubyist and the guidance they provided was great. Once the mentoring was over though I sort of left it in disrepair for most of the year. This week though I revisited it, made it more robust, and learned how to package up things a little better and get it published.

So go check it out and give me some feedback!


Tagged: ruby

2015 Ben Lakey

The words here do not reflect those of my employer.