Which Number Moved?

A team consistently hits 90% test coverage. Six months later, the number of defects reaching users hasn't changed.

Automated testing is a Tool. Coverage measures how much of that Tool is in place. Functional is the Attribute it's supposed to move: the software does what it's supposed to do, and bugs don't get in the way of your users' workflows. If your organization talks about Quality, as in Quality Assurance, Functional is the Attribute behind it.

The failure mode I see most often: organizations measure their Tools and treat those numbers as a stand-in for the Attributes they care about. Coverage stands in for Functional. Velocity stands in for Timely. Then they optimize the number.

Numbers are easy to optimize. Tests get written to touch lines of code, and they assert very little. Story points inflate until velocity climbs. Every chart goes up and to the right, and the software is no more Functional or Timely than it was before.

Tool measurements have a use. They tell you whether a Tool is in place and being used. They can't tell you whether it worked. Only the Attribute can tell you that.

So measure both, and keep them separate. Measure the Tool to confirm the change happened. Measure the Attribute to see whether it mattered. When the Tool number climbs and the Attribute stays flat, you've learned something useful: that Tool isn't moving that Attribute. Pick a different one.

I created the House of Software Value Delivery to keep Attributes and Tools visibly separate, so a Tool's number can't stand in for the outcome.

Previous
Previous

Most Transformations Are Just Changes

Next
Next

Every Tool Has a Job