Blogger: Kirk Knoernschild
Metrics can be a valuable asset to software development teams in helping to improve their efficiences and measure their effectiveness. Unfortunately, metrics don't always provide the intended value. Sometimes, metrics lie.
Metriculation is a term I use to describe how metrics are made to lie. It's derived from statisticulation, a term introduced by Darrell Huff in his book, "How to Lie with Statistics", first published way back in 1954. There are many forms of metriculation, and it's a significant factor that affects the successful use of software development metrics.
Metriculation is not the same as using metrics ineffectively, or measuring the wrong thing. For instance, using LOC (lines of code) was once thought an effective way to assess programmer productivity. LOC has nothing to do with productivity. Code generation wizards can be used to create a component with thousands of lines of code, but it might take days to correct a nasty bug that requires changing only a single LOC. LOC does have it's value, however. Using LOC to help calculate the percentage of code under test, or to help with software estimation, are examples. Using LOC to gauge developer productivity is ineffective, but it's not metriculation.
Instead, metriculation involves using metrics to mislead. Downright deceive. Lie! For instance, in an effort to undermine the value of unit testing, a developer may claim that 75% of the system defects are found in code with unit tests. This form of metriculation, known as semi-attachment, is done by finding a way to prove what you want to prove by drawing false conclusions. Based on only this information, one might be drawn to the false conclusion that unit testing does not contribute to higher quality software. Unfortunately, what the developer may not have shared is that 95% of the code is under test, meaning that the remaining 25% of defects reside in 5% of the code. Given this additional information, we could have arrived at an entirely different conclusion - that unit testing does contribute to higher code quality. In this example, the metriculation was intentional, but that's not always the case. Often times, it's due to oversight. Regardless, metriculation is dangerous.
In addition to semi-attachment, there are many other forms of metriculation, all with numerous real world examples of how they've been used to push an agenda. These include selective use, biased samples, well-chosen average, fudging the numbers, overgeneralization, faulty assumptions, whiz bang graphics, and creative charts. Upcoming Burton Group research will discuss in more detail metriculation, and how we can deal with. As that material matures, I'll continue to share interesting snippets.
Comments