For me, a title like that could mean either of two things. On the one hand, it could refer to the frustrations of living in Canada where the parliamentary system allowed the majority party to dictate to the country that we are all now citizens of the rest of the world and must learn the metric system, in spite of the fact that 80% of our foreign trade is with our neighbours to the south who use the fun but quirky imperial system of measurement. Forty years after its introduction rather than having to learn just the metric system, instead we have to learn both. We still buy our beef in pounds and measure ourselves in feet and pounds, though meat vendors take advantage of our confusion by advertising prices as “per 100gr” which looks cheaper than “per lb”. Once you’ve completed the purchase the wrapper still lists the “per lb” price because that’s the only measure we truly understand. And nobody I know would say I am 185 cm tall – they say I am 6 feet tall.
On the other hand, the term metrics frustration could refer to the fact that no matter how much you measure in your information repository, you rarely get hard and fast answers to important questions. With enough data, presented appropriately, and if you squint a bit, you might be able to tell some useful things about what is going on in your repository. Of course usually they aren’t the things you are actually looking for and they are almost never conclusive – hence the frustration. Being as this is a blog about information architecture, I will suppress my inner curmudgeon and talk about the latter, though I really can go on quite a bit about the metric system.
Metrics are numbers that help you understand your data at a higher level of abstraction. At a very high level they answer what and how questions more than why, who or when. For instance, “what is the rate of change in my requirements repository”, or “how accurate are our estimates?” A good way to think about developing and using metrics is the Goal Question Metric (GQM) approach [1]. The approach is useful because it starts with questions that are relevant to your organization. From the paper, “… measurement must be defined in a top-down fashion. It must be focused, based on goals and models. A bottom-up approach will not work because there are many observable characteristics in software … but which metrics one uses and how one interprets them is not clear without the appropriate models and goals to define the context.” The process of collecting, tracking and presenting metrics is expensive. It takes many iterations before you get comfortable with the quality of the data and your understanding of the data. In my opinion you can never be truly done because each improvement in your methodology moves the bar higher. The metrics that make sense when you are just starting off no longer suffice as you gain expertise.
The GQM method is simple: Establish organizational goals such as “increase the throughput of the call center while maintaining customer satisfaction.” Then, ask questions that break the goal down to its major components, such as “how many calls does a CSR take in an hour?” Finally, establish metrics that will help answer the question, such as average call duration, average wait between calls and standard deviations. After taking data for a while, you can look at the data and try to understand what’s going on. Often you can achieve this understanding only by looking at the actual raw data. For example, if there is an outlier (a data point well outside of the normal range) you pretty well have to look at that event or transaction to understand why it strayed so far from the rest. This is called “drill down” capability.
The problem with metrics shows up on the first page of the GQM paper, where they present examples of typical metrics including questions about productivity and defect density. Productivity in a knowledge-based industry is notoriously difficult to measure and is often just difficult to believe. Some researchers have measured differences in productivity that vary by up to two orders of magnitude depending on the employee. If this were really true then the good employees would be granted commensurately large salaries. Yes there is a wage scale, but top to bottom for a given technical position it is a factor of 3 or 4, not 10 or 20. This would be an unsustainable imbalance in a market economy if it were true. Defect density is another questionable measure because it relies on counting the number of lines of code, or the number of function points, or the number of API entry points – there are a few ways to measure the denominator. And that is precisely the point – because no one method seems to be “the answer”, people try all sorts of metrics to see which one works best.
Ultimately, that is the lesson of metrics in a knowledge-based industry: No one metric is going to tell you the answer you are looking for. Metrics that work in some circumstances will not work in others. Even from one company to the next, in the same industry, following the same methodologies, you are going to find some unexplainable differences. Even a single project team working on two consecutive projects will have different project scenarios and hence a different set of metrics. This is not a science. But we still have to measure what we can if only for one very important reason: It gets you thinking. When something comes up that seems odd, you had better try to explain it. Well OK, perhaps there are people who just accept things the way they are and don’t push any more, but an unexplained problem eats away at a good knowledge employee. You just have to know. And that’s the kind of delicious stubbornness that could one day save your company.
[1] Basili, Victor; Gianluigi Caldiera, H. Dieter Rombach (1994). “The Goal Question Metric Approach” (PDF). ftp://ftp.cs.umd.edu/pub/sel/papers/gqm.pdf.
[...] Metrics frustration [...]