Categories of metrics

In my previous post I discussed the challenges related to quality of predictions and inferences you can make from data you collect in a corporate repository.  Fortunately, the data you collect are not all required for making inferences. Some of them are extremely useful no matter how much data you have. It all depends on what you intend to do with it.

The following is a list of categories of metrics that can be used to shed light on your organization and its processes. Metrics can be related to:

- Application performance

- Process and controls

- Security

- Planning activities

- Artifact reuse

- Project management

Those are all the ones I could think of for now. Surely there are plenty more, like operational information such as human resource metrics, for example, but they are less important for the topic of this blog, which is focussed on the evolution of artifacts in a corporate repository. There is some amount of overlap between these categories. For instance, project planning involves some of the same metrics that project management uses. In fact, they had better be using some of the same metrics or you couldn’t come up with a usable plan, could you? A plan must lay out expectations for as many aspects of the coming project as possible, in language that makes sense for those involved in its execution – and this includes metrics.

Performance-related metrics are used to indicate whether the system is nearing its capacity in some way. Obvious candidates are IT-related metrics such as ethernet collision rates and phone capacity utilization, but those are operational measures that you can investigate in a different blog. The kinds of performance metrics I’m talking about indicate whether information processing applications are running efficiently from the user’s perspective. What is the largest fan-out for a relationship field compared to the stated upper limit? Are people being blocked from doing their jobs because of excessive process restrictions? How long do common operations take? How long does it take people to find the information they need?

Process-related metrics help you determine whether people are getting their jobs done. Changes in the numbers may suggest that process improvements are needed, or that process improvements instituted earlier are working. For example, how thorough is the coverage of test cases in each module? How many feature requests have not been allocated by product managers into their appropriate categories? What is the variance between task estimates and actual durations?

Security-related metrics are a challenge because they typically look for something that is not there, and when something does show up it is frequently benign. For example, you might track how many sensitive operations are done on a per-user basis over the course of a week or a month. (Useful for catching rogue traders in financial institutions). Or, whether there are anomalous phone bills for certain users. Another useful metric might indicate which users are logged on late at night.

Planning-related metrics might track how many outstanding features are waiting to be implemented in the next versions of your product. They may track forward-looking capacity utilization for employees or for critical pieces of equipment.

Reuse metrics help track the efficacy of reuse initiatives. What percentage of artifacts (source code, requirements, test cases, etc) are being used in multiple scenarios or components? How long does it take for an analyst to locate an artifact, judge whether it fits the requirements, and fold it into his or her current project?

Project-related metrics help determine how close a project (or a sub-project) is to completion. Given the track record for project success rates, these tend to be problematic, and no wonder – much of the work of modern information-intensive product development is the discovery of knowledge. Knowledge discovery, also known as research and development, can be very difficult to put on to a schedule. In addition, as I pointed out in my previous blog, the statistics are poor given the number of dimensions in the problem. This all adds up to a challenging environment for project managers. Nonetheless, you have to at least try, and here are some pretty good candidates. The defect post rate must inevitably drop off as all the previously undiscovered defects are steadily discovered and posted. The total number of open defects will also trend to zero as the project nears completion. The percentage of completed sub-projects (or tasks) will also approach 100. For most reasonably sized projects these numbers will bounce around quite a bit, so at some point you may have to simply state that the project is nearing completion and start taking decisive measures to push it to the end. For example, you may want to start pushing certain defects off the pile that are less critical and that require too much time to fix, or incur too much risk.

There are lots of metrics you can track that can suggest a course of action when they start trending away from what you expect. You may not be able to predict a project’s completion date with the kind of accuracy you would like, but you can certainly notice when things are going well or, indeed, when they are not. I will try to dig up some particularly useful metrics in each category in future posts.