Author: Gerald Harris
Automotive SPICE, along with all other conceptual frameworks for describing development processes, has the goal of improving quality, both in the final work products and in the way that these work products are created. Unfortunately, the concept of “improved quality” is a very slippery thing. We all know intuitively what is intended, but most people find it extremely difficult to provide a useful definition. One of the most important and at the same time most difficult aspects is measurability. Making quality measurable is challenging because it is, by its very nature, a qualitative and not a quantitative concept.
A need for process metrics arises automatically when an organization chooses to comply with a formal process standard such as Automotive SPICE. Metrics collected during the development process have the purpose of describing important aspects of the development process, how well it is being applied (compliance) and how successful it is (effectiveness). Well-chosen metrics will show where process compliance and effectiveness can be improved. Without good process metrics process success is a lottery.
Metrics should always be chosen with a goal in mind. There should be some specific phenomenon or behaviour which is of interest and which can be usefully described with the chosen metrics.
One possible simple example is provided by the requirement in Automotive SPICE for bi-directional traceability. A simple and easy to understand metric would be the percentage of the original customer requirements which are fully traceable through the entire development path. This number alone is unfortunately not enough to identify where problems exist or where more effort needs to be invested. If this metric can be extended to show where the chain of bi-directional traceability most often breaks down effective improvements of process compliance are vastly simplified.
The key steps in choosing process metrics are to:
Also remember that the metrics themselves are only one half of the full equation. The other half is how they inform the process improvement feedback loop.
Automation, automation, automation!!!
Engineering hours are one of the most valuable resources in any company. Collection and analysis of metrics, when done by hand, is both extremely wasteful and incredibly expensive. This cost is not reduced by distributing it in tiny increments across a large number of developers. A simple calculation shows small daily time expenses quickly add up to man-years of lost engineering time across a large organization. Ideally metrics should be derivable from the artifacts that are a natural result of the development process and should require no additional effort or activity from the engineers. To derive the maximum benefit, the collection and basic analysis of metrics should be fully automated.
Effective presentation of metrics can be a fine art of its own. Metrics must be presented in a fashion which makes both context and significance clear, the naked numbers are generally meaningless. Ideally, metrics should be presented so as to clearly identify the currently targeted goal, trends, progress and next steps. A strategy for making effective use of metrics for process improvement is an essential part of the overall development process.
A simple example to show correct usage of metrics is in order. In this example we will make the following assumptions.
When the metrics are presented, the result will be a cloud of points that can be easily shown in a two-dimensional graph.
Components in the bottom-left quadrant are lacking in both compliance and quality. Components in the bottom-right quadrant are lacking in quality but are adequately compliant. Components in the top-left quadrant are lacking in compliance but are of good quality. Components in the top-right quadrant meet or exceed both quality and compliance goals.
The technical leadership team can choose the values for the compliance and quality thresholds. If desired, two different values can be chosen, one reflecting the long-term organizational goals and the other representing a short-term improvement goal. The importance of short-term improvement goals cannot be overstated as a mechanism for making progress visible and avoiding developer frustration. These short-term goals can be reviewed and revised in regular (e.g. 3-month) intervals.
This presentation also provides immediate mechanisms for detailed enforcement.
This presentation can be shown to management with confidence because it is both easy to explain and easy to understand. It immediately shows where more work must be invested to meet compliance and/or quality standards. Furthermore, it allows the leadership team to make an informed tradeoff when investing engineering effort. Is quality the more important focus or is compliance more urgent? Where is the optimal balance?
With just a little bit of extra work and some imagination, this can easily be extended to make trends visible. One option is to add color coding to show quality-compliance improvement or degradation.
If the progress towards a chosen goal is not tracked, it is impossible to know when the goal has been reached. On the other hand, when the metrics themselves become the goal you are no longer measuring anything meaningful.
It is well known that errors can be corrected with the smallest effort and expense when they are detected as early as possible. Similarly, when a goal is being pursued within an organization, metrics can and should be used to detect progress towards that goal.
When attempting course corrections well-chosen metrics will provide quick feedback about the value and effectiveness of the chosen course correction.
Even a mediocre metric can be better than flying blind.
In a presentation, hard numbers can be very convincing. Unfortunately, poorly chosen metrics having inadequate descriptive power can be just as convincing as good metrics. The quote from Mark Twain’s autobiography applies here: “There are three kinds of lies: lies, damned lies and statistics.”
The following questions point out some of the most important pitfalls when choosing metrics.
As was stated at the beginning, process metrics are an essential tool both for the introduction of new processes and for the improvement of existing processes. Well chosen metrics are a multiplier when seeking improvements. Poorly chosen metrics are like sand in your eyes. The time and effort invested in metrics can have a huge payoff when done correctly.