Is the Project Manager Responsible for Providing Resources at all?
2022-02-18
Employee Competency 26262
Employee competence according to ISO 26262
2022-05-18

Process Metrics – Making Quality Measurable

Process Metrics

Author: Gerald Harris

The Importance of Metrics

Automotive SPICE, along with all other conceptual frameworks for describing development processes, has the goal of improving quality, both in the final work products and in the way that these work products are created. Unfortunately, the concept of “improved quality” is a very slippery thing. We all know intuitively what is intended, but most people find it extremely difficult to provide a useful definition. One of the most important and at the same time most difficult aspects is measurability. Making quality measurable is challenging because it is, by its very nature, a qualitative and not a quantitative concept.

A need for process metrics arises automatically when an organization chooses to comply with a formal process standard such as Automotive SPICE. Metrics collected during the development process have the purpose of describing important aspects of the development process, how well it is being applied (compliance) and how successful it is (effectiveness). Well-chosen metrics will show where process compliance and effectiveness can be improved. Without good process metrics process success is a lottery.

Choosing Appropriate Metrics

Metrics should always be chosen with a goal in mind. There should be some specific phenomenon or behaviour which is of interest and which can be usefully described with the chosen metrics.

One possible simple example is provided by the requirement in Automotive SPICE for bi-directional traceability. A simple and easy to understand metric would be the percentage of the original customer requirements which are fully traceable through the entire development path. This number alone is unfortunately not enough to identify where problems exist or where more effort needs to be invested. If this metric can be extended to show where the chain of bi-directional traceability most often breaks down effective improvements of process compliance are vastly simplified.

The key steps in choosing process metrics are to:

  1. Identify and describe the phenomenon and/or behavior which is of interest
  2. Identify some aspect which is both measurable and which provides a meaningful description of the phenomenon of interest
  3. Identify how the measurement is to be performed and how the data must be analyzed
  4. Determine how the results can best be stored for future reference and presented to maximize usefulness.

Also remember that the metrics themselves are only one half of the full equation. The other half is how they inform the process improvement feedback loop.

Collecting and Analyzing Metrics

Automation, automation, automation!!!

Engineering hours are one of the most valuable resources in any company. Collection and analysis of metrics, when done by hand, is both extremely wasteful and incredibly expensive. This cost is not reduced by distributing it in tiny increments across a large number of developers. A simple calculation shows small daily time expenses quickly add up to man-years of lost engineering time across a large organization. Ideally metrics should be derivable from the artifacts that are a natural result of the development process and should require no additional effort or activity from the engineers. To derive the maximum benefit, the collection and basic analysis of metrics should be fully automated.

A Simple Example

Effective presentation of metrics can be a fine art of its own. Metrics must be presented in a fashion which makes both context and significance clear, the naked numbers are generally meaningless. Ideally, metrics should be presented so as to clearly identify the currently targeted goal, trends, progress and next steps. A strategy for making effective use of metrics for process improvement is an essential part of the overall development process.

A simple example to show correct usage of metrics is in order. In this example we will make the following assumptions.

  • Two key metrics are computed from the raw metrics that are collected resulting in a two-tuple, (metric-1, metric2). Both key metrics are normalized to lie within the range of [0 – 1].
  • Each software component that contributes to the full product is measured individually. Instead of having a single two-tuple for the entire project a cloud of two-tuples are generated with each point in the cloud corresponding to a single component of the product.
  • The first metric will reflect process compliance. To keep this example concrete, it is assumed that the bi-directional traceability of requirements through the development process to tests has been chosen as a sufficient indication of compliance. At each stage along the V-model the bi-directional traceability between incoming requirements and outgoing work-products is expressed as a percentage. The final scoring for the metric is a simple weighted average of all measured compliance values for this component across the full V-model.
  • The second metric will reflect code quality. We will assume that (some selected subset of) the HIS Source Code Metrics is being used. These are considered to be outdated but are well thought out and still serve as an extremely valuable reference. A description can be found on the Wikipedia page for software metrics. The scoring generated from each metric in the selected HIS subset will be normalized to the range [0-1] and the individual values will then be combined into a single score using a simple weighted average.

When the metrics are presented, the result will be a cloud of points that can be easily shown in a two-dimensional graph.

Process KPI

Components in the bottom-left quadrant are lacking in both compliance and quality. Components in the bottom-right quadrant are lacking in quality but are adequately compliant. Components in the top-left quadrant are lacking in compliance but are of good quality. Components in the top-right quadrant meet or exceed both quality and compliance goals.

The technical leadership team can choose the values for the compliance and quality thresholds. If desired, two different values can be chosen, one reflecting the long-term organizational goals and the other representing a short-term improvement goal. The importance of short-term improvement goals cannot be overstated as a mechanism for making progress visible and avoiding developer frustration. These short-term goals can be reviewed and revised in regular (e.g. 3-month) intervals.

This presentation also provides immediate mechanisms for detailed enforcement.

  • No change in existing code is allowed to worsen the compliance or quality metrics for the component.
  • All new code which is introduced must meet or exceed the targeted threshold values.

This presentation can be shown to management with confidence because it is both easy to explain and easy to understand. It immediately shows where more work must be invested to meet compliance and/or quality standards. Furthermore, it allows the leadership team to make an informed tradeoff when investing engineering effort. Is quality the more important focus or is compliance more urgent? Where is the optimal balance?

With just a little bit of extra work and some imagination, this can easily be extended to make trends visible. One option is to add color coding to show quality-compliance improvement or degradation.

Benefits of Metrics

If the progress towards a chosen goal is not tracked, it is impossible to know when the goal has been reached. On the other hand, when the metrics themselves become the goal you are no longer measuring anything meaningful.

It is well known that errors can be corrected with the smallest effort and expense when they are detected as early as possible. Similarly, when a goal is being pursued within an organization, metrics can and should be used to detect progress towards that goal.

When attempting course corrections well-chosen metrics will provide quick feedback about the value and effectiveness of the chosen course correction.

Even a mediocre metric can be better than flying blind.

Pitfalls of Metrics

In a presentation, hard numbers can be very convincing. Unfortunately, poorly chosen metrics having inadequate descriptive power can be just as convincing as good metrics. The quote from Mark Twain’s autobiography applies here: “There are three kinds of lies: lies, damned lies and statistics.

The following questions point out some of the most important pitfalls when choosing metrics.

  • Is the chosen goal clearly defined and independent of any metrics that are collected?
  • Is the chosen metric relevant to the targeted goal?
  • Is the raw data from which the metric is derived free from bias, systematic or otherwise?
  • Does the Heisenberg Principle apply – does the data collection process impact the phenomenon being measured?
  • Is the data analysis and presentation of the metric understandable and auditable?
  • Are the conclusions drawn from the metric transparent and defensible?

Conclusions

As was stated at the beginning, process metrics are an essential tool both for the introduction of new processes and for the improvement of existing processes. Well chosen metrics are a multiplier when seeking improvements. Poorly chosen metrics are like sand in your eyes. The time and effort invested in metrics can have a huge payoff when done correctly.

Are you looking for expert advice?

Contact us!