The most important things cannot be measured. The issues that are most important, long term, cannot be measured in advance.
—W. Edwards Deming
Working software is the primary measure of progress.
—Agile Manifesto
Metrics
Metrics are agreed-upon measures used to evaluate how well the organization is progressing toward the portfolio, large solution, program, and team’s business and technical objectives.
Thanks to its work physics, Kanban systems, timeboxes, and fast feedback, Agile is inherently more measurable than its proxy-based predecessor, the waterfall process. Moreover, with Agile, the “system always runs.” So, the best measure comes directly from the objective evaluation of the working system. Continuous delivery and DevOps practices provide even more things to measure. All other measures—even the extensive set of Lean-Agile metrics outlined below—are secondary to the overriding goal of focusing on rapid delivery of high-quality solutions.
But metrics are indeed important in the enterprise context. SAFe provides metrics for each level of the Framework. The links below navigate to the entries on this page.
Portfolio Metrics
Lean Portfolio Metrics
The set of Lean portfolio metrics provided here is an example of a comprehensive but Lean group of measures that can be used to assess internal and external progress for an entire portfolio. In the spirit of ‘the simplest set of measures that can work,’ Figure 1 provides the leanest that a few Lean-Agile portfolios are using effectively to evaluate the overall performance of their transformations.
Enterprise Balanced Scorecard
The enterprise balanced scorecard provides four perspectives to measure performance for each portfolio—although the popularity of this approach has been declining over time in favor of Lean Portfolio Management (LPM), as shown in Figure 2. These measures are:
- Efficiency
- Value delivery
- Quality
- Agility
These results are then mapped into an executive dashboard, as illustrated in Figures 2 and 3.
For more on this approach, see chapter 22 of Scaling Software Agility: Best Practices for Large Enterprises [2].
Lean Portfolio Management Self-Assessment
The Lean Portfolio Management (LPM) continuously assesses and improves their methods. The LPM periodically conducts a self-assessment questionnaire to measure their performance, which automatically produces a radar chart like the one shown in Figure 4. It highlights the relative strengths and weaknesses.
Download Portfolio Self-Assessment
Large Solution Metrics
Solution Train Predictability Measure
The Agile Release Trains (ARTs) predictability measures are summarized to calculate the Solution Train’s predictability measure, as illustrated in Figure 5.
Solution Train Performance Metrics
The ARTs performance metrics are summarized to calculate the Solution Train’s performance metrics, as shown in Figure 6.
Program Metrics
Feature Progress Report
The feature progress report tracks the status of features and enablers during PI execution. It indicates which features are on track or behind at any point in time. The chart has two bars:
- Plan – Represents the total number of stories planned.
- Actual – Represents the number of stories completed. The bar is shaded red or green, depending on whether or not the item is on track.
Figure 7 gives an example of a feature progress report.
Program Predictability Measure
The team PI performance reports are summarized to determine the program predictability measure, as illustrated in Figure 8.
The report compares actual business value achieved to planned business value (see Figure 19).
For more on this approach, see chapter 15 of Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise [1].
Program Performance Metrics
The end of each PI is a natural and significant measuring point. Figure 9 shows a set of performance metrics for a program.
Cumulative Flow Diagram
The Cumulative Flow Diagram (CFD) is made up of a series of lines or areas that show the amount of work in different Kanban states. For example, the typical states of the program Kanban are:
- Funnel
- Analyzing
- Backlog
- Implementing
- Validating on staging
- Deploying to production
- Releasing
- Done
Figure 10 shows the number of features in each Kanban state by day. The thicker areas in the CFD represent potential bottlenecks.
Agile Release Train Self-Assessment
As program execution is a core value of SAFe, the ART continuously strives to improve its performance. The Release Train Engineer (RTE) fills out the self-assessment questionnaire at PI boundaries or any time the train wants to pause and reflect on their progress. Trending this data over time is a key performance indicator. Figure 11 gives an example of the results in a radar chart.
Download Agile Release Train Assessment
Continuous Delivery Pipeline Efficiency
The pipeline efficiency compares the amount of touch time versus wait time. Some of the information can be sourced automatically from tools, especially Continuous Integration, and Continuous Deployment, while other data requires manually recording in a spreadsheet. The value stream mapping technique is often applied to analyze problems identified in this report.
Note: The touch time represents when the team is adding value. Typically, touch time is only a small proportion of the total production time, most of the time is spent waiting, such as when moving work, waiting in queues and so on.
Deployments and Releases per Timebox
This metric is meant to demonstrate whether the program is making progress toward deploying and releasing more frequently. It can be viewed on a PI basis, as shown in Figure 13.
Or we can zoom in to see how releases are handled in mid-PI, as shown in Figure 14.
Recovery over Time
This report measures the number of rollbacks that occurred either physically or by turning off feature toggles. The date when a solution was deployed or released to production is also plotted here to determine if there is a relationship between the two.
Innovation Accounting and Leading Indicators
One of the goals of the Continuous Delivery Pipeline is to enable the organization to run experiments quickly to allow Customers to validate the hypotheses. As a result, both Minimal Marketable Features (MMFs) and Minimal Viable Products (MVPs) must define the leading indicators to measure progress toward the benefit hypothesis. Avoid relying on vanity metrics that do not measure real progress.
Figure 16 shows some metrics that were gathered from the SAFe website to demonstrate leading indicators for our development efforts.
DevOps Health Radar
The DevOps Health Radar is a tool to assess the progress of your program in improving the flow of value through Continuous Delivery Pipeline. Figure 17 shows the 16 sub-dimensions that programs should use to assess their maturity. It helps to identify the sub-dimensions in which we are sitting, crawling, walking, running, or flying, and identify places to improve.
Download DevOps Health Radar
You can also go to our partner’s site, Agile Transformation, to take an online version of the assessment at https://agilityhealthradar.com/safe-devops-assessment.
Team Metrics
Iteration Metrics
Each Agile Team gathers the iteration metrics they’ve agreed to collect. This occurs in the quantitative part of the team retrospective. Figure 18 illustrates the chart for the measurements of one team.
Team PI Performance Report
During the PI System Demo, the Business Owners, Customers, Agile teams, and other key stakeholders rate the actual business value (BV) achieved for each team’s PI Objectives as shown in Figure 19.
Reliable trains should operate in the 80 – 100 percent range; this allows the business and its outside stakeholders to plan effectively. Below are some notes about how the report works:
- The planned total BV does not include stretch objectives to help the reliability of the train
- The actual total BV does include stretch objectives
- The achievement percentage is the actual BV ÷ planned BV
- A team can achieve greater than 100 percent (as a result of stretch objectives achieved)
Individual team totals are rolled up into the program predictability measure (see Figure 19).
SAFe Team Self-Assessment
Agile teams continuously assess and improve their process. One such tool is a simple SAFe team practices assessment. When the team completes the spreadsheet, it will automatically produce a radar chart like the one shown in Figure 20, which highlights relative strengths and weaknesses.
Download Agile Team self-assessment
Lean Enterprise Metrics
Lean Enterprise Assessment
Of all the tools in the SAFe Metrics tool bag, the Lean Enterprise Assessment may be the most important. This assessment provides a comprehensive way to self-assess the progress of your enterprise towards the larger goal of achieving true Enterprise Business Agility. Figure 21 shows the 15 sub-dimensions that you can use to assess your progress towards the goal. The spreadsheet (you can download below) that creates this radar is organized around the five SAFe Core Competencies, each has three sub-dimensions that you can rate as “sitting, crawling, walking, running, or flying”. This should give you an understanding of where you are now, and areas to focus on for improvement.
Download Lean Enterprise Assessment
Learn More
[1] Leffingwell, Dean. Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise. Addison-Wesley, 2011.
[2] Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises. Addison-Wesley, 2007.
[3] https://www.youtube.com/watch?v=VOjPpeBh40s
Last update: 24 May 2019