Metrics

The most important things cannot be measured. The issues that are most important, long term, cannot be measured in advance.
—W. Edwards Deming

Working software is the primary measure of progress.
—Agile Manifesto

Metrics Abstract

People adjust their behavior based on the metrics used to measure their systems and evaluate their performance, so they will naturally flex their efforts to improve the measures the Enterprise puts in place. But it’s true that most measures are simply proxies for some real, and often intangible, results, so they need to be handled with care.

With its work physics, timeboxes, and fast feedback, Agile is inherently more measurable than prior documentation-oriented, indirect, waterfall-based measures of progress. Of course, the best measure by far comes directly from working software and solutions. It’s best for teams, Agile Release Trains, managers, Program Management, and portfolio managers to pivot most of their measuring attention to this critical fact. All other metrics—even the extensive set of Agile metrics outlined below—are subordinate to that objective and the overriding goal of keeping the focus on rapid delivery of quality, working Solutions.

But measures are important in the Enterprise context. To that end, SAFe provides guidance for Metrics on each level of the Framework. The links below navigate to the entries on this page.

Portfolio Metrics

Lean Portfolio Metrics

The Lean Portfolio Metrics set provided here is an example of a comprehensive but Lean set of metrics that can be used to assess internal and external progress for an entire Portfolio. In the spirit of “the simplest set of measures that can possibly work,” Figure 1 provides the leanest set that a few Lean-Agile Enterprises are using effectively to evaluate the overall performance of their transformations.

Figure 1. Lean Portfolio Metrics
Figure 1. Lean Portfolio Metrics

Back to Top

Portfolio Kanban Board

The primary motivation of the Portfolio Kanban Board is to ensure that Epics and Enablers are reasoned and analyzed prior to reaching a Program Increment boundary, are prioritized appropriately, and have established acceptance criteria to guide a high-fidelity implementation. Furthermore, the epics and enablers can be tracked to understand which ones are being worked on and which have been completed, as illustrated in Figure 2.

The Review and Analysis states are WIP limited (the numbers shown in the parentheses in Figure 2) and reflect the limit set by the PPM Team. Refer to the Portfolio Kanban article for information about each of the Kanban states.

Figure 2. Portfolio Kanban Board
Figure 2. Portfolio Kanban Board

Back to Top

Epic Burn-up Chart

The Epic Burn-up Chart tracks progress toward an epic’s completion. There are three measures:

  1. Initial epic estimate line (blue) – Estimated Story points from the lightweight business case
  2. Work completed line (red) – Actual story points rolled up from the epic’s child Features and stories
  3. Cumulative work completed line (green) – Cumulative story points completed and rolled up from the epic’s child features and stories

These are illustrated in Figure 3.

Figure 3. Epic Burn-up Chart
Figure 3. Epic Burn-up Chart

Back to Top

Epic Progress Measure

The Epic Progress Measure provides an at-a-glance view of the status of all epics in a Program portfolio.

  1. Epic X – Represents the name of the epic; business epics are blue (below) and enabler epics are red
  2. Bar length – Represents the total current estimated story points for an epic’s child features/stories; the dark green shaded area represents the actual story points completed; the light green shaded area depicts the total story points that are “in progress”
  3. Vertical red line – Represents the initial epic estimate, in story points, from the lightweight business case
  4. 0000 / 0000 – The first number represents the current estimated story points, rolled up from the epic’s child features/stories; the second number represents the initial epic estimate (same as the red line)

These measures are depicted in Figure 4.

Figure 4. Epic Progress Measure
Figure 4. Epic Progress Measure

Back to Top

Epic Success Criteria

Each epic should have Success Criteria that can be used to help establish scope and drive more detailed feature elaboration. Figure 5 shows an example of an enabler epic with associated success criteria.

 

Figure 5. An example of Success Criteria for an enabler epic
Figure 5. An example of Success Criteria for an Enabler Epic

Back to Top

Enterprise Balanced Scorecard

The Enterprise Balance Scorecard provides four perspectives to measure performance for each portfolio—although the popularity of this approach has been declining over time in favor of Lean Portfolio Metrics (see Figure 1). Nonetheless, these measures are:

  1. Efficiency
  2. Value Delivery
  3. Quality
  4. Agility

These measures are then mapped into an executive dashboard, as illustrated in Figures 6 and 7.

Figure 6. A “balanced scorecard” approach, dividing measures into four areas of interest
Figure 6. A Balanced Scorecard approach, dividing measures into four areas of interest

Figure 7. Converting the above metrics to an alphabetical rating and aggregating them can provide a look at the bigger picture for the enterprise
Figure 7. Converting the above metrics to an alphabetical rating and aggregating them can provide a look at the bigger picture for the enterprise

For more on this approach, see [2], Chapter 22.

Back to Top

Program Portfolio Management Self-Assessment

The Program Portfolio Management (PPM) team continuously assesses and improves their processes. Often this is done using a structured, periodic Self-Assessment. When the PPM team completes the spreadsheet below, it will automatically produce a radar chart like that shown in Figure 8, which highlights relative strengths and weaknesses.

Figure 8. Portfolio Self-Assessment Radar Chart
Figure 8. Program Portfolio Management Self-Assessment radar chart

Back to Top

Value Stream Metrics

Value Stream Kanban Board

The primary motivation of the Value Stream Kanban Board is to ensure that Capabilities and Enablers are reasoned and analyzed prior to reaching a PI boundary and are prioritized appropriately, and that acceptance criteria have been established to guide a high-fidelity implementation. Furthermore, the features can be tracked to understand which ones are being worked on and which have been completed, as illustrated in Figure 9.

The Review and Analysis states are WIP limited, as shown by the numbers in the parentheses. Refer to the Program and Value Stream Kanban article for information about each of the Kanban states.

 

Figure 9. Value Stream Kanban Board
Figure 9. Value Stream Kanban Board

Back to Top

Value Stream Predictability Measure

To assess the overall predictability of the Value Stream, individual predictability measures for an Agile Release Train (ART) can be aggregated to create an overall Value Stream Predictability Measure, as illustrated in Figure 10.

Figure 10. Value Stream Predictability Measure
Figure 10. Value Stream Predictability Measure

Back to Top

Value Stream Performance Metrics

To assess the overall performance of the value stream, individual performance measures for an ART can be aggregated to create an overall set of Value Stream Performance Metrics, as illustrated in Figure 11.

Figure 11. Value Stream Performance Metrics
Figure 11. Value Stream Performance Metrics

Back to Top

 

Program Metrics

Feature Progress Report

The Feature Progress Report tracks the status of features and enablers during PI execution. It indicates which features are on track or behind at any point in time. The chart has two bars:

  1. Plan – Represents the total number of stories planned for a feature.
  2. Actual – Represents the number of stories completed for a feature. The bar is shaded red or green, depending on whether the feature is on track or not.

Figure 12 gives an example of a feature progress report.

 

Figure 13. Feature Progress Report, highlighting the status of each feature compared to the PI plan
Figure 12. Feature Progress Report, highlighting the status of each feature compared to the PI plan

Back to Top

Program Kanban Board

The primary motivation of the Program Kanban Board is to ensure that features are reasoned and analyzed prior to reaching a PI boundary and are prioritized appropriately, and that acceptance criteria have been established to guide a high-fidelity implementation. Furthermore, the features can be tracked to understand which ones are being worked on and which have been completed, as illustrated in Figure 13.

The Review and Analysis states are WIP limited, as shown by the numbers in the parentheses. Refer to the Program and Value Stream Kanban article for information about each of the Kanban states.

 

Figure 14. Program Kanban Board
Figure 13. Program Kanban Board

Back to Top

Program Predictability Measure

To assess the overall predictability of the release train, the “Team PI Performance Report” is aggregated, for all teams on the train, to calculate the Program Predictability Measure, as illustrated in Figure 14. The Team PI Performance report compares actual business value achieved to planned business value (see Figure 22).

Figure 14. Program Predictability Measure, showing two of the teams on the train and program (cumulative)
Figure 14. Program Predictability Measure, showing two of the teams on the train and program (cumulative)

For more on this approach, see [1], Chapter 15.

Back to Top

Program Performance Metrics

The end of each PI is a natural and significant measuring point. Figure 15 is an example set of Performance Metrics for a program.

Figure 15. One train's chart of Performance Metrics
Figure 15. One train’s chart of Performance Metrics

Back to Top

PI Burn-down Chart

The PI Burn-down Chart shows the progress being made toward the program increment timebox. Use it to track the work planned for a PI against the work that has been accepted.

  1. The horizontal axis of the PI burn-down chart shows the iterations within the PI
  2. The vertical axis shows the amount of work (story points) remaining at the start of each iteration

Figure 16 exemplifies a train’s burn-down measure. Although the PI burn-down shows the progress being made toward the program increment timebox, it does not reveal which features may or may not be delivered during the PI. The Feature Progress Report provides that information (refer to Figure 12).

Figure 16. One train's PI Burn-down Chart
Figure 16. One train’s PI Burn-down Chart

Back to Top

Cumulative Flow Diagram

A Cumulative Flow Diagram (CFD) is made up of a series of lines or areas representing the amount of work in different stages of progression. For example, in software development, typical steps for development of a feature are:

  • Funnel
  • Backlog
  • Review
  • Analysis
  • Execution
  • Done

In the cumulative flow diagram in Figure 17, the number of features in each stage of development is plotted for each day in the chart.

Figure 18. Sample PI Cumulative Flow Diagram
Figure 17. Sample PI Cumulative Flow Diagram

Back to Top

Agile Release Train Self-Assessment

As program execution is a core value of SAFe, the Agile Release Train (ART) continuously works to improve its performance. A Self-Assessment form (below) can be used for this purpose at PI boundaries or any time the team wants to pause and assess their organization and practices. Trending this data over time is a key performance indicator for the program. Figure 18 gives an example of the results of a self-assessment radar chart.

Figure 19. ART Self-Assessment radar chart
Figure 18. ART Self-Assessment radar chart

Back to Top

Team Metrics

Iteration Metrics

The end of each iteration is the time for each Agile Team to collect whatever Iteration Metrics they have agreed upon. This occurs in the quantitative part of the team retrospective. One such team’s metrics set is illustrated in Figure 19.

Figure 20. One team’s Iteration Metrics
Figure 19. One team’s chart of Iteration Metrics

Back to Top

Team Kanban Board

A team’s Kanban process evolution is iterative. After defining the initial process steps (e.g., define – analyze – review – build – integrate – test, etc.) and WIP limits, then executing for a while, the team’s bottlenecks should surface. If not, the team refines the states or further reduces the WIP until it becomes obvious which state is “starving” or is too full, helping the team adjust toward more optimum flow.

As the assumptions are validated, WIP limits are adjusted and steps may be merged, split, or redefined. Figure 20 shows a Team Kanban Board that has surfaced bottlenecks, along with its current WIP limits (top row).

Figure 21. One team’s initial Kanban Board
Figure 20. One team’s initial Kanban Board

Back to Top

Team PI Performance Report

During the PI System Demo, the business owners, customers, Agile Teams, and other key stakeholders collaboratively rate the actual business value achieved for each team’s PI objectives as shown in Figure 21.

Reliable trains should generally operate in the 80% – 100% range; this allows the business and its outside stakeholders to plan effectively. Below are some important notes about how the report works:

  1. The Planned total (BV) does not include stretch objectives to help the reliability of the train
  2. The Actual total (Actual BV) includes stretch objectives
  3. The Achievement % is calculated by dividing the Actual BV total / Planned BV total
  4. A team can achieve greater than 100% (as a result of stretch objectives achieved)
  5. The effort required for stretch objectives is included in the Iteration plan’s load (i.e., it is not extra work the team does on weekends)
  6. Individual team totals are rolled up into the Program Predictability Measure (see Figure 14)
Figure 21. Team PI Performance Report
Figure 21. Team PI Performance Report

Back to Top

SAFe Team Self-Assessment

Agile Teams continuously assess and improve their processes. Often this is via a structured, periodic Self-Assessment. This gives the team time to reflect on and discuss the key practices that help yield results. One such assessment is a simple SAFe Team practices assessment, as appears in this spreadsheet:

When the team completes the spreadsheet, it will automatically produce a radar chart such as that shown in Figure 22, which highlights relative strengths and weaknesses.

 

Figure 22. Team Self-Assessment radar chart
Figure 22. SAFe Team Self-Assessment radar chart

 

Back to Top


Learn More

[1] Leffingwell, Dean. Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise. Addison-Wesley, 2011.

[2] Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises. Addison-Wesley, 2007.

Last update: 5 April 2016