The epiphany of integration points is that they control product development. They are the leverage points to improve the system. When timing of integration points slip, the project is in trouble.
—Dantar Oosterwal, The Lean Machine
Continuous Integration (CI) is perhaps the most essential technical practice for an ART and Value Stream, a quality heartbeat that reduces risk and establishes fast, sustainable development. If teams focus solely on features and component development, accumulated assumptions and unintended interactions in the system create waste, which exhibits itself as delays and delivery variability. But with early and often CI, teams take a system view that keeps the development process in control and enhances predictability.
However, continuous integration in complex, heterogeneous systems—those that contain mechanical subsystems, software, electrical and electronic subsystems, subassemblies from and suppliers, and so forth—is difficult. Lean system builders need a balanced approach, which allows the teams to build quality in and receive fast feedback from the integrated increments. This often requires economic trade-offs between frequency and scope of integration and testing. Proper infrastructure and continuous improvement of CI practices, along with automation techniques and design for testability, facilitates fast feedback and helps ARTs achieve ever higher velocity.
A primary source of complexity in large solutions is the inter-dependencies between components, the cross-cutting nature of the solution’s capabilities, and unintended interactions amongst solution elements. It is simply impossible to envision all those potential behaviors up front. While proactive identification and management of such dependencies is required, integrating and testing the collaborating components together, as a solution, is the only practical way to fully validate the solution.
Feature and Component Level Continuous Integration
An Agile Team creates value only when the team members frequently integrate and test the results of their combined work. If integration does not happen frequently, there is no way to ensure that different roles and functions on the team are actually able to produce a consistent output: the team increment. Therefore in SAFe, it is highly recommended that the members of an Agile team integrate their work at least once per Iteration, with a preference towards a more frequent rhythm.
This integration and testing effort is part of every iteration and should be properly planned for, by either including the effort as a part of Story estimates (included in the Story’s Definition of Done), or by having separate backlog items in the iteration backlog for integration and testing activities. One way or the other, the effort should be part of the team’s velocity, otherwise the team will fail to establish a sustainable pace—the most basic prerequisite for development predictability at scale.
However, while critical, local integration and testing isn’t enough. Even a simple change to a “tested” component may have unintended consequences that may ripple through the rest of the system, as Figure 1 illustrates.
Worse, it may been create a false sense of security (“wishful thinking,” one of the key wastes of Lean) that the system is actually working. Without a full system view, significant problems can accumulate quickly as teams build more and more functionality on top of unvalidated work, with exponential growth of problems.
Frequent integration and testing, at least once every iteration, provides the necessary means to evaluate progress and discover problems while they are still easy to fix. This produces the System Increment—an integrated, functioning set of features and components, the subject of the System Demo—that can be reviewed by the teams themselves and by Product Owners and other key stakeholders. Even more importantly, it illustrates the real, tangible value of the increment.
Large Solutions require an additional level of integration for learning and to understand the solution’s progress across multiple release trains within a value stream. The Solution Demo event is where the results of all the development efforts from multiple Agile Release Trains—along with the contributions from suppliers and other value stream participants—are made visible to the customers and other stakeholders. This is the most critical time for a fully integrated, Solution-level demonstration; a time for objective evaluation and stakeholder and customer feedback.
The solution integration and demo are the joint responsibility of the system teams at the ART and Value Stream Levels, as illustrated in Figure 2 below. These big systems can be hard to integrate. In order to be able to routinely demonstrate a solution increment, teams must typically invest in integration, testing, and supporting infrastructure. Even then, the extent of integration and testing may be less than 100 percent and may need to vary across multiple early integration points, as illustrated in Figure 2.
To assist, teams can leverage virtualization, environment emulation, mocks, stubs, reduced test suites, etc. In addition, some effort for integration and demonstration, and time to invest in the supporting environment, may need to be explicitly allocated during Pre-PI Planning.
Synchronize with Supplier and Solution Context
Suppliers play a key role in SAFe due to their unique capabilities and solutions. They can have a large impact on lead time and value delivery. In order for both organizations to achieve optimal results, close collaboration and trust are required, that are facilitated by the following recommended practices:
- Plan the integration points
- Adopt cadence whenever appropriate; establish milestones on objective evaluation of the working solution
- Collaborate with parent system’s and supplier ART’s system teams
- Build and maintain consistent integration and testing infrastructure
- Participate in the Pre- and Post-PI Planning and Solution Demos; invite suppliers to attend
- Establish collaboration and synchronization between Architecture/Systems Engineering at all all levels
- Synchronize the solution with solution context to assure the as-built solution is compatible with the deployment environment
Challenges and Trade-offs
Each Agile Release Train’s goal is to be able to fully integrate Features across all teams in each iteration. However, that can be difficult due to solution complexity and heterogeneity, availability of specialty testing personnel, laboratories, equipment, third party components, etc. Given this, teams may feel that achieving the integrated iteration goal is not realistic, at least initially. But that cannot be an excuse for continuing discontinuous, late-in-the cycle integration and testing.
When such trade-offs are required, teams need to adjust the practice accordingly, but in a manner so as to still achieve a reliable, fast feedback mechanism. Figure 3 illustrates a trade-off curve that can help teams think through the optimum scope and extent of CI in their context.
These trade-offs are achievable due to the fact that different parts of the solution have different levels of coupling, and some capabilities have various degrees of impact on different system elements. Also, different components contribute differently to the overall cost of integration and testing; some are simply easier to integrate and test than others. Below are some suggestions for how to achieve most of the benefits, even if full, fast CI isn’t immediately practical.
- Integrate different aspects at different frequencies. For example, the Baja Ride control system teams might integrate their software multiple times per iteration, and potentially every day. A full vehicle integration, however, might be feasible only every few weeks, or perhaps at a Program Increment boundary.
- Integrates all assets, but run deprecated tests. Full validation, including running all regression tests, may be too slow or costly, but based on history, teams can usually determine that a majority of the problems can be found with a few ‘vertical’ tests. The full regression suite is then reserved for less frequent intervals.
- Use of emulated environments, stubs, and mocks. Some components, equipment, third-party subsystems, etc. may not be readily available to test frequently in a real environment. Instead, teams build emulators, mocks, and stubs to simulate the rest of the system environment.
It is also important to note that frequent integration represents a special challenge for groups that are in the initial transition to leaner methods. They just haven’t done it previously, nor have they yet built the infrastructure necessary to do so. But the goal is clear, so the current state cannot be an excuse to simply reduce the scope or extent of integration. Most of these challenges should disappear in a near-future state, but only if the teams start now.
Enabling Continuous Integration
Building complex solutions in a Lean and Agile manner is a journey, a journey gated in large part by continuous integration capabilities. The following section provides some suggestions for how to move forward quickly.
Integration points are easier when all the teams are moving in the same, consistent rhythm. That’s why all iterations are aligned on the same PI boundaries. While it may be necessary to allow some minor shifts of integration/testing effort past the iteration end point, the timelier, the better. If full CI can’t be accomplished in the course of the iteration, teams can make near term trade-offs, while continuously improving their techniques and infrastructure.
Effective continuous integration depends on availability of the execution environments, CI and test tools, mocks and emulators, etc. Some of the infrastructure is built and supported by the System Team. Other aspects are more efficiently managed by the teams themselves, allowing them to more quickly understand the performance of the system at all times.
Infrastructure is of course, an investment, and as such must be evaluated against the anticipated return. An organization that doesn’t yet have the Lean culture of Built-in Quality, may struggle to understand the positive future impact of infrastructure enhancements. Therefore, new investments in development time, as well as equipment, may be considered simply a source of expense or delay. But Lean-Agile Leaders take the long view, and make the investments necessary today, to increase velocity for the marathon ahead.
Engineering Techniques in Support of CI
Continuous integration is easier when the system is designed with those concerns in mind. Designing for Testability—a technique that originates from hardware development—calls for better modularity, and separation of concerns, as well as the use of primary interfaces and physical test points. Set-Based Design can also help provide a clearer separation of some solution elements (via interfaces, protocols) that can be fixed for testing, while preserving flexibility.
The frequency of integration and testing is a primary determinate of quality. However, the cost of doing so may be a concern, especially if much of the work is performed manually. Automation of the process, or selected parts, lowers these costs and increases productivity. When accompanied by some of the other techniques discussed above, automation can significantly change the shape of the Figure 1 trade-off curve.
Support of the System Team
The System Team carries a lot of this responsibility. But it is not the only group of people, nor the ones solely responsible, for the quality of the ART increment. Integration and Testing is therefore a role collaboration, with a constant exchange of knowledge within the ART and Value Stream.
If Agile Teams are kept out of the integration loop, they will continue to deliver inconsistent increments. Similarly, if the teams do not provide the System Team with the ever-changing Solution Context, it will quickly lead to gaps in expectations, and discrepancies in environments and configurations.
CI Is a Culture
Many continuous integration discussions revolve around tooling, and tooling is a critical element. However, continuous integration is a cultural change, a new mindset that requires new responsibilities at all levels. Below are three suggestions for implementing a successful CI culture:
- Integrate often. The more frequently teams integrate, the quicker they find problems. The harder it is to do, the more often they need to do it—eliminating impediments and adding automation along the way—resulting in faster learning cycles and less rework.
- Make integration results visible. When the integration process breaks, everybody should know how and why it broke. And when it’s fixed, they should know what fixed it, too.
- Fixing a failed integration attempt is the top priority. If teams simply keep working in the presence of a failed integration effort, it negates the gains and can even create apathy for the effort. To stress this, many teams use flashing lights when a build is broken, and highly visible indicators of percentages of the time the system is broken. Attention is then directed immediately to fixing the problems before new functionality is added, thereby accelerating learning and decreasing the impact of the underlying issue.
Last update: 18 November 2015