Imagine a world where product owners, Development, QA, IT Operations, and Infosec work together, not only to help each other, but also to ensure that the overall organization succeeds. By working toward a common goal, they enable the fast flow of planned work into production, while achieving world-class stability, reliability, availability, and security.
—The DevOps Handbook
DevOps is a mindset, a culture, and a set of technical practices. It provides communication, integration, automation, and close cooperation among all the people needed to plan, develop, test, deploy, release, and maintain a Solution.
DevOps is part of the Agile Product Delivery competency of the Lean Enterprise. SAFe enterprises implement DevOps to break down silos and empower each Agile Release Train (ART) and Solution Train to release Features on demand to their end users. Over time, the separation between development and operations is significantly reduced, and trains operate with an automated Continuous Delivery Pipeline. This mechanism seamlessly defines, implements, and delivers solution elements to the end user, without handoffs or excessive external production or operations support.
The goal is simple: deliver value whenever there is business demand. This is indeed achievable, as “high-performing IT organizations deploy 30x more frequently with 200x shorter lead times. … 60x fewer failures and recover 168x faster.” 
DevOps is a combination of two words, development and operations. Without a DevOps approach, there’s often significant tension between those who create new features and those who maintain the stability of the solution in production. The development teams are measured on the business value they deliver to end users, while IT service management is measured on the health and stability of the production environment. When each group has seemingly opposing business objectives, delivery inefficiency and organizational friction may rule the day.
But DevOps ends the silo approach, providing an enterprise with the ability to develop, deploy, and release small batches of functionality to the business or customer in a flow process called the continuous delivery pipeline. DevOps is integral to every Value Stream, and, by definition, is integral to SAFe. It includes not just development and operations but everyone needed to release value, such as security, compliance, audit, marketing, legal and others.
Many SAFe concepts and principles—systems thinking, small batch sizes, short iterations, fast feedback, and more—directly support DevOps. In addition, the continuous delivery pipeline and its four aspects of Continuous Exploration (CE), Continuous Integration (CI), Continuous Deployment (CD), and Release on Demand directly support this business need.
The Goal of DevOps
From planning through delivery, the goal of DevOps is to improve collaboration across the value stream by developing and automating a continuous delivery pipeline. In doing so, DevOps:
- Increases the frequency and quality of deployments
- Improves innovation and risk-taking by making it safer to experiment
- Realizes faster time to market
- Improves solution quality and shortens the lead time for fixes
- Reduces the severity and frequency of release failures
- Improves the Mean Time to Recovery (MTTR)
SAFe’s CALMR approach to DevOps covers the five main aspects, as illustrated in Figure 1.
Each aspect is described in the sections below.
A Culture of Shared Responsibility
In SAFe, DevOps leverages the culture created by adopting the Lean-Agile values, principles, and practices of the entire framework. Just about every principle of SAFe, from Principle #1 – Take an economic view to Principle #10 – Organize around value, applies to DevOps. DevOps enables shifting some operating responsibilities upstream, while following development work downstream into deployment, and operating and monitoring the solution in production. Such a culture requires:
- Collaboration and organization – DevOps relies on the ability of Agile teams and IT Operations to collaborate effectively in an ongoing manner, ensuring that solutions are developed and delivered faster and more reliably. This is implemented, in part, by including operations personnel and capabilities on every ART.
- Risk tolerance – DevOps requires a tolerance for failure and rapid recovery, and rewards risk-taking.
- Self-service infrastructures – Self-service infrastructure empowers development and operations to act independently without blocking each other.
- Knowledge sharing – Sharing discoveries, practices, tools, and learning across teams, ARTs and the wider organization is encouraged.
- Automate everything mindset – DevOps relies heavily on automation to provide speed, consistency, and repeatable processes and environment creation, as we describe below.
DevOps simply recognizes that manual processes are the enemy of fast value delivery, high productivity, and safety. But automation is not just about saving time. Builds, testing, deployments, and packaging that are automated improve the reliability of processes that can be made routine. It also enables the creation of repeatable environments and processes, which are self-documenting and, therefore, easier to understand, improve, secure, and audit. Automation also facilitates faster learning and response to market demand and customer feedback.
The entire continuous delivery pipeline is automated to achieve a fast, Lean flow. This is accomplished, in part, by building and applying an integrated and automated ‘tool chain,’ shown in Figure 2, which typically contains the following categories of tools:
- Application Lifecycle Management (ALM) – Application and Agile lifecycle management tools create a standardized environment for communication and collaboration between development teams and related groups.
- Build – Build automation is used to script or automate the process of compiling source code into binary code.
- Testing – Automated testing tools include unit and acceptance testing, performance testing, and load testing.
- Continuous integration – CI tools automate the process of compiling code into a build after developers have checked their code into a central repository. After the CI server builds the system, it runs unit and integration tests, reports results, and typically releases a labeled version of deployable artifacts.
- Artifact Management Repository – These tools provide a software repository for storing and versioning binary files and their associated metadata.
- Continuous deployment – CD tools automate application deployments through to the various environments. They facilitate rapid feedback and continuous delivery while providing the required audit trails, versioning, and approval tracking.
- Additional tools – There are numerous other important tools that support DevOps. These often fall into the categories of configuration, logging, management and monitoring, provisioning, security and code review.
Agile teams and trains strive to achieve a state of continuous flow, enabling new features to move quickly from concept to cash. The three primary keys to implementing flow make up Principle #6 – Visualize and limit WIP, reduce batch sizes, and manage queue lengths. All three are integral to systems thinking (Principle #2), and the long-term optimization of the continuous delivery pipeline. Each is described below in the context of DevOps.
- Visualize and limit Work in Process (WIP) – Figure 3 illustrates an example of a Program Kanban board, which makes WIP visible to all stakeholders. This helps teams identify bottlenecks and balance the amount of WIP against the available development and operations capacity. Work is completed only when the new feature or functionality is running successfully in production.
- Reduce the batch sizes of work items – The second way to improve flow is to decrease the batch sizes of the work. Small batches go through the system faster and with less variability, which fosters faster learning and supports more frequent deployments. This typically involves focusing more attention on, and increasing investment in, infrastructure and automation which in turn enables small batches by reducing the transaction cost of each batch.
- Manage queue lengths – The third way to achieve faster flow is by managing, and generally reducing, queue lengths. For solution development, this means that the longer the queue of work awaiting implementation or deployment, the longer the wait time, no matter how efficiently the team is processing the work. The shorter the queue, the faster the deployment.
In a DevOps environment, problem resolution is less complex, because changes are made more frequently and in smaller batches. Telemetry, or automated collection of real-time data regarding the performance of solutions, helps to quickly assess the impact of frequent application changes. Resolution happens faster because teams don’t need to wait for a different group to troubleshoot and fix the problem.
It’s important to implement application telemetry to automatically collect data on the business and technical performance of the solution. Indeed, basing decisions on data, where ‘the facts are always friendly’, rather than on intuition, leads to an objective, blameless path toward improvement. Data should be transparent. It should be accessible to everyone, be meaningful, and visualized to easily spot problems and trends.
The goal is to build applications that:
- Collect data on business, application, infrastructure, and client layers
- Store logs in ways that enable analysis
- Use different telemetry for different stakeholders
- Broadcast measurements that are hyper-transparent
- Overlay measurements with events (deploys, releases, etc.)
- Continuously improve telemetry during and after problem-solving
It is also important to measure the performance of the continuous delivery pipeline itself (see the continuous delivery pipeline article for more details).
Recover—Enable Low-Risk Releases
To support the continuous delivery pipeline and the concept of release on demand, the system must be designed for low-risk component or service-based deploy-ability, release-ability, and fast recovery from operational failure. Techniques to achieve a more flexible release process are described in the Release on Demand article. In addition, the following techniques support fast recovery:
- Stop-the-line mentality – With a ‘stop-the-line’ mentality, everyone swarms to fix any problem until it’s resolved. When there’s a problem with the continuous delivery pipeline or a deployed solution, the same thinking must apply. Findings are then turned into improvements which are integrated into the process or product to prevent repeat problems in the future.
- Plan for and rehearse failures – When it comes to large-scale IT applications, failure is not only an option, it’s guaranteed at some point. A proactive approach to experiencing failures will increase the team’s response practices and also foster built-in resilience into the systems. (See the ‘Chaos Monkey’ in ).
- Build the environment and capability to fix forward or roll back – Since mistakes will be made, and servers will fail, teams need to develop the capability to quickly ‘fix forward’ and, where necessary, roll back to a prior known good state. In the latter case, planning and investment must be made to revert any data changes back to the prior state, and not lose any user transactions that occurred during the process.
To achieve these recovery capabilities, the organization will typically need to undertake certain enterprise-level initiatives to enhance architecture, infrastructure, and other nonfunctional considerations to support deployment readiness, release, and solutions in production.
Assessing and Improving the Flow through the Pipeline
As is described in the Metrics article, the SAFe’s DevOps Health Radar shown in Figure 4 helps ARTs and Solution Trains assess their maturity in the 16 activities of the continuous delivery pipeline.
Apply Continuous Security with DevSecOps
DevSecOps shifts security practices left, performing them earlier and continuously (Figure 5.). During continuous exploration, information security and other experts perform threat modeling to identify potential security threats based on the solution design and architect to address them. And they ensure Backlogs and Non-Functional Requirements reflect important security requirements. Security tests and tools automatically identify security vulnerabilities during integration which may also run after deployment, but prior to release. Security as code and penetration testing help prevent known vulnerabilities from getting to production. After release, security monitoring watches for breaches and attacks against vulnerabilities.
Learn More Kim, Gene and Jez Humble, Patrick Debois, John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press.  2015 State of DevOps Report https://puppet.com/resources/whitepaper/2015-state-devops-report?link=blog
Last update: 20 January 2019