In order for you to keep up with customer demand, you need to create a deployment pipeline. You need to get everything in version control. You need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on demand.

—Erik to Grasshopper, The Phoenix Project

Continuous Deployment

Continuous Deployment (CD) is an aspect of the Continuous Delivery Pipeline that automates the migration of new functionality from a staging environment to production, where it is made available for release.

CD is the third aspect in the four-part Continuous Delivery Pipeline (CDP) of Continuous Exploration (CE), Continuous Integration (CI), Continuous Deployment (CD), and Release on Demand (Figure 1).

Figure 1. Continuous deployment in the context of the CDP
Figure 1. Continuous deployment in the context of the CDP

Features must be available and verified in production before the business needs them to support Release on Demand. Therefore, it’s optimal to separate the deployment process from releasing, enabling changes to move into production without affecting the behavior of the current system. Continuous deployment allows teams to deploy small, incremental changes to production continually.

The capability to continuously deploy is critical for releasing on demand. In turn, it allows Agile Release Train (ARTs) to respond to market opportunities with the highest possible value in the shortest sustainable lead time, permitting customers to consume new functionality when they are ready.

Details

Traditional development practices treat deployment and release as the same activity. In this model, changes deployed to production are immediately available to users. Continuous deployment, however, separates the deployment and release processes. This practice fosters design thinking and fast value flow by:

  • Targeting functionality to specific customers – Enables the organization to target customers with particular functionality, allowing the organization to assess the impact of changes before deploying functionality to all customers.
  • Promoting experimentation, such as A/B Testing – Design thinking practices, such as A/B testing, require the ability to present different functionality to distinct target users, gathering feedback that helps create the optimal user experience.
  • Promoting small batches – Automating the CDP (for example, tests, builds, deploys) makes deploying in small batches economically feasible.
  • Releasing on business needs – ARTs tend to release less frequently when the deployment process is complex and error-prone. Organizations that invest in automation and relentless process improvement can release faster and with lower risk, substantially increasing Business Agility. For example, a release can be deployed in production ahead of a marketing campaign, giving the organization more flexibility in maximizing all aspects of value delivery.

To enable these capabilities, ARTs focus on reducing the transaction cost and risk of moving changes to production by automating all aspects of continuous deployment. Ensuring the deployment process is a repeatable, predictable activity without significant incidents helps teams achieve continuous deployment. Moreover, improving deployment to make value flow without interruption Principle #6) is critical for achieving business agility. See the ART Flow article for more information.

The Four Activities of Continuous Deployment

SAFe describes four activities of Continuous Deployment, as illustrated in Figure 2.

  1. Deploy – the practices necessary to deploy a solution to a production environment
  2. Verify – the practices needed to ensure solution changes operate in production as intended before releasing them to customers
  3. Monitor – the practices to monitor and report on any issues that may arise in production
  4. Respond – the practices to address any problems rapidly which may occur during deployment
Figure 2. Four activities of continuous deployment
Figure 2. Four activities of continuous deployment

Deploy

Deployment is the migration of changes into a production environment. In the CDP, deploying changes is done continuously. Partial functionality can be implemented incrementally into production.

Suppose a user story map has 27 stories to implement a new workflow. Traditional legacy practices would likely result in all 27 stories deployed in one large batch. Instead, individual stories can be deployed in ‘dark mode’ with continuous deployment and feature toggles. When the ART has deployed the full features set, the business can choose when to release them to users.

Ideally, the deployment pipeline triggers the deployment process automatically following a successful build, integration, and validation. This approach makes the workflow a fully-automated one-click process, from code-commit to production-deploy. Highly sophisticated enterprises can reliably deploy anytime, even during peak periods. This approach eliminates the need to work weekends, nights, or other off-hours to deploy.

Several practices contribute to the ability to deploy:

  • Dark launches – the ability to deploy new functionality to a production environment without releasing it to end users
  • Feature toggles – a technique to facilitate dark launches by implementing toggles in the code, which enables switching between old and new functionality
  • Deployment automation – the ability to deploy a tested solution automatically from staging to production
  • Selective deployment – the ability to deploy to specific production environments and not others based on criteria such as geography, market segment, and more
  • Self-service deployment – when automated deployment is partially implemented, self-service allows a single command to move solutions from staging to production
  • Version control – maintaining environments under version control enables fast deployment and recovery
  • Blue/green deployment – a technique that permits on-demand switching between staging and production environments

Verify

Deployments must be verified for functional integrity and robustness before releasing to end users. These two processes almost happen simultaneously when tightly coupled, making recovery decisions a primary concern. However, when they are separated, there’s room to test new functionality extensively in production before approving it for release. After migration to production, solutions undergo a final round of testing. Typically, this requires a smoke test, light user acceptance testing, and a stress and performance test, which must occur in a production setting. This verification provides the necessary sanity check that tests the behavior of the solution in its actual production Solution Context.

Continuous Integration reasonably ensures that the solution will behave as expected in production; however, surprises do occur. When verification reveals critical defects, deployments must either be rolled back or fixed quickly to prevent them from harming the production environment or disrupting the business flow.

Four practices help drive verification after deployment:

  • Production testing – testing solutions in production using feature toggles or ‘dark’ launches
  • Test automation – the ability to automate tests and run them rapidly and repeatedly
  • Test data management – managing test data in version control to ensure consistency in automated testing
  • Testing nonfunctional requirements (NFRs) – teams also test system attributes such as security, reliability, performance, scalability, and usability to ensure NFRs meet quality standards

Monitor

Verifying that deployed features didn’t break on their way into production is an essential pre-release quality check. However, teams must also ensure they can measure a feature’s performance and value over its lifespan. The insights that drive this critical feedback loop primarily come from robust monitoring capabilities, which must be in place before release.

Effective monitoring requires that full-stack telemetry is active for all features deployed through the CDP. This telemetry allows teams to verify system performance, end-user behavior, incidents, and business value rapidly and accurately in production. The data collected provides tracking and monitoring of each feature, increasing the fidelity of analysis of the business value delivered and increasing responsiveness to production issues.

While teams cannot collect some business-value metrics until release, they need to know how to obtain the measures before the release decision occurs. Some practices which help support this include:

  • Full-stack telemetry – the ability to monitor for problems across the entire stack that a system covers
  • Visual displays – tools that display automated measurements
  • Federated monitoring – consolidated monitoring across applications in the solution that creates a holistic view of problems and performance

The Measure & Grow article provides some guidance on the types of metrics that monitoring requires.

Respond

The ability to respond to and recover from unforeseen production issues is critical to supporting continuous deployment and streamlining the CDP. The reasons are obvious:

  • Production issues directly affect customers and end users, so the value of deployed solutions can quickly erode when problems occur.
  • Production issues causes rework—fixes, patches, redevelopment, retesting, redeployment, etc. That disrupts the normal flow of value through the pipeline.

Since production issues can harm delivery efficiency and lower value, teams need the capability to detect problems proactively and recover quickly. As measured by Mean Time to Restore (MTTR), fast recovery is among the most reliable leading indicators of high DevOps maturity [5]. Recovery is also one of the five elements of SAFe’s CALMR approach to DevOps.

The goal of responding and recovering is to identify potential issues before they turn into incidents and to prevent them from affecting business operations. This capability requires detecting difficulties internally before end users discover them, quickly identifying root causes, and restoring services with well-rehearsed procedures. In contrast, making hasty, reactive changes directly to production systems—‘just to keep the lights on’—invites source code and configuration differences between environments, unverified changes, and long-term risk.

Several practices support the ability to respond and recover from production issues:

  • Proactive detection – a technique for proactively creating faults in the solution to identify potential problems and situations before they occur. For example, Chaos Monkey [1], developed by Netflix, is an open-source tool that randomly terminates instances in production to ensure that engineers implement their services to be resilient to instance failures.
  • Cross-team collaboration – a mindset of cooperation across the Value Stream to identify and solve problems as they arise
  • Session replay – the ability to replay end-user sessions to research incidents and identify problems
  • Rollback and fix forward – the ability to both rollback a solution quickly to a previous environment or to fix a problem quickly through the pipeline without the need to rollback
  • Immutable infrastructure – This concept refers to never amending servers or Virtual Machines (VMs) after deployment. Instead, a new server is built from an image with the appropriate changes if something needs updating.
  • Version control – environments should be maintained under version control to rollback quickly

After teams have demonstrated that features have been deployed successfully to production and have the necessary monitoring and recovery capabilities to track and manage ongoing value, they have completed the continuous deployment stage of the CDP. In turn, this gives the enterprise the ability to release whenever warranted.

Enabling Continuous Deployment with DevOps

Continuous deployment involves critical operation activities frequently associated with the ‘Ops’ in DevOps. They focus on deploying solutions to production environments, verifying their functional integrity, and ensuring effective monitoring and post-release support.

Figure 3 illustrates that SAFe’s CALMR approach to DevOps (center) and several practice domains (inner rings) enables continuous deployment. Each of the four activities (in green) is a collaborative effort that draws upon DevOps expertise from multiple disciplines to maximize delivery speed and quality.

Figure 3. DevOps enables continuous deployment
Figure 3. DevOps enables continuous deployment

For instance, deploying solutions in the CDP involves using tools that automate the provisioning of production infrastructure, deploy solution binaries to select targets, verify production functionality, capture runtime telemetry, and proactively alert on issues. DevOps practices and tools streamline these capabilities, allowing solutions to be deployed and fully prepared for on-demand release in minutes.

All four continuous deployment activities are enabled by DevOps, though with different combinations of technical practices and tooling. See the DevOps article series for more guidance on DevOps and how it facilitates the CDP.

 


Learn More

[1] https://netflix.github.io/chaosmonkey/

[2] Kim, Gene, et al. The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. IT Revolution Press, 2013.

[3] Kim, Gene, Jez Humble, Patrick Debois, and John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press, 2016.

[4] Humble, Jez, and David Farley. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, 2010.

[5] Gregory, Janet, and Lisa Crispin. More Agile Testing: Learning Journeys for the Whole Team. Addison-Wesley Signature Series (Cohn). Pearson Education, 2014.

[6] State of DevOps Report. https://puppet.com/resources/whitepaper/state-of-devops-report

 

Last update: 9 January 2023