In order for you to keep up with customer demand, you need to create a deployment pipeline. You need to get everything in version control. You need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on demand.

—Erik to Grasshopper, The Phoenix Project [1]

Continuous Deployment

Continuous Deployment (CD) is the process that takes validated Features from a staging environment and deploys them into the production environment, where they are readied for release.

It is the third element in the four-part Continuous Delivery Pipeline of Continuous Exploration (CE), Continuous Integration (CI), Continuous Deployment, and Release on Demand, as Figure 1 shows.

Figure 1. Continuous Deployment in the context of the Continuous Delivery Pipeline

The ability to Release on Demand is a critical competency for each Agile Release Train (ART) and Solution Train. It allows businesses to respond to market opportunities with the highest value solutions in the shortest sustainable lead times, and to do so at a rate that allows customers to absorb the new functionality.

To enable a business to release on demand, the features must be waiting and verified in production before the business needs them. Therefore, the deployment process is separated from the release process—where deployed changes are moved into the production environment in a manner that does not affect the behavior of the current system. This gives the teams the ability to make smaller, incremental changes which can be deployed to production on a continuous basis but are not released to end users until the time is right.

This article, Continuous Deployment, details the skills a Lean Enterprise needs to continuously deploy potential end-user value to production.

Details

DevOps and Release on Demand is one of the five core competencies of the Lean Enterprise. It provides the enterprise with the ability to deliver increasingly valuable solutions to end users with optimal frequency. By systematically reducing time in the development cycle with Built-In Quality approaches, SAFe’s leaner and more Agile approach to the delivery process helps establish faster development flow.

Currently, however, many development teams still deploy solutions to production in large batches. This means the actual deployment and release of the new solution are likely to be manual, error-prone, and unpredictable, adversely affecting go-to-market commitments and solution quality.

For most organizations, this happens because deployment and release are the same activity. Once value is deployed to production, it is immediately in the hands of the users. This drives organizations to deploy only when the business needs the value, and only when it’s possible to do a full release regression-testing cycle, which leads to large batches.

To address this, ARTs focus their attention collectively on reducing the transaction cost and risk of production deployments by implementing continuous deployment.  By working together to reduce the deployment process to a routine, predictable, “non-event,” teams enable their organizations to achieve a continuous deployment capability that permits value to be released on demand. Facilitating low-cost migration of high-quality changes to production on a regular cadence builds predictability, responsiveness, and competitive advantage.

The Four Sub-dimensions of Continuous Deployment

SAFe describes the four basic sub-dimensions of Continuous Deployment, as Figure 2 shows: deploy to production, verify the solution, monitor for problems, and respond and recover:

  • Deploy to production covers the skills necessary to deploy a solution to a production environment
  • Verify the solution covers the skills needed to make sure the changes operate in production as intended before they are released to customers
  • Monitor for problems covers the skills to monitor and report on solutions
  • Respond and recover encompasses the skills to quickly address any problems that happen during deployment

 

Figure 2. Four sub-dimensions of Continuous Deployment

Deploy to Production

Deployment is the actual migration of changes into a production environment. In the Continuous Delivery Pipeline, changes are deployed continuously, regardless of whether the business is ready to release them to end users, and without waiting for complete features (or even stories) to be ready. ARTs need to be able to deploy features in “dark” mode—using feature toggles, for example—so that they can be validated, monitored and queued in a bona fide production setting until customers are ready to receive them.

The actual deployment process needs to be quick, painless, and highly reliable. This is achieved by automating the entire deployment process, from server provisioning and infrastructure configuration to database scripting and code migration. Therefore, it’s imperative to maintain all deployable assets in version control and script all deployment steps in a deployment automation tool.

Ideally, the deployment process is triggered automatically by the deployment pipeline upon successful build, integration, and validation. This makes the entire workflow, from code-commit to production-deploy, a fully-automated “one-click” process. Additionally, organizations should be able to deploy reliably any time of day, any day of the week, and any week of the year—even during peak periods.

The result is a rhythmic, reliable deployment of features to a bona fide production environment, where teams prepare for release to end users.

Seven skills contribute to the ability to deploy:

  • Dark launches – the ability to deploy to a production environment without releasing the functionality to end users
  • Feature toggles – a technique to facilitate dark launches by implementing toggles in the code, which enables switching between old and new functionality
  • Deployment automation – the ability to deploy a tested solution automatically from check-in to production
  • Selective deployment – the ability to deploy to specific production environments and not others based on criteria such as geography, user role, etc.
  • Self-service deployment – when automation deployment is not fully implemented, self-service deployment allows a single command to take solutions from staging to production
  • Version control – maintaining environments under version control enables fast deployment and fast recovery
  • Blue/green deployment – a technique that permits automatic switching between two environments, one for deployment and one that is live

Verify the Solution

Before being released to end users, deployments must be verified for functional integrity and robustness. When they’re coupled, deployment and release have to happen almost instantaneously, as decisions must be made immediately about whether to rollback or not. When they’re decoupled, however, there’s room to test new functionality extensively in production before marking it as ready for release. Immediately following the migration to production, solutions undergo a final round of testing, typically in the form of smoke testing and/or light user acceptance testing, but also stress and performance testing that can only be done in production. This provides a critical sanity check that tests the behavior of the solution in an actual production environment.

Continuous Integration will have already provided assurance upstream that the solution will behave as expected in production; however, surprises do occur. When verification reveals critical defects, deployments must either be rolled back or fixed quickly to prevent them from contaminating the production environment or disrupting the flow of business.

Once the deployed changes are verified and operable as intended in the production environment, they are one step closer to being able to release. Four skills help drive verification:

  • Production testing – the ability to test solutions in production when they are still ‘dark’
  • Test automation – the ability to test repeatedly via automation
  • Test data management – managing the test data in version control to create consistency in automatic testing
  • Testing nonfunctional requirements (NFRs) – system attributes such as security, reliability, performance, maintainability, scalability, and usability must also be thoroughly tested before release

Monitor for Problems

Verifying that deployed features didn’t break on their way into production is an important pre-release quality check. However, teams also need to ensure they can measure a feature’s performance and value over its entire useful life. Teams cannot stop checking for quality now—before a feature has been used. That would undermine the whole intent of the Continuous Delivery Pipeline, which is to test solution ideas (hypotheses) in a real business setting to make quick and informed judgments about how best to enable strategic business outcomes. The insights that drive this critical feedback loop come primarily from robust monitoring capabilities, which must be in place prior to release.

Effective monitoring requires that full-stack telemetry is active for all features deployed through the Continuous Delivery Pipeline. This ensures that system performance, end-user behavior, incidents and business value can be determined quickly and accurately in production. That information allows tracking and monitoring of each feature, which increases the fidelity of the assertions about business value delivered, as well as increased responsiveness to production issues.

While some business-value metrics cannot be collected until release, teams need to make sure they know how to measure them once the decision to release occurs. Three skills help support this:

  • Full-stack telemetry – the ability to monitor for problems across the full stack that a system covers
  • Visual displays – tools that display automated measurements
  • Federated monitoring – consolidated monitoring across applications in the solution that creates a holistic view of problems and performance

Respond and Recover

The ability to respond and recover from unforeseen production issues is critical to enabling continuous deployment and streamlining the Continuous Delivery Pipeline. The reasons are obvious:

  • Production issues impact business customers and end users directly, so the value of deployed solutions can erode quickly when issues occur.
  • Production issues spawn rework in the form of fixes, patches, redevelopment, retesting, redeployment, etc. That disrupts the normal flow of value through the pipeline.

Because of the severe impact production issues can have on delivery efficiency and delivered value, teams must ensure that they can proactively detect problems and quickly recover from them at all times. Also, the ability to rapidly resolve issues found when verifying and monitoring increases the ability to continuously deploy with confidence.

In fact, fast recovery is among the most reliable leading indicators of high DevOps maturity, as measured by Mean Time to Restore (MTTR). [5] Recovery also appears in CALMR as one of SAFe’s five core DevOps principles.

The goal of respond and recover is to identify potential issues before they turn into incidents and to prevent them from impacting business operations. This requires the ability to detect difficulties before end users do, quickly identify root causes, and restore services with well-rehearsed procedures. In contrast, making hasty, reactive changes directly to production systems—‘just to keep the lights on’—invites configuration drift, unverified changes, and long-term risk.

Six skills support this sub-dimension:

  • Proactive detection – a practice for proactively creating faults in the solution to identify potential problems and situations before they occur
  • Cross-team collaboration – a mindset of cooperation across the Value Stream to identify and solve problems as they arise
  • Session replay – the ability to replay end-user sessions to research incidents and identify problems
  • Rollback and fix forward – the ability to both rollback a solution quickly to a previous environment, or to fix a problem quickly through the pipeline without the need to rollback
  • Immutable infrastructure – never change the elements of the production environment itself but instead, flow the changes through the Continuous Delivery Pipeline
  • Version control – environments should be maintained under version control in order to rollback quickly

When teams have demonstrated that features have been deployed successfully to production and that the necessary monitoring and recovery capabilities are in place to track and manage ongoing value to the business, they have completed the continuous deployment stage of the Continuous Delivery Pipeline. In turn, this gives the enterprise the ability to release whenever warranted.


Learn More

[1] Kim, Gene, et al. The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. IT Revolution Press, 2013.

[2] Kim, Gene and Jez Humble, Patrick Debois, John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press. Kindle Edition.

[3] Humble, Jez and David Farley. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, 2010.

[4] Gregory, Janet and Lisa Crispin. More Agile Testing: Learning Journeys for the Whole Team. Addison-Wesley Signature Series (Cohn). Pearson Education. Kindle Edition.

[5] 2017 State of DevOps Report https://puppet.com/resources/whitepaper/state-of-devops-report

 

 

Last update: 24 August 2018