Recently, I've gone into detail on stacks and stages. I've also examined the importance of dev stacks for both teams and individual developers. Building on these topics, I wanted to talk today about approvals.

How do you promote changes to your stacks to production? More importantly, how do you gate promotions to ensure quality code? I'll look at the two major approaches to approvals - manual vs. automatic - and when and how to use each approach.

What is a Change Management Approval Process?

Traditionally, nothing strikes more fear in the heart of a dev team than pushing a change to production. Change promotion is usually an "all hands on deck" affair. Engineers and support personnel often stand at the ready, testing lives sites and monitoring dashboards for the slightest hint of trouble.

What can go wrong when pushing a change?

• A code change that wasn't thoroughly tested or reviewed can break on release.
• A code change that worked in dev might not work in production.
• A configuration change could break a production server or not be distributed to all instances in a cluster.
• A new part of your cloud infrastructure could fail to deploy correctly.
• ...and any number of other things that keep developers awake at night.

The question isn't "What can go wrong?" during a deployment. It's more like, "What can't go wrong?"

Because of this, software teams don't just shove a change into production and hope for the best. Most teams have some sort of change management approval process.

As I discussed previously, a deployment pipeline consists of a number of stages. Each stage - dev, test, staging, prod - is used to widen a change's availability and vet its quality. A change management approval process sets guidelines for when a change can flow from one stage to the next.

Types of Approvals

Traditionally, there are two types of approvals. Often, both types are used at different stages of the development process.

Manual Approvals

With a manual approval, a change requires some sort of human intervention to progress to the next stage. Often, this takes the form of a code review or buddy test, in which another member of your team reviews your changes before approving them. Once approved, the change migrates to the next stage.

A manual approval is also a good way to await feedback from stakeholders and customers. For example, you may make changes or a new feature available in a staging or demo environment that internal stakeholders and other teams can access. Once the changes have passed all tests and have secured stakeholder approval, you can approve and push them into production.

Manual approval doesn't mean that your release pipeline contains zero automation. You will likely still have steps in your deployment pipeline where you're running unit tests, smoke tests, service health checks, and other automated quality checks.

Automated Approvals

With an automated approval, a change migrates to the next stage if it passes a set of automated checks. These can include but are not limited to unit tests, service health checks, and security checks.

Automated approvals are typical in earlier stages of a release pipeline - e.g., moving from dev to test, or test to stage. They're harder to achieve in production, as they require a high degree of automated testing and verification to ensure users don't get broken bits. Automated delivery into production is often referred to as continuous delivery.

Typically, an automated approval into production will use some sort of phased release strategy. For example, you may deploy code changes to a single server (a canary). You would then test/monitor the results before deploying to all machines in a fleet. Or you may do a rolling deployment in which you deploy new code to a small percentage of your servers or serverless endpoints. If the change doesn't produce any errors (HTTP server errors, virtual machine connectivity issues, etc.), the system continues the promotion process. If there are errors, it rolls back the changes and stops the rollout.

Implementing Manual Approvals on a Pipeline in AWS

Most pipeline technologies provide some way to switch easily between manual and automated approvals.

For example, AWS CodePipeline structures a pipeline in a series of stages. Each step consists of a series of actions. In AWS, you can add a Manual Approval action to a stage.

The manual approval action will stop pipeline execution until someone approves it. AWS sends approval requests to an Amazon SNS (Simple Notification Service) topic. This means you can send the request to one or multiple potential reviewers. You can also configure the message to include a URL link. This is helpful if your team uses a code review software system like Review Board.

TinyStacks Makes Approvals Easy

At TinyStacks, our goal is to make DevOps easy. Our simplified pipeline creation tools will flow approvals automatically from stage to stage. Adding a manual approval is as simple as clicking a checkbox! Your teammates can then easily view and approve the migration to the next stage from the TinyStacks dashboard. Contact us today to see how TinyStacks can simplify your journey to DevOps maturity!

Article by Jay Allen