Every organization today has a team of engineers working hard to keep their systems up and running, and they must create processes for handling incidents that inevitably arise. Even if you have the best developers and site reliability engineers (SREs), managing and addressing these incidents will take time and effort. However, as with everything, incident response is much easier to manage if you have the right tools and strategy. A mix of best-of-the-breed continuous integration and continuous delivery (CI/CD) platforms integrated with the best incident management tool is what every organization needs.

What is Incident Management?

Incident management is a process of identifying, investigating, and resolving incidents affecting a service's availability. The incident management process begins with an outage or event that triggers an alert from monitoring tools or from manual detection by operations staff. The incident manager then assesses the impact on customers and the business before deciding how to respond based on the severity of the problem, who is impacted, and how long it will take to resolve it. Incident management is a key component of DevOps. It encompasses the steps that are taken to respond to incidents and reduce their impact on business operations.

What is PagerDuty?

PagerDuty is a cloud-based IT incident management tool that helps notify and coordinate responses to incidents across an organization. Here are some of the features and advantages of PagerDuty:

• Helps determine the urgency of an incident and assigns it to the engineers accordingly, so they can take appropriate action
• Provides live notifications when there is an update on an incident
• Allows users to assign tasks and track their progress
• Enables notifications for specific incidents

Tutorial: How to Send PagerDuty Alerts Using Harness

In this tutorial, we’ll demonstrate how to send PagerDuty alerts in case incidents are detected during a pipeline execution.

When a developer commits a code to a source control management (SCM) system, like GitHub, the CI/CD tool will build and test the code for any misconfigurations, bugs, or vulnerabilities. The code then gets deployed to a target environment, such as Kubernetes. Here, we will configure the alert notification so that when an unsuccessful deployment happens, the pipeline will notify PagerDuty to alert the developers and the team responsible for taking further actions.

Harness is the heart of this whole setup, from cloning the code to continuous integration to deploying it to the target environment and alerting the responsible people.

Prerequisites:

• Free Harness cloud account
• PagerDuty free account
• A sample application to connect with the Harness platform (you can fork my repo and use the same)
• A Kubernetes cluster from any cloud provider to deploy our application (you can use Minikube or Kind to create a single node cluster)

Note: Built-in notifications are a capability of the Harness platform, and as such, they can be applied to any stage types.

Harness Setup

If you are new to Harness, you need to have a free Harness account.

Next, sign in to your Harness account and select the CD module. Set up a simple pipeline to deploy your application. We will use a Kubernetes cluster to deploy our sample application.

Add the required connectors for your project, such as your GitHub repository (we use a sample application; you can fork and use it) and a Delegate. The Harness Delegate is a service you run in your local network or VPC to connect all of your artifact, infrastructure, collaboration, verification, and other providers with Harness.

Once you set up your project and the pipeline to deploy your application, it is time to set up notifications. On the right-hand side of the dashboard, you will see a notify icon. Click the icon and set up your incident notification strategy.

As we discussed above, I’ll show you how you can integrate the PagerDuty notifications.

Add notifications.

The setup will ask you what event you would like to receive notifications about. I am simply setting up notifications for any deployment, i.e., for any successful deployment of the application/service, notifications will be generated.

Next, we will pick PagerDuty as the notification method.

Once you pick PagerDuty as your notifications tool, you are required to provide the PagerDuty Key in the next step.

To get the PagerDuty key, you need to access your PagerDuty account.

Setting Up PagerDuty

Create an account on PagerDuty and provide a service name to continue.

In the next step, you’ll set up integrations. Go to the “All” tab and search for Harness, and select Harness as shown below.

The next step will ask you how you would like to get notified. I put my email address. See the example email below sent by PagerDuty.

Next, go to Service and click Service Directory to see your services configured. You should see the added service by you.

Click on Manage and then on View Integration to see your key.

The integration key is what we need. Copy it and save it.

Adding PagerDuty to Harness

Back in the Harness app, add the key to your PagerDuty notification tab.

Save the changes and apply them. Then, go to your Harness dashboard, and run the pipeline.

On the successful deployment of our application, we should now see the event notification on PagerDuty and receive an email response.

Similarly, you can customize your workflow and events, assign people to look into the incidents, and more. The PagerDuty integration can help your support and engineering team keep track of what is happening to resolve issues as quickly as possible.

Incident Management is Critical for CI/CD

Once you deploy your service, you need to know what is happening and report different events to your team. Having an incident management tool is an important addition to any CI/CD platform to ensure everything is going as expected. PagerDuty is an excellent example of one of the many notification tools that can be integrated with Harness, taking your incident management to the next level.