In the realm of agile development, the focus often lands on accelerating feature deployment. However, the capability to swiftly respond to critical incidents is just as crucial. Especially in testing, where the failure of key tests could significantly impact a release, staying updated on test statuses with each build is vital.

Modern applications and infrastructures require effective alerting and incident response systems to maintain high availability and operational awareness. PagerDuty stands out as a leading solution for incident management, providing teams with the tools needed to quickly address and resolve issues.

This tutorial post explores how integrating PagerDuty with Testkube can enhance your team's response to critical test failures, ensuring your development cycle remains both fast and reliable.

Alerting Via Webhooks

Throughout the evolution of technology, numerous methods have enabled systems to communicate seamlessly. From a variety of protocols to services, the goal has always been to facilitate the transfer of data across disparate systems efficiently.

Webhooks stand out as a powerful mechanism for system interaction. They serve as digital informants, alerting services to specific events. Essentially, a webhook endpoint acts as a listener for one system, ready to receive data transmitted by another.

In the context of Testkube, Webhooks are instrumental. They empower you to inform other services about test-related activities effortlessly. For example, setting up a Slack webhook with Testkube allows for the automatic dissemination of event notifications, bridging the gap between test execution and team communication.

Key Components of a Webhook Trigger in Testkube:

Events: These are the specific occurrences within Testkube you wish to monitor, such as "test-start" or "test-fail". Selecting relevant events allows for targeted notifications.
Source: Indicates the origin of the event, whether it's an individual test or a collection of tests within a TestSuite.
Destination: The webhook endpoint or the digital address to which Testkube sends the notification.

Advantages of Implementing Webhooks

Instant Alerts: Stay informed with real-time notifications about your tests' outcomes, enabling immediate action.
Automated Workflows: Trigger specific actions based on test results, such as sending detailed emails after a failure, streamlining your response mechanism.
Seamless Integration: Webhooks facilitate communication between Testkube and various tools or services like Slack for messaging or Grafana for monitoring, effortlessly fitting into your existing workflows.

By utilizing webhooks within Testkube, you can significantly enhance test observability and automation, making your testing processes more efficient and integrated.

Integrating Testkube With PagerDuty

PagerDuty is an incident response management and alerting platform for DevOps teams managing critical operations. It provides real-time alerts and escalates incidents to the appropriate personnel. By automating the incident response workflow, PagerDuty ensures that incidents are addressed promptly and efficiently, reducing downtime and improving overall system reliability.

Being the extensive tool that it is, there are a few components that we must be aware of for the purpose of this tutorial:

Services: These are applications/services within an organization that are monitored. These are connected to monitoring tools that trigger alerts based on predefined rules and thresholds.
**Integrations: **Integrations allow PagerDuty to seamlessly connect with other monitoring and collaboration tools. This allows it to collect data from multiple sources and manage them centrally.
Incidents: These are events that indicate something has gone wrong. When a monitoring tool detects an issue, it sends an alert to PagerDuty, which creates an incident that can be categorized and assigned to different personnel.

Apart from these, PagerDuty has advanced mechanisms that allow teams to send emails, calls, SMS, and push notifications to notify designated personnel about the incident. You can read more about PagerDuty and its features here.

Creating a service & integration

The first and foremost step is to create a service in PagerDuty. Creating a service in PagerDuty is fairly simple, and you can create one in a few minutes.

After your service is created, you need to create an integration. And integration basically allows PagerDuty to receive alerts from other services/tools. By default, PagerDuty provides support from multiple tools. However, in the case of Testkube, we’ll use their Events API V2, which will allow us to send custom events.

After you select this, it will give you the configurations that you will use to create the webhook in Testkube. It will share details like integration key, integration URL for change and alert events along with an example to use the API. Note that PagerDuty requires the input to come in a specific format.

With this, we have configured PagerDuty to receive alerts from Teskube. Let us now go ahead and create a Webhook in Testkube.

Creating a Testkube Webhook

Creating a webhook trigger in Testkube is straightforward. You can create Webhooks using the dashboard as well as the CLI. We’ll be using the dashboard to show you how it works.

Using Dashboard

We’ll set up a webhook for PagerDuty using the dashboard. This webhook will trigger on the test-end-failure event and send the data to the PagerDuty webhook endpoint that we got in the previous step (Integration URL)

Navigate to the webhooks section and “Create a new webhook.”

Provide a name for your webhook.
Choose the “Resource Identifier” as any test or executor on which you want the webhook to trigger. In this case, we will use a simple curl test. You can find the test in this repo - we’ve changed the HTTP code in the test so that it fails.
For “Triggered events”, choose end-test-failed, which basically means it will trigger when the chosen Pytest fails.

Click on Submit, and your webhook is ready. However, as mentioned earlier, PagerDuty requires input in a particular format as shown below:

curl --request 'POST' \
--url 'https://events.pagerduty.com/v2/enqueue' \
--header 'Content-Type: application/json' \
--data '{
  "payload": {
    "summary": "Test alert",
    "severity": "critical",
    "source": "Alert source",
    “custom_details” {“key”:”value”}
  },
  "routing_key": "routingkey",
  "event_action": "trigger"
}'

Hence, we will add a custom payload. To do that, click on your Webhook and navigate to the Action section. Here, add the following snippet under the “Custom Payload” section

{
"payload": {
    "summary": "{{ .TestExecution.TestName }} - {{ .TestExecution.ExecutionResult.Status }}" ,
    "severity": "critical",
    "source": "Testkube",
    "custom_details" : {
        "testName": "{{ .TestExecution.TestName }}",
        "testType": "{{ .TestExecution.TestType }}",
        "testStatus": "{{ .TestExecution.ExecutionResult.Status }}",
        "testEndtime": "{{ .TestExecution.EndTime }}",
        "testDuration": "{{ .TestExecution.Duration }}",
        "url": "https://app.testkube.io/organization/{{ index .Envs "TESTKUBE_CLOUD_ORG_ID" }}/environment/{{ index .Envs "TESTKUBE_CLOUD_ENV_ID" }}/dashboard/tests/{{ .TestExecution.TestName }}/executions/{{ .TestExecution.Id }}"
    }
  },
  "routing_key": "routingkey",
  "event_action": "trigger"
}

In the above payload, we are sending the following details:

Test Name
Test Type
Test Status
Test Endtime
Test Duration
URL to the specific test log

Click on save to save the configuration.

Testing the Webhook

Now that we have configured the webhook for PagerDuty, it’s time to test it. To do that, simply run the curl test.
You’ll see the webhook trigger when the test fails and send data to PagerDuty. Within PagerDuty, this will be shown as an incident based on the details we shared in the custom_payload.

By default, PagerDuty also sets up an email alert triggered when an incident is created.

And that’s how you can configure the Testkube webhook to send alerts to PagerDuty. Based on your team’s specific needs, you can configure both Webhook and Testkube.

Conclusion: Elevate Your Incident Response with Testkube and PagerDuty Integration

Harnessing the synergy between Testkube and PagerDuty through webhooks offers a formidable solution to bolster your incident management strategy. This integration not only simplifies tracking and addressing critical test failures but also fosters improved teamwork.

By utilizing Testkube's automated testing capabilities alongside PagerDuty's robust incident management system, you equip your team with the necessary resources to uphold service excellence and enhance customer satisfaction.

Discover the potential of Testkube and its versatile integrations. Embrace a system where proactive monitoring and seamless alerting transform your operational dynamics.

Connect with our vibrant community on Slack for insights, support, and networking with like-minded developers. Embark on a journey to redefine your approach to quality assurance and incident response.

Critical Test Based Alerting with PagerDuty and Testkube