Ken HamricTest and Debug Critical End to End Flows with Playwright and OpenTelemetry
Today is a tale of two traces!
Playwright Traces and OpenTelemetry Traces are brought together to create one test to rule them all. Well actually, to test both your front end AND back end in one swing.
Using the power of Playwright to drive tests of your critical flows from the browser, this integration extends your Playwright tests to leverage OpenTelemetry and trace-based testing.
You can now:
- Test, in depth! Apply assertions against the entire front end application AND back end application.
- Troubleshoot failed tests, with traces. Both Playwright traces and OpenTelemetry-based distributed traces.
- Stop the blame game. With a view of the the entire flow, and data from both Playwright and distributed traces, quickly determine the cause of the failure, enabling you to easily identify the developer that can help finish the quest.
- Be the hero and give your developer all the data. Examine both Playwright traces and OpenTelemetry traces, all from one test. The devs will love, admire and respect you for it!
- Complete the quest with better observability! As tracing data from instrumentation is used to build trace-based tests, developers will want to add more insightful and meaningful instrumentation. See your ability to fight bugs in your production system enhanced when armed with better observability data.
If you’re eager to start, these instructions provide a full example, with an instrument Cloud Native app! Want to see some examples right away of Tracetest infused Playwright tests? Join the environment and look at the three Playwright tests, and tune in for a live workshop on February 15th!
https://www.youtube.com/watch?v=mmOAZxYdTYw
The Heroes
To start, lets identify the heroes involved in this saga.
Playwright is the fastest growing web testing framework. Playwright enables end-to-end (E2E) tests to be created by synthetically driving a headless browser session and enabling assertions to verify the system performs as expected. With Playwright traces, you can review a test after the script has run and identify exactly what occurred.
OpenTelemetry is the fastest growing Cloud Native Computing Foundation (CNCF) project. It standardizes the instrumentation and collection of traces, metrics, and logs from applications, and is supported by all the major observability projects, languages, and tools. One standard to rule them all!
Lastly, located in a quiet shire at the edge of the world is a newcomer to the scene, Tracetest. Unassuming, but able to wield the power of trace-base testing and OpenTelemetry traces to shine a light in the black box, revealing any hidden gremlins, orcs or nasty spiders (Shelob anybody?) lurking in the dark places.
The Quest: Test the Untestable - Complex, Cloud Native Systems
While capable of doing much good, Cloud Native systems have introduced a level of complexity heretofore not seen in the West. Or the East.
Microservices, created by different races, in different places. Elves, Dwarves, Men… or at least C#, Go, and React developers, working from afar, from office or from home. Separated into teams, rarely talking, with distrust of the other’s strange ways and weird coding practices. Can anybody bring these groups together?
The challenge falls to the pre-deployment team, who, like the hero Aragorn, a.k.a. Strider, go by several names. DevOps Engineer. SRE. Cloud, Automation, Infrastructure, or CI/CD Engineer. Their role is to bring together these pieces into a coherent, working whole, that is prepared and ready for the rigors of a production environment and the ultimate challenge - Customers! But what can they place their hopes in?
End-to-End Tests!
Yes - true end-to-end tests are the answer. Use Playwright to cover the critical flows in the cloak of actions and assertions, testing to make sure the interactions driven from the browser will return expected results. Catch the errors in your CI/CD flow before deploying. But is this enough?
No, for evil lurks in the back end also. The Go and C# developers have complex and mysterious ways.
While the API gateway may return a 200 to the browser, deep in the system, asynchronous gRPC processes may be timing out or returning error status codes. Systems may be using wrong versions of specifications, and processes may not be listening to the correct messages from your Kafka queue. Critical back-end flows, like sending the email on sign up or writing to the database fast enough may not be occurring.
All hidden from the benevolent view of your Playwright tests by a black box, containing other black boxes, with many differing technologies and arcane logic hidden within.
Flip the Light Switch on the Black Box(es)
Much like carrying a vial of the Light of Earendil's Star, embedding a trace-based test into your Playwright test allows you to see the surrounding dangers lurking in your system. But also to set assertions that verify the correct operation of the complete system. What does this magic look like in code?
import { test, expect } from "@playwright/test";
import Tracetest, { Types } from "@tracetest/playwright";
const { TRACETEST_API_TOKEN = "" } = process.env;
let tracetest: Types.TracetestPlaywright | undefined = undefined;
test.describe.configure({ mode: "serial" });
const definition = `
type: Test
spec:
id: UGxheXdyaWdodDogaW1wb3J0cyBhIHBva2Vtb24=
name: "Playwright: imports a pokemon"
trigger:
type: playwright
specs:
- selector: span[tracetest.span.type="http"] span[tracetest.span.type="http"]
name: "All HTTP Spans: Status code is 200"
assertions:
- attr:http.status_code = 200
- selector: span[tracetest.span.type="database"]
name: "All Database Spans: Processing time is less than 100ms"
assertions:
- attr:tracetest.span.duration < 2s
outputs:
- name: MY_OUTPUT
selector: span[tracetest.span.type="general" name="Tracetest trigger"]
value: attr:name
`;
test.beforeAll(async () => {
tracetest = await Tracetest({ apiToken: TRACETEST_API_TOKEN });
tracetest.setOptions({
"Playwright: imports a pokemon": {
definition,
},
});
});
test.beforeEach(async ({ page }, { title }) => {
await page.goto("/");
await tracetest?.capture(title, page);
});
// optional step to break the playwright script in case a Tracetest test fails
test.afterAll(async ({}, testInfo) => {
testInfo.setTimeout(60000);
await tracetest?.summary();
});
test("Playwright: creates a pokemon", async ({ page }) => {
expect(await page.getByText("Pokeshop")).toBeTruthy();
await page.click("text=Add");
await page.getByLabel("Name").fill("Charizard");
await page.getByLabel("Type").fill("Flying");
await page
.getByLabel("Image URL")
.fill("https://upload.wikimedia.org/wikipedia/en/1/1f/Pok%C3%A9mon_Charizard_art.png");
await page.getByRole("button", { name: "OK", exact: true }).click();
});
test("Playwright: imports a pokemon", async ({ page }) => {
expect(await page.getByText("Pokeshop")).toBeTruthy();
await page.click("text=Import");
await page.getByLabel("ID").fill(Math.floor(Math.random() * 101).toString());
await page.getByRole("button", { name: "OK", exact: true }).click();
});
test("Playwright: deletes a pokemon", async ({ page }) => {
await page.locator('[data-cy="pokemon-list"]');
await page.locator('[data-cy="pokemon-card"]').first().click();
await page.locator('[data-cy="pokemon-card"] [data-cy="delete-pokemon-button"]').first().click();
});
Track the Uruk-hai (or other Bugs) using Traces
In much the same way that Aragorn used footprints, a dropped elven brooch, and other traces to hunt the Uruk-hai across the stretches of Middle Earth, the entire team can benefit from the dual traces left as artifacts from these Tracetest enhanced Playwright tests.
Use the Playwright traces to see exactly what was occurring in the browser, step back in time to see the browser state between each action, and view a video of the entire flow.
Then, hop into the Tracetest test run. View the full distributed trace, showing the trace spans and attributes captured from both the browser and across your back-end services. See exactly where the trace-based test assertions failed, and inspect the services above and below this point.
With both types of traces, it will be much easier to test for failures across the entire system. Once a test fails, the wealth of data will enable you to identify the area in the system that is causing the problem and quickly assign the bug to the proper team. By providing that team with the same set of traces, they will easily be able to rectify the issue, improving the time to resolve and push a correction.
Shine a Light in the Black Box with OpenTelemetry & Tracetest
Much in the same way the pieces of the sword Andúril were forged together from the shards of Narsil to serve Aragorn, modern day heroes in DevOps, Observability, and SRE teams can combine OpenTelemetry, Playwright, and Tracetest to produce a tool to bring peace and harmony to Middle Earth… or at least to your distributed system.
https://www.youtube.com/watch?v=Wv38LNleUpE
Want to undertake this quest? Do you think you’re ready? You will need:
- To use Playwright for your front-end tests. Using Cypress instead? Check this out.
- Have observability, that hopefully works and plays well with the OpenTelemetry standard, configured for your back-end systems so you can produce distributed traces.
- Use the OpenTelemetry browser instrumentation in your front-end app.
- Sign up for an account on Tracetest so you can run trace-based tests.
- See the instructions on how to include Tracetest libraries in your Playwright tests.
These instructions provide a full example, with an instrument Cloud Native app, an OpenTelemetry instrumented React front end, and Playwright tests with Tracetest included.
Want to see some examples of these Tracetest infused Playwright tests? We are running these tests in our demo environment. Join the environment and look at the three Playwright tests.
Have questions? Join the discussion with the nerds (yes, we like LOTR…) that created this supernatural fusion of Playwright and OpenTelemetry in the Tracetest Slack channel!
