Tracetest is excited to announce v0.8!

Haven't heard of Tracetest yet? Tracetest is an open-source project that enables you to write integration, system, and E2E tests against your distributed or microservice-based application. It uses the observability you enabled by implementing a distributed tracing solution to empower a totally new, modern way of testing - Trace-Based Testing.

It only takes 5 minutes to configure Tracetest to be able to send a triggering transaction to your distributed app and tell it how to get traces from your existing OpenTelemetry based tracing solution. No need to change your code or existing tracing solution!

Testing Microservices Requires Environments, Variables and Test Chaining

Testing of distributed applications happens across many environments and can involve several tests chained together to verify a complete flow through a system. The previous sentence contains an obvious statement to anyone who has written tests across multiple microservices.

When developing an entirely new approach to testing distributed applications from scratch, you start small with a core set of features and expand. This is what we did with Tracetest - we enabled it to run trace-based tests and released the initial capability this summer. We then built tests for Tracetest using Tracetest. And it was… hard!

As you can imagine, without a native capability to pass variables from one test to another, we had to rely on bash scripts, duct tape, and various forms of magic to string tests together.

Sebastian, one of our back-end engineers created the first system tests against Tracetest using Tracetest. He referred to the work of chaining without a native chaining capability as 'bull excrement'. Loved the trace-based tests, hated the duct tape.

We were not the only ones to notice the gap - Mark Watson filed a couple Github issues. One asked us to 'Provide a mechanism to make API calls in sequence' and another referenced the need for variables in the trigger of a test.

The team got busy, and we're excited to announce the v0.8 release of Tracetest! Let's discuss the major enhancements.

Test Environments and Environment Variables

Tracetest now allows you to define various environments, such as local, dev, test, QA or prod. Each environment can have variables associated with it. Here's an example:

You can then utilize both the environment and the variables when running tests. Define a test trigger using the variables with this syntax:
${env:variablename}

With this capability, you can have one test that you can run across multiple environments without altering it… one less roll of duct tape needed!

Chaining Trace-Based Tests Together

Adding variables into Tracetest formed the basis of chaining tests together and passing variables between them.

Variables can be defined in a test based on any of the attributes contained in any span.
You can use the new 'expressions' syntax to alter values stored from attributes. An example would be accessing a particular field from an attribute containing a JSON response with syntax such as: attr:http.response.body | json_path '$.name'
Tests can utilize variables defined in prior tests (or itself!)
Tests can be chained together as 'transactions'. A transaction is defined and then executed much like a test, but running one transaction results in all the chained tests executing in sequence.
A transaction fails if the response to the trigger of any of the tests fails.

Let's look at a scenario so we can understand how these concepts come together. In this scenario, we want to ensure that the 'import Pokemon' API works, and that importing one Pokemon actually increases the total number of Pokemon. We will end up having four tests chained together:

Let's build the four tests that will be involved in this chained transaction.

Building the Four Tests

First, we will get the total number of Pokemon from a 'List' test, and set an output variable. Second, we will do an import. Third, we will do another 'List' and make sure the number of Pokemon has increased. Lastly, we will delete the Pokemon to 'clean up'.

Videos, or animated gifs in this case, are worth several thousand words, so let's see a couple of examples. First, let's create a variable based on the number of Pokemon returned by the first 'List' api call. This value appears in our trace in the 'count pokeshop.pokemon' span:

You can see that we select the span and the attribute, create an output variable, name it, and add it as an output. Our plan is to use that variable, the POKEMON_COUNT, in the third test as part of our chained transaction to verify that the count has increased after importing one new Pokemon.

Next, we define the 'import' test. Rather than looking at how it was created in the UI, let's view the test definition for the completed test:

type: Test
spec:
 id: 6T0F1LOVR
 name: OSCAR - DEMO - Import
 description: Import a Pokemon
 trigger:
   type: http
   httpRequest:
     url: ${env:HTTP_HOST}/pokemon/import
     method: POST
     headers:
     - key: Content-Type
       value: application/json
     body: '{"id": ${env:IMPORTED_POKEMON_ID}}'
 specs:
 - selector: span[tracetest.span.type="http" name="HTTP GET pokeapi.pokemon" http.method="GET"]
   assertions:
   - attr:http.url contains env:POKESHOP_API
   - attr:http.route = "/pokemon/${env:IMPORTED_POKEMON_ID}"
   - attr:tracetest.span.duration < 1s
   - attr:http.response.body | json_path '$.name' = env:IMPORTED_POKEMON_NAME
 - selector: span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres"
     db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]
   assertions:
   - attr:db.result | json_path '$.name' = env:IMPORTED_POKEMON_NAME
 - selector: span[tracetest.span.type="database" name="pg.query:INSERT" db.system="postgresql"
     db.name="pokeshop" db.user="ashketchum"]
   assertions:
   - attr:tracetest.selected_spans.count = 1
 outputs:
 - name: POKEMON_DB_ID
   selector: span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres"
     db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]
   value: attr:db.result | json_path '$.id'

Notice that we are using variables in several places in this test. The trigger uses them so this one test can run in multiple environments with this line:
url: ${env:HTTP_HOST}/pokemon/import

We are also setting a variable, IMPORTED_POKEMON_NAME, in this output statement:

outputs:
  - name: POKEMON_DB_ID
    selector: span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres"
      db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]
    value: attr:db.result | json_path '$.id'

We verify that the value for the Pokemon name in our database insert is correct by comparing it to the just created variable:
‍

- selector: span[tracetest.span.type="database" name="create pokeshop.pokemon" db.system="postgres"
      db.name="pokeshop" db.user="ashketchum" db.operation="create" db.sql.table="pokemon"]
  assertions:
    - attr:db.result | json_path '$.name' = env:IMPORTED_POKEMON_NAME

Now, let's use the variable captured in our first test, POKEMON_COUNT, in the third test, creating a test spec that checks if the IMPORT in the second test increases the number of Pokemon. This assertion uses another capability added in this release, the ability to put an expression in your assertions or when you define a output variable (see expressions):

As you can see, we look at the DB result in the 'count pokeshop.pokemon' span and compare it to the original count original Pokemon count + 1.

Finally, we build a 'Delete' test to remove the added Pokemon to clean up. Now, it is time to chain these 4 tests together in a transaction.

Chaining Tests to Enable Integration Tests

Let's see how we chain the four tests together. To do this, we return to the Tracetest home page and create a transaction, selecting each of the four tests:

As you can see, we added each of the four tests to the transaction. You can adjust the order of the tests to achieve the logic flow you desire. When you save the transaction, it runs and shows you the execution results.

Tying it All Together… Literally!

This release makes Tracetest much more useful testing complex chained scenarios in multiple environments. Ready to try Tracetest in your environment? Check out our easy download, then create some tests and chain them together.

Any issues can be raised in Github, and you can communicate directly with the team in Discord. Open-source projects such as Tracetest depend on the input and support of the community and we need YOUR involvement! If you like our direction and what you are seeing from Tracetest - give us a star on Github.
‍
‍

Tracetest v0.8: Chain Tests & Environment Vars