Honeycomb + Tracetest: Observability-driven Development

Adnan Rahić - May 23 '23 - - Dev Community

Tracetest now works with Honeycomb! With Honeycomb, you can find patterns and outliers of how users experience your code in complex and unpredictable environments. Integrating Tracetest will accelerate your observability-driven development even more by allowing you to build tests in minutes based on your observability data that fully test your distributed system before pushing new code into production.

Check out this hands-on example of how Tracetest works with Honeycomb, or check out the recipe in the docs!

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683812243/Blogposts/tracetest-honeycomb/screely-1683812237868_kqlakv.png

What is Honeycomb?

Honeycomb is an observability platform for distributed services. It helps you find your most perplexing application issues.

Honeycomb is designed for high-performance engineering. It enables you to quickly interpret the billions of rows of data required to fully represent the user experience in your complex, unpredictable systems. Utilizing Honeycomb leads to better uptime, higher-quality and faster user experiences, more time for innovation, and ultimately better business outcomes - all hallmarks of high-performance engineering teams.

Honeycomb began with and is a proponent of distributed traces as the most important observability signal.

What is Tracetest?

Tracetest is an open-source project, part of the CNCF landscape. It allows you to quickly build integration and end-to-end tests, powered by your distributed traces.

Tracetest uses your existing distributed traces to power trace-based testing with assertions against your trace data at every point of the request transaction. You only need to point Tracetest to your existing trace data source or send traces to Tracetest directly!

Tracetest makes it possible to:

  • Define tests and assertions against every single microservice that a trace goes through.
  • Work with your existing distributed tracing solution, allowing you to build tests based on your already instrumented system.
  • Define multiple transaction triggers, such as a GET against an API endpoint, a GRPC request, etc.
  • Define assertions against both the response and trace data, ensuring both your response and the underlying processes worked correctly, quickly, and without errors.
  • Save and run the tests manually or via CI build jobs with the Tracetest CLI.

Tracetest + Honeycomb Integration

Tracetest now integrates with Honeycomb, providing powerful observability capabilities. You can also leverage your distributed traces to run trace-based tests with Tracetest.

If you already have OpenTelemetry instrumentation configured in your code and are using OpenTelemetry Collector with Honeycomb as a trace data store for observability, adding Tracetest to your infrastructure can enable you to write detailed trace-based tests using your existing tracing infrastructure.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683814508/Blogposts/tracetest-honeycomb/Screenshot_2023-05-11_at_14.41.05_egj1wi.png

Why is the Tracetest integration with Honeycomb important?

When running integration tests, you have no way of knowing precisely at which point an HTTP transaction goes wrong in a network of microservices! With tracing enabled, Tracetest can run tests with assertions against existing trace data throughout every service in the entire transaction. You can utilize these tests as part of your CI/CD process to ensure system functionality and to catch regressions.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683812243/Blogposts/tracetest-honeycomb/screely-1683812237868_kqlakv.png

You can enhance your testing process by using Tracetest to create tests and Honeycomb to analyze test results over time. With Honeycomb, you can track test runs and correlate corresponding services over time, giving you insights into system performance and highlighting the impact of changes on performance.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683805623/Blogposts/tracetest-honeycomb/screely-1683805616751_wdqcf7.png

Issues identified when exploring trace data via Honeycomb can be used as good candidates for new tests or assertions within Tracetest. This creates a cycle of continuous improvement.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683805259/Blogposts/tracetest-honeycomb/screely-1683805252244_nj8npd.png

Try Tracetest with Honeycomb

First, you need to have a Honeycomb account. Create your free account here. Next, you need to have your distributed application instrumented with OpenTelemetry and configured to send trace data to Honeycomb. Check out the official docs for more on getting started with Honeycomb and OpenTelemetry. Once you have your system sending trace data to Honeycomb, you are ready to set up Tracetest!

Install Tracetest

Tracetest is open-source and easy to install. Start by installing the Tracetest CLI:

brew install kubeshop/tracetest/tracetest
Enter fullscreen mode Exit fullscreen mode

Note: Check out the download page for more info.

From here, follow the official documentation to install the Tracetest server. Once the server is installed, open Tracetest Web UI in the browser and follow the instructions for connecting the OpenTelemetry Collector for Tracetest and Honeycomb.

The Collector is the recommended way to send OpenTelemetry data to an observability back-end, including Honeycomb. It is a highly-configurable binary that allows you to ingest, process, and export OpenTelemetry data.

Enabling the Tracetest/Honeycomb integration is as simple as selecting Honeycomb as your trace data store and add dedicated pipelines that send traces to both Tracetest and Honeycomb in your OpenTelemetry Collector configuration.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683804137/Blogposts/tracetest-honeycomb/screely-1683804100406_bhrkdz.png

Copy this OpenTelemetry Collector configuration and paste it into your configuration file.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 100ms

exporters:
  logging:
    logLevel: debug
  # OTLP for Tracetest
  otlp/tracetest:
    endpoint: tracetest:4317 # Send traces to Tracetest.
    tls:
      insecure: true
  # OTLP for Honeycomb
  otlp/honeycomb:
    endpoint: "api.honeycomb.io:443"
    headers:
      "x-honeycomb-team": "YOUR_API_KEY"

service:
  pipelines:
    traces/tracetest:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tracetest]
    traces/honeycomb:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, otlp/honeycomb]
Enter fullscreen mode Exit fullscreen mode

Next, edit the config to include your Honeycomb API key.

Please see the official Honeycomb docs for more info on configuring Honeycomb to ingest data from the OpenTelemetry Collector.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683805856/Blogposts/tracetest-honeycomb/screely-1683805850730_d3yvan.png

Create a Trace-based Test in Tracetest

For this example, I’ll use the official example app for Tracetest and Honeycomb.

To quickly access the example, you can run the following:

git clone https://github.com/kubeshop/tracetest.git
cd tracetest/examples/tracetest-honeycomb/
docker compose up --build
Enter fullscreen mode Exit fullscreen mode

Make sure to add your API key as shown in the section above.

To create a test in Tracetest, start by clicking Create > Create New Test > HTTP Request > Next > Add a name for your test > Next > The URL field should be http://app:8080 > Create and Run.

This will trigger the test and display a distributed trace in the Trace tab to run assertions against.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683811984/Blogposts/tracetest-honeycomb/screely-1683811978117_xexcvm.png

Proceed to add a test spec to assert all database queries return within 500 ms. Click the Test tab and proceed to click the Add Test Spec button.

In the span selector, make sure to add this selector:

span[tracetest.span.type="http" name="GET /" http.target="/" http.method="GET"]
Enter fullscreen mode Exit fullscreen mode

In the assertion field add:

attr:http.status_code = 200
attr:tracetest.span.duration < 500ms
Enter fullscreen mode Exit fullscreen mode

Save the test spec and publish the test.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683812356/Blogposts/tracetest-honeycomb/screely-1683812351444_lgn0cb.png

The http span is returning in more than 500ms and will be labeled in red.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683812243/Blogposts/tracetest-honeycomb/screely-1683812237868_kqlakv.png

This is an example of a trace-based test that can assert against every single part of an HTTP transaction, including external API calls.

However, Tracetest cannot give you a historical overview of all test runs. Let's introduce how Honeycomb makes it possible.

Monitor Trace-based Tests over Time with Honeycomb

Because you are using two pipelines in the OpenTelemetry collector, all traces your distributed system generates will be stored in Honeycomb. Additionally, if you configure the Tracetest server with Telemetry, you will see the traces the Tracetest server generates as well.

This will give you insight into every test run! Here’s a board you can create to track P99 for the response time of a Tracetest test run shown over time and compared in 2-hour intervals.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683813393/Blogposts/tracetest-honeycomb/screely-1683813387670_keknov.png

By changing the dataset, you can get the same insight into the sample app. But, the key feature would be drilling down into the specific trace for troubleshooting.

https://res.cloudinary.com/djwdcmwdz/image/upload/v1683813827/Blogposts/tracetest-honeycomb/screely-1683813821587_jxe64c.png

What's next?

Tracetest provides detailed distributed trace data, allowing you to gain insight into the behavior of your application at a granular level and ultimately create assertions against this data to write bullet-proof tests. Honeycomb provides observability and granular insight over time, where you can monitor your distributed applications and track Tracetest test run behavior over time!

Would you like to learn more about Tracetest and what it brings to the table? Check the docs and try it out today by downloading it today!

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.

Image
Maps cellular origins along with early on progression associated with the leukemia disease in Down syndrome.
Image
Throughout vitro plus vivo metabolic rate of the anabolic-androgenic anabolic steroid oxandrolone within the moose.
Image
Cyanescent Gyroporus (Gyroporaceae, Boletales) from Tiongkok.
Image
The retinal ipRGC-preoptic enterprise mediates the actual serious effect of mild on rest.
Image
Routine PET-CT tests supply early and accurate recurrence diagnosis inside asymptomatic period IIB-III most cancers sufferers.
Image
Techniques along with Challenges associated with House Reliable Spend Administration within Woldia Area, Northeastern Ethiopia.
Image
Neuro-COVID Calls for Extensive Work-up.
Image
New evidence of tyrosine neurotoxicity: give attention to mitochondrial disorder.
Image
Offering Therapy to be able to People Coping with COVID-19: Any Scoping Evaluation.
Image
Thermoperiodism Syncs Breakthrough within the Alfalfa Leafcutting Bee (Hymenoptera: Megachilidae).
Image
Genotype-related breathing development inside Duchenne Carved Dystrophy : multicentre worldwide review.
Image
Are generally Doctors Complicit within Vitamin K Insufficiency Bleeding?
Image
Face Signs and Social Actions throughout Multimodal Face-to-Face Conversation.
Image
The consequences associated with Lengthy Timeframe Spaceflight on Sensorimotor Management and Knowledge.
Image
A way regarding Calculating the Surface Free of charge Vitality involving Topical Semi-solid Serving Forms.
Image
The function involving Neurod Family genes within Mental faculties Development, Operate, and also Ailment.
Image
Applications of Natural Language Processing

nlp, ai
Image
Tests affect rules styles of drinking just before after consuming initiation using enviromentally friendly momentary assessment.
Image
Single-stage long-segment tracheal transplantation.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .