Y.MatsudaIntroduction
The OpenTelemetry ecosystem has grown significantly busier compared to a while ago. I've been contributing and writing blog posts about it for about two years now, and I'm hoping to participate in some synchronous events in the future.
Now, when it comes to instrumenting applications or writing libraries for instrumentation, how do you usually verify the local operation?
From my experience, many people run Jaeger or Grafana locally to check signals. Today, I’d like to introduce a new method: my custom tool "otel-tui," which allows you to view OpenTelemetry in your terminal!
https://github.com/ymtdzzz/otel-tui
Motivation
The Need for a "Just Right" Tool for Local Development
Based on my experience, here are some use cases for verifying signals locally:
- Ensuring traces are linked as intended
- For example, verifying implementation at the level of context
- Checking that attributes are set as expected
- Attributes specified by the vendor may not display correctly if missing
- Custom attributes may be specified
Currently, there are several ways to verify signals (mainly traces) locally:
- Run Jaeger (all-in-one with UI) or Grafana Tempo containers locally to send and view signals
- Use an exporter to output to standard output or files
However, these methods have limitations:
- Running Jaeger or Grafana Tempo containers locally is resource-intensive
- Production-grade features are often too much for local use (e.g., service maps)
- It’s challenging to view outputs in real-time
Using an exporter to output to standard output or files also has limitations:
- Low readability (e.g., grepping for Trace ID or parent Span ID to check trace linkage)
Given these challenges, I wanted a tool that could offer the best of both worlds—one that wasn’t too heavy for local use and provided readable outputs.
There're Similar Tools but...
There’s also "otel-desktop-viewer," which focuses on viewing OpenTelemetry signals locally and is very user-friendly.
However, with a significant volume of data over a long period, it can become sluggish in terms of query and screen rendering (depending on machine specs). This inspired me to create my own tool that could view data in real-time without opening a browser and maintain stable performance.
Features
Here's what the screen looks like:
Key features include:
-
Traces
- List view of service spans and search by service name
- Detailed attribute information
- Trace graph display (waterfall diagram)
-
Logs
- List view and search
- Jump from logs to corresponding traces
-
Performance
- Lightweight and real-time screen updates (asynchronous signal reception and rendering)
- No memory leaks even with a steady flow of data over a long period (data rotation)
TODOs
There are still many features missing:
- Flexible settings
- Screen update frequency
- Buffer size
- Metric collection and display
- Stability
- Occasional crashes after prolonged use
- Consistent keybindings
- UI improvements
- Navigation (e.g., back and forth between traces and log details)
- Text wrapping, waterfall diagram
- Debug logs
How to Use
To demonstrate usage, I’ll use signals from the "opentelemetry-demo."
Installing otel-tui
You can install it via Homebrew:
$ brew install ymtdzzz/tap/otel-tui
If Homebrew isn’t available, download the binary from the release page or build from source.
Verify installation with:
$ otel-tui -v
otel-tui version 0.1.2
$ otel-tui -h
Usage:
otel-tui [flags]
Flags:
--grpc int The port number on which we listen for OTLP grpc payloads (default 4317)
-h, --help help for otel-tui
--host string The host where we expose all endpoints (OTLP receivers and browser) (default "0.0.0.0")
--http int The port number on which we listen for OTLP http payloads (default 4318)
-v, --version version for otel-tui
Changing the Target of the Instrumented Application
Clone the "opentelemetry-demo" repository and configure it.
Modify Collector Target
The opentelemetry-demo consists of multiple microservices, all signals are aggregated in the OpenTelemetry Collector. Modify the collector's exporter target to point to otel-tui.
# src/otelcollector/otelcol-config-extras.yml
exporters:
otlp:
endpoint: host.docker.internal:4317
service:
pipelines:
traces:
exporters: [spanmetrics, otlp]
logs:
exporters: [otlp]
Allow Container to Host Port Communication (if not using Docker Desktop)
Set up host.docker.internal to be resolvable from within the container.
# docker-compose.minimal.yml
# OpenTelemetry Collector
otelcol:
image: ${COLLECTOR_CONTRIB_IMAGE}
container_name: otel-col
# ...
extra_hosts:
- "host.docker.internal:host-gateway"
Starting otel-tui and the Instrumented Application
Start them in separate tabs.
Exploring Data
After a short while, signals should start flowing into otel-tui.
You can switch between traces and logs using the Tab key. Navigate through panes using keys listed in the titles (e.g., Details(d)).
To filter the spans, use the search box (/ key) and search for something like "product."
Spans related to services with the prefix "product" are filtered. Details of the currently focused span are displayed in the Details pane. You can collapse the tree in the Details pane by pressing Enter.
Select a span in the Traces pane by pressing Enter to view the trace details.
Navigate the Trace Timeline using arrow keys. Details of the focused span appear in the Details pane. Related logs appear in the Logs pane.
Press Esc to return to the trace list, and Tab to switch to the logs view.
Search for a term and inspect the details. If a log has a linked trace (indicated by a link emoji 🔗), press Enter to jump to the trace details.
This tool helps verify if the instrumented application sends the intended information and if it’s correctly linked.
Conclusion
While there are still many rough edges and missing features, I find it quite user-friendly and enjoyable to use.
I’d appreciate any feedback via https://github.com/ymtdzzz/otel-tui/issues or https://twitter.com/ymtdzzz!
