Logging demo with OTEL Collector, CloudWatch and Grafana

Shakir - May 11 - - Dev Community

Hello ๐Ÿ‘‹, in this post, we would be using the OTEL Demo App with OpenTelemetry Collector and export the logs from the microservices in the app to AWS CloudWatch.

Let's get started!!! We can create a separate namespace in the kubernetes cluster with kubectl create ns otelcol-logs-demo.

And then create a secret that holds the AWS credentials.



kubectl create secret generic aws-credentials \
    --from-literal=AWS_ACCESS_KEY_ID=<access-key-id> \
    --from-literal=AWS_SECRET_ACCESS_KEY=<aws-secret-access-key> \
    -n otelcol-logs-demo


Enter fullscreen mode Exit fullscreen mode

We would be deploying open telemetry demo app, otel collector as well as Grafana with a single helm chart. The chart comes with other components too, we could disable those that we do not need for this lab. Our helm values would like below.



$ cat values.yaml 
opentelemetry-collector:
  config:
    exporters:
      awscloudwatchlogs:
        log_group_name: otel-logs-demo
        log_stream_name: otel-demo-app
        region: ap-south-2
    service:
      pipelines:
        logs:
          exporters:
            - awscloudwatchlogs
        metrics: {}
        traces: {}
  extraEnvsFrom:
    - secretRef:
        name: aws-credentials
prometheus:
  enabled: false
jaeger:
  enabled: false
opensearch:
  enabled: false


Enter fullscreen mode Exit fullscreen mode

In the values above, we have the cloudwatchlogs exporter section where we specify the region, log group and stream names. We are only setting up logs in the pipeline as we are only dealing with that for this demo. We are then setting the AWS credentials via env vars so that the collector could authenticate and send logs to cloud watch. Then, finally we are disabling certain applications which we do not need for this exercise.

Let's install the chart.



helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install my-otel-demo open-telemetry/opentelemetry-demo -n otelcol-logs-demo -f values.yaml


Enter fullscreen mode Exit fullscreen mode

Head over to CloudWatch logs in the mentioned region and you should be able to see the log group. And, inside the log group we should have the stream.

We should be able to view the logs, for ex. from cartservice as shown in the screenshot below.
Logs in console

We can now go to Grafana cloud and add the cloud watch datasource with access credentials.
CloudWatch datasource in Grafana

And should now be able to see the logs in Grafana for our log group, on the explore tab.
Logs in Grafana

As a bonus let's try adding a panel that shows the no. of messages per severity.

Add a panel with query as shown in the screenshot below.
Query in panel for logs

Go to the transform data tab and apply a transformation to parse the severity_text from the json message.
JSON transformation

And then do a group by transformation to calculate the no. of messages per severity.
Group by transformation

Can choose a visualization type such as pie chart like below.
Pie chart showing log count by severity

The options I have set in the panel are as follows:



Title: Logs by severity
Value options > Show > All values
Legend > Legend values: Value, Percent


Enter fullscreen mode Exit fullscreen mode

Ok that's it for this post, we saw how to get logs from the otel demo app via open collector to AWS CloudWatch Logs and explored it on Grafana with a sample visualization. We just need to delete the namespace with kubectl delete ns otelcol-logs-demo so that logs are not sent to AWS cloudwatch. Thanks for reading !!!

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .