Implementing Open-Source Monitoring and Observability In Kubernetes

Michael Levan - Jun 14 '23 - - Dev Community

When you begin to implement monitoring and observability solutions, you’ll be tasked with many questions, but one primary question is: “Do we go a paid route or an open-source route?”.

In many cases, the “homegrown” route is open-source and the “enterprise” route is the paid route. A paid route will consist of tools like Datadog, New Relic, and App Dynamics. The homegrown route will consist of various open-source tools that make monitoring and observability possible for your environment.

In this blog post, you’ll learn about the homegrown/open-source route.

Grafana

When it comes to real-time monitoring with graphs and alerting, Grafana is a great open-source tool that can monitor any workload you need. Because Grafana is open-source, there’s also the ability to create your own graphs. For example, there’s a graph that was created specifically for monitoring ArgoCD workloads.

You’re going to learn how to combine both Prometheus and Grafana in the next section, so don’t worry about deploying Grafana right now. However, if you decide at some point that you want to deploy Grafana without Prometheus, you can use the following code.

First, add the Helm Chart.



helm repo add grafana https://grafana.github.io/helm-charts
Enter fullscreen mode Exit fullscreen mode

Next, ensure that the Helm Chart is updated.



helm repo update
Enter fullscreen mode Exit fullscreen mode

Lastly, install the Helm Chart.



helm install -n grafana grafana grafana/grafana --create-namespace
Enter fullscreen mode Exit fullscreen mode

To access the dashboard, you can use the port-forward command.



kubectl port-forward -n grafana svc/grafana :80
Enter fullscreen mode Exit fullscreen mode

You should now see that the dashboard is up and running.

Image description

Prometheus

When you’re thinking about Metrics, Prometheus is one of the top open-source Metrics tools which falls under the observability category. Remember, Prometheus only handles Metrics, not the whole stack.

There are several ways to deploy Prometheus including with a standard Helm Chart like you saw in the Grafana section. However, there’s something called the kube-prometheus stack, which is a combination of Grafana and Prometheus. The really cool thing about kube-prometheus is that it also comes with best practice dashboards out of the box for any Kubernetes environment.

Essentially it’s packaging up what you’ll have to deploy when using Grafana and Prometheus anyways, so the best path forward would be to do the entire configuration in one shot. That’s what kube-prometheus does for you.

To install kube-prometheus, you can use a Helm Chart.

First, add the repo to your local Helm configuration.



helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Enter fullscreen mode Exit fullscreen mode

Next, ensure that the repo is updated.



helm repo update
Enter fullscreen mode Exit fullscreen mode

Install the package and put it into the monitoring Namespace.




helm install -n monitoring kube-prometheus prometheus-community/kube-prometheus-stack --create-namespace
Enter fullscreen mode Exit fullscreen mode

Once complete, you can run kubectl get all in the monitoring Namespace and you’ll see that there are several resources deployed.

Image description

To access the Grafana dashboard, run the following:



kubectl port-forward -n monitoring svc/kube-prometheus-grafana :80
Enter fullscreen mode Exit fullscreen mode

When you’re logging in for the first time, the following credentials can be used:

Username: admin

Password: prom-operator

After logging in, you can go to the Grafana Dashboards and see that there are several dashboards available out of the box.

Image description

Fluentd

When you’re thinking about how to specifically capture logs, there are several log aggregators available. Anything from paid tools to open-source to log capturing mechanisms in almost every cloud (at least the big ones). If you’d like to think about and implement an open-source solution, there’s Fluentd.

To install Fluentd, you can use a Helm Chart.



helm repo add fluent https://fluent.github.io/helm-charts

helm install fluentd fluent/fluentd
Enter fullscreen mode Exit fullscreen mode

One thing to say about Fluentd - although it’s absolutely great in terms of log capturing if you’re using a cloud platform like AWS or Azure and the logs are being captured anyways, it honestly may be a better solution to use what’s available in the cloud. The reason why is that there are a few hoops that you have to jump through to get the logs that are collected from Fluentd to a specific location, whether it’s S3 or Azure Files or another location to store data. After that, you have to parse through it.

You can take a look at the following links to help in the process of sending the logs from Fluentd to a specific location.

Tempo

As part of the observability stack, you’ll want tracing enabled, which is all about telling you the health of an application from an end-to-end perspective.

Tempo is part of the Grafana stack and you can install it with the same Helm Chart.

First, add the Grafana Helm chart if you don’t already have it.



helm repo add grafana https://grafana.github.io/helm-charts
Enter fullscreen mode Exit fullscreen mode

Next, install Tempo from the Grafana Helm repository and add it to the monitoring Namespace where the rest of the stack exists.



helm install tempo grafana/tempo-distributed -n monitoring
Enter fullscreen mode Exit fullscreen mode

You should see an output similar to the screenshot below.

Image description

If you run a kubectl get all -n monitoring, you’ll see the Tempo Pods running.



pod/tempo-compactor-5db447f8c9-wzv69                         1/1     Running   0          110s
pod/tempo-distributor-5c6b879865-dv5gp                       1/1     Running   0          110s
pod/tempo-ingester-0                                         1/1     Running   0          110s
pod/tempo-ingester-1                                         1/1     Running   0          110s
pod/tempo-ingester-2                                         1/1     Running   0          110s
pod/tempo-memcached-0                                        1/1     Running   0          110s
pod/tempo-querier-674455845b-gw94n                           1/1     Running   0          110s
pod/tempo-query-frontend-764dcf4699-wdc27                    1/1     Running   0          110s
Enter fullscreen mode Exit fullscreen mode

Loki

The last part of the observability stack is around logging. Logs are arguably the most important piece of observability. They are the make or break between understanding an application or system failure and guessing at problems as you troubleshoot.

Unfortunately, a lot of the time logs are exported somewhere and never looked at. If used properly, logs can be the best tool in your tool belt.

Loki is part of the Grafana stack and you can install it with the same Helm Chart.

First, add the Grafana Helm chart if you don’t already have it.



helm repo add grafana https://grafana.github.io/helm-charts
Enter fullscreen mode Exit fullscreen mode

Next, install Loki from the Grafana Helm repository and add it to the monitoring Namespace where the rest of the stack exists.



helm install loki grafana/loki -n monitoring
Enter fullscreen mode Exit fullscreen mode

You should see an output similar to the screenshot below.

Image description

If you run a kubectl get all -n monitoring, you’ll see the Loki Pods running.




pod/loki-backend-0                                           1/1     Running   0          2m50s

pod/loki-backend-1                                           1/1     Running   0          2m50s

pod/loki-backend-2                                           1/1     Running   0          2m49s

pod/loki-canary-26zxc                                        1/1     Running   0          2m50s

pod/loki-canary-rfx9n                                        1/1     Running   0          2m50s

pod/loki-canary-zhcmj                                        1/1     Running   0          2m50s

pod/loki-gateway-5d85cb5d7f-n9h2s                            1/1     Running   0          2m50s

pod/loki-grafana-agent-operator-7b7b8bd969-8hmfh             1/1     Running   0          2m50s

pod/loki-logs-fc4s8                                          2/2     Running   0          2m46s

pod/loki-logs-pdcn7                                          2/2     Running   0          2m46s

pod/loki-logs-sj4nl                                          2/2     Running   0          2m46s

pod/loki-read-5f7dc67977-69wgx                               1/1     Running   0          2m50s

pod/loki-read-5f7dc67977-hffbz                               1/1     Running   0          2m50s

pod/loki-read-5f7dc67977-ssgst                               1/1     Running   0          2m50s

pod/loki-write-0                                             1/1     Running   0          2m50s

pod/loki-write-1                                             1/1     Running   0          2m50s

pod/loki-write-2                                             1/1     Running   0          2m50s
Enter fullscreen mode Exit fullscreen mode




Wrapping Up

To put the tools in this blog post into categories:

  • Grafana == monitoring/graphs/visuals
  • Metrics == Prometheus
  • Logs == Fluentd and Loki (you don’t need to use both. Using just Loki with this stack is a valid choice).
  • Tracing == Tempo

With proper monitoring and observability, you should have just about everything you need to truly understand what’s happening underneath the hood in your environment.

Image
The Most Successful Asbestos Lawsuit Settlement Experts Have Been Doing 3 Things
Image
5 Injury Lawyer Projects For Every Budget
Image
langkah main slots online biar untung lagi
Image
Python for DevOps: A Comprehensive Guide from Beginner to Advanced

python, devops, beginners, programming
Image
The Most Advanced Guide To Compensation For Asbestos
Image
My journey of open-sourcing 2024

hacktoberfestchallenge, hacktoberfest, devchallenge, opensource
Image
Best Practices for Managing Users, Roles, and Permissions
Image
Deploy Custom Application in Kubernetes using Helm

helm, k8, learning, devops
Image
Daftar Judi Online Paling dipercaya serta Paling Memberikan keuntungan
Image
📚 2 Books Junior Engineers Should Read

productivity, career, learning, softwaredevelopment
Image
🚀 12 Dev Tools + Gen Z UX Secrets + CSS Animations (5-min read)

webdev, css, design, ux
Image
20 Resources To Make You More Effective At Exercise Bikes
Image
Project Blog : Intelligent Writing Assistant

webdev, programming, devops, api
Image
10 Unexpected Mental Health Tests NHS Tips
Image
Repair Bifold Door Top Pivot: It's Not As Difficult As You Think
Image
20 Myths About Ghost Immobiliser Installer: Busted
Image
7 Simple Changes That'll Make An Enormous Difference To Your Asbestos Injury Attorney
Image
How To Become A Prosperous Coffee Drip Machine Entrepreneur Even If You're Not Business-Savvy
Image
referensi blog judi slots online indonesia terbaik
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .