FluxCD, a popular open-source tool for continuous delivery and GitOps, provides native support for Prometheus metrics to facilitate comprehensive monitoring of its components. This integration allows users to gain detailed insights into the performance and state of Flux controllers, enabling more effective management and troubleshooting. This blog post will delve into the technical aspects of FluxCD's integration with Prometheus, highlighting the key metrics, configurations, and tools involved.

Flux Controller Metrics

By default, Flux controllers export Prometheus metrics at port 8080 in the standard /metrics path. These metrics provide information about the inner workings of the controllers, including:

Reconciliation Duration Metrics:

  gotk_reconcile_duration_seconds_bucket{kind, name, namespace, le}
  gotk_reconcile_duration_seconds_sum{kind, name, namespace}
  gotk_reconcile_duration_seconds_count{kind, name, namespace}

Cache Event Metrics:

  gotk_cache_events_total{event_type, name, namespace}

Controller CPU and Memory Usage:

  process_cpu_seconds_total{namespace, pod}
  container_memory_working_set_bytes{namespace, pod}

Kubernetes API Usage:

  rest_client_requests_total{namespace, pod}

Controller Runtime:

  workqueue_longest_running_processor_seconds{name}
  controller_runtime_reconcile_total{controller, result}

Custom Resource Metrics

In addition to the default controller metrics, Flux also supports custom resource metrics using kube-state-metrics. These metrics can be configured to include custom labels, making them more informative for users who interact with Flux through custom resources. For example, metrics for GitRepositories can be labeled with department names or other relevant information.

Monitoring Setup

To set up monitoring for Flux, the fluxcd/flux2-monitoring-example repository provides a comprehensive example configuration. This repository includes configurations for deploying and configuring kube-prometheus-stack, which is used to monitor Flux. The monitoring setup involves the following components:

kube-state-metrics: Generates metrics about the state of Flux objects.
Prometheus Operator: Manages Prometheus clusters atop Kubernetes.
Prometheus: Collects and stores metrics from Flux controllers and kube-state-metrics.
Promtail: Collects logs from Flux controllers.
Loki: Stores logs collected by Promtail.
Grafana: Displays Flux control plane resource usage, reconciliation stats, and logs.

Example Configuration

The fluxcd/flux2-monitoring-example repository includes a monitoring/ directory with configurations for deploying kube-prometheus-stack and loki-stack. The monitoring/controllers/ directory contains the configurations for deploying these stacks.

Flux Custom Prometheus Metrics

Flux custom Prometheus metrics can be created using kube-state-metrics. These metrics are more informative for users who interact with Flux through custom resources. For example, metrics for GitRepositories can be labeled with department names or other relevant information.

Conclusion

FluxCD's integration with Prometheus provides a robust monitoring solution for its components. By leveraging Prometheus metrics and custom resource metrics using kube-state-metrics, users can gain detailed insights into the performance and state of Flux controllers. This integration is essential for effective management and troubleshooting in Platform Engineering environments.

FluxCD Integration with Prometheus for Monitoring