Michael LevanAt the bottom-most layer of Kubernetes is an API. Any time you interact with Kubernetes, you’re interacting with an API. Whether it’s creating Pods, Deployments, Services, or simply running kubectl get pods to see what Pods are running or not running, you’re interacting with an API.
As an engineer, you have the ability to not only use that API, but create APIs, or extend the current Kubernetes API for functionality that may not already exist to fit your needs.
In this blog post, you’ll learn how to not only think about the various methods of extending the Kubernetes API, but the different tools available for building one.
What’s a CRD
A Custom Resource Definition (CRD) gives you the ability to create a new type/object in Kubernetes. For example, I can create an object called MikesType and within MikesType, I’m able to create a new resource. The resource can have a specific version along with what properties are available within the spec.
An example of a CRD is below.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: MikesType
spec:
group: mikestype.example.com
# list of Kubernetes versions supported by this CustomResourceDefinition
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
Luckily, the method of creating a CRD is with the CRD resource itself. It’s essentially extending Kubernetes with Kubernetes. When you submit the CRD to the Kubernetes cluster you’re running on, you will immediately have the new object/kind available for use.
CRD’s don’t ensure that the current state is the desired state, which is where Controllers come into play.
What’s a Controller
Unlike a CRD that extends the existing Kubernetes API (extending Kubernetes with Kubernetes), a Controller is to ensure that the object/kind you’re creating has the desired state expected. Let’s take Replicas for example. If you have a Kubernetes Deployment that has two replicas (two Pods running), the ReplicaSet Controller ensures that the Deployment always has (to the best of its ability) two Pods running. The Controller confirms that the current state is the desired state.
One thing to keep in mind with a Controller is that to ensure the current state is the desired state, there’s a loop that runs. That means if for whatever reason the current state cannot get to the desired state, the loop will keep running and trying on infinitely. An example of this would be Pod scheduling. If the Scheduler is trying to schedule the second Pod to run, but the Worker Nodes don’t have enough resources available (memory/CPU) to run the Pod, the ReplicaSet Controller will keep sending the request to create the second Pod to the Scheduler. Why? Because the Controllers job is to ensure that the current state is the desired state regardless of if the cluster can actually schedule the Pod or not.
What’s An Operator
An Operator is a mix between a Controller and a CRD. It’s a purpose-built Controller that extends the Kubernetes API for specific functionality.
As an example, the Prometheus Operator is built to ensure that you can not only use declarative methods to manage Prometheus within Kubernetes but to ensure that the current state of the cluster when it comes to Prometheus on Kubernetes is the desired state.
Operators give you the ability to:
- Represent the state of the system ensuring that the current state is the desired state.
- Create custom resources/objects in Kubernetes.
The entire goal of an Operator is to add a feature to Kubernetes that isn’t already there. Taking the Prometheus example again - Prometheus isn’t built into Kubernetes, which means the functionality has to be created and then put into Kubernetes.
💡 Side note: This is one of the greatest things about Kubernetes. The way that it’s built is to do exactly what we’re talking about here - to extend itself. That’s why there are so many plugin formats like CSI’s, CNI’s, etc… Kubernetes is meant to be extended and built to your liking.
Building An Operator
There are several options available which include:
- client-go Kubernetes library
- Kubebuilder (write in Go)
- Metacontroller
- Operator Framework
- Other options for Java, Python, and Rust (less popular, but they do work - https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
Let’s break down a few of them.
Kubebuilder
Kubebuilder automates a lot of the template away, leaving you to focus on the actual resource building. It leverages a collection of tools in the controller-runtime project. The project is included in the source code generated by the Kubebuilder CLI. It helps by taking away the heavy lifting of the Controller interacting with the Kubernetes API. It also helps with setting up shared caches and clients to provide proper interaction with the API server. controller-runtime also supports HA by providing leader election (like what Etcd uses) without you having to build it in.
From an observability perspective, Kubebuilder has built-in libraries for writing logs and it also exposes an endpoint for Prometheus.
When you’re building an Operator, one thing you’d usually have to focus on is building the CRD spec. Because this process can be incredibly lengthy and cumbersome, it’s definitely a huge bonus that Kubebuilder automatically generates one for you.
Kubebuilder, outside of generating the CRD, can also generate RBAC and sample resource manifests. It’ll then install the CRD to the cluster, builder/publish images for the operator, and run the Controller.
Kubebuilder is part of the Kubernetes SIG, so it does have a fair amount of backing compared to other tools.
https://github.com/kubernetes-sigs/kubebuilder
client-go
Client-go has no real benefits in comparison to projects like Kubebuilder. The only “benefit” would be if you don’t like the code that’s generated from Kubebuilder and you have a compelling reason to make it 100% custom. Most organizations do not have said compelling reason.
Metacontroller
Metacontroller is a great option if you don’t want to write the Controller in Go. One downside is that Metacontroller does not create the CRD’s for you. However, it is a great method of building Operators if you don’t have Go experience or if you don’t want to learn Go.
https://github.com/metacontroller/metacontroller
Operator Framework
The Operator Framework was built by Red Hat and is now under the CNCF umbrella as an Incubation Project. It works like Kubebuilder in the sense that it it uses the controller-runtime, provides a CLI, and generates boilterplate code for projects so you can focus on the actual resource building vs the templated code. It allows you to use Go, Helm or Ansible to manage Operators, which is quite interesting as it gives you a programmatic option, a declarative option, and a Configuration Management option. Operator Framework Includes the Operator Lifecycle Manager, which is essentially an Operator for Operators. It abstractions the installation and upgrades of Operators.
