In the last post, we saw how to create a Kubernetes operator using operator-sdk. As that text was quite long, I decided to write this second post to focus on the application's testing part.

Using the operator-sdk CLI to create the operator skeleton makes an initial structure for us to write your tests. For this purpose, we will use some components:

envtest: a Go library that configures an instance of etcd and the Kubernetes API, simulating the main functionalities of a Kubernetes cluster for testing purposes
Ginkgo: a testing framework based on the concept of ‌"Behavior Driven Development" (BDD)
Gomega: is a test assertion library, a vital dependency on Ginkgo.

The CLI created the file ‌controllers/suite_test.go, which contains the basic structure of the Ginkgo test suite and the envtest initialization. We need to add the code that will initialize our controller so that we can write the tests. In the following diff, you can see the change made.

We need to install the dependencies to run the tests. To do this, we will use the commands:



make envtest
./bin/setup-envtest use --bin-dir ./bin/
export PATH=$PATH:bin/k8s/1.28.0-darwin-arm64/

The first command will install the binary setup-envtest, the second downloads the executables to our project directory, and the third adds the new files to the operating system's PATH.

The next step is to write the test. The operator-sdk documentation recommends creating a kind_controller_test.go file within the "controllers" directory. In our case, application_controller_test.go. Below, you can see the file's basic structure. In the following topics, we will create each of the tests.



package controllers

import (
    "context"
    "time"

    minettodevv1alpha1 "github.com/eminetto/k8s-operator-talk/api/v1alpha1"
    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/types"
    "k8s.io/apimachinery/pkg/util/intstr"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)

// Define utility constants for object names and testing timeouts/durations and intervals.
const (
    ApplicationName      = "test-app"
    ApplicationNamespace = "test-app"

    timeout  = time.Second * 10
    duration = time.Second * 10
    interval = time.Millisecond * 250
)

var _ = Describe("Application controller", func() {
    Context("When creating an Application", func() {
        It("Should create a Deployment and a Service", func() {
        })
    })

    Context("When updating an Application", func() {
        It("Should update the Deployment", func() {
        })
    })

    Context("When deleting an Application", func() {
        It("Should delete the Deployment and Service", func() {
        })
    })
})

Application creation test

The first test we will complete is to check whether, when creating an Application, our controller also builds a Deployment and a Service. To do this, we add the following code:



Context("When creating an Application", func() {
        It("Should create a Deployment and a Service", func() {
            ctx := context.Background()

            // we need to create a namespace in the cluster
            ns := corev1.Namespace{
                ObjectMeta: v1.ObjectMeta{Name: ApplicationNamespace},
            }
            Expect(k8sClient.Create(ctx, &ns)).Should(Succeed())

            // we define an Application
            app := minettodevv1alpha1.Application{
                TypeMeta: v1.TypeMeta{
                    Kind:       "Application",
                    APIVersion: "v1alpha1",
                },
                ObjectMeta: v1.ObjectMeta{
                    Name:      ApplicationName,
                    Namespace: ApplicationNamespace,
                },
                Spec: minettodevv1alpha1.ApplicationSpec{
                    Image:    "nginx:latest",
                    Replicas: 1,
                    Port:     80,
                },
            }
            // we added the finalizer, as described in the previous post
            controllerutil.AddFinalizer(&app, finalizer)
            // we guarantee that the creation was error-free
            Expect(k8sClient.Create(ctx, &app)).Should(Succeed())
            // we guarantee that the finalizer was created successfully
            Expect(controllerutil.ContainsFinalizer(&app, finalizer)).Should(BeTrue())

            // let's now check if the deployment was created successfully
            deplName := types.NamespacedName{Name: app.ObjectMeta.Name + "-deployment", Namespace: app.ObjectMeta.Name}
            createdDepl := &appsv1.Deployment{}

            // due to the asynchronous nature of Kubernetes, we will
            // make use of Ginkgo's Eventually function. It will 
            // execute the function according to the interval value, 
            // until the timeout has ended, or the result is true
            Eventually(func() bool {
                err := k8sClient.Get(ctx, deplName, createdDepl)
                if err != nil {
                    return false
                }
                return true
            }, timeout, interval).Should(BeTrue())

            // let's check if the Deployment data was created as expected
            Expect(createdDepl.Spec.Template.Spec.Containers[0].Image).Should(Equal(app.Spec.Image))
            // the Application must be the Owner of the Deployment
            Expect(createdDepl.ObjectMeta.OwnerReferences[0].Name).Should(Equal(app.Name))

            // let's do the same with the Service, ensuring that the controller created it as expected
            srvName := types.NamespacedName{Name: app.ObjectMeta.Name + "-service", Namespace: app.ObjectMeta.Name}
            createdSrv := &corev1.Service{}

            Eventually(func() bool {
                err := k8sClient.Get(ctx, srvName, createdSrv)
                if err != nil {
                    return false
                }
                return true
            }, timeout, interval).Should(BeTrue())
            Expect(createdSrv.Spec.Ports[0].TargetPort).Should(Equal(intstr.FromInt(int(app.Spec.Port))))
            Expect(createdDepl.ObjectMeta.OwnerReferences[0].Name).Should(Equal(app.Name))
        })
    })

I added comments to the code to describe the details.

Application update test

The subsequent test checks whether, when changing an Application, the controller reflects the modification in the other objects:



Context("When updating an Application", func() {

        It("Should update the Deployment", func() {

            ctx := context.Background()

        <span class="c">// let's first retrieve the Application in the cluster</span>
        <span class="n">appName</span> <span class="o">:=</span> <span class="n">types</span><span class="o">.</span><span class="n">NamespacedName</span><span class="p">{</span><span class="n">Name</span><span class="o">:</span> <span class="n">ApplicationName</span><span class="p">,</span> <span class="n">Namespace</span><span class="o">:</span> <span class="n">ApplicationNamespace</span><span class="p">}</span>
        <span class="n">app</span> <span class="o">:=</span> <span class="n">minettodevv1alpha1</span><span class="o">.</span><span class="n">Application</span><span class="p">{}</span>
        <span class="n">Eventually</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">bool</span> <span class="p">{</span>
            <span class="n">err</span> <span class="o">:=</span> <span class="n">k8sClient</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">appName</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">app</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
                <span class="k">return</span> <span class="no">false</span>
            <span class="p">}</span>
            <span class="k">return</span> <span class="no">true</span>
        <span class="p">},</span> <span class="n">timeout</span><span class="p">,</span> <span class="n">interval</span><span class="p">)</span><span class="o">.</span><span class="n">Should</span><span class="p">(</span><span class="n">BeTrue</span><span class="p">())</span>

        <span class="c">// let's retrieve the Deployment to ensure that the data is the same as the Application</span>
        <span class="n">deplName</span> <span class="o">:=</span> <span class="n">types</span><span class="o">.</span><span class="n">NamespacedName</span><span class="p">{</span><span class="n">Name</span><span class="o">:</span> <span class="n">app</span><span class="o">.</span><span class="n">ObjectMeta</span><span class="o">.</span><span class="n">Name</span> <span class="o">+</span> <span class="s">"-deployment"</span><span class="p">,</span> <span class="n">Namespace</span><span class="o">:</span> <span class="n">app</span><span class="o">.</span><span class="n">ObjectMeta</span><span class="o">.</span><span class="n">Name</span><span class="p">}</span>
        <span class="n">createdDepl</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="n">appsv1</span><span class="o">.</span><span class="n">Deployment</span><span class="p">{}</span>

        <span class="n">Eventually</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">bool</span> <span class="p">{</span>
            <span class="n">err</span> <span class="o">:=</span> <span class="n">k8sClient</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">deplName</span><span class="p">,</span> <span class="n">createdDepl</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
                <span class="k">return</span> <span class="no">false</span>
            <span class="p">}</span>
            <span class="k">return</span> <span class="no">true</span>
        <span class="p">},</span> <span class="n">timeout</span><span class="p">,</span> <span class="n">interval</span><span class="p">)</span><span class="o">.</span><span class="n">Should</span><span class="p">(</span><span class="n">BeTrue</span><span class="p">())</span>
        <span class="n">Expect</span><span class="p">(</span><span class="n">createdDepl</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Template</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Containers</span><span class="p">[</span><span class="m">0</span><span class="p">]</span><span class="o">.</span><span class="n">Image</span><span class="p">)</span><span class="o">.</span><span class="n">Should</span><span class="p">(</span><span class="n">Equal</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Image</span><span class="p">))</span>

        <span class="c">// let's change the Application</span>
        <span class="n">app</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Image</span> <span class="o">=</span> <span class="s">"caddy:latest"</span>
        <span class="n">Expect</span><span class="p">(</span><span class="n">k8sClient</span><span class="o">.</span><span class="n">Update</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">app</span><span class="p">))</span><span class="o">.</span><span class="n">Should</span><span class="p">(</span><span class="n">Succeed</span><span class="p">())</span>

        <span class="c">// let's check if the change in the Application was reflected in the Deployment</span>
        <span class="n">Eventually</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">bool</span> <span class="p">{</span>
            <span class="n">err</span> <span class="o">:=</span> <span class="n">k8sClient</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="n">deplName</span><span class="p">,</span> <span class="n">createdDepl</span><span class="p">)</span>
            <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
                <span class="k">return</span> <span class="no">false</span>
            <span class="p">}</span>
            <span class="k">if</span> <span class="n">createdDepl</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Template</span><span class="o">.</span><span class="n">Spec</span><span class="o">.</span><span class="n">Containers</span><span class="p">[</span><span class="m">0</span><span class="p">]</span><span class="o">.</span><span class="n">Image</span> <span class="o">==</span> <span class="s">"caddy:latest"</span> <span class="p">{</span>
                <span class="k">return</span> <span class="no">true</span>
            <span class="p">}</span>
            <span class="k">return</span> <span class="no">false</span>
        <span class="p">},</span> <span class="n">timeout</span><span class="p">,</span> <span class="n">interval</span><span class="p">)</span><span class="o">.</span><span class="n">Should</span><span class="p">(</span><span class="n">BeTrue</span><span class="p">())</span>
    <span class="p">})</span>
<span class="p">})</span>

Testing the deletion of an Application



Context("When deleting an Application", func() {

        It("Should delete the Deployment and Service", func() {

            appName := types.NamespacedName{Name: ApplicationName, Namespace: ApplicationNamespace}

            // checks whether the deletion was successful

            Eventually(func() error {

                a := &minettodevv1alpha1.Application{}

                k8sClient.Get(context.Background(), appName, a)

                return k8sClient.Delete(context.Background(), a)

            }, timeout, interval).Should(Succeed())

        <span class="c">// ensures that the Application no longer exists in the cluster</span>
        <span class="c">// this test is not really necessary as the Delete happened successfully</span>
        <span class="c">// I kept this test here for educational purposes only.</span>
        <span class="n">Eventually</span><span class="p">(</span><span class="k">func</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span>
            <span class="n">a</span> <span class="o">:=</span> <span class="o">&amp;</span><span class="n">minettodevv1alpha1</span><span class="o">.</span><span class="n">Application</span><span class="p">{}</span>
            <span class="k">return</span> <span class="n">k8sClient</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">Background</span><span class="p">(),</span> <span class="n">appName</span><span class="p">,</span> <span class="n">a</span><span class="p">)</span>
        <span class="p">},</span> <span class="n">timeout</span><span class="p">,</span> <span class="n">interval</span><span class="p">)</span><span class="o">.</span><span class="n">ShouldNot</span><span class="p">(</span><span class="n">Succeed</span><span class="p">())</span>

        <span class="c">// according to this documentation: https://book.kubebuilder.io/reference/envtest.html#testing-considerations</span>
        <span class="c">// we cannot test the cluster's garbage collection to ensure that the Deployment and Service created were removed, </span>
        <span class="c">// but in the first test we check the ownership, so they will be removed as expected in a real cluster</span>
    <span class="p">})</span>
<span class="p">})</span>

Conclusions

Once again, it is possible to see how operator-sdk facilitates the operator development process by creating a structure to make it easy to test our controller.

Using envtest is also very useful as it allows us to test the functionality of a Kubernetes cluster without the need to install one, which is very important in CI/CD environments.

Another interesting point is the use of Ginkgo, which makes the tests readable and easy to understand. This project was my first time using Ginkgo, and I liked the results. I should add it to my toolbox for future projects.

I hope this post serves as an introduction to operator testing. You can find links with more advanced topics and examples in the official documentation, and I recommend reading them as the next steps.

You can find the code in this post on GitHub.

Originally published at https://eltonminetto.dev on October 5, 2023

Writing tests for a Kubernetes Operator

Application creation test

Application update test

Testing the deletion of an Application

Conclusions