After providing developers with a platform to automate deployment, scaling and operations of application containers across the host cluster, Kubernetes has really cemented its way to become a de-facto standard when it comes to container-orchestration.

With Kubernetes, you get the flexibility to roll out updates more efficiently, at whichever time and also divert the traffic to the new version if needed. It provides you options to scale your application up and down based on your requirement, you get to choose how your application interacts with other applications or the real world.

In this article, we'll take a look at:

What is a CRD
How do they work
How you can create one!
How to delete CRDs
Examples of real-life CRDs in use within Litmus

By the end of this article, you'd have a thorough understanding of what are CRs, how CRDs work and how Litmus is also implementing them for their daily use-case.

Resource and Custom Resource

Kubernetes makes it possible to provide a plethora of features like this by leveraging what is known as a resource. A resource is an endpoint in k8s API that allows you to store an API object of any kind. But what if developers need a custom object or resource based on their specific requirements? This is where a Custom Resource comes into the picture.

As a part of the Kubernetes 1.7 release, they introduced the concept of Custom Resources to extend the capabilities by adding any kind of API object useful for your application. Custom Resource Definition(CRD) is what you use to define a Custom Resource. This is a powerful way to extend Kubernetes capabilities beyond the default installation.

How does a Custom Resource Work?

As mentioned above, pretty much everything we create in Kubernetes is a resource. A Pod, a Service, a Deployment or a Secret, it's all at its very basic form a resource. These resources are often monitored by Controllers that are responsible for taking the information in the resource and turning it into something useful.

Let's take the example of what happens when you create a Pod resource. Whenever you modify the Pod resource i.e you add, change or update the resource, there is a controller that gets notified about the change. Whenever the controller sees a Pod resource being added or updated, it looks up the information in the resource and then takes a decision on what to do next. In the case of a new Pod, it'll figure out what node is responsible for running the workload and assign it to that node. This assignment is then picked up by another controller on that node, which in turn makes sure to spin up the required containers.

Custom resources are defined using the custom resource definition and that's why the abbreviation CRD and not CR. Once CRD is created, the Kubernetes API server creates a RESTful path to each version specified in the CRD. Based on the scope defined on the CRD file, the path may either be accessible globally by the entire cluster or only to a specific project. A CR, once defined by CRD, is then stored in etcd cluster. Kubernetes API server takes care of the resource lifecycle and replication. It acts similar to the native resources and when a project is deleted, all the custom resources and native resources get deleted as well.

Creating a CRD

As a pre-requisite, you'd need Kubernetes 1.7 or above and a kubectl tool installed to be able to create CRDs. Just like a lot of other files in the Kubernetes world CRDs can also be created with the help of YAML files. A CRD YAML file consists of several fields, some of the important once are listed down:

metadata.name: Name of the CRD that you're creating
spec.group: Name of the group the CRD object belongs to.
spec.versions: It holds the different versions of custom resources that will be created.
spec.names: Used to define how to describe your Custom Resource. You can add plural, singular and short names of your CR here which can later be handy while managing it through kubectl.
spec.names.kind: Defines the kind of CR that can be created using the CRD. Like Deployment,CronJob,CronTab.
spec.scope: Defines what is the scope of the CR. It can either be set to 'clustered' or 'namespaced'. It is set to 'namespaced' by default.

The manifest below shows an example CRD ourcrd.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crdname.demo.example.com
spec:
  scope: Namespaced
  group: demos.example.com
  names:
    kind: Foo
    singular: crdname
    plural: crdnames
    shortNames:
    - cn
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          description: "A foo object"
          properties:
            spec:
              type: object
              properties:
                value1:
                  type: string
                  default: "Default value"
                value2:
                  type: integer

We can create the above CRD simply using

kubectl create -f ourcrd.yaml

After creating the CRD through kubectl, the endpoint URL is generated that can be used to create and manage custom objects. This endpoint might take a couple of seconds to get created.

You might have this question, "Okay! That cool, but how/where would I possibly use this?", well, here is an example of how you can use it.

Let's create a manifest using the kind we created with the CRD. For example, we have my-kind.yaml

apiVersion: "demo.example.com/v1"
kind: Foo [The same kind we declared in the CRD]
metadata:
  name: demo-app
spec:
  value1: "Hello Developers"
  value2: 92

Here we are using the same kind we defined in our CRD, we then define the fields we want our kind object to have. Like in this example, we gave the fields as value1 and value2(As provided in the CRD) for our app. In real examples, these fields would be something like image, URI, etc.

Now if we run kubectl create -f my-kind.yaml our application can consume the data we created with the manifest above. To see what is going on we can run kubectl get cn -o yaml, we can see detailed info on what we just created.

cn is the short name we gave to our CRD above

Deleting a CRD

To delete the CRD and resources we created, simply run kubectl delete just like with any other resources. When you delete the CRD, the server uninstalls the RESTful endpoint that was created with the CRD. With the deletion of a CRD, all the custom resources created using it are also deleted and cannot be retrieved.

Real Life example of CRDs

In this last section of our journey let's actually take a look at how real projects utilize these concepts to expand upon their requirements and actually implement them.

I'm going to take the example of Litmus as the project whose CRDs we are going to look into.

Litmus is a framework for practising chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus is very easy to use; you can also set up a very quick demo environment to install and run Litmus experiments.

Few of the important CRDs in Litmus are as follows:

ChaosEngine
ChaosExperiment
ChaosResult

ChaosEngine: ChaosEngine CR is created for a given application and is tagged with appLabel. This CR ties one or more ChaosExperiments to an application.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: chaosengines.litmuschaos.io
spec:
  group: litmuschaos.io
  names:
    kind: ChaosEngine
    listKind: ChaosEngineList
    plural: chaosengines
    singular: chaosengine
  scope: Namespaced
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          type: object
          properties:
            monitoring:
              type: boolean
            jobCleanUpPolicy:
              type: string
              pattern: ^(delete|retain)$
              # alternate ways to do this in case of complex pattern matches
              #oneOf:
              #  - pattern: '^delete$'
              #  - pattern: '^retain$'
            annotationCheck:
              type: string
              pattern: ^(true|false)$
            appinfo:
              type: object
              properties:
                appkind:
                  type: string
                  pattern: ^(^$|deployment|statefulset|daemonset|deploymentconfig|rollout)$
                applabel:
                  type: string
                appns:
                  type: string
            auxiliaryAppInfo:
              type: string
            engineState:
              type: string
              pattern: ^(active|stop)$
            chaosServiceAccount:
              type: string
            components:
              ...
            experiments:
              type: array
              items:
                 ...
        status:
          type: object
  version: v1alpha1
  versions:
    - name: v1alpha1
      served: true
      storage: true

ChaosExperiment: ChaosExperiment CR is created to hold and operate the details of actual chaos on an application. It defines the type of experiment and key parameters of the experiment.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: chaosexperiments.litmuschaos.io
spec:
  group: litmuschaos.io
  names:
    kind: ChaosExperiment
    listKind: ChaosExperimentList
    plural: chaosexperiments
    singular: chaosexperiment
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          type: object
          properties:
            ...
        status:
          type: object
  version: v1alpha1
  versions:
    - name: v1alpha1
      served: true
      storage: true

ChaosResult: ChaosResult CR is created by the operator after an experiment is run. One ChaosResult CR is maintained per ChaosEngine. The ChaosResult CR is useful in making sense of a given ChaosExperiment. This CR is used for generating chaos analytics which can be extremely useful — for example when certain components are upgraded between the chaos experiments, and the results need to be easily compared

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: chaosresults.litmuschaos.io
spec:
  group: litmuschaos.io
  names:
    kind: ChaosResult
    listKind: ChaosResultList
    plural: chaosresults
    singular: chaosresult
  scope: Namespaced
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          type: object
        status:
          type: object
  version: v1alpha1
  versions:
    - name: v1alpha1
      served: true
      storage: true

To take a wider look at the CRs visit here. You'd find the CRDs of Cluster Workflows, Cron Workflows, etc

Conclusion

Voila, you have successfully learned about the concept of CRDs and how to create them. You can of course extend these and play around to create something out of your own. We welcome everyone to comment and let us know about what/how we can improve these configurations to achieve more! Every suggestion is appreciated.

A custom resource definition adds to the incredible features Kubernetes already offers to its users. CRD helps extend Kubernetes’ features helps make it an even more universal tool for container orchestration. You can use custom resources to add your own resources that help with your specific requirements. And, you can use these resources like any native Kubernetes service and leverage all the features Kubernetes has to offer like security, RBAC, API services, and CLI. You can also use dynamic registration to have the custom resources appear and disappear while the cluster is running.

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)

Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus
Submit a pull request if you identify any necessary changes.

Don't forget to share these resources with someone who you think might benefit from them. Peace out. ✌🏼

Extend your Kubernetes APIs with CRDs