In this blog, I will be talking about setting up a GitLab remote template for running a Chaos Experiment in the GitLab CI environment with LitmusChaos. Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practising chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus also has Github Chaos Actions and GitLab remote templates for the members who are using litmus experiments as part of their CI pipelines.

Pre-requisites:

• Kubernetes 1.11 or later in your CI workflow

Brief Introduction to GitLab Remote Template

GitLab is a DevOps lifecycle tool which is extremely famous for its continuous integration and deployment pipeline features in the DevOps world nowadays which are configured using a YAML file called .gitlab-ci.yml within each project. The .gitlab-ci.yaml defines the structure and determines what to execute using GitLab Runner .It deals with what decisions to be taken for when specific conditions are encountered. Like when a process is passed or failed. In GitLab we can include an external yaml file using the include keyword these external files are called templates.

Benefits of using templates:

• This helps to break down the CI/CD configuration into multiple files and increases readability for long configuration files.

• It’s also possible to have template files stored in a central repository(like litmus-templates) and projects include their configuration files. This helps avoid duplicated configuration, for example, global default variables for all projects.

Different Types of Templates

The addition of a template in GitLab is a very simple job. We just need to add the include keyword that requires the external YAML file to have the extensions .yml or .yaml, otherwise the external file is not included. include supports the following inclusion methods:

• local: It Includes a file from the local project repository and referenced using full paths relative to the root directory. we can use it like this:

include:
  - local: '/templates/.gitlab-chaos-template.yml'

• file: We can include files from another private project under the same GitLab instance using file along with include. It can be added with:

include:
  - project: 'mayadata-io/gitlab-remote-template'
    file: '/templates/pod-delete-template.yml'

• template: It used to include the pre-built GitLab template. It can be used like:

# File sourced from GitLab's template collection
include:
  - template: Auto-DevOps.gitlab-ci.yml

• remote: It can be used to include a file from a different location, using HTTP/HTTPS, referenced by using the full URL.This makes the remote template very popular among all and it is widely used in many projects. The remote file must be publicly accessible through a simple GET request as authentication schemas in the remote URL are not supported. This brings LitmusChaos Remote Templates into the picture which is used to induce chaos on an application. The yaml templates are stored in a github repository and can be used as an external template.

Example:

include:
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/pod-delete-template.yml'

Setup LitmusChaos Remote Template in your CI workflow

If you are using/willing to use litmus experiments as part of your CI pipelines and using GitLab as your CI platform then you are at the right place. LitmusChaos introduced GitLab remote templates to integrate the chaos experiments in your CI workflow. It helps to manage your configuration file and provide a simple way to perform the chaos testing on your application. We need to follow some simple steps to setup GitLab Remote Template in your CI workflow:

1. Setup a cluster: Before we begin make sure that we have a kubernetes cluster setup in our CI workflow this is more of a pre-requisite step. If we don’t have any cloud cluster like GKE or AWS we can set up a light weight simple clusters like KinD cluster.

2. Deploy an application: Once we setup a cluster we need to have an application on which we want to perform the chaos testing. If we don’t have the application running in a pod don’t worry we can set that in these two simple steps.

i. Create an Image of the application: We need to Dockerize the application for this we need to have a Dockerfile which can be used to create the application image. Let us suppose that we have a Dockerfile at location /build from root. So we will build the image with that.

ii. Run a pod with the application container: Now as we have our application image in the Docker registry. We can deploy an application pod with that image. You can refer to a sample pod manifest that can be used to create an application pod from here replace the nginx image with your image and other attributes. Now apply the pod manifest and wait for it to come in Running state.

kubectl apply -f app.yaml

3. Select a Chaos template: Now select a chaos template from the list of templates available in the chaos template repository. Based on the experiment you want to perform, take the raw link of the template. Make sure before running any experiment we need to install litmus which can be done using the install litmus template present in the same list. A chaos template looks like:

---
variables:
  APP_NS: default
  APP_KIND: deployment 
  APP_LABEL: APPLICATION_LABEL
  TARGET_CONTAINER: TARGET_CONTAINER_NAME
  TOTAL_CHAOS_DURATION: "30"
  CHAOS_INTERVAL: "10"
  FORCE: "false"
  EXPERIMENT_IMAGE: litmuschaos/go-runner
  EXPERIMENT_IMAGE_TAG: 1.7.0

.pod_delete_template:
  image: 
    name: mayadata-io/chaos-ci-lib:ci
    entrypoint: ["./pod-delete"]
  script: 
    - echo "Running Pod Delete Experiment"

Get the raw link by clicking on the raw button.

And copy the raw link

4. Add template in .gitlab-ci-yml: Now we need to add the chaos template in the GitLab configuration file for which we need to add a remote template URL with the include keyword.

include:
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/node-cpu-hog-template.yml'

stages:
  - chaosInject

Inject Node CPU Hog:
  stage: chaosInject
  extends: .node_cpu_hog_template
  before_script:
  variables:
    APP_NS: default
    APP_LABEL: "run=nginx"
    APP_KIND: deployment
    TARGET_CONTAINER: nginx

Use the link which we copied in step 3 in include:remote.

5. Add the Chaos Experiment: Once we are done with setting up the chaos infra we can add the chaos experiment which we want to induce on our application in the similar way for example the addition of pod delete chaos has been added in the below code.

.gitlab-ci.yml

include:
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/install-litmus-template.yml'
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/node-cpu-hog-template.yml'
  - remote: 'https://raw.githubusercontent.com/mayadata-io/gitlab-remote-templates/master/templates/uninstall-litmus-template.yml'

stages:
  - appDeploy
  - chaosInfraSetup
  - chaosInject
  - chaosInfraCleanup

Deploy to Staging:
  stage: appDeploy
  script:
    - kubectl run nginx --image=nginx --restart=Always

Install LitmusChaos:
  stage: chaosInfraSetup
  extends: .install_litmus_template
  before_script:
  variables:
    APP_NS: default
    EXPERIMENT_IMAGE: litmuschaos/go-runner
    EXPERIMENT_IMAGE_TAG: 1.7.0

Inject Node CPU Hog:
  stage: chaosInject
  extends: .node_cpu_hog_template
  before_script:
  variables:
    APP_NS: default
    APP_LABEL: "run=nginx"
    APP_KIND: deployment
    TARGET_CONTAINER: nginx

UnInstall LitmusChaos:
  stage: chaosInfraCleanup
  extends: .uninstall_litmus_template
  before_script:
  variables:
    APP_NS: default
    EXPERIMENT_IMAGE: litmuschaos/go-runner
    EXPERIMENT_IMAGE_TAG: 1.7.0

6. Trigger the pipeline: Once we are done with the set up we can trigger the pipeline and get the results under respective stages of the pipeline. In the same way we can add and perform any chaos experiment in the GitLab CI.

Conclusion

By using GitLab Remote template we can easily integrate LitmusChaos experiment with GitLab which helps to perform the chaos testing on an application in the CI workflow. We just need to make use of the include:remote feature of GitLab and select the desired templates from the litmuschaos remote templates. So, What are your opinions on running chaos in the CI workflow? If you’re in favor of this then does it seem to be helpful? DevOps experts are welcome to comment and suggest on this!

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)
Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus
Submit a pull request if you identify any necessary changes.