In this blog, I will be discussing how to build a multi-arch image for any desired docker architecture to run litmus chore components. For those who are new to litmus and wanted to explore more about chaos engineering, I would recommend to check out litmus blog. Now coming back to the topic we will be using docker buildx for building the images in the following sections.

Pre-requisites:

• Docker(version 19.03)

Understanding Multiarch Builds

Docker introduced the principle of multi-arch builds to support the "Build once, deploy anywhere" concept which helps to use ARM targets and reduce your bills such as AWS A1 and Raspberry Pis instances. But how do we produce them? and How do they work?
A multi-arch Docker image supports a different architecture behind the same imagetag. Let's compare the manifest of the multi-arch image with the image having a single arch.

Enable docker CLI if not already enabled using the following steps:

1. export DOCKER_CLI_EXPERIMENTAL=enabled
2. Add "experimental": "enabled", to ~/.docker/config.json (default location) at the beginning of the file and not at the end.

Image with single arch manifest

$ sudo docker manifest inspect --verbose litmuschaos/go-runner:1.9.0
{
    "Ref": "docker.io/litmuschaos/go-runner:1.9.0",
    "Descriptor": {
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "digest": "sha256:bc14ce592d7d600eb14c0b6dbfb0f6e8211f7795b5bc9def057cba6b3c994185",
        "size": 2002,
        "platform": {
            "architecture": "amd64",
            "os": "linux"
        }
    },
    "SchemaV2Manifest": {
        "schemaVersion": 2,
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "config": {
            "mediaType": "application/vnd.docker.container.image.v1+json",
            "size": 5502,
            "digest": "sha256:79fc16e2bbab6885589bd491c5ebf68c693239b8dbf13d45223b84e1a3aaa25b"
        },
        "layers": [ ... ]
    }
}

Multi arch image manifest

$ sudo docker manifest inspect --verbose litmuschaos/go-runner:1.10.0
[
    {
        "Ref": "docker.io/litmuschaos/go-runner:1.10.0@sha256:89717f4c87c7fc93c63d2d74c54c5e4457dd99dd37b1bbda3bbbabfc0f2eacb4",
        "Descriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:89717f4c87c7fc93c63d2d74c54c5e4457dd99dd37b1bbda3bbbabfc0f2eacb4",
            "size": 2207,
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        },
        "SchemaV2Manifest": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "schemaVersion": 2,
            "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:959a3aa87f6ae3add03a5a568d218e99e9394c785d924e5d6398d7cb4be0f472",
                "size": 5123
            },
            "layers": [ ... ]
        }
    },
    {
        "Ref": "docker.io/litmuschaos/go-runner:1.10.0@sha256:04cec97644444d564e256e87184e0daa73f3f335d187c65a1723f0f8dc61ad22",
        "Descriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:04cec97644444d564e256e87184e0daa73f3f335d187c65a1723f0f8dc61ad22",
            "size": 2210,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        "SchemaV2Manifest": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "schemaVersion": 2,
            "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "digest": "sha256:679ad1c72c9be392b47b3fc6c455c383e28b8fcdbcf923dd5fb9db2cfb5c3cc2",
                "size": 5125
            },
            "layers": [ ... ]
        }
    }
]

We can observe that the manifest for the multi-arch image is a simple list of manifests prepared for each platform while an image with a single arch will have only one platform in its manifest.

Build Multi-arch image for Litmus Components

For building a multi-arch image for litmus core components just follow the following simple steps:

• Setup buildx
• Build go binary for different architectures.
• Prepare Dockerfile for Multiarch.
• Build and push the multi-arch image.

Build using Buildx instance

Introduction:

Docker Buildx is a CLI plugin that extends the docker command with the full support of the features provided by Moby BuildKit builder toolkit. It has new features like creating scoped builder instances and building against multiple nodes concurrently.

Setup:

The Docker Buildx is included in Docker 19.03 to install it run the following commands:

$ export DOCKER_BUILDKIT=1
$ docker build --platform=local -o . git://github.com/docker/buildx
$ mkdir -p ~/.docker/cli-plugins
$ mv buildx ~/.docker/cli-plugins/docker-buildx

Confirm the installation:

$ sudo docker buildx version

Output

github.com/docker/buildx v0.4.2-16-g1595352 159535261de8416d2a83d600ccdd7dbfa6d71303

for any issue in installation refer installation guide.

Now the buildx is setup in your system. Buildx allows you to create new instances of isolated builders. You can use this to get a scoped environment for your CI builds that does not change the state of the shared daemon, or for isolating builds for different projects. You can create a new instance for a set of remote nodes, forming a build farm, and quickly switch between them.

Setup Builder instance

1. Using the QEMU emulation support in the kernel

sudo apt-get install qemu-user-static -y
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes i

2. Command to create new builder instances.

docker buildx create --name multibuilder

Output

multibuilder

3. To show the builder instance created

docker buildx ls

Output

NAME/NODE       DRIVER/ENDPOINT             STATUS  PLATFORMS
multibuilder *  docker-container                    
  multibuilder0 unix:///var/run/docker.sock running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
default         docker                              
  default       default                     running linux/amd64, linux/386

4. Inspect the builder

docker buildx inspect multibuilder

Name:   multibuilder
Driver: docker-container

Nodes:
Name:      multibuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Here the platforms will show the platform available to build images.

5. If the Status is not running run the following command
docker buildx inspect multibuilder --bootstrap

This will activate the Status of the builder instance.
Output

[+] Building 6.5s (1/1) FINISHED                                                                                                                                                       
 => [internal] booting buildkit                                                                                                                                                   6.5s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                5.0s
 => => creating container buildx_buildkit_multibuilder0                                                                                                                           1.5s
Name:   multibuilder
Driver: docker-container

Nodes:
Name:      multibuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/386, linux/arm/v7, linux/arm/v6, linux/s390x

6. Use the multibuilder instance

docker buildx use multibuilder

Build Go binary

We know that all the litmus core components (chaos experiment, operator, and runner) are written in Golang and we are building the go binaries and using them in the image.

Note:
You can build the go binaries and copy them in Dockerfile or we can build the binary in the Dockerfile itself and use them.

In this example, I'll be following the first way that is building binary and copying them in the Dockerfile.
The main key point here is to build different binaries with Go ENVs set to desired values like GOARCH=amd64 &GOOS=linux for which we have to build the image.

git clone https://github.com/litmuschaos/chaos-operator.git
cd chaos-operator
vi build/go-multiarch-build.sh 

The following script will help to do that:

#!/usr/bin/env bash

package=$1
if [[ -z "$package" ]]; then
  echo "usage: $0 <package-name>"
  exit 1
fi

package_split=(${package//\// })
package_name=${package_split[-1]}

# add the arch for which we want to build the image
platforms=("linux/amd64" "linux/arm64")

for platform in "${platforms[@]}"
do
    platform_split=(${platform//\// })
    GOOS=${platform_split[0]}
    GOARCH=${platform_split[1]}
    output_name=build/_output/bin/chaos-operator$GOARCH

    # The script executes for the argument passed (in package variable)
    # here the arg will be "github.com/litmuschaos/chaos-operator/cmd/manager" for creating binary
    env GOOS=$GOOS GOARCH=$GOARCH go build -o $output_name $package

    if [ $? -ne 0 ]; then
        echo 'An error has occurred! Aborting the script execution...'
        exit 1
    fi
done

Command

• ./build/go-multiarch-build.sh github.com/litmuschaos/chaos-operator/cmd/manage

In the above script, we are creating the binary of github.com/litmuschaos/chaos-operator/cmd/manage with GOARCH & GOOS set in the platforms. This will create binaries as follow:

$ ls build/_output/bin/
chaos-operatoramd64*  chaos-operatorarm64*

Prepare Dockerfile for Multiarch

Prepare a Dockerfile that should work for all the different architectures. Make sure that:
1. The base image should be multi-arch: Use the base image which is having multiarch support.

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3

Which supports both ARM64 and AMD64 arch.

2. Use TARGETARCH variable for using binary in Dockerfile: All binaries used should support multi-arch builds and are built accordingly. The Docker predefines a set of ARG variables according to the --platform we pass in buildx. This is only available when using the BuildKit backend. Here the TARGETARCH is the architecture component of TARGETPLATFORM. Check out more platform env here.

Example

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.3

ARG TARGETARCH

ENV OPERATOR=/usr/local/bin/chaos-operator \
    USER_UID=1001 \
    USER_NAME=chaos-operator

# install operator binary
COPY build/_output/bin/chaos-operator${TARGETARCH} ${OPERATOR}

COPY build/bin /usr/local/bin
RUN  /usr/local/bin/user_setup

ENTRYPOINT ["/usr/local/bin/entrypoint"]

USER ${USER_UID}

This will copy the binaries created in the previous step while building the image.

Build and push the multi-arch image

BuildKit is designed to work well for building for multiple platforms and not only for the architecture and operating system that the user invoking the build happens to run. When you invoke a build, you can set the --platform flag to specify the target platform for the build output, (for example, linux/amd64, linux/arm64, darwin/amd64).

A sample command to build and push the multiarch image with the Dockerfile created in the previous stage.

sudo docker buildx build --file build/Dockerfile --progress plane --platform linux/arm64,linux/amd64 --tag $(ORG_NAME)/$(IMAGE_NAME):$(IMAGE_TAG) . --push

The above command will build a multiarch image and push it to Dockerhub. The flags used are:

• --file: Path of the Dockerfile (Default is ‘PATH/Dockerfile’)
• --progress: Use to control verbosity. Set type of progress output (auto, plain, tty). Use plain to show container output.
• --platform: Set target platform for build.
• --tag: Set tag of the image in the ‘name:tag’ format.
• --push: Used to push the image to the registry.

for more flags checkout buildx man page.

Multiarch image in Dockerhub:

We have successfully built and pushed the multiarch images on Dockerhub. The images will look like:

Troubleshooting

1. standard_init_linux.go:211: exec user process caused "exec format error":

There could be multiple reasons for this. Try to check out the base image you're using is supporting all the platforms for which you're building the images.

Make sure while building and copying the binary in Dockerfile that the arch and os should be same. Like you're building the binary with linux/amd64 and you're trying to run it on linux/arm64.

2. rpc error: code = Unknown desc = failed to load LLB: runtime execution on platform linux/arm/v7 not supported:

This means the platform you're looking for is not available. You can verify the available platforms using docker buildx inspect <builder-name>.Now you can run the following commands to resolve the above error.

sudo apt-get install qemu-user-static -y
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes i
docker buildx rm multibuilder
docker buildx create --name multibuilder
docker buildx ls
docker buildx inspect multibuilder
docker buildx inspect multibuilder --bootstrap
docker buildx use multibuilder
docker ps -a
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t <ORG_NAME>/<IMAGE_NAME>:<IMAGE_TAG> --push .

Conclusion

In this blog, we learn how to build and push an image that can be used at different docker architecture with the same image tag on it. This can be done using a powerful docker buildx feature which helps to build image with a single command. We can create images that can run on any desired platform using buildx and can be used to reduce your bills using ARM targets Raspberry Pis instance. So have ever tried building multiarch images? How useful it could be? Please leave your thoughts in the comment box.

🚀 Special Thanks 🙌

Thanks to Michael Fornaro for helping in suggesting, reviewing & adopting the multiarch support for Litmus Chore Components.

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?
Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN
(#litmus channel on the Kubernetes workspace)
Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus
Submit a pull request if you identify any necessary changes.