In the last years the popularity of containers has exploded. Unfortunately, so have their security risks.

Most containers available today are vulnerable to supply chain attacks, because they can be published with just a simple API key. And if that that key leaks, it’s easy for an attacker to publish a container that looks legit but actually contains malware.

One of the best ways to protect users from these kinds of attacks is by signing the container image at creation time so that developers can verify that the one they have received is the real image with the code as it was intended to be. Today we are gonna see how we can sign our container images automatically, and host them in GitHub Container Registry.

Video

As usual, if you are a visual learner, or simply prefer to watch and listen instead of reading, here you have the video with the whole explanation and demo, which to be fair is much more complete than this post.

Link to the video: https://youtu.be/OqZlKbTRWOY

If you rather prefer reading, well... let's just continue :)

About Sigstore and Cosign

So, signing a container image. There are few tools that allow to do so, but one of the most exciting one is sigstore.

Sigstore is an open source security project now sponsored by the OpenSSF, the Open Software Security Foundation, which allows developers to securely build, distribute, and verify signed software artifacts.

Among the other things, sigstore contains a tool called cosign which allows you to sign container images.

Cosign supports several types of signing keys, such as text-based keys, cloud KMS-based keys or even keys generated on hardware tokens, and kubernetes-secrets, which can all be generated directly with the tool itself, and also supports adding key-value annotations to the signature (and we will see this in action in a moment).

And after you sign the image, you need a Container Registry that supports signed images, because not all do, and even the ones that do support signed images may or may not support all the different signatures. Luckily, GitHub Container Registry supports signed images, and supports cosign as well.

But enough talking, let’s see how this works with GitHub Actions and GitHub Container registry.

Cosign Installation

First thing you need to do is installing cosign to generate the keys.

You can just go to the official GitHub repo, sigstore/cosign, click on Releases, and download the version for your operative system.

Many OSes and platforms are supported, so be sure to pick the right one.

Once you have the version which is right for you, you can just run it. It is also advisable to rename the tool, in my case the executable was called cosign-windows-amd64.exe but I’ve renamed to just cosign for ease of use.

Key Generation

Now all you have to do is generate a key. For this example, I will generate a static text key using the generate-key-pair command, which requires a password to create the keys.
The password can be given to the tool via environment variable

set COSIGN_PASSWORD=123
cosign generate-key-pair

or with an interactive prompt

cosign generate-key-pair

Unfortunately, the latest version available at the moment of recording this video has a bug which makes it crash if you try to use the interactive prompt to provide the password on Windows, as you can see below.

If you don’t want to create an environment variable, you can use PowerShell and the syntax you see below, with your password piped to the command.

"myPassw0rd" | .\cosign.exe generate-key-pari

In either case, this will create for you the private and public key files that you can use to sign and validate your container images.

Cosign Key Generation for GitHub

But we can do better. As I've mentioned before, I want to sign my container images via GitHub Actions, so now I would need to create some secrets in GitHub, and copy those keys to the secret values. But the tool can do this for us directly!

For example, let’s say I want to sign images in the n3wt0n/SignedContainers repo. I can use the same command to create my keys in GitHub directly:

export GITHUB_TOKEN=ghp_xyz123

export CONSIGN_PASSWORD=pwd123

./cosign generate-key-pair github://n3wt0n/SignedContainers

First thing I need to do is creating an environment variable called GITHUB_TOKEN and its value should be a PAT with write access to your repo. (Check this video to see how to create a PAT in GitHub)

Then, I can use the command we have seen before to generate the key, but this time we pass the repo as input parameter. The syntax as you can see is github://OWNER/REPONAME

As you can see, the secrets containing the password we specified as well as the private and public keys are created in our repo, ready to use.

Keep in mind that there have been instances in which the secrets were created in GitHub but their values were empty, and therefore the Actions workflow would fail. I’m not sure why that happens, but let me know if that happened to you as well. Anyway, the solution for this is simple, just try and generate the keys again or, if the problem persists, generate the keys locally and copy them manually into your secrets.

Sign a Container Image with Cosign and GitHub Actions

Alright, now that we have our keys set up, let’s see how we can sign our images from within a GitHub Actions workflow.

Let's assume we have a fairly standard Actions workflow which just build a Docker image and pushes it to the GitHub Container Registry (you can see the whole workflow's YAML below)

The first thing we have to do is install cosign. For this we can use the pre-built action, just search for cosign and you can find the cosign installer. It has a couple of parameters, but they are optional so we don’t need them for now:

    - name: cosign-installer
      uses: sigstore/cosign-installer@v2.0.0

When we have it, we can use the cosign sign command to sign our image:

    - name: Sign the published Docker image
      env:
        COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
      run: cosign sign --key cosign.key ${{ env.REGISTRY }}/${{ github.actor }}/${{ env.IMAGE_NAME }}:${{ github.run_id }}

It uses the private key for signing, and it needs the cosign password to access it, plus of course we have to specify the full image name, with the registry name as well.

As you can see tho, the command needs they key to be in a file, while we currently have it on a GitHub secret.

The workaround for that is to add another task before the sign command, which reads the key from the secret and writes it to disk:

    - name: Write signing key to disk
      run: 'echo "$KEY" > cosign.key'
      shell: bash
      env:
        KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}

___ I don’t much like it___, so I would prefer having a cosign implementation that can read it from the secret directly.

We can now commit and run our workflow. After a few seconds, the process is completed and we can see that our image has been successfully signed.

As I’ve mentioned the step to write the key on a file to give it to cosign is quite a workaround, and it may pose some security risk especially if you do it on self-hosted runners. Hosted runners are disposed as soon as a Job finishes so it “should” be ok.

In more secure scenarios, like enterprise ones, I would recommend saving those keys in services like Azure KeyVault, Hashicorp Vault, AWS KMS, or similar to avoid this issue.

Good thing is that cosign supports reading the keys from those services directly.

If you want to see the full YAML of the workflow, check it out here on GitHub

Verify a Container Image Signature

Anyway, after an image has been signed, we can always verify it using the public key that has been generated together with the private key. You can also share your public key with developers and users of the container image so they can always verify its authenticity.

To verify the authenticity of the image, we can use the cosign verify command.

cosign verify --key cosign.pub ghcr.io/n3wt0n/signedcontainer:123

We just need to pass to it the public key file, and the name of the image we want to verify, and that’s it.

If, for comparison, we try to verify an image that hasn’t been signed with our key, we will get this error:

And this of course will happen also if the image has been changed or modified after we have signed it, so our users can be safe and trust the image if the signature is verified.

Add Annotation to a Signature

We have said before that cosign also supports__ adding key-value annotations__ to the signature. Let’s see how we can do that. Let’s say that for example I want to sign an image and also add some author metadata to it.

Since I’m running this locally this time, I will need to first login into the container registry.

docker login -u myuser -p 123 ghcr.io

Then, I can use the usual command cosign sign, but this time I use a -a flag, which stands for annotation, to add some key value pairs to my image.

cosign sign --key cosign.key -a "author=CoderDave" ghcr.io/n3wt0n/signedcontainer:123

In this case I’m adding author=CoderDave, but it can be anything and you can add multiple values as well just adding more -a parameters.

After doing that, we can use the cosign verify command as we have seen before, and it will show also the annotation I’ve added to my image.

The annotations feature can be pretty useful. For example, if you are running the signing process in GitHub Actions like we have seen before, you may want to add information about your repo, workflow run, etc to your signature to make it more complete.

Cosign and GitHub Actions: Starter Workflow

Cosign and its process works fairly well, as we have seen, but setting it all up is not very immediate. Well, once again GitHub made it simpler for use to get started.

They have in fact integrated cosign in their starter workflow. Just go to Actions > New Workflow, and pick the “Publish Docker Container” starter workflow.

[...]

      # Install the cosign tool except on PR
      # https://github.com/sigstore/cosign-installer
      - name: Install cosign
        if: github.event_name != 'pull_request'
        uses: sigstore/cosign-installer@1e95c1de343b5b0c23352d6417ee3e48d5bcd422
        with:
          cosign-release: 'v1.4.0'


    [...]

      # Sign the resulting Docker image digest except on PRs.
      # This will only write to the public Rekor transparency log when the Docker
      # repository is public to avoid leaking data.  If you would like to publish
      # transparency data even for private images, pass --force to cosign below.
      # https://github.com/sigstore/cosign
      - name: Sign the published Docker image
        if: ${{ github.event_name != 'pull_request' }}
        env:
          COSIGN_EXPERIMENTAL: "true"
        # This step uses the identity token to provision an ephemeral certificate
        # against the sigstore community Fulcio instance.
        run: cosign sign ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build-and-push.outputs.digest }}

As you can see in the extract above, taken from the actual starter workflow, there is no key specified in there.

This is because Actions supports other 2 tools which are part of sigstore: Fulcio, which is a root CA that issues signing certificates from OIDC tokens, as well as Rekor, a transparency log for certificates issued by Fulcio.

Thanks to these, you can sign your container images with the GitHub-provided OIDC token in Actions, without provisioning or managing your own private key.

It is important to note that with this keyless signing process, your username, organization name, repository name, and workflow name will be published to the Rekor public transparency log. This is the right choice for public repositories, but probably not for private repositories. And in fact GitHub has disabled this in private repositories by default to prevent potential leaks.

Conclusions

What do you think about signing your container images, and especially doing so with cosign, GitHub Actions, and the GitHub Container Registry? Let me know in the comment section below.

Also, check out this video here in which I have 3 steps for you to make your Docker Image build faster.

Like, share and follow me 🚀 for more content:

📽 YouTube
☕ Buy me a coffee
💖 Patreon
📧 Newsletter
🌐 CoderDave.io Website
👕 Merch
👦🏻 Facebook page
🐱‍💻 GitHub
👲🏻 Twitter
👴🏻 LinkedIn
🔉 Podcast