John Vester
Many enterprises that implement Kubernetes don’t realize how insecure it can truly be out of the box — and even when they do, they’re not sure what to do about it.
And since Kubernetes is an extremely complex and flexible tool that in production can consist of hundreds, if not thousands, of moving parts — understanding the vulnerabilities of Kubernetes, potential solutions, and relevant regulations and trends can be a time consuming and daunting task. For most any business, this makes it way too easy to introduce serious legal or security issues into their clusters and production applications.
With KubeCon Europe 2023, I was impressed with (and happy to see) the number of security solutions entering the market. Continuing from 2022, the biggest theme from KubeCon Europe 2023 was software supply chain security — ensuring that you know what’s running in your cluster and that it’s vulnerability free.
I was struck then, as I have been in recent years, at the importance of software supply chain security. With cloud-native apps, containerization, and Kubernetes, today’s software organizations are in an arms race with malicious attackers who seek to undermine and poison our software. So I was pleased to come across several additional projects at KubeCon Europe 2023 that are dedicated to tackling the challenge of Kubernetes security.
In this article, I’ll give a whistle-stop tour through those projects. But let’s start with a quick overview of the conference and what I consider to be the most important aspects of k8s security.
Breaking Down Kubernetes Security
First, the conference. I was a little worried that the IT recession might mean the conference wouldn’t be as meaningful as past years — but I’m happy to say that I had no need to worry! While I wasn’t able to attend in person, through remote and post-conference conversations with those that were there, the conference seemed to be as exciting as ever!
While there weren’t any standout new trends at the conference, there was a ton of tooling and discussion around Kubernetes security. To keep everything straight in my mind, I categorize k8s security into three particular aspects: attestation, provenance, and SBOM:
Attestation: Confirming your artifact’s origin and that it is what it claims to be. Attestation typically comes in the form of signing containers, packages, and so on, in a way that the receiving party can verify.
Provenance: The audit trail of where the artifact came from and who had interactions with it. Often used in connection with attestation if something doesn’t look right.
Software bill of materials (SBOM): What are the packages and dependencies inside the artifact? These are increasingly necessary for regulation and as a demonstration to customers.
With that in mind, let’s walk through the solutions.
OpenClarity
OpenClarity consists of a suite of tools that aim to bring insights into different levels of the cloud native stack, from security to observability. The two relevant components to this post are KubeClarity and VMClarity that create SBOMs and perform vulnerability analysis for Kubernetes clusters and virtual machines respectively.
KubeClarity runs on any Kubernetes cluster and provides a UI and CLI for analyzing images and generating SBOMs. By default, KubeClarity doesn’t have its own SBOM generator or vulnerability scanner, but instead supports third-party tools that you can enable in any combination, making it great for adding additional interfaces for existing toolchains.
If you would rather not install each of the components yourself, then they’re commercially available as part of Cisco Panoptica.
Support includes the following SBOM generators and vulnerability scanners:
We’ll cover some of these projects in more detail below.
VMClarity works similarly, but within the context of VMs used by cloud services to host containers and clusters. Compared to containers, VMs may be as (or even more) vulnerable to threats and they typically need complex tools for analysis. Following a similar pattern to KubeClarity, VMClarity supports aggregating multiple tools into one UI and CLI.
in-toto
in-toto is an open source project that focuses on the attestation part of software supply chain security. You use it to define a “layout” for a project, i.e., how the different components should fit together. A project ships this definition with its code, and then another user of that software can compare what they have with the attached definition to see if it matches the structure and contents they expect. If it doesn’t, then this could point to external tampering or other issues.
Trivy
Trivy is a mature and comprehensive open source tool from Aqua Security that supports scanning multiple sources, from file systems to containers and VMs. Trivy also looks beyond vulnerabilities, to scan licenses, secrets, infrastructure as code misconfiguration, and more.
It supports different SBOM output formats, attestation using in-toto, as well as scanning the attestation for vulnerabilities. To tie all these features together, Trivy has an ecosystem support, complete with IDE plugins, CI/CD, and a Kubernetes operator.
Slim.ai
Historically, Slim.ai aimed to help you understand the contents of an image and how to “slim” it down to the essentials. The platform has now expanded its feature set to include vulnerability scanning.
Slim.ai presents the data in a more user friendly way than many of the other tools in this post. On top of its open source SlimToolkit for identifying the contents of an image, Slim.ai uses Trivy for vulnerability scanning.
You can then use the “harden” feature to create a new version of the image that’s removed any vulnerabilities and extraneous packages.
The Update Framework
A common reason behind insecure software is out-of-date package versions, but how do you trust that an update is what it says it is and won’t cause you more problems?
Release signing—or attestation—was a hot topic at KubeCon among vendors, with many offering their own solutions. One in particular that stood out was CNCF’s recently graduated The Update Framework (TUF).
Confusingly named, it’s not a framework, but a specification that developers can follow when implementing their update systems to help users know they can trust the updates they receive. There’s also a Python reference implementation.
Syft
Syft is a popular open source CLI tool created by Anchore for generating an SBOM from container images and filesystems. It’s designed to provide a catalog of dependencies for other tools to use as a data source. It supports many popular programming languages, package managers, and container image formats.
With its support for different SBOM file formats and custom templates for reports, Syft has become an ideal component for vulnerability scanning toolchains.
Grype
Grype is another popular open source tool from Anchore. Working with SBOM files, Grype scans container images and filesystems for vulnerabilities. Grype supports different output formats for vulnerabilities and custom templates for output.
sigstore
sigstore is another suite of tools that focuses on attestation and provenance. Within the suite are two tools I heard mentioned a few times at KubeCon: Cosign and Rekor.
Cosign is used for signing containers through a variety of different methods. It has strong integration with other open source tools, such as Kyverno.
Rekor is a tamper-resistant ledger for metadata generated by a software supply chain project. This means that parties can query the metadata to help them make decisions based on decisions previously made by others. It’s not a very well-known project, but it is growing in popularity.
Kubescape
Kubescape is a comprehensive Kubernetes security platform and CNCF sandbox project. It can scan clusters, Helm charts, and YAML manifests to detect misconfigurations. It supports various frameworks, including NSA-CISA, MITRE ATT&CK®, and the CIS Benchmark.
Chainguard
Taking a very definite enterprise approach, Chainguard is impossible to try without first speaking with their sales department. But as far as I can tell, it consists of two main products.
Chainguard Images are specialized, hardened images that focus on minimal footprint, promise no vulnerabilities, are as small as possible, and ship with SBOMs.
Chainguard Enforce combines the creation of policies for your software supply chain with the enforcement of those policies. You write policies in YAML or by using a CLI tool to configure a Kubernetes agent that reports back to the service. You can view reports via the CLI or a GUI. Policies can include image signing, SBOM inclusion, and more.
Chainguard also appears to have several open source projects.The most popular one is apko, used for building OCI images from APK packages.
Falco
Falco is a well-known open source security solution originally created by Sysdig. It’s a CNCF incubating project and one of the few (as far as I can tell) options on this list that uses eBPF to scan for vulnerabilities.
This does mean it needs a few extra steps to install, and you can only test it on Linux systems. However, once you have Falco set up, it runs more performant than any container agent and can detect vulnerabilities at all system levels.
The future is complicated
I hope you enjoyed this overview and found a few tools that will help you to secure your enterprise.
Most experts believe that cybersecurity will become more complex and challenging as more aspects of the world are dominated by software: IoT, smart cars, drones—and now, we’ve added AI and LLMs to the mix. The arms race between attackers and defenders doesn't seem to favor the defenders. And deploying solutions is a complicated endeavor.
Now is a good time to find the tools and workflows that suit you and your organization, before you become the target.
Have a really great day!
