Series Introduction

Welcome to Part 4 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.

In this post we will be looking at:

Add container scanning to our GitHub Action workflow
Resolve security vulnerabilities

TL;DR

We add the Azure Container Scan GitHub Action to our actions' workflow file, before pushing the container image to our registry. Add a group and user to our Dockerfile and use the USER command to specify the user for the process to run; ensuring our copied files can be accessed by this user. The security attack surface is reduced by running as non-root, and this is raised in CIS; the linting provided by Dockle. We resolved some slightly older CVE’s as examples; these can be raised by Trivy and can be allowed through or not based on configuration. Both Trivy and Dockle are wrapped by the Azure Container Scan GitHub Action.

GitHub Repository

peteking / Samples.WeatherForecast-Part-4

This repository is part of the blog post series, API's from Dev to Production - Part 4 on dev.to. Based on the standard .net standard Weather API sample.

Introduction

In Part 3 we were able to get into GitHub Actions and build and push our Docker image to the GitHub Container registry. This is a great start, we are producing a Docker image in CI and it’s a consistent image, however, what about vulnerabilities?

Requirements

We will be picking-up where we left off in Part 3, which means you’ll need the end-result from GitHub Repo - Part 3 to start with.

If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.

Vulnerabilities

A software vulnerability is a glitch, flaw, or weakness that is present in software or in an Operating System. CVE (Common Vulnerabilities and Exposures), contains a list of records each containing an identification number, a description, and at least one public reference - for publicly known cybersecurity vulnerabilities.

CVE Records are used in numerous cybersecurity products and services from around the world, including the U.S. National Vulnerability Database (NVD).

There are many open source and commercial tools specifically designed for CI purposes; to be used before pushing a Docker image to a container registry. This can ensure any CVE’s of certain thresholds can be caught a build time.

Commercial tools can also scan containers at runtime and apply custom mediating actions.

CIS (Center for Internet Security)

CIS is a community of cybersecurity experts, who have developed CIS Benchmarks: more than 100 configuration guidelines across 25+ vendor product families to safeguard systems against today’s evolving cyber threats.

CIS also produced a list of hardened images - https://www.cisecurity.org/cis-hardened-image-list/

Why is this important?

It’s important because the software we design, build, and ship needs to be secure, we want to ensure safety of our end users' data and be responsible. We must take security seriously and both CVE and CIS provide a trusted source of information.

The open source community has done an amazing job of utilising the information and provided fantastic tooling we can all take advantage of.

What can we do?

In terms of CVE’s, AquaSec has created not only a great commercial product but an open source product called Trivy. This is a Docker image scanner that detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.). Trivy is easy to use, you can just install the binary and scan, all that’s required is to specify a target such as an image name of the container.

For CIS, there is an open source tool called Dockle - Dockle is a Container Image Linter for Security, Helping build the Best-Practice Docker Image, which is ideal for CI.

GitHub Actions to the rescue

Microsoft has create a GitHub Action which wraps both Trivy and Dockle called container-image-scan, you can find it on the GitHub Actions Marketplace and the open source repo, Azure/container-scan.

As of writing, the GitHub Container Registry does not scan images you push there, but other container registries do, such as the Azure Container Registry (ACR). It’s good that images will be scanned once they are in the registry, however, in terms of shifting-left, it is hugely beneficial to scan images during CI, if a vulnerability is found during build, engineers in the team are notified immediately, they can fix it themselves or consult with experts to help them.

This promotes a faster feedback loop and potentially stops images being pushed to a registry if that have vulnerabilities.

Let’s use Azure Container Scan

In out workflow file we previously called, build-and-publish.yml, it’s located in the .gihub/workflows folder. We can add an extra step, we’ve given it a name, Scan docker image, specified the azure/container-scan@v0 GitHub Action, and set the required and optional parameters. We’ve set the severity-threshold: MEDIUM, this is completely up to you, for this example, I’ve simply chosen MEDIUM. Please read the documentation on the Azure Container Scan repository and set it to what you believe is adequate for you specific situation.



- name: Scan docker image
  uses: azure/container-scan@v0
  with:
    image-name: ${{ env.image-name }}
    severity-threshold: MEDIUM
    username: ${{ github.repository_owner }}
    password: ${{ secrets.GH_CR }}

Full Workflow file



name: Build and Publish

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  workflow_dispatch:

env:
  image-name: ghcr.io/peterjking/samples-weatherforecast-part-4:${{ github.sha }}

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout the repo
        uses: actions/checkout@v2

      - name: Build docker image
        run: docker build . -t ${{ env.image-name }}

      - name: Scan docker image
        uses: azure/container-scan@v0
        with:
          image-name: ${{ env.image-name }}
          severity-threshold: MEDIUM
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GH_CR }}

      - name: Push docker image
        run: docker push ${{ env.image-name }}

If you commit the changes, your GitHub Action will kick-off and build and scan.

If you have not specified your repository secret GH_CR because you may be using a new repository perhaps for this, don’t forget to add the secret to your repo.

Scan results

CVE

Depending on when you are running this and what the state of the base image will determine what you see. For me at the time of writing, I have no CVE’s, which is great! Only a week before writing this, I made a change for a similar workflow and had a HIGH CVE, it was all to do with OpenSSL in Alpine Linux. The screenshot is below for when it came up.

We can check what CVE this is with its number, a quick Google will reveal some interesting information.

For this particular issue, I found the following link:

alpine: bump 3.12.3/3.11.7 (CVE-2020-1971) #9295

Name: API's From Dev to Production - Part 4 - Image Scanning
Rating: 1.3 (3065 reviews)
Author: peteking

ncopa posted on Dec 16, 2020

View on GitHub

Whilst writing this blog series, this issue was open.
Since then, the issue has been closed and merged :)

It was essentially a simple update, OpenSSL needed to be updated in Alpine Linux to resolve this vulnerability.

Since we have zero known vulnerabilities at the moment, from another API, I did come across another issue, again, with Alpine but earlier in December 2020. The scan results are below as another example:

In early December this came up, and this is because I was using the same base image we are using in this blog post. You should always investigate CVE and determine if they apply to you, and the reason why I’m showing you this is for that reason. Let’s go through it and see what it is.

Again, a quick Google on CVE-2020-28928, and I decide to settle on the information over at AquaSec:

https://avd.aquasec.com/nvd/cve-2020-28928/

You can see it is indeed MEDIUM, 5.5 severity, and when it was published, November 24th 2020.

If you scroll down the page you’ll see a nice thoughtful list on mitigations. If you look even closer you can see the potential mitigation is essentially memory management. Luckily for us, we are using a language and framework which prides itself on being memory safe and good memory management (on our behalf of course). Given we are using C# in this case and we are not using C or C++ for instance and using string.h etc. We can safely ignore this.

How do we ignore this CVE you ask? Well, if you check the documentation on the Azure Container Scan repository, it shows you how…

Simply create a new folder in your .github/workflows/ called containerscan, in there create a new YAML file called, allowedlist.yaml.



general:
  vulnerabilities:
    - CVE-2020-28928
  bestPracticeViolations:

Good Practice

If you do this, you need to have a regular review going forward of what you are allowing through.

Given the noted good practice above, this is a suburb example, Alpine Linux resolved this issue. How did I know this? Well, looking at Alpine’s GitHub repo tells all in this GitHub issue:

alpine:latest has CVE-2020-28928 #123

jerome-laforge posted on Dec 04, 2020

musl : 1.1.24-r9 - Layer: sha256:ace0eda3e3be35a979cec764a3321b4c7d0b9e4bb3094d20d3ff6782961a8d54

CVE-2020-28928 In musl libc through 1.2.1, wcsnrtombs mishandles particular combinations of destination buffer size and source character limit, as demonstrated by an invalid write access (buffer overflow). Solution: Upgrade musl to 1.1.24-r10

References: http://www.openwall.com/lists/oss-security/2020/11/20/4 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28928 https://lists.debian.org/debian-lts-announce/2020/11/msg00050.html https://musl.libc.org/releases.html

View on GitHub

Now, for my repo or a repo that had this vulnerability, I can safely remove it from the, allowedlist.yaml file.

CIS

In terms of CIS, we have a single 1 x WARN, and 2 x INFO.

Personally, I believe CIS-DI-0001 should really be higher, you really should in my view run your process as non-root. It’s easily solved too!

When the processes inside the container run as root, the same as the host machine, this give unprecedented privileges; we should always opt for least-privilege as a matter of practice. For example, if a user is able to gain access to the host as root

By default, root in a container is the same root (uid 0) as the host machine. If a user succeeds in breaking out of an application running as root in a container, this malicious user may be able to gain access to the host with the same root user.

Good Practice
It is good practice to launch processes with a non-root user.

TIP

Some official images already create a user for you to use, it’s always good to inspect the Dockerfile your using as your base to see.

In our case, a user is not created for us, so let’s fix that now, you can add the following code to your Dockerfile.



RUN addgroup -g 1000 dotnet && \
    adduser -u 1000 -G dotnet -s /bin/sh -D dotnet

Once we do this, when the process runs, it should be running as this, dotnet user. Therefore, we should also make sure when we COPY files we change the owner to the dotnet user using chown like so:

COPY --chown=dotnet:dotnet --from=publish /out .

If we do not do this, we run the risk of the dotnet user not being able to see the files and execute - That would be bad for us.

Full Dockerfile



ARG VERSION=5.0-alpine

FROM mcr.microsoft.com/dotnet/sdk:${VERSION} AS build
WORKDIR /app

# Copy and restore as distinct layers
COPY . .
WORKDIR /app/src/Samples.WeatherForecast.Api
RUN dotnet restore Samples.WeatherForecast.Api.csproj -r linux-musl-x64

FROM build AS publish
RUN dotnet publish \
    -c Release \
    -o /out \
    -r linux-musl-x64 \
    --self-contained=true \
    --no-restore \
    -p:PublishReadyToRun=true \
    -p:PublishTrimmed=true

# Final stage/image
FROM mcr.microsoft.com/dotnet/runtime-deps:${VERSION}

RUN addgroup -g 1000 dotnet && \
    adduser -u 1000 -G dotnet -s /bin/sh -D dotnet

WORKDIR /app
COPY --chown=dotnet:dotnetgroup --from=publish /out .

USER dotnet
EXPOSE 8080
ENV ASPNETCORE_URLS=http://+:8080
ENTRYPOINT ["./Samples.WeatherForecast.Api"]

Now you have a slightly refactored Dockerfile, let’s give this a go locally to make sure all is well.

Build the image locally:

docker build -t samples-weatherforecast:v4 .

Run the image locally:

docker run -it --rm -p 8080:8080 samples-weatherforecast:v4

Test the image in Postman or REST Client - https://marketplace.visualstudio.com/items?itemName=humao.rest-client

Once you’ve verified that it works, it’s now time to commit those changes. This should kick-off the GitHub Action and build, scan and push the Docker image container registry.

We can verify our API is running under our dotnet user but getting inside the container.

How you ask? Well, there are two ways, the CLI way or you can use Docker Desktop.

I’ll show off the Docker Desktop feature.

Open Docker Desktop → Move you mouse of your running container → Click, CLI.

A command prompt should open and you should be inside your container!

Now, execute, whoami and it should come back saying, dotnet; this is your USER.

You can also execute the top command, let’s do this too:

Cool! All running as dotnet.

Build results

Packages → Container Registry
Let’s double check that our image was pushed by checking our GitHub Packages area.

Success!

In addition, I have also pulled this image down and tested it too just like we did in API's From Dev to Production - Part 3

What have we learned?

We have learned how to scan our container images before we end up pushing them to our container registry. We understand how important this is and why we should do it, the open source tools that are available to you. We have also learned how to decrease our security attack surface by running our process as a user by adding a group and user in our Dockerfile.

There are many open source tools that are very capable, however, at times you may wish to opt for a paid product.

Here is a short list:

Snyk
Acqua
Prisma Cloud (formally Twistlock)
Anchore
...and many more, don't forget some container registries also scan images.

Up next

Part 5 in this series will be about CIS Issues:

Add a healthcheck to the Dockerfile
Add an item to the CIS allowedlist.yaml

API's From Dev to Production - Part 4 - Image Scanning