In the Kubernetes world when we want to handle sensitive data the usage is to store it in a Secret in a Kubernetes cluster. Cool, but when we want to save our Secrets in our source code (Git) repositories, a security problem appears.
We will see in this article, a solution that can helps us ;-).

Kubernetes Secrets

Secrets in Kubernetes are used to store sensitive data, like password, keys, certificates and token. Secrets are encoded in base64 and automatically decoded when they are attached and read by a Pod.

Cool, so what's the problem?

/!\ Be careful:

A secret in Kubernetes cluster is encoded in base64 but not encrypted!

Theses data are "only" encoded so if a user have access to your secrets, he can simply base64 decode to see your sensitive data (kubectl get secret my-secret -o jsonpath="{.data.password}" | base64 -D for Mac and --decode instead of -D for Linux platform).

And suddenly, since the secrets aren't encrypted, it can be unsecure to commit them to your Git repository :-(.

In the DevOps world, we can build, test, package and deploy our applications and infrastructure through a CI (Continuous Integration) & CD (Continuous Delivery) pipeline but we can also handle this through GitOps (a way of managing all our configurations and infrastructure through code in Git repositories).

So if we want to store in Git repositories our Kubernetes manifests, a problem appears. And as usual, several solutions exists , and today in this article we will see one of them: kubeseal.

Encrypt/Seal your Secrets with Kubeseal

Secrets are cool but what about encrypted secrets with kubeseal?
The goal of kubeseal, a bitnami tool, is to encrypt your Kubernetes Secret into a SealedSecret.
Thanks to that, it will be safe to store in public Git repository.
The SealedSecret can be decrypted only by the controller running in the target cluster and nobody else.

Overview

As you can see in the schema, a sealed-secret-controller run in the Kubernetes cluster. He listens when a new SealedSecret object appears, unsealed it (thanks to known certificates) and create a Kubernetes secret in the same namespace as the SealedSecret.

Be careful, if you delete the SealedSecret in your cluster, the linked Secret will be deleted too.

Usage - Step by step

After the theory, it's time to see all the steps.
Let's split the steps in two parts: server and client part.

Server part

We will:

deploy the sealed-secrets-controller in the cluster
retrieve generated certificate keypair

1. Install SealedSecret CRD, server-side controller into kube-system namespace



$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/controller.yaml

NOTE: If you can't (or don't want) to use the kube-system namespace, please consider this approach.

NOTE: if you want to install it on a GKE cluster for which your user account doesn't have admin rights, please read this guide.

Once you deploy the YAML Kubernetes manifest file, it will create the SealedSecret resource and install the controller into kube-system namespace, create a service account and necessary RBAC roles.

After a few moments, the controller will start, generate a key pair, and be ready for operation.

Resources created:



$ kubectl get all -n kube-system
NAME                                             READY   STATUS    RESTARTS   AGE
...           1/1     Running   63         2d
pod/sealed-secrets-controller-123456-abcdef   1/1     Running   2          4d

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/sealed-secrets-controller   ClusterIP   xxx.xxx.xxx.xxx   <none>        8080/TCP   4d

NAME                                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/sealed-secrets-controller   1         1         1            1           4d

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/sealed-secrets-controller-123abc456def   1         1         1       4d

If the controller Pod is not in Running state, check its logs:



$ kubectl logs sealed-secrets-controller-123456-abcdef -n kube-system

2. Retrieves the certificate keypair generated by the controller at start:



$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
NAME                       TYPE                DATA   AGE
sealed-secrets-key<ID>     kubernetes.io/tls   2      4d



$ kubectl get secret sealed-secrets-key<ID> -o yaml -n kube-system

apiVersion: v1
data:
  tls.crt: <encoded_crt>
  tls.key: <encoded_key>
kind: Secret
metadata:
  creationTimestamp: "2021-02-20T20:37:11Z"
  generateName: sealed-secrets-key
  labels:
    sealedsecrets.bitnami.com/sealed-secrets-key: active
  name: sealed-secrets-key<ID>
  namespace: kube-system
  resourceVersion: "123456"
  selfLink: /api/v1/namespaces/kube-system/secrets/sealed-secrets-key<ID>
  uid: abc123-45def-67gh-89ij-687vf6
type: kubernetes.io/tls

Here you can see tls.crt and tls.key data that you can base64 decode and store in your Vault or in your secret management tool.

Client part

We will:

install needed CLI tools
create a Kubernetes Secret
seal it into a SealedSecret
convert JSON to YAML file and clean it
deploy it in the Kubernetes cluster
check if everything have been deployed & created

0. Pre-requisites

Install following tools:

Kubeseal

For MacOS you can install it via Brew:



$ brew install kubeseal

For others platform you can follow the installation guide.

kubectl-neat (via Krew)



$ kubectl krew install neat

1. Create Kubernetes secret



$ kubectl create secret generic my-token --from-literal=my_token='123456789abc123def456ghi789' --dry-run=client -o yaml -n my-namespace > my-token.yaml

The YAML file should be like this:



$ cat my-token.yaml

apiVersion: v1
data:
  my_token: <a_token>
kind: Secret
metadata:
  creationTimestamp: null
  name: my-token
  namespace: my-namespace

Or you can create it also with Kustomize:



$ vi kustomization.yaml

namespace: my-namespace
secretGenerator:
- name: my-token
  literals:
  - my_token=<a_token>
generatorOptions:
  disableNameSuffixHash: true

$ kubectl kustomize . > my-token.yaml

2. Seal the secret (with the retrieved certificate)



$ kubeseal --cert tls.crt --format=yaml < my-token.yaml > mysealedtoken.yaml

The sealed YAML file should be like this:



apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-token
  namespace: my-namespace
spec:
  encryptedData:
    my_token: <encrypted_token>
  template:
    metadata:
      creationTimestamp: null
      name: my-token
      namespace: my-namespace

3. Clean the YAML manifest file with "kubectl neat" command



$ kubectl neat -f mysealedtoken.yaml > mycleanedsealedtoken.yaml

The cleaned sealed secret YAML file should be like this:



apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-token
  namespace: my-namespace
spec:
  encryptedData:
    my_token: <encrypted_token>
  template:
    metadata:
      creationTimestamp: null
      name: my-token
      namespace: my-namespace
    type: Opaque

4. Deploy SealedSecret in the cluster



$ kubectl apply -f mycleanedsealedtoken.yaml -n my-namespace

5. Check in your cluster

You can check/verify in your Kubernetes cluster if the SealedSecret and Secret have been correctly deployed:



$ kubectl get sealedsecret -n my-namespace
NAME           TYPE       DATA    AGE
my-token      Opaque       1      4d

$ kubectl get secret -n my-namespace
NAME          AGE
my-token    4d

Ok, my Secret have been successfully created by the controller! :-)

Details

Be careful, the SealedSecret and Secret resources must have the same namespace and name. This is a feature to prevent other users on the same cluster from re-using your sealed secrets.

Debug

As you know, some issues can appear, so in order to debug/troubleshoot the behavior of the controller, you can watch their logs:



$ kubectl logs sealed-secrets-controller-<podID> -n kube-system

A possible issue is that you sealed the secret with another certificate than the ones the controller know ;-).

Conclusion

Store your sensitive data in a Kubernetes Secret object is a common practice, but don't forget that a Secret is only encoded and not encrypted. So if you want to store them in a Git Repository (in GitHub or Gitlab repositories for example), you'll need to find a secure solution. Kubeseal is one of them and I hope this article helps you to see that it can be a solution that you can try in your side.

Store your Kubernetes Secrets in Git thanks to Kubeseal. Hello SealedSecret!