In the previous article, we've seen how to build and run a Keycloak application based on Distroless base image. However, this article only used Docker to launch containers, and you mostly use an orchestrator to do that. In this article, we will see how to run our previously created image in a Kubernetes cluster.

Challenges

Name: Keycloak on Distroless into Kubernetes
Rating: 1.3 (3065 reviews)
Author: davinkevin

Using a Keycloak image based on Distroless requires some adaptation:

Launching the application require a specific command
Stopping the application require a specific command
Probes should be defined
Database connection should be provided via env variables
Configuration file should be mounted by the orchestrator

Database

Before doing some configuration, we will "install" a PostgreSQL instance inside our cluster. This will be done with this manifest:

# database.yaml
apiVersion: v1
kind: Service
metadata:
  name: database
  labels:
    app: database
spec:
  ports:
    - name: pg-port
      port: 5432
      targetPort: 5432
      protocol: TCP
  selector:
    app: database
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: database
spec:
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
        - image: postgres:13.3-alpine
          imagePullPolicy: IfNotPresent
          name: database
          env:
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database
                  key: password
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: name
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - su - postgres -c "pg_ctl stop -m fast"
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - -e
                - exec pg_isready -U $POSTGRES_USER -d $POSTGRES_DB -h 127.1 -p 5432
          ports:
            - name: pg-port
              containerPort: 5432
              protocol: TCP

NOTE: This manifest is some kind of hello-world at PostgreSQL level and SHOULD NOT be used for production… because it doesn't manage any state. Every data will be lost if the pod restart. This is just for this example.

Launching Keycloak

In the previous step, we fetched the java command used to launched Keycloak. We will reuse it, in the Kubernetes manifest. For information, the command was:

java '-D[Standalone]' -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/keycloak -Djboss.server.base.dir=/opt/jboss/keycloak/standalone -Djboss.bind.address=172.17.0.2 -Djboss.bind.address.private=172.17.0.2 -c=standalone-ha.xml

We will set $.spec.template.spec.containers[0].command and $.spec.template.spec.containers[0].args to the previously used ENTRYPOINT in the Dockerfile.

# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          # We define Java as the main command of the image.
          command: [ "java" ]
          # We put back all the parameters from the previous step.
          args:
            - "-D[Standalone]"
            - "-server"
            - "-Xms64m"
            - "-Xmx512m"
            - "-XX:MetaspaceSize=96M"
            - "-XX:MaxMetaspaceSize=256m"
            - "-Djava.net.preferIPv4Stack=true"
            - "-Djboss.modules.system.pkgs=org.jboss.byteman"
            - "-Djava.awt.headless=true"
            - "--add-exports=java.base/sun.nio.ch=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED"
            - "--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED"
            - "-Dorg.jboss.boot.log.file=/opt/jboss/keycloak/standalone/log/server.log"
            - "-Dlogging.configuration=file:/opt/jboss/keycloak/standalone/configuration/logging.properties"
            - "-jar"
            - "/opt/jboss/keycloak/jboss-modules.jar"
            - "-mp"
            - "/opt/jboss/keycloak/modules"
            - "org.jboss.as.standalone"
            - "-Djboss.home.dir=/opt/jboss/keycloak"
            - "-Djboss.server.base.dir=/opt/jboss/keycloak/standalone"
            - "-c=standalone.xml"
            - "-b=0.0.0.0"
            - "-bprivate=0.0.0.0"
            - "-bmanagement=0.0.0.0"

NOTE: Like previously, the parameter can and should be adapted to your needs, for performance and security reasons.

Providing configuration to Keycloak

Another part of the Keycloak configuration is the standalone.xml. Our goal here will be to mount the file inside the running container, at the right place, to be used as source of configuration at boot time.

# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      # We declare a volume containing the configuration of keycloak
      volumes:
        - name: keycloak-config
          # The configMap targeted here will be declared in the next step of the article
          configMap:
            name: keycloak
      containers:
        - name: keycloak
          # And then, we mount the file from the previously seen volume at the 
          # location where keycloak want to find it…  
          volumeMounts:
            - name: keycloak-config
              mountPath: /opt/jboss/keycloak/standalone/configuration/standalone.xml
              subPath: standalone.xml

Probes

In Kubernetes world, the orchestrator needs to know when the system is live and ready. For this, we have to define some liveness and readyness probes allowing Kubernetes to act and react to problem with Keycloak if required. Again, this is just some values in our manifest:

# keycloak.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          # Liveness, if in error, keycloak will be restarted
          livenessProbe:
            httpGet:
                path: /auth/
                port: http
            initialDelaySeconds: 30
            timeoutSeconds: 5
          # Readiness, if in success, traffic will be routed to this pod
          readinessProbe:
            httpGet:
                path: /auth/realms/master
                port: http
            initialDelaySeconds: 30
            timeoutSeconds: 1

NOTE: RedHat team will introduce more advanced and precise liveness and readyness probes in future version of Keycloak 😇.

Lifecycle Pre-Stop Command

To gracefully shutdown Keycloak, we usually have to execute a specific shell script. In our case, without any shell, this will be problematic… like for ENTRYPOINT, we will extract the java command from the shell script and use it in our manifest.

# We launch the keycloak original image and launch a bash inside it
$ docker run -it --rm --entrypoint=bash jboss/keycloak:13.0.1
# We use our magic trick to see commands executed by the jboss-cli.sh we will use
bash-4.4$ awk -i inplace 'NR==2 {print "set -x"} 1' /opt/jboss/keycloak/bin/jboss-cli.sh
# Finally, we launch the script with parameters we want to execute in our pre-hook
bash-4.4$ /opt/jboss/keycloak/bin/jboss-cli.sh --connect command=:shutdown --timeout=20
++ dirname /opt/jboss/keycloak/bin/jboss-cli.sh
+ DIRNAME=/opt/jboss/keycloak/bin
+ GREP=grep
+ . /opt/jboss/keycloak/bin/common.sh
++ '[' x = x ']'
++ COMMON_CONF=/opt/jboss/keycloak/bin/common.conf
++ '[' -r /opt/jboss/keycloak/bin/common.conf ']'
+ cygwin=false
+ darwin=false
+ case "`uname`" in
++ uname
+ false
++ cd /opt/jboss/keycloak/bin/..
++ pwd
+ RESOLVED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' x/opt/jboss/keycloak = x ']'
++ cd /opt/jboss/keycloak
++ pwd
+ SANITIZED_JBOSS_HOME=/opt/jboss/keycloak
+ '[' /opt/jboss/keycloak '!=' /opt/jboss/keycloak ']'
+ export JBOSS_HOME
+ '[' x = x ']'
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
+ '[' x = x ']'
+ '[' x '!=' x ']'
+ JAVA=java
+ setDefaultModularJvmOptions
+ setModularJdk
+ java --add-modules=java.se -version
+ MODULAR_JDK=true
+ '[' true = true ']'
++ echo
++ grep '\-\-add\-modules'
+ DEFAULT_MODULAR_JVM_OPTIONS=
+ '[' x = x ']'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
+ DEFAULT_MODULAR_JVM_OPTIONS=' --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
+ false
+ false
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing'
+ JAVA_OPTS='  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true'
++ eval echo '"/opt/jboss/keycloak/modules"'
+++ echo /opt/jboss/keycloak/modules
+ JBOSS_MODULEPATH=/opt/jboss/keycloak/modules
++ echo --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true
++ grep logging.configuration
+ LOG_CONF=
+ '[' x = x ']'
+ exec java --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true -Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties -jar /opt/jboss/keycloak/jboss-modules.jar -mp /opt/jboss/keycloak/modules org.jboss.as.cli --connect command=:shutdown --timeout=20
Failed to connect to the controller: The controller is not available at localhost:9990: java.net.ConnectException: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: WFLYPRT0053: Could not connect to remote+http://localhost:9990. The connection failed: Connection refused

The execution ends in error, which is normal because, in this case, no Keycloak instance are running. The java command is displayed, starting with + exec java. All of this can be moved into our yaml manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          lifecycle:
            preStop:
              exec:
                command:
                  - "java"
                  - '--add-exports=java.base/sun.nio.ch=ALL-UNNAMED'
                  - '--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED'
                  - '--add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED'
                  - '-Djboss.modules.system.pkgs=com.sun.java.swing'
                  - '-Dcom.ibm.jsse2.overrideDefaultTLS=true'
                  - '-Dlogging.configuration=file:/opt/jboss/keycloak/bin/jboss-cli-logging.properties'
                  - "-jar"
                  - "/opt/jboss/keycloak/jboss-modules.jar"
                  - "-mp"
                  - "/opt/jboss/keycloak/modules"
                  - "org.jboss.as.cli"
                  - "--connect"
                  - '--commands=shutdown --timeout=20'

Environment Configuration

Finally, we have to provide all the configuration to Keycloak. Until now, we only define how the value will be linked but not which value at all. To do that, we will first include some environment values into the Keycloak manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  template:
    spec:
      containers:
        - name: keycloak
          env:
            - name: DB_ADDR
              value: database
            - name: DB_DATABASE
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: name
            - name: DB_USER
              valueFrom:
                configMapKeyRef:
                  name: database
                  key: user
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database
                  key: password

Here, we are mainly targeting values required for database connection. To provide it to the deployment, we will use a kustomization.yaml file. It will gather every manifests presented until now and add concret values:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Install everything in the kubernetes namespace named Keycloak
namespace: keycloak

# Load both keycloak.yaml and database.yaml
resources:
  - keycloak.yaml
  - database.yaml

# Generate two ConfigMap
configMapGenerator:
  # One for Keycloak, containing the standalone.xml
  - name: keycloak
    files:
      - standalone.xml
  # One for database, containing user and name
  - name: database
    literals:
      - user=keycloak
      - name=keycloak

# Generate a Secret
secretGenerator:
  # For the database, containing just the password of the database
  - name: database
    literals:
      - password=sPCwZjuq8CMvrBn7

NOTE: The password is here stored in clear, you can choose to use some tooling to encrypt the secret at git level (with SOPS) or cluster level (with SealedSecret).

Deploying Keycloak

In this step, we will deploy our version of Keycloak in a Kubernetes Cluster. For this example, I choose to use docker-for-mac and its Kubernetes integration. You can, of course, use any Kubernetes distribution (Google Kubernetes Engine, Azure Kubernetes Service, Amazon Elastic Kubernetes Service…).

$ ls
database.yaml  keycloak.yaml  kustomization.yaml  standalone.xml
# We deploy all the manifests
$ kubectl apply -k .
configmap/database-56h9f7gfdh created
configmap/keycloak-96k2tfg747 created
secret/database-8g8gk22d26 created
service/database created
service/keycloak created
deployment.apps/database created
deployment.apps/keycloak created
# We create a port-forward to the keycloak pod
$ kubectl -n keycloak  port-forward pod/keycloak-567797b6bb-6vqvz 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

To access it, we create a port-forward between my computer and the cluster. Thanks to that, I'm able to access the UI:

And Voila!

Conclusion

In this article, we've seen how to deploy our custom Keycloak based on Distroless Java. This is a challenging setup, but at the end we can use a more secure version of Keycloak without any shell, which prevent any attack using shell as a vector of code execution for example!

I hope you liked it, you can find all the sample files from this article in this GitLab repository: davinkevin/keycloak-distroless.