This turned out to be a relatively lengthy post, but it represents the culmination of my recent study of Kubernetes, where I aimed to gain a comprehensive understanding of its core concepts and features. Throughout my devops research, I came to realise that prioritizing security considerations before any deployment is a more practical and effective approach to implementing technology. As a result, my study focused on two main areas: the fundamental structural components of Kubernetes and the essential security practices associated with it.

Due to resource constraints I was limited to how much of Kubernetes I could explore. However, I used two tools provided me enough access to learn it at a moderate level. These included:

Minikube: A lightweight Kubernetes implementation that creates a virtual machine on any OS and deploys a simple cluster containing only one node.
Kind: A tool for running local Kubernetes clusters using Docker container "nodes."

With Minikube, you are limited to a single cluster, which, for my purposes, was not an issue, as it is more than capable of running a simple three-tier application for reference.

In contrast, Kind offers more flexibility, allowing you to create multiple clusters and control various aspects of them

Nonetheless, this is my account of Kubernetes at a basic yet sufficient level, highlighting it's essential concepts, features, and tools.

Kubernetes Overview

Kubernetes is a fleet container management system that automatically manages the state of a multi-container system. Some key features include:

Service Discovery and Load Balancing: Kubernetes can expose a container based on its DNS or IP. It can also automatically divert traffic to a different container if one is particularly busy.
Storage Orchestration: Automatically mounts a storage option of your choice.
Automated Rollouts and Rollbacks: Kubernetes will automatically meet your desired state in a controlled manner.
Automatic Bin Packing: You can specify your CPU and memory requirements, and Kubernetes will automatically allocate the necessary resources to each container.
Self-Healing: Kubernetes automatically restarts containers that have been shut down.
Secret Management: Manages sensitive information such as passwords and tokens.
Batch Execution: Kubernetes can manage batch and CI workloads.
Horizontal Scaling: Easily scale applications up or down based on demand.
IPv4/IPv6: Allocation of IPv4/IPv6 addresses to pods and services.

Infrastructure

Clusters

Kubernetes coordinates a highly available cluster of computers that work together as a single unit. The Kubernetes cluster architecture consists of:

Control Plane: This coordinates the cluster and handles tasks such as scheduling applications, maintaining the desired state, scaling applications, and rolling out updates. This is governed by the Master Node, which contains several components:
- API Server: The front end for the control plane, handling all REST commands.
- Scheduler: Assigns work to nodes based on resource availability.
- Controller Manager: Manages controllers that regulate the state of the cluster.
- etcd: A distributed key-value store that holds the cluster's state and configuration.
Node Workers: A node is a VM or physical machine that serves as a worker in a cluster. Each node has a kubelet, which is an agent that handles communication between the node and the control plane.
Pods: Within each node, there exists one or more Pods that run application workloads. Each pod can contain multiple containers.

In production workloads, it's recommended to have at least three nodes to ensure you don't experience any downtime if a node goes down. If one node fails, a control plane instance is lost, so having redundant nodes is a sensible choice.

I've come to learn, that as production workloads really start to scale, it's also sensible to have multiple control planes for the same reasons.

Services

Services provide the ability to connect to a deployment. They are the abstraction that allows pods to die and replicate in Kubernetes without impacting your application. Some types of services include:

ClusterIP: The default service type, exposing the service on a cluster-internal IP. Used for internal communication within the cluster.
NodePort: Exposes the service on a static port on each node's IP address, allowing external access.
LoadBalancer: Creates an external load balancer to distribute traffic to the service.
ExternalName: Maps the service to an external DNS name.

Jobs

A job is a resource that creates one or more pods for the purpose of completing a task and then gets destroyed. Kubernetes offer 3 types of job:

Single-Completion Job: Self-explanatory, A single pod is created to complete a task.
Parallel Job: Allows several pods to run concurrently.
Cron Job: Runs jobs on a scheduled basis.

Namespaces

Namespaces are a means of creating a virtually separated section of a Kubernetes cluster. This is useful for:

Isolating Resources: Several teams may work on the same cluster and would want to keep their resources separate from other teams, preventing possible conflicts.
Environment Separation: Allocate space for development, testing, and production.
Access Control: Define who can access and manage resources.
Organizational Clarity: Keep resources organized and easy to manage.

Deployments

A deployment is a higher-level abstraction that manages the lifecycle of a set of replicas of a pod. In other words, it manages the desired state of a given application, defining how many replicas are needed to ensure constant uptime and efficient healing.

Some key features of deployments include:

Declarative Definition: You can declare the desired state of your deployment in a Deployment configuration. Kubernetes then automatically tries to match that state.
Rolling Updates: You can update your applications without experiencing any downtime. Kubernetes replaces old pods with new ones, ensuring that some instances are always available.
Rollback: If an update fails, you can easily revert to a stable version, as Kubernetes keeps track of deployment history.
Scaling: It's very easy to make changes to deployment requirements. Kubernetes will add or destroy pods automatically to match the desired number of replicas.
Self-Healing: If a pod fails or is destroyed, the deployment controller automatically creates a new pod to replace it, maintaining the replica count.

A simple deployment may look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: example-container
        image: example-image

Notes:

kind specifies the type of resource, in this case a deployment.
metadata - standard metadata section
spec - contains information regarding your desired state like the number of desired replicas, container image etc..

Scaling

Kubernetes offers two types of scaling: Horizontal and Vertical. Both of which are common terms in cloud architecture.

Horizontal scaling refers to the addition or removal of pods to handle load requirements.
Vertical scaling refers to the scaling of individual resources. So allocating more or less CPU/memory to existing pods. Kubernetes does not automatically handle this type of scaling.

Autoscaling functionality can be defined as a resource, A horizontal scaler, for example, might look like this:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: example-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-deployment
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

You can also manually scale a deployment using the command:

kubectl scale <deployment> --replicas=<num>

Rolling Updates

To maintain uptime, Kubernetes provides the ability to update pods by scheduling the creation of new, updated pods and waiting for those to deploy before destroying the old pods. Rolling updates allow the following actions:

Promote an app from one environment to another.
Roll back to previous versions.
Implement CI/CD with zero downtime.

Updates are triggered by any change to your deployment like the changing of application image which can be done with the following command:

kubectl set image <deployment_name> <image>

Once an update is underway there are several useful commands that can check the status of the update, pause the update or even revert to previous versions:

kubectl rollout status <deployment_name>
kubectl rollout undo <deployment_name>
kubectl rollout history deployment/<deployment_name>

Config

There are several ways to set environment variables in Kubernetes: Dockerfile, Kubernetes YAML, Kubernetes ConfigMaps, and Kubernetes Secrets.

A benefit of using ConfigMaps and Secrets is that they can be reused across containers. Both ConfigMaps and Secrets are API objects that store key-value pairs, but Secrets are used to store confidential/sensitive information.

ConfigMaps can be configured as either environment variables or volumes. If you make a change to a ConfigMap configured as an environment variable the change isn't reflected until the relevant pods are manually refreshed using kubectl rollout. If configured as volume however, the pod recognizes the changes almost immediately.

Immutable ConfigMaps are a suitable option for configurations that are expected to be constant and not change over time. Marking a ConfigMap as immutable provides performance benefits, as the kubelet in a node does not need to watch for changes.

Storage

Volumes

A volume is a directory that allows data to be stored beyond the lifecycle of a container but not beyond a pod. When a pod is destroyed, so is the volume. This is useful if you wish to share data between containers within a pod.

Persistent Volumes

Persistent Volumes (PV) are pieces of storage in the cluster that have been either manually or dynamically provisioned using Storage Classes, allowing data to outlive the lifecycle of individual pods. PVs are cluster resources, similar to nodes.

Persistent volumes are created when a user makes a Persistent Volume Claim (PVC). These 'claims' specify size, access modes, and other requirements and are used by pods to request storage resources.

PVs support different access modes:

ReadWriteOnce (RWO): Can be mounted as read-write by a single node.
ReadOnlyMany (ROX): The volume can be mounted as read-only by many nodes.
ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes.

Security

Pod Security Standards

There are three tiers of security allowance in Kubernetes:

Privileged: Pods can run with any privilege.
Baseline: Provides several sets of security improvements but does not restrict functionality.
Restricted: Enforces several of the baseline rules halting deployment if they are in violation.

In the restricted tier, specific privileges and configurations are enforced, meaning that if a pod violates these rules, it will not be allowed to deploy. In contrast, the baseline tier provides recommendations, and while it may notify you of potential security improvements, it does not prevent deployment.

To enforce pod security, the following command is used on the relevant namespace:

kubectl annotate namespace example-namespace
  pod-security.kubernetes.io/enforce=restricted

This command sets the Pod Security Admission controller to enforce the restricted level of security for all pods created in the specified namespace.

Security at the Namespace Level

The main way security is enforced at the namespace level is through RBAC (Role-Based Access Control). RBAC is controlled and applied with the following:

Roles: Define a set of permissions for resources in a namespace.
RoleBindings: Associate a user or group with a role.

Example of Roles and RoleBindings

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: example-namespace
  name: example-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: example-rolebinding
  namespace: example-namespace
subjects:
- kind: User
  name: example-user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: example-role
  apiGroup: rbac.authorization.k8s.io

Seccomp

Seccomp is built into the Linux kernel and stands for Secure Computing Mode. It provides a mechanism to limit the attack surface of applications by allowing only a predefined set of system calls.

What are System Calls?

System calls are the primary means of communication between user-space applications and the Linux kernel. When a process is run with a seccomp profile, it can only execute system calls that are allowed by that profile.

Some common system calls include:

File Operations: open(), read(), write(), close().
Process Management: fork(), exec(), wait().
Networking: socket(), bind(), connect().

Seccomp Modes

Seccomp offers two modes:

Strict Mode: Only a limited set of system calls are allowed.
Filtered Mode: More flexible, allowing the use of BPF (Berkeley Packet Filter) to define complex rules for system call filtering.

You can run a node with default profiling by running the kubelet with the --default-seccomp flag. The default profile provides a strong set of security defaults while maintaining the functionality of the workload.

Policies

Policies are configurations that manage other configurations or runtime behaviors. These can be used to manage network traffic, resource allocation, or consumption.

They can be applied using one of the following methods:

Admission Controllers
- Admission controllers run in the API server and can validate or mutate API requests. They intercept requests to the API prior to the persistence of the object but after the request is authenticated and authorized. They limit requests to create, delete, and modify objects.
Validating Admission Policies
- VAPs allow the creation of configurable validation checks using the declarative language CEL (Common Expression Language). For example, they can be used to deny the use of the 'latest' image tag.
Dynamic Admission Control
- Dynamic Admission Controllers run outside the API server as separate applications that receive webhook requests to perform validation or mutation of API requests. They can perform complex checks, including those that require other cluster resources or external data.
Kubelet Configurations
- Some kubelet configurations can act as policies, such as PID limits and reservations or Node Resource Managers.

Useful Commands

It quickly became very repetitive typing kubectl out so frequently so I sought a means of shortening my call to it and came across the ability to create/provide aliases for processes.

Creating an alias

PowerShell:

New-Alias <desired-alias> <command>

Bash:

alias desired-alias="command"

Useful kubectl Commands

kubectl cluster-info Prints info relating to the current or named cluster.
kubectl expose deployment <name> --type=<service_type> Creates a service.
kubectl get <resource> Lists all resources of a given type.
kubectl describe <resource> Shows details of a resource or group of resources.
kubectl create <resource> <name> --image=<image> Creates a deployment.
kubectl apply <path_to_yaml_file> Executes a deployment definition.
kubectl edit <resource_type> <resource> Edits a resource.
kubectl port-forward <type/name> host:container Forwards a port.
kubectl logs <pod_name> Gets logs from a container in a pod.
kubectl exec -ti <pod_name> <command> Executes a commands within a container.
- -t (TTY) Allocates a pseudo-TTY, allowing interaction.
- -i (interactive) Keeps the standard input open.
kubectl label <resource_type> <resource> <label> Adds a label to a resource.

Final Notes

As of this post, aside from the Minikube examples, I have yet to deploy my own application on Kubernetes. However, based on what I have learned over the past few months, I can see how straightforward the process can be. I now understand why Kubernetes is such a popular tool for modern application deployment, given its automated node management, self-healing capabilities, and robust networking features. Additionally, I see how easilyKubernetes integrates with CI/CD tools and Infrastructure as Code (IaC) practices and as such I plan to deploy a previous project on Kubernetes.

Understanding Kubernetes