MOCO (MySQL Operator for Kubernetes) is a robust, cloud-native solution designed to simplify the management of MySQL clusters in Kubernetes environments. It automates the provisioning, scaling, backup, and maintenance of MySQL instances while ensuring high availability and reliability. MOCO leverages Kubernetes resources to create, monitor, and manage MySQL clusters.

MOCO supports specific versions of MySQL and Kubernetes. As of the latest information, it supports MySQL versions 8.0.28, 8.0.37, 8.0.39, 8.0.40, and 8.4.3, and Kubernetes versions 1.29, 1.30, and 1.31

How MOCO Works

Provisioning Clusters

MOCO provisions MySQL clusters by creating Kubernetes StatefulSets for each cluster. The process involves:
Defining a MySQLCluster custom resource (CR) with desired configurations.
MOCO controller creates StatefulSets and persistent volume claims (PVCs) for the cluster nodes.
Configuring MySQL instances with semi-synchronous replication for high availability.

Deployment Services

MOCO deploys the following services:

Primary Service: Routes traffic to the primary node.
Replica Service: Routes traffic to replica nodes for read queries.
Backup Service: Handles backups through sidecars integrated into the StatefulSet.

Types of Deployment

Single Primary with Replicas: Default mode with one primary and multiple replicas.
Multi-Region Clusters: For cross-region replication.

Backup and restore

MOCO can take full and incremental backups regularly. The backup data are stored in Amazon S3 compatible object storages.
MOCO supports:

Scheduled backups to object storage (e.g., S3).
On-demand backups triggered through the Kubernetes API.
Restorations from backups via simple CR updates.
Point-in-Time Recovery: Ensures robust data protection in disaster recovery scenarios.

Object storage bucket
Bucket is a management unit of objects in S3. MOCO stores backups in a specified bucket.

MOCO does not remove backups. To remove old backups automatically, you can set a lifecycle configuration to the bucket.

Quick setup

You can choose between two installation methods.

MOCO depends on cert-manager. If cert-manager is not installed on your cluster, install it as follows:

Install using raw manifests:
$ curl -fsLO https://github.com/cybozu-go/moco/releases/latest/download/moco.yaml $ kubectl apply -f moco.yaml

Install using Helm chart:
$ helm repo add moco https://cybozu-go.github.io/moco/ $ helm repo update $ helm install --create-namespace --namespace moco-system moco moco/moco

Customize manifests
If you want to edit the manifest, config/ directory contains the source YAML for kustomize.

Creating a Cluster

An empty cluster always has a writable instance called the primary. All other instances are called replicas. Replicas are read-only and replicate data from the primary.

The following YAML is to create a three-instance cluster. It has an anti-affinity for Pods so that all instances will be scheduled to different Nodes. It also sets the limits for memory and CPU to make the Pod Guaranteed.

apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
  namespace: default
  name: test
spec:
  # replicas is the number of mysqld Pods.  The default is 1.
  replicas: 3
  podTemplate:
    spec:
      # Make the data directory writable. If moco-init fails with "Permission denied", uncomment the following settings.
      # securityContext:
      #   fsGroup: 10000
      #   fsGroupChangePolicy: "OnRootMismatch"  # available since k8s 1.20
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - mysql
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - test
            topologyKey: "kubernetes.io/hostname"
      containers:
      # At least a container named "mysqld" must be defined.
      - name: mysqld
        image: ghcr.io/cybozu-go/moco/mysql:8.4.3
        # By limiting CPU and memory, Pods will have Guaranteed QoS class.
        # requests can be omitted; it will be set to the same value as limits.
        resources:
          limits:
            cpu: "10"
            memory: "10Gi"
  volumeClaimTemplates:
  # At least a PVC named "mysql-data" must be defined.
  - metadata:
      name: mysql-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

By default, MOCO uses preferredDuringSchedulingIgnoredDuringExecution to prevent Pods from being placed on the same Node.
There are other example manifests in thier examples directory.

Using the cluster

kubectl moco
From outside of your Kubernetes cluster, you can access MOCO MySQL instances using kubectl-moco. kubectl-moco is a plugin for kubectl. Pre-built binaries are available on GitHub releases.

The following is an example to run mysql command interactively to access the primary instance of test MySQLCluster in foo namespace.

$ kubectl moco -n foo mysql -it test

Connecting to mysqld over network
MOCO prepares two Services for each MySQLCluster. For example, a MySQLCluster named test in foo Namespace has the following Services.

Service Name DNS Name Description moco-test-primary moco-test-primary.foo.svc Connect to the primary instance. moco-test-replica moco-test-replica.foo.svc Connect to replica instances.

Cluster status

You can see the health and availability status of MySQLCluster as follows:

$ kubectl get mysqlcluster NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS test True True 0 3

The cluster is available when the primary Pod is running and ready.
The cluster is healthy when there is no problems.
PRIMARY is the index of the current primary instance Pod.
SYNCED REPLICAS is the number of ready Pods.
ERRANT REPLICAS is the number of instances having errant transactions. You can also use kubectl describe mysqlclusterto see the recent events on the cluster

Logs

Error logs from mysqld can be viewed as follows:

$ kubectl logs moco-test-0 mysqld
Slow logs from mysqld can be viewed as follows:

$ kubectl logs moco-test-0 slow-log

Switchover
Switchover is an operation to change the live primary to one of the replicas.

MOCO automatically switch the primary when the Pod of the primary instance is to be deleted.

Users can manually trigger a switchover with kubectl moco switchover CLUSTER_NAME.

Failover
Failover is an operation to replace the dead primary with the most advanced replica. MOCO automatically does this as soon as it detects that the primary is down.

The most advanced replica is a replica who has retrieved the most up-to-date transaction from the dead primary. Since MOCO configures loss-less semi-synchronous replication, the failover is guaranteed not to lose any user data.

Re-initializing an errant replica
Delete the PVC and Pod of the errant replica, like this:

$ kubectl delete --wait=false pvc mysql-data-moco-test-0 $ kubectl delete --grace-period=1 pods moco-test-0
Depending on your Kubernetes version, StatefulSet controller may create a pending Pod before PVC gets deleted. Delete such pending Pods until PVC is actually removed.

Ref - https://cybozu-go.github.io/moco/index.html

Getting started on MOCO, the MySQL Operator for Kubernetes Part 1