Enable Persistence in EMQX Cluster

Objective

Configure persistent storage for the Core nodes of an EMQX cluster using the persistentVolumeClaimSpec field.

Configure EMQX Cluster Persistence

The .spec.coreTemplate.spec.persistentVolumeClaimSpec field in the apps.emqx.io/v3beta1 EMQX CRD configures persistent storage for each Core node. This field uses the Kubernetes PersistentVolumeClaimSpec schema and semantics.

EMQX Operator 3.0 manages Core nodes with a single StatefulSet. Each Core Pod has a stable identity and a stable PVC across image updates and rolling updates. When you configure this field, EMQX Operator configures the /opt/emqx/data volume of the EMQX container to be backed by a Persistent Volume Claim (PVC), which provisions a Persistent Volume (PV) using a specified StorageClass.

PVC Lifecycle

Core node PVCs are tied to StatefulSet Pod ordinals. For example, the PVC for emqx-core-0 stays attached to emqx-core-0 during image updates and rolling updates, so the node keeps using the same data volume.

EMQX Operator configures Kubernetes to delete Core node PVCs when they are no longer needed:

• When you scale down Core nodes, PVCs for the removed Pod ordinals are deleted.

  For example, scaling from 5 Core replicas to 3 deletes the PVCs for ordinals 3 and 4. Before scaling down the StatefulSet, EMQX Operator rebalances Durable Storage data away from those Core nodes to preserve data and durability.

• When you delete the EMQX custom resource, Kubernetes deletes the Core StatefulSet and its associated PVCs.

• During rolling updates, PVCs are preserved because the StatefulSet name and Pod ordinals do not change.

This automatic cleanup depends on the Kubernetes StatefulSetAutoDeletePVC feature gate, which is enabled by default in Kubernetes 1.27 and later. If your cluster administrator has disabled the feature gate, Kubernetes ignores the deletion policy, and you must clean up unused PVCs manually.

For more details about PVs and PVCs, refer to the Persistent Volumes documentation.

1. Save the following content as a YAML file and deploy it using kubectl apply.

   apiVersion: apps.emqx.io/v3beta1
   kind: EMQX
   metadata:
     name: emqx
   spec:
     image: emqx/emqx:6.3.2
     config:
       roots:
         license:
           key: "..."
     coreTemplate:
       spec:
         persistentVolumeClaimSpec:
           storageClassName: standard
           resources:
             requests:
               storage: 1Gi
           accessModes:
             - ReadWriteOnce
         replicas: 3
     listenersServiceTemplate:
       spec:
         type: LoadBalancer
     dashboardServiceTemplate:
       spec:
         type: LoadBalancer

   TIP

   Use the storageClassName field to choose the appropriate StorageClass for EMQX data. Run kubectl get storageclass to list the StorageClasses that already exist in the Kubernetes cluster, or create a StorageClass according to your needs.

2. Wait for the EMQX cluster to become ready.

   Check the status of the EMQX cluster with kubectl get and ensure that STATUS is Ready. This may take some time.

   $ kubectl get emqx emqx
   NAME   STATUS   AGE
   emqx   Ready    10m

Verify Persistence

Verify that Kubernetes reattaches the same PVC when a Core Pod is replaced. Do not delete the EMQX resource for this test: EMQX Operator configures its StatefulSet to delete associated PVCs when the StatefulSet is deleted.

1. Record the UID of the PVC attached to the first Core Pod:

   pvc_name=emqx-core-data-emqx-core-0
   pvc_uid_before=$(kubectl get pvc "${pvc_name}" -o jsonpath='{.metadata.uid}')
   kubectl get pvc "${pvc_name}"

2. Delete the Pod and wait for the StatefulSet to recreate it:

   kubectl delete pod emqx-core-0
   kubectl wait --for=condition=Ready pod/emqx-core-0 --timeout=10m

3. Compare the PVC UID after the Pod is ready:

   pvc_uid_after=$(kubectl get pvc "${pvc_name}" -o jsonpath='{.metadata.uid}')
   test "${pvc_uid_before}" = "${pvc_uid_after}" && echo "The Core Pod reused the same PVC."

   Matching UIDs confirm that Kubernetes preserved the original PVC, so the replacement Pod continues to use the same persistent volume.