Perform a Rolling Update of an EMQX Cluster

Use EMQX Operator 3.0 to perform a graceful rolling update of an EMQX cluster.

Before You Begin

• Install EMQX Operator 3.0 and configure kubectl to access the Kubernetes cluster.
• Make sure that the Kubernetes environment can provision an external address for a LoadBalancer Service.
• Prepare a valid EMQX Enterprise license that supports at least 3,000 concurrent connections. Replace the license placeholder in this example with your license key. For more information, see Manage License.
• Install jq to inspect the node evacuation status.
• Install MQTTX CLI to generate test connections.
• For a Core-Replicant cluster, configure at least two Core replicas. EMQX Operator rejects a Core-Replicant configuration with fewer than two Core replicas.

How Rolling Updates Work

EMQX Operator performs a rolling update when a change to an EMQX custom resource modifies a Pod template, such as the image, image pull policy, resource requests, or Core and Replicant templates. The rollout proceeds in the following stages:

1. Update Core nodes: The Operator updates one Core Pod at a time in the same StatefulSet. It recreates each Pod with the desired template and waits until the Pod is ready before continuing. The Replicant rollout can start after at least one updated Core node is ready.
2. Update Replicant nodes: The Operator uses a Deployment-style rollout, creating updated Replicant Pods up to maxSurge and evacuating outdated Replicant Pods up to maxUnavailable. These limits control rollout speed and availability. The Operator keeps at least one Core node from the previous revision until the Replicant Pods from that revision have migrated, so each revision has a Core node available during the transition.
3. Complete the rollout: After the Replicant migration, the Operator finishes updating the Core nodes. The rollout completes when all desired Pods are ready and no outdated Replicant Pods remain.

By default, the Operator evacuates MQTT connections and sessions at configured rates before removing a Pod. To disable node evacuation, set .spec.updateStrategy.evacuationStrategy.type to Disabled.

The following procedure changes Pod template annotations to demonstrate this mechanism without upgrading EMQX. To upgrade EMQX, update .spec.image; the same rolling update mechanism applies.

Configure the Update Strategy

1. Create an apps.emqx.io/v3beta1 EMQX CR and configure the update strategy.

apiVersion: apps.emqx.io/v3beta1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx/emqx:6.3.2
  config:
    roots:
      license:
        key: "..."
  updateStrategy:
    type: RollingUpdate
    evacuationStrategy:
      # MQTT client evacuation rate, connections per second:
      connectionEvictionRate: 1000
      # MQTT Session evacuation rate, sessions per second:
      sessionEvictionRate: 1000
      # Time to wait before deleting a Pod:
      waitTakeover: 10
    replicants:
      maxUnavailable: 1
      maxSurge: 1
  coreTemplate:
    spec:
      replicas: 2
  replicantTemplate:
    spec:
      replicas: 3
  listenersServiceTemplate:
    spec:
      type: LoadBalancer

WARNING

maxUnavailable and maxSurge cannot both be 0. If maxUnavailable is 100%, maxSurge must be greater than 0. For field definitions, defaults, and validation rules, see ReplicantsUpdateStrategy.

2. Save the above content as emqx-update.yaml and deploy it using kubectl apply:

$ kubectl apply -f emqx-update.yaml
emqx.apps.emqx.io/emqx created

3. Check the status of the EMQX cluster.

Make sure that STATUS is Ready. This may take a while.

$ kubectl get emqx
NAME      STATUS   AGE
emqx      Ready    8m33s

Establish Test Connections

Use MQTTX CLI to establish MQTT connections for observing connection evacuation during the rolling update. MQTTX CLI supports automatic reconnection.

Get the external address of the emqx-listeners Service. The command supports load balancers that publish either an IP address or a hostname.

export EMQX_HOST="$(kubectl get service emqx-listeners -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')"

After EMQX_HOST contains an address, use MQTTX to open 3,000 connections to the default TCP listener on port 1883:

mqttx bench conn -h "${EMQX_HOST}" -p 1883 -c 3000
[10:05:21 AM] › ℹ  Start the connect benchmarking, connections: 3000, req interval: 10ms
✔  success   [3000/3000] - Connected
[10:06:13 AM] › ℹ  Done, total time: 31.113s

Trigger the Update

1. Update an annotation in the Core and Replicant Pod templates to trigger a rolling update. The command combines a timestamp with a random suffix to reduce the chance of reusing the same annotation value.

ROLLOUT_ID="$(date +%s)-${RANDOM}"

kubectl patch emqx emqx --type=merge -p \
  "{\"spec\":{\"coreTemplate\":{\"metadata\":{\"annotations\":{\"docs.emqx.com/rollout-id\":\"${ROLLOUT_ID}\"}}},\"replicantTemplate\":{\"metadata\":{\"annotations\":{\"docs.emqx.com/rollout-id\":\"${ROLLOUT_ID}\"}}}}}"

Expected output:

emqx.apps.emqx.io/emqx patched

2. Check the status of the update process.

$ kubectl get emqx emqx -o json | jq ".status.nodeEvacuations"
[
  {
    "nodeName": "emqx@10.244.4.56",
    "initialConnections": 33,
    "initialSessions": 0,
    "connectionEvictionRate": 200,
    "sessionEvictionRate": 200,
    "state": "waiting_takeover",
    "sessionRecipients": [
      "emqx@10.244.4.57",
      "emqx@10.244.4.58"
    ]
  }
]

Field	Description
nodeName	The node currently being evacuated.
state	Node evacuation phase.
sessionRecipients	MQTT session recipients.
sessionEvictionRate	MQTT session eviction rate on this node (sessions per second).
connectionEvictionRate	MQTT connection eviction rate on this node (connections per second).
initialSessions	Initial number of sessions on this node.
initialConnections	Initial number of connections on this node.

Estimate node evacuation progress by checking the connections and sessions counters in the corresponding EMQX node status.

3. Wait for the update to complete.

$ kubectl get emqx
NAME      STATUS   AGE
emqx      Ready    8m33s

Make sure that the STATUS is Ready. Depending on the number of MQTT clients and sessions, the update process may take a while.

After the update is completed, you can verify that all Pods are running the desired template using kubectl get pods.

Rolling Update Illustration

The following diagram illustrates a possible connection distribution during a Replicant rolling update with 3,000 connections. It is not based on measured Grafana data.

Label/Prefix	Description
Total	Example total connection count, shown as the top line in the diagram.
emqx-replicant-86f864f9	Example name prefix for the set of outdated Replicant Pods.
emqx-replicant-648c45c7	Example name prefix for the set of updated Replicant Pods.

During a Replicant rollout, clients disconnected from an evacuated Pod may reconnect to other serving Pods, including updated Pods. The lines illustrate this redistribution only; they are not measured values. Actual connection counts can fluctuate based on the node evacuation rate, available cluster capacity, and client reconnection behavior.