# Perform a Rolling Update of an EMQX Cluster Use EMQX Operator 3.0 to perform a graceful rolling update of an EMQX cluster. ## Before You Begin - [Install EMQX Operator 3.0](../getting-started.md) and configure `kubectl` to access the Kubernetes cluster. - Make sure that the Kubernetes environment can provision an external address for a `LoadBalancer` Service. - Prepare a valid EMQX Enterprise license that supports at least 3,000 concurrent connections. Replace the license placeholder in this example with your license key. For more information, see [Manage License](./configure-emqx-license.md). - Install `jq` to inspect the node evacuation status. - Install [MQTTX CLI](https://mqttx.app/cli) to generate test connections. - For a Core-Replicant cluster, configure at least two Core replicas. EMQX Operator rejects a Core-Replicant configuration with fewer than two Core replicas. ## How Rolling Updates Work EMQX Operator performs a rolling update when a change to an EMQX custom resource modifies a Pod template, such as the image, image pull policy, resource requests, or Core and Replicant templates. The rollout proceeds in the following stages: 1. **Update Core nodes:** The Operator updates one Core Pod at a time in the same StatefulSet. It recreates each Pod with the desired template and waits until the Pod is ready before continuing. The Replicant rollout can start after at least one updated Core node is ready. 2. **Update Replicant nodes:** The Operator uses a Deployment-style rollout, creating updated Replicant Pods up to `maxSurge` and evacuating outdated Replicant Pods up to `maxUnavailable`. These limits control rollout speed and availability. The Operator keeps at least one Core node from the previous revision until the Replicant Pods from that revision have migrated, so each revision has a Core node available during the transition. 3. **Complete the rollout:** After the Replicant migration, the Operator finishes updating the Core nodes. The rollout completes when all desired Pods are ready and no outdated Replicant Pods remain. By default, the Operator evacuates MQTT connections and sessions at configured rates before removing a Pod. To disable node evacuation, set `.spec.updateStrategy.evacuationStrategy.type` to `Disabled`. The following procedure changes Pod template annotations to demonstrate this mechanism without upgrading EMQX. To upgrade EMQX, update `.spec.image`; the same rolling update mechanism applies. ## Configure the Update Strategy 1. Create an `apps.emqx.io/v3beta1` EMQX CR and configure the update strategy. ```yaml apiVersion: apps.emqx.io/v3beta1 kind: EMQX metadata: name: emqx spec: image: emqx/emqx:6.3.2 config: roots: license: key: "..." updateStrategy: type: RollingUpdate evacuationStrategy: # MQTT client evacuation rate, connections per second: connectionEvictionRate: 1000 # MQTT Session evacuation rate, sessions per second: sessionEvictionRate: 1000 # Time to wait before deleting a Pod: waitTakeover: 10 replicants: maxUnavailable: 1 maxSurge: 1 coreTemplate: spec: replicas: 2 replicantTemplate: spec: replicas: 3 listenersServiceTemplate: spec: type: LoadBalancer ``` ::: warning `maxUnavailable` and `maxSurge` cannot both be `0`. If `maxUnavailable` is `100%`, `maxSurge` must be greater than `0`. For field definitions, defaults, and validation rules, see [ReplicantsUpdateStrategy](../reference/v3beta1-reference.md#replicantsupdatestrategy). ::: 2. Save the above content as `emqx-update.yaml` and deploy it using `kubectl apply`: ```bash $ kubectl apply -f emqx-update.yaml emqx.apps.emqx.io/emqx created ``` 3. Check the status of the EMQX cluster. Make sure that `STATUS` is `Ready`. This may take a while. ```bash $ kubectl get emqx NAME STATUS AGE emqx Ready 8m33s ``` ## Establish Test Connections Use MQTTX CLI to establish MQTT connections for observing connection evacuation during the rolling update. MQTTX CLI supports automatic reconnection. Get the external address of the `emqx-listeners` Service. The command supports load balancers that publish either an IP address or a hostname. ```bash export EMQX_HOST="$(kubectl get service emqx-listeners -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')" ``` After `EMQX_HOST` contains an address, use MQTTX to open 3,000 connections to the default TCP listener on port `1883`: ```bash mqttx bench conn -h "${EMQX_HOST}" -p 1883 -c 3000 [10:05:21 AM] › ℹ Start the connect benchmarking, connections: 3000, req interval: 10ms ✔ success [3000/3000] - Connected [10:06:13 AM] › ℹ Done, total time: 31.113s ``` ## Trigger the Update 1. Update an annotation in the Core and Replicant Pod templates to trigger a rolling update. The command combines a timestamp with a random suffix to reduce the chance of reusing the same annotation value. ```bash ROLLOUT_ID="$(date +%s)-${RANDOM}" kubectl patch emqx emqx --type=merge -p \ "{\"spec\":{\"coreTemplate\":{\"metadata\":{\"annotations\":{\"docs.emqx.com/rollout-id\":\"${ROLLOUT_ID}\"}}},\"replicantTemplate\":{\"metadata\":{\"annotations\":{\"docs.emqx.com/rollout-id\":\"${ROLLOUT_ID}\"}}}}}" ``` Expected output: ```text emqx.apps.emqx.io/emqx patched ``` 2. Check the status of the update process. ```bash $ kubectl get emqx emqx -o json | jq ".status.nodeEvacuations" [ { "nodeName": "emqx@10.244.4.56", "initialConnections": 33, "initialSessions": 0, "connectionEvictionRate": 200, "sessionEvictionRate": 200, "state": "waiting_takeover", "sessionRecipients": [ "emqx@10.244.4.57", "emqx@10.244.4.58" ] } ] ``` | Field | Description | |-------------------------|----------------------------------------------------------------------| | `nodeName` | The node currently being evacuated. | | `state` | Node evacuation phase. | | `sessionRecipients` | MQTT session recipients. | | `sessionEvictionRate` | MQTT session eviction rate on this node (sessions per second). | | `connectionEvictionRate`| MQTT connection eviction rate on this node (connections per second). | | `initialSessions` | Initial number of sessions on this node. | | `initialConnections` | Initial number of connections on this node. | Estimate node evacuation progress by checking the `connections` and `sessions` counters in the corresponding [EMQX node status](../reference/v3beta1-reference.md#emqxnode). 3. Wait for the update to complete. ```bash $ kubectl get emqx NAME STATUS AGE emqx Ready 8m33s ``` Make sure that the `STATUS` is `Ready`. Depending on the number of MQTT clients and sessions, the update process may take a while. After the update is completed, you can verify that all Pods are running the desired template using `kubectl get pods`. ## Rolling Update Illustration The following diagram illustrates a possible connection distribution during a Replicant rolling update with 3,000 connections. It is not based on measured Grafana data. 0 600 1,200 1,800 2,400 3,000 Update progress Total 3,000 emqx-replicant-86f864f9-0 emqx-replicant-86f864f9-1 emqx-replicant-86f864f9-2 emqx-replicant-648c45c7-0 emqx-replicant-648c45c7-1 emqx-replicant-648c45c7-2 Illustrative Replicant rolling update, maxSurge = 1, maxUnavailable = 1 | Label/Prefix | Description | |----------------------|-----------------------------------------------------| | Total | Example total connection count, shown as the top line in the diagram. | | `emqx-replicant-86f864f9` | Example name prefix for the set of outdated Replicant Pods. | | `emqx-replicant-648c45c7` | Example name prefix for the set of updated Replicant Pods. | During a Replicant rollout, clients disconnected from an evacuated Pod may reconnect to other serving Pods, including updated Pods. The lines illustrate this redistribution only; they are not measured values. Actual connection counts can fluctuate based on the node evacuation rate, available cluster capacity, and client reconnection behavior.