EMQX Docs

English

简体中文

English

简体中文

Appearance

Contact Us

Create and Manage Cluster ​

You can create an EMQX cluster either manually or automatically. This page will introduce you to both manual and automatic clustering methods and guide you in creating and managing EMQX clusters using these two different approaches.

Basic Concepts ​

You can learn about the basic knowledge of the EMQX cluster and how it works in Cluster and Architecture. To create a cluster, you also need to be familiar with the following concepts:

Node Name ​

EMQX nodes are identified by their names. All nodes are set with a unique node name in the format of name@host, where the host must be an IP address or fully qualified domain name (FQDN). For example:

  • For EMQX node deployed on server s1.emqx.io, the node name should be emqx@s1.emqx.io;
  • If this server has a static IP (192.168.0.10), the node name should be emqx@192.168.0.10.

TIP

EMQX node names are immutable, as they are baked into the database schema and data files. Therefore, it is recommended to use static FQDNs for EMQX node names.

Node Discovery ​

Node discovery is a necessary process in creating a cluster, allowing individual EMQX nodes to discover each other and communicate, regardless of their location or IP address.

Manual and Auto Clustering ​

Based on the node discovery strategies, the way to create clusters can be divided into manual clustering and automatic clustering.

Manual clustering is the method that creates an EMQX cluster by manually specifying which nodes should be part of the cluster. Auto clustering is another method that allows multiple EMQX nodes to form a cluster automatically without manual configuration. Auto clustering simplifies the process of setting up an EMQX cluster and makes it easier to add or remove nodes from the cluster dynamically. EMQX supports auto clustering based on static node list, DNS Record, etcd, and Kubernetes.

The following table shows different node discovery strategies and cluster creation methods supported by EMQX:

StrategyDescription
manualManually create a cluster with commands
staticAutocluster through static node list
dnsAutocluster through DNS A and SRV records
etcdAutocluster through etcd
k8sAutocluster provided by Kubernetes
singletonClustering is disabled. The node rejects all connection attempts to and from other nodes.

EMQX supports automatic cluster creation based on the Ekka library. Ekka is a cluster management library developed for Erlang/OTP applications. Except for automatic discovery of Erlang nodes (Service Discovery) and automatic clustering (Autocluster), it also implements functionalities such as automatic healing of network partitions (Network Partition Autoheal) and automatic removal of downed nodes (Autoclean).

You can define the way of clustering by configuring the node discovery strategy in the emqx.conf configuration file. The manual clustering is configured by default.

bash
cluster {
    ## Options: manual | static | dns | etcd | k8s | singleton
    discovery_strategy  =  manual
}

Before You Start ​

This section provides you with the guidance about how to configure nodes and network environments before creating a cluster.

Configure Node Names ​

Before creating a cluster, you need to know how to name the nodes to join the cluster. Suppose you want to create a cluster for 2 nodes deployed in s1.emqx.io and s2.emqx.io respectively, you can follow the steps below to create the cluster.

Configure the node name in the emqx.conf configuration file of the 1st node, for example:

bash
node.name = emqx@s1.emqx.io

You can also override the node name with an environment variable. For example in docker run command's -e option, or systemd's emqx.service file, define the environment variable as below:

bash
EMQX_NODE__NAME='emqx@s1.emqx.io'

Repeat the above step for the other node to join the cluster.

Now you have named 2 nodes to join the cluster, emqx@s1.emqx.io and emqx@s2.emqx.io. You can create a cluster either manually or automatically.

Set Node Cookies ​

For security concerns, you should change the default cookie settings to a Secret cookie in emqx.conf on all nodes to join the cluster. All nodes to join the cluster should use the same Secret cookie. For details about the magic cookie used, see Distributed Erlang - Security.

node {
  cookie = "<a Secret cookie>"
}

Configure Network Environment ​

Ensure that the network connections between nodes are functioning properly. If there is a firewall or security group between nodes, you need to open the ports for internal cluster communication, including:

  • 4370: Erlang distributed transport port
  • 5370: Cluster RPC port, suitable for physical machine environments
  • 5369: Cluster RPC port, suitable for Docker environments

If multiple EMQX nodes are deployed on a single server, each node will use different cluster communication ports. For details on firewall configurations, see Intra-cluster communication port.

Quick Start ​

This section demonstrates how to quickly create a cluster in a Docker network using two different clustering methods:

TIP

If you plan to run EMQX on Docker environments across multiple physical machines and form a cluster, additional setup will be required. Please refer to Configure Network Environment to map the necessary cluster communication ports in the container and ensure these ports are open in the firewall.

Execute the emqx ctl cluster status command on any node to view the cluster status. If the cluster status is normal, the following information will be output:

bash
$ docker exec -it emqx1 emqx ctl cluster status
Cluster status: #{running_nodes =>
                    ['emqx@node1.emqx.com','emqx@node2.emqx.com'],
                stopped_nodes => []}

Now you have completed a simple cluster creation process. Next, you can modify and deploy according to the instructions in the following sections to select the cluster creation method you need.

Manual Clustering ​

This section explains the procedure of creating a cluster manually. During the manual clustering process, you must manually configure each node in the cluster, including setting up network connections between the nodes. Compared to auto clustering, manual clustering allows for finely-tuned configuration of custom network topologies and is very suitable in situations where auto clustering mechanisms are unavailable or inappropriate.

TIP

Manual clustering can only be used for core nodes. If you are using a core-replica node deployment architecture, please use auto clustering to manage the cluster.

Suppose you have two nodes, emqx@node1.emqx.com and emqx@node2.emqx.com. You can manually create a cluster for them through the following steps:

  1. Set the cluster discovery strategy to manual:

    bash
    cluster {
    ## Options: manual | static | dns | etcd | k8s | singleton
    discovery_strategy  =  manual
}

  2. After starting the two nodes, execute the cluster join command on one of the nodes:

    bash
    $ ./bin/emqx ctl cluster join emqx@node1.emqx.com

Join the cluster successfully.
Cluster status: [{running_nodes,['emqx@node1.emqx.com','emqx@node2.emqx.com']}]

    TIP

    • This command must run on the node to join the cluster, that is, as a request rather than invite.

    • After emqx@s2.emqx.io joins emqx@s1.emqx.io to form a cluster, it will clear the local data and synchronize the data in emqx@s1.emqx.io.

    • If emqx@s2.emqx.io wants to join another cluster, it must first leave the current cluster. On how to leave the cluster, see Leave Cluster.

  3. Query the status of the cluster on any node:

    bash
    $ ./bin/emqx ctl cluster status

Cluster status: [{running_nodes,['emqx@node1.emqx.com','emqx@node2.emqx.com']}]

Now you have successfully created a cluster with two nodes, you can read the Query Cluster Status, Manage Cluster Nodes, and Configure Network Protocols sections on how to monitor the cluster status and how to manage the cluster.

Auto Clustering ​

This section explains how to create a cluster automatically by various auto-clustering methods.

Autocluster by Static Node List ​

In EMQX, autocluster by static node list is to use a pre-defined static node list on each node to join the cluster. After starting, the nodes will create a cluster automatically according to the node list.

Static clustering is the easiest way to create an EMQX cluster automatically with no dependencies on other network components or services. As long as each node can communicate with each other through the TCP protocol, they can form an EMQX cluster.

To enable this feature, configure the cluster mode and node list in emqx.conf:

Example code:

bash
cluster {
    discovery_strategy = static
    static {
        seeds = ["emqx@s1.emqx.io", "emqx@s2.emqx.io"]
    }
}

Where,

  • discovery_strategy is the node discovery strategy, set it to static.
  • seeds is an array, where you can add the node to join the cluster, multiple nodes can be separated with ,.

After all nodes are started, the cluster will be automatically established.

Autocluster by DNS Records ​

DNS is short for Domain Name System. When a DNS server receives a domain name query request, it returns the corresponding IP address of that domain name, which is the so-called A (Address) record. DNS allows a domain name to have multiple A records, i.e., multiple IP addresses, thus forming a mapping where one name corresponds to multiple IP addresses. EMQX's DNS auto clustering utilizes this one-to-many mapping to locate all the nodes in the cluster, allowing each independent node to join the cluster.

Configure DNS Services ​

Most public cloud services have DNS services. After assigning a domain name, you only need to add the IP address of each EMQX node to the A record of this domain to finish the configuration. If EMQX is deployed in a private cloud or internal network, you will need to deploy your own DNS system, for example, with software BIND.

Configure Autocluster by DNS Records ​

After the DNS service is ready, you can add all nodes to join the cluster in emqx.conf with the cluster.dns configuration item:

Example code:

bash
cluster {
    discovery_strategy = dns
    dns {
        name = "localhost"
        ## support DNS A record and DNS SRV record
        record_type = a
    }
}

Where,

  • discovery_strategy is the node discovery strategy, set it to dns.
  • cluster.dns.name is a a string, input the localhost.
  • cluster.dns.record_type is a enum, optional value: a or srv.

After all nodes are started, the cluster will be automatically established.

Autocluster Using etcd ​

etcd is an open-source project initiated by CoreOS. It is widely used in distributed systems for service discovery and connection establishing, which is exactly what EMQX auto clustering needs.

After you deploy an etcd server (cluster) in your network, EMQX can automatically create the cluster via etcd. For how to install and configure etcd, see etcd Install.

To enable autocluster using etcd, you can work with the cluster.etcd configuration items in emqx.conf.

Example code:

bash
cluster {
    discovery_strategy = etcd
    etcd {
        server = "http://127.0.0.1:2379"
        prefix = emqxcl
        node_ttl = 1m
    }
}

Where:

  • discovery_strategy is the node discovery strategy, set it to etcd.
  • cluster.etcd.server is the server address, multiple nodes can be separated with ,.
  • cluster.etcd.prefix is the etcd key prefix used for EMQX service discovery.
  • cluster.etcd.node_ttl is a duration, indicating the expiration time of the etcd key associated with the node, default: 1m.

After completing the configuration, you can start the EMQX nodes one by one, and use the etcdctl tool to observe the changes on the etcd server:

bash
$ etcdctl ls /emqxcl/emqxcl --recursive

/emqxcl/emqxcl/nodes
/emqxcl/emqxcl/nodes/emqx@s1.emqx.io
/emqxcl/emqxcl/nodes/emqx@s2.emqx.io

The result shows that all nodes are started normally and joined the cluster automatically.

Autocluster on Kubernetes ​

The EMQX Kubernetes Operator helps you quickly create and manage EMQX clusters on a Kubernetes environment quickly, greatly simplifying the EMQX cluster deployment and management process by turning deployment and management efforts into a low-cost, labeled, repeatable job.

If you want to deploy and manage EMQX by yourself, you can still use Kubernetes API for node discovery and auto clustering. To use this feature, you need first to create RBAC for the EMQX Pod to allow EMQX to get cluster node information from the Kubernetes APIServer via the endpoints resource. On how to configure, see Using RBAC Authorization.

To enable EMQX autocluster on Kubernetes, you can work with the cluster.k8s configuration item in emqx.conf.

bash
cluster {
    discovery_strategy = k8s
    K8s {
        apiserver = "http://10.110.111.204:8080"
        service_name = emqx
        address_type = ip
        namespace = default
    }
}

Where:

  • discovery_strategy is the node discovery strategy, set it to k8s
  • cluster.K8s.apiserver is the Kubernetes API endpoint URL, default: http://10.110.111.204:8080
  • cluster.K8s.service_name is the EMQX service name, default: emqx
  • cluster.K8s.address_type is the address type to connect the discovered nodes, default: ip, optional values: ip, dns, hostname
  • [optional] cluster.K8s.suffix is the node name suffix, only needed when cluster.K8s.address_type is set to dns, default: pod.local
  • cluster.K8s.namespace is the Kubernetes namespace, it is a string object, default: default

Start all nodes one by one after the configuration, and the cluster will be automatically established.

TIP

When working EMQX autocluster on Kubernetes, Calico rather than Fannel plugin is recommended.

Manage Cluster ​

After you create a cluster, you can monitor the cluster status and manage the cluster nodes.

Query Cluster Status ​

Run the command below on any cluster node to query the cluster status:

bash
$ ./bin/emqx ctl cluster status

Cluster status: [{running_nodes,['emqx@s1.emqx.io','emqx@s2.emqx.io']}]

Leave Cluster ​

There are two ways to leave a cluster:

  1. Run the cluster leave command: This lets the current node leave the cluster. It notifies the other nodes in the cluster and stops participating in cluster operations. The node will complete any ongoing tasks before leaving.
  2. Run the cluster force-leave <node@host> command: This removes a node from the cluster. The target node will be forcefully removed from the cluster. This command is typically used when a node fails or becomes unresponsive.

For example, in the previously built cluster, if emqx@s2.emqx.io wants to leave the cluster, you can run the command below on emqx@s2.emqx.io:

bash
./bin/emqx ctl cluster leave

Or run the command below on emqx@s1.emqx.io to remove emqx@s2.emqx.io from the cluster:

bash
./bin/emqx ctl cluster force-leave emqx@s2.emqx.io

Configure Network Protocols ​

After the cluster is created, you can continue to set the network protocols for the nodes. EMQX supports connecting the nodes via TCP or TLS. The connection method can be configured in emqx.conf:

To use TCP IPv4 and TCP IPv6, you can set with the cluster.proto_dist in emqx.conf.

  • TCP IPv4: inet_tcp (Default)
  • TCP IPv6: inet6_tcp

To enable SSL, you first need to set the cluster.proto_dist to inet_tls, then configure the ssl_dist.conf file in the etc folder and specify the TLS certificate. For details, see Using TLS for Erlang Distribution.