# Master-Backup Mode

## How it works

Deploy EMQX Neuron on two servers and let Keepalived handle failure detection and failover. If the EMQX Neuron service on the primary node fails, or the whole server goes down, the backup node takes over; when the primary recovers, the role switches back.

![Master-backup topology: Keepalived on the two nodes exchanges VRRP unicast advertisements, and the backup starts EMQX Neuron to take over acquisition and forwarding when the primary fails](./_assets/master-backup-topology.jpg)

::: warning The two nodes are mutually exclusive
**Only one node collects data at a time.** The backup keeps its EMQX Neuron service stopped and starts it only when taking over. Running both at once produces duplicate collection. This is also why a short window of data loss and duplication exists around a switchover — see [Data loss and duplication](#data-loss-and-duplication).
:::

## Environment

| Item | Requirement |
| --- | --- |
| Servers | 2, one primary and one backup |
| Resources per server | At least 1 CPU core and 1 GB memory |
| Operating system | Ubuntu 18.04 or later, or CentOS 7 or later |
| Software | EMQX Neuron (deb, rpm, or Docker) and Keepalived |
| Network | Internal connectivity between the two nodes; security groups and firewalls must allow VRRP and the EMQX Neuron service port |

The examples below use two Ubuntu 22.04 x86_64 servers:

| Role | Internal IP | Interface |
| --- | --- | --- |
| Primary | `10.0.0.127` | `eth0` |
| Backup | `10.0.0.223` | `eth0` |

Replace the IP addresses and interface names in the configuration with your own.

## Installing and configuring EMQX Neuron

### Installation

Install EMQX Neuron on both nodes — see [Installing from a Package](../installation/package.md) or [Docker](../installation/docker.md). Then enable it at boot:

```bash
sudo systemctl enable neuronex
```

### Configuring data collection

Both nodes need an **identical** collection setup so the backup can take over seamlessly.

1. Configure collection on the primary — for example a Modbus TCP southbound driver — and confirm it collects data.
2. Copy `/opt/neuronex/data/` from the primary to the same path on the backup, overwriting what is there. You can also configure the backup by hand.
3. Stop the EMQX Neuron service on the backup to reach the initial state of "primary running, backup standing by":

   ```bash
   sudo systemctl stop neuronex
   ```

::: tip
Configuration is **not synchronized automatically** between the two nodes. When the primary's configuration changes later, sync it manually — see [Configuration file synchronization](#configuration-file-synchronization).
:::

## Keepalived Installation and Configuration

### Keepalived Installation
Install Keepalived on both the master and backup nodes:

```bash
# Install Keepalived
sudo apt-get install keepalived
```


### Keepalived Configuration on Master Node

Create `keepalived.conf`, `master.sh`, `fault.sh`, `check_alive.sh` files in the `/etc/keepalived/` directory of the master node.

1. Configure Keepalived on the master node, the configuration file is `/etc/keepalived/keepalived.conf`, the content is as follows:

```shell
! Configuration File for keepalived
global_defs {
   vrrp_skip_check_adv_addr
   #vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

# Define the script content for the instance execution
vrrp_script check_ex_alived {
        script "/etc/keepalived/check_alive.sh"
        interval 5
        fall 3 # require 3 failures for KO
}


# Define a virtual router instance
vrrp_instance VI_1 {
    # Define the initial state, which can be MASTER or BACKUP
    state MASTER
    # nopreempt
    # Define the working interface
    interface eth0

    virtual_router_id 51
	# Define the weight
    priority 100
	# Define the advert frequency, unit is second
    advert_int 1
	# Define the communication authentication mechanism
    authentication {
        auth_type PASS
        auth_pass abcdefgh
    }

    # Define the virtual VIP address, which is not used
    virtual_ipaddress {
        192.160.127.254/17
    }
    unicast_peer {
        10.0.0.223  # Backup node IP address
    }
    # Track script, usually used to execute the script content defined in the vrrp_script
    track_script {
        check_ex_alived
    }

    # Notify script, which will be executed after the host state becomes Master|Backup|Fault
    notify_fault "/etc/keepalived/fault.sh"
    notify_master "/etc/keepalived/master.sh"
}

```

::: tip

Since the IP address of the backup node in this example is `10.0.0.223`, `unicast_peer` in the keepalived.conf file is `10.0.0.223`, please modify it according to actual conditions.

Since the network card bound to the host IP address `10.0.0.127` is `eth0`, `interface` in the keepalived.conf file is `eth0`. Please modify it according to the actual situation.

:::


2. Configure `master.sh` script on the master node, the configuration file directory is `/etc/keepalived/master.sh`, the content is as follows:

```shell
#!/bin/bash

systemctl start neuronex
```

3. Configure `fault.sh` script on the master node, the configuration file directory is `/etc/keepalived/fault.sh`, the content is as follows:

```shell
#!/bin/bash

systemctl stop neuronex
```

4. Configure `check_alive.sh` script on the master node, the configuration file directory is `/etc/keepalived/check_alive.sh`, the content is as follows:

```shell
#!/bin/bash

if ! curl 127.0.0.1:8085  >/dev/null 2>&1; then echo "neuronex start failed"; exit 1; fi
```

5. Start Keepalived on the master node

```shell
sudo systemctl start keepalived

# Set to start automatically on boot
sudo systemctl enable keepalived
```


### Configuring Keepalived on the backup node

The backup node's `keepalived.conf` is **mostly the same** as the primary's. Copy it over and change the following:

| Setting | Primary | Backup |
| --- | --- | --- |
| `state` | `MASTER` | `BACKUP` |
| `priority` | `100` | `90` |
| `nopreempt` | commented out (preemption on) | enabled |
| `unicast_peer` | `10.0.0.223` (the peer) | `10.0.0.127` (the peer) |
| `vrrp_script` / `track_script` | required | **remove** — the backup does not monitor its own service |
| Notify scripts | `notify_fault` + `notify_master` | `notify_master` + `notify_backup` |

The resulting `vrrp_instance` block on the backup:

```shell
vrrp_instance VI_1 {
    state BACKUP
    nopreempt
    interface eth0
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass abcdefgh
    }
    virtual_ipaddress {
        192.160.127.254/17
    }
    unicast_peer {
        10.0.0.127  # the primary node's IP
    }
    notify_master "/etc/keepalived/master.sh"
    notify_backup "/etc/keepalived/backup.sh"
}
```

The backup needs only two scripts in `/etc/keepalived/`:

| Script | Contents | When it runs |
| --- | --- | --- |
| `master.sh` | `systemctl start neuronex` | The backup is promoted to MASTER and starts serving |
| `backup.sh` | `systemctl stop neuronex` | The backup is demoted to BACKUP and stands down |

Start Keepalived and enable it at boot:

```bash
sudo systemctl start keepalived
sudo systemctl enable keepalived
```

## Master-Backup Switch Logic

After the above configuration steps, both the master and backup nodes have started the Keepalived service, and the master node is in the `MASTER` state, and the backup node is in the `BACKUP` state. The master node EMQX Neuron service is running normally, and the backup node EMQX Neuron service is stopped. When the following situations occur:

1. **Master Node EMQX Neuron Service Failure**

  - Fault detection:

    Keepalived periodically executes the `check_alive.sh` script through vrrp_script to detect the status of the EMQX Neuron service.

    If the `check_alive.sh` script detects that the EMQX Neuron service fails, it returns a failure status.

    Keepalived confirms the service failure within the specified time (15 seconds in this example) based on the interval and fall parameters.

  - Priority adjustment:

    Keepalived lowers the priority of the master node.

    Keepalived executes the `fault.sh` script to stop the master node EMQX Neuron service.

    The master node sends a VRRP announcement, announcing its new priority.

  - Backup node switch:

    The backup node receives the VRRP announcement from the master node and finds that the priority of the master node is lower than its own priority (for example, 0 < 90).

    The backup node switches to the `MASTER` state.

    The backup node executes the `master.sh` script to start the EMQX Neuron service and take over the workload of the master node.


2. **Master Node Server Failure**

  - Fault detection:

    The master node server completely crashes, and both Keepalived and EMQX Neuron stop running.

    The master node cannot send a VRRP announcement, and the backup node cannot receive the status information from the master node.

  - Backup node switch:

    The backup node does not receive the VRRP announcement from the master node within the advert_int * 3 time (3 seconds in this example), and considers the master node to be faulty.

    The backup node automatically switches to the `MASTER` state.

    The backup node executes the `master.sh` script to start the EMQX Neuron service and take over the workload of the master node.


3. **Master Node Recovery**

  - Service recovery:

    The EMQX Neuron service of the master node recovers, and the `check_alive.sh` script detects that the EMQX Neuron is running normally, returning a successful status.

    Keepalived restores the priority of the master node.

    The master node sends a VRRP announcement, announcing its priority.

  - Master node becomes `MASTER`:

    The backup receives the primary's VRRP advertisement, sees that its priority (100) is higher than its own (90), and steps down to `BACKUP`.

    The backup runs `backup.sh` to stop its own EMQX Neuron service.

    The primary becomes `MASTER` again and takes over the workload.

::: tip About preemption
On the primary, `nopreempt` is commented out, so **preemption is on** — once it recovers, its higher priority wins `MASTER` back automatically.

On the backup, `nopreempt` **is** enabled, which stops it from seizing `MASTER` from a primary that is already running when Keepalived starts.

If you would rather the primary not take over automatically — to avoid the data churn of a second switchover — uncomment `nopreempt` on the primary as well.
:::


## Test and Verification

### Simulate Master Node EMQX Neuron Service Failure

1. Stop the master node EMQX Neuron service through the following command:

    ```shell
    sudo systemctl stop neuronex
    ```

2. Check the EMQX Neuron status of the master node and the backup node through the following command:

    ```shell
    sudo systemctl status neuronex
    ```

3. Access the backup node EMQX Neuron Dashboard page, the backup node EMQX Neuron service runs normally, and the southbound driver collects data normally. Access the master node EMQX Neuron Dashboard page, the master node EMQX Neuron service stops.

    - Backup node EMQX Neuron service runs normally
![alt text](_assets/backup_run.png)

    - Master node EMQX Neuron service stops
![alt text](_assets/master_down.png)

### Simulate Master Node Server Failure

1. Turn off the master node server, and check the EMQX Neuron status of the backup node through the following command:

    ```shell
    sudo systemctl status neuronex
    ```

2. Access the backup node EMQX Neuron Dashboard page, the backup node EMQX Neuron service runs normally, and the southbound driver collects data normally. Access the master node EMQX Neuron Dashboard page, the master node EMQX Neuron service stops.


### Master Node Recovery

1. Power on the master node server. Because Keepalived and EMQX Neuron were configured to start automatically in the previous steps, the master node starts both services on boot. Check the EMQX Neuron status of the master node through the following command:

    ```shell
    sudo systemctl status neuronex
    ```

2. Access the master node EMQX Neuron Dashboard page, the master node EMQX Neuron service runs normally.

3. Access the backup node EMQX Neuron Dashboard page, the backup node EMQX Neuron service has stopped.


## Other Notes

### Deployment Mode

This guide deploys with systemd, so the scripts use `systemctl`.

With a Docker deployment, swap the commands in `master.sh`, `backup.sh`, and `fault.sh` for their Docker equivalents:

| Script | systemd | Docker |
| --- | --- | --- |
| `master.sh` | `systemctl start neuronex` | `docker start neuronex` |
| `backup.sh` / `fault.sh` | `systemctl stop neuronex` | `docker stop neuronex` |

### Configuration file synchronization

The master-backup mode does not support automatic synchronization of configuration files between the master and backup EMQX Neuron. If you need the configuration files of the master and backup EMQX Neuron to be consistent, you need to manually synchronize the configuration files.

If the configuration file of the master node EMQX Neuron has changed after running for a period of time, and you do not want to start both the master and backup EMQX Neuron services at the same time (this will cause duplicate data collection), then you can manually synchronize the configuration file while keeping the backup node EMQX Neuron service stopped:

- Package installation Mode:  

    Copy the configuration file `/opt/neuronex/data/` of the master node EMQX Neuron to the same directory of the backup node.

- Docker deployment Mode:

    Copy the configuration file `/opt/neuronex/data/` of the master node EMQX Neuron to the directory mounted to the host by the docker container.


### Data loss and duplication

To build a complete high-availability function for EMQX Neuron, the complete high-availability means that any single EMQX Neuron node failure will not lose or duplicate data. This requires configuring three EMQX Neuron services, implementing data high availability through distributed databases and cluster modes. This method requires a high cost and requires the factory equipment and network to support high availability to be fully effective. In actual factory scenarios, it is often difficult to meet this condition, that is, the PLC does not support master-backup or the factory network does not support master-backup, and there is still a single point of failure, which cannot achieve full data link high availability.

The high-availability solution in the current example still has some data loss or duplication issues.

About the data loss issue, when the EMQX Neuron on the master node fails, keepalived needs some time to detect it, where the detection interval and failure count can be configured, as follows:

```shell

vrrp_script check_ex_alived {
        script "/etc/keepalived/check_alive.sh"
        interval 5
        fall 3 # require 3 failures for KO
}
```

The current configuration is to detect every 5 seconds and trigger the master-backup switch after 3 failures. Therefore, during this period, the EMQX Neuron on both the `MASTER` and `BACKUP` nodes is not running, resulting in short-term data loss.

In addition, about the data duplication issue, after the master node recovers, the backup node detects the master node recovery and stops its own EMQX Neuron, so there will be a short period of time when both nodes' EMQX Neuron are running, causing short-term duplicate data collection.

## Common Issues


- You can use the `ping` command to check if the master and backup nodes are communicating normally.

- You can check the status of the master and backup nodes and whether they are switching normally by viewing the keepalived log.

    ```shell
    journalctl -u keepalived -f
    ``` 

- Master node log example:
![alt text](_assets/master_log.png)


- Backup node log example:
![alt text](_assets/backup_log.png)