# Performance Tuning (Linux)

Due to the typically large number of devices and data in IoT applications, EMQX, as an MQTT server, is responsible for handling and delivering messages generated by a massive number of devices. In this scenario, optimizing EMQX system performance becomes particularly crucial.

Optimization aims to maximize the following aspects of performance:

- **Message Processing Capability**: Enhancing EMQX's ability to process messages quickly and efficiently, ensuring it can rapidly receive, process, and forward device-generated messages.
- **Throughput**: Increasing throughput to ensure the system can handle and deliver messages from devices in a timely manner.
- **Stability**: Reducing latency under high loads, improving system responsiveness, and lowering the risk of system crashes or failures.

This page provides general tuning suggestions for benchmarking and deployment.

## Turn Off Swap

Linux swap partitions may cause nondeterministic memory latency to an Erlang virtual machine, significantly affecting the system stability.
It is recommended to turn off the swap permanently.

- To turn off swap immediately, execute the command `sudo swapoff -a`.

- To turn off swap permanently, comment out the `swap` line in `/etc/fstab` and reboot the host.

## Linux Kernel Tuning

The system-wide limit on max opened file handles:

```bash
# 2 millions system-wide
sysctl -w fs.file-max=2097152
sysctl -w fs.nr_open=2097152
echo 2097152 > /proc/sys/fs/nr_open
```

The limit on opened file handles for the current session:

```bash
ulimit -n 2097152
```

### `/etc/sysctl.conf`

Persist `fs.file-max` configuration to `/etc/sysctl.conf`:

```bash
fs.file-max = 2097152
```

Set the maximum number of file handles for the service in `/etc/systemd/system.conf`:

```bash
DefaultLimitNOFILE=2097152
```

### `emqx.service`

Set the maximum number of file handles for emqx service in one of the below paths depending on which Linux distribution is used.

- `/usr/lib/systemd/system/emqx.service`
- `/lib/systemd/system/emqx.service`

```bash
LimitNOFILE=2097152
```

### `/etc/security/limits.conf`

Persist the maximum number of opened file handles for users in `/etc/security/limits.conf`:

```bash
*      soft   nofile      2097152
*      hard   nofile      2097152
```

### Disable Transparent HugePages (THP)

EMQX includes built-in database workloads. As with other database systems, it is strongly recommended to disable Transparent HugePages (THP) before starting EMQX.

```bash
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
```

If you experience the following symptoms after running EMQX for a long period on a high-memory machine (>16 GB), disable THP to rule out THP-related issues:

- Unstable message latency.
- Unexpected memory usage spikes.
- EMQX `long_schedule` warning logs.
- EMQX `runq_overload` alarms.

If you are running a cluster, disable THP on a subset of nodes first for comparison. Note that some workloads may benefit from having THP enabled.

To make these changes persistent across reboots, consult your OS documentation for the appropriate method.

## TCP Network Tuning

Increase the number of incoming connections backlog:

```bash
sysctl -w net.core.somaxconn=32768
sysctl -w net.ipv4.tcp_max_syn_backlog=16384
sysctl -w net.core.netdev_max_backlog=16384
```

Local port range

```bash
sysctl -w net.ipv4.ip_local_port_range='1024 65535'
```

TCP Socket read/write buffer:

```bash
sysctl -w net.core.rmem_default=262144
sysctl -w net.core.wmem_default=262144
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.core.optmem_max=16777216

#sysctl -w net.ipv4.tcp_mem='16777216 16777216 16777216'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16777216'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16777216'
```

TCP connection tracking:

```bash
sysctl -w net.nf_conntrack_max=1000000
sysctl -w net.netfilter.nf_conntrack_max=1000000
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
```

TIME-WAIT Bucket Pool, Recycling, and Reuse:

```bash
sysctl -w net.ipv4.tcp_max_tw_buckets=1048576

# Enabling following option is not recommended. It could cause connection reset under NAT
# sysctl -w net.ipv4.tcp_tw_recycle=1
# sysctl -w net.ipv4.tcp_tw_reuse=1
```

Timeout for FIN-WAIT-2 Sockets:

```bash
sysctl -w net.ipv4.tcp_fin_timeout=15
```

Reduce TCP packet retransmission count:

```bash
sysctl -w net.ipv4.tcp_retries2=5
```

## Erlang VM Tuning

Starting from EMQX 6.3.0, EMQX automatically sets Erlang VM resource limits based on the CPU resources available to the node. Configure the following settings in `etc/emqx.conf`. The settings take effect after the node restarts.

### Port and Process Limits

`node.max_ports` controls the maximum number of files and sockets that the Erlang VM can open simultaneously. The default value is `auto`, which sets the Erlang VM port limit (`+Q`) as follows:

- On nodes with 1 to 8 available logical CPUs, the limit is 65,536 ports per CPU.
- On nodes with more than 8 available logical CPUs, the limit is 1,048,576 ports.

::: warning Important Notice
When upgrading from an earlier EMQX version, nodes with 8 or fewer available logical CPUs will start with a lower port limit. If your deployment requires more connections than the automatically calculated limit supports, explicitly set `node.max_ports` and restart the node before upgrading.
:::

EMQX sets the Erlang process limit (`+P`) to twice the resolved `node.max_ports` value. If you explicitly configure `node.process_limit`, only a value greater than the derived process limit takes effect.

If the automatically calculated port limit does not meet the connection requirements of a high-concurrency workload, set `node.max_ports` explicitly. For example:

```hocon
node.max_ports = 2097152
```

Before increasing `node.max_ports`, ensure that the operating system file descriptor limit and available memory can support the configured value. You can view the effective port and process limits on the node monitoring page in EMQX Dashboard.

### Erlang Schedulers

`node.schedulers` controls the number of Erlang schedulers through the Erlang VM `+S` flag. The default value is `auto`, which uses the number of logical processors available to the Erlang VM, including the CPU resources available to a container.

Set `node.schedulers` to a positive integer only when you need to override the detected value, for example, to reserve CPU capacity for other workloads on the same host.

## EMQX Tuning

### Listener Acceptor

Tune the acceptor pool size and `max_connections` limit in `etc/base.hocon`.

To optimize connection handling, you can adjust the acceptor pool size and the `max_connections` limit in the `etc/emqx.conf` configuration file.

For example, to configure TCP listeners:

```bash
## TCP Listener
listeners.tcp.$name.acceptors = 64
listeners.tcp.$name.max_connections = 1024000
```

`acceptors`: The number of acceptor processes handling incoming connections.

`max_connections`: The maximum number of concurrent connections allowed.

### Distribution Port Buffer Size

In large clusters with many replicant nodes, it is suggested to tune the distribution port buffer size by setting the `node.dist_buffer_size` parameter on core nodes.

```bash
# Buffer size in KB. The following sets the maximum to approximately 2 GB.
node.dist_buffer_size=2097151
```

This adjustment helps core nodes better handle traffic spikes caused by mass client reconnections.

Additionally, if you observe warning log messages like the following, increasing this buffer size can mitigate the issue:

```
[warning] msg: busy_dist_port ...
```


## Client Machine Tuning

Tune the client machine to benchmark EMQX:

```
sysctl -w net.ipv4.ip_local_port_range="500 65535"
echo 1000000 > /proc/sys/fs/nr_open
ulimit -n 100000
```


### MQTT Benchmarking

Test tools for concurrent connections: [emqtt_bench](https://github.com/emqx/emqtt_bench).