Configuration Files

Users can configure EMQX with configuration files or environment variables. This section mainly introduces the EMQX configuration files and provides the basic configuration instructions for the most commonly used functions in EMQX. For comprehensive configuration items with detailed explanations, see EMQX Open Source Configuration Manual and EMQX Enterprise Configuration Manual.

Config Directories

After EMQX is installed, it creates a set of directories to manage its configuration and runtime data. These directories are separated into two main categories:

• Static Configuration Directory (etc): Ready-only and contains immutable or static configuration files.
• Dynamic Configuration Directory (data/configs): Writable and stores runtime-generated or dynamically updated configuration files.

Static Configuration Directory (etc)

The etc directory holds the configuration files that define EMQX's initial setup. These files are typically modified during deployment or upgrades and are read-only at runtime to ensure stability. The location of the etc directory depends on the installation method:

Installation	Path
Installed with RPM or DEB package	/etc/emqx
Running in docker container	/opt/emqx/etc
Extracted from portable compressed package	./etc

Dynamic Configuration Directory (data/configs)

At runtime, EMQX allows dynamic reconfiguration through the Dashboard, REST API, or CLI. Changes made using these tools are stored in the data/configs directory to ensure persistence across sessions. The location of this directory also depends on the installation method:

Installation	Path
Installed with RPM or DEB package	/var/lib/emqx/configs
Running in docker container	/opt/emqx/data/configs
Extracted from portable compressed package	./data/configs

TIP

It is possible to change the data directory by modifying the node.data_dir setting in the configuration or the EMQX_NODE__DATA_DIR environment variable. However, when running a cluster, all nodes must use the same directory path.

Although not encouraged, the content of the configuration files can overlap. In case of overlapping, the conflict is resolved by a predefined override rule, see Config Override Rules.

Config Examples

While the Schema section provides a detailed reference, configuration examples can be helpful for understanding and applying settings in EMQX. You can find several example configurations in the etc/examples directory.

Base Configuration File

Starting from EMQX 5.8.4, there is a base configuration file named base.hocon in the etc directory. This file contains default settings that can be overridden by higher-level configuration files at runtime.

For example, you may want to start the deployment with a basic authentication configuration, and then override it with a more complex configuration at runtime from the Dashboard UI.

For immutable configurations such as node and cluster configs, it is NOT recommended to set them in the base.hocon file. See the Immutable Configurations File for more details.

TIP

The base.hocon file is not synchronized across the cluster and only applies to the node where it is located.

Configuration Rewrite File

In data/configs directory, the cluster.hocon file contains configuration items for the entire cluster. Configuration changes made from Dashboard, REST API, and CLI will be persisted to this file.

If a node in the cluster is restarted or if new nodes are added, the node will automatically copy and apply the cluster.hocon file from another node in the cluster. For this reason, it is not recommended to modify the file manually.

Configurations in this file are applied on top of those in the base.hocon file. For details on the configuration override hierarchy, see Config Override Rules.

Since EMQX version 5.1, any changes to the cluster configuration will trigger a backup of the cluster.hocon file before it is overwritten. These backups are timestamped with the node's local time, and up to 10 backup files can be retained.

Immutable Configuration File

For backward compatibility, the emqx.conf file remains the primary configuration file for critical system settings, including node and cluster configurations. This file has a higher priority than both base.hocon and cluster.hocon, but a lower priority than environment variables.

For more details on configuration overrides, refer to the Config Override Rules section.

Configuration Paths

In EMQX, configuration values can be referenced using dot-separated paths, similar to a tree structure. Starting from the root (always a Struct), each segment in the path refers to a field name or a Map key. For array elements, a 1-based index is used.

Here are some examples of configuration paths:

node.name = "emqx.127.0.0.1"
zone.zone1.max_packet_size = "10M"
authentication.1.enable = true

HOCON Configuration Format

From EMQX v5.0, EMQX uses Human-Optimized Config Object Notation (HOCON) as the configuration file format.

HOCON is a format for human-readable data and a superset of JSON. With features like inheritance, combined, and quotes, HOCON further simplifies the configuration work.

HOCON syntax：

HOCON values can be represented as JSON-like objects, for example:

node {
  name = "emqx@127.0.0.1"
  cookie = "mysecret"
  cluster_call {
    retry_interval  =  1m
  }
}

or in flattening:

node.name = "127.0.0.1"
node.cookie = "mysecret"
node.cluster_call.retry_interval = "1m"

This cuttlefish-like flattening format is backward compatible with the previous EMQX versions, but it is used differently:

HOCON recommends adding quotes at both ends of the string. Strings without special characters can also be unquoted, for example foo, foo_bar, while cuttlefish regards all characters to the right of = as values.

For more information about HOCON syntax, please refer to HOCON Documentation.

Environment Variables

Besides configuration files, you can also use environment variables to configure EMQX.

For example, environment variable EMQX_NODE__NAME=emqx2@127.0.0.1 will override the following configuration:

# emqx.conf
node {
  name = "emqx@127.0.0.1"
}

Configuration items and environment variables can be converted by the following rules:

1. Since the . separator in the configuration file cannot be used in environment variables, EMQX uses double underscores __ as the configuration separator;
2. To distinguish the converted configuration items from other environment variables, EMQX also adds a prefix EMQX_ to the environment variable;
3. The value of the environment variable is parsed according to the HOCON value, making it possible to use the environment variable to pass the value of complex data types, but please note that special characters such as ： and = need to be wrapped in double quotes ".

Conversion example:

# Environment variables

## localhost:1883 will be parsed into a struct `{"localhost": 1883}`, so it needs to be wrapped in double quotes
export EMQX_LISTENERS__SSL__DEFAULT__BIND='"127.0.0.1:8883"'

## Pass the HOCON array directly by character
export EMQX_LISTENERS__SSL__DEFAULT__SSL_OPTIONS__CIPHERS='["TLS_AES_256_GCM_SHA384"]'


# Configuration file
listeners.ssl.default {
    ...
    bind = "127.0.0.1:8883"
    ssl_options {
      ciphers = ["TLS_AES_256_GCM_SHA384"]
    }
  }
}

TIP

EMQX will ignore undefined root paths, for example, EMQX_UNKNOWN_ROOT__FOOBAR , because UNKNOWN_ROOT is not a pre-defined root path.

When a known root path is set with an unknown field name, EMQX will output a warning log at startup, for example, when enable is incorrectly configured as enabled, it will output:

[warning] unknown_env_vars: ["EMQX_AUTHENTICATION__ENABLED"]

Config Override Rules

In EMQX, configuration values are applied hierarchically, with the following override rules:

• Within the same file, values defined later will override earlier ones.
• Higher-level configurations will replace lower-level ones.

The order of configuration priority is as follows:

base.hocon < cluster.hocon < emqx.conf < environment variables.

This means that the settings in base.hocon have the lowest priority and can be overridden by settings in higher-priority files. Environment variables that start with EMQX_ have the highest priority.

TIP

Before version 5.8.4, the base.hocon file did not exist. The priority order remains the same, but without base.hocon.

Changes made through EMQX Dashboard UI, HTTP API, or CLI are persisted in cluster.hocon at runtime and will take effect immediately. However, changes may get reverted after a node restart if the same configuration items are set differently in emqx.conf or environment variables.

To avoid confusion, do not overlap configuration settings between emqx.conf and cluster.hocon.

TIP

1. If you're using an older version of EMQX (e.g., 5.0.2/v5.0.22 or earlier, where the cluster-override.conf file still exists), the priority order for configuration settings is: emqx.conf < ENV < HTTP API (cluster-override.conf).
2. When upgrading from version 5.0.2/v5.0.22 or earlier to the latest version, the priority order remains unchanged, and cluster.hocon will not be created to maintain compatibility.
3. The cluster-override.conf mechanism is removed in version 5.1.

Override

In the following configuration, the debug value of level defined in the last line will overwrite the previously defined error, but the enable field remains unchanged:

log {
  console {
    enable = true
    level = error
  }
}

## Set the console log printing level to debug, and keep the other configurations
log.console.level = debug

The packet size limit was first set to 1MB, then overridden to 10MB:

zones {
  zone1 {
    mqtt.max_packet_size = 1M
  }
}
zones.zone1.mqtt.max_packet_size = 10M

List Element Override

EMQX array has two expression ways:

• List, for example, [1, 2, 3]
• Map (subscribing), for example: {"1"=1, "2"=2, "3"=3}

The following 3 formats are equivalent:

authentication.1 = {...}
authentication = {"1": {...}}
authentication = [{...}]

Based on this feature, we can easily override the value of an element in an array, for example:

authentication  = [
  {
    enable = true,
    backend = "built_in_database",
    mechanism = "password_based"
  }
]

# The `enable` field of the first element can be overridden in the following way:
authentication.1.enable = false

TIP

Arrays (in list format) will be fully overwritten and the original value cannot be kept, for example:

authentication = [
  {
    enable = true
    backend = "built_in_database"
    mechanism="password_based"
  }
]

## With the following method, all fields except `enable` of the first element will be lost.
authentication = [{ enable = true }]

Zone Override

A zone in EMQX is a concept for grouping configurations. Zones can be associated with listeners by setting the zone field to the name of the desired zone. MQTT clients connected to listeners associated with a zone will inherit the configurations from that zone, which may override global settings.

TIP

By default, listeners are linked to a zone named default. The default zone is a logical grouping and does not exist in the configuration files.

The following configuration items can be overridden at the zone level:

• mqtt: MQTT connection and session settings, such as allowing a greater maximum packet size for MQTT messages in a specific zone.
• force_shutdown: Policies for forced shutdowns.
• force_gc: Fine-tuning for Erlang process garbage collection.
• flapping_detect: Detection of client flapping.
• durable_sessions: Session persistence settings, such as enabling durable storage for MQTT sessions in a specific zone.

In EMQX version 5, the default configuration file does not include any zones, which differs from version 4, where there are two default zones: internal and external.

To create a zone, you need to define it in emqx.conf, for example:

zones {
  # Multiple zones can be defined
  my_zone1 {
    # Zones share the same configuration schema as the global configurations
    mqtt {
      # Allow a larger packet size for connections in this zone
      max_packet_size = 10M
    }
    force_shutdown {
      # Configuration specific to this zone
      ...
    }
    durable_sessions {
      # Enable durable storage for sessions in this zone
      enable = true
      ...
    }
  }
  my_zone2 {
    ...
  }
}

In a listener, set the zone field to associate it with a zone that has been created.

listeners.tcp.default {
    bind = 1883
    zone = my_zone1
    ...
}

Schema

To make the HOCON objects type-safe, EMQX introduced a schema for it. This schema defines data types, field names, and metadata, allowing for configuration value validation and more.

The EMQX Open Source Configuration Manual and EMQX Enterprise Configuration Manual are generated from the schema.

TIP

The zone configuration schema is not included in the configuration manual because it is identical for each group. For example, zones.my_zone1.mqtt {...} has the same schema as mqtt {...}.

Primitive Data Types

Primitive data types in the configuration manual are largely self-explanatory, requiring minimal documentation. Below is a comprehensive list of all primitive types you will encounter.

Integer

Represents a whole number. Examples include 42, -3, 0.

Integer(Min..Max)

An integer that falls within a specified range. For example, 1..+inf means from 1 to positive infinity (+inf), indicating that only positive integers are acceptable.

Enum(symbol1, symbol2, ...)

Defines an enumerated type that can only take one of the predefined symbols. For instance, Enum(debug,info,warning,error) defines acceptable logging levels.

String

The String data type represents a sequence of characters and supports several formats for diverse use cases:

• Unquoted: Ideal for simple identifiers or names that avoid special characters (see below for details).

• Quoted Strings: For strings that include special characters or whitespace, use double quotes ("), utilizing the backslash (\) to escape characters as needed. Example: "line1\nline2".

• Triple-quoted String: Enclosed with triple quotes ("""), these strings do not require escapes (except for \), simplifying the inclusion of complex content. Note that quotes adjacent to the triple-quotes must be escaped to be considered part of the string.

• Triple-quoted String with Indentation: Surrounded by """~ and ~""", introduced since EMQX 5.6, this format allows for indentation within the string for better layout in the configuration file, ideal for multi-line or formatted text.

Special Considerations for Unquoted Strings:

• Avoid "forbidden characters": $, ", {, }, [, ], :, =, ,, +, #, `, ^, ?, !, *, &, \, or whitespace.
• Do not start with // (which introduces a comment).
• Do not begin with true, false, or null in a way that could be misinterpreted as boolean or null values.

Guidelines for Triple-quoted Strings:

• To include a quote character adjacent to the triple-quotes, escape it or use the ~ delimiter for clarity.
• Multiline strings support indentation with spaces (not tabs) for readability. The indentation level is determined by the smallest number of leading spaces on any line.

For example:

rule_xlu4 {
  sql = """~
    SELECT
      *
    FROM
      "t/#"
  ~"""
}

For additional details on string quoting conventions in HOCON, consult the HOCON specification.

For insights into EMQ's specialized adaptations of the triple-quoted string with indentation, refer to the emqx/hocon.git README.

String("constant")

A constant string value, effectively acting as a single-value enumeration (Enum). This could be used to define a static value that doesn't change, such as a specific setting or mode.

Boolean

Either true or false, which is case sensitive.

Float

A floating-point number, supporting decimals. Examples include 3.14, -0.001.

Duration

Represents a span of time in a human-readable format. Examples and explanation of format.

Duration(s)

Specifies a Duration type with a precision level of seconds. Further details and examples.

Secret

A type intended for sensitive information, such as passwords and tokens. Explanation of its usage and importance.

Complex Data Types

Complex data types in EMQX's HOCON configuration are designed to encapsulate data structures that can include both other complex types and primitive values. These data types enable flexible and hierarchical data representation.

Struct Struct(name)

Represents a structure with fields enclosed between curly braces {}. The name parameter is a reference to a schema that specifies the structure's fields and their respective types.

Map Map($name-\>Type)

Similar to Struct, a Map holds key-value pairs without predefined names for the fields.

The $name variable indicates that the keys can be any string (except for a string with dot . in it), representing the name of an entity or attribute. The Type specifies that all values in the map must be of the same data type, allowing for uniform collections of data items.

OneOf OneOf(Type1, Type2, ...)

Defines a union type that can include two or more types to indicate that one struct field can be any one of the member types. For example, this allows a configuration entry to be either String(infinity) or a Duration.

Array Array(Type)

Defines an array consisting of elements that adhere to the specified Type.

TIP

If a Map field name is a positive integer number, it is interpreted as an alternative representation of an Array. For example:

myarray.1 = 74
myarray.2 = 75

will be interpreted as myarray = [74, 75], which is handy when trying to override array elements.

Variform Expressions

Variform is a lightweight, expressive language designed for string manipulation and runtime evaluation. It is not a full-fledged programming language but a specialized tool that can be embedded within configurations for EMQX to perform string operations dynamically.

TIP

Variform expressions are only applicable to certain configuration items. Do not use them unless specifically stated.

NULL value:

In Variform expressions, a value-binding reference or sub-expression evaluation may result in an undefined value, which is represented as an empty string ('').

It is important to note that if a JSON-decoded field is null, it is treated as undefined value (hence ''), but not string value "null".

Syntax

To illustrate:

function_call(clientid, another_function_call(username))

This expression combines or manipulates clientid and username to generate a new string value.

Variform supports below literals:

• Boolean: true or false.
• Integer: For example, 42.
• Float: For example, 3.14.
• String: ASCII characters between single quotes ' or double quotes ".
• Array: Elements between [ and ], separated by a comma ,.
• Variable: Referencing to predefined values, for example clientid.
• Function: Predefined functions, for example, concat([...]).

Variform does not support the following:

• Arithmetic operations
• Loops
• User-defined variables
• User-defined functions
• Exception handling and error recovery
• Escape sequence in string literals. Call the unescape function to unescape special characters.

Below is a configuration example with a Variform expression embedded.

mqtt {
    client_attrs_init = [
        {
            # Extract the prefix of client ID before the first -
            expression = "nth(1, tokens(clientid, '-'))"
            # And set as client_attrs.group
            set_as_attr = group
        }
    ]
}

TIP

When an unescape function is required in the expression, it's a good idea to use triple quote (""") strings in HOCON config so there is no need to perform double escaping.

For example

#### For multi-line client ID, take the first line.
expression = """nth(1, tokens(clientid, unescape('\n')))"""

Pre-defined Functions

EMQX includes a rich set of string, array, random, and hashing functions similar to those available in rule engine string functions. These functions can be used to manipulate and format the extracted data. For instance, lower(), upper(), and concat() help in adjusting the format of extracted strings, while hash() and hash_to_range() allow for creating hashed or ranged outputs based on the data.

Below are the functions that can be used in the expressions:

• String functions:

  • String Operation Functions
  • A new function any_to_string/1 is also added to convert any intermediate non-string value to a string.

• Array functions: nth/2

• Random functions: rand_str, rand_int

• Schema-less encode/decode functions:

  • bin2hexstr(Data)
  • hexstr2bin(Data)
  • base64_decode(Data)
  • base64_encode(Data)
  • int2hexstr(Integer): Encode an integer to hex string. e.g. 15 as 'F' (uppercase).

• Hash functions:

  • hash(Algorihtm, Data): Algorithm can be one of: md4 | md5, sha (or sha1) | sha224 | sha256 | sha384 | sha512 | sha3_224 | sha3_256 | sha3_384 | sha3_512 | shake128 | shake256 | blake2b | blake2s
  • hash_to_range(Input, Min, Max): Use sha256 to hash the Input data and map the hash to an integer between Min and Max inclusive ( Min =< X =< Max)
  • map_to_rage(Input, Min, Max): Map the input to an integer between Min and Max inclusive (Min =< X =< Max)

• Compare functions:

  • num_eq(A, B): Return 'true' if two numbers are the same, otherwise 'false'.
  • num_neq(A, B): Return 'true' if two numbers are NOT the same, otherwise 'false'.
  • num_gt(A, B): Return 'true' if A is greater than B, otherwise 'false'.
  • num_gte(A, B): Return 'true' if A is not less than B, otherwise 'false'.
  • num_lt(A, B): Return 'true' if A is less than B, otherwise 'false'.
  • num_lte(A, B): Return 'true' if A is not greater than B, otherwise 'false'.
  • str_eq(A, B): Return 'true' if two strings are the same, otherwise 'false'.
  • str_neq(A, B): Return 'true' if two strings are NOT the same, otherwise 'false'
  • str_gt(A, B): Return 'true' if A is behind B in lexicographic order, otherwise 'false'.
  • str_gte(A, B): Return 'true' if A is not before B in lexicographic order, otherwise 'false'.
  • str_lt(A, B): Return 'true' if A is before B in lexicographic order, otherwise 'false'.
  • str_lte(A, B): Return 'true' if A is not after B in lexicographic order, otherwise 'false'.
  • is_empty_var(V): Check if a variable is empty. Empty in Variform means the value is not present (undefined), JSON's null (but not string "null"), or an empty string "".
  • not(Bool): Return true if Bool is false, and return false if the condition is true. It also accepts string parameters. If the input is a string, the output is also a string.

• System functions:

  • getenv(Name): Return the value of the environment variable Name with the following constraints:
    • Prefix EMQXVAR_ is added before reading from OS environment variables. For example, getenv('FOO_BAR') is to read EMQXVAR_FOO_BAR.
    • Values are immutable once loaded from the OS environment.

Conditions

The variform expression so far has no comprehensive control flows. Below functions can help to preform basic control of which value to return from the expression.

• iif(Condition, ThenExpression, ElseExpression): returns ThenExpression if Condition yields true or a non-empty string value, otherwise returns ElseExpression.
• coalesce(Arg1, Arg2, ...): returns the first non-empty argument.
• coalesce([Element1, Element2, ...]): returns the first non-empty element.

Error Handling

As the default behavior of scripting environments like Bash, Variform expression is designed to yield an empty string ("") in scenarios where errors occur, such as unbound variables or exceptions during runtime.

• Unbound Variables: If an expression references a variable that has not been defined or is out of scope (unbound), the expression will be evaluated as an empty string.
• Runtime Exceptions: Any exceptions that occur during the execution of an expression, whether due to incorrect function usage, invalid data types, or other unforeseen issues, will result in the expression yielding an empty string. For example, the array index is out of range.

Example Expressions

• nth(1, tokens(clientid, '.')): Extract the prefix of a dot-separated client ID.
• strlen(username, 0, 5): Extract a partial username.
• coalesce(regex_extract(clientid,'[0-9]+'),'vin-1000'): Extract digits from client ID using a regular expression. If the regular expression yields empty string, then return '000'.
• iif(true, "Value if true", "Value if false"): Returns Value if true
• iif("", "Value if true", "Value if false"): Returns Value if false
• iif("hello", "Value if true", "Value if false"): Returns Value if true
• iif(regex_match(clientid,'^foo\.+*'),'foo','bar'): Returns foo if clientid starts with foo., otherwise bar.