Data Integration Overview

The EMQX Cloud Data Integration is used to configure the rules for handling and responding to EMQX message flows and device events. The Data Integrations not only provides a clear and flexible "configurable" architecture solution, but also simplifies the development process, improves user usability, and reduces the coupling degree between the business system and EMQX Cloud. It also provides a superior infrastructure for customization of EMQX Cloud's proprietary capabilities.

Note

This feature is not available for the Serverless plan

In EMQX Cloud, using Data Integration has the following requirements:

• Standard Deployment
  • The resource only supports public access, so you need to ensure that the resource has public access and open security groups before creating the resource.
  • Only Webhook and MQTT bridges are open for the resource creation.
• Professional Deployment
  • The resource technically only supports internal network access, so you need to configure VPC peering before creating the resource and open the security group.
  • If public access is needed in a professional environment, NAT Gateway can offer the abilities.
• BYOC Deployment
  • It is recommended to access resources through an internal network to enhance network security and performance. Before creating resources, you need to configure a peering connection between the VPC where the resources are located and the VPC where the BYOC deployment is located in the cloud platform console. Additionally, you should open relevant security groups. For detailed steps, please refer to the Create BYOC Deployment - VPC Peering Configuration section.
  • If you need to access resources through the public network, please configure a NAT gateway for the VPC where the BYOC deployment is located in your cloud platform console.

TIP

Serverless deployments don't support Data Integrations.

General Flow

The primary dimension of Data Integrations is the resource. A resource is a service provided by the cloud provider. It should already be set in the system architecture. Then choose the specific resource from the browse page.

After the resource is settled, you need to create a rule to handle the data collected from the device. The rule can collect and process data in the way you want by using a SQL clause.

When a rule is tested and created, action needs to be attached to the rule. It will send the data to the target resource.

Resource

The resource is a cloud service offering that you have purchased from the cloud platform, either a message queue service like Kafka or an RDS storage service.

Rule

The rules describe "where the data comes from" and "how to filter and process the data". The rules use SQL-like statements to customize the written data and can use SQL tests to simulate the exported data.

Action

Actions solve the problem of "where to send the processed data". A rule can correspond to one or more actions, and the defined resources need to be set in the action, that is, to which service the data will be sent.

Write Rules Using SQL

Learn and understand how to use SQL to write rules.