Source Connectors

Source is used to read data from external systems. EMQX Neuron supports loading data sources into three modes: Stream, Scan Table, and Lookup Table.

General data sources can be loaded as Stream. Tables (including Scan table and Lookup table) are methods of retaining a large amount of data status, and are generally used in combination with Stream , to cope with complex processing scenarios.

Tips

When using Rule, at least one source must be a Stream.

Data source type

Currently EMQX Neuron has the following built-in data source types and the loading modes they support:

source type	Description	Stream	Scan Table	Lookup Table
Neuron	Read data from EMQX Neuron's data collection module	✅	✅	❌
MQTT	Read data from MQTT topic	✅	✅	❌
HTTP pull	Pull data from HTTP server	✅	✅	❌
HTTP push	Push data to EMQX Neuron via HTTP	✅	✅	❌
Memory	Read data from the EMQX Neuron memory to form a rule pipeline	✅	✅	✅
SQL	Query data from the database	✅	✅	✅
File	Read data from a file	✅	✅	❌
Video	Query data from video stream	✅	✅	❌
Simulator	Generate simulated data for testing	✅	✅	❌
Redis	Read data from Redis	❌	❌	✅
CAN	Read data from CAN bus	✅	✅	❌
Kafka	Read data from Kafka	✅	❌	❌
WebSocket	Read data from WebSocket	✅	✅	❌

Define and run

In EMQX Neuron, after creating a stream or table on the Sources page, it actually only creates a logical definition of a data source rather than a real data input for physical operation. The data flow will not actually run until a rule using that data source is started.

Tips

Streams or tables created on the Sources page can be used in the from clause of SQL in multiple rules.

Users can define whether to share the data source by specifying the SHARED field when creating a Sources.

Data source decoding

Users can define the decoding method by specifying the stream format field when creating a data source (Source). Currently supports json, binary, protobuf, delimited. If you want to use a custom format, you can also choose custom.

Tips

Before the data source enters rule processing, it will be decoded first, and the decoded data will be used as the input of the rule.

data structure

EMQX Neuron supports structured/unstructured streams, with the default being unstructured. That is, when Sources -> Create Stream, the Whether it is a stream with structure option is not checked. For details, see [Stream Parameter Configuration](./stream.md#Stream Parameter Configuration).