Databricks

Collected tag data can also land in a lakehouse for long-term storage and analysis. EMQX has no direct Databricks sink. The path runs through Amazon S3: EMQX writes messages into S3 and Databricks reads them through an external location.

Field devices → EMQX Neuron → EMQX Cloud / EMQX Enterprise → Amazon S3 → Databricks

The edge maintains one MQTT path and nothing more. How the data lands and how it is modelled are configured on the EMQX and Databricks sides, leaving EMQX Neuron untouched.

Prerequisites

• A working EMQX Cloud deployment or self-hosted EMQX Enterprise, with EMQX Neuron connected and publishing — see EMQX Cloud
• A Databricks account with permission to create workspaces and external locations
• AWS access credentials with read and write permission on the target S3 bucket

TIP

Data integration is a feature of EMQX Enterprise and EMQX Cloud. Open-source EMQX does not include it.

What EMQX Neuron publishes

The default values-format message has a fixed shape. The whole message lands in S3 as-is, and the query in Databricks parses it against this structure:

{
  "node": "modbus-tcp",
  "group": "group-1",
  "timestamp": 1790219933821,
  "values": { "temperature": 23.5, "pressure": 1013 },
  "errors": {},
  "metas": {}
}

The default topic is /neuron/{application}/{driver}/{group}, and each subscription can override it. With several plants or lines, name topics hierarchically — factory-a/line-1/modbus-tcp, for example — so one wildcard subscription on the EMQX side covers them all. See Send Data to the Cloud · Upload topic.

On the Databricks side

1. Create a workspace and note the S3 bucket it is associated with.
2. Under Catalog → External locations, create an external location pointing at the path the data will land in, such as s3://<bucket>/neuron-data.
3. Prepare AWS access credentials with read and write permission on that bucket.

Create the connector and sink in EMQX

The connector type is Amazon S3:

Field	Value
Host	s3.{region}.amazonaws.com
Port	443
Access key ID, secret access key	The AWS credentials from the previous step

What matters in the sink is that the object key lands under the path the external location points at:

Field	Value
Bucket	The bucket associated with the Databricks workspace
Object key	neuron-data/${clientid}_${timestamp}.json
Object content	${payload}

The rule SQL can take the whole message here — no flattening needed:

SELECT * FROM "/neuron/#"

Query from Databricks

The data lands in S3 as JSON files and is queried through the external location:

SELECT
  payload:node        AS node,
  payload:group       AS group_name,
  payload:timestamp   AS ts,
  payload:values      AS tag_values
FROM json.`s3://<bucket>/neuron-data/`;

Tag values sit one level down under values, keyed by tag name, so new tags simply appear in the JSON and this query needs no change. Load it into a Delta table when you move on to sustained analysis.

Further reading

• EMQX documentation: Databricks data integration
• When the set of tags is stable and feeds reports directly, see Snowflake
• For every available downstream system, see EMQX data integration
• When the volume is high, filter and aggregate at the edge before publishing — see Processing Data Before Delivery