Message Transformation
Message Transformation allows you to modify and format messages based on user-defined rules before they are processed further or delivered to subscribers. This feature is based on dynamic message transformation using Schema. It is highly customizable and supports multiple encoding formats and advanced transformations to meet complex downstream data processing needs.
Workflow
When a message is published, it undergoes the following workflow:
Schema Validation: When a message is published and passes authorization, it is first checked against Schema Validation. If the message passes, it moves to the next step.
Message Transformation Pipeline:
- Transformation Matching: The message is matched against a list of user-defined transformations based on its topic. Multiple transformations can be set for different topics or topic filters.
- Transformation Execution: The matched transformations are executed in the order they are configured. The pipeline supports various encoders and decoders, such as JSON, Protobuf, and Avro, and allows for variform expressions to enrich or modify the message.
- Post-Transformation Processing: Once the message successfully passes through the transformation pipeline, it proceeds to the next steps, such as triggering the rule engine or dispatching the message to subscribers.
Failure Handling: If a transformation fails, user-configured actions are executed:
- Discard Message: Terminate the publish and discard the message, returning a specific reason code (131 - Implementation Specific Error) for QoS 1 and QoS 2 messages via PUBACK.
- Disconnect and Discard Message: Discard the message and disconnect the publishing client.
- Ignore: No additional actions are taken.
A log entry can be generated whenever a transformation fails, regardless of the configured action. Users can configure the log's output level, which defaults to
warning
.Additionally, a transformation failure can trigger a rule engine event (
$events/message_transformation_failed
), enabling users to implement custom handling, such as republishing the erroneous message to another topic or sending it to Kafka for further analysis.
Create Message Transformation Rules
If you have subscribed to the Smart Data Hub, you can access the Message Transform page by clicking Smart Data Hub -> Message Transform from the left menu. On this page, you can create message transformation rules.
On the Message Transform page, click the + New button in the top-right corner to enter the New Message Transform page and configure the following settings:
Name: Enter a name for the transformation to identify the message transformation rule.
Note: Optional. Add a description for the message transformation rule.
Message Source Type: Set the topic(s) of the messages to be transformed. Multiple topics or topic filters can be set.
Message Format Transformation:
- Source Format: Specify the valid payload decoding format to be applied when entering the transformation pipeline. Options include
None
(no decoding),JSON
,Avro
, orProtobuf
. These formats will convert the binary input payload into a structured mapping. IfAvro
orProtobuf
is chosen, the Schema must first be defined in the Schema Registry. In pipelines configured with multiple transformation rules, decoding doesn't need to be applied at each step. For example, if TransformationT1
has already decoded the payload, subsequent transformations likeT2
can skip decoding and directly use the correctly formatted payload. - Target Format: Specify the encoding format to convert the final message payload into a binary value when the transformation pipeline completes. Encoding options are the same as the Source Format:
None
(no encoding),JSON
,Avro
, orProtobuf
. Only the last transformation in the pipeline needs to ensure the payload is encoded as binary; intermediate transformations do not need to handle binary encoding.
TIP
If the source format and target format are the same, no message format transformation will occur. Avro/Protobuf formats and None cannot be transformed between each other. For detailed rules on format conversion, refer to Message Format Transformation Setting Rules.
- Source Format: Specify the valid payload decoding format to be applied when entering the transformation pipeline. Options include
Message Properties Transformation:
Properties: Specify the target location for the transformed value (generated by an expression). Valid targets include
payload
,topic
,qos
,retain
(set the corresponding flags), anduser_property
(forUser-Property
MQTT attributes).When using
user_property
, you must specify a concrete key (e.g.,user_property.my_custom_prop
).payload
can either be used as-is to overwrite the entire message payload, or you can specify a nested key path to treat the payload as a nested JSON object (e.g.,payload.x.y
).Target Value: Define the value to be written to the specified property. This value can be copied from other fields, such as
qos
,retain
,topic
,payload
, orpayload.x.y
, or it can be generated using variform expressions.
TIP
For setup, refer to Message Properties Transformation Setting Rules.
Transform Failure Opearation:
- Actions After Failure: Choose the action to be executed if the transformation fails:
- Drop Message: Terminate the publishing process and discard the message, returning a specific reason code for QoS 1 and QoS 2 messages via PUBACK.
- Disconnect and Drop Message: Discard the message and disconnect the client that published it.
- Ignore: Perform no additional action.
- Logs Level: Define the output log level for validation failures in the deployment logs. Options include
warning
orerror
.
- Actions After Failure: Choose the action to be executed if the transformation fails:
(Optional) You can preview your transformation before creating it by clicking Preview. This opens a dialog where you can enter contextual information for the incoming message, such as QoS, payload, whether the retain flag is set, and the publisher’s username and client ID. After providing the necessary details, click Run Transformation to execute the transformation with the specified context and view the output.
Click Confirm to complete the configuration.
Once the transformation is created, it will be enabled by default and displayed in the message transformation list. You can manage the created message transformations as described in Manage Message Transformation Rules.
Message Format Transformation Setting Rules
Note: ✓
(allowed setting); ✗
(not allowed setting)
Source Format \ Target Format | JSON | Avro | Protobuf | None |
---|---|---|---|---|
JSON | ✓ | ✓ | ✓ | ✓ |
Avro | ✓ | ✓ | ✓ | ✗ |
Protobuf | ✓ | ✓ | ✓ | ✗ |
None | ✓ | ✗ | ✗ | ✓ |
Message Properties Transformation Setting Rules
Note: ✓
(supports setting all properties); ✗
(does not support setting)
user_property
cannot be set as a whole, but individualuser_property
sub-properties can be manipulated.
Source Format \ Target Format | JSON | Avro | Protobuf | None |
---|---|---|---|---|
JSON | ✓ | ✓ | ✓ | ✓ |
Avro | Cannot directly set payload , but can add sub-properties to payload | Cannot directly set payload , but can add sub-properties to payload | Cannot directly set payload , but can add sub-properties to payload | ✗ |
Protobuf | Cannot directly set payload , but can add sub-properties to payload | Cannot directly set payload , but can add sub-properties to payload | Cannot directly set payload , but can add sub-properties to payload | ✗ |
None | Can directly set payload , cannot add sub-properties to payload | ✗ | ✗ | Can directly set payload , cannot add sub-properties to payload |
Manage Message Transformation Rules
You can perform the following management operations on the message transformation rule list page:
- Enable/Disable Message Transformation Rule: Click the toggle button in the Enabled column.
- Edit Message Transformation Rule: Click the "Edit" icon in the Actions column to enter the editing page, make modifications, and save.
- Delete Message Transformation Rule: Click the "More" icon in the Actions column, select Delete from the dropdown menu, and confirm the deletion.
- Adjust Message Transformation Rule Order: Use the mouse to drag a row in the list to reorder it, or use the quick move buttons in the More options menu in the Actions column.
View Statistical Metrics
Click on the name in the message transformation rule list to enter the details page, where you can view the statistics related to message transformation execution:
- Statistics:
- Matched: The total number of times the message transformation rule has been triggered since it was enabled.
- Successes: The number of successful message transformations.
- Failures: The number of failed message transformations.
- Rate Indicators:
- Current Rate: The transformation rate in the last 1 minute.
- Rate in Last 5 Min: The transformation rate in the last 5 minutes.
- Maximum Rate: The highest historical transformation rate.