Skip to content

Message Transformation

Message Transformation allows you to modify and format messages based on user-defined rules before they are processed further or delivered to subscribers. This feature is based on dynamic message transformation using Schema. It is highly customizable and supports multiple encoding formats and advanced transformations to meet complex downstream data processing needs.

Workflow

When a message is published, it undergoes the following workflow:

  1. Schema Validation: When a message is published and passes authorization, it is first checked against Schema Validation. If the message passes, it moves to the next step.

  2. Message Transformation Pipeline:

    • Transformation Matching: The message is matched against a list of user-defined transformations based on its topic. Multiple transformations can be set for different topics or topic filters.
    • Transformation Execution: The matched transformations are executed in the order they are configured. The pipeline supports various encoders and decoders, such as JSON, Protobuf, and Avro, and allows for variform expressions to enrich or modify the message.
    • Post-Transformation Processing: Once the message successfully passes through the transformation pipeline, it proceeds to the next steps, such as triggering the rule engine or dispatching the message to subscribers.
  3. Failure Handling: If a transformation fails, user-configured actions are executed:

    • Discard Message: Terminate the publish and discard the message, returning a specific reason code (131 - Implementation Specific Error) for QoS 1 and QoS 2 messages via PUBACK.
    • Disconnect and Discard Message: Discard the message and disconnect the publishing client.
    • Ignore: No additional actions are taken.

    A log entry can be generated whenever a transformation fails, regardless of the configured action. Users can configure the log's output level, which defaults to warning.

    Additionally, a transformation failure can trigger a rule engine event ($events/message_transformation_failed), enabling users to implement custom handling, such as republishing the erroneous message to another topic or sending it to Kafka for further analysis.

Create Message Transformation Rules

If you have subscribed to the Smart Data Hub, you can access the Message Transform page by clicking Smart Data Hub -> Message Transform from the left menu. On this page, you can create message transformation rules.

  1. On the Message Transform page, click the + New button in the top-right corner to enter the New Message Transform page and configure the following settings:

    • Name: Enter a name for the transformation to identify the message transformation rule.

    • Note: Optional. Add a description for the message transformation rule.

    • Message Source Type: Set the topic(s) of the messages to be transformed. Multiple topics or topic filters can be set.

    • Message Format Transformation

      • Source Format: Specify the valid payload decoding format to be applied when entering the transformation pipeline. Options include None (no decoding), JSON, Avro, or Protobuf. These formats will convert the binary input payload into a structured mapping. If Avro or Protobuf is chosen, the Schema must first be defined in the Schema Registry. In pipelines configured with multiple transformation rules, decoding doesn't need to be applied at each step. For example, if Transformation T1 has already decoded the payload, subsequent transformations like T2 can skip decoding and directly use the correctly formatted payload.
      • Target Format: Specify the encoding format to convert the final message payload into a binary value when the transformation pipeline completes. Encoding options are the same as the Source Format: None (no encoding), JSON, Avro, or Protobuf. Only the last transformation in the pipeline needs to ensure the payload is encoded as binary; intermediate transformations do not need to handle binary encoding.

      TIP

      If the source format and target format are the same, no message format transformation will occur. Avro/Protobuf formats and None cannot be transformed between each other. For detailed rules on format conversion, refer to Message Format Transformation Setting Rules.

    • Message Properties Transformation:

      • Properties: Specify the target location for the transformed value (generated by an expression). Valid targets include payload, topic, qos, retain (set the corresponding flags), and user_property (for User-Property MQTT attributes).

        When using user_property, you must specify a concrete key (e.g., user_property.my_custom_prop).

        payload can either be used as-is to overwrite the entire message payload, or you can specify a nested key path to treat the payload as a nested JSON object (e.g., payload.x.y).

      • Target Value: Define the value to be written to the specified property. This value can be copied from other fields, such as qos, retain, topic, payload, or payload.x.y, or it can be generated using variform expressions.

    • Transform Failure Opearation

      • Actions After Failure: Choose the action to be executed if the transformation fails:
        • Drop Message: Terminate the publishing process and discard the message, returning a specific reason code for QoS 1 and QoS 2 messages via PUBACK.
        • Disconnect and Drop Message: Discard the message and disconnect the client that published it.
        • Ignore: Perform no additional action.
      • Logs Level: Define the output log level for validation failures in the deployment logs. Options include warning or error.
  2. (Optional) You can preview your transformation before creating it by clicking Preview. This opens a dialog where you can enter contextual information for the incoming message, such as QoS, payload, whether the retain flag is set, and the publisher’s username and client ID. After providing the necessary details, click Run Transformation to execute the transformation with the specified context and view the output.

  3. Click Confirm to complete the configuration.

Once the transformation is created, it will be enabled by default and displayed in the message transformation list. You can manage the created message transformations as described in Manage Message Transformation Rules.

Message Format Transformation Setting Rules

Note: (allowed setting); (not allowed setting)

Source Format \ Target FormatJSONAvroProtobufNone
JSON
Avro
Protobuf
None

Message Properties Transformation Setting Rules

Note: (supports setting all properties); (does not support setting)

user_property cannot be set as a whole, but individual user_property sub-properties can be manipulated.

Source Format \ Target FormatJSONAvroProtobufNone
JSON
AvroCannot directly set payload, but can add sub-properties to payloadCannot directly set payload, but can add sub-properties to payloadCannot directly set payload, but can add sub-properties to payload
ProtobufCannot directly set payload, but can add sub-properties to payloadCannot directly set payload, but can add sub-properties to payloadCannot directly set payload, but can add sub-properties to payload
NoneCan directly set payload, cannot add sub-properties to payloadCan directly set payload, cannot add sub-properties to payload

Manage Message Transformation Rules

You can perform the following management operations on the message transformation rule list page:

  • Enable/Disable Message Transformation Rule: Click the toggle button in the Enabled column.
  • Edit Message Transformation Rule: Click the "Edit" icon in the Actions column to enter the editing page, make modifications, and save.
  • Delete Message Transformation Rule: Click the "More" icon in the Actions column, select Delete from the dropdown menu, and confirm the deletion.
  • Adjust Message Transformation Rule Order: Use the mouse to drag a row in the list to reorder it, or use the quick move buttons in the More options menu in the Actions column.

View Statistical Metrics

Click on the name in the message transformation rule list to enter the details page, where you can view the statistics related to message transformation execution:

  • Statistics:
    • Matched: The total number of times the message transformation rule has been triggered since it was enabled.
    • Successes: The number of successful message transformations.
    • Failures: The number of failed message transformations.
  • Rate Indicators:
    • Current Rate: The transformation rate in the last 1 minute.
    • Rate in Last 5 Min: The transformation rate in the last 5 minutes.
    • Maximum Rate: The highest historical transformation rate.