Skip to content

Schema Registry

Schema Registry provides a centralized schema for managing and validating topic message data and the ability to serialize and deserialize data over the network. Publishers and subscribers of MQTT topics can use the Schema to ensure data consistency and compatibility. Schema Registry is a key component of the rule engine. It can be adapted to multiple scenarios of device access and rule design, helping to ensure data quality, compliance, application development efficiency and system performance.

Understand Schema Registry

The Schema defines the structure of the data. It defines the allowed data types, formats, and relationships. A schema is a blueprint for data that describes the structure of a data record, the data types of individual fields, the relationships between fields, and any constraints or rules that apply to the data.

Schemas can be used in a variety of data processing systems, including databases, messaging services, and distributed event and data processing frameworks. They help ensure that data is consistent and accurate, and can be efficiently processed and analyzed by different systems and applications. Data sharing and interoperability between different systems and organizations are facilitated.

Users can define Schema in the Schema Registry, and then use the defined Schema in rules for forwarding client data to different data services through data integration. At the same time, you can also send the data in the application or data service to the client through the Schema to realize two-way data flow.

schema

Schema Registry has multiple advantages, including data validation, compatibility checking, version control and iterative evolution. It also simplifies the development and maintenance of data pipelines and reduces the risk of data compatibility issues, data corruption and data loss.

Create Schema

If you have subscribed to the Smart Data Hub, you can access the Schema Registry page by clicking Smart Data Hub -> Schema Registry in the left menu of the deployment. On this page, you can create a Schema. The EMQX Platform supports creating Schemas in Avro, Protobuf, and JSON Schema formats.

  1. Click the + New button in the top right corner of the page to go to the New Schema Registry page. Set the following fields:

    • Name: Used to identify the Schema.

      • The name can be used in Schema validation and data transformation rules.
      • The name can also be used in the SQL encoding and decoding functions for data integration rules, for example: SELECT schema_encode("<name>", payload).
    • Description: Optional, add any description information.

    • Type: Select the Schema type from the dropdown menu. Available options are Avro, Protobuf, and JSON Schema.

    • Schema: Enter the Schema corresponding to the selected type. Examples:

      • Avro:
      json
      {
        "type": "record",
        "name": "Device",
        "fields": [
          { "name": "id", "type": "string" },
          { "name": "temp", "type": "int" }
        ]
      }
      • Protobuf:
      proto
      message Device {
        required string id = 1;
        required uint32 temp = 2;
      }
      • JSON Schema; you can generate JSON Schema from JSON data:

      generate_JSON_schema

      json
      {
          "$schema": "http://json-schema.org/draft-06/schema#",
          "type": "object",
          "properties": {
            "temp": {
              "type": "integer"
            },
            "id": {
              "type": "string"
            }
          },
         "required": [
            "temp",
            "id"
         ]
      }
  2. After clicking Confirm, the schema will be successfully created, and you will be returned to the Schema Registry page.

Manage Schema

The Schema Registry page shows all created Schemas. You can manage them here.

Edit Schema

In the Schema list, click the edit icon under Actions to modify the description, Schema type, and Schema.

Delete Schema

In the Schema list, click the delete icon under Actions, confirm the deletion, and the Schema will be deleted.