# End-to-End Tracing Span Details EMQX provides end-to-end tracing capabilities based on the OpenTelemetry standard. This allows you to monitor the entire lifecycle of MQTT messages and client activities within the EMQX cluster. This page describes the spans generated by EMQX and explains what they reveal about the broker’s internal operations. ## Client Lifecycle Spans These spans trace the primary lifecycle events of an MQTT client. - **`client.connect`**: A root span that traces the client connection process. It starts when a client initiates a connection to the broker and ends when the connection is established or rejected. - **`client.disconnect`**: Traces the client disconnection process. It starts when a client sends a `DISCONNECT` packet or when the connection is closed for other reasons (e.g., network error, keep-alive timeout). - **`client.subscribe`**: Traces a client's subscription request. It covers the entire process of the broker receiving the `SUBSCRIBE` packet, processing the subscriptions, and sending a `SUBACK` packet. - **`client.unsubscribe`**: Traces a client's unsubscription request. It covers the process from receiving the `UNSUBSCRIBE` packet to sending the `UNSUBACK` packet. ## Authentication and Authorization Spans These spans reveal how EMQX performs authentication and authorization checks. - **`client.authn`**: Traces the authentication process for a client. This span is a child of the `client.connect` span. - **`client.authn_backend`**: Traces a specific backend call during authentication (e.g., querying a database, calling an HTTP service). This is a child span of `client.authn` and is useful for identifying performance bottlenecks in authentication backends. - **`client.authz`**: Traces the authorization process for a client, which occurs on publish or subscribe operations. - **`client.authz_backend`**: Traces a specific backend call during authorization. This is a child span of `client.authz`. ## Message Lifecycle Spans These spans trace the journey of an MQTT message through the broker. ### Ingress (Client to Broker) - **`client.publish`**: A root span that traces a message published by a client to the broker. It starts when the broker receives the `PUBLISH` packet. - **`message.route`**: A child span of `client.publish` that traces the routing of the message within the broker to find matching subscribers. - **`message.forward`**: If the message needs to be delivered to a subscriber on another node in the cluster, this span traces the forwarding of the message to that node. It is a child of `message.route`. - **`message.handle_forward`**: On the receiving node, this span traces the handling of a forwarded message. ### Egress (Broker to Client) - **`broker.publish`**: Traces how the broker prepares and publishes a message to a subscriber. It is a child of either `message.route` or `message.handle_forward`. ### QoS Acknowledgement Spans These spans trace the QoS 1 and QoS 2 acknowledgement flows. #### Broker to Publisher - **`broker.puback`**: Traces the broker sending a `PUBACK` to a publisher (QoS 1). - **`broker.pubrec`**: Traces the broker sending a `PUBREC` to a publisher (QoS 2). - **`broker.pubcomp`**: Traces the broker sending a `PUBCOMP` to a publisher (QoS 2), completing the QoS 2 flow from the publisher's side. #### Publisher to Broker - **`client.pubrel`**: Traces the broker receiving a `PUBREL` from a publisher (QoS 2). #### Broker to Subscriber - **`broker.pubrel`**: Traces the broker sending a `PUBREL` to a subscriber (QoS 2). #### Subscriber to Broker - **`client.puback`**: Traces the broker receiving a `PUBACK` from a subscriber (QoS 1). - **`client.pubrec`**: Traces the broker receiving a `PUBREC` from a subscriber (QoS 2). - **`client.pubcomp`**: Traces the broker receiving a `PUBCOMP` from a subscriber (QoS 2), completing the QoS 2 flow from the subscriber's side. ## Rule Engine Spans These spans trace the execution within the EMQX Rule Engine. - **`broker.rule_engine.apply`**: Traces how a message is evaluated against rules. This is a child of the `message.route`. - **`broker.rule_engine.action`**: Traces the execution of a specific action triggered by a matched rule. This is a child of `broker.rule_engine.apply`. ## Broker Internal Spans These spans trace internal broker operations not initiated directly by clients. - **`broker.disconnect`**: Traces when the broker actively disconnects a client (e.g., due to an administrative action). - **`broker.subscribe`**: Traces an internal subscription process initiated by the broker itself (e.g., due to an administrative action). - **`broker.unsubscribe`**: Traces an internal unsubscription process. ## Trace Sampling and Filtering EMQX's OpenTelemetry integration includes a flexible sampler that allows you to control which traces are generated. This helps to manage the volume of trace data and focus on specific clients, topics, or event types. The decision to sample a trace is made based on the following hierarchy: 1. **Trace Context Source**: Determines whether EMQX extracts trace context from incoming MQTT packets. This is controlled by the `follow_traceparent` boolean switch. - If `true` (the default), EMQX attempts to extract trace context from the incoming request (e.g., from the `traceparent` user property in an MQTT packet). This allows you to link traces that originate from upstream instrumented applications. - If `false`, EMQX ignores any incoming trace context and always starts a new trace. 2. **Remote Sampling Decision**: If `follow_traceparent` is `true` and an incoming request contains a trace context that is already marked as "sampled", EMQX will respect this upstream decision and sample the trace, unless overridden by other rules. 3. **Whitelist Rules**: If the trace is not already sampled by a remote parent, you can define specific rules to force sampling for certain clients or topics. This is the most direct way to ensure you are tracing the activity you care about. - **ClientID whitelist**: Forces sampling for all root-level activities (connect, subscribe, publish, etc.). - **Topic whitelist**: Forces sampling for messages published to matching topics. > **Note:** This rule applies to the start of a trace (e.g., the `client.publish` span). It does not apply to the `broker.publish` span, which is responsible for message delivery to subscribers. 4. **Ratio-Based Sampling**: If no whitelist rule matches, the decision falls back to ratio-based sampling, which is controlled by the `sample_ratio` configuration. You can configure this ratio (from `0.0` to `1.0`) to control the percentage of traces that are captured. A value of `1.0` means 100% of traces will be captured, while `0.0` means none will be (unless they match a whitelist rule). 5. **Event Type Switches**: Even if a trace is selected by the ratio-based sampler, it is only generated if the relevant event type switch is enabled. These switches act as a global on/off for categories of spans. The available switches are: - `client_connect_disconnect`: A boolean switch to enable or disable tracing for client connect and disconnect events. - `client_subscribe_unsubscribe`: A boolean switch to enable or disable tracing for client subscribe and unsubscribe events. - `client_messaging`: A boolean switch to enable or disable tracing for client message publishing. - `trace_rule_engine`: A boolean switch to enable or disable tracing for the rule engine. 6. **Message Trace Level**: For spans related to QoS acknowledgements (e.g., `PUBACK`, `PUBREC`), you can control their creation based on a QoS level using the `msg_trace_level` switch. - `msg_trace_level`: This setting can be configured to a specific QoS level (0, 1, or 2) to control which acknowledgement spans are created based on the QoS of the original message. For example, if `msg_trace_level` is set to `1`, `PUBACK` spans will be created for QoS 1 messages. For QoS 2 messages, this setting will generate `PUBREC` spans, but not `PUBREL` or `PUBCOMP` spans. This helps to reduce the verbosity of traces for high-QoS message flows.