Prometheus Monitoring

The EMQX Platform provides a Prometheus API that allows you to easily call the API to monitor key metrics of the EMQX Platform. This page introduces how to configure the Prometheus service, read key metrics from the EMQX Platform API, and use Grafana templates to visualize the metrics.

Note

This feature is only available with the EMQX Dedicated edition.

API Configuration

Find the Deployment API Key on the Overview page of your deployment to obtain the API address. Click + New Application to get the APP ID and APP Secret.

Deployment Metrics URI

GET /deployment_metrics

Returns cluster metrics for Prometheus collection.

Query Parameters:

None

Request Message

None

Request Example

curl -u app_id:app_secret -X GET {api}/deployment_metrics

Response

Metrics	Description	Type
deployment_emqx_sessions_count	The current number of sessions for the cluster	Gauge
deployment_emqx_connections_count	The current number of connections for the cluster	Gauge
deployment_emqx_sent_msg_rate	The rate of messages sent per second	Gauge
deployment_emqx_received_msg_rate	The rate of message receiving.	Gauge
deployment_emqx_metrics_cluster_sessions_count	Number of sessions in the cluster	Gauge
deployment_emqx_metrics_cluster_sessions_max	Historical maximum number of sessions in the cluster	Gauge
deployment_emqx_metrics_delayed_count	Number of delayed messages	Gauge
deployment_emqx_metrics_delayed_max	Historical maximum number of delayed messages	Gauge
deployment_emqx_metrics_live_connections_count	Number of live connections	Gauge
deployment_emqx_metrics_live_connections_max	Historical maximum number of live connections	Gauge
deployment_emqx_disconnected_durable_sessions	Number of disconnected sessions that are preserved	Gauge
deployment_emqx_dropped_msg_rate	Rate of messages that were dropped	Gauge
deployment_emqx_license_quota	Maximum number of connections allowed by the license	Gauge
deployment_emqx_node_count	The number of nodes in the EMQX cluster	Gauge
deployment_emqx_persisted_rate	Rate of messages that were persisted to storage	Gauge
deployment_emqx_retained_msg_count	Current count of retained messages	Gauge
deployment_emqx_shared_subscriptions	Number of shared subscriptions	Gauge
deployment_emqx_subscriptions	Total number of subscriptions	Gauge
deployment_emqx_subscriptions_durable	Number of durable subscriptions that persist across sessions	Gauge
deployment_emqx_subscriptions_ram	Number of non-durable subscriptions stored in RAM	Gauge
deployment_emqx_topics	Total number of topics	Gauge
deployment_emqx_transformation_failed_rate	Rate of message transformation failures	Gauge
deployment_emqx_transformation_succeeded_rate	Rate of successful message transformations	Gauge
deployment_emqx_validation_failed_rate	Rate of message validation failures	Gauge
deployment_emqx_validation_succeeded_rate	Rate of successful message validations	Gauge
deployment_emqx_metrics_topics_count	Number of current topics	Gauge
deployment_emqx_metrics_topics_max	Maximum number of topics	Gauge
deployment_emqx_metrics_subscriptions_max	Maximum number of subscriptions	Gauge
deployment_emqx_metrics_subscriptions_count	Current number of subscriptions	Gauge
deployment_emqx_metrics_subscribers_max	Maximum number of subscribers	Gauge
deployment_emqx_metrics_subscribers_count	Current number of subscribers	Gauge
deployment_emqx_metrics_connections_count	Current number of connections	Gauge
deployment_emqx_metrics_connections_max	Maximum number of connections	Gauge
deployment_emqx_metrics_channels_count	Current number of channels	Gauge
deployment_emqx_metrics_channels_max	Maximum number of channels	Gauge
deployment_emqx_metrics_sessions_count	Current number of sessions	Gauge
deployment_emqx_metrics_sessions_max	Maximum number of sessions	Gauge
deployment_emqx_metrics_suboptions_count	Current number of subscription options	Gauge
deployment_emqx_metrics_suboptions_max	Maximum number of subscription options	Gauge
deployment_emqx_metrics_subscriptions_shared_count	Current number of shared subscriptions	Gauge
deployment_emqx_metrics_subscriptions_shared_max	Maximum number of shared subscriptions	Gauge
deployment_emqx_metrics_retained_count	Current number of retained messages	Gauge
deployment_emqx_metrics_retained_max	Historical maximum number of retained messages	Gauge
deployment_emqx_cluster_status	The status of the EMQX cluster	Gauge
deployment_emqx_metrics_durable_subscriptions_max	The max number of durable subscriptions for the current cluster	Counter
deployment_emqx_metrics_durable_subscriptions_count	The current number of durable subscriptions for the current cluster	Counter
deployment_emqx_metrics_client_disconnected_reason	Listener shutdown counts labeled by reason	Counter
deployment_emqx_metrics_bytes_received	Number of received bytes	Counter
deployment_emqx_metrics_bytes_sent	Number of sent bytes	Counter
deployment_emqx_metrics_packets_received	Number of received packets	Counter
deployment_emqx_metrics_packets_sent	Number of sent packets	Counter
deployment_emqx_metrics_packets_connack_auth_error	Number of sent CONNACK messages with auth error	Counter
deployment_emqx_metrics_packets_connack_error	Number of sent CONNACK packets with error	Counter
deployment_emqx_metrics_packets_connack_sent	Number of sent CONNACK packets	Counter
deployment_emqx_metrics_packets_publish_received	Number of received PUBLISH packets	Counter
deployment_emqx_metrics_packets_publish_sent	Number of sent PUBLISH packets	Counter
deployment_emqx_metrics_packets_publish_inuse	Number of PUBLISH packets in use	Counter
deployment_emqx_metrics_packets_publish_auth_error	Number of PUBLISH packets with auth error	Counter
deployment_emqx_metrics_packets_publish_error	Number of PUBLISH packets with error	Counter
deployment_emqx_metrics_packets_publish_dropped	Number of dropped PUBLISH packets	Counter
deployment_emqx_metrics_packets_puback_received	Number of received PUBACK packets	Counter
deployment_emqx_metrics_packets_puback_sent	Number of sent PUBACK packets	Counter
deployment_emqx_metrics_packets_puback_inuse	Number of PUBACK packets in use	Counter
deployment_emqx_metrics_packets_puback_missed	Number of missed PUBACK packets	Counter
deployment_emqx_metrics_packets_pubrec_received	Number of received PUBREC packets	Counter
deployment_emqx_metrics_packets_pubrec_sent	Number of sent PUBREC packets	Counter
deployment_emqx_metrics_packets_pubrec_inuse	Number of PUBREC packets in use	Counter
deployment_emqx_metrics_packets_pubrec_missed	Number of missed PUBREC packets	Counter
deployment_emqx_metrics_packets_pubrel_received	Number of received PUBREL packets	Counter
deployment_emqx_metrics_packets_pubrel_sent	Number of sent PUBREL packets	Counter
deployment_emqx_metrics_packets_pubrel_missed	Number of missed PUBREL packets	Counter
deployment_emqx_metrics_packets_pubcomp_received	Number of received PUBCOMP packets	Counter
deployment_emqx_metrics_packets_pubcomp_sent	Number of sent PUBCOMP packets	Counter
deployment_emqx_metrics_packets_pubcomp_inuse	Number of PUBCOMP packets in use	Counter
deployment_emqx_metrics_packets_pubcomp_missed	Number of missed PUBCOMP packets	Counter
deployment_emqx_metrics_packets_subscribe_received	Number of received SUBSCRIBE packets	Counter
deployment_emqx_metrics_packets_subscribe_error	Number of SUBSCRIBE packets with error	Counter
deployment_emqx_metrics_packets_subscribe_auth_error	Number of SUBSCRIBE packets with auth error	Counter
deployment_emqx_metrics_packets_suback_sent	Number of sent SUBACK packets	Counter
deployment_emqx_metrics_packets_unsubscribe_received	Number of received UNSUBSCRIBE packets	Counter
deployment_emqx_metrics_packets_unsubscribe_error	Number of UNSUBSCRIBE packets with error	Counter
deployment_emqx_metrics_packets_unsuback_sent	Number of sent UNSUBACK packets	Counter
deployment_emqx_metrics_packets_pingreq_received	Number of received PINGREQ packets	Counter
deployment_emqx_metrics_packets_pingresp_sent	Number of sent PINGRESP packets	Counter
deployment_emqx_metrics_packets_disconnect_received	Number of received DISCONNECT packets	Counter
deployment_emqx_metrics_packets_disconnect_sent	Number of sent DISCONNECT packets	Counter
deployment_emqx_metrics_packets_auth_received	Number of received AUTH packets	Counter
deployment_emqx_metrics_packets_auth_sent	Number of sent AUTH packets	Counter
deployment_emqx_metrics_delivery_dropped_too_large	Number of dropped too large deliveries	Counter
deployment_emqx_metrics_delivery_dropped_queue_full	Number of dropped deliveries due to full queue	Counter
deployment_emqx_metrics_delivery_dropped_qos0_msg	Number of dropped QoS 0 messages	Counter
deployment_emqx_metrics_delivery_dropped_expired	Number of expired message deliveries	Counter
deployment_emqx_metrics_delivery_dropped_no_local	Number of deliveries with no local clients	Counter
deployment_emqx_metrics_delivery_dropped	Total number of dropped deliveries	Counter
deployment_emqx_metrics_messages_delayed	Number of delayed messages	Counter
deployment_emqx_metrics_messages_delivered	Number of delivered messages	Counter
deployment_emqx_metrics_messages_dropped	Number of dropped messages	Counter
deployment_emqx_metrics_messages_dropped_expired	Number of expired messages	Counter
deployment_emqx_metrics_messages_dropped_no_subscribers	Number of messages dropped due to no subscribers	Counter
deployment_emqx_metrics_messages_forward	Number of forwarded messages	Counter
deployment_emqx_metrics_messages_publish	Number of published messages	Counter
deployment_emqx_metrics_messages_qos0_received	Number of QoS 0 messages received	Counter
deployment_emqx_metrics_messages_qos2_received	Number of QoS 2 messages received	Counter
deployment_emqx_metrics_messages_qos1_received	Number of QoS 1 messages received	Counter
deployment_emqx_metrics_messages_qos0_sent	Number of QoS 0 messages sent	Counter
deployment_emqx_metrics_messages_qos1_sent	Number of QoS 1 messages sent	Counter
deployment_emqx_metrics_messages_qos2_sent	Number of QoS 2 messages sent	Counter
deployment_emqx_metrics_messages_received	Total number of received messages	Counter
deployment_emqx_metrics_messages_sent	Total number of sent messages	Counter
deployment_emqx_metrics_messages_retained	Number of retained messages	Counter
deployment_emqx_metrics_messages_acked	Number of acknowledged messages	Counter
deployment_emqx_metrics_client_auth_anonymous	Number of anonymous client authentications	Counter
deployment_emqx_metrics_client_connect	Number of client connections	Counter
deployment_emqx_metrics_client_authenticate	Number of client authentications	Counter
deployment_emqx_metrics_client_connack	Number of CONNACK packets sent	Counter
deployment_emqx_metrics_client_connected	Number of currently connected clients	Counter
deployment_emqx_metrics_client_disconnected	Number of client disconnections	Counter
deployment_emqx_metrics_client_subscribe	Number of client subscriptions	Counter
deployment_emqx_metrics_client_unsubscribe	Number of client unsubscriptions	Counter
deployment_emqx_metrics_session_created	Number of sessions created	Counter
deployment_emqx_metrics_session_discarded	Number of discarded sessions	Counter
deployment_emqx_metrics_session_resumed	Number of resumed sessions	Counter
deployment_emqx_metrics_session_terminated	Number of terminated sessions	Counter
deployment_emqx_metrics_authentication_failure	Number of failed authentications	Counter
deployment_emqx_metrics_authentication_success	Number of successful authentications	Counter
deployment_emqx_metrics_authentication_success_anonymous	Number of successful anonymous authentications	Counter
deployment_emqx_metrics_authorization_allow	Number of successful authorizations	Counter
deployment_emqx_metrics_authorization_cache_hit	Number of cache hits during authorization	Counter
deployment_emqx_metrics_authorization_cache_miss	Number of cache misses during authorization	Counter
deployment_emqx_metrics_authorization_deny	Number of authorization denies	Counter
deployment_emqx_metrics_authorization_matched_allow	Number of successful authorization matches	Counter
deployment_emqx_metrics_authorization_matched_deny	Number of authorization deny matches	Counter
deployment_emqx_metrics_authorization_nomatch	Number of authorization no match	Counter
deployment_emqx_metrics_authorization_superuser	Number of superuser authorizations	Counter
deployment_emqx_metrics_client_authorize	client.authorize hook trigger times	Counter
deployment_emqx_metrics_packets_connect	Number of sent CONNECT packets	Counter
deployment_emqx_metrics_session_takenover	session.takenover hook trigger times	Counter

Data Integration Metrics URI

GET /deployment_metrics/data_integration

Returns data integration metrics for Prometheus collection.

Query Parameters:

None

Request Message

None

Request Example

curl -u app_id:app_secret -X GET {api}/deployment_metrics/data_integration

Response

• Each resource status
• Each rule status
• The count of different match statuses for each rule ("matched", "passed", "failed", "exception", "no_result")
• The count of all action processing statuses under each rule ("success", "failed", "taken")

Metrics	Description	Type	Variable Labels
deployment_emqx_resource_status	The current status of a specific resource.	Gauge	resource_id
deployment_emqx_rule_status	The current status of a specific rule.	Gauge	rule_id
deployment_emqx_rule_matched_count	The total matches for a specific rule.	Gauge	rule_id, match_status
deployment_emqx_rule_action_execution_count	The execution count for actions of a specific rule.	Gauge	rule_id, execution_status
deployment_emqx_action_dropped_count	The count of dropped events for a specific action	Counter	drop_reason
deployment_emqx_action_enable	Number of emqx nodes with action enabled	Gauge	status
deployment_emqx_action_events_count	The count of events for a specific action	Counter	event, action_id
deployment_emqx_action_queued_count	The count of queued events for a specific action	Gauge	action_id, queue_status
deployment_emqx_action_retried_count	The count of retried events for a specific action	Counter	action_id, retry_status
deployment_emqx_action_status	The current status of a specific action	Gauge	action_id

Authentication and Authorization Metrics URI

GET /deployment_metrics/auth

Returns authentication and authorization metrics for Prometheus collection.

Query Parameters:

None

Request Message

None

Request Example

curl -u app_id:app_secret -X GET {api}/deployment_metrics/auth

Response

Metrics	Description	Type	Variable Labels
deployment_emqx_authn_enable	Is the specified auth method enabled	Gauge	id
deployment_emqx_authn_match_count	Statistics for each auth method	Gauge	id, status
deployment_emqx_authn_status	The availability status of each auth n method	Gauge	id
deployment_emqx_authz_enable	Is the specified autz method enabled	Gauge	id
deployment_emqx_authz_match_count	Statistics for each autz method	Gauge	id, status
deployment_emqx_authz_status	The availability status of each auth z method	Gauge	id

Smart Data Hub Metrics URI

GET /deployment_metrics/data_hub

Returns smart data hub metrics for Prometheus collection.

Query Parameters:

None

Request Message

None

Request Example

curl -u app_id:app_secret -X GET {api}/deployment_metrics/data_hub

Response

Metrics	Description	Type	Variable Labels
emqx_message_transformation_enable	Message transformation enable status	Gauge	validation_name
emqx_message_transformation_stats	Transformation statisitcs	Counter	stats, validation_name
emqx_schema_validation_enable	Schema validation enable status	Gauge	validation_name
emqx_schema_validation_stats	Schema validation statistics	Counter	stats, validation_name

Prometheus Configuration

1. Install Prometheus:

   wget -c https://github.com/prometheus/prometheus/releases/download/v2.35.0-rc0/prometheus-2.35.0-rc0.linux-amd64.tar.gz
   tar xvfz prometheus-*.tar.gz

2. Modify the configuration file.

   Go to the monitoring directory specified for your Prometheus service and modify the scrape_configs section of the configuration file prometheus.yml as shown in the example below.

   scrape_configs:
     - job_name: 'emqx_cloud_deployment_metrics'
       scheme: 'https'
       static_configs:
         - targets: [ 'xxxx:8443' ]
       metrics_path: "/api/v5/deployment_metrics"
       params:
         type: [ "prometheus" ]
       basic_auth:
         username: 'APP ID'
         password: 'APP Secret'
     - job_name: 'emqx_cloud_deployment_data_integration_metrics'
       scheme: 'https'
       static_configs:
         - targets: [ 'xxxx:8443' ]
       metrics_path: "/api/v5/deployment_metrics/data_integration"
       params:
         type: [ "prometheus" ]
       basic_auth:
         username: 'APP ID'
         password: 'APP Secret'

3. Launch and check service status.

   Launch Prometheus:

    ./prometheus --config.file=prometheus.yml

   Access your Prometheus service via your local IP with the corresponding port, e.g. x.x.x.x:9090, and check Status-Targets to confirm that the new scrape_config file has been read. If the status shows an exception, you should check the configuration file and restart the Prometheus service.

   

Grafana Configuration

1. Install and launch Grafana:

    wget https://dl.grafana.com/enterprise/release/grafana-enterprise-8.4.6.linux-amd64.tar.gz
    tar -zxvf grafana-enterprise-8.4.6.linux-amd64.tar.gz
    ./bin/grafana-server

2. Configure Grafana.

   Access Grafana Dashboard via the local IP + the corresponding port, e.g. x.x.x.x:3000. The initial ID and password are admin. You can change the password when logging in for the first time.

   

3. Import Grafana Data Templates.

   The EMQX Platform provides template files for Grafana dashboards. These templates include displays of all EMQX Platform monitoring data. Users can directly import these templates into Grafana to generate charts showing the monitoring status of EMQX. Click here to get the template file.

   You can upload the local grafana_template.json file via "Upload JSON file" or manually configure it via "Import via panel json".

   

Metrics Details

After completing the system setup and running it for a period, the data collected by EMQX Platform Prometheus will be displayed on Grafana. This includes historical statistics on the number of clients, subscriptions, topics, messages, packets, and other business information. You can view charts corresponding to each metric and detailed information at specific points in time.

Prometheus tracks the following metrics data for your EMQX Platform deployment:

Metrics	Type	Description
deployment_emqx_connections_count	gauge	Number of cluster connections
deployment_emqx_sessions_count	gauge	Total number of sessions
deployment_emqx_messages_rate	gauge	Total message rate
deployment_emqx_messages_send_rate	gauge	Message send rate
deployment_emqx_messages_receive_rate	gauge	Message receive rate
deployment_emqx_metrics_bytes_received	gauge	Number of bytes received
deployment_emqx_metrics_bytes_sent	gauge	Number of bytes sent
deployment_emqx_metrics_retained_count	gauge	Number of retained messages
deployment_emqx_metrics_messages_received	counter	Number of Messages received
deployment_emqx_metrics_messages_qos0_received	counter	Number of QoS=0 messages received
deployment_emqx_metrics_messages_qos1_received	counter	Number of QoS=1 messages received
deployment_emqx_metrics_messages_qos2_received	counter	Number of QoS=2 messages received
deployment_emqx_metrics_messages_dropped	counter	Number of messages dropped at receive stage
deployment_emqx_metrics_messages_dropped_expired	counter	Number of messages dropped due to expiration
deployment_emqx_metrics_messages_dropped_no_subscribers	counter	Number of messages dropped due to no subscribers
deployment_emqx_metrics_messages_sent	counter	Number of messages sent
deployment_emqx_metrics_messages_qos0_sent	counter	Number of QoS=0 messages sent
deployment_emqx_metrics_messages_qos1_sent	counter	Number of QoS=1 messages sent
deployment_emqx_metrics_messages_qos2_sent	counter	Number of QoS=2 messages sent
deployment_emqx_metrics_delivery_dropped	counter	Number of messages dropped at the delivery stage
deployment_emqx_metrics_delivery_dropped_too_large	counter	Number of messages dropped due to size
deployment_emqx_metrics_delivery_dropped_queue_full	counter	Number of messages dropped due to full queue
deployment_emqx_metrics_delivery_dropped_no_local	counter	Number of messages dropped due to No Local
deployment_emqx_metrics_delivery_dropped_expired	counter	Number of messages dropped due to expiration
deployment_emqx_metrics_topics_count	gauge	Number of topics
deployment_emqx_metrics_subscriptions_count	gauge	Number of subscriptions
deployment_emqx_metrics_subscriptions_shared_count	gauge	Number of shared subscriptions
deployment_emqx_metrics_subscribers_count	gauge	Number of subscribers
deployment_emqx_metrics_authentication_success	counter	Number of successful authentications
deployment_emqx_metrics_authentication_failure	counter	Number of failed authentications
deployment_emqx_metrics_client_authorize	counter	Number of client authorizations
deployment_emqx_metrics_authorization_allow	counter	Number of successful authorizations
deployment_emqx_metrics_authorization_deny	counter	Number of authorization denials
deployment_emqx_metrics_authorization_matched_allow	counter	Number of successful authorization matches
deployment_emqx_metrics_authorization_matched_deny	counter	Number of authorization denial matches
deployment_emqx_metrics_authorization_nomatch	counter	Number of authorization mismatches
deployment_emqx_metrics_packets_received	counter	Number of packets received
deployment_emqx_metrics_packets_sent	counter	Number of packets sent
deployment_emqx_metrics_packets_connect_received	counter	Number of CONNET packets received
deployment_emqx_metrics_packets_connack_sent	counter	Number of CONNACK packets sent
deployment_emqx_metrics_packets_disconnect_received	counter	Number of DISCONNECT packets received
deployment_emqx_metrics_packets_disconnect_sent	counter	Number of DISCONNECT packets sent
deployment_emqx_metrics_packets_publish_received	counter	Number of PUBLISH packets received
deployment_emqx_metrics_packets_publish_sent	counter	Number of PUBLISH packets sent
deployment_emqx_metrics_packets_subscribe_received	counter	Number of SUBSCRIBE packets received
deployment_emqx_metrics_packets_unsubscribe_received	counter	Number of UNSUBSCRIBE packets received
deployment_emqx_metrics_packets_pingreq_received	counter	Number of PINGREQ packets received
deployment_emqx_metrics_packets_pingresp_sent	counter	Number of PINGRESP packets sent
deployment_emqx_metrics_packets_auth_received	counter	Number of AUTH packets received
deployment_emqx_metrics_packets_auth_sent	counter	Number of AUTH packets sent
deployment_emqx_rule_status	gauge	Rule status
deployment_emqx_rule_matched_rate	gauge	Rule match rate
deployment_emqx_rule_matched_count	gauge	Number of rule matches
deployment_emqx_rule_action_execution_count	gauge	Number of rule action executions