Monitor EMQX cluster by Prometheus and Grafana

Task Target

Deploy EMQX Exporter and monitor EMQX cluster by Prometheus and Grafana.

Deploy Prometheus and Grafana

Prometheus' deployment documentation can refer to Prometheus Grafana' deployment documentation can refer to Grafana

Deploy EMQX Cluster

Here are the relevant configurations for EMQX Custom Resource. You can choose the corresponding APIVersion based on the version of EMQX you wish to deploy. For specific compatibility relationships, please refer to EMQX Operator Compatibility:

Create API secret

emqx-exporter and Prometheus will pull metrics from EMQX dashboard API, so you need to sign in to dashboard to create an API secret.

Note that it is different to create a secret between EMQX 5 and EMQX 4.4 on the dashboard.

EMQX 5 create a new API KEY.
EMQX 4.4 create a new User instead of Application

Deploy EMQX Exporter

The emqx-exporter is designed to expose partial metrics that are not included in the EMQX Prometheus API. It is compatible with EMQX 4.4 and EMQX 5, both open-source and enterprise.

yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: emqx-exporter
  name: emqx-exporter-service
spec:
  ports:
    - name: metrics
      port: 8085
      targetPort: metrics
  selector:
    app: emqx-exporter
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-exporter
  labels:
    app: emqx-exporter
spec:
  selector:
    matchLabels:
      app: emqx-exporter
  replicas: 1
  template:
    metadata:
      labels:
        app: emqx-exporter
    spec:
      securityContext:
        runAsUser: 1000
      containers:
        - name: exporter
          image: emqx-exporter:latest
          imagePullPolicy: IfNotPresent
          args:
            # "emqx-dashboard-service-name" is the service name that creating by operator for exposing 18083 port
            - --emqx.nodes=${emqx-dashboard-service-name}:18083
            - --emqx.auth-username=${paste_your_new_api_key_here}
            - --emqx.auth-password=${paste_your_new_secret_here}
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
          ports:
            - containerPort: 8085
              name: metrics
              protocol: TCP
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
            requests:
              cpu: 100m
              memory: 20Mi

Set the arg "--emqx.nodes" to the service name that creating by operator for exposing 18083 port. Check out the service name by call kubectl get svc.

Save the above content as emqx-exporter.yaml, replace --emqx.auth-username and --emqx.auth-password with your new creating API secret, then execute the following command to deploy the emqx-exporter:

bash

kubectl apply -f emqx-exporter.yaml

Check the status of emqx-exporter pod。

bash

$ kubectl get po -l="app=emqx-exporter"

NAME      STATUS   AGE
emqx-exporter-856564c95-j4q5v   Running  8m33s

Configure Prometheus Monitor

Prometheus-operator uses PodMonitor and ServiceMonitor CRD to define how to monitor a set of pods or services dynamically.

yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: emqx
  labels:
    app.kubernetes.io/name: emqx
spec:
  podMetricsEndpoints:
    - interval: 5s
      path: /api/v5/prometheus/stats
      # the name of emqx dashboard containerPort
      port: dashboard
      relabelings:
        - action: replace
          # user-defined cluster name, requires unique
          replacement: emqx5
          targetLabel: cluster
        - action: replace
          # fix value, don't modify
          replacement: emqx
          targetLabel: from
        - action: replace
          # fix value, don't modify
          sourceLabels: ['pod']
          targetLabel: "instance"
  selector:
    matchLabels:
      # the label is the same as the label of emqx pod
      apps.emqx.io/instance: emqx
      apps.emqx.io/managed-by: emqx-operator
  namespaceSelector:
    matchNames:
      # modify the namespace if your EMQX cluster deployed in other namespace
      #- default
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: emqx-exporter
  labels:
    app: emqx-exporter
spec:
  selector:
    matchLabels:
      # the label is the same as the label of emqx exporter svc
      app: emqx-exporter
  endpoints:
    - port: metrics
      interval: 5s
      path: /metrics
      relabelings:
        - action: replace
          # user-defined cluster name, requires unique
          replacement: emqx5
          targetLabel: cluster
        - action: replace
          # fix value, don't modify
          replacement: exporter
          targetLabel: from
        - action: replace
          # fix value, don't modify
          sourceLabels: ['pod']
          regex: '(.*)-.*-.*'
          replacement: $1
          targetLabel: "instance"
        - action: labeldrop
          # fix value, don't modify
          regex: 'pod'
  namespaceSelector:
    matchNames:
      # modify the namespace if your exporter deployed in other namespace
      #- default

`path` indicates the path of the indicator collection interface. In EMQX 5, the path is: `/api/v5/prometheus/stats`. `selector.matchLabels` indicates the label of the matching Pod: `apps.emqx.io/instance: emqx`.

The value of targetLabel `cluster` represents the name of current cluster, make sure its uniqueness.

yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: emqx
  labels:
    app.kubernetes.io/name: emqx
spec:
  podMetricsEndpoints:
  - interval: 5s
    params:
      type:
        - prometheus
    path: /api/v4/emqx_prometheus
    # the name of emqx dashboard containerPort
    port: dashboard
    relabelings:
      - action: replace
        # user-defined cluster name, requires unique
        replacement: emqx4
        targetLabel: cluster
      - action: replace
        # fix value, don't modify
        replacement: emqx
        targetLabel: from
      - action: replace
        # fix value, don't modify
        sourceLabels: ['pod']
        targetLabel: "instance"
  selector:
    matchLabels:
      # the label is the same as the label of emqx pod
      apps.emqx.io/instance: emqx
      apps.emqx.io/managed-by: emqx-operator
  namespaceSelector:
    matchNames:
      # modify the namespace if your EMQX cluster deployed in other namespace
      #- default
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: emqx-exporter
  labels:
    app: emqx-exporter
spec:
  selector:
    matchLabels:
      # the label in emqx exporter svc
      app: emqx-exporter
  endpoints:
    - port: metrics
      interval: 5s
      path: /metrics
      relabelings:
        - action: replace
          # user-defined cluster name, requires unique
          replacement: emqx4
          targetLabel: cluster
        - action: replace
          # fix value, don't modify
          replacement: exporter
          targetLabel: from
        - action: replace
          # fix value, don't modify
          sourceLabels: ['pod']
          regex: '(.*)-.*-.*'
          replacement: $1
          targetLabel: "instance"
        - action: labeldrop
          # fix value, don't modify
          regex: 'pod'
  namespaceSelector:
    matchNames:
      # modify the namespace if your exporter deployed in other namespace
      #- default

path indicates the path of the indicator collection interface. In EMQX 4, the path is: /api/v4/emqx_prometheus. selector.matchLabels indicates the label of the matching Pod: apps.emqx.io/instance: emqx-ee. The value of targetLabel cluster represents the name of current cluster, make sure its uniqueness.

Save the above content as monitor.yaml and execute the following command:

bash

$ kubectl apply -f monitor.yaml

View EMQX Indicators on Prometheus

Open the Prometheus interface, switch to the Graph page, and enter emqx to display as shown in the following figure:

Switch to the Status -> Targets page, the following figure is displayed, and you can see all monitored EMQX Pod information in the cluster:

Import Grafana Templates

Import all dashboard templates. Open the main dashboard EMQX and enjoy yourself!

Monitor EMQX cluster by Prometheus and Grafana ​

Task Target ​

Deploy Prometheus and Grafana ​

Deploy EMQX Cluster ​

Create API secret ​

Deploy EMQX Exporter ​

Configure Prometheus Monitor ​

View EMQX Indicators on Prometheus ​

Import Grafana Templates ​

Monitor EMQX cluster by Prometheus and Grafana

Task Target

Deploy Prometheus and Grafana

Deploy EMQX Cluster

Create API secret

Deploy EMQX Exporter

Configure Prometheus Monitor

View EMQX Indicators on Prometheus

Import Grafana Templates