# Configure EMQX Logs Collection

# Task target

  • How to collect EMQX cluster logs through Telegraf and export them to the standard output of the container

# Deploy telegraf-operator

Telegraf is a server-based agent for collecting and sending all metrics and events from databases, systems, and IoT sensors. It supports four types of plugins, including input, output, aggregator and processor. More articles about Telegraf can be found at: telegraf (opens new window), The documentation for telegraf-operator can be found in: telegraf-operator (opens new window)

Execute the following command to deploy telegraf-operator

helm repo add influxdata https://helm.influxdata.com/
helm upgrade --install telegraf-operator influxdata/telegraf-operator
1
2

# Global Configuration - classes

The global configuration is mounted via secret, specifying the class name as logs

agent Used to configure telegraf agent, refer to the detailed definition: telegraf agent (opens new window)

inputs.tail The tail plug-in used to configure the input is defined in detail in: tail (opens new window)

outputs.file The file plug-in used to configure the output is defined in detail in: file (opens new window)

apiVersion: v1
kind: Secret
metadata:
  name: telegraf-operator-classes
  namespace: default
stringData:
  logs: |+
    [agent]
      interval = "60s"
      flush_jitter = "5s"
      flush_interval = "15s"
      debug = true
      quiet = false
      metric_batch_size = 128
      metric_buffer_limit = 256

    [[inputs.tail]]
      files = ["/opt/emqx/log/emqx.log.[1-9]"]
      from_beginning = false
      max_undelivered_lines = 64
      character_encoding = "utf-8"
      data_format = "grok"
      grok_patterns = ['^%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02T15:04:05.999999999-07:00"} \[%{LOGLEVEL:level}\] (?m)%{GREEDYDATA:messages}$']
      [inputs.tail.tags]
        collection = "log"

    [[outputs.file]]
      files = ["stdout"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Save the above as: classes.yaml

  • Create secret
kubectl apply -f classes.yaml
1
  • Check the created secret
kubectl get secret telegraf-operator-classes
1

The output is similar to:

NAME                        TYPE     DATA   AGE
telegraf-operator-classes   Opaque   1      11h
1
2

# Deploy EMQX cluster

Telegraf uses annotations to inject sidecar for Pod log collection, for a detailed definition of annotations refer to the documentation: telegraf annotations (opens new window)

Save the above as: emqx-telegraf.yaml

  • Deploy EMQX cluster
kubectl apply -f emqx-telegraf.yaml
1
  • Checking EMQX Cluster Status
  • Check the collected logs

The collected EMQX logs are output to the standard output.