# Collect EMQX Logs in Kubernetes
# Task target
How to collect EMQX cluster logs through Telegraf and export them to the standard output of the container.
# Deploy Telegraf Operator
Telegraf is a server-based agent for collecting and sending all metrics and events from databases, systems, and IoT sensors. It supports four types of plugins, including input, output, aggregator and processor. More articles about Telegraf can be found at telegraf (opens new window), The documentation for telegraf-operator can be found in telegraf-operator (opens new window).
Execute the following command to deploy telegraf-operator
helm repo add influxdata https://helm.influxdata.com/
helm upgrade --install telegraf-operator influxdata/telegraf-operator
2
# Global Configuration - classes
The global configuration is mounted via secret, specifying the class name as logs, where
agent
is to configure telegraf agent, refer to the detailed definition: telegraf agent (opens new window)inputs.tail
is the tail plug-in used to configure the input and is defined in detail in tail (opens new window)outputs.file
is a file plug-in used to configure the output is defined in detail in file (opens new window)
apiVersion: v1
kind: Secret
metadata:
name: telegraf-operator-classes
namespace: default
stringData:
logs: |+
[agent]
interval = "60s"
flush_jitter = "5s"
flush_interval = "15s"
debug = true
quiet = false
metric_batch_size = 128
metric_buffer_limit = 256
[[inputs.tail]]
files = ["/opt/emqx/log/emqx.log.[1-9]"]
from_beginning = false
max_undelivered_lines = 64
character_encoding = "utf-8"
data_format = "grok"
grok_patterns = ['^%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02T15:04:05.999999999-07:00"} \[%{LOGLEVEL:level}\] (?m)%{GREEDYDATA:messages}$']
[inputs.tail.tags]
collection = "log"
[[outputs.file]]
files = ["stdout"]
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Save the above as classes.yaml
- Create secret
kubectl apply -f classes.yaml
- Check the created secret
kubectl get secret telegraf-operator-classes
The output is similar to:
NAME TYPE DATA AGE
telegraf-operator-classes Opaque 1 11h
2
# Deploy EMQX Cluster
Telegraf uses annotations to inject sidecar for Pod log collection, for a detailed definition of annotations refer to the documentation: telegraf annotations (opens new window)
Here are the relevant configurations for EMQX Custom Resource. You can choose the corresponding APIVersion based on the version of EMQX you wish to deploy. For specific compatibility relationships, please refer to EMQX Operator Compatibility:
# Check the Telegraf Logs
kubectl logs -f $pod_name -c telegraf