每周一篇——PLG(Promtail+Loki+Grafana)轻量日志方案
每周一篇——PLG轻量日志方案（Promtail+Grafana已改好可抄）
| 组件 | 作用 | 备注 |
| Loki | 日志存储 | 替代ES - 全文搜索会比ES慢 - CPU、内存、存储开销会比ES小 |
| Promtail | 收集日志agent | 替代filebeat |
| Grafana | 日志查询 | 替代Kibana |
Loki 存储结构
Grafana展示
id: 16970，基于这个进行二次开发
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "Container/Pod Log Quick Search(Loki as DataSource)",
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": 16970,
"graphTooltip": 0,
"id": 38,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "left",
"barAlignment": 0,
"drawStyle": "bars",
"fillOpacity": 100,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 24,
"x": 0,
"y": 0
},
"id": 3,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.1.5",
"targets": [
{
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"editorMode": "code",
"expr": "sum(count_over_time({namespace=\"$namespace\", pod=~\"$pod\"} |= \"$searchable_pattern\" [$__interval]))",
"legendFormat": "",
"queryType": "range",
"refId": "A"
}
],
"transparent": true,
"type": "timeseries"
},
{
"datasource": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd",
"description": "",
"gridPos": {
"h": 24,
"w": 24,
"x": 0,
"y": 3
},
"id": 2,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": true
},
"targets": [
{
"datasource": {
"0": "a",
"1": "0",
"2": "4",
"3": "a",
"4": "7",
"5": "4",
"6": "a",
"7": "8",
"8": "-",
"9": "8",
"10": "d",
"11": "b",
"12": "9",
"13": "-",
"14": "4",
"15": "a",
"16": "f",
"17": "f",
"18": "-",
"19": "a",
"20": "0",
"21": "a",
"22": "4",
"23": "-",
"24": "c",
"25": "1",
"26": "5",
"27": "2",
"28": "4",
"29": "d",
"30": "a",
"31": "2",
"32": "a",
"33": "f",
"34": "d",
"35": "d"
},
"expr": "{namespace=\"$namespace\", pod=~\"$pod\"} |~ \"(?i)$searchable_pattern\" ",
"hide": false,
"refId": "A"
}
],
"title": "Search Result",
"type": "logs"
}
],
"refresh": "1m",
"schemaVersion": 38,
"style": "dark",
"tags": [
"Loki",
"logging"
],
"templating": {
"list": [
{
"allValue": ".*",
"current": {
"selected": false,
"text": "fogcloud",
"value": "fogcloud"
},
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"definition": "label_values({namespace=~\".+\"}, namespace)",
"hide": 0,
"includeAll": false,
"label": "Namespace",
"multi": false,
"name": "namespace",
"options": [],
"query": "label_values({namespace=~\".+\"}, namespace)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": ".*",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"definition": "label_values({namespace=\"$namespace\"}, pod)",
"hide": 0,
"includeAll": true,
"label": "Pod",
"multi": false,
"name": "pod",
"options": [],
"query": "label_values({namespace=\"$namespace\"}, pod)",
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"current": {
"selected": true,
"text": "",
"value": ""
},
"hide": 0,
"label": "Search (case insensitive)",
"name": "searchable_pattern",
"options": [
{
"selected": true,
"text": "",
"value": ""
}
],
"query": "",
"skipUrlSync": false,
"type": "textbox"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
]
},
"timezone": "",
"title": "K8s-应用日志",
"uid": "7ecZpUkRk",
"version": 7,
"weekStart": ""
}
K8s报表
Linux报表
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "Container/Pod Log Quick Search(Loki as DataSource)",
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": 16970,
"graphTooltip": 0,
"id": 40,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "left",
"barAlignment": 0,
"drawStyle": "bars",
"fillOpacity": 100,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 24,
"x": 0,
"y": 0
},
"id": 3,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.1.5",
"targets": [
{
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"editorMode": "code",
"expr": "sum(count_over_time({app=~\"$app\",filename=~\"$Filename\", environment=\"$environment\"} |= \"$searchable_pattern\" [$__interval]))",
"legendFormat": "",
"queryType": "range",
"refId": "A"
}
],
"transparent": true,
"type": "timeseries"
},
{
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"description": "",
"gridPos": {
"h": 24,
"w": 24,
"x": 0,
"y": 3
},
"id": 2,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": true
},
"targets": [
{
"datasource": {
"0": "a",
"1": "0",
"2": "4",
"3": "a",
"4": "7",
"5": "4",
"6": "a",
"7": "8",
"8": "-",
"9": "8",
"10": "d",
"11": "b",
"12": "9",
"13": "-",
"14": "4",
"15": "a",
"16": "f",
"17": "f",
"18": "-",
"19": "a",
"20": "0",
"21": "a",
"22": "4",
"23": "-",
"24": "c",
"25": "1",
"26": "5",
"27": "2",
"28": "4",
"29": "d",
"30": "a",
"31": "2",
"32": "a",
"33": "f",
"34": "d",
"35": "d"
},
"editorMode": "code",
"expr": "{app=\"$app\", filename=~\"$Filename\", environment=\"$environment\"} |~ \"(?i)$searchable_pattern\" ",
"hide": false,
"range": true,
"refId": "A"
}
],
"title": "Search Result",
"type": "logs"
}
],
"refresh": "1m",
"schemaVersion": 38,
"style": "dark",
"tags": [
"Loki",
"logging"
],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "dev",
"value": "dev"
},
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"definition": "",
"hide": 0,
"includeAll": false,
"label": "Env",
"multi": false,
"name": "environment",
"options": [],
"query": {
"label": "environment",
"refId": "LokiVariableQueryEditor-VariableQuery",
"stream": "",
"type": 1
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
},
{
"allValue": ".*",
"current": {
"selected": false,
"text": "deye-app",
"value": "deye-app"
},
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"definition": "",
"hide": 0,
"includeAll": false,
"label": "App",
"multi": false,
"name": "app",
"options": [],
"query": {
"label": "app",
"refId": "LokiVariableQueryEditor-VariableQuery",
"stream": "{app=~\".+\"}",
"type": 1
},
"refresh": 2,
"regex": "deye-app",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": ".*",
"current": {
"selected": false,
"text": "/data/logs/auth-rpc/auth-rpc.log",
"value": "/data/logs/auth-rpc/auth-rpc.log"
},
"datasource": {
"type": "loki",
"uid": "a04a74a8-8db9-4aff-a0a4-c1524da2afdd"
},
"definition": "",
"hide": 0,
"includeAll": true,
"label": "Filename",
"multi": false,
"name": "Filename",
"options": [],
"query": {
"label": "filename",
"refId": "LokiVariableQueryEditor-VariableQuery",
"stream": "{app=~\".+\"}",
"type": 1
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"current": {
"selected": true,
"text": "",
"value": ""
},
"hide": 0,
"label": "Search (case insensitive)",
"name": "searchable_pattern",
"options": [
{
"selected": true,
"text": "",
"value": ""
}
],
"query": "",
"skipUrlSync": false,
"type": "textbox"
}
]
},
"time": {
"from": "now-15m",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
]
},
"timezone": "",
"title": "Linux-应用日志",
"uid": "7ecZpUkRk1",
"version": 15,
"weekStart": ""
}
部署
整套PLG的部署可以参考，https://cloud.tencent.com/developer/article/1915207
注意loki的数据盘 需要做持久化存储，线下可以选nfs、ceph、线上可以s3、nas之类的
loki存储配置
auth_enabled: false
chunk_store_config:
max_look_back_period: 0s
compactor:
shared_store: filesystem
working_directory: /data/loki/boltdb-shipper-compactor
ingester:
chunk_block_size: 262144
chunk_idle_period: 3m
chunk_retain_period: 1m
lifecycler:
ring:
replication_factor: 1
max_transfer_retries: 0
wal:
dir: /data/loki/wal
limits_config:
enforce_metric_name: false
max_entries_limit_per_query: 5000
reject_old_samples: true
reject_old_samples_max_age: 168h # 保存7天数据
memberlist:
join_members:
- 'loki-memberlist'
schema_config:
configs:
- from: "2020-10-24"
index:
period: 24h
prefix: index_
object_store: filesystem
schema: v11
store: boltdb-shipper
server:
grpc_listen_port: 9095
http_listen_port: 3100
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h
shared_store: filesystem
filesystem:
directory: /data/loki/chunks
table_manager:
retention_deletes_enabled: true
retention_period: 0s
promtail-配置K8s(daemonset)
server:
log_level: info
log_format: logfmt
http_listen_port: 3101
http_server_write_timeout: 60s
http_server_read_timeout: 60s
querier:
query_timeout: 30s
engine:
timeout: 30s
clients:
- url: http://loki:3100/loki/api/v1/push
positions:
filename: /run/promtail/positions.yaml
scrape_configs:
# See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
- job_name: kubernetes-pods
pipeline_stages:
- cri: {}
- multiline:
firstline: '^\d{4}-\d{2}-\d{2}'
max_lines: 128
max_wait_time: 3s
- replace:
expression: (\n\n)
replace: "\n"
- regex:
expression: "^(INFO|WARN|ERROR).*" # 正则表达式匹配日志级别开头的行并提取出级别
source: stream
target: log_level
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_controller_name
regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
action: replace
target_label: __tmp_controller_name
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- __meta_kubernetes_pod_label_app
- __tmp_controller_name
- __meta_kubernetes_pod_name
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: app
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_instance
- __meta_kubernetes_pod_label_instance
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: instance
- source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_component
- __meta_kubernetes_pod_label_component
regex: ^;*([^;]+)(;.*)?$
action: replace
target_label: component
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node_name
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement: $1
separator: /
source_labels:
- namespace
- app
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- action: replace
regex: true/(.*)
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
- __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
- __meta_kubernetes_pod_container_name
target_label: __path__
limits_config:
tracing:
enabled: false
promtail-配置linux
[root@deye-app-test-001 ~]$ cat /data/promtail/config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /data/promtail/positions.yaml
clients:
- url: http://10.180.200.16:38727/loki/api/v1/push #loki地址
scrape_configs:
- job_name: system
pipeline_stages:
#日志换行，匹配开头
- multiline:
firstline: '^\d{4}-\d{2}-\d{2}'
max_lines: 128
max_wait_time: 3s
#静态配置
static_configs:
- targets:
- localhost
labels:
job: linux-promtail-agent
host_name: deye-app-test-001 # 主机名，各主机不一样
environment: dev # 这个环境标签 -> {app="$app", filename=~"$Filename", environment="$environment"} |~ "(?i)$searchable_pattern"
app: deye-app # 此处名称设置为统一，配置上方便，搜索上需要过滤
__path__: /data/logs/*/*.log
Ansible 批量安装promtail
---
- name: Copy and execute
hosts: "ems-web-001"
tasks:
- name: Execute
shell:
cmd: mkdir -p /data/promtail/
- name: Copy
copy:
src: ./file/config.yaml
dest: /data/promtail/config.yaml
mode: 0644
- name: Copy
copy:
src: ./file/promtail
dest: /usr/bin/promtail
mode: 0755
- name: Copy
copy:
src: ./file/promtail.service
dest: /usr/lib/systemd/system/promtail.service
mode: 0644
- name: Execute
shell:
cmd: systemctl enable promtail
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /data/promtail/positions.yaml
clients:
- url: http://10.180.200.16:38727/loki/api/v1/push
scrape_configs:
- job_name: system
pipeline_stages:
#日志换行，匹配开头
- multiline:
firstline: '^\d{4}-\d{2}-\d{2}'
max_lines: 128
max_wait_time: 3s
#静态配置
static_configs:
- targets:
- localhost
labels:
job: linux-promtail-agent
host_name: ems-web-001
environment: test
app: nginx-gateway
__path__: /data/logs/openresty/*.log
[Unit]
Descriptinotallow=promtail
Documentatinotallow=https://github.com/grafana/loki/tree/master
After=network.target
[Service]
Type=simple
User=root
ExecStart=/usr/bin/promtail -config.file=/data/promtail/config.yaml
Restart=on-failure
[Install]
WantedBy=multi-user.target
还有二进制promtail 可执行文件，file/promtail
ansible-playbook deploy.yml