Grafana可观测性平台搭建实战：监控告警与多数据源联动

小编 • • 网站运维

可观测性平台的架构规划

Grafana可观测性平台通过统一的数据采集、存储和可视化层，将指标（Metrics）、日志（Logs）和链路追踪（Traces）三大支柱整合到一个面板。核心组件包括：Prometheus作为时序数据库存储指标，Loki作为日志聚合系统，Tempo或Jaeger存储分布式链路追踪，Grafana作为统一可视化前端，Alertmanager处理告警路由。规划时需根据业务规模确定采集频率、数据保留周期和存储容量。

Prometheus指标采集与存储配置

Prometheus采用拉模式采集指标，需配置scrape_configs定义采集目标。以下是一个覆盖主机、容器和应用的完整配置：

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  retention: 30d

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  # 主机指标
  - job_name: 'node'
    static_configs:
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '(.*):.*'
        replacement: '$1'

  # 容器指标
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  # 应用指标
  - job_name: 'app'
    metrics_path: /metrics
    static_configs:
      - targets: ['app1:8080', 'app2:8080']

  # 黑盒探测
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com
          - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - target_label: __address__
        replacement: blackbox-exporter:9115

retention设置为30天，配合远程存储方案（如Thanos或Mimir）可实现长期存储。relabel_configs用于在采集时为指标添加自定义标签，方便后续按主机或服务维度筛选。

Loki日志聚合系统部署

Loki是Grafana生态的日志聚合系统，采用类似Prometheus的标签索引方式，只索引日志的元数据而非全文，存储成本远低于Elasticsearch。部署架构使用Promtail作为日志采集Agent：

# promtail-config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # 系统日志
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: system
          host: node-1
          __path__: /var/log/*.log

  # 容器日志（Docker）
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        target_label: container
        regex: '/(.*)'

  # 应用日志（带多行解析）
  - job_name: app
    static_configs:
      - targets: [localhost]
        labels:
          job: app
          app: myservice
          __path__: /opt/app/logs/*.log
    pipeline_stages:
      - multiline:
          firstline: '^\d{4}-\d{2}-\d{2}'
          max_wait_time: 3s
      - regex:
          expression: '^(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P\w+) (?P.*)$'
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: '2006-01-02 15:04:05'

pipeline_stages中的multiline配置处理Java异常堆栈等多行日志，regex提取日志级别和消息内容作为标签，timestamp解析日志时间戳确保日志时间准确。

Grafana数据源配置与Dashboard设计

Grafana支持多种数据源，可观测性平台需要同时配置Prometheus、Loki和Tempo。通过provisioning机制自动配置：

# datasources.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      manageAlerts: true

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      maxLines: 1000

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      tracesToLogs:
        datasourceUid: loki
        filterByTraceID: true
        tags: ['service.name', 'http.status_code']

  - name: Alertmanager
    type: alertmanager
    access: proxy
    url: http://alertmanager:9093
    jsonData:
      implementation: prometheus

tracesToLogs配置实现了链路追踪到日志的联动跳转：在Tempo中查看某条Trace时，可以一键跳转到Loki查看同一时间段、同一服务的日志。这是可观测性平台”三支柱联动”的核心价值。

告警规则与Alertmanager路由

告警规则定义在Prometheus中，告警路由和通知方式在Alertmanager中配置。一个完整的告警体系需要区分告警级别、接收渠道和抑制规则：

# /etc/prometheus/rules/infra_alerts.yml
groups:
- name: infrastructure
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 10m
    labels:
      severity: warning
      team: ops
    annotations:
      summary: "CPU使用率过高 {{ $labels.instance }}"
      description: "CPU使用率: {{ $value }}%，持续超过10分钟"

  - alert: DiskSpaceLow
    expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
    for: 5m
    labels:
      severity: critical
      team: ops
    annotations:
      summary: "磁盘空间不足 {{ $labels.instance }} {{ $labels.mountpoint }}"
      description: "磁盘使用率: {{ $value }}%"

  - alert: ServiceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
      team: ops
    annotations:
      summary: "服务不可达 {{ $labels.instance }}"
      description: "Job: {{ $labels.job }}, Target: {{ $labels.instance }}"

# alertmanager.yml
route:
  receiver: default
  group_by: ['alertname', 'instance', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "critical"
      receiver: critical-channel
      group_wait: 10s
      repeat_interval: 1h
    - matchers:
        - severity = "warning"
      receiver: warning-channel
      repeat_interval: 4h

receivers:
  - name: default
    webhook_configs:
      - url: 'http://webhook.example.com/alert'

  - name: critical-channel
    webhook_configs:
      - url: 'http://dingtalk.example.com/critical'
    email_configs:
      - to: 'ops-team@example.com'
        send_resolved: true

  - name: warning-channel
    webhook_configs:
      - url: 'http://dingtalk.example.com/warning'

inhibit_rules:
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"
    equal: ['instance', 'alertname']

inhibit_rules配置了告警抑制：当同一实例的critical告警触发时，自动抑制对应的warning告警，避免告警风暴。group_by控制告警分组维度，同一组告警合并为一条通知。

日志告警与SLO监控

除了基于指标的告警，日志告警能捕获指标无法覆盖的异常。Loki支持通过LogQL规则定义日志告警：

# Loki ruler配置
ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  alertmanager_url: http://alertmanager:9093
  enable_alertmanager_v2: true

# /loki/rules/logs_alerts.yml
groups:
- name: log_alerts
  rules:
  - alert: HighErrorRate
    expr: |
      sum by(job, app) (
        rate({job="app", level="ERROR"}[5m])
      ) / sum by(job, app) (
        rate({job="app"}[5m])
      ) > 0.1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "应用错误率过高 {{ $labels.app }}"
      description: "错误日志占比超过10%"

  - alert: OOMKilled
    expr: count_over_time({job="system"} |= "Out of memory: Killed process" [10m]) > 0
    labels:
      severity: critical
    annotations:
      summary: "进程被OOM Killer终止"

SLO（Service Level Objective）监控是可观测性平台的高级用法。通过错误预算（Error Budget）衡量服务可用性：

# SLO规则示例
- alert: SLOBurnRateHigh
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      / sum(rate(http_requests_total[1h]))
    ) > 0.01
    and
    (
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m]))
    ) > 0.01
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "SLO错误预算消耗过快"
    description: "5xx错误率超过1%的SLO目标"

这个规则使用多窗口多燃烧率策略：1小时窗口和5分钟窗口同时超过阈值才告警，减少误报同时保证灵敏度。

平台运维与性能调优

可观测性平台本身的性能同样需要关注。Prometheus在大规模部署时可能遇到内存占用过高、查询慢的问题。优化手段包括：减少采集频率、使用记录规则预计算常用查询、启用查询缓存。Loki的写入性能受限于索引，大规模日志场景建议使用BoltDB Shipper模式或启用查询缓存。Grafana面板过多会影响加载速度，建议按业务域拆分Dashboard，单面板Panel数量控制在20个以内。定期清理历史数据和归档冷数据到对象存储，控制热存储规模在合理范围。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/grafana-ke-guan-ce-xing-ping-tai-da-jian-shi-zhan-jian-kong/

赞 (0)
小编小编
0
上一篇 4小时前
下一篇 4小时前

相关推荐