监控告警体系设计实战：从Prometheus指标采集到故障自动响应全链路

监控体系的分层架构设计

生产环境的监控不是装一个Prometheus就完事。合理的监控体系需要分四层构建：基础设施层（CPU/内存/磁盘/网络）、应用层（QPS/延迟/错误率）、业务层（订单量/支付成功率/用户活跃度）、用户体验层（页面加载/核心交互延迟）。每一层的告警阈值、响应级别、处理流程各不相同。混为一谈的告警只会让值班工程师在噪音中漏掉真正关键的故障信号。

Prometheus指标采集与Exporter部署

Prometheus采用Pull模式采集指标，每台目标节点部署对应的Exporter暴露metrics端点。核心组件的部署配置：

# prometheus.yml - 主配置文件
global:
  scrape_interval: 15s        # 全局采集间隔
  evaluation_interval: 15s    # 规则评估间隔
  scrape_timeout: 10s

scrape_configs:
  # Node Exporter - 基础设施监控
  - job_name: node
    file_sd_configs:
      - files: ["/etc/prometheus/targets/nodes/*.json"]
        refresh_interval: 30s
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

  # 应用指标（Spring Boot Actuator）
  - job_name: app
    metrics_path: /actuator/prometheus
    file_sd_configs:
      - files: ["/etc/prometheus/targets/apps/*.json"]

  # Blackbox Exporter - 端点探测
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
      - files: ["/etc/prometheus/targets/probes/*.json"]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

目标节点的服务发现采用文件动态配置：

// /etc/prometheus/targets/nodes/production.json
[
  {
    "targets": ["10.0.1.10:9100", "10.0.1.11:9100", "10.0.1.12:9100"],
    "labels": {
      "env": "production",
      "cluster": "web-cluster",
      "team": "infra"
    }
  }
]

告警规则编写与多级阈值设计

告警规则的核心原则是：减少噪音、分级响应、避免告警疲劳。每条规则必须明确严重等级和响应时限：

# /etc/prometheus/rules/infrastructure.yml
groups:
  - name: infrastructure
    rules:
      # P0 - 立即响应（5分钟内）
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 2m
        labels:
          severity: critical
          team: infra
          response_time: 5m
        annotations:
          summary: "节点 {{ $labels.instance }} 宕机"
          description: "节点已离线超过2分钟"

      # P1 - 30分钟内响应
      - alert: DiskSpaceCritical
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90
        for: 10m
        labels:
          severity: warning
          team: infra
          response_time: 30m
        annotations:
          summary: "{{ $labels.instance }} 磁盘使用率超过90%"
          runbook: "https://wiki.internal/runbook/disk-full"

      # P2 - 当日处理
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
        for: 30m
        labels:
          severity: info
          team: infra
          response_time: 4h
        annotations:
          summary: "{{ $labels.instance }} 内存使用率持续高于85%"

应用层告警以RED方法（Rate-Errors-Duration）为框架：

# /etc/prometheus/rules/application.yml
groups:
  - name: application_red
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "服务 {{ $labels.service }} 5xx错误率超过5%"

      - alert: HighLatencyP99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
          ) > 2.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "服务 {{ $labels.service }} P99延迟超过2秒"

Alertmanager路由与抑制策略

告警路由决定谁在什么时间收到什么级别的通知：

# alertmanager.yml
route:
  group_by: ["alertname", "cluster", "service"]
  group_wait: 30s          # 同组告警等待30s合并
  group_interval: 5m       # 同组新告警间隔5m
  repeat_interval: 4h      # 未恢复告警4h重复通知
  receiver: default
  routes:
    - match:
        severity: critical
      receiver: critical-oncall
      repeat_interval: 30m
    - match:
        severity: warning
      receiver: warning-channel
    - match:
        team: dba
      receiver: dba-team
# 抑制规则：节点宕机时，抑制该节点上的所有应用告警
inhibit_rules:
  - source_match:
      alertname: NodeDown
    target_match:
      severity: warning
    equal: ["instance"]

故障自动响应与Runbook自动化

手动处理重复性故障是SRE团队效率的最大杀手。将常见故障场景的处置流程自动化，是提升MTTR的关键：

# Webhook接收器触发自动修复
from flask import Flask, request
import subprocess

app = Flask(__name__)

@app.route("/webhook/alert", methods=["POST"])
def handle_alert():
    alert = request.json
    alerts = alert.get("alerts", [])

    for a in alerts:
        labels = a.get("labels", {})
        alertname = labels.get("alertname", "")

        if alertname == "DiskSpaceCritical":
            instance = labels.get("instance", "")
            subprocess.run([
                "ssh", instance,
                "find /var/log -name '*.log.*' -mtime +7 -delete && "
                "journalctl --vacuum-time=3d"
            ])
            return "auto-remediated", 200

        elif alertname == "HighMemoryUsage":
            instance = labels.get("instance", "")
            service = labels.get("service", "")
            subprocess.run([
                "ssh", instance,
                f"systemctl restart {service}"
            ])
            return "service-restarted", 200

    return "no-handler", 200

自动化修复必须设边界——只处理已知、低风险的修复动作，复杂故障仍需人工介入。每次自动修复操作都记录到审计日志，并在修复后发送确认通知。监控告警体系的目标不是消除告警，而是让每一条告警都有价值、每一次响应都有成效。