监控告警体系：Prometheus+Alertmanager生产级配置指南

监控告警体系为什么选Prometheus技术栈

网站运维中，监控告警是SRE稳定性工程的基石。系统故障从发生到感知的时间窗口，直接决定业务受损程度。Prometheus作为云原生监控的事实标准，具备多维数据模型、灵活的PromQL查询语言、拉取式采集架构和原生服务发现能力，与Kubernetes生态深度整合。搭配Alertmanager实现告警去重、分组、路由和静默，构成完整的监控告警链路。

Prometheus Server生产级部署

生产环境推荐使用docker-compose部署Prometheus，便于版本管理和配置迁移：

# docker-compose.yml
version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.54.1
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--storage.tsdb.retention.size=50GB"
      - "--query.max-samples=50000000"
      - "--query.timeout=2m"
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    restart: unless-stopped

volumes:
  prometheus_data:

Prometheus核心配置与采集优化

prometheus.yml的配置直接决定采集覆盖面和性能开销：

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s
  external_labels:
    cluster: "prod"
    env: "production"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - "/etc/prometheus/rules/*.yml"

scrape_configs:
  # Node Exporter - 主机指标
  - job_name: "node"
    scrape_interval: 15s
    static_configs:
      - targets:
          - "web-01:9100"
          - "web-02:9100"
          - "db-01:9100"
        labels:
          team: "infra"

  # 应用指标 - 支持Kubernetes服务发现
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

采集频率不是越高越好。15秒间隔适合关键指标（CPU、内存、请求延迟），低优先级指标如磁盘容量可设为60秒。过高的采集频率会显著增加TSDB写入压力和网络开销。

告警规则编写规范

告警规则的核心矛盾：灵敏度和噪声是一对矛盾体。规则太松导致漏报，太紧导致告警疲劳。编写原则是分级分类——P0级故障必须立即触发，P2级问题允许一定观察窗口。

# rules/infra_alerts.yml
groups:
  - name: node_alerts
    rules:
      # P0 - 主机宕机
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
          team: infra
        annotations:
          summary: "主机 {{ $labels.instance }} 不可达"
          description: "已持续1分钟无法采集指标"

      # P1 - CPU使用率
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
          team: infra
        annotations:
          summary: "{{ $labels.instance }} CPU使用率超过85%"
          description: "当前值: {{ $value | printf \"%.1f\" }}%"

      # P1 - 内存压力
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 3m
        labels:
          severity: warning
          team: infra
        annotations:
          summary: "{{ $labels.instance }} 内存使用率超过90%"

      # P2 - 磁盘空间
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes) * 100 > 85
        for: 10m
        labels:
          severity: info
          team: infra
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} 磁盘使用率超过85%"

Alertmanager路由与去重配置

Alertmanager解决的核心问题：同一个故障触发多条规则时，如何避免告警轰炸？通过group_by聚合相同特征的告警，group_wait控制首次等待时间，group_interval控制后续批次间隔：

# alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: "smtp.example.com:587"
  smtp_from: "alertmanager@example.com"
  smtp_auth_username: "alertmanager@example.com"
  smtp_auth_password: "smtp-password"

route:
  group_by: ["alertname", "cluster", "namespace"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "default"
  routes:
    # P0告警直接电话+短信
    - match:
        severity: critical
      receiver: "oncall-phone"
      group_wait: 10s
      repeat_interval: 30m
    # P1告警发企业IM
    - match:
        severity: warning
      receiver: "team-webhook"
      group_wait: 30s
      repeat_interval: 2h
    # P2告警仅邮件
    - match:
        severity: info
      receiver: "email-only"
      repeat_interval: 12h

receivers:
  - name: "default"
    webhook_configs:
      - url: "http://webhook-adapter:8080/alert"
        send_resolved: true

  - name: "oncall-phone"
    webhook_configs:
      - url: "http://phone-gateway:8080/call"

  - name: "team-webhook"
    webhook_configs:
      - url: "https://hooks.feishu.cn/your-webhook-url"

  - name: "email-only"
    email_configs:
      - to: "ops-team@example.com"

# 维护窗口静默
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["instance"]

inhibit_rules的配置很关键——当同台主机触发P0告警时，自动抑制该主机上的P1告警，避免噪声干扰。

Grafana可视化与告警看板

Prometheus数据最终需要通过Grafana呈现。部署Grafana并对接Prometheus数据源后，建议构建三个层级的看板：全局概览看板展示集群整体健康度、活跃告警数、SLI趋势；服务维度看板按服务拆分展示请求量、延迟P99、错误率（RED方法论），每个服务一个Row；主机维度看板完整展示Node Exporter指标，CPU/Memory/Disk/Network四行布局。

# Grafana Provisioning数据源配置
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true
    jsonData:
      timeInterval: "15s"
      httpMethod: POST

告警质量治理与持续优化

告警体系上线后，持续治理比初始配置更重要。每月执行告警复盘：统计各规则触发频次，识别高频低价值告警。阈值不是一次定好的——根据历史数据调整for持续时间和阈值边界。核心指标是告警确认率（ACK Rate），目标值高于80%。低于50%的规则需要重新评估是否应该调松阈值或直接下线。建立告警分级SLA：P0在5分钟内响应，P1在30分钟内处理，P2在下一个工作日跟进。持续迭代，让告警体系真正成为稳定性保障而不是噪声来源。