Prometheus监控告警体系搭建与Alertmanager通知路由配置实战

监控系统是SRE稳定性工程的基础设施。一套完整的监控告警体系需要解决三个问题：采集什么指标、如何存储和查询指标、告警如何触达并减少噪音。Prometheus通过Pull模式采集时序数据，配合Alertmanager实现告警路由、抑制和去重，已成为云原生监控事实标准。本文从零搭建一套Prometheus + Node Exporter + Alertmanager监控告警体系，覆盖部署、指标采集、告警规则编写和通知路由配置全流程。

Prometheus架构与数据模型

Prometheus采用时序数据库（TSDB）存储指标数据，每个指标由指标名和标签集合唯一标识。数据模型为：metric_name{label1=”value1″, label2=”value2″} timestamp value。

核心组件分工：Prometheus Server负责指标采集和存储；Exporter提供被监控目标的指标端点；Alertmanager负责告警路由和通知推送；Grafana负责数据可视化。采集模式为Pull，Prometheus主动访问目标端点拉取数据，这与传统的Push模式（如StatsD）形成对比。

指标类型分为四种：Counter（单调递增计数器）、Gauge（可增可减的瞬时值）、Histogram（分桶统计分布）、Summary（客户端计算分位数）。选择正确的指标类型直接影响告警规则的准确性。

部署Prometheus Server与Node Exporter

以Docker Compose方式部署，便于管理和升级。先创建配置文件目录结构：

mkdir -p /opt/prometheus/{config,rules,data}
mkdir -p /opt/alertmanager/config

编写docker-compose.yml：

version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.54.0
    container_name: prometheus
    restart: always
    ports:
      - "9090:9090"
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - ./data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"

  node-exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node-exporter
    restart: always
    ports:
      - "9100:9100"
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: always
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml

编写prometheus.yml主配置文件：

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]
        labels:
          env: "production"

编写PromQL告警规则

告警规则文件放在rules目录下。以下创建node_alerts.yml，覆盖CPU、内存、磁盘和节点存活四类告警：

groups:
  - name: node_alerts
    rules:
      # CPU使用率超过80%持续5分钟
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU使用率过高 {{ $labels.instance }}"
          description: "CPU使用率: {{ $value }}%, 阈值: 80%"

      # 可用内存低于10%
      - alert: LowMemory
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "内存不足 {{ $labels.instance }}"
          description: "可用内存: {{ $value }}%, 阈值: 10%"

      # 磁盘使用率超过85%
      - alert: DiskSpaceHigh
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘空间不足 {{ $labels.instance }}"
          description: "挂载点 {{ $labels.mountpoint }} 使用率: {{ $value }}%"

      # 节点离线
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "节点离线 {{ $labels.instance }}"
          description: "已持续1分钟无法采集指标"

规则中的for字段指定告警的Pending Duration，即条件满足后需要持续多久才触发告警。这能过滤掉瞬时抖动。severity标签用于后续Alertmanager路由分发。

Alertmanager路由与通知配置

Alertmanager的alertmanager.yml配置告警路由和通知渠道：

global:
  resolve_timeout: 5m

route:
  receiver: "default"
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "critical"
      receiver: "oncall-pagerduty"
    - matchers:
        - severity = "warning"
      receiver: "oncall-slack"

receivers:
  - name: "default"
    email_configs:
      - to: "ops@yunthe.com"
        from: "alertmanager@yunthe.com"
        smarthost: "smtp.yunthe.com:587"
        auth_username: "alertmanager@yunthe.com"
        auth_password: "password"
        require_tls: true

  - name: "oncall-slack"
    slack_configs:
      - api_url: "https://hooks.slack.com/services/xxx"
        channel: "#alerts-warning"
        send_resolved: true
        title: "{{ .CommonLabels.alertname }}"
        text: "{{ .CommonAnnotations.description }}"

  - name: "oncall-pagerduty"
    webhook_configs:
      - url: "https://events.pagerduty.com/integration/xxx/enqueue"
        send_resolved: true

inhibit_rules:
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"
    equal: ["instance"]

关键参数说明：group_by控制相同告警的聚合粒度；group_wait是告警发出前的等待时间；group_interval是同组新告警的通知间隔；repeat_interval是重复通知间隔。inhibit_rules配置告警抑制，当同一实例存在critical级别告警时，抑制其warning级别告警，减少噪音。

告警收敛与降噪策略

告警噪音是监控系统的头号敌人。生产环境通常面临告警风暴问题——一次网络抖动触发数十条告警。Alertmanager提供三层收敛机制：分组（Grouping）将相关告警合并为一条通知；抑制（Inhibition）在高级别告警存在时屏蔽低级别告警；静默（Silence）临时屏蔽特定标签的告警。

静默操作可通过API动态设置：

curl -X POST http://localhost:9093/api/v2/silences   -H "Content-Type: application/json"   -d '{
    "matchers": [
      {"name": "instance", "value": "10.0.0.5:9100", "isRegex": false}
    ],
    "startsAt": "2026-08-19T10:00:00Z",
    "endsAt": "2026-08-19T12:00:00Z",
    "createdBy": "admin",
    "comment": "维护窗口静默"
  }'

合理设置repeat_interval避免告警反复打扰。建议critical级别设为1h，warning级别设为4h。过短的repeat_interval导致告警疲劳，过长则可能错过持续恶化的故障。