Prometheus告警规则设计与多渠道通知实战

Prometheus告警体系架构解析

网站运维的核心诉求是在故障影响用户之前发现问题。Prometheus Alertmanager配合Prometheus Server构成完整的告警链路：Prometheus负责规则评估和告警触发，Alertmanager负责告警去重、分组、路由和通知。两条告警规则如果同时触发同一组标签，Alertmanager会自动合并为一条通知，避免告警风暴淹没运维人员。

告警规则编写与分级标准

告警规则的质量直接决定监控体系的有效性。规则太少会漏掉关键故障，规则太多则产生告警疲劳。以下是一套经过生产验证的告警分级体系：

# 告警规则 - 基础设施层
groups:
- name: infrastructure.alerts
  rules:
  # P0 - 立即响应（5分钟内）
  - alert: NodeDown
    expr: up{job="node_exporter"} == 0
    for: 2m
    labels:
      severity: p0
      team: infra
    annotations:
      summary: "节点 {{ $labels.instance }} 宕机"
      runbook: "https://wiki/runbooks/node-down"

  - alert: DiskSpaceCritical
    expr: |
      (node_filesystem_avail_bytes{mountpoint!~"/snap.*"}
       / node_filesystem_size_bytes) < 0.05
    for: 5m
    labels:
      severity: p0
    annotations:
      summary: "磁盘剩余不足5%: {{ $labels.instance }} {{ $labels.mountpoint }}"

  # P1 - 30分钟内响应
  - alert: HighMemoryUsage
    expr: |
      (1 - node_memory_MemAvailable_bytes
       / node_memory_MemTotal_bytes) > 0.90
    for: 10m
    labels:
      severity: p1
    annotations:
      summary: "内存使用率超90%: {{ $labels.instance }}"

  # P2 - 工作时间内处理
  - alert: DiskSpaceWarning
    expr: |
      (node_filesystem_avail_bytes{mountpoint!~"/snap.*"}
       / node_filesystem_size_bytes) < 0.15
    for: 30m
    labels:
      severity: p2
    annotations:
      summary: "磁盘剩余不足15%: {{ $labels.instance }} {{ $labels.mountpoint }}"

# 告警规则 - 应用层
- name: application.alerts
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      / sum(rate(http_requests_total[5m])) > 0.05
    for: 3m
    labels:
      severity: p0
    annotations:
      summary: "5xx错误率超过5%，当前值: {{ $value | humanizePercentage }}"

  - alert: HighLatencyP99
    expr: |
      histogram_quantile(0.99,
        sum(rate(http_request_duration_seconds_bucket[5m]))
        by (le, service)) > 2.0
    for: 5m
    labels:
      severity: p1
    annotations:
      summary: "P99延迟超过2秒: {{ $labels.service }}"

Alertmanager路由与分组配置

Alertmanager的路由配置决定告警通知的去向和方式。合理的分组策略可以大幅减少告警噪音：

# alertmanager.yml
route:
  group_by: ['alertname', 'cluster', 'service']  # 按告警名+集群+服务分组
  group_wait: 30s        # 同组首条告警等待30s，合并后续告警
  group_interval: 5m     # 同组已发送通知后，5分钟内不重复
  repeat_interval: 4h    # 4小时后重复通知（未恢复的情况下）
  receiver: 'default'
  routes:
  # P0走电话+钉钉
  - match:
      severity: p0
    receiver: 'p0-critical'
    group_wait: 10s
    repeat_interval: 30m
  # P1走钉钉+邮件
  - match:
      severity: p1
    receiver: 'p1-warning'
    repeat_interval: 2h
  # P2仅邮件
  - match:
      severity: p2
    receiver: 'p2-info'
    repeat_interval: 8h

receivers:
- name: 'default'
  webhook_configs:
  - url: 'http://alertmanager-webhook:8080/alert'

- name: 'p0-critical'
  # 钉钉机器人
  webhook_configs:
  - url: 'http://alertmanager-webhook:8080/dingtalk/p0'
  # 电话通知
  webhook_configs:
  - url: 'http://alertmanager-webhook:8080/call/p0'

- name: 'p1-warning'
  webhook_configs:
  - url: 'http://alertmanager-webhook:8080/dingtalk/p1'
  email_configs:
  - to: 'ops-team@company.com'
    from: 'alertmanager@company.com'
    smarthost: 'smtp.company.com:587'

- name: 'p2-info'
  email_configs:
  - to: 'ops-team@company.com'

# 告警抑制：节点宕机时抑制该节点上的其他告警
inhibit_rules:
- source_match:
    severity: p0
    alertname: NodeDown
  target_match:
    severity: p1
  equal: ['instance']

钉钉告警通知Webhook实现

import json, hmac, hashlib, base64, time, urllib.request
from email.mime.text import MIMEText
import smtplib

class DingTalkNotifier:
    def __init__(self, webhook_url, secret):
        self.webhook_url = webhook_url
        self.secret = secret

    def _sign(self):
        ts = str(round(time.time() * 1000))
        string_to_sign = f"{ts}\n{self.secret}"
        hmac_code = hmac.new(
            self.secret.encode(), string_to_sign.encode(),
            hashlib.sha256
        ).digest()
        sign = urllib.parse.quote_plus(base64.b64encode(hmac_code).decode())
        return ts, sign

    def send(self, alerts):
        ts, sign = self._sign()
        url = f"{self.webhook_url}×tamp={ts}&sign={sign}"

        # 构建Markdown格式告警消息
        msg_parts = ["### 告警通知\n"]
        for alert in alerts:
            status = "🔥 FIRING" if alert['status'] == 'firing' else "✅ RESOLVED"
            msg_parts.append(
                f"**{status}** {alert['labels'].get('alertname', 'Unknown')}\n"
                f"> 实例: {alert['labels'].get('instance', 'N/A')}\n"
                f"> 摘要: {alert['annotations'].get('summary', 'N/A')}\n"
            )

        payload = {
            "msgtype": "markdown",
            "markdown": {
                "title": "告警通知",
                "text": "\n".join(msg_parts)
            }
        }

        req = urllib.request.Request(
            url,
            data=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json"}
        )
        urllib.request.urlopen(req, timeout=10)

告警质量治理与优化

告警体系上线后需要持续治理，核心指标：

# 告警健康度评估维度
1. 告警信噪比 = 有效告警数 / 总告警数（目标 >0.7）
2. 告警响应率 = 已处理告警 / 总告警（目标 >0.95）
3. 平均响应时间 = 告警触发到确认的耗时（P0 <5min）
4. 告警抑制率 = 被抑制的告警 / 原始告警（合理范围 30-60%）

# 优化策略
- 合并同类告警：同一服务的多个实例告警合并为一条
- 添加for子句：所有规则至少for 2-5分钟，过滤瞬时抖动
- 配置告警抑制：上级故障抑制下级告警（节点宕机抑制进程告警）
- 定期review：每月回顾告警统计，删除持续触发的无效规则
- 对齐SLO：告警阈值应基于SLO计算，而非凭经验设定

一套设计良好的告警体系，P0告警每月不超过10条，每条都对应真实故障；运维人员日常收到的P2告警可以批量处理，不会被频繁打断。告警不是越多越安全，精准才是核心指标。