Prometheus自定义告警规则引擎与多维度指标聚合配置

告警规则引擎的架构设计

Prometheus的告警体系由规则评估引擎和Alertmanager两部分组成。规则引擎负责周期性计算告警条件，生成告警实例；Alertmanager负责告警的路由、分组、抑制和通知。规则引擎的工作机制：Prometheus按evaluation_interval（默认15秒）周期性执行每条规则。每条规则是一个PromQL表达式，评估结果为即时向量。如果向量非空则产生告警实例，向量为空则告警恢复。

groups:
  - name: service_sla_alerts
    interval: 30s
    rules:
      - alert: ServiceHighLatency
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="api-server"}[5m])) by (le, service)) > 2
        for: 5m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "{{ $labels.service }} P99延迟超过2秒"

      - alert: ServiceErrorRateSpike
        expr: |
          sum(rate(http_requests_total{job="api-server", status=~"5.."}[5m])) / sum(rate(http_requests_total{job="api-server"}[5m])) > 0.05
        for: 3m
        labels:
          severity: critical

多维度指标的聚合策略

Prometheus的核心优势是多维数据模型——每个时间序列由指标名和一组标签唯一标识。告警规则设计的关键在于正确使用聚合操作符。按服务聚合：关注服务整体健康度，屏蔽单实例抖动。适合P99延迟、整体错误率等SLI指标。按实例聚合：关注个体故障，快速定位问题节点。适合CPU使用率、磁盘空间等资源类指标。跨维度关联：将不同维度的指标组合计算复合告警，避免低流量时错误率波动导致的误报。

- alert: HighErrorRateWithTraffic
  expr: |
    (sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
    / sum(rate(http_requests_total[5m])) by (service)) > 0.03
    and sum(rate(http_requests_total[5m])) by (service) > 100
  for: 5m
  labels:
    severity: critical

告警分级与静默策略

生产环境的告警需要分级处理。P0-Critical服务完全不可用，5分钟内响应；P1-Warning指标异常但服务仍可用，30分钟内响应；P2-Info潜在风险，日常巡检处理。Alertmanager的路由配置实现分级通知：

route:
  receiver: default-slack
  group_by: [service, severity]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: oncall-phone
      group_wait: 10s
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: team-slack
      repeat_interval: 2h

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [service]

Recording Rule优化查询性能

复杂聚合告警规则的PromQL查询可能非常耗时。Recording Rule将中间计算结果预写入新的时间序列，告警规则直接查询预计算结果，大幅降低评估延迟。

groups:
  - name: api_server_metrics
    interval: 30s
    rules:
      - record: api:http_request:p99_latency
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
      - record: api:http_request:error_rate_5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service)

- alert: ServiceHighLatency
  expr: api:http_request:p99_latency > 2
  for: 5m

Recording Rule的命名建议采用level:metric:operations的层级格式。每条Recording Rule都会产生独立的时间序列，建议仅对被多条规则引用或计算量大的表达式创建Recording Rule。

告警规则测试与灰度发布

上线新告警规则前必须经过充分测试。

# 验证规则文件语法
promtool check rules alert_rules.yml

# 使用单元测试框架验证
rule_files:
  - alert_rules.yml
tests:
  - interval: 1m
    input_series:
      - series: http_request_duration_seconds_bucket{service="api",le="2.5"}
        values: "10 20 30 40 50"
    alert_rule_test:
      - eval_time: 5m
        alertname: ServiceHighLatency

灰度上线新规则的做法是先将severity设为info级别运行一周，观察触发频率和准确率，确认无误后再调整为正式级别。这种渐进式发布策略能有效避免告警风暴。