Prometheus+Alertmanager告警体系搭建：从指标采集到智能降噪的完整方案

小编 • • 网站运维

监控系统是运维基础设施的核心组件。Prometheus通过拉取（Pull）模式采集目标服务的时序指标，Alertmanager负责告警路由、分组、抑制和静默，两者配合构成完整的可观测性告警链路。本文记录一套覆盖指标采集、规则评估、告警路由与通知发送的完整方案构建过程，包含多渠道告警分发和重复告警降噪策略。

Prometheus架构与指标采集模式

Prometheus采用时序数据库存储指标数据，每个指标由名称和一组标签唯一标识。数据模型为metric_name{label1=value1,label2=value2} value timestamp。采集方式有两种：Pull模式由Prometheus主动拉取目标端点的/metrics接口数据；Push模式通过Pushgateway接收短任务指标。

部署Prometheus Server的Docker Compose配置：

version: "3.8"

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data: {}

services:
  prometheus:
    image: prom/prometheus:v2.54.0
    container_name: prometheus
    restart: always
    networks: [monitoring]
    ports: ["9090:9090"]
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--storage.tsdb.retention.size=50GB'
      - '--web.enable-lifecycle'
      - '--alertmanager.url=http://alertmanager:9093'

  node_exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node_exporter
    restart: always
    networks: [monitoring]
    ports: ["9100:9100"]
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: always
    networks: [monitoring]
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--storage.path=/alertmanager'

Prometheus配置文件prometheus.yml定义采集目标和评估规则：

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets:
          - 'node-exporter:9100'
          - '192.168.10.11:9100'
          - '192.168.10.12:9100'

  - job_name: 'nginx_exporter'
    static_configs:
      - targets: ['192.168.10.11:9113', '192.168.10.12:9113']

  - job_name: 'mysql_exporter'
    static_configs:
      - targets: ['192.168.20.10:9104']

  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.yunthe.com
          - https://www.kou5.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

告警规则编写与PromQL表达式实践

告警规则文件rules/alerts.yml：

groups:
  - name: node_alerts
    rules:
      - alert: NodeHighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "CPU使用率过高 {{ $labels.instance }}"
          description: "节点 {{ $labels.instance }} CPU使用率持续5分钟超过85%，当前值: {{ $value | printf \"%.1f\" }}%"

      - alert: NodeHighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 3m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "内存使用率过高 {{ $labels.instance }}"
          description: "节点 {{ $labels.instance }} 内存使用率超过90%"

      - alert: NodeDown
        expr: up{job="node_exporter"} == 0
        for: 1m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "节点离线 {{ $labels.instance }}"

  - name: http_probe_alerts
    rules:
      - alert: SiteUnavailable
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "站点不可访问 {{ $labels.instance }}"
          description: "站点 {{ $labels.instance }} 持续2分钟探测失败"

      - alert: SiteSlowResponse
        expr: probe_duration_seconds > 3
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "站点响应缓慢 {{ $labels.instance }}"
          description: "站点 {{ $labels.instance }} 响应时间超过3秒"

每条规则由expr（PromQL表达式）、for（持续时间）、labels（用于路由的标签）和annotations（告警描述模板）组成。for: 5m表示表达式持续5分钟满足条件才触发告警，避免瞬时抖动引发误报。

Alertmanager路由分组与多渠道告警通知

Alertmanager配置文件alertmanager.yml：

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.yunthe.com:25'
  smtp_from: 'alert@yunthe.com'

route:
  receiver: 'default-alerts'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "critical"
      receiver: 'critical-alerts'
      group_wait: 10s
      repeat_interval: 1h
      continue: false

    - matchers:
        - alertname = "NodeDown"
      receiver: 'phone-alert'
      group_wait: 0s
      repeat_interval: 30m

receivers:
  - name: 'default-alerts'
    email_configs:
      - to: 'ops-team@yunthe.com'
        send_resolved: true

  - name: 'critical-alerts'
    email_configs:
      - to: 'ops-team@yunthe.com'
        send_resolved: true
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=CRITICAL_KEY'
        send_resolved: true

  - name: 'phone-alert'
    webhook_configs:
      - url: 'http://localhost:5001/api/call'
        send_resolved: false

inhibit_rules:
  - source_matchers:
      - alertname = "NodeDown"
    target_matchers:
      - severity = "warning"
    equal: ['instance']

核心参数：group_by按告警名称和标签分组，同类告警合并为一封通知。group_wait: 30s设置首次告警延迟，30秒内产生的同组告警合并发送。group_interval: 5m设置同组新增告警的发送间隔。repeat_interval: 4h设置未恢复告警的重复通知间隔。inhibit_rules定义抑制规则——当NodeDown告警触发时，同一实例上的所有warning级别告警被抑制，避免告警风暴。

告警降噪策略与抖动问题诊断

告警抖动（Flapping）指指标在阈值附近波动导致告警反复触发和恢复。解决方案：调整for持续时间提高容错、使用统计类PromQL函数平滑数据、配置Alertmanager的repeat_interval降低通知频率。

对于CPU使用率抖动，用avg_over_time替代即时值：

# 原始规则（容易抖动）
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85

# 改进：使用7分钟滑动平均
expr: avg_over_time((100 - (rate(node_cpu_seconds_total{mode="idle"}[5m]) * 100))[7m:1m]) > 85

对于计划内维护期间暂停告警，通过API创建动态静默（Silence）：

curl -X POST http://localhost:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "instance", "value": "192.168.10.11", "isRegex": false}
    ],
    "startsAt": "2026-08-03T10:00:00Z",
    "endsAt": "2026-08-03T12:00:00Z",
    "createdBy": "matao",
    "comment": "计划内维护"
  }'

完成以上配置后，监控系统覆盖了主机资源（CPU、内存、磁盘）、中间件（Nginx、MySQL、Redis）和业务可用性（HTTP探测）三个层面的监控。通过Alertmanager的分组、抑制和静默机制，告警噪声得到有效控制，运维团队收到的每条告警都指向需要实际行动的问题。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/prometheusalertmanager-gao-jing-ti-xi-da-jian-cong-zhi-biao/

赞 (0)
小编小编
0
上一篇 21小时前
下一篇 21小时前

相关推荐