Prometheus监控告警体系搭建与Alertmanager告警规则配置实战

小编 • • 网站运维

Prometheus是云原生监控体系的事实标准，配合Alertmanager构成完整的监控告警链路。DevOps实践中，监控覆盖指标采集、数据存储、告警触发、通知分发四个环节。一个可用的监控告警体系需要在故障发生后的分钟级别发出告警，同时控制误报率在合理范围内。本文覆盖Prometheus部署、告警规则编写、Alertmanager路由配置和告警抑制策略。

Prometheus架构与核心组件部署

Prometheus生态由Prometheus Server（时序数据库+采集引擎）、Alertmanager（告警路由分发）、Exporter（指标暴露）和Grafana（可视化展示）组成。数据模型为时序数据，每个指标由metric name和label set唯一标识。

# Docker Compose部署Prometheus + Alertmanager + Grafana
# docker-compose.yml
version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.54.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules/:/etc/prometheus/rules/
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
      - alertmanager_data:/alertmanager

  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123

volumes:
  prometheus_data:
  alertmanager_data:
  grafana_data:

Prometheus配置文件与采集目标定义

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  # Prometheus自身指标
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  # Node Exporter - 主机指标
  - job_name: "node"
    static_configs:
      - targets:
          - "192.168.1.10:9100"
          - "192.168.1.11:9100"
          - "192.168.1.12:9100"

  # 应用自定义指标
  - job_name: "app"
    metrics_path: /metrics
    static_configs:
      - targets: ["app-server:8080"]

  # Kubernetes服务发现
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__

PromQL告警规则编写：CPU、内存、磁盘、服务可用性

告警规则文件定义在rules目录下。每条规则包含名称、条件表达式、持续时间（for）、标签和注释：

# rules/host_alerts.yml
groups:
  - name: host_alerts
    rules:
      # CPU使用率持续5分钟超过80%
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU使用率过高 {{ $labels.instance }}"
          description: "实例 {{ $labels.instance }} CPU使用率已达 {{ $value }}%"

      # 内存使用率超过90%
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "内存使用率过高 {{ $labels.instance }}"
          description: "实例 {{ $labels.instance }} 内存使用率 {{ $value }}%"

      # 磁盘空间不足
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘空间不足 {{ $labels.instance }}"
          description: "挂载点 {{ $labels.mountpoint }} 已使用 {{ $value }}%"

      # 节点离线
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "节点离线 {{ $labels.instance }}"
          description: "节点 {{ $labels.instance }} 已离线超过1分钟"

      # 服务可用性下降
      - alert: ServiceUnavailable
        expr: up{job="app"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "服务不可用 {{ $labels.instance }}"
          description: "应用实例 {{ $labels.instance }} 不可达"

Alertmanager告警路由与通知渠道配置

Alertmanager支持按标签路由告警到不同通知渠道，并支持告警分组、抑制和静默：

# alertmanager.yml
route:
  receiver: "default"
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # critical级别走电话+邮件
    - matchers:
        - severity = critical
      receiver: "critical-team"
      group_wait: 10s
      repeat_interval: 1h
    # warning级别走钉钉
    - matchers:
        - severity = warning
      receiver: "dingtalk"
      repeat_interval: 2h

receivers:
  - name: "default"
    webhook_configs:
      - url: "http://dingtalk-webhook:8060/dingtalk/ops/send"

  - name: "critical-team"
    webhook_configs:
      - url: "http://dingtalk-webhook:8060/dingtalk/critical/send"
    email_configs:
      - to: "ops-team@company.com"
        from: "alertmanager@company.com"
        smarthost: "smtp.company.com:587"
        auth_username: "alertmanager@company.com"
        auth_password: "password"
        require_tls: true

  - name: "dingtalk"
    webhook_configs:
      - url: "http://dingtalk-webhook:8060/dingtalk/ops/send"

# 告警抑制：当NodeDown触发时，抑制该节点的其他告警
inhibit_rules:
  - source_matchers:
      - alertname = NodeDown
    target_matchers:
      - severity = warning
    equal: ["instance"]

告警抑制与静默策略

告警抑制（inhibition）用于在高层故障发生时屏蔽衍生告警。例如节点离线时，该节点上的CPU、内存、磁盘告警无意义，应自动抑制。配置中通过inhibit_rules定义source和target的匹配关系，equal字段指定标签匹配条件。

告警静默（silence）用于维护窗口期间临时屏蔽告警。通过Alertmanager API或Web UI创建：

# 创建静默规则：屏蔽instance=192.168.1.10的所有告警，持续2小时
curl -X POST http://localhost:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "instance", "value": "192.168.1.10", "isRegex": false}
    ],
    "startsAt": "2026-08-20T10:00:00Z",
    "endsAt": "2026-08-20T12:00:00Z",
    "createdBy": "ops",
    "comment": "维护窗口，临时静默"
  }'

Prometheus监控告警体系调优建议

告警规则中for字段控制从条件触发到告警发出的等待时间。设置过短会导致瞬时波动产生噪声，过长会延误故障发现。CPU、内存类指标建议3-5分钟，服务可用性类建议1-2分钟。

scrape_interval默认15秒适合大多数场景。高频指标（如HTTP请求QPS）可单独配置更短的采集间隔，但需注意Prometheus存储压力。30天retention配合合理的采集间隔，单节点可承载约200万活跃时序。

repeat_interval控制同一告警的重复通知间隔，避免长时间未处理的告警被遗忘。critical级别建议1小时重复，warning级别建议2-4小时。group_by配置将相关告警合并为一组通知，减少告警风暴。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/prometheus-jian-kong-gao-jing-ti-xi-da-jian-yu-alertmanager/

赞 (0)
小编小编
0
上一篇 2026年8月20日
下一篇 2026年8月20日

相关推荐

Prometheus监控告警体系搭建与Alertmanager通知路由配置实战

小编 • • 网站运维

监控系统是SRE稳定性工程的基础设施。一套完整的监控告警体系需要解决三个问题：采集什么指标、如何存储和查询指标、告警如何触达并减少噪音。Prometheus通过Pull模式采集时序数据，配合Alertmanager实现告警路由、抑制和去重，已成为云原生监控事实标准。本文从零搭建一套Prometheus + Node Exporter + Alertmanager监控告警体系，覆盖部署、指标采集、告警规则编写和通知路由配置全流程。

Prometheus架构与数据模型

Prometheus采用时序数据库（TSDB）存储指标数据，每个指标由指标名和标签集合唯一标识。数据模型为：metric_name{label1=”value1″, label2=”value2″} timestamp value。

核心组件分工：Prometheus Server负责指标采集和存储；Exporter提供被监控目标的指标端点；Alertmanager负责告警路由和通知推送；Grafana负责数据可视化。采集模式为Pull，Prometheus主动访问目标端点拉取数据，这与传统的Push模式（如StatsD）形成对比。

指标类型分为四种：Counter（单调递增计数器）、Gauge（可增可减的瞬时值）、Histogram（分桶统计分布）、Summary（客户端计算分位数）。选择正确的指标类型直接影响告警规则的准确性。

部署Prometheus Server与Node Exporter

以Docker Compose方式部署，便于管理和升级。先创建配置文件目录结构：

mkdir -p /opt/prometheus/{config,rules,data}
mkdir -p /opt/alertmanager/config

编写docker-compose.yml：

version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.54.0
    container_name: prometheus
    restart: always
    ports:
      - "9090:9090"
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - ./data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"

  node-exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node-exporter
    restart: always
    ports:
      - "9100:9100"
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: always
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/config/alertmanager.yml:/etc/alertmanager/alertmanager.yml

编写prometheus.yml主配置文件：

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]
        labels:
          env: "production"

编写PromQL告警规则

告警规则文件放在rules目录下。以下创建node_alerts.yml，覆盖CPU、内存、磁盘和节点存活四类告警：

groups:
  - name: node_alerts
    rules:
      # CPU使用率超过80%持续5分钟
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU使用率过高 {{ $labels.instance }}"
          description: "CPU使用率: {{ $value }}%, 阈值: 80%"

      # 可用内存低于10%
      - alert: LowMemory
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "内存不足 {{ $labels.instance }}"
          description: "可用内存: {{ $value }}%, 阈值: 10%"

      # 磁盘使用率超过85%
      - alert: DiskSpaceHigh
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘空间不足 {{ $labels.instance }}"
          description: "挂载点 {{ $labels.mountpoint }} 使用率: {{ $value }}%"

      # 节点离线
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "节点离线 {{ $labels.instance }}"
          description: "已持续1分钟无法采集指标"

规则中的for字段指定告警的Pending Duration，即条件满足后需要持续多久才触发告警。这能过滤掉瞬时抖动。severity标签用于后续Alertmanager路由分发。

Alertmanager路由与通知配置

Alertmanager的alertmanager.yml配置告警路由和通知渠道：

global:
  resolve_timeout: 5m

route:
  receiver: "default"
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "critical"
      receiver: "oncall-pagerduty"
    - matchers:
        - severity = "warning"
      receiver: "oncall-slack"

receivers:
  - name: "default"
    email_configs:
      - to: "ops@yunthe.com"
        from: "alertmanager@yunthe.com"
        smarthost: "smtp.yunthe.com:587"
        auth_username: "alertmanager@yunthe.com"
        auth_password: "password"
        require_tls: true

  - name: "oncall-slack"
    slack_configs:
      - api_url: "https://hooks.slack.com/services/xxx"
        channel: "#alerts-warning"
        send_resolved: true
        title: "{{ .CommonLabels.alertname }}"
        text: "{{ .CommonAnnotations.description }}"

  - name: "oncall-pagerduty"
    webhook_configs:
      - url: "https://events.pagerduty.com/integration/xxx/enqueue"
        send_resolved: true

inhibit_rules:
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"
    equal: ["instance"]

关键参数说明：group_by控制相同告警的聚合粒度；group_wait是告警发出前的等待时间；group_interval是同组新告警的通知间隔；repeat_interval是重复通知间隔。inhibit_rules配置告警抑制，当同一实例存在critical级别告警时，抑制其warning级别告警，减少噪音。

告警收敛与降噪策略

告警噪音是监控系统的头号敌人。生产环境通常面临告警风暴问题——一次网络抖动触发数十条告警。Alertmanager提供三层收敛机制：分组（Grouping）将相关告警合并为一条通知；抑制（Inhibition）在高级别告警存在时屏蔽低级别告警；静默（Silence）临时屏蔽特定标签的告警。

静默操作可通过API动态设置：

curl -X POST http://localhost:9093/api/v2/silences   -H "Content-Type: application/json"   -d '{
    "matchers": [
      {"name": "instance", "value": "10.0.0.5:9100", "isRegex": false}
    ],
    "startsAt": "2026-08-19T10:00:00Z",
    "endsAt": "2026-08-19T12:00:00Z",
    "createdBy": "admin",
    "comment": "维护窗口静默"
  }'

合理设置repeat_interval避免告警反复打扰。建议critical级别设为1h，warning级别设为4h。过短的repeat_interval导致告警疲劳，过长则可能错过持续恶化的故障。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/prometheus-jian-kong-gao-jing-ti-xi-da-jian-yu-alertmanager/

赞 (0)
小编小编
0
上一篇 2026年8月19日
下一篇 2026年8月19日

相关推荐