监控系统是DevOps实践的基石。一套完整的监控告警体系需要解决三个核心问题：指标采集的全面性、告警规则的准确性、通知路由的及时性。本文记录基于Prometheus + Alertmanager构建生产级监控告警体系的完整流程，覆盖Prometheus指标采集、Alertmanager告警路由、Grafana可视化三大部分。

监控体系架构与组件规划

整体架构采用Prometheus拉取模式（Pull Mode），各业务节点运行node_exporter暴露主机指标，应用通过/metrics端点暴露业务指标。Alertmanager负责告警去重、分组和路由分发。

# 架构组件
# Prometheus Server:  192.168.20.10:9090  (中心采集与存储)
# Alertmanager:       192.168.20.11:9093  (告警路由分发)
# Grafana:           192.168.20.12:3000  (可视化看板)
# Node Exporter:     各业务节点:9100     (主机指标采集)
# Blackbox Exporter: 192.168.20.10:9115  (HTTP/TCP探测)

# Docker Compose部署
cat > /opt/monitoring/docker-compose.yml << 'EOF'
version: '3.8'
services:
  prometheus:
    image: prom/prometheus:v2.54.0
    container_name: prometheus
    ports: ["9090:9090"]
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=90d'
      - '--web.enable-lifecycle'
    restart: always

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    restart: always

  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    ports: ["3000:3000"]
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=Grafana2026
    volumes:
      - grafana_data:/var/lib/grafana
    restart: always

volumes:
  prometheus_data:
  grafana_data:
EOF

cd /opt/monitoring && docker compose up -d

Prometheus采集配置与服务发现

prometheus.yml是采集核心配置，需定义抓取目标、间隔和标签规则。生产环境建议使用文件服务发现（file_sd_configs）实现动态管理。

# /opt/monitoring/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

# 告警规则文件
rule_files:
  - /etc/prometheus/rules/*.yml

# Alertmanager配置
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['192.168.20.11:9093']

# 抓取目标
scrape_configs:
  # Prometheus自身监控
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # 主机指标采集（文件服务发现）
  - job_name: 'node_exporter'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/nodes.yml']
        refresh_interval: 30s
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '(.*):.*'
        replacement: '$1'

  # HTTP探测（Blackbox Exporter）
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
      - files: ['/etc/prometheus/targets/http_probes.yml']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 192.168.20.10:9115

  # 应用业务指标
  - job_name: 'app_metrics'
    metrics_path: /metrics
    file_sd_configs:
      - files: ['/etc/prometheus/targets/apps.yml']
        refresh_interval: 30s

文件服务发现的目标列表：

# /opt/monitoring/prometheus/targets/nodes.yml
- targets:
    - 192.168.20.21:9100
    - 192.168.20.22:9100
    - 192.168.20.23:9100
  labels:
    env: production
    region: cn-east-1

# /opt/monitoring/prometheus/targets/http_probes.yml
- targets:
    - https://www.example.com
    - https://api.example.com/health
  labels:
    env: production

采集启动后通过http://localhost:9090/targets查看各job的采集状态。

告警规则编写与PromQL实战

告警规则使用PromQL表达式定义触发条件。关键原则是：告警必须可操作、有明确的阈值和持续时间，避免频繁误报。规则文件放在/etc/prometheus/rules/目录下。

# /opt/monitoring/prometheus/rules/host_alerts.yml
groups:
  - name: host_alerts
    rules:
      # CPU使用率持续5分钟超过80%
      - alert: HighCPUUsage
        expr: |
          100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "CPU使用率过高: {{ $labels.instance }}"
          description: "实例 {{ $labels.instance }} CPU使用率超过80%，当前值: {{ $value }}%"

      # 内存使用率超过90%
      - alert: HighMemoryUsage
        expr: |
          (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 3m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "内存使用率告警: {{ $labels.instance }}"
          description: "内存使用率超过90%，当前: {{ $value | printf \"%.1f\" }}%"

      # 磁盘空间不足
      - alert: DiskSpaceLow
        expr: |
          (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} 
          / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 > 85
        for: 5m
        labels:
          severity: warning
          team: ops
        annotations:
          summary: "磁盘空间不足: {{ $labels.instance }} {{ $labels.mountpoint }}"
          description: "磁盘使用率超过85%，当前: {{ $value | printf \"%.1f\" }}%"

# /opt/monitoring/prometheus/rules/service_alerts.yml
groups:
  - name: service_alerts
    rules:
      # 服务宕机（实例数低于预期）
      - alert: ServiceDown
        expr: up{job="node_exporter"} == 0
        for: 1m
        labels:
          severity: critical
          team: ops
        annotations:
          summary: "服务不可用: {{ $labels.instance }}"
          description: "目标 {{ $labels.instance }} 已离线超过1分钟"

      # HTTP探测失败
      - alert: HTTPProbeFailed
        expr: probe_success{job="blackbox_http"} == 0
        for: 2m
        labels:
          severity: critical
          team: dev
        annotations:
          summary: "HTTP探测失败: {{ $labels.instance }}"
          description: "URL {{ $labels.instance }} 连续2分钟探测失败"

      # HTTP响应延迟过高
      - alert: HTTPLatencyHigh
        expr: |
          histogram_quantile(0.95, 
          rate(probe_duration_seconds_bucket{job="blackbox_http"}[5m])) > 2
        for: 5m
        labels:
          severity: warning
          team: dev
        annotations:
          summary: "HTTP响应延迟过高: {{ $labels.instance }}"
          description: "P95延迟超过2秒，当前: {{ $value | printf \"%.2f\" }}s"

配置变更后热加载：curl -X POST http://localhost:9090/-/reload。

Alertmanager多渠道路由与通知模板

Alertmanager负责接收Prometheus发送的告警，按规则去重、分组后路由到不同通知渠道。核心配置包括路由树（route tree）、抑制规则（inhibit_rules）和通知模板。

# /opt/monitoring/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m

# 告警模板
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# 路由树
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s        # 首次等待30秒再发送，收集同组告警
  group_interval: 5m     # 同组告警间隔5分钟
  repeat_interval: 4h    # 重复发送间隔4小时
  receiver: 'default-webhook'

  routes:
    # critical级别 -> 立即发送到企业微信+电话告警
    - matchers:
        - severity = "critical"
      receiver: 'critical-wechat-phone'
      group_wait: 0s
      repeat_interval: 1h

    # warning级别 -> 发送到企业微信
    - matchers:
        - severity = "warning"
      receiver: 'warning-wechat'
      group_wait: 30s
      repeat_interval: 4h

    # dev团队告警 -> 发送到钉钉
    - matchers:
        - team = "dev"
      receiver: 'dev-dingtalk'

# 抑制规则：critical触发时屏蔽同目标的warning
inhibit_rules:
  - source_matchers:
      - severity = "critical"
    target_matchers:
      - severity = "warning"
    equal: ['instance', 'cluster']

# 接收器配置
receivers:
  - name: 'default-webhook'
    webhook_configs:
      - url: 'http://192.168.20.12:9094/webhook'
        send_resolved: true

  - name: 'critical-wechat-phone'
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=CRITICAL_KEY'
        send_resolved: true
    # 电话告警通过webhook触发外部呼叫系统

  - name: 'warning-wechat'
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=WARNING_KEY'
        send_resolved: true

  - name: 'dev-dingtalk'
    webhook_configs:
      - url: 'https://oapi.dingtalk.com/robot/send?access_token=DEV_TOKEN'
        send_resolved: true

企业微信通知模板，告警消息格式化输出：

# /opt/monitoring/alertmanager/templates/wechat.tmpl
{{ define "wechat.default.message" }}
{{- if gt (len .Alerts.Firing) 0 -}}
**告警触发** ({{ .Alerts.Firing | len }}条)
{{ range .Alerts.Firing }}
---
> **级别**: {{ .Labels.severity }}
> **告警**: {{ .Labels.alertname }}
> **实例**: {{ .Labels.instance }}
> **描述**: {{ .Annotations.description }}
> **触发时间**: {{ .StartsAt.Format "2006-01-02 15:04:05" }}
{{ end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}

**告警恢复** ({{ .Alerts.Resolved | len }}条)
{{ range .Alerts.Resolved }}
---
> **告警**: {{ .Labels.alertname }}
> **实例**: {{ .Labels.instance }}
> **恢复时间**: {{ .EndsAt.Format "2006-01-02 15:04:05" }}
{{ end }}
{{- end }}
{{ end }}

告警治理与噪音抑制策略

监控体系上线后最大的挑战不是覆盖不足，而是告警风暴。几个有效的噪音抑制策略：

1. 合理设置for持续时间。 瞬时尖峰往往不代表真实问题。CPU偶尔飙到90%然后迅速回落不应当告警。设for: 5m确保只有持续异常才触发。

2. 分级告警策略。 将告警分为critical/warning/info三级，critical走即时通知渠道（电话/短信），warning走异步渠道（企业微信/钉钉），info仅记录不通知。

3. 告警抑制规则。 当上层故障触发时，自动屏蔽下层的衍生告警。例如数据库宕机会导致大量应用连接超时告警，应通过inhibit_rules屏蔽：

inhibit_rules:
  # 数据库宕机时屏蔽应用层连接告警
  - source_matchers:
      - alertname = "DatabaseDown"
    target_matchers:
      - alertname = "AppConnectionError"
    equal: ['cluster']

  # 节点宕机时屏蔽该节点上的所有服务告警
  - source_matchers:
      - alertname = "NodeDown"
    target_matchers:
      - alertname =~ "Service.*"
    equal: ['instance']

4. 维护期静默（Silence）。 计划性维护期间使用Alertmanager API创建静默规则：

curl -X POST http://localhost:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "instance", "value": "192.168.20.21", "isRegex": false}
    ],
    "startsAt": "2026-08-05T10:00:00Z",
    "endsAt": "2026-08-05T12:00:00Z",
    "createdBy": "ops-team",
    "comment": "计划维护：扩容磁盘"
  }'

这套Prometheus + Alertmanager监控告警体系覆盖主机指标、服务健康、HTTP探测三大维度，通过文件服务发现实现节点动态管理，多渠道路由确保不同级别告警精准触达对应团队。日常运维中需定期审查告警规则有效性——统计每个告警的触发频率和恢复时间，对高频低效告警调整阈值或降级处理，保持告警体系的信噪比。