Prometheus + Grafana监控告警体系搭建：指标采集与告警规则配置

DevOps实践中的监控告警体系架构

监控告警体系是SRE稳定性工程的基础设施。Prometheus负责时序数据采集与存储，Grafana负责数据可视化，Alertmanager负责告警路由与通知。三者配合形成完整的监控闭环：数据采集、可视化展示、异常告警。Docker自动化部署场景下，整个体系可在数分钟内搭建完成。

Prometheus部署与配置

使用Docker Compose部署Prometheus、Grafana和Alertmanager：

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports: ['9090:9090']
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    ports: ['9093:9093']
    volumes:
      - ./alertmanager/config.yml:/etc/alertmanager/config.yml
    command:
      - '--config.file=/etc/alertmanager/config.yml'

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports: ['3000:3000']
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your_password

volumes:
  prometheus_data:
  grafana_data:

Prometheus主配置文件prometheus.yml：

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - alert_rules.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100', 'web-01:9100', 'web-02:9100']

  - job_name: 'app'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['app-01:8080', 'app-02:8080']

  - job_name: 'mysql'
    static_configs:
      - targets: ['mysqld-exporter:9104']

  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://www.example.com
          - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

核心告警规则配置

告警规则文件alert_rules.yml定义触发条件：

groups:
  - name: host_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU使用率过高 {{ $labels.instance }}"
          description: "CPU使用率: {{ $value }}%"

      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "内存不足 {{ $labels.instance }}"
          description: "内存使用率: {{ $value }}%"

      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘空间不足 {{ $labels.instance }}"
          description: "挂载点 {{ $labels.mountpoint }} 使用率: {{ $value }}%"

  - name: app_alerts
    rules:
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "服务不可用 {{ $labels.instance }}"
          description: "Job: {{ $labels.job }}"

      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100 > 5
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "HTTP 5xx错误率过高"
          description: "错误率: {{ $value }}%"

for字段表示条件持续多久后触发告警，避免瞬时波动导致误报。severity标签用于告警分级，Alertmanager根据该标签路由到不同通知渠道。

Alertmanager告警路由与通知配置

route:
  receiver: 'default'
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - matchers:
        - severity = critical
      receiver: 'pagerduty'
    - matchers:
        - severity = warning
      receiver: 'slack'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops@example.com'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'
        send_resolved: true

  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: 'your-routing-key'
        send_resolved: true

group_by控制告警合并维度，相同alertname和severity的告警合并为一条通知。group_wait是首次告警等待时间，group_interval是同一组告警的发送间隔，repeat_interval是重复告警的发送间隔。send_resolved: true表示告警恢复时也发送通知。

Grafana数据源与Dashboard配置

Grafana启动后，添加Prometheus数据源，URL填写http://prometheus:9090。常用PromQL查询语句用于Dashboard面板：

# CPU使用率
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

# 磁盘使用率
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100

# 网络流量
rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m])

# HTTP请求QPS
sum(rate(http_requests_total[5m])) by (status)

# P99响应时间
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

建议为每类服务创建独立Dashboard，设置变量实现多实例切换。告警阈值需根据实际业务负载调优，避免告警风暴。CI/CD流水线中可将Prometheus配置和Grafana Dashboard纳入版本管理，实现监控配置的自动化部署。