监控告警体系搭建：Prometheus+Grafana+Alertmanager实战部署

监控告警是SRE稳定性工程的基础设施。Prometheus负责指标采集与存储，Grafana提供可视化面板，Alertmanager处理告警路由与通知。三者的组合构成了云原生监控的标准方案。

监控架构设计与组件规划

典型的Prometheus监控架构包含以下层级：Exporter采集目标指标，Prometheus Server拉取并存储时序数据，Alertmanager根据规则评估触发告警，Grafana查询数据渲染面板。对于大规模集群，引入Thanos或VictoriaMetrics解决长期存储和联邦查询问题。

组件版本要求：Prometheus 2.45+、Grafana 10+、Alertmanager 0.26+。部署环境以CentOS 7/Ubuntu 22.04为例，采用二进制方式安装。

Prometheus Server安装与配置

# 下载安装
cd /usr/local/src
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xzf prometheus-2.45.0.linux-amd64.tar.gz
mv prometheus-2.45.0.linux-amd64 /usr/local/prometheus

# 创建数据目录和系统用户
mkdir -p /data/prometheus
useradd -r -s /sbin/nologin prometheus
chown -R prometheus:prometheus /data/prometheus /usr/local/prometheus

核心配置文件prometheus.yml：

global:
  scrape_interval: 15s
  evaluation_interval: 15s

# 告警规则文件
rule_files:
  - "rules/*.yml"

# Alertmanager配置
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

# 采集目标
scrape_configs:
  # Prometheus自身指标
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter - 主机指标
  - job_name: 'node'
    static_configs:
      - targets: ['10.0.0.11:9100', '10.0.0.12:9100', '10.0.0.13:9100']

  # MySQL Exporter
  - job_name: 'mysql'
    static_configs:
      - targets: ['10.0.0.20:9104']

  # 应用自定义指标
  - job_name: 'app'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['10.0.0.30:8080', '10.0.0.31:8080']

创建systemd服务单元：

cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Server
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/prometheus/prometheus \
  --config.file=/usr/local/prometheus/prometheus.yml \
  --storage.tsdb.path=/data/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.enable-lifecycle
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus

告警规则编写与Alertmanager部署

创建告警规则文件：

mkdir -p /usr/local/prometheus/rules

cat > /usr/local/prometheus/rules/host.yml << 'EOF'
groups:
  - name: host_alerts
    rules:
      # CPU使用率超过80%持续5分钟
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU使用率过高 {{ $labels.instance }}"
          description: "CPU使用率: {{ $value }}%, 持续超过5分钟"

      # 内存使用率超过90%
      - alert: HighMemoryUsage
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "内存不足 {{ $labels.instance }}"
          description: "内存使用率: {{ $value }}%"

      # 磁盘空间不足
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘空间不足 {{ $labels.instance }} {{ $labels.mountpoint }}"
          description: "磁盘使用率: {{ $value }}%"

      # 节点宕机
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "节点宕机 {{ $labels.instance }}"
          description: "节点已失联超过1分钟"
EOF

安装并配置Alertmanager：

cd /usr/local/src
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xzf alertmanager-0.26.0.linux-amd64.tar.gz
mv alertmanager-0.26.0.linux-amd64 /usr/local/alertmanager

cat > /usr/local/alertmanager/alertmanager.yml << 'EOF'
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'severity']
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical-webhook'
      group_wait: 0s
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: 'dingtalk'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://localhost:8060/dingtalk/default/send'

  - name: 'critical-webhook'
    webhook_configs:
      - url: 'http://localhost:8060/dingtalk/critical/send'

  - name: 'dingtalk'
    webhook_configs:
      - url: 'http://localhost:8060/dingtalk/warning/send'
        send_resolved: true
EOF

Grafana数据源配置与面板搭建

# 安装Grafana
yum install -y https://dl.grafana.com/oss/release/grafana-10.0.0-1.x86_64.rpm
systemctl enable grafana-server
systemctl start grafana-server
# 默认端口3000，初始密码admin/admin

Grafana中添加Prometheus数据源后，推荐导入社区Dashboard ID：

Node Exporter Full（ID: 1860）：主机CPU、内存、磁盘、网络全景监控
MySQL Overview（ID: 7362）：MySQL性能指标
Spring Boot Statistics（ID: 11378）：Java应用JVM和请求指标

自定义面板时使用PromQL查询关键指标：

# CPU使用率（多核平均）
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# 磁盘IOPS
rate(node_disk_reads_completed_total[1m]) + rate(node_disk_writes_completed_total[1m])

# 网络带宽（入站）
rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[1m]) * 8

# HTTP请求P99延迟（应用自定义指标）
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

告警收敛与故障应急响应流程

告警风暴是运维常见痛点。Alertmanager的group_by机制按告警名称和级别分组，避免同一故障触发多条通知。配合inhibit规则，当critical告警触发时自动抑制关联的warning告警：

# alertmanager.yml 添加inhibit规则
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['instance']

建立故障分级响应机制：critical级别触发电话加短信加IM通知，要求5分钟内响应；warning级别仅IM通知，30分钟内处理。所有告警事件记录到工单系统，形成MTTD（平均检测时间）和MTTR（平均恢复时间）的闭环度量。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/jian-kong-gao-jing-ti-xi-da-jian/