Prometheus + Alertmanager监控告警体系搭建实战

小编 • • 网站运维

SRE工程中监控告警是稳定性保障的第一道防线。本文覆盖Prometheus部署与自动发现、Alertmanager分级通知路由、Grafana可视化对接、高可用部署方案与告警收敛实践。

为什么需要自建监控告警体系

SRE工程中，监控告警是稳定性保障的第一道防线。裸机部署Prometheus + Alertmanager组合，能实现秒级指标采集、灵活告警规则和分级通知路由，成本远低于商业SaaS方案。这套体系覆盖从指标采集、规则评估到通知分发的完整链路，适合10-500台服务器规模的基础设施监控。这篇实战指南覆盖Prometheus部署配置、Alertmanager告警路由、Grafana可视化对接三个核心环节。

Prometheus部署与基础配置

安装与启动

# 下载Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v3.3.0/prometheus-3.3.0.linux-amd64.tar.gz
tar xzf prometheus-3.3.0.linux-amd64.tar.gz
cd prometheus-3.3.0.linux-amd64

# 创建数据和配置目录
mkdir -p /data/prometheus /etc/prometheus

# 基础配置文件
cat > /etc/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    env: 'prod'

rule_files:
  - '/etc/prometheus/rules/*.yml'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

scrape_configs:
  - job_name: 'node'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/node/*.json']
        refresh_interval: 30s
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):.*'
      - source_labels: [__meta_datacenter]
        target_label: dc

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
      - files: ['/etc/prometheus/targets/blackbox/*.json']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115
EOF

# 启动
./prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/data/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=80GB \
  --web.enable-lifecycle

关键配置项说明

scrape_interval设15s是平衡精度和存储的折中值。高频指标（如HTTP延迟P99）可以单独配置更短的采集间隔。retention.time=30d配合retention.size=80GB做双重限制，避免磁盘写满。

Node Exporter批量部署与自动发现

# 在所有被监控节点安装Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xzf node_exporter-1.8.2.linux-amd64.tar.gz

# 创建systemd service
cat > /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Node Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.processes \
  --web.listen-address=:9100
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl enable node_exporter && systemctl start node_exporter

使用file_sd_configs实现自动发现：

# /etc/prometheus/targets/node/nodes.json
[
  {
    "targets": ["10.0.1.10:9100", "10.0.1.11:9100", "10.0.1.12:9100"],
    "labels": {
      "job": "node",
      "datacenter": "bj-1",
      "env": "prod"
    }
  },
  {
    "targets": ["10.0.2.10:9100", "10.0.2.11:9100"],
    "labels": {
      "job": "node",
      "datacenter": "sh-1",
      "env": "prod"
    }
  }
]

新增节点只需更新JSON文件，Prometheus每30s自动刷新目标列表。

Alertmanager告警路由与分级通知

安装Alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.28.0/alertmanager-0.28.0.linux-amd64.tar.gz
tar xzf alertmanager-0.28.0.linux-amd64.tar.gz

cat > /etc/prometheus/alertmanager.yml << 'EOF'
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alert@example.com'
  smtp_auth_username: 'alert@example.com'
  smtp_auth_password: 'your_password'

route:
  group_by: ['alertname', 'cluster', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical-team'
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: 'warning-channel'
      repeat_interval: 6h
    - match_re:
        alertname: ^(Watchdog|InfoInhibitor)$
      receiver: 'devnull'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops-team@example.com'
    webhook_configs:
      - url: 'http://localhost:8080/alerts'
        send_resolved: true

  - name: 'critical-team'
    email_configs:
      - to: 'oncall@example.com'
    # 企业微信/飞书webhook
    webhook_configs:
      - url: 'https://open.feishu.cn/open-apis/bot/v2/hook/your-hook-id'
        send_resolved: true

  - name: 'warning-channel'
    email_configs:
      - to: 'ops-group@example.com'

  - name: 'devnull'

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'cluster', 'namespace']
EOF

# 启动
./alertmanager --config.file=/etc/prometheus/alertmanager.yml

inhibit_rules的作用：当同一告警的critical级别触发时，自动抑制warning级别通知，避免重复告警。

核心告警规则模板

# /etc/prometheus/rules/infra.yml
groups:
  - name: node_alerts
    rules:
      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "节点 {{ $labels.instance }} 不可达"
          description: "节点已宕机超过1分钟"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes) < 0.15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘空间不足 {{ $labels.instance }} {{ $labels.mountpoint }}"
          description: "可用空间低于15%，当前值 {{ $value | printf \"%.1f\" }}%"

      - alert: HighLoadAvg
        expr: node_load15 / count(node_cpu_seconds_total{mode="idle"}) without (cpu, mode) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "负载过高 {{ $labels.instance }}"
          description: "15分钟负载超过CPU核心数2倍"

  - name: http_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.service }} 5xx错误率超过5%"

Grafana Dashboard对接与常用面板

# Docker启动Grafana
docker run -d --name grafana \
  -p 3000:3000 \
  -v /data/grafana:/var/lib/grafana \
  -e GF_SECURITY_ADMIN_PASSWORD=yourpassword \
  grafana/grafana:11.0.0

# 添加Prometheus数据源
# UI: Configuration -> Data Sources -> Add Prometheus
# URL: http://prometheus-host:9090

推荐导入以下社区Dashboard（Grafana Dashboard ID）：

  • 1860 - Node Exporter Full（主机指标全景）
  • 7362 - Blackbox Exporter（HTTP探测）
  • 15991 - Nginx VTS Stats（Nginx指标）

自定义面板核心PromQL

# CPU使用率（按核心排除idle）
100 - avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

# 内存使用率
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# 磁盘I/O延迟P99
histogram_quantile(0.99, sum(rate(node_disk_io_time_seconds_bucket[5m])) by(le, device))

# 网络带宽利用率
rate(node_network_receive_bytes_total{device=~"eth0"}[5m]) * 8 / 1e9

高可用部署方案

单点Prometheus不可靠。生产环境推荐以下高可用架构：

方案：双副本 + 远程写入

# 两台Prometheus实例采集相同目标，互为备份
# Alertmanager集群模式（3节点）
./alertmanager \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=alertmgr-1:9094 \
  --cluster.peer=alertmgr-2:9094

# Prometheus远程写入到Victoriametrics或Thanos Receive做长期存储
# prometheus.yml 添加：
remote_write:
  - url: "http://victoriametrics:8428/api/v1/write"
    queue_config:
      max_samples_per_send: 10000
      capacity: 20000
      retry_on_http_429: true

两套Prometheus + Alertmanager集群：即使一个实例故障，另一个仍能正常采集和告警。告警去重由Alertmanager的group_by机制保障。

告警收敛与降噪实践

线上告警噪音是运维效率的最大杀手。收敛策略：

  • group_by按alertname+cluster+namespace聚合，避免同一故障触发N条告警
  • group_wait=30s给系统缓冲时间，短时抖动自动恢复
  • repeat_interval分级设置：critical 1h、warning 6h
  • 使用inhibit_rules：critical触发时抑制同组warning
  • 添加Watchdog规则作为心跳检测，确保告警链路本身正常

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/prometheusalertmanager-jian-kong-gao-jing-ti-xi-da-jian-shi/

赞 (0)
小编小编
0
上一篇 20小时前
下一篇 20小时前

相关推荐