监控告警体系是SRE稳定性工程的基础设施。Prometheus因其多维数据模型和强大的PromQL查询语言，已成为云原生监控的事实标准。本文覆盖从指标采集、存储配置、告警规则编写到Alertmanager路由分发的完整链路。
架构设计与组件选型
一个完整的Prometheus监控体系包含以下组件：Prometheus Server负责指标采集和存储；Node Exporter采集主机级指标；Blackbox Exporter做HTTP/TCP/ICMP探测；Alertmanager负责告警去重、分组和路由分发；Grafana做数据可视化。
架构示意:
Exporter --> Prometheus Server --> Alertmanager --> 钉钉/邮件/企业IM
|
Node Exporter (主机指标)
Blackbox Exporter (HTTP探测)
App /metrics (应用指标)
Prometheus Server安装与核心配置
使用Docker部署Prometheus，配合docker-compose管理：
# docker-compose.yml
version: '3.8'
services:
prometheus:
image: prom/prometheus:v2.52.0
container_name: prometheus
restart: always
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./rules:/etc/prometheus/rules
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=30d'
- '--storage.tsdb.retention.size=50GB'
- '--web.enable-lifecycle'
volumes:
prometheus_data:
prometheus.yml主配置文件：
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093']
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node'
static_configs:
- targets:
- 'web-01:9100'
- 'web-02:9100'
- 'db-01:9100'
- job_name: 'app'
metrics_path: /metrics
static_configs:
- targets: ['app-01:8080', 'app-02:8080']
- job_name: 'blackbox_http'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://api.example.com/health
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: target
- target_label: __address__
replacement: blackbox:9115
注意scrape_interval的取值策略：15秒适合大多数场景。过于频繁的采集如5秒会增加Prometheus的CPU和存储压力。对于关键指标可以通过job级override设置更短的采集间隔。
告警规则编写与PromQL实战
告警规则是监控体系的核心。规则文件示例：
# /etc/prometheus/rules/infra.yml
groups:
- name: infra_alerts
rules:
- alert: HighCpuUsage
expr: |
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
team: infra
annotations:
summary: "CPU使用率过高: {{ $labels.instance }}"
description: "实例 {{ $labels.instance }} CPU使用率已达 {{ $value | printf \"%.1f\" }}%"
- alert: LowMemory
expr: |
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
for: 3m
labels:
severity: critical
team: infra
annotations:
summary: "内存不足: {{ $labels.instance }}"
- alert: DiskSpaceLow
expr: |
(1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 > 85
for: 10m
labels:
severity: warning
team: infra
annotations:
summary: "磁盘空间不足: {{ $labels.instance }} {{ $labels.mountpoint }}"
- alert: ServiceDown
expr: up == 0
for: 1m
labels:
severity: critical
team: ops
annotations:
summary: "服务不可达: {{ $labels.job }} {{ $labels.instance }}"
规则编写的要点：for字段设置持续时间避免瞬时波动触发告警；severity区分告警级别便于路由分发；annotations中的$value变量在告警触发时会被实际值替换；PromQL中用rate()处理counter类型指标，用avg by()做维度聚合。
应用自定义指标暴露
Prometheus的Pull模式要求应用主动暴露/metrics端点。Python应用使用prometheus_client库实现CI/CD流水线中的监控埋点：
from prometheus_client import Counter, Histogram, Gauge, generate_latest
from flask import Flask, Response
app = Flask(__name__)
REQUEST_COUNT = Counter(
'http_requests_total', 'Total HTTP requests',
['method', 'endpoint', 'status']
)
REQUEST_LATENCY = Histogram(
'http_request_duration_seconds', 'HTTP request latency',
['endpoint'],
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)
@app.before_request
def before_request():
REQUEST_LATENCY.labels(endpoint=request.path).observe(0.15)
@app.route('/metrics')
def metrics():
return Response(generate_latest(), mimetype='text/plain')
Histogram的buckets选择需要根据实际延迟分布调整。如果大部分请求在100ms内但有少量慢请求在2-5秒，合理的buckets配置应覆盖这个范围。不要直接用默认buckets，它们针对通用场景设计，对特定业务不一定合适。
Alertmanager告警路由配置
Alertmanager负责告警的去重、分组、静默和路由。核心配置：
# alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'default'
routes:
- match:
severity: critical
receiver: 'critical-webhook'
group_wait: 10s
repeat_interval: 1h
- match:
team: infra
receiver: 'infra-webhook'
- match_re:
severity: 'warning|info'
mute_time_intervals: ['offhours']
receiver: 'default'
mute_time_intervals:
- name: offhours
time_intervals:
- times:
- start_time: '22:00'
end_time: '09:00'
weekdays: ['mon:fri']
location: 'Asia/Shanghai'
receivers:
- name: 'default'
webhook_configs:
- url: 'http://dingtalk-webhook:8060/dingtalk/ops/send'
send_resolved: true
- name: 'critical-webhook'
webhook_configs:
- url: 'http://dingtalk-webhook:8060/dingtalk/critical/send'
send_resolved: true
group_by决定了哪些告警会被合并成一条通知。将alertname和instance放入group_by，同一实例的同类告警只发一条。group_wait控制首次告警的缓冲时间，在窗口期内触发的同组告警会合并发送，减少告警风暴。send_resolved: true使告警恢复时也发送通知，对运维人员确认故障修复很有用。
日志分析与指标关联
混沌工程实践中，监控指标和日志的关联分析是快速定位根因的关键。Prometheus指标告诉你什么坏了，日志告诉你为什么坏。推荐使用Loki作为日志后端，与Grafana集成实现指标和日志的联动查询：
# Panel 1: Prometheus指标（PromQL）
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
# Panel 2: Loki日志查询（LogQL）
{app="myapp"} |= "ERROR" | json | line_format "{{.msg}} {{.error}}"
Grafana支持在两个Panel间设置时间联动，点击指标图表的异常时段，日志Panel自动同步到同一时间范围，实现从指标异常到日志详情的快速下钻。这种关联查询能力在故障应急响应中能将平均定位时间（MTTI）缩短60%以上。
常见配置问题排查
采集目标显示down：检查目标端点是否可达、防火墙规则、Exporter是否运行。在Prometheus Web UI的Status页面可以看到每个job的采集状态和最后一次错误信息。
告警不触发：在Prometheus Web UI的Alerts页面查看规则状态。pending表示已满足表达式条件但还在for指定的等待时间内；firing表示已发送到Alertmanager。如果规则显示firing但没收到通知，检查Alertmanager的路由配置和receiver的webhook地址。
存储空间增长过快：高基数标签是主要元凶。检查是否有标签值过于离散（如user_id、request_id作为标签）。Prometheus的每个唯一标签组合都是一条独立时间序列，高基数标签会导致序列数爆炸。原则是标签值的变化频率不应超过指标采集频率。
原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/prometheus-jian-kong-gao-jing-ti-xi-da-jian-cong-zhi-biao/
