Prometheus监控告警体系搭建与Grafana可视化仪表盘配置实战

小编 • • 网站运维

Prometheus是云原生监控领域的事实标准，采用Pull模式采集指标数据，通过多维标签模型和PromQL查询语言提供灵活的指标聚合能力。配合Grafana可视化仪表盘和Alertmanager告警通道，可以构建覆盖基础设施、应用服务、业务指标的完整监控告警体系。本文从Prometheus架构设计到Grafana仪表盘配置，给出一套可直接部署的监控告警方案。

Prometheus监控架构与数据模型设计

Prometheus的核心架构包含五个组件：Prometheus Server负责指标采集和存储、Exporter负责暴露监控指标、Alertmanager负责告警路由和通知、Pushgateway负责短生命周期任务指标推送、Grafana负责数据可视化。

Prometheus的数据模型基于时序数据库，每个指标由指标名称和一组键值对标签唯一标识。指标类型分为Counter（只增不减的计数器）、Gauge（可增可减的瞬时值）、Histogram（分布统计）、Summary（分位数统计）。合理选择指标类型是设计监控体系的第一步。

指标命名遵循 namespace_subsystem_name_unit 规范，例如 http_requests_total 表示HTTP请求总数， node_cpu_seconds_total 表示CPU使用时间。标签用于多维度筛选，例如 http_requests_total{method="GET",status="200"} 可以精确筛选GET方法且状态码200的请求。

Prometheus服务端安装与配置

使用Docker Compose部署Prometheus Server是最快捷的方式：

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:v2.51.0
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules.yml:/etc/prometheus/rules.yml
      - prom_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'

  alertmanager:
    image: prom/alertmanager:v0.27.0
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

  grafana:
    image: grafana/grafana:10.4.0
    ports: ["3000:3000"]
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your_secure_password

volumes:
  prom_data:
  grafana_data:

prometheus.yml核心配置：

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100']
        labels:
          env: 'production'

  - job_name: 'app'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['app-server:8080']

  - job_name: 'dynamic-targets'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/*.yml']
        refresh_interval: 30s

Exporter数据采集与指标暴露

Node Exporter采集主机层面的CPU、内存、磁盘、网络指标，在每台被监控服务器上部署：

docker run -d --name node-exporter --net=host --pid=host \\
  --restart=always -v /:/host:ro,rslave \\
  quay.io/prometheus/node-exporter:latest --path.rootfs=/host

对于Spring Boot应用，通过 micrometer-registry-prometheus 依赖暴露指标：

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: prometheus,health,info
  metrics:
    tags:
      application: ${spring.application.name}

自定义业务指标通过Micrometer API暴露：

@Service
public class OrderMetrics {
    private final Counter orderCounter;
    private final Timer orderProcessTimer;

    public OrderMetrics(MeterRegistry registry) {
        this.orderCounter = Counter.builder("order_created_total")
            .tag("type", "default")
            .description("Total orders created")
            .register(registry);

        this.orderProcessTimer = Timer.builder("order_process_duration")
            .tag("stage", "processing")
            .description("Order processing duration")
            .register(registry);
    }

    public void recordOrder() {
        orderCounter.increment();
    }

    public void recordProcessTime(long millis) {
        orderProcessTimer.record(millis, TimeUnit.MILLISECONDS);
    }
}

PromQL查询语法与告警规则配置

PromQL是Prometheus的查询语言，支持聚合、运算、函数操作。常用查询示例：

# CPU使用率（5分钟平均值）
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

# 磁盘使用率
1 - node_filesystem_avail_bytes / node_filesystem_size_bytes

# HTTP请求QPS（按状态码分组）
sum(rate(http_requests_total[5m])) by(status)

# P99延迟
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by(le))

# 错误率
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100

告警规则定义在rules.yml中：

groups:
  - name: infra-alerts
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU使用率过高 {{ $labels.instance }}"
          description: "CPU使用率 {{ $value }}% 超过80%阈值持续5分钟"

      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "磁盘空间不足 {{ $labels.instance }} {{ $labels.mountpoint }}"

  - name: app-alerts
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100 > 5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "HTTP错误率过高"

      - alert: ServiceDown
        expr: up{job="app"} == 0
        for: 1m
        labels:
          severity: critical

Grafana可视化仪表盘与告警通知集成

Grafana配置Prometheus数据源后，通过Dashboard JSON或手动添加Panel构建可视化。常用Dashboard模板可从grafana.com/dashboards导入，Node Exporter Full（ID: 1860）和Spring Boot Statistics（ID: 12900）是使用最广泛的模板。

自定义Panel的PromQL示例配置：

{
  "title": "服务请求QPS趋势",
  "type": "timeseries",
  "datasource": "Prometheus",
  "targets": [{
    "expr": "sum(rate(http_requests_total[5m])) by(service)",
    "legendFormat": "{{service}}",
    "refId": "A"
  }],
  "fieldConfig": {
    "defaults": {
      "unit": "reqps"
    }
  }
}

Alertmanager配置告警通知路由，支持邮件、钉钉、企业微信等渠道：

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - matchers: ['severity=critical']
      receiver: 'critical'
      group_wait: 10s
    - matchers: ['severity=warning']
      receiver: 'warning'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://dingtalk-webhook/alert'

  - name: 'critical'
    webhook_configs:
      - url: 'http://dingtalk-webhook/critical'
    email_configs:
      - to: 'ops-team@company.com'
        send_resolved: true

  - name: 'warning'
    webhook_configs:
      - url: 'http://dingtalk-webhook/warning'

告警抑制（Inhibit）规则避免告警风暴，例如服务宕机时抑制其下游应用的告警：

inhibit_rules:
  - source_matchers: ['alertname=ServiceDown', 'severity=critical']
    target_matchers: ['severity=warning']
    equal: ['instance']

这套配置在生产环境中运行稳定，Prometheus存储30天历史数据，Grafana提供实时可视化，Alertmanager将关键告警通过钉钉和邮件同步推送，覆盖了从基础设施到应用层的全链路监控需求。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/prometheus-jian-kong-gao-jing-ti-xi-da-jian-yu-grafana-ke/

赞 (0)
小编小编
0
上一篇 15小时前
下一篇 15小时前

相关推荐