Grafana可视化监控仪表盘设计与多数据源集成配置实战

Grafana是开源的可视化与监控分析平台，支持Prometheus、InfluxDB、MySQL、Elasticsearch、Loki等数十种数据源，通过灵活的查询编辑器和面板系统构建运维仪表盘。在SRE实践中，Grafana仪表盘是故障响应的视觉中枢，将分散在多个监控系统中的指标、日志、链路追踪数据汇聚为统一的可观测性视图。本文从数据源配置、仪表盘设计、告警规则、权限管理四个维度展开实战配置。

Grafana数据源配置与Provisioning自动化管理

Grafana支持通过配置文件（Provisioning）自动管理数据源，避免手动在Web界面逐个添加。生产环境中数据源配置应纳入版本控制，与基础设施代码同步管理。

# docker-compose部署Grafana
cat > docker-compose.yml << 'EOF'
version: '3.8'
services:
  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=ChangeMeNow!
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_SERVER_DOMAIN=grafana.internal
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=smtp.internal:587
      - GF_SMTP_USER=alert@internal
      - GF_SMTP_PASSWORD=smtp_password
      - GF_SMTP_FROM_ADDRESS=alert@internal
    volumes:
      - grafana_data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
      - ./dashboards:/var/lib/grafana/dashboards
    restart: unless-stopped

volumes:
  grafana_data:
EOF

docker compose up -d

# 数据源Provisioning配置
# provisioning/datasources/datasources.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: '15s'
      httpMethod: POST

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      maxLines: 1000

  - name: MySQL-Production
    type: mysql
    access: proxy
    url: mysql-prod:3306
    database: metrics
    user: grafana_reader
    secureJsonData:
      password: ${MYSQL_GRAFANA_PASSWORD}
    jsonData:
      sslMode: required

  - name: Elasticsearch
    type: elasticsearch
    access: proxy
    url: http://elasticsearch:9200
    database: "logstash-*"
    jsonData:
      esVersion: 8.0
      timeField: "@timestamp"
      interval: Daily

仪表盘面板类型选择与PromQL查询设计

Grafana提供时间序列图、仪表盘、柱状图、热力图、表格、日志视图等多种面板类型。合理的面板类型选择直接影响数据可读性。CPU使用率适合时间序列图，队列积压适合柱状图，请求延迟分布适合热力图，服务健康状态适合状态面板。

# 关键PromQL查询模板

# 1. CPU使用率（按节点分组）
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 2. 内存使用率
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
  / node_memory_MemTotal_bytes * 100

# 3. 磁盘空间使用率（按挂载点）
100 - (node_filesystem_avail_bytes{fstype!="tmpfs"} 
  / node_filesystem_size_bytes{fstype!="tmpfs"} * 100)

# 4. HTTP请求QPS（按状态码分类）
sum by(status)(rate(http_requests_total[1m]))

# 5. P99延迟（5分钟窗口）
histogram_quantile(0.99, 
  sum by(le)(rate(http_request_duration_seconds_bucket[5m])))

# 6. 错误率（5xx占总请求比例）
sum(rate(http_requests_total{status=~"5.."}[5m])) 
  / sum(rate(http_requests_total[5m])) * 100

# 7. Pod重启次数（过去1小时）
increase(kube_pod_container_status_restarts_total[1h])

# 8. 网络流量（接收/发送速率）
rate(node_network_receive_bytes_total{device!="lo"}[1m]) * 8  # 转换为bps
rate(node_network_transmit_bytes_total{device!="lo"}[1m]) * 8

# 9. 连接数监控（按状态）
node_netstat_Tcp_CurrEstab
node_netstat_Tcp_TimeWait

# 10. 服务可用性（过去5分钟成功请求比例）
sum(rate(http_requests_total{status!~"5.."}[5m])) 
  / sum(rate(http_requests_total[5m])) * 100

面板变量（Variables）是Grafana仪表盘的核心功能，允许用户通过下拉菜单动态筛选数据。典型变量包括集群名称、命名空间、服务名、节点名等：

# 仪表盘变量配置示例
# 变量1：集群选择
name: cluster
query: label_values(kube_node_info, cluster)

# 变量2：命名空间选择（依赖集群变量）
name: namespace
query: label_values(kube_pod_info{cluster="$cluster"}, namespace)

# 变量3：服务选择（依赖命名空间变量）
name: service
query: label_values(kube_pod_info{cluster="$cluster",namespace="$namespace"}, pod)

# 变量4：时间间隔
name: interval
type: interval
options:
  - text: 1m, value: 1m
  - text: 5m, value: 5m
  - text: 15m, value: 15m
  - text: 1h, value: 1h

Grafana统一告警规则与通知路由配置

Grafana 8.0+引入了内置告警系统，支持统一管理来自Prometheus、Loki等多个数据源的告警规则。告警规则通过Provisioning配置文件管理，配合通知策略实现多级路由。

# 告警规则Provisioning配置
# provisioning/alerting/rules.yml
apiVersion: 1
groups:
  - name: system-health
    folder: Infrastructure
    interval: 30s
    rules:
      - uid: high-cpu-usage
        title: CPU使用率过高
        condition: A
        data:
          - refId: A
            relativeTimeRange:
              from: 300
            datasourceUid: prometheus
            model:
              expr: "100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)"
              instant: true
          - refId: B
            relativeTimeRange:
              from: 300
            datasourceUid: prometheus
            model:
              expr: "85"
              instant: true
        noDataState: NoData
        for: 5m
        annotations:
          summary: "CPU使用率超过85%"
          description: "节点 {{ $labels.instance }} CPU使用率持续5分钟超过85%，当前值: {{ $values.A }}"
        labels:
          severity: warning
          team: infra

      - uid: disk-space-critical
        title: 磁盘空间严重不足
        condition: A
        data:
          - refId: A
            relativeTimeRange:
              from: 300
            datasourceUid: prometheus
            model:
              expr: "100 - (node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"} * 100) > 90"
              instant: true
        noDataState: NoData
        for: 2m
        annotations:
          summary: "磁盘空间使用率超过90%"
          description: "节点 {{ $labels.instance }} 挂载点 {{ $labels.mountpoint }} 磁盘使用率超过90%"
        labels:
          severity: critical
          team: infra

# 通知策略配置
# provisioning/alerting/contact-points.yml
apiVersion: 1
contactPoints:
  - uid: webhook-infra
    name: 基础设施团队
    type: webhook
    settings:
      url: http://alertmanager.internal/api/webhook
      httpMethod: POST

  - uid: dingtalk
    name: 钉钉告警群
    type: dingding
    settings:
      url: https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN
      message: |
        {{ range .alerts }}
        **{{ .status }}**: {{ .annotations.summary }}
        {{ .annotations.description }}
        {{ end }}

  - uid: email
    name: 邮件通知
    type: email
    settings:
      addresses: ops-team@internal

# 通知路由策略
# provisioning/alerting/notification-policy.yml
apiVersion: 1
policies:
  - receiver: webhook-infra
    group_by: ['alertname', 'cluster']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: dingtalk
        matchers:
          - severity="critical"
        group_wait: 10s
        repeat_interval: 1h
      - receiver: email
        matchers:
          - team="app"
        group_wait: 1m

仪表盘JSON模型与版本管理最佳实践

Grafana仪表盘以JSON格式存储，支持导出导入和版本控制。将仪表盘JSON纳入Git仓库管理，配合CI/CD流水线实现仪表盘的自动化部署。团队协作时，通过Pull Request审核仪表盘变更，确保监控视图的一致性。

# 仪表盘Provisioning配置
# provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: Infrastructure
    folder: Infrastructure
    type: file
    updateInterval: 30s
    options:
      path: /var/lib/grafana/dashboards/infrastructure

  - name: Applications
    folder: Applications
    type: file
    updateInterval: 30s
    options:
      path: /var/lib/grafana/dashboards/applications

# 通过API导出仪表盘
curl -u admin:password http://localhost:3000/api/dashboards/uid/abc123 |   jq '.dashboard' > dashboards/infrastructure/node-overview.json

# 通过API导入仪表盘
curl -u admin:password -X POST   http://localhost:3000/api/dashboards/db   -H "Content-Type: application/json"   -d @dashboards/infrastructure/node-overview.json

# 批量导入社区仪表盘
# Node Exporter Full
wget https://grafana.com/api/dashboards/1860/revisions/37/download -O node-full.json
# MySQL Overview
wget https://grafana.com/api/dashboards/7362/revisions/5/download -O mysql-overview.json

仪表盘设计遵循"单一焦点"原则：每个仪表盘围绕一个核心主题（如节点概览、服务延迟、数据库性能），每个面板回答一个具体问题。面板过多会导致视觉过载，建议单页不超过12个面板。关键指标面板放置在顶部，详情面板放在下方。颜色编码统一：绿色代表正常、黄色代表警告、红色代表异常，确保团队对颜色含义有统一认知。