Kubernetes容器编排环境下的监控告警体系是SRE稳定性工程的基础设施。Prometheus提供多维度指标采集与存储，Grafana负责可视化面板展示，AlertManager实现告警路由与通知分发。三者在Kubernetes集群中的生产级部署需要解决服务发现、动态配置、告警收敛等实际问题。本文给出完整的Helm部署方案和告警规则编写指南。

Prometheus监控组件Helm部署与服务发现配置

使用kube-prometheus-stack Helm Chart一次性部署Prometheus、Grafana、AlertManager和Node Exporter：

# 添加Prometheus社区仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 创建监控命名空间
kubectl create namespace monitoring

# 自定义values文件
cat > /tmp/prom-values.yaml << 'EOF'
prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: 100GB
    scrapeInterval: 30s
    evaluationInterval: 30s

    # 存储配置
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 200Gi

    # 资源限制
    resources:
      requests:
        cpu: "1"
        memory: 4Gi
      limits:
        cpu: "4"
        memory: 8Gi

    # 额外的Scrape配置
    additionalScrapeConfigs:
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          resources:
            requests:
              storage: 10Gi
    config:
      global:
        resolveTimeout: 5m
      route:
        group_by: ['alertname', 'namespace', 'severity']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 4h
        receiver: 'default'
        routes:
          - matchers: ['severity="critical"']
            receiver: 'pagerduty'
            group_wait: 10s
            repeat_interval: 1h
          - matchers: ['severity="warning"']
            receiver: 'slack'
      receivers:
        - name: 'default'
          webhook_configs:
            - url: 'http://alert-router:8080/webhook'
        - name: 'slack'
          slack_configs:
            - api_url: 'https://hooks.slack.com/services/xxx'
              channel: '#alerts-warning'
        - name: 'pagerduty'
          pagerduty_configs:
            - routing_key: 'your-routing-key'

grafana:
  adminPassword: "YourSecurePassword2026"
  persistence:
    enabled: true
    size: 20Gi
    storageClassName: fast-ssd
  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
        - name: Prometheus
          type: prometheus
          url: http://prometheus-operated:9090
          isDefault: true
EOF

# 安装
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    -n monitoring \
    -f /tmp/prom-values.yaml \
    --version 65.0.0

Prometheus告警规则编写与Recording Rules优化

告警规则通过PrometheusRule CRD注入，支持热更新无需重启Prometheus：

cat > /tmp/alert-rules.yaml << 'EOF'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: k8s-production-alerts
  namespace: monitoring
  labels:
    prometheus: kube-prometheus-stack
spec:
  groups:
    - name: kubernetes-node-alerts
      interval: 30s
      rules:
        # 节点CPU使用率告警
        - alert: NodeHighCPUUsage
          expr: |
            100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
          for: 10m
          labels:
            severity: warning
            category: resource
          annotations:
            summary: "节点CPU使用率过高: {{ $labels.instance }}"
            description: "节点 {{ $labels.instance }} CPU使用率持续10分钟超过85%，当前值: {{ $value }}%"

        # 节点内存压力告警
        - alert: NodeMemoryPressure
          expr: |
            (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "节点内存压力: {{ $labels.instance }}"
            description: "可用内存不足10%，当前使用率: {{ $value }}%"

        # Pod CrashLoopBackOff检测
        - alert: PodCrashLoopBackOff
          expr: |
            increase(kube_pod_container_status_restarts_total[1h]) > 5
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Pod频繁重启: {{ $labels.namespace }}/{{ $labels.pod }}"
            description: "容器 {{ $labels.container }} 在1小时内重启超过5次"

    - name: kubernetes-resource-alerts
      interval: 30s
      rules:
        # PVC存储空间不足
        - alert: PVCFillingUp
          expr: |
            kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100 > 80
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "PVC存储空间不足: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
            description: "PVC使用率超过80%，当前: {{ $value }}%"

        # API Server错误率
        - alert: APIServerHighErrorRate
          expr: |
            sum(rate(apiserver_request_total{code=~"5.."}[5m])) 
            / sum(rate(apiserver_request_total[5m])) * 100 > 5
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Kubernetes API Server错误率过高"
            description: "5xx错误率超过5%，当前: {{ $value }}%"

        # HPA达到最大副本数
        - alert: HPAMaxedOut
          expr: |
            kube_hpa_status_current_replicas == kube_hpa_spec_max_replicas
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "HPA已达到最大副本数: {{ $labels.namespace }}/{{ $labels.hpa }}"
            description: "当前副本数已达上限 {{ $value }}，可能需要调整资源配额"
EOF

kubectl apply -f /tmp/alert-rules.yaml

Recording Rules用于预计算高频查询指标，降低Prometheus查询负载：

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: recording-rules
  namespace: monitoring
spec:
  groups:
    - name: precomputed-metrics
      interval: 1m
      rules:
        - record: namespace:pod_cpu_usage:sum
          expr: |
            sum by(namespace) (
              rate(container_cpu_usage_seconds_total{container!=""}[5m])
            )

        - record: namespace:pod_memory_usage:sum
          expr: |
            sum by(namespace) (
              container_memory_working_set_bytes{container!=""}
            )

        - record: cluster:node_cpu_utilization:ratio
          expr: |
            sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) 
            / sum(rate(node_cpu_seconds_total[5m]))

Grafana可视化面板配置与Dashboard自动化管理

生产环境Dashboard通过ConfigMap管理，实现版本控制和自动加载：

cat > /tmp/grafana-dashboards.yaml << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: production-dashboards
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  k8s-cluster-overview.json: |
    {
      "dashboard": {
        "title": "Kubernetes集群总览",
        "panels": [
          {
            "title": "节点CPU使用率",
            "type": "stat",
            "targets": [{
              "expr": "cluster:node_cpu_utilization:ratio * 100",
              "legendFormat": "集群平均"
            }],
            "fieldConfig": {
              "defaults": {
                "thresholds": {
                  "steps": [
                    {"color": "green", "value": 0},
                    {"color": "yellow", "value": 70},
                    {"color": "red", "value": 85}
                  ]
                }
              }
            }
          },
          {
            "title": "Pod状态分布",
            "type": "piechart",
            "targets": [{
              "expr": "sum by(phase) (kube_pod_status_phase)",
              "legendFormat": "{{phase}}"
            }]
          }
        ]
      }
    }
EOF

kubectl apply -f /tmp/grafana-dashboards.yaml

自定义业务指标采集与ServiceMonitor配置

业务应用通过/metrics端点暴露Prometheus格式指标，使用ServiceMonitor自动发现：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-service-monitor
  namespace: production
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: api-service
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: pod_name
        - sourceLabels: [__meta_kubernetes_namespace]
          targetLabel: namespace

业务应用侧需暴露指标，以Go语言为例：

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var (
    httpRequestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP请求处理耗时",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "path", "status"},
    )

    activeConnections = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "active_connections",
            Help: "当前活跃连接数",
        },
    )
)

func main() {
    prometheus.MustRegister(httpRequestDuration)
    prometheus.MustRegister(activeConnections)

    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

告警收敛策略与On-Call排障流程

AlertManager的告警收敛通过group_by、group_wait、group_interval、repeat_interval四个参数协同实现。生产环境推荐配置：group_by按alertname+namespace分组，group_wait设为30秒避免短时间风暴，repeat_interval根据严重等级区分——critical 1小时、warning 4小时。

inhibitions规则用于抑制低优先级告警，例如节点宕机时抑制该节点上的所有Pod告警：

inhibit_rules:
  - source_matchers: ['alertname="NodeDown"']
    target_matchers: ['alertname=~"Pod.*"']
    equal: ['instance']

告警信息必须包含可执行的排障指引。annotations中的description字段应包含：指标当前值、阈值、影响范围、排查命令。运维人员收到告警后按标准流程操作——先确认告警真实性（Prometheus UI查询指标），再执行应急预案（扩容/降级/切流），最后进行根因分析并更新Runbook。