Kubernetes监控告警体系构建：Prometheus+Grafana+AlertManager全链路部署

Kubernetes集群的稳定性高度依赖完善的监控告警体系。Prometheus负责指标采集与存储，Grafana负责数据可视化，AlertManager负责告警路由与通知。三组件联动构建从指标采集到告警触发的全链路监控，覆盖Pod、Node、Service三个层级。

监控体系架构设计与组件选型

采用kube-prometheus-stack Helm Chart一站式部署，集成Prometheus、Grafana、AlertManager、Node Exporter和kube-state-metrics。相比手动部署各组件，Helm方式自动处理RBAC、ServiceMonitor和默认告警规则配置。

# 添加Helm仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 创建监控命名空间
kubectl create namespace monitoring

# 部署kube-prometheus-stack
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --set prometheus.prometheusSpec.retention=30d \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=fast-ssd \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi \
    --set grafana.adminPassword='Gr@fana2026!' \
    --set alertmanager.config.slack_api_url='https://hooks.slack.com/services/xxx'

retention设为30天确保历史数据可追溯。存储使用100Gi SSD，可存储约200万条时间序列。如果集群规模较大（50+节点），建议部署Thanos或VictoriaMetrics做长期存储。

Prometheus服务发现与指标采集配置

kube-prometheus-stack默认通过ServiceMonitor CRD管理采集目标。为自定义应用添加监控，创建ServiceMonitor：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-server-monitor
  namespace: monitoring
  labels:
    release: kube-prometheus-stack  # 必须匹配Prometheus的selector
spec:
  selector:
    matchLabels:
      app: api-server  # 匹配应用的Service标签
  namespaceSelector:
    matchNames:
      - production
  endpoints:
    - port: metrics  # Service中定义的metrics端口名
      interval: 15s
      scrapeTimeout: 10s
      path: /actuator/prometheus
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          targetLabel: node
        - sourceLabels: [__meta_kubernetes_namespace]
          targetLabel: namespace

interval: 15s表示每15秒采集一次。对于关键指标可缩短至5s，但会增加Prometheus负载。relabelings将Kubernetes元数据标签附加到指标上，便于在Grafana中按namespace和node维度过滤。

Grafana仪表盘搭建与关键指标面板

部署完成后通过port-forward访问Grafana：

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

导入官方Dashboard ID快速搭建监控面板。推荐使用的Dashboard：

# 通过Grafana API批量导入Dashboard
DASHBOARDS=(
  "15760"  # Kubernetes Cluster Overview
  "1860"   # Node Exporter Full
  "13105"  # nginx-ingress-controller
  "4701"   # JVM Micrometer
)

for id in "${DASHBOARDS[@]}"; do
  curl -X POST http://admin:password@localhost:3000/api/dashboards/import \
    -H "Content-Type: application/json" \
    -d "{"dashboard": $(curl -s https://grafana.com/api/dashboards/${id}/revisions/latest | jq .dashboard), "folderId": 0, "overwrite": true}"
done

核心监控面板需要覆盖以下指标：集群CPU/内存使用率、Pod重启次数、PV使用率、API Server请求延迟和错误率、etcd性能指标。Pod重启次数持续增长通常意味着OOM或Liveness Probe配置不当。

AlertManager告警规则配置与通知路由

创建PrometheusRule定义告警规则：

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: critical-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: k8s-critical
      rules:
        - alert: NodeCPUHigh
          expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Node CPU usage > 85%"
            description: "Node {{ $labels.instance }} CPU usage is {{ $value }}%"

        - alert: PodCrashLooping
          expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"
            description: "Container {{ $labels.container }} restarted {{ $value }} times in 15min"

        - alert: PVAlmostFull
          expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.85
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "PV {{ $labels.persistentvolumeclaim }} > 85% full"

for字段表示条件持续多久才触发告警，避免瞬时抖动误报。AlertManager配置通知路由，将critical级别告警发送到企业微信/钉钉，warning级别发送到邮件组：

alertmanager:
  config:
    route:
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'default'
      routes:
        - matchers: ['severity="critical"']
          receiver: 'wechat'
        - matchers: ['severity="warning"']
          receiver: 'email'

    receivers:
      - name: 'wechat'
        webhook_configs:
          - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxx'
            send_resolved: true
      - name: 'email'
        email_configs:
          - to: 'ops-team@company.com'
            send_resolved: true

repeat_interval: 4h表示告警未恢复时每4小时重复发送一次。send_resolved: true确保告警恢复时发送恢复通知，避免运维人员手动确认。

常见监控盲区与故障排查

问题：部分Pod指标缺失

检查ServiceMonitor的selector是否正确匹配到应用的Service。使用kubectl get servicemonitor -A确认ServiceMonitor状态。Prometheus Web UI的Status -> Service Discovery页面可查看所有采集目标及其状态。如果target显示DOWN，检查应用的/metrics端口是否正常暴露。

问题：Prometheus存储空间增长过快

高基数指标（如包含user_id、request_id的标签）会导致时间序列爆炸。使用promtool tsdb analyze分析高基数指标。在Recording Rules中预聚合高频查询指标，减少查询时计算量。