Kubernetes容器编排高可用集群故障排查与混沌工程实践指南

小编 • • 网站运维

Kubernetes容器编排下的高可用架构痛点

Kubernetes已成为容器编排的事实标准，但高可用集群的运维远非”部署完就稳定”那么简单。SRE稳定性工程的核心挑战在于：集群规模扩大后，etcd脑裂、API Server过载、节点NotReady等故障的排查窗口急剧缩短；CI/CD流水线的高频发布引入的变更风险需要混沌工程来提前暴露；DevOps实践中监控告警体系的设计直接决定MTTR（平均恢复时间）。本文围绕Kubernetes生产环境的高可用集群搭建、故障排查和混沌工程实践展开。

高可用etcd集群部署与脑裂防护

etcd是Kubernetes控制面的存储核心，其稳定性直接决定集群可用性。生产环境必须部署3个或5个etcd节点，配合负载均衡实现故障自愈。

# etcd高可用集群部署配置
# etcd-cluster.yaml
apiVersion: v1
kind: Pod
metadata:
  name: etcd
  namespace: kube-system
spec:
  containers:
  - name: etcd
    image: quay.io/coreos/etcd:v3.5.17
    command:
    - /bin/sh
    - -c
    - |
      etcd \
        --name etcd-$(hostname) \
        --data-dir /var/lib/etcd \
        --listen-client-urls https://0.0.0.0:2379 \
        --advertise-client-urls https://$(hostname):2379 \
        --listen-peer-urls https://0.0.0.0:2380 \
        --initial-advertise-peer-urls https://$(hostname):2380 \
        --initial-cluster etcd-node1=https://node1:2380,etcd-node2=https://node2:2380,etcd-node3=https://node3:2380 \
        --initial-cluster-token ha-etcd-cluster \
        --initial-cluster-state new \
        --client-cert-auth \
        --trusted-ca-file /etc/etcd/ca.crt \
        --cert-file /etc/etcd/server.crt \
        --key-file /etc/etcd/server.key \
        --peer-client-cert-auth \
        --peer-trusted-ca-file /etc/etcd/ca.crt \
        --peer-cert-file /etc/etcd/peer.crt \
        --peer-key-file /etc/etcd/peer.key \
        --auto-compaction-retention 1 \
        --max-request-bytes 10485760 \
        --quota-backend-bytes 8589934592
    volumeMounts:
    - name: etcd-data
      mountPath: /var/lib/etcd
    - name: etcd-certs
      mountPath: /etc/etcd
  volumes:
  - name: etcd-data
    hostPath:
      path: /var/lib/etcd
  - name: etcd-certs
    secret:
      secretName: etcd-certs

脑裂防护的关键是确保etcd集群始终维持法定多数（quorum）。当某个节点网络分区后，少数派节点应自动停止服务而非继续接受写入。配置--initial-cluster-state new确保节点只在有效集群中启动。

故障应急响应：节点NotReady的标准化排查流程

节点NotReady是生产Kubernetes集群最常见的告警类型。根因从kubelet到网络到磁盘IO都有可能，需要按流程逐步排除。

#!/bin/bash
# k8s-node-troubleshoot.sh - 节点NotReady标准化排查脚本

NODE_NAME=$1
if [ -z "$NODE_NAME" ]; then
    echo "Usage: $0 <node_name>"
    exit 1
fi

echo "=== 1. 节点状态 ==="
kubectl get node $NODE_NAME -o wide

echo -e "\n=== 2. kubelet状态 ==="
ssh $NODE_NAME "systemctl status kubelet --no-pager -l"

echo -e "\n=== 3. kubelet日志最近错误 ==="
ssh $NODE_NAME "journalctl -u kubelet -n 50 --no-pager | grep -i error"

echo -e "\n=== 4. 容器运行时状态 ==="
ssh $NODE_NAME "systemctl status containerd --no-pager -l"

echo -e "\n=== 5. 磁盘使用率 ==="
ssh $NODE_NAME "df -h / /var/lib/kubelet /var/lib/containerd"

echo -e "\n=== 6. 内存压力 ==="
ssh $NODE_NAME "free -h && cat /proc/memory_pressure"

echo -e "\n=== 7. 网络连通性 ==="
ssh $NODE_NAME "ping -c 3 $API_SERVER_IP"

echo -e "\n=== 8. PLEG状态 ==="
ssh $NODE_NAME "journalctl -u kubelet -n 200 --no-pager | grep -i pleg"

echo -e "\n=== 9. 系统负载 ==="
ssh $NODE_NAME "uptime && iostat -x 1 3"

根据排查数据，NotReady的常见根因分布：PLEG问题（容器运行时卡死）占35%、磁盘压力（imagefs满）占25%、网络分区占20%、内存不足占15%、证书过期占5%。

监控告警体系设计：Prometheus+Thanos长期存储方案

大规模Kubernetes集群的监控数据量巨大，单Prometheus实例无法长期保存。Thanos提供分布式存储和全局查询能力。

# thanos-values.yaml - Helm部署配置
receive:
  enabled: true
  replicaCount: 3
  persistence:
    enabled: true
    size: 500Gi
    storageClass: ssd

compactor:
  enabled: true
  retentionResolutionRaw: 7d
  retentionResolution5m: 30d
  retentionResolution1h: 365d

query:
  enabled: true
  replicaCount: 2
  stores:
    - thanos-receive:10908
    - thanos-store:10905

storegateway:
  enabled: true
  replicaCount: 2
  persistence:
    enabled: true
    size: 100Gi

objstore:
  type: S3
  config:
    bucket: "k8s-metrics-longterm"
    endpoint: "s3.internal.example.com"
    access_key: "${S3_ACCESS_KEY}"
    secret_key: "${S3_SECRET_KEY}"

混沌工程：Chaos Mesh实战注入场景

混沌工程不是”搞破坏”，而是用受控实验验证系统韧性。Chaos Mesh是Kubernetes原生混沌工程平台，支持网络、IO、Pod、时间等多维度故障注入。

# 网络延迟注入 - 模拟etcd集群间网络抖动
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: etcd-network-delay
  namespace: chaos-testing
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - kube-system
    labelSelectors:
      component: etcd
  delay:
    latency: "200ms"
    jitter: "50ms"
    correlation: "25"
  duration: "5m"

---
# Pod故障注入 - 模拟API Server随机重启
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: apiserver-pod-kill
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - kube-system
    labelSelectors:
      component: kube-apiserver
  scheduler:
    cron: "@every 10m"

混沌实验的执行流程：

  1. 定义稳态假设：如”API Server重启后30秒内Pod调度恢复正常”
  2. 注入故障：使用Chaos Mesh创建NetworkChaos/PodChaos
  3. 观测指标：通过Prometheus监控调度延迟、API响应时间
  4. 验证假设：对比注入前后指标是否符合预期
  5. 修复改进：根据暴露的薄弱点加固系统

CI/CD流水线与Kubernetes发布的稳定性保障

高频发布是稳定性风险的主要来源。通过GitOps+渐进式发布将变更风险降至最低：

# Argo Rollouts 渐进式发布配置
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web-api
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - analysis:
          templates:
          - templateName: error-rate-check
      - setWeight: 30
      - pause: {duration: 5m}
      - analysis:
          templates:
          - templateName: latency-check
      - setWeight: 60
      - pause: {duration: 5m}
      - setWeight: 100
      canaryService: web-api-canary
      stableService: web-api-stable
---
# 分析模板 - 错误率检查
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
spec:
  metrics:
  - name: error-rate
    interval: 30s
    count: 6
    successLimit: 5
    provider:
      prometheus:
        address: http://thanos-query:9090
        query: |
          sum(rate(http_requests_total{service="web-api",code=~"5.."}[1m]))
          /
          sum(rate(http_requests_total{service="web-api"}[1m]))
        successCondition: "result[0] < 0.01"

Argo Rollouts配合AnalysisTemplate实现发布过程的自动化质量门禁——当5xx错误率超过1%时自动回滚，无需人工介入。这套方案在10分钟级发布窗口内完成验证，将发布事故率降低至原来的1/5。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/kubernetes-rong-qi-bian-pai-gao-ke-yong-ji-qun-gu-zhang-pai/

赞 (0)
小编小编
0
上一篇 16小时前
下一篇 16小时前

相关推荐