Kubernetes集群高可用架构设计：从控制平面到故障自愈实践

互联网

Kubernetes已成为容器编排领域的事实标准，但生产环境中的K8s集群面临诸多挑战：控制平面组件故障、节点宕机、网络分区等问题都可能导致服务中断。本文将系统讲解Kubernetes高可用架构的设计原则，并结合实际案例分享故障自愈的实践经验。

一、高可用K8s架构设计原则

Kubernetes高可用架构的核心目标是消除单点故障，确保集群在任何单个组件发生故障时仍能正常运行。高可用设计需要从以下几个维度考虑：

1. 控制平面高可用

控制平面包含四个核心组件，每个都需要部署多副本：

  • kube-apiserver：无状态服务，可水平扩展，通过负载均衡器分发请求
  • etcd：分布式键值存储，建议3或5节点组成Raft集群
  • kube-scheduler：通过Leader选举机制保证只有一个实例工作
  • kube-controller-manager：同样采用Leader选举机制
# etcd集群搭建示例（3节点）
# 节点1配置
cat > /etc/etcd/etcd.conf <

2. 负载均衡层设计

kube-apiserver前面需要部署负载均衡器，常用方案包括HAProxy+Keepalived或云厂商的SLB：

# HAProxy配置示例
cat > /etc/haproxy/haproxy.cfg < /etc/keepalived/keepalived.conf <

二、工作节点高可用与Pod调度策略

1. Pod反亲和性与拓扑分布

确保Pod分布在不同的节点和可用区，避免单节点故障导致服务不可用：

# Pod反亲和性配置
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: web-app
              topologyKey: kubernetes.io/hostname
          - weight: 50
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: web-app
              topologyKey: topology.kubernetes.io/zone
      containers:
      - name: web
        image: nginx:1.25
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

2. 使用PodDisruptionBudget保障最小可用副本

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-app
---
# 同时配置HPA实现自动伸缩
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

三、故障自愈机制设计

1. 节点故障自动处理

当节点发生故障时，K8s需要能够自动检测并恢复Pod。以下是关键配置：

# 节点问题检测器配置
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-problem-detector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: node-problem-detector
  template:
    metadata:
      labels:
        app: node-problem-detector
    spec:
      serviceAccountName: node-problem-detector
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: node-problem-detector
        image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.18
        command:
        - /bin/node-problem-detector
        - --logtostderr
        - --config.system-log-monitor=/config/kernel-monitor.json
        - --config.system-log-monitor=/config/docker-monitor.json
        - --config.custom-plugin-monitor=/config/network-problem-monitor.json
        volumeMounts:
        - name: log
          mountPath: /var/log
        - name: config
          mountPath: /config
          readOnly: true
      volumes:
      - name: log
        hostPath:
          path: /var/log
      - name: config
        configMap:
          name: node-problem-detector-config

2. 自动化故障恢复脚本

#!/bin/bash
# 节点自动恢复脚本，配合cron定时执行

# 检测NotReady节点并处理
NOT_READY_NODES=$(kubectl get nodes -o jsonpath='{range .items[?(@.status.conditions[?(@.type=="Ready")].status=="Unknown")]}{.metadata.name}{"
"}{end}')

for NODE in $NOT_READY_NODES; do
    echo "$(date): Node $NODE is NotReady"
    
    # 检查节点是否已经长时间NotReady（超过5分钟）
    NOT_READY_TIME=$(kubectl get node $NODE -o jsonpath='{.status.conditions[?(@.type=="Ready")].lastTransitionTime}')
    
    if [ ! -z "$NOT_READY_TIME" ]; then
        CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
        TIME_DIFF=$(( $(date -d "$CURRENT_TIME" +%s) - $(date -d "$NOT_READY_TIME" +%s) ))
        
        if [ $TIME_DIFF -gt 300 ]; then
            echo "$(date): Node $NODE has been NotReady for over 5 minutes, cordon and drain"
            kubectl cordon $NODE
            kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --force --timeout=60s
            
            # 如果是云环境，可以调用API重启节点
            # aws ec2 reboot-instances --instance-ids $INSTANCE_ID
        fi
    fi
done

# 检测并清理CrashLoopBackOff的Pod
CRASHING_PODS=$(kubectl get pods --all-namespaces -o jsonpath='{range .items[?(@.status.containerStatuses[0].state.waiting.reason=="CrashLoopBackOff")]}{.metadata.namespace}/{.metadata.name}{"
"}{end}')

for POD in $CRASHING_PODS;
do
    NAMESPACE=$(echo $POD | cut -d'/' -f1)
    PODNAME=$(echo $POD | cut -d'/' -f2)
    echo "$(date): Pod $POD is in CrashLoopBackOff, deleting to trigger restart"
    kubectl delete pod $PODNAME -n $NAMESPACE
done

四、监控与告警体系

高可用集群离不开完善的监控告警体系。以下是基于Prometheus的监控配置：

# Prometheus规则：K8s集群健康告警
groups:
- name: k8s-cluster-health
  rules:
  - alert: NodeNotReady
    expr: kube_node_status_condition{condition="Ready",status!="true"} == 1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Node {{ $labels.node }} is NotReady"
      description: "Node {{ $labels.node }} has been NotReady for more than 5 minutes."

  - alert: PodCrashLooping
    expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Pod {{ $labels.pod }} is crash looping"
      description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has restarted {{ $value }} times in the last hour."

  - alert: APIServerDown
    expr: up{job="kubernetes-apiservers"} == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "API Server is down"
      description: "API Server {{ $labels.instance }} has been down for more than 2 minutes."

  - alert: EtcdHighNumberOfFailedProposals
    expr: rate(etcd_server_proposals_failed_total[5m]) > 5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "etcd high number of failed proposals"

五、灾备与数据备份策略

#!/bin/bash
# etcd定期备份脚本
ETCD_ENDPOINTS="https://192.168.1.10:2379"
ETCD_CERT="/etc/etcd/ssl/server.crt"
ETCD_KEY="/etc/etcd/ssl/server.key"
ETCD_CA="/etc/etcd/ssl/ca.crt"
BACKUP_DIR="/data/etcd-backup"
RETENTION_DAYS=7

# 创建备份目录
mkdir -p $BACKUP_DIR

# 执行备份
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/etcd-snapshot-$TIMESTAMP.db"

ETCDCTL_API=3 etcdctl   --endpoints=$ETCD_ENDPOINTS   --cacert=$ETCD_CA   --cert=$ETCD_CERT   --key=$ETCD_KEY   snapshot save $BACKUP_FILE

# 验证备份
ETCDCTL_API=3 etcdctl snapshot status $BACKUP_FILE --write-table

# 清理旧备份
find $BACKUP_DIR -name "etcd-snapshot-*.db" -mtime +$RETENTION_DAYS -delete

# 同步到远程存储（可选）
# rsync -avz $BACKUP_FILE backup-server:/backup/etcd/

总结

Kubernetes高可用是一个系统工程，需要从架构设计、调度策略、故障自愈、监控告警和数据备份等多个层面综合考虑。控制平面的多副本部署是基础，Pod的反亲和性和PDB保障了工作负载的可用性，而节点问题检测器和自动化恢复脚本则实现了故障的自动发现和处理。配合完善的监控告警和灾备方案，可以构建出一个真正生产可用的K8s高可用集群。

在实际运维中，建议定期进行混沌工程演练（如使用Chaos Mesh），主动注入故障来验证集群的自愈能力。只有在故障真正发生前做好充分准备，才能在关键时刻保证业务的连续性。