Kubernetes容器编排实战：从Pod调度到生产级集群运维全攻略

Kubernetes生产部署的核心挑战

Kubernetes容器编排已经成为云原生基础设施的事实标准，但从”能跑通”到”生产可用”之间，隔着调度策略、资源管理、网络策略、监控告警等一系列工程问题。本文围绕生产环境中最常遇到的编排与运维场景，给出可直接操作的配置方案。

Pod调度：让工作负载精确落到期望节点

Kubernetes默认调度器基于预选+优选算法分配Pod，但生产环境需要更精确的控制。

节点亲和性：把数据库Pod调度到SSD节点

apiVersion: v1
kind: Pod
metadata:
  name: mysql-primary
  labels:
    app: mysql
    role: primary
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disk-type
            operator: In
            values: ["ssd", "nvme"]
          - key: node-role
            operator: In
            values: ["database"]
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: mysql
          topologyKey: kubernetes.io/hostname
  containers:
  - name: mysql
    image: mysql:8.0
    resources:
      requests:
        cpu: "4"
        memory: "16Gi"
      limits:
        cpu: "8"
        memory: "32Gi"
    volumeMounts:
    - name: data
      mountPath: /var/lib/mysql
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: mysql-pvc-ssd

污点与容忍：专用节点的隔离方案

# 给GPU节点打污点，只有声明容忍的Pod才能调度上去
kubectl taint nodes gpu-node-01 nvidia.com/gpu=true:NoSchedule

# GPU训练Pod的容忍配置
tolerations:
- key: "nvidia.com/gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

资源管理：Requests与Limits的正确姿势

资源配额设置不当是Kubernetes集群不稳定的首要原因。CPU是可压缩资源，内存是不可压缩资源——理解这个区别至关重要。

# 资源配额最佳实践
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:          # 默认limits
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:    # 默认requests
      cpu: "100m"
      memory: "128Mi"
    max:               # 最大允许值
      cpu: "4"
      memory: "8Gi"
    min:               # 最小允许值
      cpu: "50m"
      memory: "64Mi"
---
# 命名空间级别的配额限制
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
  namespace: production
spec:
  hard:
    requests.cpu: "32"
    requests.memory: "64Gi"
    limits.cpu: "64"
    limits.memory: "128Gi"
    pods: "100"
    services: "20"

CPU requests建议设为实际P50使用量的1.2倍，limits设为P99使用量的1.5倍。内存requests和limits建议保持一致，避免OOM Kill。

HPA自动扩缩容：从CPU到自定义指标

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"
  - type: External
    external:
      metric:
        name: rabbitmq_queue_messages
        selector:
          matchLabels:
            queue: "order-processing"
      target:
        type: AverageValue
        averageValue: "50"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60

scaleUp的stabilizationWindowSeconds设为30秒，快速响应流量增长；scaleDown设为300秒，避免流量波动导致的频繁缩容。

网络策略：零信任的Pod间访问控制

默认情况下Kubernetes集群内所有Pod可以互相通信，这在生产环境中是不可接受的：

# 默认拒绝所有入站流量
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---
# 仅允许frontend访问backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-frontend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

故障排查：kubectl诊断速查

Pod CrashLoopBackOff排查：

# 查看Pod事件
kubectl describe pod <pod-name> -n production

# 查看容器日志（上一个崩溃的容器）
kubectl logs <pod-name> -c <container> --previous -n production

# 查看容器退出码
# 0=正常退出 1=应用错误 137=OOMKill 139=SegFault 143=SIGTERM

节点NotReady排查：

# 检查kubelet状态
systemctl status kubelet

# 查看节点条件
kubectl describe node <node-name> | grep -A 5 Conditions

# 检查网络插件
kubectl get pods -n kube-system | grep -E "(calico|flannel|cilium)"

Pending状态Pod排查：

# 查看调度失败原因
kubectl get events --field-selector involvedObject.name=<pod-name>

# 检查资源是否足够
kubectl top nodes
kubectl describe resourcequota -n production

监控体系：Prometheus + Grafana生产部署

# Prometheus ServiceMonitor配置
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-server-monitor
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-server
  endpoints:
  - port: metrics
    interval: 15s
    path: /actuator/prometheus
  namespaceSelector:
    matchNames: [production]
---
# 告警规则
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-server-alerts
  namespace: production
spec:
  groups:
  - name: api-server
    rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total[5m]))
        > 0.01
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "API错误率超过1%"
        description: "当前5xx错误率: {{ $value | humanizePercentage }}"

Kubernetes容器编排的运维难度远超部署本身。调度策略、资源配额、网络隔离、监控告警这些环节缺一不可，任何一个短板都可能成为生产事故的导火索。