混沌工程实战：Chaos Mesh注入网络延迟故障与系统韧性验证

小编 • • 网站运维

混沌工程通过主动注入故障来验证系统韧性，Chaos Mesh是Kubernetes生态中功能最完整的混沌工程平台。在微服务架构下，网络分区、延迟抖动、DNS解析失败等故障难以在测试环境自然触发，Chaos Mesh提供了声明式故障注入能力，能在不修改应用代码的情况下模拟真实故障场景。

Chaos Mesh架构与安装

Chaos Mesh基于Kubernetes CRD（Custom Resource Definition）实现，通过控制器模式管理故障注入的生命周期：

# Helm安装Chaos Mesh
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
    -n chaos-testing --create-namespace \
    --set chaosDaemon.runtime=containerd \
    --set dashboard.securityMode=false

# 验证安装
kubectl get pods -n chaos-testing
# chaos-controller-manager-xxx   1/1   Running
# chaos-daemon-xxx               1/1   Running
# chaos-dashboard-xxx            1/1   Running

# 确认CRD已注册
kubectl get crd | grep chaos
# networkchaos.chaos-mesh.org
# podchaos.chaos-mesh.org
# iochaos.chaos-mesh.org
# stresschaos.chaos-mesh.org
# dnschaos.chaos-mesh.org

Chaos Mesh的核心组件包括chaos-controller-manager（调度和CRD管理）、chaos-daemon（在每个节点执行故障注入）、chaos-dashboard（Web可视化界面）。故障注入通过Linux tc（traffic control）、iptables、ebpf等内核机制实现。

网络延迟故障注入

NetworkChaos是最常用的故障类型，模拟网络延迟、丢包、重复包、带宽限制等场景。以下实验模拟订单服务调用支付服务时50%的请求延迟2000ms：

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency-injection
  namespace: chaos-testing
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      "app.kubernetes.io/name": "payment-service"
  direction: to
  target:
    mode: all
    selector:
      namespaces:
        - production
      labelSelectors:
        "app.kubernetes.io/name": "order-service"
  delay:
    latency: "2000ms"
    correlation: "50"
    jitter: "500ms"
  duration: "5m"
  scheduler:
    cron: "@every 1h"

部署并观察效果：

kubectl apply -f payment-latency-injection.yaml

kubectl get networkchaos -n chaos-testing
# NAME                      AGE   STATUS
# payment-latency-injection 10s   Injected

# 从order-service容器内验证延迟
kubectl exec -it deploy/order-service -- sh -c \
  "time curl -o /dev/null -s http://payment-service:8080/api/health"
# 正常: real 0.05s
# 故障注入后: real 2.03s

# 在tc层面验证故障注入
kubectl exec -it deploy/payment-service -- tc qdisc show dev eth0
# qdisc netem 1: root refcnt 2 limit 1000 delay 2.0s 500ms 50%

网络丢包与分区故障模拟

模拟服务间30%丢包，验证重试机制和熔断器是否生效：

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-packet-loss
  namespace: chaos-testing
spec:
  action: loss
  mode: all
  selector:
    namespaces: [production]
    labelSelectors:
      app: payment-service
  direction: to
  target:
    mode: all
    selector:
      namespaces: [production]
      labelSelectors:
        app: order-service
  loss:
    loss: "30"
    correlation: "25"
  duration: "3m"

网络分区故障模拟更极端——完全隔离两个服务之间的通信：

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-network-partition
  namespace: chaos-testing
spec:
  action: partition
  mode: all
  selector:
    namespaces: [production]
    labelSelectors:
      app: payment-service
  direction: to
  target:
    mode: all
    selector:
      namespaces: [production]
      labelSelectors:
        app: gateway
  duration: "2m"
  scheduler:
    cron: "0 14 * * 1-5"

分区故障下，gateway完全无法访问payment-service。此时需要验证：网关的fallback降级逻辑是否生效、熔断器是否在阈值触发后open、消息队列是否接住了未能即时处理请求、监控告警是否在30秒内发出通知。

Pod级故障注入

PodChaos模拟容器级别的故障，包括Pod Kill、Container Kill、Pod OOM等：

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: payment-pod-kill
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [production]
    labelSelectors:
      app: payment-service
  scheduler:
    cron: "@every 10m"

---
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: payment-memory-stress
  namespace: chaos-testing
spec:
  mode: all
  selector:
    namespaces: [production]
    labelSelectors:
      app: payment-service
  stressors:
    memory:
      workers: 2
      size: "512MB"
  duration: "5m"

Pod Kill实验验证Kubernetes的自动恢复能力：HPA是否自动扩容新Pod、就绪探针是否阻止流量打到未就绪的Pod、session保持是否会话中断。

DNS故障注入

DNS解析故障在微服务环境中影响面广但难以复现。DNSChaos模拟DNS服务器返回错误或延迟：

apiVersion: chaos-mesh.org/v1alpha1
kind: DNSChaos
metadata:
  name: dns-random-error
  namespace: chaos-testing
spec:
  action: error
  mode: all
  selector:
    namespaces: [production]
    labelSelectors:
      app: order-service
  patterns:
    - "payment-service.production.svc.cluster.local"
    - "redis-cluster.production.svc.cluster.local"
  duration: "3m"

DNSChaos通过修改Pod的DNS配置，只影响目标Pod，不干扰集群其他服务。order-service在故障期间无法解析payment-service，需要在本地配置兜底IP或服务网格的DNS缓存。

编排式自动化实验

单次故障注入测试价值有限，生产级混沌工程需要编排式实验——按顺序执行多步骤故障场景。Chaos Mesh支持Workflow：

apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: payment-resilience-test
  namespace: chaos-testing
spec:
  entry: main
  templates:
    - name: main
      templateType: Serial
      children: [baseline-check, inject-latency, verify-slo, cleanup]

    - name: baseline-check
      templateType: Task
      task:
        container:
          image: busybox
          command: ["sh", "-c", "curl -sf http://gateway/api/health"]

    - name: inject-latency
      templateType: NetworkChaos
      networkChaos:
        action: delay
        mode: all
        selector:
          labelSelectors:
            app: payment-service
        delay:
          latency: "3000ms"
        duration: "5m"

    - name: verify-slo
      templateType: Task
      task:
        container:
          image: curlimages/curl
          command: ["sh", "-c"]
          args:
            - |
              FAIL=0
              for i in $(seq 1 60); do
                CODE=$(curl -o /dev/null -s -w "%{http_code}" http://gateway/api/order)
                if [ "$CODE" != "200" ] && [ "$CODE" != "503" ]; then
                  FAIL=1; break
                fi
                sleep 5
              done
              exit $FAIL

    - name: cleanup
      templateType: Task
      task:
        container:
          image: bitnami/kubectl
          command: ["kubectl", "delete", "networkchaos", "-n", "chaos-testing", "--all"]

Workflow按照baseline-check -> inject-latency -> verify-slo -> cleanup的顺序执行。verify-slo步骤在故障注入期间持续验证SLO（可用性99%+、错误率<1%），只有全部通过才判定实验成功。

实验结果分析与告警联动

Chaos Mesh原生集成Prometheus，实验期间的关键指标可通过Grafana对比分析：

# Chaos Mesh暴露的Prometheus指标
# chaos_mesh_experiments: 当前活跃的实验数
# chaos_mesh_experiments_phase: 实验阶段(Running/Finished/Paused)

# Grafana面板叠加业务指标与故障时间轴
# Panel 1: HTTP请求成功率（叠加NetworkChaos时间轴）
# Panel 2: P99响应延迟（叠加PodKill时间轴）

# 自动化判定脚本
python3 << 'EOF'
import requests

PROMQL = {
    "success_rate": 'sum(rate(http_requests_total{status=~"2.."}[1m])) / sum(rate(http_requests_total[1m]))',
    "p99_latency": 'histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[1m])) by (le))',
}

def query(promql):
    r = requests.get("http://prometheus:9090/api/v1/query", params={"query": promql})
    return float(r.json()["data"]["result"][0]["value"][1])

experiment_sr = query(PROMQL["success_rate"])
experiment_p99 = query(PROMQL["p99_latency"])
assert experiment_sr >= 0.99, f"成功率低于SLO: {experiment_sr}"
assert experiment_p99 <= 1.0, f"P99超SLO: {experiment_p99}s"
print(f"实验期: 成功率={experiment_sr:.4f}, P99={experiment_p99:.3f}s - PASS")
EOF

混沌工程不是制造混乱，而是在受控条件下发现系统脆弱点。每次实验的产出应该是一个明确的结论：系统在X故障下表现如何、SLO是否达成、需要改进什么。将实验纳入CI/CD流水线，在每次发版前自动执行回归实验，才能真正建立系统韧性的信心。

原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/hun-dun-gong-cheng-shi-zhan-chaosmesh-zhu-ru-wang-luo-yan/

赞 (0)
小编小编
0
上一篇 8小时前
下一篇 8小时前

相关推荐