混沌工程通过主动注入故障验证系统韧性，是SRE稳定性工程的重要实践。Chaos Mesh作为Kubernetes原生的混沌实验平台，支持Pod级别、网络级别、IO级别的故障注入，配合稳态验证实现自动化的弹性测试。本文给出Chaos Mesh部署、实验配置和CI集成的完整方案。
Chaos Mesh架构与故障注入类型
Chaos Mesh基于Kubernetes CRD实现，通过Chaos Controller管理实验生命周期。支持的故障类型覆盖以下维度：
- PodChaos：Pod杀死、容器终止
- NetworkChaos：网络延迟、丢包、分区、带宽限制
- StressChaos：CPU/内存压力注入
- IOChaos：文件读写延迟、错误注入
- DNSChaos：DNS解析失败或返回错误结果
- TimeChaos：系统时钟偏移
安装Chaos Mesh使用Helm：
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update
helm install chaos-mesh chaos-mesh/chaos-mesh \
-n chaos-testing --create-namespace \
--set chaosDashboard.security=false \
--set chaosDaemon.runtime=containerd
kubectl get pods -n chaos-testing
Pod级故障注入实验配置
PodChaos模拟容器崩溃和Pod驱逐场景。以下实验每次随机杀死目标Deployment的一个Pod：
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: pod-kill-experiment
namespace: chaos-testing
spec:
action: pod-kill
mode: one
selector:
namespaces:
- production
labelSelectors:
"app.kubernetes.io/name": "order-service"
scheduler:
cron: "@every 5m"
duration: "60s"
StressChaos对容器注入CPU和内存压力，测试资源限制下的服务降级行为：
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
name: cpu-stress-test
namespace: chaos-testing
spec:
mode: all
selector:
namespaces:
- production
labelSelectors:
app: payment-service
stressors:
cpu:
workers: 4
load: 80
duration: "120s"
scheduler:
cron: "0 2 * * *"
网络延迟与分区故障模拟
网络故障是分布式系统最常见的异常来源。注入网络延迟（模拟跨可用区调用延迟）：
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: network-delay-experiment
namespace: chaos-testing
spec:
action: delay
mode: all
selector:
namespaces:
- production
labelSelectors:
app: api-gateway
delay:
latency: "200ms"
correlation: "50"
jitter: "50ms"
direction: to
target:
selector:
namespaces:
- production
labelSelectors:
app: user-service
mode: all
duration: "300s"
模拟网络分区（隔离指定Pod的所有网络通信）：
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: network-partition
namespace: chaos-testing
spec:
action: partition
mode: all
selector:
namespaces:
- production
labelSelectors:
app: inventory-service
direction: both
target:
selector:
namespaces:
- production
labelSelectors:
app: order-service
mode: all
duration: "60s"
网络分区实验验证订单服务在库存服务不可达时的降级策略——是返回缓存数据、触发熔断还是直接报错。
稳态验证与自动化CI集成
混沌工程的核心不是注入故障本身，而是验证注入后系统是否保持稳态。Chaos Mesh 2.x的Workflow机制支持自定义检查步骤：
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
name: resilience-test-workflow
namespace: chaos-testing
spec:
entry: main
templates:
- name: main
templateType: Serial
children:
- check-baseline
- inject-network-delay
- verify-steady-state
- cleanup
- name: check-baseline
templateType: Task
task:
container:
image: curlimages/curl:latest
command:
- /bin/sh
- -c
- |
SUCCESS_RATE=$(curl -s http://prometheus:9090/api/v1/query \
--data-urlencode 'query=rate(http_requests_total{status!~"5.."}[1m])/rate(http_requests_total[1m])' \
| jq -r '.data.result[0].value[1]')
if [ "$(echo "$SUCCESS_RATE > 0.999" | bc)" != "1" ]; then
echo "Baseline check failed: success rate $SUCCESS_RATE"
exit 1
fi
- name: inject-network-delay
templateType: NetworkChaos
networkChaos:
action: delay
mode: all
selector:
namespaces: ["production"]
labelSelectors:
app: api-gateway
delay:
latency: "500ms"
duration: "300s"
- name: verify-steady-state
templateType: Task
task:
container:
image: curlimages/curl:latest
command:
- /bin/sh
- -c
- |
sleep 30
SUCCESS_RATE=$(curl -s http://prometheus:9090/api/v1/query \
--data-urlencode 'query=rate(http_requests_total{status!~"5.."}[1m])/rate(http_requests_total[1m])' \
| jq -r '.data.result[0].value[1]')
if [ "$(echo "$SUCCESS_RATE > 0.99" | bc)" != "1" ]; then
echo "Steady state violated: success rate dropped to $SUCCESS_RATE"
exit 1
fi
echo "Steady state maintained under fault injection"
Workflow串行执行四个步骤：基线检查、故障注入、稳态验证、资源清理。任一步骤失败即中止实验并告警。将Workflow嵌入CI流水线，每次服务发版前自动运行：
# GitLab CI集成
chaos_test:
stage: test
image: bitnami/kubectl:latest
script:
- kubectl apply -f chaos-workflow.yaml -n chaos-testing
- |
while true; do
STATUS=$(kubectl get workflow resilience-test-workflow \
-n chaos-testing -o jsonpath='{.status.conditions[?(@.type=="Ready")].reason}')
if [ "$STATUS" = "Completed" ]; then break; fi
if [ "$STATUS" = "Failed" ]; then
echo "Chaos experiment failed"
exit 1
fi
sleep 10
done
only:
- main
自动化混沌测试从手动实验走向CI流水线，是系统从”被动救火”到”主动发现”的关键一步。实际落地时建议从非生产环境开始，逐步扩展实验范围和故障类型，配合告警阈值调优，避免混沌实验本身引发二次故障。
原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/hun-dun-gong-cheng-shi-zhan-chaosmesh-gu-zhang-zhu-ru-yu-xi/
