Kubernetes多环境部署的痛点
Kubernetes集群中同一应用需要部署到开发、测试、生产多个环境时,最直接的做法是为每个环境维护一套独立的YAML清单。这种方式在应用数量和集群规模增长后迅速失控——配置散落在多个目录,修改一个环境变量需要同步到十几份文件中。Helm Chart配合Values层叠策略能将多环境配置收敛到单一Chart中,通过层叠覆盖机制让不同环境共享基础配置、各自只声明差异项。
Values层叠机制的核心原理
Helm的Values层叠遵循”后加载覆盖先加载”的规则,覆盖顺序从低到高为:
1. Chart内的values.yaml(基础默认值)
2. 父Chart的values.yaml(子Chart场景)
3. -f参数指定的Values文件(从左到右依次覆盖)
4. –set命令行参数(最高优先级)
利用这个覆盖规则,可以把通用配置写在values.yaml,环境差异写在独立的override文件中:
my-app/
├── Chart.yaml
├── values.yaml # 基础配置
├── values-dev.yaml # 开发环境覆盖
├── values-staging.yaml # 测试环境覆盖
├── values-prod.yaml # 生产环境覆盖
├── templates/
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── hpa.yaml
│ └── ingress.yaml
└── .helmignore
基础Values配置设计
values.yaml存放所有环境共享的配置,只放最保守的默认值:
# values.yaml
replicaCount: 1
image:
repository: registry.example.com/my-app
pullPolicy: IfNotPresent
tag: ""
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
service:
type: ClusterIP
port: 8080
ingress:
enabled: false
className: ""
annotations: {}
hosts: []
tls: []
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 3
targetCPUUtilizationPercentage: 80
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /readyz
port: http
initialDelaySeconds: 5
periodSeconds: 10
configMap:
LOG_LEVEL: "info"
APP_ENV: "development"
环境覆盖Values的差异化配置
开发环境只需声明与基础配置不同的部分:
# values-dev.yaml
replicaCount: 1
image:
tag: "dev-latest"
pullPolicy: Always
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 256Mi
ingress:
enabled: true
className: nginx
hosts:
- host: my-app.dev.example.com
paths:
- path: /
pathType: Prefix
configMap:
LOG_LEVEL: "debug"
APP_ENV: "development"
生产环境覆盖资源规格、副本数、监控告警等关键参数:
# values-prod.yaml
replicaCount: 3
image:
tag: "v2.1.0"
pullPolicy: IfNotPresent
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rate-limit: "100"
hosts:
- host: my-app.example.com
paths:
- path: /
pathType: Prefix
tls:
- hosts:
- my-app.example.com
secretName: my-app-tls
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 15
readinessProbe:
initialDelaySeconds: 10
periodSeconds: 5
configMap:
LOG_LEVEL: "warn"
APP_ENV: "production"
Deployment模板引用Values
模板中使用.Values引用配置,同时要处理层叠覆盖时嵌套对象的合并问题:
# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "my-app.fullname" . }}
labels:
{{- include "my-app.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{- include "my-app.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
labels:
{{- include "my-app.selectorLabels" . | nindent 8 }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.port }}
protocol: TCP
envFrom:
- configMapRef:
name: {{ include "my-app.fullname" . }}-config
{{- with .Values.livenessProbe }}
livenessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.readinessProbe }}
readinessProbe:
{{- toYaml . | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
模板中checksum/config注解确保ConfigMap变更时Pod自动滚动更新。这是一个容易被忽略但生产环境必须处理的细节——ConfigMap内容变了但Pod不会自动重启,加上hash注解才能触发Deployment的RollingUpdate。
多环境部署命令与CI/CD集成
部署时通过-f参数叠加Values文件:
# 开发环境
helm upgrade --install my-app ./my-app \
-f ./my-app/values.yaml \
-f ./my-app/values-dev.yaml \
-n dev
# 生产环境
helm upgrade --install my-app ./my-app \
-f ./my-app/values.yaml \
-f ./my-app/values-prod.yaml \
-n prod
在GitLab CI或GitHub Actions中,环境名称由分支或标签决定:
# GitLab CI 配置示例
deploy:
stage: deploy
script:
- |
ENV="${CI_ENVIRONMENT_NAME}"
helm upgrade --install ${CI_PROJECT_NAME} ./chart \
-f ./chart/values.yaml \
-f ./chart/values-${ENV}.yaml \
--set image.tag=${CI_COMMIT_SHORT_SHA} \
-n ${ENV}
environment:
name: ${CI_COMMIT_BRANCH}
rules:
- if: $CI_COMMIT_BRANCH == "main"
environment: production
- if: $CI_COMMIT_BRANCH == "staging"
environment: staging
–set image.tag=${CI_COMMIT_SHORT_SHA}确保每次部署使用对应的Git提交镜像,覆盖Values文件中的tag字段,这是层叠策略在CI/CD中的典型应用。
嵌套对象覆盖的陷阱与解决
Helm Values层叠有一个关键行为:嵌套对象是整体覆盖而非深度合并。例如values-prod.yaml中只定义了resources.limits,合并后resources.requests会变成空值而非保留基础配置中的值。
解决方法是确保每个覆盖文件中的嵌套对象是完整的:
# 错误写法:只覆盖limits,requests会丢失
resources:
limits:
cpu: 2000m
memory: 2Gi
# 正确写法:完整声明整个resources对象
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 2000m
memory: 2Gi
对于深层嵌套场景,可以借助Helm的tpl函数或合并字典的方式实现深度合并:
# templates/_helpers.tpl
{{- define "my-app.mergedResources" -}}
{{- $base := .Values.resources -}}
{{- $override := dig "resources" (dict) .Values.environmentOverride -}}
{{- mergeOverwrite $base $override | toYaml -}}
{{- end -}}
mergeOverwrite执行深度合并,子字典中的键只覆盖同名键,不影响兄弟键。但要注意mergeOverwrite只在字典间生效,列表仍是整体替换。
Secrets的环境隔离
Secret对象不应写在Values文件中。每个环境的密钥应存储在外部密钥管理系统(如Vault、AWS Secrets Manager),通过External Secrets Operator同步到集群:
# templates/externalsecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: {{ include "my-app.fullname" . }}-secrets
spec:
refreshInterval: 1h
secretStoreRef:
name: {{ .Values.environment }}-vault
kind: SecretStore
target:
name: {{ include "my-app.fullname" . }}-secrets
data:
- secretKey: DB_PASSWORD
remoteRef:
key: secret/data/{{ .Values.environment }}/my-app
property: db_password
- secretKey: API_KEY
remoteRef:
key: secret/data/{{ .Values.environment }}/my-app
property: api_key
这种方式下,密钥的生命周期与Chart完全解耦。开发者只需在values文件中声明environment: production,ExternalSecret控制器自动从对应Vault路径拉取密钥。轮换密钥在Vault侧完成,集群中的Secret自动同步更新,无需重新部署应用。
原创文章,作者:小编,如若转载,请注明出处:https://www.yunthe.com/kubernetes-rong-qi-bian-pai-shi-zhan-helmchart-duo-huan/