Kubernetes Pod手动扩缩容无法应对突发流量。Horizontal Pod Autoscaler(HPA)根据CPU、内存或自定义指标自动调整副本数,是保障服务稳定性和控制成本的核心机制。本文梳理HPA的工作原理、指标适配器安装和自定义指标配置的完整流程。
Kubernetes HPA工作原理:控制循环与扩缩容算法
HPA控制器以轮询方式(默认15秒间隔)从指标API获取Pod指标,计算期望副本数:
期望副本数 = ceil(当前副本数 * (当前指标值 / 目标指标值))
例如当前5个Pod平均CPU利用率为80%,目标设为50%,则期望副本数 = ceil(5 * 80/50) = 8。
HPA引入两个冷却参数防止副本数频繁抖动:--horizontal-pod-autoscaler-downscale-stabilization(默认5分钟,缩容时取窗口内最大值)和冷却延迟时间。
Metrics Server安装:提供CPU内存基础指标
HPA依赖Metrics Server采集各Pod的resource usage。安装Metrics Server:
# 应用官方YAML
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# 验证安装
kubectl get pods -n kube-system | grep metrics-server
kubectl top nodes
kubectl top pods
如果Pod使用率无数据,检查Metrics Server启动参数是否包含 --kubelet-insecure-tls(测试环境用)或正确配置了kubelet证书。
基于CPU利用率的HPA配置:标准扩缩容策略
定义一个Deployment并设置resources requests/limits,HPA依赖requests值计算CPU利用率百分比:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-server
spec:
replicas: 2
selector:
matchLabels:
app: api-server
template:
metadata:
labels:
app: api-server
spec:
containers:
- name: api
image: registry.example.com/api:v1.0
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-server-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api-server
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 70
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 60
behavior 字段(autoscaling/v2引入)提供细粒度控制:扩容时每15秒最多增加100%副本(快速响应流量增长),缩容时每60秒最多减少50%,且5分钟内取最大值避免抖动。
Prometheus Adapter安装:支持自定义指标HPA
CPU和内存指标无法覆盖所有场景。HTTP请求QPS、消息队列积压量、自定义业务指标需要Prometheus采集后通过Adapter暴露给HPA。
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prom-adapter prometheus-community/prometheus-adapter \
--namespace kube-system \
--set prometheus.url=http://prometheus-server.monitoring.svc:9090 \
--set prometheus.port=80 \
--set rules.default=false
创建自定义指标规则,将Prometheus查询映射为Kubernetes custom.metrics.k8s.io API:
apiVersion: v1
kind: ConfigMap
metadata:
name: prom-adapter-rules
namespace: kube-system
data:
http_requests_per_second: |
seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: 'sum(rate(https_requests_total{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
自定义指标HPA配置:基于QPS的弹性伸缩实战
使用自定义指标替换或补充CPU指标,实现更贴合业务特征的扩缩容:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-server-custom-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api-server
minReplicas: 3
maxReplicas: 50
metrics:
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: "1000"
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
上述配置表示:当Pod平均QPS超过1000时触发扩容,同时CPU超过60%也触发扩容(取两者较大值)。type: Pods 的指标来自每个Pod各自上报的Prometheus指标,AverageValue 表示每个Pod的处理量目标值。
HPA排障命令与常见问题处理
HPA不生效时,按顺序排查:
# 查看HPA状态和指标
kubectl describe hpa api-server-custom-hpa
# 查看HPA事件
kubectl get hpa api-server-custom-hpa -o yaml
# 检查自定义指标API是否可用
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests_per_second"
# 检查Metrics Server状态
kubectl get apiservice | grep metrics
常见问题:“the HPA was unable to compute the replica count” 表示指标获取失败——检查Pod是否设置了resources.requests(HPA计算Utilization依赖.requests值);“missing request” 表示Deployment未配置resources.requests,HPA无法计算CPU利用率百分比。
原创文章,作者:小编,如若转载,请注明出处:https://www.yunthe.com/kuberneteshpa-tan-xing-shen-suo-pei-zhi-xiang-jie-zi-ding/