Linux服务器高可用集群故障自动切换实战：Keepalived + HAProxy配置指南

服务器单点故障为什么必须消灭

生产环境中，单台服务器宕机意味着业务全线中断。无论是硬件故障、内核崩溃还是OOM Killer误杀进程，单点部署的恢复时间通常在30分钟以上，而高可用集群的故障切换可以在1-3秒内完成。Keepalived + HAProxy是当前轻量级高可用方案中部署成本最低、运维复杂度最小的组合，适合中小规模业务场景。

Keepalived VRRP原理与集群拓扑

Keepalived基于VRRP（Virtual Router Redundancy Protocol）协议实现IP漂移。集群中两台服务器分别充当MASTER和BACKUP，共享一个虚拟IP（VIP）。MASTER正常时持有VIP，故障时BACKUP自动接管。

集群拓扑：

客户端 → VIP(192.168.1.100)
           ├→ MASTER(192.168.1.10) HAProxy
           └→ BACKUP(192.168.1.11) HAProxy
                    ├→ 后端节点1(192.168.1.21:8080)
                    ├→ 后端节点2(192.168.1.22:8080)
                    └→ 后端节点3(192.168.1.23:8080)

Keepalived配置详解

MASTER节点配置（/etc/keepalived/keepalived.conf）：

global_defs {
    router_id LVS_MASTER
    script_user root
    enable_script_security
}

vrrp_script chk_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    weight -20
    fall 3
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass K8s_Cluster_Secret
    }

    virtual_ipaddress {
        192.168.1.100/24
    }

    track_script {
        chk_haproxy
    }

    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault  "/etc/keepalived/notify.sh fault"
}

BACKUP节点将state改为BACKUP，priority改为90，其余配置相同。

健康检查脚本

#!/bin/bash
# /etc/keepalived/check_haproxy.sh
if ! killall -0 haproxy; then
    logger "HAProxy process not running, triggering failover"
    exit 1
fi

# 检查HAProxy是否能正常转发
if ! curl -sf http://127.0.0.1:8404/stats >/dev/null; then
    logger "HAProxy stats check failed"
    exit 1
fi

exit 0

健康检查间隔2秒，连续3次失败后降低优先级20，触发主备切换。恢复正常后连续2次成功才恢复优先级，避免抖动。

HAProxy负载均衡配置

# /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    maxconn 4096
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s
    retries 3

frontend http_in
    bind *:80
    default_backend web_servers

backend web_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server node1 192.168.1.21:8080 check inter 3s fall 3 rise 2
    server node2 192.168.1.22:8080 check inter 3s fall 3 rise 2
    server node3 192.168.1.23:8080 check inter 3s fall 3 rise 2

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats auth admin:Str0ngP@ss

故障切换通知机制

通知脚本在主备切换时发送告警：

#!/bin/bash
# /etc/keepalived/notify.sh
STATE=$1
HOSTNAME=$(hostname)
VIP=192.168.1.100

case $STATE in
    master)
        MSG="${HOSTNAME} became MASTER for VIP ${VIP}"
        ;;
    backup)
        MSG="${HOSTNAME} became BACKUP for VIP ${VIP}"
        ;;
    fault)
        MSG="${HOSTNAME} entered FAULT state for VIP ${VIP}"
        ;;
esac

# 发送企业微信告警
curl -s -X POST "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"msgtype\":\"text\",\"text\":{\"content\":\"${MSG}\"}}"

logger "$MSG"

脑裂问题诊断与防护

脑裂发生在MASTER和BACKUP同时持有VIP的场景，通常由网络分区引起。防护手段：

1. 增加仲裁节点：部署第三台服务器作为仲裁，MASTER必须获得仲裁节点确认才持有VIP。

2. 防火墙阻断重复VIP：在交换机层面配置ARP检查，同一MAC不允许出现在不同端口。

3. 自动检测脚本：

#!/bin/bash
# 脑裂检测，每分钟执行
VIP=192.168.1.100

# 如果我是BACKUP但持有VIP，说明脑裂
if grep -q "state BACKUP" /etc/keepalived/keepalived.conf; then
    if ip addr show eth0 | grep -q "$VIP"; then
        logger "SPLIT BRAIN detected! BACKUP holding VIP"
        ip addr del ${VIP}/24 dev eth0
        systemctl restart keepalived
    fi
fi

性能调优与内核参数

Keepalived的advert_int默认1秒，生产环境不要设低于0.5秒，避免网络抖动导致误切换。HAProxy的maxconn需要根据后端服务承载能力设置，推荐值为后端单节点最大并发数的80%。连接超时timeout server根据业务P99延迟设置，API服务建议10秒，长连接场景设为300秒。

系统层面需调整内核参数：

# /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

执行sysctl -p生效。这些参数确保VIP切换后ARP更新及时，高并发连接不被内核丢弃。

故障演练验证方案

部署完成后必须经过故障演练：

1. 停止MASTER的Keepalived进程，观察BACKUP接管时间
2. 停止MASTER的HAProxy进程，观察健康检查触发切换
3. 拔掉MASTER网线，模拟网络分区
4. 后端节点逐个下线，验证HAProxy自动剔除
5. 全部后端下线，验证HAProxy返回503而非超时

每种场景的切换时间应控制在3秒以内，客户端连接在切换后应自动重连成功。