Spring Boot微服务对接万亿参数大模型的API网关设计：限流、缓存与熔断实战

大模型API网关的特殊挑战

万亿参数大模型（如Kimi K3的2.8万亿参数）的推理成本远高于传统API。一次128K上下文的请求消耗的GPU算力等价于数十次普通API调用。当微服务架构中的多个业务服务同时调用大模型API时，如果没有网关层的流量控制，轻则推理服务过载导致全局限流，重则GPU显存溢出服务宕机。Spring Cloud Gateway作为微服务API网关，需要针对大模型API的特殊性做三层防护：请求级别的令牌桶限流、模型级别的并发连接控制、以及服务级别的熔断降级。

网关架构设计

# application.yml 路由配置
spring:
  cloud:
    gateway:
      routes:
      - id: llm-vllm-kimi-k3
        uri: http://vllm-service:8000
        predicates:
        - Path=/api/v1/chat/**
        filters:
        - name: TokenBucketRateLimiter
          args:
            tokens-per-second: 5
            bucket-capacity: 20
        - name: CircuitBreaker
          args:
            name: llm-cb
            fallbackUri: forward:/fallback/llm
        metadata:
          model: kimi-k3
          max-context: 131072

Token感知的令牌桶限流器

大模型API的限流不能简单按请求次数。一个128K上下文的请求和一个1K上下文的请求对GPU的消耗差距超过100倍。需要实现Token感知的限流策略：不同服务分配不同的令牌桶配额（user-service 50 tokens/s, content-service 30 tokens/s），从请求体Content-Length估算Token消耗，超出配额返回429状态码并附带retry-after头。

@Component
public class TokenAwareRateLimiterFilter implements GlobalFilter, Ordered {

    private final Map<String, RateLimiter> limiters = new ConcurrentHashMap<>();
    private final Map<String, Integer> serviceQuotas = Map.of(
        "user-service", 50,
        "content-service", 30,
        "search-service", 20
    );

    @Override
    public Mono<Void> filter(ServerWebExchange exchange,
                           GatewayFilterChain chain) {
        String serviceId = extractServiceId(exchange);
        int quota = serviceQuotas.getOrDefault(serviceId, 10);
        RateLimiter limiter = limiters.computeIfAbsent(
            serviceId, k -> new TokenBucketLimiter(quota, quota * 2)
        );
        int estimatedTokens = estimateTokenCost(exchange);
        if (limiter.tryConsume(estimatedTokens)) {
            return chain.filter(exchange);
        }
        exchange.getResponse().setStatusCode(HttpStatus.TOO_MANY_REQUESTS);
        return exchange.getResponse().writeWith(
            Mono.just(exchange.getResponse().bufferFactory()
                .wrap("{\"error\":\"rate_limit_exceeded\"}".getBytes())));
    }

    private int estimateTokenCost(ServerWebExchange exchange) {
        int len = exchange.getRequest().getHeaders().getContentLength();
        return Math.max(1, (int)(len * 0.3));
    }

    @Override
    public int getOrder() { return -100; }
}

模型推理结果缓存层

大模型推理延迟在数百毫秒到数秒之间，相同或相似的请求应该命中缓存。使用Caffeine构建语义缓存：maximumSize 10000条，expireAfterWrite 1小时，recordStats开启统计。缓存Key由模型名加消息内容Hash加参数构成，temperature=0的确定性输出缓存价值最高。语义Hash策略：去掉标点和空格后的内容做精确匹配，避免相似问题重复推理。

@Component
public class InferenceCacheFilter implements GlobalFilter, Ordered {

    private final Cache<String, CachedResponse> cache = Caffeine.newBuilder()
        .maximumSize(10_000)
        .expireAfterWrite(Duration.ofHours(1))
        .recordStats()
        .build();

    @Override
    public Mono<Void> filter(ServerWebExchange exchange,
                           GatewayFilterChain chain) {
        String cacheKey = buildSemanticCacheKey(exchange);
        CachedResponse cached = cache.getIfPresent(cacheKey);
        if (cached != null && !cached.isExpired()) {
            return writeCachedResponse(exchange, cached);
        }
        return chain.filter(exchange).then(Mono.fromRunnable(() -> {
            cacheResponseIfNeeded(exchange);
        }));
    }

    @Override
    public int getOrder() { return -50; }
}

熔断降级：GPU不可用时快速切换轻量模型

当vLLM服务过载或宕机时，网关需要快速降级到更小的模型而不是返回错误。Resilience4j熔断器配置：failureRateThreshold 50%, slowCallRateThreshold 60%, slowCallDurationThreshold 30秒, waitDurationInOpenState 30秒, slidingWindowSize 20次。降级控制器将model字段从kimi-k3替换为qwen3-8b并降低max_tokens到2048防止轻量模型超时，对调用方返回model=kimi-k3-degraded保持透明。

@RestController
@RequestMapping("/fallback")
public class LLMFallbackController {

    private final WebClient webClient;

    public LLMFallbackController(WebClient.Builder builder) {
        this.webClient = builder
            .baseUrl("http://vllm-small:8000/v1").build();
    }

    @PostMapping("/llm")
    public Mono<String> fallback(ServerWebExchange exchange) {
        String originalBody = exchange.getAttribute("cachedRequestBody");
        String fallbackBody = originalBody
            .replace("kimi-k3", "qwen3-8b");
        return webClient.post()
            .uri("/chat/completions")
            .contentType(MediaType.APPLICATION_JSON)
            .bodyValue(fallbackBody)
            .retrieve()
            .bodyToMono(String.class)
            .map(body -> body.replace("qwen3-8b", "kimi-k3-degraded"))
            .onErrorResume(e -> Mono.just("{\"error\":\"service_degraded\"}"));
    }
}

请求队列与优先级调度

当GPU并发数达到上限时，后续请求进入优先级队列等待。PriorityBlockingQueue按业务优先级排序：realtime-chat优先级10, code-assist优先级8, content-gen优先级5, batch-analysis优先级2。Semaphore控制GPU并发槽位，每50ms轮询队列处理。超时请求直接返回错误。

监控与告警指标

网关层暴露自定义Prometheus指标：gateway.llm.requests按model标签计数，gateway.llm.latency记录推理端到端延迟分布，gateway.llm.queue.size实时监控队列深度。关键告警规则：队列深度超过100触发告警，熔断器状态切换到OPEN触发P0告警，降级请求占比超过30%触发P1告警。这些指标对大模型服务的SLA保障至关重要，需要接入Grafana仪表盘实时监控。