大模型API集成的架构选型
Spring Boot应用集成大模型API时，最直接的方案是使用Spring AI或OpenFeign封装HTTP调用。但在生产环境高并发场景下，裸调API会遇到几个问题：上游模型推理延迟波动大（P99可达30秒+）、Token流式传输保持长连接、单用户请求成本高导致资源耗尽风险。微服务架构下需要从连接管理、限流熔断、异步编排三个维度构建治理体系。
OpenFeign客户端配置：超时与重试
Spring Boot调用大模型API，OpenFeign的超时配置必须区别于普通REST接口。大模型推理延迟远高于常规接口，默认超时设置会导致大量超时失败：
// FeignClient定义
@FeignClient(
name = "llm-service",
url = "${llm.api.base-url}",
configuration = LLMFeignConfig.class
)
public interface LLMFeignClient {
@PostMapping("/v1/chat/completions")
ChatCompletionResponse chatCompletion(
@RequestBody ChatCompletionRequest request
);
@PostMapping(value = "/v1/chat/completions",
consumes = MediaType.APPLICATION_JSON_VALUE,
produces = MediaType.TEXT_EVENT_STREAM_VALUE)
Flux<ServerSentEvent<String>> chatCompletionStream(
@RequestBody ChatCompletionRequest request
);
}
// Feign配置类
@Configuration
public class LLMFeignConfig {
@Bean
public Request.Options requestOptions() {
return new Request.Options(
10, TimeUnit.SECONDS, // 连接超时
120, TimeUnit.SECONDS, // 读超时（推理可能很慢）
true // 跟随重定向
);
}
@Bean
public Retryer retryer() {
// 最多重试2次，初始间隔500ms，最大间隔2s
return new Retryer.Default(500, 2000, 2);
}
}
Resilience4j限流与熔断配置
大模型API调用失败率高、延迟大，必须配置熔断器防止级联故障。限流器保护上游API不被突发流量打垮：
// application.yml
resilience4j:
circuitbreaker:
instances:
llmService:
sliding-window-type: COUNT_BASED
sliding-window-size: 20
failure-rate-threshold: 50
slow-call-rate-threshold: 80
slow-call-duration-threshold: 30s
wait-duration-in-open-state: 30s
permitted-number-of-calls-in-half-open-state: 5
ratelimiter:
instances:
llmService:
limit-for-period: 100
limit-refresh-period: 1s
timeout-duration: 5s
bulkhead:
instances:
llmService:
max-concurrent-calls: 50
max-wait-duration: 10s
关键参数说明：slow-call-duration-threshold设为30秒是因为大模型推理的正常延迟就在这个量级。failure-rate-threshold 50%意味着20次请求中10次失败才触发熔断。bulkhead限制并发调用50个，防止连接池耗尽。
Sentinel动态限流：按Token配额管控
大模型API按Token计费，传统QPS限流无法控制成本。需要基于Token消耗量限流。Sentinel支持自定义热点参数限流：
// Token配额限流规则
@PostConstruct
public void initFlowRules() {
List<FlowRule> rules = new ArrayList<>();
// 按用户维度的Token配额限流
FlowRule userTokenRule = new FlowRule()
.setResource("llm-token-usage")
.setGrade(RuleConstant.FLOW_GRADE_QPS)
.setCount(100000) // 每用户每日10万Token上限
.setLimitApp("default")
.setControlBehavior(RuleConstant.CONTROL_BEHAVIOR_WARM_UP)
.setWarmUpSec(60);
rules.add(userTokenRule);
FlowRuleManager.loadRules(rules);
}
// 在调用层埋点
@SentinelResource(
value = "llm-chat",
blockHandler = "chatBlockHandler",
fallback = "chatFallback"
)
public ChatCompletionResponse chat(ChatRequest request) {
// 计算预估Token数
int estimatedTokens = estimateTokenCount(request.getPrompt());
try (Entry entry = SphU.entry("llm-token-usage",
1, estimatedTokens)) {
return llmFeignClient.chatCompletion(
convertRequest(request)
);
}
}
// 限流降级处理
public ChatCompletionResponse chatBlockHandler(
ChatRequest request, BlockException ex) {
return new ChatCompletionResponse(
"当前请求量过大，请稍后重试",
HttpStatus.TOO_MANY_REQUESTS
);
}
WebFlux异步流式响应处理
SSE流式传输在Servlet容器中会占用线程池，Tomcat默认200线程很快耗尽。WebFlux基于Netty的EventLoop模型，用少量线程处理大量并发SSE连接：
@RestController
@RequestMapping("/api/chat")
public class ChatController {
private final LLMService llmService;
@PostMapping(produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent<String>> streamChat(
@RequestBody ChatRequest request
) {
return llmService.streamChat(request)
.map(token -> ServerSentEvent
.<String>builder()
.data(token)
.build())
.concatWith(Flux.just(
ServerSentEvent
.<String>builder()
.data("[DONE]")
.build()
));
}
}
// Service层
@Service
public class LLMService {
private final WebClient webClient;
public Flux<String> streamChat(ChatRequest request) {
return webClient.post()
.uri("/v1/chat/completions")
.bodyValue(convertToAPIRequest(request))
.retrieve()
.bodyToFlux(String.class)
.filter(data -> !"[DONE]".equals(data))
.map(this::extractToken)
.onErrorResume(WebClientRequestException.class, e -> {
return Flux.just("[连接异常，请重试]");
})
.timeout(Duration.ofSeconds(120));
}
}
WebFlux方案下，1000个并发SSE连接只需要4-8个EventLoop线程，而Servlet方案需要1000个线程。内存占用差距巨大。
请求级缓存与语义去重
相同Prompt重复调用大模型API是典型的资源浪费。引入语义缓存，对相似请求返回缓存结果：
@Service
public class SemanticCacheService {
private final RedisTemplate<String, String> redisTemplate;
// 对Prompt做hash，缓存完整响应
public Optional<String> getCachedResponse(String prompt) {
String cacheKey = "llm:cache:" + DigestUtils.md5Hex(prompt);
String cached = redisTemplate.opsForValue().get(cacheKey);
return Optional.ofNullable(cached);
}
public void cacheResponse(String prompt, String response) {
String cacheKey = "llm:cache:" + DigestUtils.md5Hex(prompt);
// 缓存1小时，Token响应不会频繁变化
redisTemplate.opsForValue().set(
cacheKey, response, 1, TimeUnit.HOURS
);
}
}
语义缓存命中时直接返回，省去一次大模型API调用，响应延迟从秒级降到毫秒级。对于FAQ类场景，缓存命中率可达40-60%。
可观测性：调用链追踪与Token计量
大模型API调用需要细粒度的可观测性。在Micrometer中注册自定义指标，追踪每次调用的Token消耗、延迟和成本：
@Service
public class LLMMetricsService {
private final Counter tokenCounter;
private final Timer responseTimer;
private final MeterRegistry registry;
public LLMMetricsService(MeterRegistry registry) {
this.registry = registry;
this.tokenCounter = Counter.builder("llm.tokens.consumed")
.tag("model", "default")
.register(registry);
this.responseTimer = Timer.builder("llm.response.time")
.tag("endpoint", "chat-completions")
.register(registry);
}
public void recordCall(int tokens, long latencyMs) {
tokenCounter.increment(tokens);
responseTimer.record(latencyMs, TimeUnit.MILLISECONDS);
}
}
在Grafana看板中同时展示QPS、P50/P99延迟、Token消耗速率、熔断器状态，配合告警规则在Token消耗异常或熔断器频繁Open时及时通知。
原创文章，作者：小编，如若转载，请注明出处：https://www.yunthe.com/springboot-ji-cheng-da-mo-xing-api-gao-bing-fa-chang-jing/
