A national-scale LLM serving platform that stays up under load

// 01 the problem

The platform had to serve large language models to government departments across the country, with a design target of 5,000 concurrent users and a commercial rollout behind it. The models ran on a small on-premise GPU cluster, not a hyperscaler: seven nodes, 96 GB of VRAM per GPU, no spare capacity to waste. Three things made it hard at once. Prompts were long (17,000–20,000 tokens of shared context) so prefill, not decode, was the bottleneck. A recurring GPU-level crash took a serving node down under load with no obvious trigger. And every model or routing change touched production directly.

// 02 architecture

Requests enter through LiteLLM (API keys, routing, usage), which forwards to one HAProxy pool per model. Pools balance with leastconn and a per-instance maxconn so a slow request cannot pile up behind a fast one. Each pool fronts vLLM instances pinned to GPU nodes. Prometheus, Grafana and a two-minute hang detector close the loop.

// 03 what I did and why

1. Moved to two-tier routing (LiteLLM → HAProxy pools → vLLM) and collapsed model configuration into one source of truth (+207/−188 lines in one change). One file now defines every pool.
2. Benchmarked before deciding. With guidellm and AIPerf I measured speculative decoding at +25–50% throughput and −40–60% time-to-first-token on the 35B MoE model, plus MoE kernel tuning that won 12 of 12 comparisons. Warm-up compilation left the serving path (13 in-flight compiles → 0); CUDA-graph capture memory fell from 1.87 to 0.22 GiB.
3. Root-caused the crash instead of restarting around it. Reproduced it with a load generator in 5.5 minutes, ruled out hardware, prefix caching and structured output, and isolated the condition: speculative decoding plus the FlashInfer backend on a hybrid model. Removed it from that pool, accepting a 20–25% throughput cost for stability.
4. Fixed the load balancer's reload path. HAProxy mounted its config as a file; Git updates replaced the inode so reloads read stale config. Mounting the directory fixed it; a verification script checks every pool before reload.
5. Added capacity without downtime. A new GPU node went from bare metal to live traffic in 53 minutes. Retiring an unused model freed disk on two nodes (94% → 61%, 92% → 45%).
6. Set up the next optimisation with a baseline. Prefix-cache hit rate measured at 28.4% and 24.7%; sticky routing by conversation is deployed and verified, pending client header adoption.

// 04 measured results