Table of Contents
Preparing narration.
Five identical servers sitting behind one load balancer, each getting exactly one-fifth of the requests. So why is one of them pinned at 90% CPU while another one sits half idle?
Well, because an equal share of requests is not an equal share of work.
Adaptive load balancing is a routing strategy that picks a destination using live health signals like error rate, latency and utilization, instead of a fixed rotation. The balancer keeps scoring every backend while traffic flows, shifts weight toward the ones behaving well, and pulls weight away from the ones going bad.
That's the definition. But the definition is the boring part, so let's get into what the balancer is actually measuring, how fast it reacts, and why this suddenly matters a lot more in 2026 than it did five years ago.
What is a load balancer?
A load balancer is just a thing sitting in front of your servers, deciding which one gets the next request. That's it.
The simplest version is round robin. Request 1 goes to server A, request 2 to server B, request 3 to server C, then back to A again. It's a rotation, and it never once looks at what is actually happening inside those servers.
Static algorithms like this quietly assume two things: every request costs the same, and every server has the same capacity right now. Both assumptions survive about five minutes of real production traffic.
Think of a supermarket. Round robin is the sign saying "next customer to the next till, in order." Adaptive is a floor manager who watches which till is genuinely moving, spots the one stuck behind a price check, and sends people elsewhere.
Same queue, very different Saturday.
How does an adaptive load balancer decide where to send a request?
It collects signals, turns them into a score per route, and turns those scores into weights.
Three signals do most of the work:
- Error rate. Is this backend returning failures? Usually the heaviest signal, and usually time-decayed, so a spike from ten minutes ago stops dominating the decision.
- Latency. How slow is it right now, both against its peers and against its own recent baseline? A route that always takes 2 seconds is fine. A route that usually takes 200ms and is now taking 2 seconds is in trouble.
- Utilization. How much of its capacity is already committed, so that no single fast route gets hammered into becoming a slow one.
Those collapse into one number per route, and a higher number means a bigger share of the traffic.
And the weights are not recalculated per request, because that would drop real work onto the hot path. They get recalculated on a background loop, and each incoming request simply reads the numbers that were computed a moment ago.
Bifrost, the open-source AI gateway from Maxim, is the clearest published example of this that I have come across. Its adaptive load balancer scores routes on error penalty (50% of the score), a token-aware latency score (20%), and utilization (5%), plus a momentum bias that speeds up recovery once a bad route starts behaving again. Weights recalculate every 5 seconds, and route selection adds under 10 microseconds to the hot path, per its documentation.
The entire source code is on GitHub, so you can go read how the scoring is implemented rather than taking a feature page's word for it. Most load balancers describe their algorithm as "intelligent" and then stop talking.
The other half of the mechanism is state. A good adaptive balancer doesn't just have a dial, it has an opinion about what each route currently is: healthy, degraded, failed, or recovering.
And the recovering state is the one people forget. A route that failed does not get cut off forever, it gets a thin trickle of live traffic so the balancer can find out when it is better. Without that, your balancer is just a fancy circuit breaker that never closes again.
Adaptive load balancing vs round robin and least connections
Everyone puts these in a table. I would rather just tell you where each one stops working.
Round robin rotates blindly. Fine when every server and every request is genuinely identical, which is basically never.
Weighted round robin lets you say "server A is beefier, give it double." Better, but you set those weights by hand, based on what was true when you deployed. It has no idea what is true at 3am during a traffic spike.
Least connections picks whichever server has the fewest open connections. This one is genuinely dynamic, and it is a solid default. But an open connection is a rough proxy for load, since one connection doing heavy work counts exactly the same as one connection idling.
Least response time, usually implemented with an exponentially weighted moving average of latency, gets close to adaptive. It measures the thing you actually care about.
Adaptive load balancing is the version that stops relying on any single number. It combines errors, latency and capacity, keeps a health state per route, and has explicit behavior for pulling a route out and easing it back in.
Does the extra machinery pay off? Envoy's own benchmark for its Peak EWMA policy puts it at a 99.9% success rate under a 1-second timeout, against 99% for least-loaded and 95% for round robin (Envoy docs). That gap between adaptive and round robin is made entirely of user-visible failures.
Alright, so far this is a decades-old idea
And it genuinely is. Adaptive load balancing has been in networking gear and reverse proxies forever, and if you run a normal web app behind NGINX, least connections is probably good enough and you can stop reading here.
So why is the term suddenly everywhere again?
Because AI traffic breaks nearly every assumption the older algorithms were built on. That is where this stops being system-design trivia and starts being your on-call pager, so let's get into it.
Why LLM traffic makes static load balancing fall apart?
Four things go wrong at once.
Requests are wildly different sizes. A "summarize this sentence" call and a "read these 40 pages and reason about them" call hit the same endpoint, and one of them costs a hundred times more. When I built Scholarian, a deep-research pipeline over academic papers, it ended up fetching and ranking over 10,000 papers across 250+ search sessions. Some sessions were one cheap query. Some were a long chain of very expensive ones. A rotation cannot tell those apart, so it cheerfully fires the expensive one at the route that is already drowning.
The limits are not counted in requests. Model providers rate-limit you on requests per minute and tokens per minute, and it is usually the token ceiling you hit first. So a balancer counting requests is watching the wrong meter, and you find out about it through a 429 error in production.
The backends are not yours. You cannot SSH into OpenAI. There is no CPU graph, no memory reading, nothing except the latency and error rate you observe from outside. Observed behavior is the only signal you have, and observed behavior is exactly what adaptive balancing runs on.
Every API key is its own bottleneck. Rate limits are per key, so teams end up holding several keys per provider. Now you are not balancing across servers anymore, you are balancing across a grid of providers and keys, each with separate limits and separate health.
What adaptive load balancing looks like inside an AI gateway?
The gateways handling this properly split the decision in two.
The first level picks the provider and model for a request, based on live capacity and error rates across all of them. The second level picks which API key inside that provider actually gets used, weighted by how each individual key is performing.
That two-level shape matters because the failures are different at each level. A provider goes down for everybody at once. A single key just quietly hits its own token ceiling while its siblings are perfectly fine. One balancer trying to handle both would be making the wrong call half the time.
I went through the practical side of this in my post on Bifrost's enterprise features, where the adaptive routing sits right next to audit logs and guardrails. Worth a read if you want the version with an actual dashboard in front of you.
When you should not reach for adaptive load balancing
Now the honest part, because none of this is free.
You need real telemetry before you need adaptive routing. At a contract role earlier this year I built an internal tool that captured our AI product's logs and generated reports on latency and probable slowdowns, and the uncomfortable lesson was that most of the wins came from simply seeing the numbers. Half the routing problems people want an adaptive balancer to solve turn out to be one bad prompt template or one undersized instance, and a dashboard finds those faster than an algorithm hides them.
Low traffic means no signal. Scoring on error rate and latency needs enough requests per window to mean anything. At 5 requests a minute, an adaptive balancer is mostly reacting to noise, and reacting to noise is worse than not reacting at all.
It can paper over a capacity problem. If every route is degraded, adaptive balancing will smoothly and confidently send you to the least-bad option, forever, while the real answer was "add capacity" or "get off the free tier."
And it is one more moving part. More state, more tuning, one more thing to reason about at 3am. If round robin across two identical boxes is working for you, keep it.
So what's the final message?
If you run a plain web service on infrastructure you control, least connections is fine and adaptive load balancing is over-engineering.
If you are routing to model providers, it is not optional anymore. You are balancing across backends you cannot inspect, with limits measured in tokens, with per-key ceilings, and with failure modes that arrive as a slow degradation instead of a clean crash. A fixed rotation has no mechanism to even notice that. Use a gateway that scores routes on live behavior and moves the traffic for you, and go spend that attention on your product instead. And also, if what Bifrost is doing intrigued you, you can easily book a demo, and see how it fits in your organization.
That is my take, and your setup might look nothing like mine. If you have built this kind of routing yourself, or you have watched an adaptive balancer make a genuinely stupid decision, drop it in the comments, I want to hear it.
You can find me on X where I post about most of what I am building, and the rest of my writing lives at swapnoneel.site.