AI traffic management: Load balancing vs model routing

AI traffic looks like an API call, but it behaves nothing like traditional API traffic. In this episode of Pop Goes the Stack, F5's Lori MacVittie, Joel Moses, and Scott Calvet unpack why classic load balancing assumptions break down for inference and agentic workloads, and what “model routing” needs to become if we’re serious about performance, cost, and reliability. The core distinction is simple: traditional load balancing mostly optimizes distribution and availability under the assumption that requests are broadly interchangeable. Model routing has to inspect intent. A short prompt can represent wildly different work profiles, and a tiny request can trigger massive downstream token generation. Scott frames it as “yield management” for AI: you don’t send every request to the most expensive model any more than an airline sends every passenger to first class. From there, they get practical about the variables AI introduces. Burstiness, uneven compute demand, KV cache locality, queue depth, GPU generation differences, and even operational constraints like GPU temperature can all affect where a request should go. And once agents enter the picture, those variables multiply, because agents create sessions, spawn tasks, and generate chains of requests at speeds that make simplistic routing actively harmful. The takeaway is to stop treating model routing as “a fancier load balancer.” It’s traffic management with semantics and governance. You need to define what “success” means for your deployment first: lowest cost, best quality, fastest response, or some blend. Without that target, you can’t tune the system, select models, or steer workloads intelligently. Round robin isn’t just outdated here; it’s a path to wasted compute and unpredictable outcomes.