What Is LLM Routing? A Plain-English Answer

llm router is the practice of letting software choose which model answers each request, instead of hard-coding one model in your code. A routing layer sits between your application and every model provider: it reads the request, grades it, and sends it to the model that fits. What that trip costs depends on the model — the live rate card for GLM-5.3 shows what a capable mid-tier charges per million tokens; this piece is the simpler version — what the thing is, why it exists, and whether you should care.

Start with the mess that created the category. A few years ago, “picking a model” meant picking one company. Today, models come from everywhere — and they come with wildly different prices, speeds, and failure habits. The cheap model that answers your support tickets in milliseconds is the same tool that will confidently invent an API you never shipped. The frontier model that writes your hardest code is the same tool that will bankrupt a high-volume chat feature on price. The moment you have more than one model that could answer a request, someone has to decide which one does — and that decision is exactly what LLM routing automates.

LLM routing, defined in one sentence

LLM routing is a decision layer: which model answers this request, right now, under these rules? The router is software that sits in front of your model calls. Your application sends every request to the same place — one endpoint, one key — and the router decides the destination for you.

That’s the entire category in one paragraph. Everything else is how the decision gets made and how much control you keep over it.

Why anyone needs it: the three problems one choice creates

Routing exists because three things are true at once, and no single model fixes all three.

There are too many models to pick from by hand. At the last count we verified, a single API key at OrcaRouter opens access to more than 200 models across every major provider — OpenAI, Anthropic, Google, Meta, Mistral, xAI, DeepSeek, Qwen, GLM, MiniMax, and more [OrcaRouter]. Nobody audits 200 models by hand for every request. The job is too big to do in a switch statement, and it gets bigger every quarter as new models ship.

The price spread is enormous. The difference between a small model and a frontier model is often a 10x-to-100x gap in cost per token. If you route everything to the smartest model, you overpay on the easy stuff — summaries, classification, routing-quality intros. If you route everything to the cheapest model, you under-deliver on the hard stuff. One fixed choice forces you to pick a single compromise.

Models fail, and fail differently. A model can be down for an outage, exhausted by rate limits, or simply time out on a slow day. When your one model is down, your whole feature is down. The failure mode is predictable — you just can’t avoid it by picking a “better” model, because every model fails sometimes.

How a router actually decides

A router is only as good as its decision process. The standard one works in three steps.

Step 1: grade the prompt. Before anything is sent anywhere, the router scores the request — how complex it is, what kind of task it looks like, how much is riding on a correct answer. This step needs to be fast, because it’s pure overhead on every single call; OrcaRouter’s adaptive routing grades each prompt in under a millisecond before choosing a destination [OrcaRouter].

Step 2: apply your rules. You set the guardrails: a hard cost ceiling, a latency ceiling, a “never use this model for legal output” rule, a “this customer must always get the frontier model” override. The router filters the model pool down to whatever clears your bar.

Step 3: pick the cheapest model that clears the bar. This is the sentence that makes the whole category make sense. The router doesn’t pick the best model, and it doesn’t pick the cheapest model — it picks the cheapest model that is good enough for this specific request. Easy request, cheap model. Hard request, expensive model. The same prompt-level judgment you’d apply manually, applied on every single call, automatically.

This adaptive mode — where every prompt is graded and then routed to the cheapest model that meets your standards — is the behavior OrcaRouter ships as its orcarouter/auto mode [OrcaRouter]. The three-step pattern is the idea; the implementation details are where products differ.

Who needs LLM routing

The honest answer is narrower than the marketing: you need routing the day you call more than one model. Not before.

If your product calls exactly one model for exactly one task, routing is an abstraction you don’t need yet — go ship the feature. The moment you add a second model, you have made a routing decision by hand: you hard-coded which one. Hard-coding works while your answer is right, and it breaks the day a model gets cheaper, slower, or discontinued.

The teams that hit routing first are usually the ones under pressure on three sides:

• Cost pressure — token bills that scaled faster than revenue. Routing sends easy traffic to cheap models and reserves expensive ones for the requests that need them.

• Reliability pressure — an outage that took a feature down, or rate limits that hit exactly when traffic spiked. Routing adds automatic failover, so a request that would have failed is retried on another model instead [OrcaRouter].

• Choice pressure — a new model ships every few weeks and nobody has time to re-evaluate the catalog. Routing makes the model an input to the decision, not a line of code.

There’s a simpler tell: if you’ve ever written code that tries one model and falls back to another when it errors, you already built a one-line router. Routing is that instinct, systematized.

What LLM routing is NOT

A term this simple gets sloppy the moment it gets popular. Three confusions come up constantly.

It is not load balancing. Load balancing spreads identical work across identical servers to use capacity evenly. Routing sends each request to a different kind of tool based on what the request needs. One is about distribution; the other is about judgment. A router may also balance load, but that’s a side effect, not the point.

It is not a cache. Caching stores a previous answer and replays it for identical requests. Routing still calls a model on every request — it just chooses which one. They’re complementary: cache the repeats, route the new stuff. But they solve different problems, and conflating them leads to measuring the wrong thing.

It is not just an API key swap. The most common misconception is that routing is a unified key in front of a bunch of models — a convenience wrapper. A key swap gets you access; it doesn’t get you a decision. Routing is the part that decides which model, when, under what rules. The unified access is table stakes; the decision is the product.

Common belief	What’s actually true
It’s load balancing across models	It’s per-request judgment about which model fits
It’s a cache that avoids calling models	It still calls a model every time — just the right one
It’s a single key to call many models	That’s unified access; the routing decision is the actual feature
It means “pick the best model”	It means “pick the cheapest model that clears the bar”

The takeaway

LLM routing is a small idea that pays off at scale: decide which model answers each request, instead of hard-coding one. It exists because there are too many models, the price spread between them is enormous, and every single one fails sometimes. A router grades the prompt in under a millisecond, applies your rules, and picks the cheapest model that clears the bar — with automatic failover when a model is down, and 0% markup on list prices so the economics stay yours [OrcaRouter]. You need it the day you call more than one model, and not before. Until then, the definition is the takeaway: routing isn’t about having more models. It’s about using the ones you have — differently, per request, at a price you can defend.

Sourcing note: all product facts — one API key for 200+ models, 0% markup, adaptive routing that grades each prompt in under 1ms, and automatic failover — are OrcaRouter’s own published claims, verified August 22, 2026.