LLM routing terms explained

A service that receives your model request and decides where it goes: which model, which provider serving that model, or both. A hosted router usually also bills you for the inference, so one key and one balance reach many models.

OpenRouter and the You.com Inference Router are hosted routers.

See also: LLM gateway, Model selection routing

A layer in front of provider accounts you already hold. It adds control (keys per team, budgets, logs, caching, rate limits, retries) but you keep paying each provider. Some gateways are software you run yourself; others are hosted.

LiteLLM and Portkey can be self-hosted; Cloudflare and Vercel run hosted gateways.

See also: LLM router, Model host

A company that runs model weights on its own hardware and sells inference. A router sends requests to hosts; a host is where the tokens are actually generated.

See also: LLM router

What happens when a call fails: the router or gateway retries with the next model or provider on a list you set, so a provider outage or rate limit doesn't become your outage. Each failed attempt still adds time to that request.

See also: Load balancing

Spreading requests for the same model across several deployments or keys, by weight or by health, so no single one hits its rate limit and traffic keeps flowing when one slows down.

See also: Fallback

Choosing a model per request from what the request is: a short question to a cheap fast model, a hard coding task to a stronger one. Done by rules you write or by a classifier the router runs.

See also: Provider routing

Choosing which provider serves a model when several host the same one, by price, throughput, latency or data policy.

See also: Model selection routing, Zero data retention (ZDR)

The time a router or gateway adds on top of a direct call to the provider. Measure it as the difference in time to first token, at p50 and p95, from where your app runs.

Using your own provider API key through a hosted router or gateway, so the provider bills you directly. Some services charge a fee on BYOK traffic; others don't.

A commitment that prompts and responses are not stored after the request is served. Check which products and endpoints it covers: it is often limited to some APIs, plans or providers.

A model whose trained weights are published, so many hosts can serve it and you can run it yourself. Its licence still sets what you may do with it.

Giving a model live search results or page contents so it answers from current sources and can cite them, instead of only from its training data.

The You.com Inference Router runs web_search and web_extract tools server-side and returns the sources in web_results.