How to choose an LLM router
Start from the one need you can't trade away, then shortlist two candidates and test them against the checklist below. Most teams have one deciding need; the rest are preferences.
01
Cost and fallbacks
What do you pay on top of tokens, and what happens when a provider fails?
Fees sit in different places: on credit purchases, on traffic through your own keys, or nowhere because you run the layer yourself. Fallbacks matter as much as price, because a rate limit at the wrong moment costs more than a few percent.
Look for
- Where the fee is charged: on credits, on BYOK traffic, or none
- Fallback across models and across providers, set per request
- Budgets or spend limits per key or team
Start with: OpenRouter, Vercel AI Gateway, LiteLLM · OpenRouter vs Vercel AI Gateway →
02
Latency
How much time does the layer add, and where does it run?
Few vendors publish an overhead figure, and those that do measure it their way. The number that counts is yours: time to first token through the router minus a direct call, from the region your app runs in.
Look for
- Where its servers run relative to your app and your providers
- Caching for repeated requests
- Provider sorting by latency or throughput
Start with: OpenRouter, Cloudflare AI Gateway, Portkey AI Gateway
03
Compliance
Where do prompts go, who stores them, and can you prove it?
Read the retention terms for the exact product you'll call, not the company's general policy: zero data retention often covers some APIs and not others. If you can't get it in writing for your endpoint, keep traffic in your own network.
Look for
- Zero data retention that covers inference, enforceable per request or account-wide
- A list of which upstream providers retain or train on data
- Whether logging is off by default
Start with: OpenRouter, LiteLLM · OpenRouter vs LiteLLM →
04
Self-hosting
Must the layer run inside your own infrastructure?
Self-hosting gives you full control of keys, logs and network paths, and moves the work of running, scaling and patching it to your team. You still need an account with every provider you call.
Look for
- Licence of the code you'll run, and what sits behind an enterprise licence
- Virtual keys, budgets and an admin UI
- A hosted option if you later want to stop running it
Start with: LiteLLM, Portkey AI Gateway · LiteLLM vs Portkey →
05
Open-weight models
Do you run open-weight models, and do they need the live web?
If your models are open-weight and your answers need current facts, a router that runs web search inside the call saves you a separate search API and tool loop. The You.com Inference Router does this: declare web_search or web_extract, and the response lists the sources used. If you need closed frontier models too, pair it with a broader router.
Look for
- Which open-weight models the key can call (list them through the API)
- Search and page fetching run server-side, with sources returned
- Control over freshness, country and the domains searched
Start with: You.com Inference Router, OpenRouter · OpenRouter vs You.com router →
Checklist: test two candidates
Run each candidate through these with a trial key before you commit. Most take under an hour.
- List the models your key can call. Call the models endpoint with a trial key; check every model your workload needs is there.
- Measure added latency. Send the same 50 prompts direct and through the router from your app's region; compare time to first token at p50 and p95.
- Force a failure. Point a request at a model or provider that will fail and check the fallback takes over, and how long it takes.
- Price a real month. Take last month's token counts per model and price them on the vendor's pricing page, including any fee on credits or BYOK.
- Get the retention terms in writing. Find the page that covers the exact API you'll call. If it isn't listed, ask the vendor before sending production data.
- Check your SDK works unchanged. Switch only the base URL and key in your OpenAI or Anthropic SDK and run your test suite.
- Check the date on every claim. Pricing and terms change. Re-read the source page on the day you decide; every fact on this site links to its source and the date it was checked.