Questions readers ask: Is an LLM router the same thing as an LLM gateway?

An LLM router and an LLM gateway are not the same thing. A router decides which model should handle a request, while a gateway governs who can make that request and under what rules. Most products bundle both, which makes their labels confusing. Knowing what each function does makes it easier to judge what a product provides and what the team still has to do.

Why "LLM router" and "LLM gateway" are genuinely different concepts, not just synonyms

A vendor calls its product a gateway, but its documentation spends pages explaining routing. A procurement conversation then treats the two terms as interchangeable, leaving an engineering lead unsure whether the proposed system covers model selection, access control, or both. Products often bundle these functions, so one product name can hide several different jobs in the architecture.

One question concerns which model should receive a request, given the task and the available providers. Another concerns whether that request is permitted and what controls must apply to it. These questions can have different answers within the same product. A system may make complex model choices but offer little governance, or enforce detailed access rules but use only a simple fallback chain.

What a router actually does — selecting the right model for each request

A router sits between an application and a pool of models, inspecting incoming requests and choosing where to send them. Not every request takes the same path. Its work has two parts, selection and recovery. Selection chooses the model and provider, while recovery determines what happens when that target fails, rate-limits, or times out. A routing system therefore needs to address both the suitability of its first choice and the availability of alternatives.

How does it make the initial choice? Rule-based routing matches request metadata, including task type, user tier, or headers, to a model through deterministic rules maintained by hand. Semantic or complexity-based routing embeds the prompt and compares it with labeled examples to assign a model tier. That allows routine requests to reach cheaper models while demanding prompts go to frontier models. Learned routing instead uses a trained classifier to predict which model will suffice. The RouteLLM framework from LMSYS and UC Berkeley includes transformer-based and matrix factorization approaches that use human preference data for that prediction.

Recovery introduces a different set of signals. Performance-adaptive routing uses live error rates and latency to shift traffic away from degraded providers, while failover routing activates a priority chain only after the primary target fails. Failover is the most widely deployed pattern, but it is not equivalent to learned or cost-optimized selection. Failure triggers it regardless of how hard the task is estimated to be. None of these routing decisions, by themselves, authorizes a request, enforces a budget, creates a compliance log, or transforms the prompt.

What a gateway actually does — governing every call, not just choosing its destination

A gateway wraps LLM calls in policies that apply regardless of which model the application or router chooses. Authentication validates credentials, and authorization determines which users, teams, or applications may reach which models. Rate limits cap request activity by caller or time window, while budget enforcement constrains spending before a bill arrives. These controls answer a different question from model selection, even when they operate on the same request.

The gateway also records and modifies interactions. Full request and response logging captures prompts and outputs for audit trails, cost attribution, and debugging, giving compliance teams the record they require. Prompt transformation can inject system context, strip PII, or apply content filters before a call reaches the model, with output transformation occurring afterward. Semantic caching reuses responses for semantically similar requests to avoid redundant calls. These functions can reduce costs, but their mechanism differs from choosing a cheaper model.

TLS termination, identity binding, request inspection, policy evaluation, and audit record emission sit outside the router’s selection job. In agentic workloads, gateways are also gaining support for MCP and agent-to-agent (A2A) protocol traffic, extending the control point beyond ordinary model calls. Routing can still be a component inside that gateway. It does not define the whole component, nor does its inclusion make the gateway merely a more powerful router. The router optimizes selection, while the gateway governs use.

The proxy underneath both — the transport layer most definitions skip

A proxy intercepts an application’s HTTP request and forwards it to a provider, handling connection pooling, TLS, and streaming passthrough. At the code level, this is request forwarding and response streaming, with decisions about which model is suitable or whether the caller is authorized left open. Transport alone provides neither judgment. A bare proxy can capture requests for cost attribution and latency measurement before routing or governance logic is added.

The proxy handles how traffic gets there, the router determines where it goes, and the gateway establishes who can send it under which rules. A gateway product can bundle all three without erasing the distinction between them. Consequently, a claim that a product proxies LLM traffic says little, on its own, about the sophistication of its routing or the completeness of its governance.

How the functions map to different owners, failure modes, and procurement conversations

Each function brings its own work to keep the system running, even when one team owns the whole deployment. Routing work concerns model suitability, quality, latency, and the cost of selection. Gateway work concerns access policy, spending limits, audit records, and attribution. Who gets paged depends on which responsibility has failed. Treating the bundled product as one undifferentiated control can obscure that distinction during both procurement and incident review.

The failures are asymmetric. A routing failure can send a difficult prompt to an unsuitable model or send routine work to an unnecessarily expensive one, producing quality degradation or unexpected cost spikes. Every call may still be authorized and logged and follow the policy. Conversely, a gateway failure can leave access ungoverned, audit trails missing, or spending unattributed even while the router consistently selects appropriate models. Success at either layer does not establish success at the other. The core function of each layer lacks the checks needed to spot every failure in the other.

That asymmetry exposes a procurement myth: “If I have a gateway, routing is solved. Five configured providers and a single round-robin policy do not establish task-aware model selection. Composable routing strategies, live signal inputs, and an evaluation feedback loop let routing do more, and a cost spike or quality regression may reveal that these capabilities are missing. The inverse assumption is equally consequential. A standalone router without gateway controls provides no audit trail, access control, or rate limiting, a gap acceptable in a single-team prototype but not in multi-team or compliance-sensitive production.

Why most products on the market bundle both, and what that means for reading vendor docs

Bundling is a rational response to production needs. Requests need transport, model selection, and policy enforcement, so products can provide them behind one interface rather than requiring each team to assemble separate components. LiteLLM describes its gateway as including cost tracking, budget controls, routing, and governance. Portkey’s official buyer’s guide lists unified routing and governance at scale as separate things the product can do.

Standalone routing still has a clear role. That makes a standalone router relevant when a team needs learned selection that its bundled gateway does not provide. Direct dispatch does not, however, establish that governance is covered. The architecture must establish where the required policies are enforced, independently of the router’s ability to reach a model.

A gateway claim should lead to documentation of authentication, rate limiting, audit logging, and budget enforcement alongside routing fallback. A router claim should prompt a check of whether the product assumes an upstream gateway and who must provide it. When a product claims both roles, its transport, selection, and governance features can be evaluated separately. The label is a starting point for checking coverage. This approach accommodates bundled products without treating bundling itself as a defect.

What neither a router nor a gateway answers — the grounding gap in LLM infrastructure

Is the information supplied to the model current and verifiable? A financial reporting assistant can select the right model, admit the right caller, and preserve a full audit trail while reasoning over a revenue definition superseded by a policy change months earlier. Neither routing nor governance detects or corrects that stale premise through its core function. Prompt transformation may inject context, but the ability to insert information does not establish its accuracy or freshness. Grounding therefore needs its own place in the infrastructure before model reasoning, especially for agents that query the live web, file reports, or conduct multi-step research. A real-time search and research API layer can supply current, cited source material, complementing the control plane.

A decision framework — which layer to build or buy first, and in what order

For teams deciding what to deploy first, the functional distinction provides a practical sequence. Start with gateway controls when multiple teams share LLM access, have to meet compliance requirements, or need cost attribution. Visibility comes before optimization in that setting. Authentication, audit logging, rate limits, and budget enforcement make it possible to govern the system whose model choices will later be tuned.

More capable routing is warranted when model cost becomes a meaningful budget line and the query distribution shows an opportunity for selection. At low volume, the gateway’s built-in round-robin or simple fallback may be sufficient. A learned or semantic router can help when there are enough requests for the savings to matter and different model tiers work better for different requests. The decision is therefore not whether sophisticated routing is available, but whether the application’s requests justify it.

Map each claimed capability to the proxy, router, or gateway function, and check whether documentation supports the per-team budget enforcement and audit logging implied by the word “gateway. Routing and API key management alone do not cover all the governance work a product needs to do. Choosing which model handles which query is a routing discussion, while deciding who may call models and what can be proved afterward is a gateway discussion.

Most production deployments need both discussions, but they also need grounding when the application has to reason over current, real-world information. Treat that as the third infrastructure layer, with requirements of its own beyond what the control plane already provides. Its timing follows the application’s information needs. The resulting sequence is to establish governance for shared production access, improve selection when the workload justifies it, and supply grounding when current evidence is required.