Skip to main content
Ragen has to turn “answer this question with gpt-5.4” into an HTTP call to somebody’s API. It makes that call itself, from the application processes, using a route table that names which upstream serves each model id.
There is no gateway mode to choose. Ragen used to route every call through a LiteLLM proxy, selected by an LLM_GATEWAY variable. That path and that variable are gone — Portkey, vLLM, Ollama, LiteLLM and anything else speaking OpenAI’s API are attached per model, as a route with provider: openai-compatible. See Attaching an external gateway.
Because Ragen calls providers itself, the provider credentials must be present in the web, api and worker processes — not only in one container. If you are upgrading a deployment that relied on the proxy, supply them to all three. Check before deploying, with a real call per model:
You do not need a gateway to run Ragen. You want one if you want what a gateway is actually for — spend dashboards, provider fallback, caching, retries across vendors. Ragen does not build those, deliberately.

​
Calling providers directly

Ragen reads a route table naming which upstream serves each model id, and takes credentials from the environment.
infra/llm-gateway/routes.yaml
Point LLM_ROUTES_PATH at your own file to serve a different set. The shipped table is Ragen’s own installation — a default, not a law. A route carries no credentials and no display name. Credentials come from the environment; display names and capability flags are a separate concern.

​
Credentials per provider

Vertex may need no credential variable at all. VERTEX_CREDENTIALS accepts either the service-account JSON itself or a path to it — both work. Leave it unset on a workload that has its own identity (Cloud Run, GKE Workload Identity, GCE) and Google’s application default credentials apply, with no key to rotate or leak. This is the one provider where “no credentials configured” is routinely correct.A route’s own location beats VERTEX_LOCATION, which matters because some models are served from the global endpoint only and 404 in a region.

​
Attaching an external gateway

Ragen deliberately does not build spend dashboards, provider fallback, cross-vendor retries or response caching. Those are what a gateway is for, and anything speaking OpenAI’s /v1/chat/completions and /v1/embeddings can serve some or all of your models.
Portkey is the gateway we recommend. It is open source, Node and TypeScript like Ragen itself, runs as a single container you can host, and routes on request headers — which is exactly what per-model routing needs. Nothing below is Portkey-specific plumbing in Ragen: it is the same openai-compatible seam every other upstream uses, which is why recommending one costs you nothing if you later prefer another.

​
Portkey

Run it yourself — one container, no account:
Point a route at it, and name the connection:
infra/llm-gateway/routes.yaml
.env.local
LLM_PORTKEY_API_KEY is the upstream key — your OpenAI or Anthropic key — because the self-hosted gateway forwards it rather than holding credentials of its own. x-portkey-provider tells Portkey which upstream that key belongs to. Chat and embeddings both route this way; there is nothing extra to configure for the embedding side.

​
Using Portkey’s hosted service

With a saved Portkey config — which is where its fallbacks, retries and caching are defined — name the config instead of the provider:
.env.local
Here the API key is Portkey’s own, and the provider credentials live in the Portkey config rather than in Ragen.
Header names are Portkey’s, not Ragen’s. Check their documentation for the version you run — LLM_PORTKEY_HEADERS passes through whatever JSON object you give it, so a header change on their side needs no change here.

​
Check it before you rely on it

One real call per configured model. A gateway that is reachable but misrouted answers this and nothing else until a user asks a question.

​
Anything else OpenAI-compatible

vLLM, Ollama, Text Generation Inference, a hosted API, or LiteLLM itself — a proxy Ragen no longer has a mode for is still a perfectly good openai-compatible upstream. The pattern is identical; only the connection name changes.
The connection name becomes the environment variables, upper-cased:
That is the whole integration. A new upstream is a route and two variables — no code change.

​
Per model, not per deployment

connection is a property of the route, so different models can come from different places at once: a local vLLM for the cheap ones, a hosted API for the hard ones, a gateway in front of a third. Ragen does not care that they differ.

​
Before you rely on it

Misconfiguration does not fail at boot. It fails at the first model call, as a 5xx, because that is the first moment anything checks whether a credential works.
--probe makes one real call per configured model. It exists because “configured” and “works” turned out to be different questions — a provider can have every variable set and still fail to authenticate, and a role-based deployment (Bedrock’s default credential chain, Vertex’s application default credentials) has no variable to inspect in the first place.

​
If a provider stops working

There is no mode to switch back to, so the fix is per model rather than global: point the affected route at something else and leave every other route alone. Which shape the replacement takes depends on what you are pointing at, and the two are not interchangeable: Only the second is “two variables and done”; the first needs whatever credentials that provider family requires, which is why Credentials per provider is the table to check before assuming a swap is free. Either way it is the same mechanism as attaching a gateway, which is why there is no separate rollback procedure: routing is configuration, and it always was.