> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ragen.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Gateway — How Ragen Reaches Model Providers

> Ragen calls model providers directly through a route table. Attach Portkey, vLLM, Ollama or any OpenAI-compatible endpoint per model.

Ragen has to turn "answer this question with `gpt-5.4`" into an HTTP call to somebody's API. It makes that call itself, from the application processes, using a **route table** that names which upstream serves each model id.

<Note>
  **There is no gateway mode to choose.** Ragen used to route every call through a LiteLLM proxy, selected by an `LLM_GATEWAY` variable. That path and that variable are gone — Portkey, vLLM, Ollama, LiteLLM and anything else speaking OpenAI's API are attached *per model*, as a route with `provider: openai-compatible`. See [Attaching an external gateway](#attaching-an-external-gateway).
</Note>

<Warning>
  Because Ragen calls providers itself, the provider credentials must be present in the **web, api and worker** processes — not only in one container. If you are upgrading a deployment that relied on the proxy, supply them to all three. Check before deploying, with a real call per model:

  ```bash theme={null}
  npm run gateway:preflight -- --probe
  ```
</Warning>

<Tip>
  You do not need a gateway to run Ragen. You want one if you want what a gateway is actually for — spend dashboards, provider fallback, caching, retries across vendors. Ragen does not build those, deliberately.
</Tip>

## Calling providers directly

Ragen reads a **route table** naming which upstream serves each model id, and takes credentials from the environment.

```yaml infra/llm-gateway/routes.yaml theme={null}
version: 1
routes:
  gpt-5.4:
    provider: azure
    model: gpt-5.4
  gemini-2.5-flash:
    provider: vertex
    model: gemini-2.5-flash
  claude-sonnet-5:
    provider: bedrock
    model: eu.anthropic.claude-sonnet-5
```

Point `LLM_ROUTES_PATH` at your own file to serve a different set. The shipped table is Ragen's own installation — a default, not a law.

A route carries no credentials and no display name. Credentials come from the environment; display names and capability flags are a separate concern.

### Credentials per provider

| `provider`          | Environment                                                                                          |
| ------------------- | ---------------------------------------------------------------------------------------------------- |
| `azure`             | `AZURE_API_KEY`, `AZURE_API_BASE`, optionally `AZURE_API_VERSION`                                    |
| `vertex`            | `VERTEX_PROJECT`, `VERTEX_LOCATION`, and optionally `VERTEX_CREDENTIALS`                             |
| `bedrock`           | `AWS_BEDROCK_REGION` — credentials come from the AWS default chain, so an instance role or SSO works |
| `openai`            | `OPENAI_API_KEY`, optionally `OPENAI_BASE_URL`                                                       |
| `openai-compatible` | See [Attaching a different gateway](#attaching-a-different-gateway)                                  |

<Note>
  **Vertex may need no credential variable at all.** `VERTEX_CREDENTIALS` accepts either the service-account JSON itself or a path to it — both work. Leave it unset on a workload that has its own identity (Cloud Run, GKE Workload Identity, GCE) and Google's application default credentials apply, with no key to rotate or leak. This is the one provider where "no credentials configured" is routinely correct.

  A route's own `location` beats `VERTEX_LOCATION`, which matters because some models are served from the `global` endpoint only and 404 in a region.
</Note>

## Attaching an external gateway

Ragen deliberately does not build spend dashboards, provider fallback, cross-vendor retries or response caching. Those are what a gateway is for, and anything speaking OpenAI's `/v1/chat/completions` and `/v1/embeddings` can serve some or all of your models.

<Tip>
  **Portkey is the gateway we recommend.** It is open source, Node and TypeScript like Ragen itself, runs as a single container you can host, and routes on request headers — which is exactly what per-model routing needs. Nothing below is Portkey-specific plumbing in Ragen: it is the same `openai-compatible` seam every other upstream uses, which is why recommending one costs you nothing if you later prefer another.
</Tip>

### Portkey

Run it yourself — one container, no account:

```bash theme={null}
docker run -d --name portkey -p 8787:8787 portkeyai/gateway
```

Point a route at it, and name the connection:

```yaml infra/llm-gateway/routes.yaml theme={null}
version: 1
routes:
  gpt-4o-mini:
    provider: openai-compatible
    connection: portkey
    model: gpt-4o-mini
  text-embedding-3-small:
    provider: openai-compatible
    connection: portkey
    model: text-embedding-3-small
```

```bash .env.local theme={null}
LLM_PORTKEY_BASE_URL=http://localhost:8787/v1
LLM_PORTKEY_API_KEY=<the upstream provider key Portkey should forward>
LLM_PORTKEY_HEADERS={"x-portkey-provider":"openai"}
```

`LLM_PORTKEY_API_KEY` is the **upstream** key — your OpenAI or Anthropic key — because the self-hosted gateway forwards it rather than holding credentials of its own. `x-portkey-provider` tells Portkey which upstream that key belongs to.

Chat and embeddings both route this way; there is nothing extra to configure for the embedding side.

#### Using Portkey's hosted service

With a saved Portkey config — which is where its fallbacks, retries and caching are defined — name the config instead of the provider:

```bash .env.local theme={null}
LLM_PORTKEY_BASE_URL=https://api.portkey.ai/v1
LLM_PORTKEY_API_KEY=<your Portkey API key>
LLM_PORTKEY_HEADERS={"x-portkey-config":"pc-ragen-xxxx"}
```

Here the API key is Portkey's own, and the provider credentials live in the Portkey config rather than in Ragen.

<Note>
  Header names are Portkey's, not Ragen's. Check their documentation for the version you run — `LLM_PORTKEY_HEADERS` passes through whatever JSON object you give it, so a header change on their side needs no change here.
</Note>

#### Check it before you rely on it

```bash theme={null}
npm run gateway:preflight -- --probe
```

One real call per configured model. A gateway that is reachable but misrouted answers this and nothing else until a user asks a question.

### Anything else OpenAI-compatible

[vLLM](https://docs.vllm.ai), [Ollama](https://ollama.com), Text Generation Inference, a hosted API, or LiteLLM itself — a proxy Ragen no longer has a mode for is still a perfectly good `openai-compatible` upstream. The pattern is identical; only the connection name changes.

```yaml theme={null}
routes:
  my-model:
    provider: openai-compatible
    connection: ollama
    model: llama3.1
```

The connection name becomes the environment variables, upper-cased:

```bash theme={null}
LLM_OLLAMA_BASE_URL=http://localhost:11434/v1
LLM_OLLAMA_API_KEY=unused
LLM_OLLAMA_HEADERS={"x-example":"1"}   # optional
```

That is the whole integration. A new upstream is a route and two variables — no code change.

### Per model, not per deployment

`connection` is a property of the route, so different models can come from different places at once: a local vLLM for the cheap ones, a hosted API for the hard ones, a gateway in front of a third. Ragen does not care that they differ.

## Before you rely on it

Misconfiguration does **not** fail at boot. It fails at the first model call, as a 5xx, because that is the first moment anything checks whether a credential works.

```bash theme={null}
npm run gateway:preflight -- --probe
```

`--probe` makes one real call per configured model. It exists because "configured" and "works" turned out to be different questions — a provider can have every variable set and still fail to authenticate, and a role-based deployment (Bedrock's default credential chain, Vertex's application default credentials) has no variable to inspect in the first place.

## If a provider stops working

There is no mode to switch back to, so the fix is per model rather than global: point the affected route at something else and leave every other route alone. Which shape the replacement takes depends on what you are pointing at, and the two are not interchangeable:

| Replacement                                                                                                       | Route                                                                                                                                                                                             |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Another provider family, or the same one in another region — Bedrock instead of Azure, Vertex in another location | Its **native** provider (`bedrock`, `vertex`, `azure`, `openai`, `anthropic`), with that family's own credentials in the environment. A `location` on the route overrides the deployment-wide one |
| An upstream that speaks OpenAI's API — a gateway you run, vLLM, Ollama, a hosted aggregator                       | `provider: openai-compatible` with a `connection` name, whose base URL and key are two environment variables derived from that name                                                               |

Only the second is "two variables and done"; the first needs whatever credentials that provider family requires, which is why [Credentials per provider](#credentials-per-provider) is the table to check before assuming a swap is free.

Either way it is the same mechanism as [attaching a gateway](#attaching-an-external-gateway), which is why there is no separate rollback procedure: routing is configuration, and it always was.