Skip to main content
Ragen calls language models in four roles — answering questions, rephrasing queries, summarising documents, and generating embeddings. This page is about which model fills each role. Model gateway is about how Ragen reaches it: a route table naming the upstream for each model id. Two things have to agree, and this is the mistake worth avoiding before you make it: the variables below name a model id, and that id has to exist in the route table. A name with no route is a name nothing can serve.

​
Model Roles

Ragen calls language models in four distinct roles. Each role has its own environment variable and its own default, and each default points to a cloud-hosted model. Set all four when you want full control over costs, privacy, or latency.
VECTOR_SIZE must match the output dimensions of your EMBEDDINGS_MODEL. If they disagree, Qdrant will reject every vector upsert and document ingest will silently fail. The default bge-multilingual-gemma2 model produces 3584-dimensional vectors; cohere-embed-multilingual-v3 produces 1024. Check your model’s specification rather than assuming.Changing EMBEDDINGS_MODEL or VECTOR_SIZE after you have ingested documents invalidates the entire Qdrant collection. You must re-index all documents from scratch if you switch embedding models.
A complete set of model roles:

​
Supported Providers

Every major provider is reachable. The model is named in the route table; the credentials come from the environment, per provider family.

OpenAI

GPT-4o, o3, and all OpenAI models.

Anthropic

Claude 3.5, Claude 4, and Haiku variants.

Google Vertex AI

Gemini models via a GCP service account.

AWS Bedrock

Claude, Cohere, and Titan models via Bedrock.
IAM permissions: aws-iam-policy.json.

Azure OpenAI

GPT models deployed to your Azure subscription.
The deployment name is the route’s model, not an environment variable.

Scaleway

Mistral, BGE embeddings, and other Scaleway Generative APIs.

Portkey (recommended gateway)

Spend dashboards, provider fallback, caching and retries in front of any upstream. Self-host it as one container, or use the hosted service.
See Model gateway.

OpenRouter

Hundreds of models behind one key. OpenRouter speaks OpenAI’s API, so it attaches as a connection, not as a provider of its own:
Give the route provider: openai-compatible and connection: openrouter. A bare OPENROUTER_API_KEY is read by nothing.

vLLM (local)

Self-hosted models via the vLLM OpenAI-compatible server.See the Local Models page for full setup instructions.

Ollama (local)

Local models running in Ollama on your own hardware. OpenAI-compatible, so it attaches as a connection:
See the Local Models page for full setup instructions.

​
Registering a Model

A model id you set in DEFAULT_MODEL, REPHRASE_MODEL or EMBEDDINGS_MODEL has to appear in the route table, which says which upstream serves it and under what name there. The shipped table is infra/llm-gateway/routes.yaml; point LLM_ROUTES_PATH at your own file to serve a different set.
infra/llm-gateway/routes.yaml
The left-hand key is Ragen’s name for the model — the one you put in an environment variable. model is the upstream’s name for it, which is often different: Bedrock prefixes the region and the vendor (eu.anthropic.claude-sonnet-5), and an Azure deployment name is whatever you called it. A route carries no credentials and no display name. Credentials come from the environment; display names and capability flags live in the model catalogue.
The chat model picker shows what your credentials can actually serve. It is the route table intersected with the credentials present in the app processes — so a route pointing at a provider you have no key for does not appear, rather than appearing and failing on the first click. A model you set in an environment variable but never routed will not appear either.Check the whole set before you rely on it, with one real call per model:

​
Additional Model Settings