Skip to main content
When you point every model role at a local endpoint, nothing your users type and none of your document content leaves your network. “Fully offline” is a configuration you arrive at intentionally, not something you inherit: you must set all four model roles, disable outbound reranking, set DOCLING_STRICT=1, and check the what still reaches outward table below.

​
How the Routing Works

Each application process calls model providers itself, using a route table that names the upstream for every model id. Pointing Ragen at a local server is therefore a route plus two environment variables — no proxy in the middle, and no application code change.
The route table is read by all three processes, so the credentials and base URLs have to be present in all three — not in one container. The embedding model is the one exception worth planning around: it determines the shape of the Qdrant index. Changing EMBEDDINGS_MODEL requires updating VECTOR_SIZE to match the new model’s output dimensions and re-indexing every document you have already ingested.

​
Pick a Server

Both vLLM and Ollama expose an OpenAI-compatible API, which is all a route needs: provider: openai-compatible plus a connection name.
vLLM is the right choice when more than one person will be using Ragen concurrently. Its continuous batching lets many requests run in parallel on the same GPU.Rough VRAM arithmetic: weights need about 2 bytes per parameter at bf16, or ~0.6 bytes at 4-bit. An 8B model is roughly 16 GB unquantised and 6 GB quantised. Budget extra headroom for the KV cache — RAG prompts carry retrieved chunks and can easily reach 4–8k tokens, so the cache grows larger than a simple chatbot workload would suggest.

​
Set All Four Model Roles

The most common mistake is pointing only DEFAULT_MODEL at a local server, watching chat work, and assuming the installation is isolated. It is not. Ragen calls a model in four separate places, each with its own environment variable and its own cloud-hosted default. Leaving REPHRASE_MODEL unset is the most common miss. It sends the user’s question and the full conversation history to a cloud model on every single turn — which is exactly the traffic an isolated deployment exists to prevent.

​
Wiring vLLM

Start the vLLM server with a served model name that matches what you will put in the route table:
Add a route for it, and name the connection:
infra/llm-gateway/routes.yaml
The connection name becomes the environment variables, upper-cased:
To run vLLM as a container alongside the rest of the stack, add it to docker-compose.override.yml on the same network:
docker-compose.override.yml
If vLLM runs on a separate GPU host, replace http://vllm:8000 with that host’s address — the app processes only need network access to it. After editing the route table, confirm the model answers a real call:

​
Wiring Ollama

Pull the model you want to use:
Add a route for it. Ollama’s OpenAI-compatible API lives under /v1, and the model name is Ollama’s own tag:
infra/llm-gateway/routes.yaml
If Ollama runs on the Docker host rather than inside the compose network, use http://host.docker.internal:11434 on Docker Desktop. On Linux, use the host’s IP address on the Docker bridge network.
Point the four model role variables at your local model name:

​
Setting Up Local Embeddings

Set VECTOR_SIZE to match your embedding model’s output dimensions before you upload the first document. Qdrant creates the collection with a fixed vector size, and there is no way to change it afterwards short of deleting the collection and re-indexing everything. A mismatch between VECTOR_SIZE and the model’s actual output causes Qdrant to reject every upsert — and the failure looks like a broken ingest, not a configuration mistake.
Both vLLM and Ollama expose a /v1/embeddings endpoint. Give the embedding model its own route — a separate model id, usually served by a separate process, because an embedding server and a chat server rarely want the same GPU:
infra/llm-gateway/routes.yaml
A connection name becomes environment variables by upper-casing it and replacing anything that is not a letter or digit with _, so vllm-embed reads LLM_VLLM_EMBED_BASE_URL. Set the matching environment variables. VECTOR_SIZE must equal the model’s output dimensionality — not an assumption, the actual number from the model card:
Common embedding model dimensions for reference:

​
What Still Reaches Outward

Setting local model targets closes most outbound traffic, but several other paths remain. Work through this table to confirm your installation is fully isolated.

​
Things That Break Differently on Local Models

Switching to a local model can change behaviour in ways that are not obvious errors. Know what to look for before you commit to a model.
Multi-query expansion asks the model to return a JSON object that matches a schema. vLLM constrains generation to the schema natively. Ollama supports response_format, but coverage varies by model and version.A failure here does not surface as an error — Ragen falls back to searching the user’s raw question without expansion, so a follow-up like “and what about the second one?” is retrieved literally with no conversation history behind it. Watch the logs: grep for Rephrase-and-expand failed after switching models.
MCP connectors and built-in tools require a model that supports function calling. Many open models do; some do not. When a model does not support function calling, the tools simply never fire — there is no error, the features are just absent.
A text-only local model cannot process an image attached to a message. Set MULTIMODAL_TEXT_ONLY_MODELS to the model’s ID and MULTIMODAL_FALLBACK_MODEL to a vision-capable model, and Ragen will swap models automatically when a request includes an image.
RAG prompts are long — the question, the conversation history, and several retrieved document chunks can easily reach 4–8k tokens. A model with a short context window will either reject the request or silently truncate the earliest part of it. Truncation reads as “the answer ignored my document” rather than a context length error. Check your model’s context window against a realistic prompt before deploying.

​
Verifying Your Isolated Install

After completing the configuration, run these checks to confirm every path is local:
Then ask one question in the UI and ingest one document, and watch the model server’s log. A fully local install shows:
  • A request for the rephrase (before retrieval)
  • A request for the answer (after retrieval)
  • A batch of embedding requests for the ingested document
If the rephrase request never arrives at the local server, REPHRASE_MODEL is still pointing at a cloud model.