Skip to main content
When you point every model role at a local endpoint, nothing your users type and none of your document content leaves your network. “Fully offline” is a configuration you arrive at intentionally, not something you inherit: you must set all four model roles, disable outbound reranking, set DOCLING_STRICT=1, and check the what still reaches outward table below. If you would rather use EU-hosted models than your own GPUs, see 100% EU Stack.

​
How the Routing Works

Each application process calls model providers itself, using a route table that names the upstream for every model id. Pointing Ragen at a local server is therefore a route plus two environment variables — no proxy in the middle, and no application code change.
The route table is read by all three processes, so the credentials and base URLs have to be present in all three — not in one container. The embedding model is the one exception worth planning around: it determines the shape of the Qdrant index. Changing EMBEDDINGS_MODEL requires updating VECTOR_SIZE to match the new model’s output dimensions and re-indexing every document you have already ingested.

​
Pick a Server

Both vLLM and Ollama expose an OpenAI-compatible API, which is all a route needs: provider: openai-compatible plus a connection name.
vLLM is the right choice when more than one person will be using Ragen concurrently. Its continuous batching lets many requests run in parallel on the same GPU.Rough VRAM arithmetic: weights need about 2 bytes per parameter at bf16, or ~0.6 bytes at 4-bit. An 8B model is roughly 16 GB unquantised and 6 GB quantised. Budget extra headroom for the KV cache — RAG prompts carry retrieved chunks and can easily reach 4–8k tokens, so the cache grows larger than a simple chatbot workload would suggest.

​
Set All Four Model Roles

The most common mistake is pointing only DEFAULT_MODEL at a local server, watching chat work, and assuming the installation is isolated. It is not. Ragen calls a model in four separate places, each with its own environment variable and its own cloud-hosted default. Leaving REPHRASE_MODEL unset is the most common miss. It sends the user’s question and the full conversation history to a cloud model on every single turn — which is exactly the traffic an isolated deployment exists to prevent.

​
Wiring vLLM

Start the vLLM server with a served model name that matches what you will put in the route table:
Add a route for it, and name the connection:
infra/llm-gateway/routes.yaml
The connection name becomes the environment variables, upper-cased:
To run vLLM as a container alongside the rest of the stack, add it to docker-compose.override.yml on the same network:
docker-compose.override.yml
If vLLM runs on a separate GPU host, replace http://vllm:8000 with that host’s address — the app processes only need network access to it. After editing the route table, confirm the model answers a real call:

​
Wiring Ollama

Pull the model you want to use:
Add a route for it. Ollama’s OpenAI-compatible API lives under /v1, and the model name is Ollama’s own tag:
infra/llm-gateway/routes.yaml
The picker lists only models the built-in catalogue knows. To offer a local model there under its own name, give its route display and capabilities: [chat]. See Registering a model.
If Ollama runs on the Docker host rather than inside the compose network, use http://host.docker.internal:11434 on Docker Desktop. On Linux, use the host’s IP address on the Docker bridge network.
Point the four model role variables at your local model name:

​
Setting Up Local Embeddings

Set VECTOR_SIZE to match your embedding model’s output dimensions before you upload the first document. Qdrant creates the collection with a fixed vector size, and there is no way to change it afterwards short of deleting the collection and re-indexing everything. A mismatch between VECTOR_SIZE and the model’s actual output causes Qdrant to reject every upsert — and the failure looks like a broken ingest, not a configuration mistake.
Both vLLM and Ollama expose a /v1/embeddings endpoint. Give the embedding model its own route — a separate model id, usually served by a separate process, because an embedding server and a chat server rarely want the same GPU:
infra/llm-gateway/routes.yaml
A connection name becomes environment variables by upper-casing it and replacing anything that is not a letter or digit with _, so vllm-embed reads LLM_VLLM_EMBED_BASE_URL. Set the matching environment variables. VECTOR_SIZE must equal the model’s output dimensionality — not an assumption, the actual number from the model card:
Common embedding model dimensions for reference:

​
Models Ragen Calls by Name

The four variables above are not the whole list. A few features call a model by a fixed id, not through an environment variable: On an isolated install, give each of these ids a route to a local model, so none of them falls through to a cloud provider whose key happens to be set. Ragen ships such a table, infra/llm-gateway/routes.appliance.yaml, along with an env template, infra/appliance/appliance.env.example. The template sets every model variable, VECTOR_SIZE, DOCLING_STRICT=1, a local encryption key, and leaves every outbound setting empty. The appliance table also gives each model a name. You see three of them in the model picker: The cloud-id aliases have a display name but no chat capability. They answer the calls the application makes by those ids. They stay out of the picker, so you do not see Bielik listed three times. When you swap a model, change its model: and display: together.

​
Running Everything in Containers

docker-compose.appliance.yml sits on top of the two standard compose files and adds an Ollama service:
On a machine with an NVIDIA GPU and the NVIDIA Container Toolkit, add -f docker-compose.appliance.gpu.yml.
On an NVIDIA DGX Spark, or any Linux host with an NVIDIA GPU, one script does all of this and checks the GPU, the disk and the image architectures on the way. See NVIDIA DGX Spark.
A one-shot job pulls the models the appliance table names into the ollama_models volume and skips any that are already there. After the first pull, or with a volume restored from elsewhere, it needs no network. The web, api and worker containers wait for it, and they are pointed at the appliance table, the compose Ollama and DOCLING_STRICT=1. It needs Docker Compose 2.24 or newer. Copy infra/appliance/appliance.env.example to .env.local at the root of the checkout, then fill in every secret it lists. The containers read that file, and the API and the worker refuse to start without the secrets. docker-compose.appliance.yml switches off the telemetry and update checks of the services next to Ragen: Qdrant, Prisma, Ollama and Docling. The environment file does not, so run the stack with the compose file. See What still reaches outward.

​
Checking a Local Model Server

The setup checklist on the sign-in page asks each openai-compatible server behind DEFAULT_MODEL, REPHRASE_MODEL, SUMMARY_MODEL and EMBEDDINGS_MODEL for its model list (GET /models). It reports two problems separately:
  • A server that does not answer. It names the LLM_<CONNECTION>_BASE_URL to check.
  • A model the server does not have. For Ollama it gives the ollama pull command to run.
The model picker and GET /v1/models also leave out a model whose server answered without it, so nobody picks a model whose first answer is certain to fail. A server that does not answer hides nothing; the checklist reports it instead. Usage reports price a model whose route describes itself (display) at zero, because its cost is your hardware, not tokens. Both checklist findings are recommendations, so sign-in still works while a model server is down. Cloud routes are not probed this way, because proving a cloud model works costs a billed call. Use npm run gateway:preflight -- --probe for those.

​
What Still Reaches Outward

Setting local model targets closes most outbound traffic, but several other paths remain. Work through this table to confirm your installation is fully isolated. The egress check below found each of the last five rows, on a stack where no container had any other way out. They used to be switched off only in docker-compose.appliance.yml. The standard stack now switches them off too, so the appliance inherits them. Turning Docling’s UI on in a standard install is your call. The appliance pins it off.

​
Things That Break Differently on Local Models

Switching to a local model can change behaviour in ways that are not obvious errors. Know what to look for before you commit to a model.
Multi-query expansion asks the model to return a JSON object that matches a schema, and so do Brain extraction, RAG scoring and the guardrail policy judge. Ragen sends the schema only when you set structuredOutputs: true on the route. Without the flag, Ragen asks the model for plain JSON, and the model has to guess the shape. See Structured outputs.With the flag on, vLLM constrains generation to the schema natively. Ollama supports response_format, but coverage varies by model and version.A failure here does not surface as an error — Ragen falls back to searching the user’s raw question without expansion, so a follow-up like “and what about the second one?” is retrieved literally with no conversation history behind it. Watch the logs: grep for Rephrase-and-expand failed after switching models.
MCP connectors and built-in tools require a model that supports function calling. Many open models do; some do not. When a model does not support function calling, the tools simply never fire — there is no error, the features are just absent.
A text-only local model cannot process an image attached to a message. Set MULTIMODAL_TEXT_ONLY_MODELS to the model’s ID and MULTIMODAL_FALLBACK_MODEL to a vision-capable model, and Ragen will swap models automatically when a request includes an image.
RAG prompts are long — the question, the conversation history, and several retrieved document chunks can easily reach 4–8k tokens. A model with a short context window will either reject the request or silently truncate the earliest part of it. Truncation reads as “the answer ignored my document” rather than a context length error. Check your model’s context window against a realistic prompt before deploying.

​
Verifying Your Isolated Install

After completing the configuration, run these checks to confirm every path is local:
The egress check tests the reference local-model stack, not your installation. It brings up its own copy, with a generated environment, so a pass shows that the shipped compose files and route table send nothing out. It does not show that your own routes, environment or extra services are clean. Use it after changing the compose files or the appliance route table, and use the checks above for your own install. Run it from a checkout of the repository:
It starts the local-model stack on a Docker network with no route out, signs up an account, ingests a document and asks a question about it. It fails if any container tried to reach anything off that network, and names the container and the host. It runs in its own Compose project from a temporary directory, so your own environment file never reaches it. It uses small models by default; --full-models uses the real route table. It needs Docker Compose 2.24 or newer and about 13 GB of disk for the images. Then ask one question in the UI and ingest one document, and watch the model server’s log. A fully local install shows:
  • A request for the rephrase (before retrieval)
  • A request for the answer (after retrieval)
  • A batch of embedding requests for the ingested document
If the rephrase request never arrives at the local server, REPHRASE_MODEL is still pointing at a cloud model.