Ollama and vLLM: Serving a Model

Why host a model yourself?

Calling a model from a cloud provider is simple: one key, one request, one bill per token. Hosting an open model (one whose weights can be downloaded) on your own machine meets different needs:

• confidentiality: data never leaves your infrastructure;
• cost control: a predictable machine cost rather than a cost proportional to usage;
• independence: no forced version changes, and offline operation is possible.

The trade-offs are real: you need suitable hardware (often a GPU), and you have to install, monitor, update, and secure it. The open models you can run on a single card are also generally less capable than the largest commercial models on difficult tasks. Hence the “local first, cloud as a fallback” architecture covered later.

Ollama: getting started in three commands

Ollama is an open-source tool that downloads, manages, and serves models. On Linux, the official installation is done with a script, then:

ollama serve &                 # starts the server (often run as a service)
ollama pull qwen2.5:0.5b       # downloads a small model
ollama run qwen2.5:0.5b "Explain a for loop in one sentence."
ollama list                    # downloaded models
ollama ps                      # models loaded in memory, on GPU or CPU

Models are identified by a name and a tag (qwen2.5:0.5b) that specifies the size and sometimes the quantization. Without a GPU, Ollama runs on CPU, more slowly: that is enough for learning with a small model.

Ollama’s HTTP API

By default, the server listens on 127.0.0.1:11434. Your application queries it over HTTP:

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "qwen2.5:0.5b",
  "prompt": "Give a synonym for fast.",
  "stream": false
}'

The JSON response contains the text (response), a done flag, and measurements: prompt_eval_count and eval_count (tokens read and generated), eval_duration and total_duration (in nanoseconds). Other routes exist: /api/chat for a conversation, /api/tags for the list of models, /api/embed for embeddings.

Ollama also exposes an OpenAI-compatible API under /v1 (for example /v1/chat/completions): a client library designed for OpenAI works once you change the base URL.

vLLM: serving at scale

vLLM is an open-source inference engine designed for throughput on GPU. It dynamically groups in-flight requests (continuous batching) and manages the attention cache memory in pages (PagedAttention), which lets it serve many simultaneous requests efficiently.

vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192 --api-key "$VLLM_API_KEY"
curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"

vLLM loads models in Hugging Face format (identified as organization/model) and by default serves an OpenAI-compatible API on port 8000. An official Docker image (vllm/vllm-openai) makes deployment easier.

Which one to choose?

Criterion	Ollama	vLLM
Setup	very simple	more technical
Hardware	CPU or GPU	mainly GPU
Model format	GGUF (quantized)	Hugging Face, AWQ and GPTQ quantizations…
Strength	prototyping, small workloads	many simultaneous requests

Since both speak the same OpenAI dialect, you can start with Ollama and move to vLLM when the load requires it, without rewriting the application: only the URL and model names change. Measure your own load before deciding: no generic figure replaces a test on your hardware.

Key takeaways

• Self-hosting brings confidentiality and control, at the price of operations.
• Ollama: pull, run, list, ps, API on port 11434.
• vLLM: throughput-oriented on GPU, OpenAI-compatible API on port 8000.
• The OpenAI-compatible API lets you switch servers without rewriting the application.