One install and one pull command puts an open-weight model behind an OpenAI-compatible API on your own hardware — private by default, free per token, and offline-friendly.
The whole path from nothing to code calling a local model. Model choice against your hardware is the step that decides whether this feels magical or miserable.
Install from ollama.com — native installers for macOS and Windows, one script for Linux. It runs as a background service listening on localhost:11434, so nothing is exposed beyond your machine by default.
GPU acceleration is automatic where drivers exist (Apple Silicon, NVIDIA, most AMD). CPU-only works too; it's just slower — fine for small models and batch jobs.
The rule of thumb: the model file must fit in your VRAM (or unified memory) with headroom, or it spills to CPU and slows dramatically. 8GB comfortably runs 7–8B models at 4-bit quantization; 16GB handles 13–14B; 32GB+ opens up the 30B class.
Start with a current small instruct model for chat and a code model if that's your use. Pull them and talk to one interactively to sanity-check speed before wiring anything up.
ollama pull llama3.2 ollama pull qwen2.5-coder:7b ollama run llama3.2
Ollama serves two APIs on port 11434: its native one, and an OpenAI-compatible /v1 endpoint — meaning any library or tool that speaks the OpenAI SDK works by changing the base URL and using any string as the API key.
That compatibility is the unlock: editors, agents, and existing apps point at your box instead of a cloud endpoint, and prompts containing sensitive data never leave the machine.
curl http://localhost:11434/v1/chat/completions \
-d '{"model":"llama3.2","messages":[{"role":"user","content":"hi"}]}' Bake a persona or default parameters into a variant with a Modelfile — FROM a base model plus a SYSTEM prompt and parameters like temperature, then ollama create yourname. Your app then requests that model name and gets consistent behavior without repeating the system prompt everywhere.
Context length is a parameter too (num_ctx); raising it costs memory, so raise it deliberately, not by default.
To serve other machines, set OLLAMA_HOST=0.0.0.0 — but understand the API has no authentication. Never do this on a machine with a public IP without putting an authenticating reverse proxy in front, or better, share it over a private network like a tailnet so only your devices reach it.
One box with a decent GPU serving models to every laptop in the house or office over a private network is the setup most people actually want.
Generation is painfully slow.
The model doesn't fit in GPU/unified memory and is spilling to CPU. Use a smaller model or heavier quantization (q4), and check ollama ps to see actual placement.
'Model requires more system memory' error.
The model plus context doesn't fit at all. Pick the smaller parameter count or reduce num_ctx.
Other machines can't reach the API.
Ollama binds localhost by default. Set OLLAMA_HOST=0.0.0.0 in the service environment and open the port only on a private network.
First request after idle takes forever.
The model loads on demand and unloads after inactivity. Set OLLAMA_KEEP_ALIVE to keep the model resident if latency matters.
Responses are weirdly bad vs. the same model in the cloud.
You're likely running a heavier quantization than the benchmark. Try a larger quant (q8) or the full-precision tag if memory allows, and check the system prompt isn't being doubled.
Honesty note: articles are drafted with AI assistance, then a human checks each one against the vendor's current setup flow before it publishes — nothing goes live unreviewed. Vendor interfaces still change; if a step looks different, the underlying record is what matters.