AI & Machine Learning
Ollama
Run open-weight language models on your own hardware — one command, no per-token bill, nothing leaving your network.
Overview
Ollama is the simplest way to run open-weight large language models locally. One command downloads a model — Llama, Mistral, Qwen, Gemma, and dozens more — and runs it on your own machine or server. It packages the model’s weights, tokeniser and configuration into a single pull, works out whether to use your GPU or fall back to CPU, and handles quantisation so a model that would otherwise need a data centre can fit on a laptop or a modest server. The friction that used to sit between a published open model and a running one — CUDA versions, weight formats, serving code — is gone.
What makes Ollama genuinely useful for building software, rather than just for tinkering, is the local REST API it exposes. Once a model is running, your application talks to it over HTTP exactly as it would talk to a hosted provider. Ollama also ships an OpenAI-compatible endpoint, so code written against the OpenAI SDK can point at a local model by changing a base URL and nothing else. A Modelfile lets you pin a system prompt, parameters and a base model into a reusable, named variant — your own configured model, version-controlled alongside the code that uses it.
We use Ollama in two places. First, in development: every engineer on a project can run the same model locally, iterate on prompts and retrieval without a metered API in the loop, and work on a train or a plane. Second, in privacy-sensitive and on-premises deployments, where the data genuinely cannot leave the building and a hosted API is off the table for legal or contractual reasons. We are equally clear about its ceiling — Ollama is built for single-machine, development-scale inference, and when a workload needs real concurrency you graduate to a served cluster. Ollama’s job is to make that first stretch fast and cheap, and to prove the model choice before you spend on serving infrastructure.
Best for — Teams that need private, local LLM inference — for development, for data that cannot leave the network, or for small internal tools — and want a fast path to running open models before committing to production serving.
Why teams choose Ollama
No per-token cost
Once a model runs on hardware you already own, inference is effectively free at the margin. Development iteration, batch jobs and internal tools stop metering every call, which changes how freely a team can experiment.
Data never leaves your control
Prompts and responses stay on your machine or your server. For confidential, regulated or contractually restricted data, that is not a nice-to-have — it is the difference between being able to use an LLM at all and not.
A hosted-API-shaped local endpoint
Ollama’s OpenAI-compatible API means application code does not care whether it is talking to a local model or a cloud one. You can build against a local model and switch to a hosted provider later — or the reverse — by changing a base URL.
Why businesses choose Ollama
- You need LLM inference on data that genuinely cannot leave your network, and a hosted API is ruled out by law, contract or policy.
- You want your team developing against a real model locally, with no metered API and no rate limits slowing the iteration loop.
- You are building small-scale internal tools where the request volume is modest and the cost and privacy of self-hosting clearly beat a hosted provider.
- You want to prove an open model against your actual task on cheap local hardware before committing budget to a production serving stack.
What we build with Ollama
The capabilities this technology is genuinely strong at — and what we most often build with it.
One-command model pulls
A single `ollama run` or `ollama pull` fetches a model’s weights, configuration and tokeniser and starts it serving. Switching between Llama, Mistral, Qwen, Gemma and others is a matter of a model name, not an afternoon of setup.
Automatic quantisation and hardware handling
Ollama ships quantised builds and detects your GPU, falling back to CPU when it must. That is what lets a model which would otherwise need a data centre run on a workstation — we choose the quantisation level against the quality you actually need.
Local REST and OpenAI-compatible API
A running model is reachable over HTTP on localhost, with both Ollama’s native API and an OpenAI-compatible endpoint. Application code — including anything already written for the OpenAI SDK — talks to the local model by pointing at that base URL.
Modelfiles for reproducible variants
A Modelfile pins a base model, a system prompt, and parameters like temperature and context length into a named, buildable variant. We keep these in version control so development, staging and on-prem all run an identical, auditable model definition.
Embeddings for local retrieval
Ollama serves embedding models as well as chat models, so a retrieval pipeline — chunk, embed, store in a vector database, query — can run end to end on your own hardware without a single call to an external service.
Streaming and structured output
The API streams tokens as they are generated for responsive interfaces, and supports constraining output to JSON so a local model can drive tool-calling and structured extraction the same way a hosted one would.
Use cases
Team-wide local development
Every engineer runs the same model on their own machine, iterating on prompts, retrieval and agent logic with no metered API in the loop and no rate limits — then the same code points at a served model in production.
On-premises assistants for sensitive data
An internal question-answering or drafting tool over confidential documents, running entirely inside a client’s network, so no prompt or document ever reaches a third-party provider.
Offline and air-gapped inference
LLM features on machines with no reliable internet, or in secure environments that forbid outbound connections — field hardware, isolated networks, or regulated estates where an air gap is mandatory.
Proving a model before you scale
A cheap local trial of an open model against your real task and data, to settle which model and prompt actually work before you spend on GPUs and a vLLM serving cluster.
When Ollama is the right choice
- Right for local development and prototyping: give the whole team a real LLM on their own machines, iterate on prompts and retrieval with no per-token cost and no rate limits, and keep experiments fast and disposable.
- Right when data cannot leave your network — regulated, confidential, or contractually restricted material — and you need inference that runs entirely on hardware you control, on-premises or in your own private cloud.
- Right for offline and air-gapped work, and for small-scale internal tools where a handful of colleagues query a model occasionally rather than thousands of users hitting it at once.
- Wrong for high-throughput production serving: Ollama is designed for one machine at a time, not for hundreds of concurrent requests. For that you want vLLM, TGI, or a managed serving cluster, and we will build you that instead.
- Wrong when you need frontier-model quality with no operations burden. A quantised model running locally will trail the best hosted models on hard reasoning, and running your own inference means you own the hardware, the updates and the uptime. If those trade-offs do not pay off, a hosted API is the honest answer.
Ollama: pros and cons
Strengths
- The fastest route from a published open model to a running one: a single command pulls weights, config and tokeniser and starts serving, with GPU or CPU detection and quantisation handled for you.
- Local and private by default — no data leaves the machine, no API key, no rate limit, and no per-token bill during development or for internal use.
- A clean local REST API plus an OpenAI-compatible endpoint, so existing SDKs and application code work against a local model with a one-line change.
- Modelfiles make a configured model — base model, system prompt, parameters — reproducible and version-controllable, so every environment runs exactly the same thing.
Trade-offs
- Your hardware sets the ceiling: large models and long context windows need serious GPU VRAM. On a laptop you are limited to smaller, more heavily quantised models, and quality drops with them.
- Quantised local models trail frontier hosted models on hard reasoning, coding and instruction-following. For the most demanding tasks, a smaller local model is a real step down in quality, and you should know that going in.
- It is built for single-machine, development-scale serving, not high concurrency. Throughput and batching under load are not its job — that is where vLLM or TGI take over.
- You take on the operations: model updates, GPU drivers, capacity, monitoring and the hardware itself. What you save in per-token cost you partly pay back in the effort of running your own inference.
Architecture
Ollama runs as a local server that loads a model into memory and exposes it over HTTP. Your application never links the model in directly; it makes API calls, exactly as it would to a hosted provider. We treat that boundary as the whole point of the design — the application is written against an API, and whether that API is served by Ollama on localhost, by a vLLM cluster in your cloud, or by a hosted vendor becomes a deployment decision rather than a rewrite. Building the client behind a thin provider interface keeps that door open.
For anything beyond a bare chat call we compose Ollama into a pipeline. A retrieval-augmented setup embeds your documents with an Ollama embedding model, stores the vectors in a vector database, and injects retrieved context into the prompt at query time — all of it local. Modelfiles define the exact model variant each environment runs, version-controlled alongside the code, so there is no drift between an engineer’s laptop and the on-prem box. Where we expect to graduate to a served cluster later, we design the retrieval and orchestration layers to be serving-engine-agnostic from day one.
Performance
Ollama’s performance is dominated by two things: the hardware and the model. On a GPU with enough VRAM to hold the model, generation is quick and interactive; on CPU, or when a model spills out of VRAM, it slows sharply. Quantisation is the main lever — a more aggressively quantised model uses less memory and runs faster, at a measurable cost in quality. We size that trade-off deliberately, benchmarking candidate models at candidate quantisation levels against your actual task rather than trusting a leaderboard.
Where Ollama does not compete is concurrency. It serves requests on a single machine and is not built to batch hundreds of simultaneous users efficiently — throughput under load is precisely what dedicated serving engines like vLLM exist to solve, with continuous batching and paged attention. So we are clear about the boundary: Ollama is excellent for one user, or a few, iterating quickly; when the workload is many concurrent requests with tight latency targets, that is the signal to move to a served cluster, and we plan for that transition rather than pushing Ollama past what it was designed for.
Security
The core security property is also the reason most clients reach for Ollama: prompts and responses never leave the machine. For regulated or confidential data, that removes an entire category of exposure — there is no third-party provider retaining your inputs, no data-residency question, no vendor to trust with material you are not allowed to share. This is what makes local inference viable where a hosted API is contractually or legally impossible.
That privacy is only real if the deployment is locked down. Ollama’s API has no authentication of its own, so we never expose it directly to a network — it sits behind an authenticating gateway or stays bound to localhost, reached only by services we control. We treat model provenance as part of the supply chain, pulling weights from sources we trust and pinning versions, and on-premises we handle the host hardening, patching and access control as deliberately as any other production system, because with self-hosting those responsibilities are yours rather than a provider’s.
Scalability
It is important to be honest about the shape of Ollama’s scaling: it does not. Ollama scales down beautifully — onto every engineer’s laptop and onto a single on-prem server — but it is not the tool for scaling up to heavy concurrent production traffic. Trying to make it serve thousands of simultaneous users is using it against its grain, and the answer is not more Ollama instances behind a load balancer but a serving engine designed for the job.
So we treat Ollama as the first rung of a ladder. It proves the model, the prompts and the retrieval on cheap hardware and in development, and because the application talks to it through a standard API, graduating to vLLM or TGI on a GPU cluster changes the serving layer and little else. That deliberate transition — local Ollama for development and small-scale or private workloads, a dedicated served cluster when concurrency demands it — is how we get the cost and privacy benefits early without painting a project into a corner it cannot serve out of later.
Ollama integrations & ecosystem
The technologies we most often pair with it — each links to how we work with it.
How we work
We start by testing whether local inference is the right call at all. That means benchmarking a shortlist of open models, at sensible quantisation levels, against your real task and data — not a public leaderboard — and weighing the result honestly against a hosted API. If a hosted model is genuinely the better answer for your quality needs and volume, we will say so before you invest in hardware. When local does win, we lock the model choice into a version-controlled Modelfile so every environment runs the same thing.
From there we build the application against Ollama’s API through a thin provider interface, so the model is a swappable dependency rather than something wired through the codebase. We keep retrieval and orchestration serving-engine-agnostic, which is what lets a project move from local Ollama to a vLLM cluster without a rewrite when it needs to. The engineers who design this are the ones who run it: we optimise for the deployment you will actually operate — its hardware, its updates, its failure modes — not for a demo that only ever ran on one laptop.
The service behind it
Delivered throughAI DevelopmentWhat we build with Ollama
The disciplines this technology most often shows up in — from a first build to taking over and stabilising an existing one.
How we deliver
- 01
Discover
We map the system, the constraints and the business it serves — including the parts nobody documented.
Architecture brief
- 02
Architect
Decisions get made, written down and defended before a line of production code exists.
Decision records
- 03
Build
Short cycles against working software. You see progress in the product, not in a status deck.
Shipping increments
- 04
Operate
Monitoring, incident response and iteration. The system is alive, so the engagement is too.
Runbooks & SLOs
Industries we use Ollama in
Domain knowledge changes what gets built. A few of the sectors we know before the first meeting.
Also in AI & Machine Learning
Why teams choose us for Ollama
We tell you when local is the wrong call
Running your own inference pays off in some cases and not others. We benchmark honestly and will recommend a hosted API when a quantised local model would not meet your quality bar — before you buy a single GPU.
We operate what we build
Self-hosted inference means real operations — hardware, drivers, updates, uptime. We run the systems we ship, so we design for the deployment you will actually maintain, not a laptop demo that never faced load.
We build the whole ladder
Ollama for development and private, small-scale work; vLLM or TGI when concurrency demands it. Because we build both, we design the application to move between them cleanly rather than trapping you on one rung.
Senior engineers only
Model choice, quantisation trade-offs and retrieval quality reward judgement and punish guesswork. The people making those calls on your project have made them before and have run the results in production.
Typical timeline
- 01
Feasibility and model selection
One to two weeks benchmarking candidate open models at candidate quantisation levels against your real task, sizing the hardware, and deciding honestly between local Ollama and the alternatives.
- 02
First working slice
Two to three weeks building one real feature end to end against Ollama’s API — including retrieval if it is needed — so the model choice is proven against reality, not a plan.
- 03
Iterative build and hardening
Feature-by-feature delivery, tightening prompts, retrieval quality and the deployment, with the API boundary kept clean so the serving layer stays swappable.
- 04
Deployment and handover
Standing up the on-premises or private deployment behind an authenticating gateway, documenting the Modelfiles and operations, and — where relevant — planning the path to a served cluster.
How pricing works
- Fixed-scope builds for a defined local or on-premises LLM feature — a private assistant, a retrieval tool, an offline capability — quoted once we understand the model, the data and the hardware.
- A focused feasibility engagement: benchmarking open models against your task and data on representative hardware, with an honest recommendation on local Ollama versus a served cluster versus a hosted API.
- Monthly senior engagement for ongoing work where the model, prompts and retrieval evolve, including the eventual graduation from local inference to a production serving stack.
Hire Ollama engineers
Need Ollama capacity on your own team? We embed named senior engineers into your existing team — reporting to your leads, working in your rituals — so you add capacity without a hiring cycle.
Hire Ollama engineersCommon questions
Is Ollama suitable for production?
For the right kind of production, yes — a private, on-premises tool with modest, low-concurrency usage, or an internal assistant used by a handful of colleagues. What it is not built for is high-throughput serving with many simultaneous users; Ollama serves on a single machine and does not batch heavy concurrent load efficiently. When you need that, we move you to vLLM or TGI on a GPU cluster, and we design the application so that move does not require a rewrite.
How does a local Ollama model compare to a hosted model like Claude or GPT?
On quality, a quantised model running locally will generally trail the best hosted frontier models on hard reasoning, coding and instruction-following — that gap is real and you should plan around it. What you gain in exchange is privacy, no per-token cost, and no rate limits. For many tasks a good open model is more than sufficient; for the most demanding ones a hosted API is worth it. We benchmark against your actual task rather than guessing which side of that line you fall on.
What hardware do we need to run Ollama?
It depends entirely on the model. Small, quantised models run acceptably on a modern laptop, even on CPU. Larger models and long context windows need a GPU with enough VRAM to hold the model — that is the binding constraint, and it grows quickly with model size. Part of our feasibility work is sizing the hardware against the model you actually need, so you are not surprised by a VRAM requirement after the fact.
Can we switch from Ollama to a hosted provider later, or the other way round?
Yes, and we build for it. Ollama exposes an OpenAI-compatible API, so application code written against the OpenAI SDK can point at a local model — or a hosted one — by changing a base URL. We put a thin provider interface in front of the model regardless, so the model is a swappable dependency. Starting local for development and privacy, then graduating to a served cluster or a hosted API, is a deployment change rather than a rebuild.
Is our data really private if we use Ollama?
The inference itself is fully local — prompts and responses never leave the machine, with no third-party provider involved. That is genuinely private in a way a hosted API cannot be. The caveat is the deployment: Ollama’s API has no built-in authentication, so it must sit behind an authenticating gateway or stay bound to localhost, never exposed raw to a network. Get that right, and it is a sound basis for handling data that legally or contractually cannot leave your control.
Building on Ollama?
A technical conversation with the engineers who would do the work. If we are not the right fit, we will say so on the call.