The quiet rise of the local LLM.
“Can we run it on our own hardware?” went from a hobbyist question to the most common thing clients ask us in 2026. Here’s when the answer is yes — and what local models are genuinely good at today.
Three things changed at once. The open-weight models got genuinely good — Qwen is now the most-downloaded model family in the world, and MIT-licensed frontier-adjacent weights from DeepSeek and Zhipu ship the day they launch. The hardware caught up — NPU-equipped “AI PCs” and big-memory workstations run a quantised 70B-class model at conversational speed, and the tooling matured from weekend-project (Ollama) to production-grade (vLLM). And the regulators arrived: in banking, telecoms, health and anywhere the EU AI Act reaches, data residency has turned “the model comes to the data” from preference into requirement. Industry surveys now put a majority of enterprise inference on-premises or at the edge — up from around one-eighth in 2023.checked 2026-07-06
The economics: rent first, buy at volume
At low volume the cloud wins, full stop — you can’t amortise a GPU server against a few hundred thousand tokens a day, and API prices keep falling. The crossover comes with sustained volume: as a rule of thumb, a business consistently pushing around ten million tokens a day through a 70B-class model breaks even on its own hardware within 12–18 months, because API costs scale linearly with usage while local costs flatten once the box is paid for. Below that line, buy tokens. Above it — or the moment compliance forbids the data leaving the building — the calculation flips.checked 2026-07-06
What local models are genuinely good at — today
RAG over contracts, policies and internal knowledge. The workhorse local use case — private by construction.
Redaction, classification, extraction on customer records — the data never crosses your firewall.
Factory floors, field kit, air-gapped sites. Milliseconds instead of a cloud round-trip; works offline.
First-line drafting and routing at fixed cost — predictable volume is exactly what local pricing rewards.
Completion and review where the codebase is the crown jewels. Good, not frontier — know the gap.
Multi-hour autonomous runs and the hardest reasoning still belong to the big hosted models. Route accordingly.
Whether a local model belongs in your stack is — as ever — a measurement question: your volumes, your data rules, your hardware, your numbers. That’s a Reality Check, not a hunch.
Local is a compliance answer first, a cost answer second. If your data can’t leave the building, the decision is already made — everything else is sizing.
Don’t buy hardware to feel sovereign. Below ∼10M steady tokens a day, an owned GPU server is an expense pretending to be a strategy. Rent until the volume argues otherwise.
Someone has to run the thing. A local model is a small production service: updates, monitoring, capacity, an owner. Budget the human, not just the box.
Hybrid is the grown-up architecture. Local for the private, high-volume, predictable work; hosted frontier for the hard 10%. The same routing logic that saves money on cheap APIs applies to your own silicon.