model: at a local server and model inference stays on your
machine. Other steps in the workflow can still contact services, such as
a URL fetch, an MCP server or a hosted model chosen for another task.
This page is the ladder, from the server built into nika to production
servers.
1. Built in: nika model pull and nika model serve
The nika binary carries its own local inference: no daemon to install,
one models folder (~/.nika/models).
Pull a GGUF from the Hugging Face Hub
--yes for scripts). An interrupted pull resumes, and
HF_TOKEN authenticates gated repositories. The pull is a command you
run, not a workflow step, so no permits: block governs it.Serve it
127.0.0.1:8712:
loopback only, one generation at a time, Ctrl-C stops it. Its banner
prints the two lines a workflow needs.Point a workflow at it
llamacpp provider, so the file needs no
new syntax.nika model list shows what is on disk and nika model rm <owner/repo>
reclaims it.
Gear One: the pick nika welcome proposes
nika welcome and nika doctor propose one local pull sized to your
memory, under the name Nika Gear One. The pick is a table shipped in
the binary:
nika model serve cannot load: on a 16 to 23 GB
machine, either pull the lite pick for the built-in server, or serve the
Gemma file with llama.cpp’s llama-server or LM Studio (below).
2. The one-command path: Ollama
Install Ollama, then:nika init scaffold starts smaller, with ollama/qwen2.5:0.5b):
nika doctor --ping confirms the wiring: it connects to the local ports
you configured and reads their model lists, without running a model.
Audit first, then run on your own machine. nika check clears a meeting-notes workflow before any model is called; then a local Ollama model (ollama/llama3.2:3b) turns a sample transcript into typed action items, written to action-items.json with a hash-chained trace. No provider key, no cloud call for the model. Captured from the real CLI: the check on nika 0.121.0, the local-model run on nika 0.109.2. The file is a shorter demo version of the meeting actions example.
3. Any Hugging Face GGUF through Ollama
Ollama pulls directly from the Hugging Face Hub. Any public GGUF repo works, no account needed:Q4_K_M is the sane default (quality per
GB), Q8_0 when you have RAM to spare, Q2/Q3 only when memory is tight.
The Hub’s GGUF filter lists
every compatible repo.
4. LM Studio: the visual browser
Prefer a GUI? LM Studio browses Hugging Face, downloads models with a click, and serves an OpenAI-compatible endpoint. Start its server, then:5. Servers: llama.cpp and vLLM
For shared machines and production:- llama.cpp
llama-serverserves a GGUF:model: llamacpp/<model> - vLLM serves full-precision Hub models at
datacenter throughput:
model: vllm/<hub-repo-id>
NIKA_<ID>_BASE_URL (for
example NIKA_VLLM_BASE_URL) points a provider at another host or port.
Checking what’s wired
$0:
--max-cost-usd never blocks local or mock work: it bounds what the catalog
can meter. Cost honesty explains why.
Thinking time is budgeted for
A local thinking model can need minutes for one completion on consumer hardware. When a task declares notimeout:, a buffered provider call gets
600 s, local or cloud alike: the transport cannot tell a model that is
thinking from a dead connection. A streaming call has no total deadline;
the idle-read guard ends a stream that stays silent for 600 s.
When a task needs a different bound, its own timeout: governs the
provider deadline directly:
Swapping between local and cloud
The file does not change shape. One line moves the workflow between a laptop and an API:permits: block with no
net.http entry means the workflow cannot reach the network even if the
model could. Local model + closed permits = fully air-gapped AI work.