Skip to main content
Point model: at a local server and model inference stays on your machine. Other steps in the workflow can still contact services, such as a URL fetch, an MCP server or a hosted model chosen for another task. This page is the ladder, from the server built into nika to production servers.

1. Built in: nika model pull and nika model serve

The nika binary carries its own local inference: no daemon to install, one models folder (~/.nika/models).
1

Pull a GGUF from the Hugging Face Hub

The size prints before the download, and a file of 2 GiB or more asks you to confirm (--yes for scripts). An interrupted pull resumes, and HF_TOKEN authenticates gated repositories. The pull is a command you run, not a workflow step, so no permits: block governs it.
2

Serve it

A foreground, OpenAI-compatible server starts on 127.0.0.1:8712: loopback only, one generation at a time, Ctrl-C stops it. Its banner prints the two lines a workflow needs.
3

Point a workflow at it

In another terminal:
The served model rides the llamacpp provider, so the file needs no new syntax.
nika model list shows what is on disk and nika model rm <owner/repo> reclaims it.
In 0.124.3 the built-in server loads Qwen3-family GGUFs only. A GGUF of another family is refused when it loads; serve it with one of the servers further down this page.

Gear One: the pick nika welcome proposes

nika welcome and nika doctor propose one local pull sized to your memory, under the name Nika Gear One. The pick is a table shipped in the binary: None of these picks is marked verified in 0.124.3. The standard pick is a Gemma 3 model, which nika model serve cannot load: on a 16 to 23 GB machine, either pull the lite pick for the built-in server, or serve the Gemma file with llama.cpp’s llama-server or LM Studio (below).

2. The one-command path: Ollama

Install Ollama, then:
That’s it. In any workflow (the same tag the Homebrew install suggests; the nika init scaffold starts smaller, with ollama/qwen2.5:0.5b):
nika doctor --ping confirms the wiring: it connects to the local ports you configured and reads their model lists, without running a model.
qwen3.5:4b is small enough for most laptops and good enough for structured extraction. It thinks before it answers, and the thinking counts against max_tokens: size the ceiling for both, and if a run fails NIKA-INFER-004, the thinking ate the budget. Swap the tag for anything in the Ollama library: llama3.2, mistral, phi4, larger Qwen variants. Rehearse with model: mock/echo when you want zero daemon and zero keys: mock simulates the model reply; it does not disable tools or effects.

Audit first, then run on your own machine. nika check clears a meeting-notes workflow before any model is called; then a local Ollama model (ollama/llama3.2:3b) turns a sample transcript into typed action items, written to action-items.json with a hash-chained trace. No provider key, no cloud call for the model. Captured from the real CLI: the check on nika 0.121.0, the local-model run on nika 0.109.2. The file is a shorter demo version of the meeting actions example.

3. Any Hugging Face GGUF through Ollama

Ollama pulls directly from the Hugging Face Hub. Any public GGUF repo works, no account needed:
Pick a quantization explicitly with a tag:
Then reference it in the workflow exactly as pulled:
Rule of thumb for quantizations: Q4_K_M is the sane default (quality per GB), Q8_0 when you have RAM to spare, Q2/Q3 only when memory is tight. The Hub’s GGUF filter lists every compatible repo.

4. LM Studio: the visual browser

Prefer a GUI? LM Studio browses Hugging Face, downloads models with a click, and serves an OpenAI-compatible endpoint. Start its server, then:

5. Servers: llama.cpp and vLLM

For shared machines and production:
  • llama.cpp llama-server serves a GGUF: model: llamacpp/<model>
  • vLLM serves full-precision Hub models at datacenter throughput: model: vllm/<hub-repo-id>
Both speak the same OpenAI-compatible dialect; the prefix picks the provider, nothing else in the file changes. NIKA_<ID>_BASE_URL (for example NIKA_VLLM_BASE_URL) points a provider at another host or port.

Checking what’s wired

Doctor prints a fix for each missing piece and never applies it. Then audit the file before a run:
A local model has no catalog price, so the cost line marks it unpriced, never $0:
--max-cost-usd never blocks local or mock work: it bounds what the catalog can meter. Cost honesty explains why.

Thinking time is budgeted for

A local thinking model can need minutes for one completion on consumer hardware. When a task declares no timeout:, a buffered provider call gets 600 s, local or cloud alike: the transport cannot tell a model that is thinking from a dead connection. A streaming call has no total deadline; the idle-read guard ends a stream that stays silent for 600 s. When a task needs a different bound, its own timeout: governs the provider deadline directly:
Full semantics: YAML reference · Timeouts.

Swapping between local and cloud

The file does not change shape. One line moves the workflow between a laptop and an API:
The permits boundary applies either way: a permits: block with no net.http entry means the workflow cannot reach the network even if the model could. Local model + closed permits = fully air-gapped AI work.

Providers

Model and access, the wires, and how one request reaches every provider.

First workflow

Five minutes from install to a checked, runnable file.

Model bench

Pick your default local model on a measured table, not vibes.

Cost honesty

Why a local run is unpriced, never « free », and how floors work.