Skip to main content
T2 diamond · engineering / model selection: three infer.model: seats fan out the SAME question in parallel; a nika:jq fan-in builds a markdown table from measured facts — each answer’s latency comes from tasks.X.duration_ms, the run’s own clock. No judge model, no scores: the table states facts, the quality call stays yours.

The job

Picking a default model by vibes is how you end up paying for quality you don’t need — or shipping answers a smaller model would have matched. This workflow is the bench you run BEFORE deciding: same prompt, three seats, one report on disk that you can diff run over run.

The shape

The file

model-bench.nika

Two findings the bench surfaced on its own authoring run

Both were caught live while writing this example — they are what the bench is FOR:
  1. A thinking model with a bounded max_tokens can return an empty answer — it spends the whole budget reasoning before a single output token. The row reads 0 chars: that is a bench result, not a bug. Seat ollama/qwen3.5:4b and reproduce it on your machine.
  2. On a --resume, rehydrated rows say cached — a latency that was not re-measured is never re-printed as a fresh number. The report stays honest across resumes.

Run it

What it needs: Ollama with the three contenders pulled (qwen2.5:14b, llama3.2:3b, qwen2.5:0.5b). There is no mock rehearsal: each task pins its own model, --model replaces only a workflow’s default model, and this file has none, so nika try model-bench needs the same three models.
1

Rehearse it

It runs the same three pinned models: pull them first (next step), or the rehearsal fails on each one that is missing (NIKA-INFER-001).
2

Get your own copy

Save the file above as model-bench.nika in a new folder. It reads no sample file.
3

Check it, then run it

The run writes out/model-bench.md.The first call to each model pays its load time: run the bench twice and read the warm numbers. Swap any seat for a cloud model (mistral/…, anthropic/…, openai/…) to bench it against your local models, with its key and --max-cost-usd.