T2 diamond · engineering / model selection: threeinfer.model:seats fan out the SAME question in parallel; anika:jqfan-in builds a markdown table from measured facts — each answer’s latency comes fromtasks.X.duration_ms, the run’s own clock. No judge model, no scores: the table states facts, the quality call stays yours.
The job
Picking a default model by vibes is how you end up paying for quality you don’t need — or shipping answers a smaller model would have matched. This workflow is the bench you run BEFORE deciding: same prompt, three seats, one report on disk that you can diff run over run.The shape
The file
model-bench.nika
Two findings the bench surfaced on its own authoring run
Both were caught live while writing this example — they are what the bench is FOR:- A thinking model with a bounded
max_tokenscan return an empty answer — it spends the whole budget reasoning before a single output token. The row reads0 chars: that is a bench result, not a bug. Seatollama/qwen3.5:4band reproduce it on your machine. - On a
--resume, rehydrated rows saycached— a latency that was not re-measured is never re-printed as a fresh number. The report stays honest across resumes.
Run it
What it needs: Ollama with the three contenders pulled (
qwen2.5:14b, llama3.2:3b, qwen2.5:0.5b). There is no mock rehearsal: each task pins its own model, --model replaces only a workflow’s default model, and this file has none, so nika try model-bench needs the same three models.1
Rehearse it
NIKA-INFER-001).2
Get your own copy
Save the file above as
model-bench.nika in a new folder. It reads no sample file.3
Check it, then run it
out/model-bench.md.The first call to each model pays its load time: run the bench twice and read the warm numbers. Swap any seat for a cloud model (mistral/…, anthropic/…, openai/…) to bench it against your local models, with its key and --max-cost-usd.