Skip to main content
How do you pin what a workflow returns without spending tokens? nika test <file> runs it under the mock provider — offline, deterministic, zero keys — then diffs the output values against <file>.golden.json (deep JSON equality, path-by-path). Schema acceptance is a separate gate: when infer.schema is present the mock must synthesize a valid instance before that JSON is compared. Predict: if you change only the prompt text, does the golden still match? Not if the recorded values change. mock/echo echoes the prompt, so the golden fails. A schema mock that still synthesizes the same instance can match even when the prompt changed — the golden never means “shape only”.

​
Why the mock makes this possible

nika test swaps every infer: / agent: call to the mock provider, whatever model: the file declares — your ollama/qwen3.5:4b (or openai/…, or any cloud model) workflow tests offline, unchanged. The mock is not a stub that returns "ok". It is schema-conformant: when a task declares a schema:, the mock synthesizes an instance that validates against it — required fields present, types respected. Every schema workflow runs offline, end to end, through the same runtime, bindings, and typed-outputs validation as a live run.
triage.nika
The golden pins the workflow’s outputs: block — a workflow without one pins {}. Declare typed outputs: and the golden guards the whole callable contract.

​
The golden lifecycle

First run — no golden exists yet, so nika test teaches instead of guessing (exit 3):
Create it with --update, review it once, commit it:
The golden is small, readable JSON — the mock-synthesized, schema-conformant outputs:
triage.nika.golden.json
From now on nika test compares. A match is exit 0; drift renders a per-path diff and exits 1:
If the drift was intended (you changed the schema, added an output), re-pin with --update and commit the new golden — the diff shows up in code review, exactly like a snapshot test.

​
Proving a rule against cases

A nika:decide bundle is a rubric with consequences, so it wants a corpus, not one happy path. The kernel enforces the bundle’s own laws, but a fixture’s class is only a label: nothing asserts that a positive fixture actually recommends. nika test is how you assert it today. The pattern that works: the rule lives once, the cases live one per file.
Each case is a small workflow: an evidence snapshot in const:, the kernel in the middle, the outcome out. No model, no network, no clock, so the only thing under test is the rule.
refund-fresh-returned.nika
Pin it once, read the outcome, commit both files:
refund-fresh-returned.nika.golden.json
The rule is now guarded. Change one character in the shared bundle, a weight of 6000 that becomes 600, and every case that leaned on it goes red at once, naming the flip:
That line is the point. Not “a test failed” but this rule used to recommend this case and now defers. If the flip was intended, re-baseline and let the golden diff carry it into review, which is where a rubric change belongs.
Keep the golden narrow. Pinning outcome (or the outcome plus one dimension interval) makes each red line legible. Pinning the whole receipt works, but the receipt carries every weight_bp and every contribution, so a single weight edit churns every golden in the corpus and buries the one flip that matters.

​
Two limits, stated plainly

  • nika test takes no --var. Its flags are --update · --answer · --color · --hyperlink · --plain · --ascii. One file is one input, so a ten-case corpus is ten small files sharing one bundle. That is more files than a parameterised table would be, and it is also ten reviewable goldens.
  • Under the mock, the model leg is a constant. nika test swaps every infer: to the mock provider, which synthesizes a schema-conformant value. A case that runs infer: then nika:decide therefore proves the law, and proves nothing about the extraction: the facts were handed to it. Keep the model out of the case files, pin the evidence in const:, and prove the rubric. Extraction quality is a separate exercise against real seats, where it costs money and is worth what it costs.

​
Exit codes · the CI contract

nika test refuses to run a dirty file the same way nika run does: a workflow with check findings exits 2 before anything executes.

​
Wire it into CI

Offline and deterministic means no keys in CI, no flakes, no spend:
.github/workflows/nika-test.yml (fragment)
Commit the *.golden.json files in the same directory as the workflows they guard. A PR that edits a workflow either keeps its golden green or shows the re-pinned golden in the diff — both are reviewable.