nika test <file> runs it under the mock provider — offline,
deterministic, zero keys — then diffs the output values against
<file>.golden.json (deep JSON equality, path-by-path). Schema
acceptance is a separate gate: when infer.schema is present the
mock must synthesize a valid instance before that JSON is compared.
Predict: if you change only the prompt text, does the golden still
match? Not if the recorded values change. mock/echo echoes the
prompt, so the golden fails. A schema mock that still synthesizes the
same instance can match even when the prompt changed — the golden
never means “shape only”.
Why the mock makes this possible
nika test swaps every infer: / agent: call to the mock provider,
whatever model: the file declares — your ollama/qwen3.5:4b (or
openai/…, or any cloud model) workflow tests offline, unchanged.
The mock is not a stub that returns "ok". It is schema-conformant:
when a task declares a schema:, the mock synthesizes an instance that
validates against it — required fields present, types respected. Every
schema workflow runs offline, end to end, through the same runtime,
bindings, and typed-outputs validation as a live run.
triage.nika
The golden lifecycle
First run — no golden exists yet, sonika test teaches instead of
guessing (exit 3):
--update, review it once, commit it:
triage.nika.golden.json
nika test compares. A match is exit 0; drift renders a
per-path diff and exits 1:
--update and commit the new golden — the diff shows up in code
review, exactly like a snapshot test.
Proving a rule against cases
Anika:decide bundle is a rubric
with consequences, so it wants a corpus, not one happy path. The kernel
enforces the bundle’s own laws, but a fixture’s class is only a label:
nothing asserts that a positive fixture actually recommends. nika test is
how you assert it.
The pattern that works: the rule lives once, each case is a named golden.
One workflow takes the case’s facts as inputs:; --case <name> pins its
golden as <file>.<name>.golden.json, and --var binds that case’s inputs.
refund.nika
refund.bundle.json · the rule the cases prove
refund.bundle.json · the rule the cases prove
A refund within 30 days scores 6000 basis points, a returned item 4000, and
the bundle recommends at 8000. Its own fixtures carry the mandatory
contradictory case.
refund.bundle.json
refund.nika.fresh-returned.golden.json
"defer": a stale purchase and an item still out both fall
below the threshold. The rule is now guarded. Change one character in the
shared bundle, a weight of 6000 that becomes 600, and every case that
leaned on it goes red, naming the flip:
--update and let the golden diff carry it into review, which is where a
rubric change belongs.
What to know about cases
--varneeds--case. A bound value needs a golden of its own, sonika test refund.nika --var days=5without a case name exits 3 and says so. A file with no--casekeeps the single<file>.golden.json.- A case is a command line. The golden records the outcome, not the
inputs that produced it, so keep each case’s line in a script beside the
workflow (
decisions/cases.sh, without--update) and run that script in CI. One small workflow per case, with its facts inconst:, remains a valid layout when you prefer files to lines. - Pin the instant with
--at.--at <RFC 3339 instant>places the run at that time, and${{ run.scheduled_for }}reads it, as in a run fired for that slot. Without--atthat value is the current time, so a golden that pins it drifts on the next run. - Under the mock, the model leg is a constant.
nika testswaps everyinfer:to the mock provider, which synthesizes a schema-conformant value. A case that runsinfer:thennika:decidetherefore proves the law, and proves nothing about the extraction: the facts were handed to it. Keep the model out of the cases, bind the evidence as inputs, and prove the rubric. Extraction quality is a separate exercise against real models, where it costs money and is worth what it costs.
Exit codes · the CI contract
nika test refuses to run a dirty file the same way nika run does: a
workflow with check findings exits 2 before anything executes.
Wire it into CI
Offline and deterministic means no keys in CI, no flakes, no spend:.github/workflows/nika-test.yml (fragment)
*.golden.json files in the same directory as the workflows
they guard. A PR that edits a workflow either keeps its golden green or
shows the re-pinned golden in the diff — both are reviewable.
Related
- Concepts · Workflows: typed
inputs:in, typedoutputs:out — the callable contract the golden pins. - Reference · CLI: every
nika testflag. - Reference · Builtins: the
nika:decidebundle grammar, what the kernel enforces, and the receipt the cases above pin. - Guides · Troubleshooting: when
nika checkfindings block a test (exit 2).