Skip to main content
T2 chain · data engineering: the on_error: recover: example. The difference between « re-run the whole night » and « fix three rows tomorrow morning » is one block: bad data is data, not an exception.

The job

The nightly batch dies at row 14,203 and the whole pipeline dies with it. Here the malformed-CSV case recovers to an empty batch, the schema gate splits good from bad, rejects land in a quarantine file with their validation errors, and the good rows aggregate per currency, in jq, deterministically.

The shape

The file

etl-quarantine.nika

How it works

1

recover: is the safety net

on_error: recover: ${{ tasks.empty_batch.output }}: when the CSV won’t parse, the task yields the fallback instead of failing. Downstream sees an empty batch, not a dead pipeline.
2

The gate splits, when: routes

nika:validate returns {valid, errors} · two when: branches route the run: good rows aggregate, bad batches quarantine WITH their error report.
3

jq does the accounting

group_by(.currency) + add: totals are arithmetic, not model output. No LLM call anywhere in this workflow.

Constructs you just used

Run it

What it needs: The sample orders CSV. No model, no key, no network.
1

Rehearse it

The sample files come with it and nothing is written in your folder.
2

Get your own copy

Save the file above as etl-quarantine.nika in a new folder, then download the sample it reads:
3

Check it, then run it

The run writes out/quarantine/orders-rejected.json.The sample is deliberately dirty: one row carries a currency outside the enum, so the run takes the quarantine branch. Change that row from CHF to EUR and run again: the good branch writes out/reports/daily-totals.json instead.

Make it yours

  • Add a nika:notify task on the quarantine branch: data-quality alerts only when something was actually rejected.
  • Chain Release notes’ pattern to post the daily totals to your team channel.
  • Validate against your real row schema: the gate is just JSON Schema.

Next · Release radar

The feed-mode example: RSS/Atom parsed natively, diffed against last run’s state.