> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nika.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL quarantine

> T2 chain · data engineering: bad batches degrade to a quarantine file instead of killing the pipeline.

> **T2 chain · data engineering**: the `on_error: recover:` example.
> The difference between « re-run the whole night » and « fix three
> rows tomorrow morning » is one block: bad data is **data**, not an
> exception.

## The job

The nightly batch dies at row 14,203 and the whole pipeline dies with
it. Here the malformed-CSV case recovers to an empty batch, the schema
gate splits good from bad, rejects land in a quarantine file with
their validation errors, and the good rows aggregate per currency,
in jq, deterministically.

## The shape

```mermaid theme={"system"}
flowchart LR
  empty_batch["empty_batch · nika:jq"]:::invoke
  raw["raw · nika:read"]:::invoke
  rows["rows · nika:convert"]:::invoke
  check["check · nika:validate"]:::invoke
  good["good · nika:jq"]:::invoke
  quarantine["quarantine · nika:write"]:::invoke
  report["report · nika:write"]:::invoke
  raw --> rows
  rows --> check
  rows --> good
  check --> good
  check --> quarantine
  good --> report
  classDef invoke fill:#22d3ee22,stroke:#22d3ee,color:#22d3ee
```

## The file

```yaml etl-quarantine.nika theme={"system"}
nika: etl-quarantine

const:
  batch_csv: "examples/fixtures/orders.csv"
permits:
  tools: ["nika:convert", "nika:jq", "nika:read", "nika:validate", "nika:write"]
  # The batch comes in at ONE path and leaves at TWO · each named exactly.
  # Note what is NOT granted: `out/**` would have handed the whole artifact
  # tree to a pipeline that only ever writes these two files.
  fs:
    read: ["examples/fixtures/orders.csv"]
    write: ["out/quarantine/orders-rejected.json", "out/reports/daily-totals.json"]

tasks:
  # A deterministic empty fallback · the recover target when parsing dies.
  empty_batch:
    invoke:
      tool: "nika:jq"
      args: { input: [], expression: "." }

  raw:
    invoke:
      tool: "nika:read"
      args: { path: "${{ const.batch_csv }}" }

  rows:
    with:
      csv: ${{ tasks.raw.output }}
    invoke:
      tool: "nika:convert"
      args:
        input: "${{ with.csv }}"
        from: csv
        to: json
        has_header: true
    on_error:
      recover: ${{ tasks.empty_batch.output }}    # malformed CSV → empty batch · pipeline lives

  check:
    with:
      rows: ${{ tasks.rows.output }}
    invoke:
      tool: "nika:validate"
      args:
        data: "${{ with.rows }}"
        format: json
        schema:
          type: array
          items:
            type: object
            required: [order_id, amount, currency]
            properties:
              order_id: { type: string }
              amount: { type: string }
              currency: { type: string, enum: [EUR, USD, GBP] }

  good:
    with:
      rows: ${{ tasks.rows.output }}
      valid: ${{ tasks.check.output.valid }}
    when: ${{ with.valid == true }}
    invoke:
      tool: "nika:jq"
      args:
        input: "${{ with.rows }}"
        expression: 'group_by(.currency) | map({currency: .[0].currency, orders: length, total: (map(.amount | tonumber) | add)})'

  quarantine:
    with:
      valid: ${{ tasks.check.output.valid }}
      errors: ${{ tasks.check.output.errors }}
    when: ${{ with.valid == false }}
    invoke:
      tool: "nika:write"
      args:
        path: out/quarantine/orders-rejected.json
        content: "${{ with.errors }}"
        create_dirs: true

  report:
    with:
      totals: ${{ tasks.good.output }}
    when: ${{ with.totals != null && size(with.totals) > 0 }}   # good is SKIPPED (null) on the quarantine path · guard before size()
    invoke:
      tool: "nika:write"
      args:
        path: out/reports/daily-totals.json
        content: "${{ with.totals }}"
        create_dirs: true

outputs:
  # The gate's verdict rows · [] on a clean batch, the validator's error
  # objects on a dirty one (measured both ways · never null).
  rejected: ${{ tasks.check.output.errors }}
  # Deliberately untyped · on the quarantine path `good` is SKIPPED and
  # this is null — a declared `type: array` would refuse the branch the
  # file exists to demonstrate.
  totals: ${{ tasks.good.output }}
```

## How it works

**1. recover: is the safety net**

`on_error: recover: ${{ tasks.empty_batch.output }}`: when the
CSV won't parse, the task yields the fallback instead of failing.
Downstream sees an empty batch, not a dead pipeline.

**2. The gate splits, when: routes**

`nika:validate` returns `{valid, errors}` · two `when:` branches
route the run: good rows aggregate, bad batches quarantine WITH
their error report.

**3. jq does the accounting**

`group_by(.currency)` + `add`: totals are arithmetic, not model
output. No LLM call anywhere in this workflow.

## Constructs you just used

| Construct | Where | Reference |
| - | - | - |
| `on_error: recover:` | `rows` | [Error model](https://docs.nika.sh/reference/error-codes.md) |
| `nika:validate` gate | `check` | [Builtins](https://docs.nika.sh/reference/builtins.md) |
| `when:` branch routing | `good` · `quarantine` | [Workflows](https://docs.nika.sh/concepts/workflows.md) |
| jq aggregation | `good` | [Builtins](https://docs.nika.sh/reference/builtins.md) |

## Run it

> [!NOTE]
>
> **What it needs:** The sample orders CSV. No model, no key, no network.

**1. Rehearse it**

```bash theme={"system"}
nika try etl-quarantine
```

The sample files come with it and nothing is written in your folder.

**2. Get your own copy**

Save the file above as `etl-quarantine.nika` in a new folder, then download the sample it reads:

```bash theme={"system"}
FIXTURES=https://raw.githubusercontent.com/supernovae-st/nika/v0.124.1/crates/nika-pack/pack/examples/fixtures
mkdir -p examples/fixtures
curl -fsSLo examples/fixtures/orders.csv "$FIXTURES/orders.csv"
```

**3. Check it, then run it**

```bash theme={"system"}
nika check etl-quarantine.nika
nika run etl-quarantine.nika
```

The run writes `out/quarantine/orders-rejected.json`.

The sample is deliberately dirty: one row carries a currency outside the enum, so the run takes the quarantine branch. Change that row from `CHF` to `EUR` and run again: the good branch writes `out/reports/daily-totals.json` instead.

## Make it yours

* Add a `nika:notify` task on the quarantine branch: data-quality alerts only when something was actually rejected.
* Chain [Release notes](https://docs.nika.sh/examples/release-notes.md)' pattern to post the daily totals to your team channel.
* Validate against your real row schema: the gate is just JSON Schema.

> [card] [Next · Release radar](https://docs.nika.sh/examples/release-radar.md)
>
> The feed-mode example: RSS/Atom parsed natively, diffed against
> last run's state.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.