> ## Documentation Index > Fetch the complete documentation index at: https://docs.nika.sh/llms.txt > Use this file to discover all available pages before exploring further. # Corpus digest > T4 epic · research: plan a focus per file · card each document in a fresh window · one sourced brief. Context folding. > **T4 epic · research / any corpus one window cannot hold**: context > folding. A fast pass plans a focus per file **from the paths alone**, > each document is carded in its **own fresh window**, jq assembles and > ranks the deck, a thinking pass answers from the cards. No model > window ever holds two documents. ## The job « What do these forty documents actually establish? » is either one giant prompt that rots halfway through, or it is this pipeline: the corpus stays in the dataflow, each model call sees one slice, and the synthesis reads a deck of typed cards it can actually hold. ## The shape ```mermaid theme={"system"} flowchart LR files["files · nika:glob"]:::invoke plan["plan · typed"]:::infer texts["texts · ∥ for_each · nika:read"]:::invoke slices["slices · nika:jq"]:::invoke cards["cards · ∥ for_each · typed"]:::infer deck["deck · nika:jq"]:::invoke brief["brief · thinking"]:::infer files --> plan files --> texts files --> slices texts --> slices plan --> slices slices --> cards slices --> deck cards --> deck deck --> brief classDef infer fill:#5b8cff22,stroke:#5b8cff,color:#5b8cff classDef invoke fill:#22d3ee22,stroke:#22d3ee,color:#22d3ee ``` ## The file ```yaml corpus-digest.nika.yaml theme={"system"} nika: v1 workflow: id: corpus-digest description: "glob a corpus → plan a focus per file → card each doc in a fresh window → jq ranks the deck → one sourced brief" run: clock: system # the `timeout:` deadlines below ride the wall clock · said out loud (see 03) # A small NON-thinking seat, on purpose: two of the four model stages are # forms to fill, and a thinking seat DELIBERATES on forms (measured: the # corpus default qwen3.5:4b burned caps of 600, 2000 and 3000 tokens # reasoning before the JSON and never closed it · NIKA-INFER-002). One # resident seat also spares the model-swap thrash of per-task overrides # on a single GPU (measured: alternating two seats sent every stage cold # and timed the synthesis out). Upgrade path: point `brief` at a # thinking seat via its own `model:` when your machine holds both. model: ollama/llama3.2:3b # local · zero key · `--model mock/echo` rehearses the whole file inputs: question: type: string default: "What do these documents establish, and where do they disagree?" description: "The question the brief must answer · free text (--var question=…)" const: # The committed rehearsal tree — two small markdown files. Point this at # your real corpus and change `permits.fs.read` in the same edit. corpus_root: "./examples/fixtures/docs" permits: # Read-only by construction: no `nika:write`, no `fs.write`, no network. # The digest's product is its typed `outputs:` — this file cannot touch # anything, which is exactly what you want from a tool that reads your # documents. tools: ["nika:glob", "nika:jq", "nika:read"] fs: # `nika:glob` opens the ROOT of its pattern, so the bound covers the # directory and everything under it (see localization-factory for the # long version of this note). read: ["./examples/fixtures/docs/**"] tasks: # ── the corpus is discovered, never declared ──────────────────────── files: invoke: tool: "nika:glob" args: pattern: "${{ const.corpus_root }}/**/*.md" # ── plan · a focus per file, from the MAP, never the territory ───── # The planner sees paths, not contents: filenames are cheap tokens and # they are enough to divide the work. Nothing the model says here can # read a file, and nothing it says below can name one. plan: with: question: ${{ inputs.question }} files: ${{ tasks.files.output }} timeout: "8m" # a local seat may be slow · bounded, not instant infer: max_tokens: 3000 # the form is short · the cap is margin, not hope prompt: | Assign one focus of 3 to 8 words to each file path. The focuses serve this question: ${{ with.question }} Paths · ${{ with.files }} Return the assignments only. schema: # `additionalProperties: false` on every object node — the plan is # read by jq below, so its shape is a promise, not a hope. type: object additionalProperties: false required: [assignments] properties: assignments: type: array items: type: object additionalProperties: false required: [path, focus] properties: path: { type: string } focus: { type: string } # ── read every file · disk wave, no model in sight ───────────────── texts: with: files: ${{ tasks.files.output }} for_each: ${{ with.files }} max_parallel: 8 # local disk · the only cost is file handles invoke: tool: "nika:read" args: { path: "${{ item }}" } # ── the join · discovered path + planned focus + text ────────────── # The glob list is the spine: `texts` is index-aligned with `files`, so # transpose pairs them, and the plan joins BY PATH through a lookup # table. A path the planner invented matches nothing; a file the planner # skipped falls back to the question itself. Model output steers focus — # it never steers which files exist or get read. slices: with: files: ${{ tasks.files.output }} texts: ${{ tasks.texts.output }} assignments: ${{ tasks.plan.output.assignments }} question: ${{ inputs.question }} invoke: tool: "nika:jq" args: input: ["${{ with.files }}", "${{ with.texts }}", "${{ with.assignments }}"] expression: >- . as [$files, $texts, $plan] | ($plan | map({ key: .path, value: .focus }) | from_entries) as $focus | [$files, $texts] | transpose | map({ path: .[0], focus: ($focus[.[0]] // "${{ with.question }}"), text: .[1] }) # ── card each document · one FRESH window per slice ──────────────── # This is the fold. Each iteration's prompt holds exactly one document # and its focus — never a sibling, never the history. Ten documents cost # ten small windows, not one window ten documents deep. cards: with: slices: ${{ tasks.slices.output }} for_each: ${{ with.slices }} max_parallel: 3 # rate-limit the provider, not the disk fail_fast: false # finish the batch · one bad doc is not a batch failure timeout: "8m" # per ITERATION · same reasoning as `plan` on_error: recover: null # null holds the index open so the zip below stays aligned infer: max_tokens: 3000 # a card, not an essay prompt: | Fill the card for ONE document. Claims are one sentence each, quotes are verbatim and short, relevance is 0 to 5 for the focus. Return the card only. Focus · ${{ item.focus }} Document (${{ item.path }}) · ${{ item.text }} schema: # No `path` in the card: the model summarizes, it does not get to # say which file it was reading. Identity is re-attached by # transpose below — deterministically. type: object additionalProperties: false required: [claims, quotes, relevance] properties: claims: type: array items: { type: string } quotes: type: array items: { type: string } relevance: type: integer # ── the deck · deterministic merge, rank, and prune ──────────────── # jq decides; the model explained. Failed cards leave by VALUE (the # nulls `recover:` held in place), the rest are re-keyed to their path # and sorted by the model's own relevance score — but the SORT is ours. deck: with: slices: ${{ tasks.slices.output }} cards: ${{ tasks.cards.output }} invoke: tool: "nika:jq" args: input: ["${{ with.slices }}", "${{ with.cards }}"] expression: >- transpose | map(select(.[1] != null)) | map({ path: .[0].path, claims: .[1].claims, quotes: .[1].quotes, relevance: .[1].relevance }) | sort_by(-.relevance) # ── the brief · the synthesis reads the deck, never the corpus ───── brief: with: question: ${{ inputs.question }} deck: ${{ tasks.deck.output }} # The synthesis only runs on a NON-EMPTY deck. `recover: null` above # keeps a failed card from aborting the batch — but if EVERY card # died, an ungated brief would synthesize from nothing and sound # confident doing it. The gate turns that silent hazard into a # visible skip. when: ${{ size(with.deck) > 0 }} timeout: "12m" # judgment gets patience · still bounded infer: max_tokens: 2000 prompt: | Question · ${{ with.question }} Deck of source cards · ${{ with.deck }} Answer the question from the cards alone. Cite the `path` of every card you lean on. Where cards disagree, say so — do not resolve a conflict the sources did not resolve. thinking: # A reasoning budget, honoured by seats that support extended # thinking and ignored by the ones that do not — declaring it # never fails a run. enabled: true budget_tokens: 4000 # Both the brief and the deck, so the fold is inspectable: the deck is # everything the synthesis was allowed to see. Read with `--output json`. outputs: brief: ${{ tasks.brief.output }} deck: value: ${{ tasks.deck.output }} description: "The ranked cards the brief was written from · one per document that survived" files: value: ${{ tasks.files.output }} description: "Every path the glob discovered · the spine of the fold" ``` ## How it works Ten documents cost ten small windows plus one deck-sized synthesis, not one window ten documents deep. The fold is the point: adding a file adds an iteration, it never grows a prompt. The planner sees paths, never contents, and its assignments join the glob list **by path** in jq. A hallucinated path joins nothing and drives nothing; a skipped file falls back to the question itself. The card schema has no `path` field on purpose: the model summarizes, transpose re-attaches identity deterministically. The deck the synthesis reads is everything it was allowed to see, and the typed `outputs:` expose that deck for inspection. ## Constructs you just used | Construct | Where | Reference | | ------------------------------------------- | ---------------- | -------------------------------------------------- | | `for_each` over a planned collection | `cards` | [YAML syntax](/reference/yaml-syntax) | | join-by-key in `nika:jq` (`from_entries`) | `slices` | [Builtins](/reference/builtins) | | `fail_fast: false` + `recover: null` fan-in | `cards` → `deck` | [YAML syntax · `on_error`](/reference/yaml-syntax) | | `thinking:` budget on the synthesis | `brief` | [The 4 verbs](/concepts/verbs) | ## Make it yours * Point `const.corpus_root` and the matching `permits.fs.read` entry at your real folder · the two move together (`NIKA-AUTH-007`). * Chunk instead of file-split: for documents that are each too big for a window, add a jq task that slices `texts` into sections and fan out over the sections · the fold works at any grain. * Feed the deck onward: the typed `outputs:` make this callable, so a parent workflow can consume `{brief, deck}` as a contract · pair it with [Deep research brief](/examples/deep-research-brief) to digest what the agent fetched. The same plan-then-execute skeleton with an agent middle: when the work needs judgment calls between fetches, the loop earns its place.