Skip to content

feat: per-row file output — write a folder of documents, not just a table - #76

Merged
mirpo merged 6 commits into
mainfrom
feat/per-row-file-output
Jul 25, 2026
Merged

feat: per-row file output — write a folder of documents, not just a table#76
mirpo merged 6 commits into
mainfrom
feat/per-row-file-output

Conversation

@mirpo

@mirpo mirpo commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

A write step's source now picks its mode: from: writes one file with every row (a dataset or report — today's behavior), forEach: writes one file per row (a folder of documents). That closes the content half of the office loop — N rows → N documents, next to N rows → 1 table.

  - name: reply_digest
    from: drafts                          # → one replies.md table of all drafts
    write: replies.md

  - name: reply_files
    forEach: drafts                       # → one file per draft
    write: replies/{{.item.subject}}.md   # path is a template, rendered per row
    content: "{{.item.reply}}"            # body is raw text, not JSON
  • content: is the file body as raw text — that is what makes the output a document rather than a record. Omit it and the single row is serialized by extension instead (.json, .csv, …), one row per file, reusing the same writers as aggregate mode.
  • The path template may nest (out/{{.item.kind}}/{{.item.id}}.md); missing directories are created.
  • Exactly one of from/forEach is now required on a write step, and a per-row path without a template is rejected — otherwise every row would overwrite the same file. Both templates are parsed at config time so a typo fails in validate, not mid-run.

Filename safety, split by origin

A path rendered from model output is a real hazard, so slashes are treated differently depending on where they come from: the path is rendered against a copy of the row whose strings are already filename-safe, so a slash you write in the template still nests directories, while a slash inside the data cannot redirect the file.

This came out of a failing test — sanitizing the final path segment (the original plan) is too late, because filepath.Split has already turned a data slash into a directory.

Also: a name that renders empty falls back to the row number, and two rows rendering the same name get numbered (-2) with a warning rather than silently clobbering each other.

Commits

  1. feat: config for per-row write steps (forEach + content)
  2. feat: fs.SanitizeFilename — both slash kinds replaced regardless of platform, so a name written on one OS cannot escape its directory on another
  3. feat: per-row fan-out in the write step (+ extracted serializeRows, shared by both modes)
  4. test: e2e per-row pipeline
  5. docs: inbox-triage emits per-row reply drafts + README

Verification

  • Full go test ./... + golangci-lint green; datamatic validate passes on every example.
  • New tests cover: all 10 config/guard cases, the sanitizer, per-row content bodies, sanitize+collision+empty-name fallback, content-less serialization, and an end-to-end read → LLM → folder-of-documents run.
  • Live-run on Ollama (qwen3:1.7b): write exported 4 rows to 4 file(s), dataset/replies/ holding one file per ticket named from its subject (including one with spaces and a comma), each body the reply text — alongside the aggregate board.csv and replies.md.

mirpo added 6 commits July 25, 2026 12:05
A write step now takes exactly one source, and that source picks the mode:
'from' exports every row into one aggregate file (as before), 'forEach'
renders the write path per row to emit one file each. 'content' supplies the
file body as raw text and is only valid in per-row mode.

Per-row paths stay templates through preprocessing — they are rendered at
runtime — but both templates are parsed up front so a typo fails at config
time. A per-row path without a template is rejected, since every row would
overwrite the same file.
A per-row write path is rendered from model output, so a field containing a
slash or a colon would silently redirect the file. Both slash kinds are
replaced regardless of platform, so a name written on one OS cannot escape
its directory on another; an unusable name returns empty and the caller
falls back to the row index.
A write step with forEach renders its path per row and writes one file each —
the folder-of-documents mode. With content: the body is the rendered template
as raw text; without it, the single row is serialized by extension, reusing
the same writers as the aggregate mode (extracted as serializeRows).

Filename safety is split by origin: the path is rendered from a copy of the
row whose strings are already filename-safe, so a slash written in the
template still nests directories while a slash inside the data cannot
redirect the file. A name that renders to nothing falls back to the row
number, and two rows rendering one name are numbered rather than clobbered.
Read rows from a CSV, generate a body per row, and emit one file per row —
proving the read → LLM → folder-of-documents loop end to end.
Adds a reply_files step so the example shows both write modes side by side:
replies.md aggregates every draft into one table, replies/<subject>.md is one
editable document per ticket. Documents the mode split in README.
content: is written verbatim, so a quoted template produced files with no
trailing newline — awkward for text files (cat runs them together, git flags
'no newline at end of file'). The block scalar adds it, and the example now
shows that idiom; quoting still gives byte-exact control.
@mirpo
mirpo merged commit e4b634f into main Jul 25, 2026
3 checks passed
@mirpo
mirpo deleted the feat/per-row-file-output branch July 25, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant