feat: per-row file output — write a folder of documents, not just a table - #76
Merged
Conversation
A write step now takes exactly one source, and that source picks the mode: 'from' exports every row into one aggregate file (as before), 'forEach' renders the write path per row to emit one file each. 'content' supplies the file body as raw text and is only valid in per-row mode. Per-row paths stay templates through preprocessing — they are rendered at runtime — but both templates are parsed up front so a typo fails at config time. A per-row path without a template is rejected, since every row would overwrite the same file.
A per-row write path is rendered from model output, so a field containing a slash or a colon would silently redirect the file. Both slash kinds are replaced regardless of platform, so a name written on one OS cannot escape its directory on another; an unusable name returns empty and the caller falls back to the row index.
A write step with forEach renders its path per row and writes one file each — the folder-of-documents mode. With content: the body is the rendered template as raw text; without it, the single row is serialized by extension, reusing the same writers as the aggregate mode (extracted as serializeRows). Filename safety is split by origin: the path is rendered from a copy of the row whose strings are already filename-safe, so a slash written in the template still nests directories while a slash inside the data cannot redirect the file. A name that renders to nothing falls back to the row number, and two rows rendering one name are numbered rather than clobbered.
Read rows from a CSV, generate a body per row, and emit one file per row — proving the read → LLM → folder-of-documents loop end to end.
Adds a reply_files step so the example shows both write modes side by side: replies.md aggregates every draft into one table, replies/<subject>.md is one editable document per ticket. Documents the mode split in README.
content: is written verbatim, so a quoted template produced files with no trailing newline — awkward for text files (cat runs them together, git flags 'no newline at end of file'). The block scalar adds it, and the example now shows that idiom; quoting still gives byte-exact control.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A write step's source now picks its mode:
from:writes one file with every row (a dataset or report — today's behavior),forEach:writes one file per row (a folder of documents). That closes the content half of the office loop — N rows → N documents, next to N rows → 1 table.content:is the file body as raw text — that is what makes the output a document rather than a record. Omit it and the single row is serialized by extension instead (.json,.csv, …), one row per file, reusing the same writers as aggregate mode.out/{{.item.kind}}/{{.item.id}}.md); missing directories are created.from/forEachis now required on a write step, and a per-row path without a template is rejected — otherwise every row would overwrite the same file. Both templates are parsed at config time so a typo fails invalidate, not mid-run.Filename safety, split by origin
A path rendered from model output is a real hazard, so slashes are treated differently depending on where they come from: the path is rendered against a copy of the row whose strings are already filename-safe, so a slash you write in the template still nests directories, while a slash inside the data cannot redirect the file.
This came out of a failing test — sanitizing the final path segment (the original plan) is too late, because
filepath.Splithas already turned a data slash into a directory.Also: a name that renders empty falls back to the row number, and two rows rendering the same name get numbered (
-2) with a warning rather than silently clobbering each other.Commits
feat:config for per-row write steps (forEach + content)feat:fs.SanitizeFilename— both slash kinds replaced regardless of platform, so a name written on one OS cannot escape its directory on anotherfeat:per-row fan-out in the write step (+ extractedserializeRows, shared by both modes)test:e2e per-row pipelinedocs:inbox-triage emits per-row reply drafts + READMEVerification
go test ./...+golangci-lintgreen;datamatic validatepasses on every example.qwen3:1.7b):write exported 4 rows to 4 file(s),dataset/replies/holding one file per ticket named from its subject (including one with spaces and a comma), each body the reply text — alongside the aggregateboard.csvandreplies.md.