feat: add read: file source — process local files as rows (D.1) - #72
Merged
Conversation
A native `read:` step turns local files into rows, so you can run a workflow
over your own data without a shell costume:
- glob / directory / a single .txt|.md -> one row per file: {path, name, content}
- .csv / .tsv -> one row per record, columns as fields
- .jsonl -> one row per line (parsed JSON)
`read:` is a new inferred step type (like prompt/run/jq). Format is inferred
from the extension and overridable with `format:`. The read path resolves
relative to CWD (your files); rows materialize to outputFolder like a transform,
so `from:`/`forEach:` consume them unchanged. Glob results are always sorted for
determinism; an empty match is an error.
New example process-my-files (read a folder of support tickets -> triage each),
live-verified on qwen3:1.7b. TDD throughout: fs helpers, ReadStep.Run,
step-type inference, preprocess validation, and a runner e2e.
Additive — shell + duckdb stays the tool for parquet/complex sources.
Follow-ups: unify the image glob into read: + an `image:` attach field (kills
imagePath); review/simplify the complex getSourceDataFromLine source-reading.
- Resolve the format dispatch once before the loop instead of re-switching per file; the loop body is now just loadFile(path). - Reuse transform's limitedEmit for write+count instead of hand-rolling WriteJSON + written++ in each format branch; emitJSONLines emits via the same callback. - fs.GlobFiles: plain `var files []string` instead of the `candidates[:0]` in-place reuse trick. - Update the getSourceDataFromLine doc comment to mention read steps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
read:— run a workflow over your own local filesThe headline of Block D (file-in): a native
read:step that turns local files into rows, so you can process your own data with no shell/download costume.Formats (by extension, overridable with
format:).txt/.md{path, name, content}.csv/.tsv.jsonlDesign
read:is a new inferred step type (likeprompt/run/jq).outputFolderlike a transform, sofrom:/forEach:consume them unchanged.Unlocked (the n8n-lite bridge)
Triage a folder of documents, enrich a CSV, extract from your
.md/.txt, re-process an existing.jsonl— batch LLM over your own data.Tests (TDD)
fshelpers (glob+sort+dir, CSV, text) — pure, goldenReadStep.Run— files / csv / jsonl / empty-globread+count/imagePath)forEach→ one row per fileFull suite +
golangci-lintclean. Additive — shell + DuckDB stays the tool for parquet/complex sources.Follow-ups (next PR): Phase 2 — unify the image glob into
read:+ animage:attach field on prompt steps (removesimagePath); and review/simplify the complexgetSourceDataFromLinesource-reading path.