Skip to content

[codex] add batched collection traversal#66

Merged
samlaycock merged 2 commits into
mainfrom
codex/each-batch-traversal
May 20, 2026
Merged

[codex] add batched collection traversal#66
samlaycock merged 2 commits into
mainfrom
codex/each-batch-traversal

Conversation

@samlaycock

Copy link
Copy Markdown
Owner

Summary

Adds fx.eachBatch(items, batchSize, f, options?) for processing collections in sequential batches while preserving result order and global item indexes.

Why

Issue #55 calls out that callers currently need to either collect all traversal work at once or manually chunk large inputs before using fx.each / fx.sequence. This helper gives the public API a first-class batch traversal path for large workloads that need to limit memory pressure and external API load.

Implementation

  • Adds synchronous batch-size validation with RangeError: batch size must be a positive finite integer.
  • Runs batches sequentially and applies the existing concurrency option inside each batch.
  • Preserves typed failure and dependency inference through the existing Task<A, E, R> style.
  • Streams batches from the input iterable rather than prebuilding all chunks.
  • Exports the helper through fx.eachBatch.

Docs and Examples

  • Updates docs/concurrency.md and docs/api-reference.md.
  • Adds fx.eachBatch usage to examples/collections.ts.
  • Switches examples/batch-job.ts to the new batch traversal helper.
  • Adds a changeset for the public API addition.

Validation

  • bun fmt
  • bun lint:fix
  • bun lint
  • bun test
  • bun typecheck
  • bun run examples:smoke

Closes #55.

Add fx.eachBatch for processing iterables in sequential batches while preserving result order and global item indexes.

Validate invalid batch sizes synchronously with a RangeError and reuse existing concurrency options within each batch.

Cover ordering, bounded concurrency, type inference, docs, examples, and changeset for Issue #55.
@samlaycock samlaycock marked this pull request as ready for review May 19, 2026 20:24
@greptile-apps

greptile-apps Bot commented May 19, 2026

Copy link
Copy Markdown

Greptile Summary

Adds fx.eachBatch(items, batchSize, f, options?) for processing iterables in sequential batches with optional per-batch concurrency, synchronous batch-size validation, and preserved result order and global item indexes.

  • batchIterable generator is constructed inside Effect.suspend, ensuring the iterator is recreated fresh on every run (covering retry/repeat scenarios); validateBatchSize is intentionally left outside Effect.suspend for synchronous throws.
  • The new export is wired through src/index.ts, documented in docs/concurrency.md and docs/api-reference.md, and exercised by four new runtime tests (order, concurrency bound, re-runnability, invalid-size rejection) plus a compile-time type assertion.
  • examples/batch-job.ts is migrated to the new helper; examples/collections.ts demonstrates it alongside existing traversal APIs.

Confidence Score: 5/5

The implementation is correct and the one previous concern about generator exhaustion on re-run is properly resolved with Effect.suspend.

The core logic is sound: the batchIterable generator is created inside Effect.suspend so it is freshly constructed on every execution, global item indexes are threaded through correctly, and the depth-1 flat() matches the double-nesting produced by the two Effect.forEach calls. Batch-size validation throws synchronously as advertised. Test coverage includes order preservation, concurrency bounding, task re-runnability, and invalid-input rejection.

No files require special attention.

Important Files Changed

Filename Overview
src/concurrency.ts Adds validateBatchSize, batchIterable generator, and eachBatch export; batchIterable is correctly constructed inside Effect.suspend so the iterator is fresh on each run; normalizeConcurrency is called once per batch rather than once per run (minor style issue)
src/index.ts Exports eachBatch from concurrency module — straightforward one-liner addition
test/runtime.test.ts Adds four tests: result order, per-batch concurrency bound, task re-runnability (covers the Effect.suspend fix), and invalid-batch-size rejection; coverage is solid
test/types.test.ts Adds compile-time type assertion that eachBatch returns boolean[] — correct
examples/batch-job.ts Replaces fx.sequence(ids.map(...)) with fx.eachBatch(ids, 100, ...) — straightforward migration to the new helper
examples/collections.ts Adds a batched task demonstrating eachBatch with per-batch concurrency; exports result correctly
docs/concurrency.md Documents batch traversal semantics, concurrency behaviour, and synchronous batch-size validation — accurate and consistent with implementation

Reviews (2): Last reviewed commit: "fix: make eachBatch task reusable" | Re-trigger Greptile

Comment thread src/concurrency.ts
Create the batch iterator inside Effect.suspend so repeated executions of the same eachBatch task reprocess re-iterable inputs instead of seeing an exhausted generator.

Keep batch-size validation synchronous and add regression coverage for running the same task twice.
@samlaycock samlaycock merged commit 230bd6a into main May 20, 2026
2 checks passed
@samlaycock samlaycock deleted the codex/each-batch-traversal branch May 20, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P2: Add batched collection traversal for large workloads

1 participant