Skip to content

feat(genext2fs): add @deroll/genext2fs, Node.js bindings for xgenext2fs - #194

Merged
tuler merged 2 commits into
prerelease/v2from
claude/genext2fs-nodejs-binding-5ij7ae
Aug 4, 2026
Merged

feat(genext2fs): add @deroll/genext2fs, Node.js bindings for xgenext2fs#194
tuler merged 2 commits into
prerelease/v2from
claude/genext2fs-nodejs-binding-5ij7ae

Conversation

@tuler

@tuler tuler commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Binds cartesi/genext2fs to Node.js as an N-API addon, so a tar archive can be turned into an ext2 drive image from inside a Node process instead of shelling out to a container or an installed CLI.

import { tarToExt2 } from "@deroll/genext2fs";

await tarToExt2("rootfs.tar", "rootfs.ext2", { blockSize: 4096, faketime: true });

Approach

xgenext2fs is a command line program, not a library: a single 100 KB C file that parses argv, prints to stderr and calls exit() on every error. Rather than fork it, both upstream projects are vendored as git submodules and compiled unmodified by node-gyp:

Submodule Pin Why
deps/genext2fs v1.5.6 the generator itself
deps/libarchive v3.8.9 the Cartesi fork's add2fs_from_tarball() calls archive_read_* unconditionally, so libarchive is a hard requirement (the README's "builtin tarball parser" note is stale)

Three pieces make that work:

  • src/xgenext2fs_lib.c textually includes xgenext2fs.c with main renamed, exit() redirected through longjmp(), and stdout/stderr redirected into temporary files — which is what turns the CLI into a callable function whose failures unwind and whose diagnostics come back as strings. Upstream is not patched at all.
  • src/libarchive_formats.c narrows archive_read_support_format_all/support_filter_all to tar + none, so only 16 of libarchive's 217 translation units are compiled and the addon links against nothing but libc. No zlib/bz2/lzma, no system libarchive; gzip is inflated in JS instead.
  • config/ holds hand-written replacements for the two generated config.h files, since node-gyp cannot run ./configure. POSIX features are asserted directly, with __APPLE__/__linux__ branches for the few that genuinely differ.

Submodules over copying the sources: provenance is exact, bumps are a git checkout, and it matches the existing @deroll/cmio pattern. The published tarball carries only the files actually compiled.

Because xgenext2fs keeps parser state in globals and chdir()s while reading directory layers, calls are serialized behind a mutex in the addon; async calls still generate off the main thread.

API

Four functions:

  • tarToExt2(tar, output, options?) — the main use case, archive as a path or bytes, gzip inflated transparently
  • createImage(output, options?) — images assembled from several layers (tarball / directory / devtable, each optionally at a path inside the image)
  • tarToExt2Sync / createImageSync — the same work on the calling thread

There is deliberately no raw-argv escape hatch: of xgenext2fs' 22 long options, 19 map onto ImageOptions or a layer type, --version is the exported version, and --help is meaningless in a library. Auditing that list also corrected CreatorOsmasix is not recognized by lookup_creator_os() and silently falls back to Linux, GNU is an accepted alias for hurd, and a raw number is accepted for an OS the tool has no name for.

Errors reject with the tool's own diagnostic, carrying status, stdout and stderr.

Two things worth a look before merging

License. xgenext2fs is GPL-2.0-only (not "or later") and this package compiles it in, so @deroll/genext2fs is GPL-2.0-only — unlike every other package in the repo, which are Apache-2.0. Called out in package.json, the package README, the docs and CLAUDE.md, but whether that belongs in this monorepo is a call for a human.

Upstream sizing bug. With no -b, xgenext2fs under-estimates the image by 3–13 blocks for some archives and dies mid-build with couldn't allocate a block (no free space). The standalone CLI does the same — which is why Cartesi's tooling always passes an explicit -b. createImage absorbs it: on that specific failure it retries with a size derived from the failed attempt (the partial file is truncated to blocks * blockSize, so the estimate is readable from disk). Deterministic, so images stay reproducible. autoSize: false opts out and sizeInBlocks pins it.

Also in this PR

  • Release workflow: a genext2fs-prebuild matrix (linux/macOS × x64/arm64) gated on the same "is this version already on npm" check as cmio/cm, with prebuilds collected into the package before publish. Unlike its siblings this job needs no system packages.
  • Docs: a genext2fs section (intro, API, options) plus top-nav and sidebar entries.
  • Root README: a note that the native bindings need --recurse-submodules.
  • A changeset, and the package registered in .changeset/pre.json.

Verification

19 tests pass — e2fsck-clean images, byte-identical output across runs with faketime, gzip and Uint8Array input matching path input, the full option surface in a single call, concurrent conversions, autoSize growth versus autoSize: false, error status/stderr, and the addon staying usable after a failed run. I also packed the tarball, extracted it into a clean directory and confirmed the from-source build works end to end, which is the real check on the files list and the submodule strategy. bun run lint, bun run build and bun run test are green across the workspace, and the docs site builds with the twoslash snippets typechecked.

Not included: an entry in apps/examples — happy to add one if you want it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZ516LsVnnu4ySXKkXsNbB

@changeset-bot

changeset-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 927b1ba

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@deroll/genext2fs Minor
@deroll/docs Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
deroll Ready Ready Preview Aug 4, 2026 8:45pm
deroll-explorer Ready Ready Preview Aug 4, 2026 8:45pm

@tuler
tuler changed the base branch from main to prerelease/v2 August 4, 2026 04:41
@tuler tuler changed the title chore: enter alpha pre-release feat(genext2fs): add @deroll/genext2fs, Node.js bindings for xgenext2fs Aug 4, 2026
claude added 2 commits August 4, 2026 20:41
Binds cartesi/genext2fs to Node.js as an N-API addon, so a tar archive can
be turned into an ext2 drive image from inside a Node process instead of
shelling out to a container or an installed CLI.

xgenext2fs is a command line program, not a library: a single 100 KB C file
that parses argv, prints to stderr and calls exit() on every error. Rather
than fork it, both upstream projects are vendored as git submodules and
compiled unmodified:

  deps/genext2fs   cartesi/genext2fs v1.5.6
  deps/libarchive  libarchive v3.8.9, required by the fork's tar reader

src/xgenext2fs_lib.c textually includes xgenext2fs.c with main renamed,
exit() redirected through longjmp() and stdout/stderr redirected into
temporary files, which is what turns the CLI into a callable function whose
failures unwind and whose diagnostics become strings. Only the 16 libarchive
translation units needed to read an uncompressed tar are compiled, and
src/libarchive_formats.c narrows support_format_all/support_filter_all to
them, so the addon links against nothing but libc. Neither build system runs
under node-gyp, so the two generated config.h files are replaced by the
hand-written ones under config/.

The TypeScript layer exposes tarToExt2/tarToExt2Buffer for the main use
case, createImage for multi-layer images, and genext2fs(args) as a raw argv
escape hatch, each with a blocking counterpart. Options map one to one onto
the CLI flags, gzipped archives are inflated in JS, and calls are serialized
behind a mutex because xgenext2fs keeps parser state in globals.

xgenext2fs' own size estimate comes out a few blocks short for some
archives; createImage retries deterministically with a slightly larger
explicit size rather than surfacing a mid-build failure.

Note the package is GPL-2.0-only, inherited from xgenext2fs, unlike the
Apache-2.0 packages around it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JZ516LsVnnu4ySXKkXsNbB
tarToExt2Buffer/tarToExt2BufferSync went in on the premise that they saved
a round trip to disk. They did not: xgenext2fs only ever writes to a file,
so they built into a temporary directory and read the result back, which
is exactly what a caller can do in two lines when they want the bytes.

The genext2fs(args) raw-argv escape hatch existed for options the typed
surface did not model. There are none: of the 22 long options, 19 map onto
ImageOptions or a layer type, --version is the exported `version`, and
--help is meaningless in a library. Checking that list also turned up two
mistakes in CreatorOs — "masix" is not recognized by lookup_creator_os and
falls back to Linux, "GNU" is an accepted alias for hurd, and a raw number
is accepted for an OS the tool has no name for.

What is left is tarToExt2/createImage plus their blocking counterparts.
Tests moved onto the remaining API rather than being dropped, and gained
coverage the escape hatch used to provide: the full option surface in one
call, colon rejection in layer paths, error status/stderr, and autoSize
growth versus autoSize: false.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JZ516LsVnnu4ySXKkXsNbB
@tuler
tuler force-pushed the claude/genext2fs-nodejs-binding-5ij7ae branch from b3db476 to 927b1ba Compare August 4, 2026 20:43
@tuler
tuler merged commit c60f2c4 into prerelease/v2 Aug 4, 2026
5 checks passed
@tuler
tuler deleted the claude/genext2fs-nodejs-binding-5ij7ae branch August 4, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants