Swarm FS

Swarm FS is an upload layer on top of Bee that stamps chunks client-side and tracks which slots in a postage batch are occupied by each file. This makes it possible to "delete" a file by reclaiming its slots for future uploads — something Swarm has no native concept of.

Installation

npm install --global swarm-fs

Usage

Environment variables can be set in your shell profile for global use, or in a local .env file if you prefer per-project configuration:

export SWARMFS_UPLOAD_URL="http://localhost:1633/chunks"
export SWARMFS_SIGNER="<private key hex>"
export SWARMFS_BATCH_ID="<batch id hex>"
export SWARMFS_BATCH_DEPTH=<depth of your batch>
export SWARMFS_REDUNDANCY_LEVEL=0  # 0=none, 1=MEDIUM, 2=STRONG, 3=INSANE, 4=PARANOID

# Upload a file or directory and print the manifest root hash
swarm-fs upload <file|dir>

# Upload with client-side encryption
swarm-fs upload <file|dir> --encrypt

# Upload with erasure coding for redundancy (levels 1–4: MEDIUM, STRONG, INSANE, PARANOID)
swarm-fs upload <file|dir> --redundancy=1

# Upload with both encryption and erasure coding
swarm-fs upload <file|dir> --encrypt --redundancy=2

# List all tracked files and manifests
swarm-fs list

# Delete a file or manifest by root hash, reclaiming its slots
swarm-fs delete <root hash>

# Show slot usage and most utilized bucket
swarm-fs status

# Benchmark chunk splitting speed (no upload, no state changes)
swarm-fs bench:split <file|dir>

# Benchmark chunk splitting + stamp signing speed (no upload, no state changes)
# Requires SWARMFS_SIGNER, SWARMFS_BATCH_ID, and SWARMFS_BATCH_DEPTH
swarm-fs bench:sign <file|dir>

State is stored in ~/.swarmfs/, with files named by the first 8 hex characters of the batch ID (e.g. swarmfs-a1b2c3d4.free, swarmfs-a1b2c3d4.db). Multiple postage batches can be used independently by switching SWARMFS_BATCH_ID.

Design

Swarm FS stamps chunks client-side, which means it controls which (bucket, slot) each chunk is assigned to before sending the pre-signed chunk to the Bee API. This makes it possible to maintain a local index that tracks exactly which slots are occupied by each file, and to deliberately target previously freed slots when uploading new files.

All state lives under a .swarmfs/ directory.

swarmfs.free — SlotMap

A bitmap tracking slot occupancy across the entire batch. Each bit represents one slot: 0 = free, 1 = occupied. Pre-allocated at init time as all zeros, so the full batch capacity is available from the start.

per bucket (65536 entries, indexed by bucket number):
  [slot bitmap: slotsPerBucket / 8 bytes]

The fixed-size layout allows O(1) access by bucket index. Allocating a slot scans the bucket's bitmap for the first 0 bit and sets it to 1. Freeing a slot clears the corresponding bit. File size is determined by batch depth: for depth 24 (256 slots per bucket), the file is 2 MB.

swarmfs.db — FileRegistry

A SQLite database tracking all uploaded files, their root chunk hash, and the slots they occupy.

CREATE TABLE files (
    id               INTEGER PRIMARY KEY AUTOINCREMENT,
    path             TEXT NOT NULL,
    root_hash        BLOB NOT NULL,    -- 32 bytes
    chunks           BLOB NOT NULL,    -- repeated [bucket uint16, slot uint16] pairs
    kind             TEXT NOT NULL DEFAULT 'file',  -- 'file' or 'manifest'
    redundancy_level INTEGER NOT NULL DEFAULT 0     -- 0=none, 1=MEDIUM, 2=STRONG, 3=INSANE, 4=PARANOID
);
CREATE INDEX idx_root_hash ON files(root_hash);

Chunk refs are packed as 4 bytes each ([bucket uint16][slot uint16]), so overhead is approximately 4 bytes per chunk — about 0.1% of the file size — regardless of file size. The index on root_hash makes lookup and deletion O(log N) in the number of tracked files.

Manifest upload flow

upload <dir> packages a directory as a Swarm website so it can be browsed via the Bzz gateway (/bzz/<root-hash>/).

It builds a Mantaray v0.2 trie — the same binary format Bee uses natively:

1. Pass 1 — upload content: walk the directory, split each file into chunks, stamp and upload each chunk, collect the root hash per file
2. Pass 2 — build the trie: construct a Mantaray node for every file path (no leading slash, e.g. index.html), plus a '/' metadata node carrying website-index-document if index.html is present — this is what Bee reads to serve the index page
3. Upload the trie: serialize each trie node into a chunk, stamp and upload it, record the manifest root hash in swarmfs.db

After upload, the directory is accessible at <gateway>/bzz/<root-hash>/ and <gateway>/bzz/<root-hash>/path/to/file.

Upload flow

1. Split the file into chunks client-side
2. For each chunk, allocate a slot from swarmfs.free for the matching bucket
3. Sign the stamp with the chosen (bucket, slot) and send the pre-signed chunk to Bee
4. Wrap the file in a single-entry Mantaray manifest (with a website-index-document pointer) so the file is directly browseable at <gateway>/bzz/<root-hash>/
5. Record the manifest root hash and all (bucket, slot) pairs in swarmfs.db

Deletion flow

1. Look up the file's chunk list in swarmfs.db
2. Return all its (bucket, slot) pairs to swarmfs.free under their respective buckets
3. Remove the file entry from swarmfs.db
4. Optionally upload tombstone chunks to overwrite the slots on the network