feat(storage): add chunked serialization and parallel deserialization

[geerniman]

Loading a large index is I/O bound: the sequential Deserialize path reads and decompresses the file with a single thread, leaving multi-core machines and high-bandwidth storage underutilized. Split the index body into independently readable frames, record their physical placement in a chunked_layout footer key, and restore them concurrently through a caller-supplied positioned reader and thread pool.

Serialize(SerializeWriter&, chunk_size) writes each component either as a whole frame or as a head / chunked io-data / tail triple, while SerializeWholeBody writes the same components as one continuous body. ParallelDeserialize pre-allocates the io extents on the main thread, fills the frames from the pool without locking because tasks touch disjoint ranges, and finalizes on the main thread. Layout coverage and per-frame byte counts are validated, so a corrupted frame fails with an error instead of corrupting memory. Uncompressed chunked files stay readable by the sequential path; compressed ones are rejected explicitly rather than misparsed.

Add ReserveIO / GetIOSize / WriteRaw / DeserializeTail hooks across the datacell, layout and io abstractions, skip zero-filling in ResizeForOverwrite so the reserve step does not fault in every page from the main thread, and extend the parallel fill to file-backed io types. reader_io stays excluded because its write path is a no-op; its components fall back to whole frames.

1M x 1024 vectors with sq8 base and fp32 reorder load 2.7x-5.2x faster at 16 threads across block_memory, memory, mmap and buffer io, returning results identical to the in-memory index query by query.

Change Type

• Bug fix
• New feature
• Improvement/Refactor
• Documentation
• CI/Build/Infra

Linked Issue

• Fixes: [improve](hgraph): parallel chunked index loading to speed up deserialization #2747

What Changed

• Add chunked serialization: Serialize(SerializeWriter&, chunk_size) writes each
  component as a whole frame or a head / chunked io-data / tail triple, and records the
  physical placement in a chunked_layout footer key; SerializeWholeBody writes the
  same components as one continuous body.
• Add ParallelDeserialize: restores an index from a chunked file with one task per
  frame via a caller-supplied DeserializeReader and ThreadPool. Extents are
  pre-allocated on the main thread and filled lock-free (disjoint ranges); a probe path
  also loads existing layout-less uncompressed files concurrently.
• Add SerializeWriter / DeserializeReader public interfaces and
  GetIOSize / ReserveIO / WriteRaw / DeserializeTail hooks across the datacell,
  layout and io abstractions; skip zero-filling in ResizeForOverwrite.
• Currently implemented for HGraph; other index types return an unsupported-operation error.

Test Evidence

• make fmt
• make lint
• make test
• make cov, run tests, and collect coverage
• Other (describe below)

Test details:

Built with tests enabled and ran the Catch2 unittests binary directly.

Feature tests: 26 cases / 3268 assertions passed
  [parallel_deserialize] [chunked_serialize] [chunked_layout]
  [whole_body] [chunked_stream_writer]

Full unittest suite: 792 cases / 85,354,504 assertions passed

Benchmark (1M x 1024, sq8 + fp32 reorder, 32MiB frames, zstd, page-cache warm, 48-core):
  16-thread ParallelDeserialize vs sequential Deserialize(footer, body):
    block_memory 4.64x | memory 5.19x | mmap 3.50x | buffer 2.68x
  Probe path on an uncompressed layout-less file (5.0 GiB): 2.67x at 16 threads
  Every configuration returns results identical to the in-memory index, query by query.

[vsag-bot]

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao

[mergify]

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

• label~=^kind/

🟢 Require version label

• label~=^version/

🟢 Require linked issue for feature/bug PRs

• body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

[wxyucs]

/kind feature
/version 1.1
/assign

[LHT129]

关于 ParallelDeserialize(DeserializeReader&, ThreadPool&)：建议不要在 Index 公共 API 中暴露由调用方传入 ThreadPool 的重载，只保留 ParallelDeserialize(DeserializeReader&)。

索引由 Engine 创建时已经具备绑定线程池的能力；这里复用 Engine/Index 已受管的线程池即可。额外让调用方注入线程池会使资源归属和并发控制出现两套入口，也会让索引层 API 与 Engine 的职责重复。若确有内部复用需求，可以将带线程池的实现保留在非公开内部接口。

[LHT129]

This is a substantial and well-engineered PR. The architecture is carefully thought out with clear separation of concerns. The code has already gone through multiple review rounds and the quality is high.

A few remaining observations:

1. [note] The ResizePhysicalForOverwrite in MMapRegion ignores previous_logical_size (cast to void), while BlockMemoryBackend uses it to clear the unwritten tail. The asymmetry is correct but could surprise someone adding a third backend.

2. [note] The measure_head_size functions use explicit base-class qualification to bypass derived overrides. This is intentional and well-documented, but creates a hidden coupling. The ByteIO serialize framing test pins this, which is good.

3. [note] Test coverage is excellent: compressed/plain round trips, reorder/dedup/MCI/conjugate indexes, file-backed IOs, empty indexes, corrupted inputs, tampered layouts, configuration mismatches, and dribbling decompressed streams.

[LHT129]

Thank you for this well-engineered PR. The chunked serialization and parallel deserialization implementation is thorough, with strong validation at every layer (ChunkedManifest::Validate, pre-validation in parallel_deserialize_manifest, per-component consumption checks), clear documentation of invariants, and comprehensive test coverage including negative/fuzzing tests.

Summary of changes:

• New public interfaces: DeserializeReader (positioned reads + optional decompression) and SerializeWriter (byte sink + optional per-frame compression)
• ChunkedStreamWriter: splits components into head/io-data-frames/tail at precomputed boundaries
• ChunkedManifest: JSON-serializable layout with Validate() enforcing physical invariants (no overlaps, no gaps, exact tiling)
• Three-phase parallel load: Prepare (parse footer, pre-allocate extents), Fill (thread pool, one task per frame), Finalize (tails + post-load setup)
• ResizeForOverwrite optimization across BlockMemoryBackend, MMapRegion (fallocate), ContiguousBackend, ByteIO, and FixedLayout
• Probe path for legacy uncompressed files without recorded layout
• TaskBatch/FirstTaskError for safe concurrent task dispatch with exception propagation
• 939 lines of new tests covering round-trip, compressed/uncompressed, mmap-backed, tampered-footer rejection, dribbling streams, and compatibility

Review notes (non-blocking):

1. io_size_field_bytes() runtime measurement (src/algorithm/hgraph/hgraph_serialize.cpp:2833): The function measures the byte size of StreamWriter::WriteObj(uint64_t) by constructing a CountingStreamWriter and performing a dry-run write. The result is a compile-time constant (8 bytes), and the ByteIO Serialize Framing Pins Head Measurement test already guards against format changes. Consider using constexpr uint64_t kIOSizeFieldBytes = sizeof(uint64_t) directly and keeping the test as a regression guard — this would make the intent clearer that it is a format constant, not something that varies per call. The runtime cost is negligible (called at most 4 times per serialization), so this is purely about code clarity.

2. ChunkedManifest::FindComponent linear search (src/storage/chunked_manifest.cpp:4094): The function does an O(n) linear scan through the components vector. It is called from parallel_deserialize_manifest for each Byte-granularity component during pass 2 (at most 4 calls) and for each component during the pre-validation loop. With the current component count (~10), this is perfectly fine. If the component count grows significantly, consider using an unordered_map or sorting + binary search.

The implementation is solid — the validation layering (ChunkedManifest::Validate for physical invariants, then pre-validation for semantic constraints) is well-designed, the thread-safety invariants are clearly documented, and the test coverage is thorough. Nice work.

[LHT129]

Review Summary

This is a well-engineered PR that adds chunked serialization and parallel deserialization to the HGraph index. The architecture is carefully layered: public SerializeWriter/DeserializeReader interfaces at the API boundary, ChunkedStreamWriter/ChunkedManifest in the storage layer, and the parallel dispatch logic in hgraph_parallel_deserialize.cpp. The probe fallback path for legacy files, the if constexpr compile-time gating, and the Fallocate raw syscall choice all show solid engineering judgment.

Key strengths:

• Backward compatibility: The probe path handles layout-less uncompressed files, and Deserialize(StreamReader&) still works for compressed chunked bodies (rejects with a clear error).
• Concurrency safety: Pre-allocated io extents for lock-free concurrent fill, TaskBatch with first-error propagation, and the conjugate graph mutex invariant are all well thought out.
• Test coverage: Round-trip tests cover compressed/uncompressed, 1/4/16 threads, corruption rejection, empty index, MCI, conjugate graph, file-backed IO, and dribbling streams.
• Documentation: Both Chinese and English docs with usage examples are included.

Suggestions (see inline comments for details):

1. Consider debug assertions in ChunkedStreamWriter::Write() for frame boundary overflow
2. TaskBatch destructor could log when it swallows exceptions during stack unwinding
3. The conjugate graph mutex invariant comment should be referenced from the header
4. Validate() overlap detection is O(n^2) — acceptable for now but worth noting
5. Fallocate error message could distinguish ENOSPC from other failures
6. measure_head_size base-class dispatch is fragile — consider a regression test

No blocking issues found. The PR is ready for merge once the author reviews the suggestions.

[LHT129]

This is a new review round on the current state (958418b). The PR has matured significantly through prior rounds — most structural concerns have been addressed. The remaining issues are style consistency (alternative tokens) and one exception-type mismatch.

Alternative tokens (not, and, or)

The VSAG codebase consistently uses !, &&, ||. The following new files still contain alternative tokens:

• src/algorithm/hgraph/hgraph_parallel_deserialize.cpp: not on lines 92, 126, 176, 197, 442, 561, 674; and on lines 197, 642, 645
• src/storage/chunked_stream_writer.cpp: not on lines 48, 115; and on line 182
• src/storage/chunked_manifest.cpp: not on lines 68, 154; or on line 68; and on lines 163, 180, 188, 239
• src/storage/parallel_deserialize_utils.h: not on line 202

The or token on chunked_manifest.cpp:68 was flagged in a prior review round but is still present.

DeserializeReader::ReadDecompressed default throws std::runtime_error

include/vsag/deserialize_reader.h:61 — The default ReadDecompressed throws std::runtime_error, while every other error path in the deserialization pipeline throws VsagException. If a caller accidentally invokes ParallelDeserialize on a compressed file with a plain reader, the std::runtime_error propagates through FirstTaskError::Capture() (which catches ...) and surfaces as a bare std::runtime_error without a proper ErrorType. The existing comment acknowledges this is intentional to avoid depending on internal headers, but the inconsistency means callers must catch both VsagException and std::runtime_error to handle all deserialization failures uniformly.

[geerniman]

Review round addressed in be5551a (rebased onto current main, fd36393 — clean, no conflicts). Both items from the CHANGES_REQUESTED are fixed, plus the actionable suggestions from the inline round. Itemized:

Blocking item 1 — alternative tokens. All not/and/or operators in the four listed files (hgraph_parallel_deserialize.cpp, chunked_stream_writer.cpp, chunked_manifest.cpp, parallel_deserialize_utils.h) replaced with !/&&/|| — 24 sites total. Re-scanned every file this branch adds: remaining occurrences are inside string literals and prose comments only. Test files keep the existing upstream test convention (REQUIRE(not ...) appears throughout main's own test suite).

Blocking item 2 — exception family. Fixed at the pipeline boundary rather than the public header: all three ReadDecompressed call sites in the parallel path now go through read_decompressed_checked(), which passes VsagException through untouched and rethrows any other std::exception as VsagException(INTERNAL_ERROR, ...) with offset/size context. ParallelDeserialize now surfaces exactly one exception family; include/vsag/deserialize_reader.h stays free of internal-type dependencies by design. New unit test covers both the conversion and typed pass-through.

Implemented suggestions: TaskBatch destructor logs a warning when it suppresses a recorded failure during unwinding (with FirstTaskError::HasError()); fallocate failure message distinguishes actionable ENOSPC from other errnos; debug assertions in ChunkedStreamWriter (compressor no-op detection at the first frame boundary, frame-overrun invariant); kPoisonByte named constant; mutex-invariant note on the conjugate_graph_mutex_ declaration in hgraph.h; WriteRaw scope doc; macOS scope note on Fallocate; DEFAULT_SERIALIZE_CHUNK_SIZE documented as a recommendation; new test "ParallelDeserialize Rejects Non-Empty Index".

Fact-checked, no change needed (details in the respective threads): the offset + size < offset UB pattern is not present — Validate() already uses explicit subtraction guards; hgraph.h:244 already carries override; NonContinuousIO is never a backing io for chunked components (IVF buckets only) and shares the pinned ByteIO::Serialize framing.

Verification on be5551a: clean rebuild; targeted suites green (61 unit cases / 69,853 assertions including the two new tests; chunked FT cases 10,202 assertions); full unit-test run and CircleCI in flight.

[LHT129]

LGTM

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request