fix(documents): cache hydrated documents by version to end read-path CPU storm by ericvicenti · Pull Request #858 · seed-hypermedia/seed

ericvicenti · 2026-07-09T23:00:20Z

Summary

Production hyper.media was intermittently unavailable (requests timing out at 20–60s, flapping in and out). Root cause: Document.Hydrate replays a document's entire block-move op-log on every read to rebuild tree state (treeOpSet.State → isAncestor), which is superlinear in edit history. Under load, repeated expensive hydrations of the same hot documents blocked each other and grew an unbounded request queue until the daemon collapsed.

This PR adds a version-keyed hydrate cache + singleflight in front of the two hot read paths (GetDocument, getResource). A resolved document version is content-addressed and immutable, so the hydrated proto for a given (iri, version) never changes and is safe to cache indefinitely. singleflight collapses concurrent identical hydrations into a single computation.

Evidence

Root cause reproduced locally — Hydrate is O(n²) in tree depth (500 moves → 12ms, 1k → 47ms, 2k → 205ms, 4k → 873ms), with a CPU profile byte-identical to a production profile (95% of daemon CPU in isAncestor/State/Hydrate).

Production telemetry (/debug/journeys): ~50% of GetDocument load is version-pinned (immutable) and ~60% concentrates in just 26 documents re-hydrated ~495×/hour each — all redundant re-computation.

Benchmark: cache hit ~0.47ms vs ~884ms uncached for a 4000-move deep doc (~1,880×).

Deployed to prod as a hotfix (side-loaded image, watchtower paused) and verified:

Metric	Before	After
`hyper.media` home	timeout 20–30s	0.9–1.7s, stable 200
`design/stories/authoring`	78s	0.05s
`debate-tema-7` doc	16.7s	0.009s
8 concurrent reads	piled to 60s+	all <90ms
Flaps in 10+ min	constant	zero

Correctness

The cache is self-invalidating: a latest read always resolves current heads via loadDocument, so after an edit it forms a new key and recomputes; old version-pinned entries stay valid because the content is immutable. Returned protos are defensive clones so shared cache entries are never mutated by callers. The visibility/deny check runs before the cache, so access control is unchanged.

New tests (hydrate_cache_test.go): cached output equals direct hydrate; independent clones; keys on version (invalidates on edit); concurrent singleflight correctness. All pass under -race. Benchmarks in hydrate_bench_test.go document the O(n²) scaling and the cache speedup.

Files

hydrate_cache.go (new) — version-keyed LRU (2048) + singleflight
documents.go, resources.go — wire the cache into GetDocument and getResource
docmodel/docmodel.go — export Document.Version() (content-addressed cache key)
*_test.go (new) — correctness + benchmarks

Deploy / durability

The fix is currently side-loaded onto the prod box (docker save | ssh | docker load) with watchtower paused so it can't revert to the old registry image. It survives reboots (compose uses the local :latest tag) but nothing auto-updates while watchtower is down.

To make it durable: seedhypermedia/site:latest is published only by a version tag (*.*.*) or a manual run of release-docker-images.yml from main — not by merging this PR. After merge, publish :latest (tag a release or dispatch the release workflow from main), then re-enable watchtower (docker start autoupdater on the prod box) so it picks up the registry image.

Follow-ups (not in this PR)

Cap concurrent miss-hydrations with a semaphore for CPU headroom.
Make isAncestor use O(1) map lookups instead of the btree hot path, so even cache misses are cheap.

🤖 Generated with Claude Code

…CPU storm Document.Hydrate replays the entire block-move op-log on every read to rebuild tree state (treeOpSet.State -> isAncestor), which is superlinear in edit history and dominated daemon CPU in production (95% of a prod CPU profile, ~320% CPU sustained on 4 cores). Prod telemetry showed ~50% of GetDocument load is version-pinned (immutable) and 60% concentrated in ~26 documents re-hydrated ~495x/hour each. Add a version-keyed LRU + singleflight cache in front of Hydrate on the two hot read paths (GetDocument, getResource). A resolved version is content-addressed and immutable, so the hydrated proto is deterministic and safe to cache indefinitely; a latest read always resolves current heads via loadDocument, so the cache self-invalidates on edits. singleflight collapses concurrent identical hydrations into one computation. Benchmarks: cache hit ~0.47ms vs ~884ms uncached for a 4000-move deep doc (~1880x). Correctness tests verify cached output equals direct hydrate, returned protos are independent clones, and the cache keys on version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012PcXpTnVq45xwRBUMgczVp

…DRATE_BENCH TestHydrateScaling and TestLoadVsHydrate build documents with thousands of deeply-nested block moves to demonstrate the superlinear hydrate cost. That is intentionally slow and, under the race detector, exceeds CI's 10-minute test budget (the docmodel package timed out). They are diagnostic/proof tools rather than regression tests, so skip them unless RUN_HYDRATE_BENCH=1 is set (and never under -short). The cache regression coverage in hydrate_cache_test.go still runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012PcXpTnVq45xwRBUMgczVp

ericvicenti · 2026-07-10T00:07:39Z

Merging because @burdiyan says its okay, and because it seems to improve our production CPU bottleneck

…ee cycle (#860) * perf(docmodel): O(1) block lookups in isAncestor to cut hydrate CPU Document.Hydrate rebuilds tree state via treeOpSet.State() -> isAncestor, which did a comparator-heavy B-tree lookup per ancestor step. In a production CPU profile of the hyper.media daemon ~57% of all CPU was in that B-tree machinery (Less/hintsearch/nodeAscend). The #858 version cache cut how often we hydrate but not the per-hydrate cost, so cache misses (latest reads of just-synced docs) still saturate CPU. blockTreeState.blocks is point-lookup only (no ordered iteration), so switch it from *btree.Map[string,blockState] to a plain map[string]blockState. treeOpSet's ordered maps stay B-trees; no CRDT semantics change. Deep-doc hydrate 320.6ms -> 51.1ms (6.3x); flat 2.22ms -> 1.47ms. go test (incl -race) passes. * fix(docmodel): stop infinite loop hydrating documents with a block-tree cycle A document whose block tree contains a cycle — in production, a block that was its own parent — made isAncestor loop forever. isAncestor walks up a block's parent chain in an unbounded `for {}` with no visited guard, so a self-parent (X.Parent == X) spins a CPU core indefinitely. This hung GetDocument/Hydrate, and since the read path retries on timeout, a single poisoned document pinned a core per concurrent request. Three production documents in one space were affected; each read of them burned a core forever, which was the real cause of the gateway's CPU falling over at trivial request rates. How the corruption arose: State() prevents cycles by calling isAncestor(block, parent) before applying each move, but that inspects the pre-move state where the block is not yet self-referential, so a move with block == parent slipped through and made the graph cyclic. The bad changes are already signed and immutable, so hydration must be robust to them. Fixes: - isAncestor now bounds its walk by the number of blocks (a valid ancestor chain visits each block at most once), so it always terminates; a cyclic path is treated as "not an ancestor". This un-bricks existing documents. - State() explicitly marks a self-parent move (block == parent) invisible. - Move() rejects creating a self-parent move, so new corruption can't form. Validated against the real exported change history of the affected production document: hydration went from non-terminating (>400s, effectively infinite) to ~6ms. Adds regression tests for the self-parent cycle (hydrate must terminate), a two-node cycle in isAncestor, and the write-path guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012PcXpTnVq45xwRBUMgczVp --------- Co-authored-by: juligasa <11684004+juligasa@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

ericvicenti and others added 2 commits July 10, 2026 00:22

ericvicenti requested review from burdiyan and juligasa July 9, 2026 23:41

ericvicenti merged commit 082c508 into main Jul 10, 2026
2 checks passed

juligasa mentioned this pull request Jul 10, 2026

perf(docmodel): O(1) block lookups in isAncestor to cut hydrate CPU #859

Closed

ericvicenti mentioned this pull request Jul 10, 2026

fix(docmodel): stop infinite loop hydrating documents with a block-tree cycle #860

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(documents): cache hydrated documents by version to end read-path CPU storm - #858

fix(documents): cache hydrated documents by version to end read-path CPU storm#858
ericvicenti merged 2 commits into
mainfrom
hotfix/hydrate-cache

ericvicenti commented Jul 9, 2026

Uh oh!

ericvicenti commented Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ericvicenti commented Jul 9, 2026

Summary

Evidence

Correctness

Files

Deploy / durability

Follow-ups (not in this PR)

Uh oh!

ericvicenti commented Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant