feat(vpp-offload): a steered keep-vpp restart diffs a seeded mirror without waiting out the replay

[lunarthegrey]

Summary

A steered keep-vpp restart that has both route ledgers from the stopping daemon now verifies and re-asserts its steer within seconds of the route source's first route. Until now it waited for the source's whole replay to go quiet: about 11 minutes for 1.35M IPv4 routes from FRR, plus IPv6.

DeferredResync::AwaitingSeededDiff gets #324's seeded-mirror door. The four legs are the same as for the unsteered door:

• the mirror holds a seed no route-source GC has reconciled;
• the feed session is up, in its first session;
• the completeness authority's current word is yes;
• the floor.

This diff lands on a VPP that is carrying traffic, so the door adds a guard. The resync walk's own plan is put to the guard before any of it is queued (a new ConvergenceEngine::begin_resync_judged), and the door opens only if, in each address family, withdrawals, replacements and additions together are at most 1% of that family's routes VPP holds (STEER_MAX_DRIFT). Anything bigger means the two ledgers disagree. The early door is then shut for the rest of that restart, and the diff waits for quiet as before.

Every other case keeps the quiet gate unchanged:

• an unseeded mirror;
• no authority;
• a reconnected session (first session only, like feat(vpp-offload): an unsteered adoption diffs a seeded mirror without waiting out the replay #324);
• a seed a GC already reconciled;
• a refused guard.

Why

The replay re-advertises every route, and #153 counts each element as activity, so the quiet gate never sees quiet until the replay ends. For all of that time:

• VPP's FIB took no route changes (no deltas apply during a deferral), so a route that changed kept its stale path in a VPP carrying steered traffic.
• fib-synced read Degraded resync deferred.
• The post-verify Action::Steer re-assert waited too. That re-assert is also what moves keep rules from the queue-0 form to the RSS form (fix(vpp-offload): spread keep traffic over RSS, place queue 0's IRQ when it cannot, watch the kernel path #325).

Why landing this diff on a VPP that carries traffic is safe

Compare the early diff with the one that would run after the replay. They differ by what the replay has not re-advertised yet, and by how long what the early one lands stays wrong:

• Withdrawals are counted. These are routes VPP holds that the seed lacks and the replay would bring back. An early diff removes them from VPP until the replay reaches them.
• Replacements are counted. These are routes VPP holds through other paths than the seed resolves to now. They include routes whose next hop resolves to nothing at the walk, which the drain would remove as unresolvable. An early diff moves their traffic until the replay says otherwise.
• Additions are counted (review finding). Both diffs add them, but not for as long. Under the quiet gate a seeded route the source has dropped is in VPP for seconds: the diff runs 2 quiet seconds after the replay and the GC removes it at 5. The early door leaves it there for the whole replay, and a stale more-specific pulls traffic off the less specific route VPP was using. The authority counts routes, not which ones, so it cannot tell two ledgers from different stops apart if they agree in size.

When both ledgers come from one clean stop, all of that is near zero:

• The fast-path ledger snapshots the mirror, and vpp-offload's snapshots what VPP acknowledged from that same mirror. They differ only by what VPP had not received or acknowledged at that moment, which is churn-sized.
• Shadowed, kernel-delivered and link-local-only routes stay out of the walk's upserts, so they do not count. Every upsert of a prefix the ledger does not record VPP holding is an addition, whether or not its next hops resolve at the walk. The upsert keeps its next hops, and a neighbour learned before the drain installs it (review finding). Additions also count whole: the drain withdraws a replace whose next hops resolve to nothing, which frees a slot, so a room estimate would undercount (review finding). So routes the previous process left unresolvable or withheld do count, because VPP's preserved ledger records only what it installed. On a healthy box that is a handful (the status row names every unresolvable route); a box with more waits for quiet as before.
• Re-advertising an unchanged route is no FIB write and no VPP delta (programmer.rs no-change shortcut). The walk also drops the deltas the seed queued (feat(vpp-offload): an unsteered adoption diffs a seeded mirror without waiting out the replay #324).
• What the seed holds that the source dropped leaves VPP as an ordinary delta when the source's InitiationComplete GC withdraws it, the same as on the eBPF tier.
• The replay's real changes now reach VPP as deltas as they arrive. The quiet gate held them back for the whole replay.

The guard: needed, and the plan is available before the diff lands

"One stop, two snapshots" is a premise vpp-offload cannot check:

• A stop can write one ledger and not the other (each has its own 5 s budget).
• A start that runs without vpp-offload consumes the fast-path ledger but leaves VPP's.
• The walk resolves routes through the neighbours the feed holds a few seconds into the process. A next hop the resolver has not re-announced yet would remove every route through it from a steered VPP.

So the guard measures the disagreement instead of assuming it is small.

• One walk, judged before it is committed. begin_resync_judged holds the walk's ops, puts the finished plan to a judge, and applies them to the pending map in the order found only on a yes. On a no, nothing is queued, and the phase and the ledger are untouched. The route source mutates the mirror while it streams, so the first version (a preview walk, then a second walk that committed) let a route replaced at constant count, or a next hop lost between the two, become whatever the second walk found (review finding). The walk discards the source's queued deltas immediately before it, refused or not. Nothing is lost: no delta applies during a deferral, and every way out of a deferral runs a full walk first. ResyncPlan gains per-family withdrawals, replaces and additions.
• The plan is per family. ResyncPlan carries a Disturbance { withdrawals, replaces, additions } for IPv4 and one for IPv6, from the one shared walk, and the seed records its per-family size. Each family is held to its own table, so 400k IPv6 routes cannot cover a diff that rewrites a small IPv4 table, or the reverse (review finding). The refusal, its event (family) and the status row name the family; IPv4 is checked first.
• The bound is STEER_MAX_DRIFT of the family's adopted table, rounded down, with no floor. That is 13,500 at 1.35M routes and 146 on the lab's 14.6k. It is the authority's own line between churn and a missing table. It is the same constant on purpose: churn that needs one widened needs the other widened too. On a table small enough that 1% rounds to a handful, a refusal costs only a replay of a second or two.
• A refusal is latched for the deferral, not retried. Only the replay corrects the disagreement, and each retry would be another full walk of the mirror on the supervision thread. It is logged, recorded as a seeded_door_refused event, and appended to the fib-synced deferral line.
• Cost: none beyond the resync walk itself. The held ops are what the walk would have queued, and on an accepted seeded diff they are churn-sized.

Order at the door.

1. VPP's route counts are re-read (seed_still_holds), as the quiet release does. Nothing that takes time sits between this and the walk, so the window is the quiet door's own (review finding).
2. The one walk runs.
3. Its judge re-reads the legs, because the walk reads the whole mirror; a session that flapped meanwhile is not acted on, and its new epoch shuts the door on the next tick. The judge then applies the guard.
4. On a yes, the held ops are committed.

Interactions checked

• The fallback to AwaitingFallback. If the seed no longer holds at the early door, the dump path used to inherit an already-released quiet gate and unsteered at once. An early release has no quiet to hand over. Without a fix, traffic would have stayed for the whole replay on a FIB known to differ from the seed. The handover now carries the door as AwaitingFallback::seeded_door: the unsteer goes out at once, and the dump reads an idle VPP. After the dump, the release is re-checked by what it rested on: the door's legs, not the churn budget, since the replay streams through the door by design. The diff that follows is the unsteered one. The inherited door never releases a diff onto traffic. If a revocation put traffic back over a kept dump, that diff still waits for the quiet release, as before (the_inherited_door_never_diffs_onto_traffic).
• Verify after the diff, and fix(vpp-offload): hold a steer that diverts more until VPP has caught up and a verify covers the table #333's holds. Verify runs on the supervision thread with no drain in between, so it compares the ledger with VPP regardless of how loud the stream is. A seed whose paths VPP disagrees with still raises PreservedLedgerRejected. That path's "the eBPF tier is loaded" claim (supervisor and runbook) now also covers the door: its authority counted complete the mirror the eBPF tier forwards on. The VerifyPassed → Action::Steer re-assert is a reconcile of the same plan, so steer_hold does not hold it (expands_diversion is false), and that reconcile is where keep forms migrate (fix(vpp-offload): spread keep traffic over RSS, place queue 0's IRQ when it cannot, watch the kernel path #325). The tests show it landing while the replay streams.
• preserve(). A refused deferral leaves the seed untouched (a refused walk changes no ledger), so a stop during it still preserves the ledger, as before.
• Status, journal and events. A refusal is appended to the deferral row under every authority posture. Every adopted-resync release is now recorded as a new resync_released event (door: quiet/seeded, path, steered, routes, what the diff queued), and the journal line carries door too. routes is the mirror the walk read, counted just before it, not the count the tick began with: on the dump path the replay streams through a dump of seconds (review finding). The existing quiet-path journal texts are unchanged.

Tests

tests/vpp_runtime.rs, driver + runtime + fake VPP:

• a_steered_seeded_restart_verifies_and_re_steers_while_the_replay_streams: a 200-element replay every tick. The box reaches Steered during the replay with steering never down (["steer"]), no dump, and no route op.
• a_steered_seeded_release_lets_changes_reach_vpp_during_the_replay (production RouteFeed): nothing VPP holds is sent again, and a withdrawal plus an addition made after the release reach VPP while the replay streams.
• a_steered_seeded_diff_without_every_leg_waits_for_the_replay_to_go_quiet covers five cases: unseeded mirror, no authority, an authority that disputes the mirror, a reconnected session, and a reconciled seed. Each stays deferred with nothing previewed and nothing sent, then releases through the quiet door.
• a_steered_seeded_diff_within_the_allowance_lands_while_the_replay_streams (1 of 200 withdrawn plus 1 added) and ..._beyond_the_allowance_waits_for_the_replay_to_go_quiet (3 withdrawn, 3 replaced, or 3 added). The second checks that the refusal is reported in status and the row text, that the preview ran once rather than every tick, and that the quiet door later lands the same diff.
• a_session_that_flaps_during_the_walk_shuts_the_door: the judge re-reads the legs when the walk is done; nothing is queued, and nothing is latched.
• routes_replaced_as_the_walk_begins_are_judged_on_that_walk: 3 of 200 routes replaced at constant count as the walk begins. The walk's own plan is refused, with pending_ops == 0.
• a_next_hop_lost_after_the_walk_does_not_change_what_it_committed: the committed unchanged table stays in VPP.
• an_unresolved_addition_counts_though_its_neighbour_arrives_later: 50 absent routes go through a next hop whose neighbour is learned after the walk. The door refuses, nothing is queued, and the quiet door later installs all 50 once the neighbour delta arrives.
• a_full_table_counts_the_additions_its_unresolvable_replaces_make_room_for: a full table of 200 with 2 unresolvable replaces and 2 additions is refused; the quiet door later shows the drain freeing and refilling those slots.
• a_next_hop_not_yet_relearned_keeps_the_early_door_shut: a missing neighbour counts as 6 replaces, so the door refuses. Once the neighbour is back, the quiet diff sends nothing.
• a_seed_that_no_longer_holds_at_the_early_door_unsteers_at_once: the carry-over, with ["unsteer", "steer"], one dump and the stranger route withdrawn, all during the replay.
• the_inherited_door_never_diffs_onto_traffic.
• one_family_over_its_bound_shuts_the_door_whatever_the_other_holds (dual-stack): 3 IPv4 withdrawals against 100 IPv4 and 200 IPv6 routes, and the reverse. Both are within 1% of the combined table, and both are refused, naming the family.
• a_release_after_a_dump_records_the_mirror_the_walk_read: the mirror grows from 6 to 13 routes during the dump, and resync_released says 13. This test installs the test binary's event log and reads it back.

Elsewhere:

• tests/vpp_engine.rs: a_judged_resync_commits_only_the_plan_its_judge_saw. A refusal leaves pending empty and the phase and ledger untouched; an acceptance queues exactly the plan judged. It also covers per-family counts and a new route through an unresolvable next hop, which still counts as an addition.
• Unit tests: the_seeded_diff_guard_counts_what_lands_on_traffic covers additions (counted whole), each family against its own table both ways, and the invariant allowance ≤ adopted × STEER_MAX_DRIFT < allowance + 1 at every size from 0 to 50,000 and at large sizes. Also a_refused_seeded_door_explains_the_wait_on_the_deferral_row.

Negative control against origin/main. I put the new vpp_runtime tests into an origin/main tree, with only the lines naming the new status field removed, and ran them once:

• Fail on the old code (7): restart_verifies_and_re_steers, release_lets_changes_reach_vpp, within_the_allowance_lands, seed_that_no_longer_holds…unsteers_at_once, and inherited_door all stay AdoptedResyncing because the old code waits for the replay. beyond_the_allowance and the session-flap test fail because the old code never counts the diff before it lands (measured in the first round, when the flap test was named …during_the_preview…).
• Pass on the old code (2), by design: without_every_leg_waits and next_hop_not_yet_relearned pin behaviour that must not change, and the old code always waits. The engine test and the two unit tests do not compile on the old code, because there is no preview or guard to test.

Mutation checks. Each one removes one leg of the new code, and the named test(s) fail:

Mutation	Fails
door ignores mirror_seeded	without_every_leg (+ #324's unseeded test, the door unit test)
door ignores the authority	without_every_leg, door unit test
door accepts any epoch	without_every_leg, flaps_during_the_preview, #324's reconnect test
no guard	beyond_the_allowance, next_hop_not_yet_relearned, guard unit test
guard ignores replaces	beyond_the_allowance (re-path case), guard unit test
walk stops counting replaces	beyond_the_allowance, next_hop_not_yet_relearned, engine test
no re-check of the legs after the walk	flaps_during_the_walk
refusal not latched	beyond_the_allowance ("previewed once")
a refusing judge still commits	routes_replaced…, full_table…, beyond_the_allowance, one_family…, the engine test and others
held ops dropped on commit	engine test, within_the_allowance
no carry-over to the dump path	seed_that_no_longer_holds…
inherited door may diff onto traffic	inherited_door_never_diffs_onto_traffic
additions not counted	beyond_the_allowance (added case), guard unit test
IPv6 counts folded into IPv4	one_family_over_its_bound…
dump path records the pre-dump count	a_release_after_a_dump_records…

Review round (Codex, 3 findings). I ran the new and changed tests against the previous head, 87cb6bc, with only the new family and additions field references stubbed out. Three fail there, each for its finding:

• beyond_the_allowance: the ss-beyond-added case lands early.
• one_family_over_its_bound…: both ss-v4-in-v6 and ss-v6-in-v4 land early when each is run first.
• a_release_after_a_dump…: it records [Some(6)].

Some rows of the table are from the review rounds; the walk-related rows were re-run against the one-walk code.

Review round 3 (Codex, 3 findings). I ran the new tests against 8b81a11. Each fails there, for its finding:

• routes_replaced_as_the_walk_begins… lands early: the preview approved, and the commit walk found the replacement.
• a_next_hop_lost_after_the_walk…: 200 withdrawn from the steered VPP.
• a_full_table_counts_the_additions… lands early: the clamp hid the additions.

The fingerprint finding needs no test of its own: there is no longer a step between the count re-read and the walk.

Review round 4 (Codex, 1 finding). an_unresolved_addition_counts_though_its_neighbour_arrives_later fails on 02485f6: the door opens early and the box steers while the replay streams. The engine test's inverted unresolvable-route assertion also fails there.

Verified

In pf-bpf-test:bookworm (Linux arm64, real BPF ELF, --cpus=4, PACKETFRAME_BPF_REQUIRED=1), on git archive HEAD, rerun at 6acfccd after the review fixes:

• cargo fmt --all --check: clean.
• cargo clippy --workspace --all-targets --all-features -- -D warnings: clean.
• cargo test --workspace --no-run plus privileged-tests.sh bundle: built.
• cargo test --workspace: all pass, including the 18 new tests.
• PACKETFRAME_BENCH_QUICK=1 privileged-tests.sh run native: all pass except tc_parse_error_bumps_both_shared_and_tc_counter. That one is the known linuxkit 7.0 failure (tc TEST_RUN returns EINVAL for an IPv4 frame cut inside its IP header). It fails on main too, and CI's 5.15 and 6.6 pass it.

Also checked: host (macOS) cargo fmt --check, host cargo clippy --workspace --all-targets --all-features -D warnings, the same clippy cross-checked for x86_64-unknown-linux-gnu, and host cargo test -p packetframe-vpp-offload.

Docs

• CHANGELOG.md (Unreleased).
• docs/runbooks/vpp-offload.md: a new "The steered diff and a seeded mirror" section. Updates to the release-gate door count, keep-vpp step 3, the fast-path ledger paragraph, "What you see", the PreservedLedgerRejected claim, and the "planned restart looks Degraded" section.
• docs/runbooks/packetframe-fib.md: the steered bullet.
• docs/runbooks/event-log.md: the two new kinds and the restart sequence.

Hardware validation (lab EFG)

This needs the lab as it is: the 5.15 UniFi kernel, fast-path forwarding-mode packetframe-fib with route-source bgp fed by the lab FRR (~14.6k routes), route-ledger on (the default), and vpp-offload with require-table-complete on plus its integrity-authority frr … line, steered on one port.

The lab's replay is a burst of a few seconds, so the drill keeps the stream busy across the restart. It loops clear bgp ipv4 unicast <pf-peer> soft out on the lab FRR. Each loop re-sends the whole table to PacketFrame as identical UPDATEs: the same shape as the production replay, where every element counts as activity and nothing in the mirror changes. No prefixes are touched, and the guard sees nothing to count. It also keeps the BGP listener from ever being quiet for 5 s, so the seed stays unreconciled until the churn stops.

0. Fetch both builds (Mac)

# New build: this PR's head
gh workflow run hardware-artifacts.yml -R unredacted/packetframe --ref claude/steered-seeded-diff
gh run list -R unredacted/packetframe --workflow hardware-artifacts.yml --branch claude/steered-seeded-diff -L 1
gh run watch <new-run-id> -R unredacted/packetframe
gh run download <new-run-id> -R unredacted/packetframe -n hwtest-aarch64-unknown-linux-gnu -D ~/pf-hw/ssd-new
# Old build: main at 46d82fd, this PR's base (pick the run whose head SHA is 46d82fd)
gh run list -R unredacted/packetframe --workflow hardware-artifacts.yml --branch main -L 10
gh run download <old-run-id> -R unredacted/packetframe -n hwtest-aarch64-unknown-linux-gnu -D ~/pf-hw/ssd-old
(cd ~/pf-hw/ssd-new && shasum -a 256 -c packetframe-hwtest-aarch64-unknown-linux-gnu.tar.gz.sha256)
(cd ~/pf-hw/ssd-old && shasum -a 256 -c packetframe-hwtest-aarch64-unknown-linux-gnu.tar.gz.sha256)
scp ~/pf-hw/ssd-old/packetframe-hwtest-aarch64-unknown-linux-gnu.tar.gz root@<lab-efg>:/data/pf-ssd-old.tar.gz
scp ~/pf-hw/ssd-new/packetframe-hwtest-aarch64-unknown-linux-gnu.tar.gz root@<lab-efg>:/data/pf-ssd-new.tar.gz

1. Unpack, and the two drill scripts (EFG, as root)

Use fresh directories, so each holds exactly one .deb. Never glob a reused bundle directory.

rm -rf /data/pf-ssd-old /data/pf-ssd-new && mkdir -p /data/pf-ssd-old /data/pf-ssd-new
tar -xzf /data/pf-ssd-old.tar.gz -C /data/pf-ssd-old
tar -xzf /data/pf-ssd-new.tar.gz -C /data/pf-ssd-new
ls /data/pf-ssd-old/packetframe-hwtest-aarch64-unknown-linux-gnu/*.deb   # exactly one, ~hwtest.<run>.46d82fd
ls /data/pf-ssd-new/packetframe-hwtest-aarch64-unknown-linux-gnu/*.deb   # exactly one, ~hwtest.<run>.<this PR's head>
OLD_DEB=$(ls /data/pf-ssd-old/packetframe-hwtest-aarch64-unknown-linux-gnu/*.deb)
NEW_DEB=$(ls /data/pf-ssd-new/packetframe-hwtest-aarch64-unknown-linux-gnu/*.deb)
PORT=<the steered vpp-offload port>
PF_PEER=<PacketFrame's neighbour address, as `vtysh -c 'show bgp summary'` lists it>

cat > /data/ssd-churn.sh <<'EOF'
#!/bin/sh
# Re-send the whole table to PacketFrame every second until /data/ssd-stop-churn exists.
while [ ! -e /data/ssd-stop-churn ]; do
  vtysh -c "clear bgp ipv4 unicast $1 soft out" >/dev/null 2>&1
  sleep 1
done
EOF
cat > /data/ssd-watch.sh <<'EOF'
#!/bin/sh
# Once a second until /data/ssd-stop-watch: VPP's pid, the steered port's
# rule count, and vpp-offload's fib-synced and steering rows.
while [ ! -e /data/ssd-stop-watch ]; do
  printf '%s vpp=%s rules=%s | %s\n' "$(date +%T)" "$(pgrep -x vpp_main)" \
    "$(ethtool -n "$1" 2>/dev/null | grep -c '^Filter:')" \
    "$(packetframe status 2>/dev/null | grep -E 'fib-synced|steering' | tr -s ' ' | tr '\n' '|')"
  sleep 1
done >> "$2" 2>&1
EOF

2. Phase A: the old build waits for the churn

systemctl stop packetframe && packetframe detach --keep-vpp && dpkg -i "$OLD_DEB" && systemctl daemon-reload && systemctl start packetframe
# Wait until `packetframe status` shows fib-synced healthy and the port steered, then:
pgrep -x vpp_main; ethtool -n "$PORT" | grep -c '^Filter:'
rm -f /data/ssd-stop-watch /data/ssd-stop-churn /data/ssd-watch-A.log
setsid -f sh /data/ssd-watch.sh "$PORT" /data/ssd-watch-A.log
setsid -f sh -c 'journalctl -f -u packetframe -o short-precise > /data/ssd-journal-A.log 2>&1'
setsid -f sh /data/ssd-churn.sh "$PF_PEER"
sleep 10
vtysh -c "show bgp neighbors $PF_PEER" | sed -n '/Message statistics/,/Total/p'   # run twice, a few seconds apart: Updates sent climbs by thousands
date +%T; systemctl stop packetframe && packetframe detach --keep-vpp && systemctl start packetframe; date +%T
sleep 180
date +%T; touch /data/ssd-stop-churn          # note this time: CHURN-STOP-A
sleep 60
touch /data/ssd-stop-watch; pkill -f 'journalctl -f -u packetframe'
packetframe events --since 15m --module vpp-offload
packetframe events --since 15m --module fast-path

Expected on the old build:

• Journal at start: adopted resync deferred until the route source is loaded and quiet; the preserved FIB keeps forwarding untouched meanwhile, and steering stays up throughout.
• ssd-watch-A.log: fib-synced DEGRADED — resync deferred: … (release floor N met); the diff runs once the source is live and has gone quiet … on every line for the whole 180 s.
• Only after CHURN-STOP-A (a few seconds later, once FRR's queue drains and the stream has been quiet 2 s): route source loaded and quiet; diffing it against the preserved ledger while VPP keeps forwarding — no dump and no unsteer, and only the differences go to VPP, then verify passed, then steering UP: allowlisted traffic is now diverted to VPP (the eBPF tier remains the failover).
• Events: adoption_path (path: preserved-ledger) at the start; verify_passed (seeded: true) and steering_up timestamped after CHURN-STOP-A; route_ledger_reconciled once the stream has been quiet for 5 s after CHURN-STOP-A. There is no resync_released, because that kind is new.
• Every watch line has the same vpp= pid and the same rules= count, apart from the lines where status cannot answer during the stop/start.

3. Install the new build

The keep-vpp restart onto the new build adopts A's ledgers. With no churn it takes the seeded door as well.

systemctl stop packetframe && packetframe detach --keep-vpp && dpkg -i "$NEW_DEB" && systemctl daemon-reload && systemctl start packetframe
# Wait until fib-synced is healthy and the port steered.
pgrep -x vpp_main; ethtool -n "$PORT" | grep -c '^Filter:'

Both builds carry the same VERSION and identical state and ledger formats (their deb versions differ only in the ~hwtest suffix), so this is a same-version keep-vpp restart.

4. Phase B: the new build releases at the first attestation

rm -f /data/ssd-stop-watch /data/ssd-stop-churn /data/ssd-watch-B.log
setsid -f sh /data/ssd-watch.sh "$PORT" /data/ssd-watch-B.log
setsid -f sh -c 'journalctl -f -u packetframe -o short-precise > /data/ssd-journal-B.log 2>&1'
setsid -f sh /data/ssd-churn.sh "$PF_PEER"
sleep 10
vtysh -c "show bgp neighbors $PF_PEER" | sed -n '/Message statistics/,/Total/p'
date +%T; systemctl stop packetframe && packetframe detach --keep-vpp && systemctl start packetframe; date +%T   # START-B
sleep 180
date +%T; touch /data/ssd-stop-churn          # CHURN-STOP-B
sleep 60
touch /data/ssd-stop-watch; pkill -f 'journalctl -f -u packetframe'
packetframe events --since 15m --module vpp-offload
packetframe events --since 15m --module fast-path

Expected on the new build, all within seconds of START-B and well before CHURN-STOP-B:

• Journal at start: adopted resync deferred until the route source is loaded and quiet; over a mirror the fast-path seeded from its route ledger it runs sooner, once the route source streams and the completeness authority attests the seed, if the diff disturbs no more of either family's table than one clean stop's two snapshots can differ by; …
• fast-path: route mirror seeded from the route ledger: ….
• vpp-offload: the route mirror was seeded from the fast-path route ledger, the route source is streaming in its first session, the completeness authority agrees with it, and the diff disturbs no more of either family's table than one clean stop's two snapshots can differ by: diffing it against the preserved ledger now … with door="seeded". Expect withdrawals=0 replaces=0 and unchanged ≈ the table: the soft-out churn changes nothing. additions should be 0, or the number of unresolvable routes fib-synced named before START-B, since VPP's preserved ledger does not record those.
• Then verify passed, then steering UP: ….
• Events: route_ledger_seeded and adoption_path (path: preserved-ledger), then resync_released (door: seeded, steered: true, path: preserved-ledger) → verify_passed (seeded: true) → steering_up, all timestamped before CHURN-STOP-B. Then route_ledger_reconciled only once the stream has been quiet for 5 s after CHURN-STOP-B: the seed stayed unreconciled through the churn, which is the door's first leg.
• ssd-watch-B.log: resync deferred for seconds only, briefly resyncing the FIB the previous process verified and preserved …, then fib-synced healthy while the churn still runs. Every line has the same vpp= pid as before START-B and the same rules= count.
• No seeded_door_refused. If one appears, its family, withdrawals, replaces, additions and allowance (146 on this table) say how far the two ledgers disagreed. Please send them: a refusal on an ordinary lab restart would mean the bound is too tight.

5. Optional: a real change reaches VPP during the replay

Only if the lab can originate a TEST-NET prefix through the same next hop as the feed (as D2b/D3 did, a kernel route on the upstream router plus redistribute kernel). During Phase B's churn, after steering UP, originate 198.51.100.0/24 there. Within seconds, while the churn still runs:

vppctl -s /run/packetframe/vpp/api.sock.cli show ip fib 198.51.100.0/24

It should show the route via the feed's next hop. On the old build it would only appear after the churn stopped. Remove it and confirm it leaves VPP the same way.

6. Clean up, and what to send back

touch /data/ssd-stop-churn /data/ssd-stop-watch; pkill -f 'journalctl -f -u packetframe'
packetframe events --since 2h --json > /data/ssd-events.json
ls -la /data/ssd-journal-A.log /data/ssd-journal-B.log /data/ssd-watch-A.log /data/ssd-watch-B.log /data/ssd-events.json
rm -f /data/ssd-churn.sh /data/ssd-watch.sh /data/ssd-stop-churn /data/ssd-stop-watch

Open points

• The 1% bound. It is borrowed from STEER_MAX_DRIFT with a stated reason, and it now covers additions and each family separately, so it is stricter than before. If the lab or the primary ever logs seeded_door_refused on an ordinary restart, it is too tight. The disagreement it would show is the VPP drain backlog at the moment of the stop, plus any routes the previous process left unresolvable or withheld, which now count as additions.
• The latch. A transient disagreement, for example a neighbour not re-announced yet at the first attestation, costs the full replay rather than a retry. I judged a paced retry (one full walk per retry, during the replay) not worth it before seeing a refusal in practice.
• No hardware drill for the guard. On a lab restart the authority's drift check catches most ways of making the ledgers disagree before the guard sees them. The guard is covered by the unit tests, the driver tests and the mutations above.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-11T10:09:31.414526Z	6acfccd	Manual request
🔒 Security Review	✅ Completed 2026-10-11T10:12:17.799266Z	6acfccd	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87cb6bcb20

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lunarthegrey]

@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 8b81a1163e

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b81a1163e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lunarthegrey]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 02485f620e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 02485f620e

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[lunarthegrey]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 6acfccd044

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6acfccd044

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.