Add: generation-safe two-frame local endpoint

[Crane-Liu]

Summary

• add two generation-safe task frames behind a separate mailbox control frame
• let one bounded progress owner poll active run A while validating and owning
  successor frame B
• defer B native prepare and launch until FIFO promotion and A native
  finalization fence, preserving the one-unfinished-native-run invariant
• keep launch acceptance as a sticky per-frame word and validate protocol,
  run, slot, generation, and dispatch identity before reuse
• terminalize both lanes when native reuse or child quiescence is uncertain;
  registry controls remain coherent with active and staged frame ownership
• retain the single-frame compatibility path for SUB, nested/remote workers,
  simulation, A5, and depth-one local chip endpoints

Scope boundary

FRAME_STAGED means the child validated and owns an immutable mailbox payload;
it is not HBG/TMR native preparation. Runtime-specific preparation overlap and
resource banking remain follow-up B4/B5 work.

The production two-frame path is enabled only for direct A2/A3 onboard chip
children that advertise pipeline depth >= 2. HBG and TMR share the common
endpoint contract while continuing to serialize their native run lifecycles.

CI robustness follow-ups

• make endpoint capacity immutable before scheduler startup and fence the
  publication-failure regression on the recorded successor failure, removing
  the macOS test data/timing race
• make HCCL destroy markers atomic and add arrival/departure plus post-teardown
  per-follower release handoff on A2/A3 and A5, preventing a fast rank-0 cleanup
  from stranding a slower peer or splitting sequential rootinfo generations

Validation

• Python unit suite: 1034 passed, 9 skipped
• C++ unit suite: 74/74 passed after the final rebase
• file-marker generation/cleanup regression: 50/50 repeated passes
• changed-file pre-commit hooks: all passed, including clang-tidy, cpplint,
  markdownlint, Ruff, and pyright
• A2/A3 and A5 host runtimes: HBG and TMR all compile/link successfully
• A2/A3 onboard HBG two-frame scene: standalone 3/3 repeated passes plus
  pytest pass
• A2/A3 onboard final pipeline task task_20260802_040006_317367021979: HBG
  two-frame scene and TMR L3 dependency smoke passed
• A2/A3 onboard final teardown task task_20260802_050029_390675130917:
  communicator lifecycle and two-rank TMR allreduce both passed

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f4908fe0-6501-40ce-81e0-56e367f73676

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The change adds generation-safe pipeline leases, whole-run FIFO admission, prepared endpoint activation, and two-frame local mailbox execution. Run-aware scheduling, Python bindings, chip startup wiring, mailbox layouts, and unit/end-to-end tests are updated accordingly.

Changes

Pipeline admission and prepared execution

Layer / File(s)	Summary
Lease and run-scoped queue contracts src/common/hierarchical/types.*, src/common/worker/pipeline_slot_pool.h, docs/task-flow.md	Run phases and task slots carry pipeline leases; queue operations are scoped by run; admission depth limits lease acquisition.
FIFO run admission and dispatch src/common/hierarchical/orchestrator.*, src/common/hierarchical/scheduler.*, src/common/hierarchical/worker.*	Runs reserve leases, activate in FIFO order, cancel unstarted work on failure, and route active or preparable work through run-aware scheduler queues.
Two-frame mailbox and activation protocol src/common/hierarchical/worker_manager.*	Worker endpoints support multiple frames, bounded inflight dispatch, prepared execution, backend-ready gating, activation, lease propagation, and terminal state handling.
Chip loop and binding integration python/simpler/worker.py, python/simpler/task_interface.py, python/bindings/*	Python startup derives supported depth and frame counts, forwards lease data through mailbox execution, exports mailbox constants, and releases the GIL around blocking operations.
Unit and end-to-end validation tests/ut/cpp/hierarchical/*, tests/ut/py/test_worker/*, tests/st/a2a3/host_build_graph/worker_async_*/*	Tests cover FIFO admission, prepared activation, frame concurrency, lease wire fields, depth configuration, and asynchronous A2A3 execution.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

• [Bug] READY NEXT_LEVEL group can be overtaken indefinitely by later singles (no Tier-0 equivalent at L3) #1551 — The scheduler changes are related to run-aware FIFO dispatch, though the issue’s requested head-group reservation is not implemented here.

Possibly related PRs

• hw-native-sys/simpler#1459 — Earlier per-run lifecycle and isolation work overlaps the updated orchestrator run handling.
• hw-native-sys/simpler#1540 — Directly related generation-safe depth-two pipeline lease plumbing.
• hw-native-sys/simpler#1574 — Directly related two-frame endpoint and whole-run FIFO admission behavior.

Poem

A rabbit hops through frames of two,
With leases fresh and generations true.
FIFO paws the waiting line,
Backend-ready stars align.
“Activate!” the burrow sings—
Done or failed, it frees its strings.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 2.82% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: a generation-safe two-frame local endpoint.
Description check	✅ Passed	The description directly explains the two-frame endpoint, FIFO prepare/activation contract, compatibility scope, and validation results.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/common/hierarchical/orchestrator.h`:
- Around line 132-134: Integrate can_dispatch_run into the production
run-admission/dispatch path so every run is checked against the documented
lease-ownership gate before being dispatched. Locate the scheduler method that
admits or starts runs and invoke can_dispatch_run with the candidate run_id,
rejecting or deferring runs that fail the check; leave active_run_id and
preparable_run_id unchanged.

In `@src/common/hierarchical/worker_manager.cpp`:
- Around line 305-315: Prevent failed dispatches from leaving an unretiable
publish sequence. In the dispatch preparation flow, validate the endpoint,
prepared slot, and activation capability before calling
WorkerThread::enqueue_dispatch; for failures that can still occur after ID
allocation in dispatch_process or run_prepared_with_activation, add and invoke a
WorkerThread sequence-cancellation/poisoning hook so the allocated ID advances
or is retired and subsequent dispatches do not block.
- Around line 328-338: Update WorkerThread::stop and the activation-wait path to
synchronize shutdown_ with activation_mu_, the mutex used by the activation_cv_
predicate. Set or publish shutdown_ while holding activation_mu_ before calling
activation_cv_.notify_all(), and ensure the waiter reads it under that same lock
so shutdown cannot be missed and stop() can join all threads.

In
`@tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py`:
- Around line 129-133: Drop the loop-variable tensor references before freeing
host buffers in both cleanup blocks: update
tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py
lines 129-133 and
tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py lines
178-182 to clear the respective buffer/tensor loop bindings alongside the named
locals, then run the existing free_host_buffer sweep.
- Around line 115-119: Add a short sleep inside the polling loop around the
state reads in the worker async endpoint test, after each unsuccessful poll, so
the loop yields the GIL while retaining the existing deadline and
state-detection behavior.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 684-697: Replace the unbounded orchestrator.wait_run calls after
the readiness assertions with the bounded wait_run_for variant, applying the
existing test timeout convention to both first_run and second_run. Preserve the
release_normal cleanup and ensure the test reports a failure rather than hanging
when readiness was not achieved.
- Around line 575-587: Add steady-clock deadlines to both simulated-child
polling loops in the thread lambda, covering waits for PREPARE_READY and
ACTIVATE; exit or fail cleanly when either deadline expires. Also bound the
caller’s wait for completion so child.join() and caller.join() cannot hang when
readiness or done never occurs, while preserving the existing state transitions
and assertions.

In `@tests/ut/py/test_worker/test_host_worker.py`:
- Around line 1620-1625: Update the test’s finally block to join the submitter
thread before closing or unlinking shared resources, matching the cleanup
pattern in test_submit_close_race_accepts_and_drains_admitted_run. Ensure the
join occurs even when the test body fails, so submitter has exited before
hw.close(), state_shm.close(), and state_shm.unlink().

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1b1a4a18-1617-45e9-bef5-e2d973cedbfd

📥 Commits

Reviewing files that changed from the base of the PR and between 33e5450 and a264f8e.

📒 Files selected for processing (24)

• docs/task-flow.md
• python/bindings/task_interface.cpp
• python/bindings/worker_bind.h
• python/simpler/task_interface.py
• python/simpler/worker.py
• src/common/hierarchical/orchestrator.cpp
• src/common/hierarchical/orchestrator.h
• src/common/hierarchical/scheduler.cpp
• src/common/hierarchical/scheduler.h
• src/common/hierarchical/types.cpp
• src/common/hierarchical/types.h
• src/common/hierarchical/worker.cpp
• src/common/hierarchical/worker.h
• src/common/hierarchical/worker_manager.cpp
• src/common/hierarchical/worker_manager.h
• src/common/worker/pipeline_slot_pool.h
• tests/st/a2a3/host_build_graph/worker_async_endpoint/kernels/orchestration/long_vector_orch.cpp
• tests/st/a2a3/host_build_graph/worker_async_endpoint/test_worker_async_endpoint.py
• tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py
• tests/ut/cpp/hierarchical/test_orchestrator.cpp
• tests/ut/cpp/hierarchical/test_pipeline_contract.cpp
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• tests/ut/py/test_callable_identity.py
• tests/ut/py/test_worker/test_host_worker.py