Skip to content

Refactor: make stream launch exact - #1700

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
Crane-Liu:codex/worker-async-b6b-exact-launch
Aug 6, 2026
Merged

Refactor: make stream launch exact#1700
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
Crane-Liu:codex/worker-async-b6b-exact-launch

Conversation

@Crane-Liu

@Crane-Liu Crane-Liu commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add move-only launch permits and identity-bound receipts
  • move launch-safe materialization into prepared execution ownership
  • consume prepared state and produce active state at the exact AICore/AICPU submission boundary
  • retain and poison uncertain partial launches while keeping pre-launch failures rollback-safe
  • keep prepared-successor rollback isolated from active-run collectors and terminal state
  • keep device-wall capture failure non-fatal without freeing a shared in-flight buffer
  • place simulation launch side effects after permit consumption
  • update stale lifecycle documentation and remove dead runner-global kernel arguments

CI budget

The a2a3 full-scene lane had reached 587 seconds on #1695 with a 600-second session limit. The earlier #1700 run reported every case passed at 100% and then exited 124 at the same session wall. The main a2a3 sweep now uses 1200 seconds, matching the a5 sweep; isolated smoke and SDMA limits remain unchanged.

Validation

  • current base: 29b119d
  • LCW arm64 CANN editable build: passed
  • C++ no-hardware targeted tests: test_native_run_launch_signal and test_native_run_execution passed
  • pre-commit: headers, English-only, platform literals, large-file, YAML, EOF, whitespace, clang-format, Markdown, Ruff, and pyright hooks passed
  • standalone clang-tidy was blocked by a shared compile-database cache rebuild race; it reported no finding attributable to this diff before the cache failure
  • earlier a2a3 hardware lifecycle/stream/FIFO coverage on the exact-launch implementation: 10/10 scenes and 6/6 hardware unit tests

Simulation scene execution was intentionally omitted; simulation targets were compile-validated.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR replaces the enqueue/poll/drain API with prepared and active execution objects. It adds identity-bound launch permits and receipts, transactional AICore/AICPU submission, per-execution kernel arguments, explicit cleanup, and unit tests for launch and receipt validation.

Changes

Native execution lifecycle

Layer / File(s) Summary
Execution contracts and ownership primitives
src/common/platform/..., src/common/worker/...
Adds PreparedExecution, ActiveExecution, LaunchPermit, LaunchReceipt, launch progress, and identity tracking. Updates runner interfaces and native-run ownership.
Prepared execution setup and kernel arguments
src/a2a3/..., src/a5/...
Separates preparation from launch. Stores runtime geometry, resources, profiling state, and device buffer addresses in per-execution kernel arguments.
Transactional launch, polling, and cleanup
src/common/platform/onboard/host/c_api_shared.cpp, src/common/platform/sim/host/c_api_shared.cpp, src/a2a3/..., src/a5/...
Submits AICore before AICPU through exact_launch_transaction. Returns launch progress and ownership state. Polls, drains, abandons, and cleans up executions explicitly.
Launch and signal validation
tests/ut/cpp/...
Adds tests for identity mismatches, partial launches, receipt publication, one-shot permits, and stale receipts.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Poem

A rabbit prepares each run,
Then launches cores in order, one by one.
Permits guard the waiting gate,
Receipts confirm the launch state.
Cleanup follows when work is done—
Hop, hop, the new flow runs!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.98% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: making stream launch behavior exact through a refactor.
Description check ✅ Passed The description directly explains the launch refactor, execution lifecycle changes, failure handling, and validation results.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Crane-Liu
Crane-Liu force-pushed the codex/worker-async-b6b-exact-launch branch from 337c960 to 1183755 Compare August 5, 2026 12:14

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/common/platform/onboard/host/c_api_shared.cpp (1)

609-623: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

A failure from abandon_prepared_execution is discarded.

The catch block at line 613 sets resources_rc = -1. The runner_resources_owned block at line 618 then assigns resources_rc again and overwrites that failure. Accumulate the first failure instead.

♻️ Proposed fix
     if (state->runner_resources_owned) {
         try {
-            resources_rc = state->runner->abandon_native_run_resources(state->descriptor.pipeline_slot);
+            int abandon_rc = state->runner->abandon_native_run_resources(state->descriptor.pipeline_slot);
+            if (resources_rc == 0) resources_rc = abandon_rc;
         } catch (...) {
             resources_rc = -1;
         }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/c_api_shared.cpp` around lines 609 - 623,
Preserve the failure recorded by abandon_prepared_execution in the surrounding
cleanup flow: update the runner_resources_owned block to assign its
abandon_native_run_resources failure only when resources_rc has not already
failed, rather than unconditionally overwriting resources_rc.
src/a2a3/platform/onboard/host/device_runner.cpp (1)

291-294: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Correct the stale error message.

The message names enqueue_run, which no longer exists, and states that activate_launch_shape must run first. activate_launch_shape now runs later, inside launch_run at line 587. The message misdirects triage.

📝 Proposed message fix
     if (block_dim < 1) {
-        LOG_ERROR("enqueue_run reached with unresolved block_dim; activate_launch_shape must run first");
+        LOG_ERROR("prepare_execution computed block_dim < 1 from worker_count=%d", runtime.get_worker_count());
         return -1;
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/a2a3/platform/onboard/host/device_runner.cpp` around lines 291 - 294,
Update the LOG_ERROR message in the block_dim validation guard to identify the
current caller/context instead of the removed enqueue_run and avoid claiming
activate_launch_shape must run first, since it executes later within launch_run.
Keep the existing error condition and return behavior unchanged.
🧹 Nitpick comments (6)
src/common/platform/onboard/host/device_runner_base.h (1)

578-602: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Update the doc comments to the new method names.

The comments still name poll_run(), drain_run(), and enqueue_run(). These methods no longer exist. The current names are poll_execution, drain_execution, and prepare_execution / launch_execution.

♻️ Proposed doc fix
     /**
      * Prepare host-owned execution state and submit the Runtime to the device.
      * Success means the platform's real kernel-launch boundary was crossed;
-     * all state needed by poll_run() and drain_run() remains owned by the
+     * all state needed by poll_execution() and drain_execution() remains owned by the
      * runner until drain completes.
      */
@@
     /**
      * Query the active run without waiting. Returns one of the
      * SIMPLER_NATIVE_RUN_POLL_* values. This may run concurrently with
-     * drain_run() on the compatibility executor.
+     * drain_execution() on the compatibility executor.
      */
@@
     /**
      * Wait for the enqueued run, publish DFX, and release its execution
-     * resources. Called on the same executor thread as enqueue_run().
+     * resources. Called on the same executor thread as launch_execution().
      */
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/device_runner_base.h` around lines 578 -
602, Update the documentation comments around prepare_execution, poll_execution,
and drain_execution to reference only the current method names: replace
poll_run() with poll_execution(), drain_run() with drain_execution(), and
enqueue_run() with the prepare_execution()/launch_execution() flow. Do not alter
method signatures or behavior.
src/a2a3/platform/sim/host/device_runner.cpp (1)

524-574: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Simulated launch paths run side effects outside the launch transaction. Both simulated runners publish platform bases, start collectors, and reset run_completion_ before exact_launch_transaction consumes the permit. A rejected permit returns NotStarted while the collectors already run and the completion counter already expects threads that are never created. The onboard a5 path performs this setup inside the AICore submission callback.

  • src/a2a3/platform/sim/host/device_runner.cpp#L524-L574: move the base publication, the collector starts, and the run_completion_.reset call into the AICore submission callback.
  • src/a5/platform/sim/host/device_runner.cpp#L450-L499: apply the same move for the base publication, the collector starts, and the run_completion_.reset call.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/a2a3/platform/sim/host/device_runner.cpp` around lines 524 - 574, Move
platform-base publication, collector startup, and run_completion_.reset from the
pre-transaction setup into the AICore submission callback in
src/a2a3/platform/sim/host/device_runner.cpp lines 524-574, and apply the same
change in src/a5/platform/sim/host/device_runner.cpp lines 450-499. Ensure these
side effects occur only after exact_launch_transaction accepts the permit, while
preserving their existing ordering relative to submission.
src/a5/platform/onboard/host/device_runner.cpp (1)

147-147: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove the unused local.

selected_pipeline_slot is assigned and never read. pipeline_slot is used directly.

♻️ Proposed fix
-    const uint32_t selected_pipeline_slot = pipeline_slot;
     auto prepare_rollback = RAIIScopeGuard([this, &execution]() {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/a5/platform/onboard/host/device_runner.cpp` at line 147, Remove the
unused local selected_pipeline_slot in the device_runner flow and keep using
pipeline_slot directly where the value is needed. Update the nearby logic in the
same function to reference the existing pipeline_slot symbol so the assignment
is eliminated without changing behavior.
tests/ut/cpp/common/test_native_run_execution.cpp (2)

112-129: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert that a consumed permit blocks a second submission.

The test proves that first is invalidated after the move at line 119. It does not prove the one-shot property that the name states: that an already-consumed permit refuses to submit.

Add a second transaction with the consumed permit and confirm no submission occurs.

✅ Proposed additional assertion
     EXPECT_EQ(success.progress, LaunchProgress::Complete);
     EXPECT_FALSE(first.valid());
+
+    int submissions = 0;
+    LaunchTransactionResult replay = exact_launch_transaction(
+        kIdentity, std::move(first),
+        [&]() {
+            ++submissions;
+            return 0;
+        },
+        [&]() {
+            ++submissions;
+            return 0;
+        }
+    );
+    EXPECT_EQ(replay.progress, LaunchProgress::NotStarted);
+    EXPECT_EQ(submissions, 0);
+    EXPECT_FALSE(replay.receipt.valid());
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/cpp/common/test_native_run_execution.cpp` around lines 112 - 129,
Extend NativeRunExecutionTest.PermitIsOneShot after the successful
exact_launch_transaction call by submitting the already-consumed first permit
again, then assert the second transaction is rejected and does not report
LaunchProgress::Complete. Preserve the existing validity checks and use the
test’s established transaction helper and result type.

29-50: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Extend identity-mismatch coverage to dispatch_id and pipeline_slot.

The tests exercise a mismatched run_epoch at line 31 and a stale generation at line 108. dispatch_id and pipeline_slot are never mismatched.

If the permit check or LaunchReceipt::matches compared only a subset of the four fields, every test in this file would still pass. The whole prepared-execution contract rests on full identity binding, so pin all four fields.

✅ Proposed parameterized mismatch coverage
TEST(NativeRunExecutionTest, EveryIdentityFieldIsBound) {
    const NativeRunIdentity mismatches[] = {
        {kIdentity.run_epoch + 1, kIdentity.generation, kIdentity.dispatch_id, kIdentity.pipeline_slot},
        {kIdentity.run_epoch, kIdentity.generation + 1, kIdentity.dispatch_id, kIdentity.pipeline_slot},
        {kIdentity.run_epoch, kIdentity.generation, kIdentity.dispatch_id + 1, kIdentity.pipeline_slot},
        {kIdentity.run_epoch, kIdentity.generation, kIdentity.dispatch_id, kIdentity.pipeline_slot + 1},
    };
    for (const NativeRunIdentity &other : mismatches) {
        int submissions = 0;
        LaunchTransactionResult result = exact_launch_transaction(
            other, NativeRunExecutionTestPeer::mint(kIdentity),
            [&]() {
                ++submissions;
                return 0;
            },
            [&]() {
                ++submissions;
                return 0;
            }
        );
        EXPECT_EQ(result.progress, LaunchProgress::NotStarted);
        EXPECT_EQ(submissions, 0);
        EXPECT_FALSE(result.receipt.valid());
    }
}

Also applies to: 105-109

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/cpp/common/test_native_run_execution.cpp` around lines 29 - 50,
Extend NativeRunExecutionTest identity-mismatch coverage to validate every
NativeRunIdentity field. Replace or augment the existing run_epoch and
generation cases with a test such as EveryIdentityFieldIsBound that exercises
mismatches in run_epoch, generation, dispatch_id, and pipeline_slot, asserting
each transaction remains NotStarted, performs no submissions, and returns an
invalid receipt.
src/a2a3/platform/onboard/host/device_runner.h (1)

305-305: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Doxygen blocks were not updated when KernelArgsHelper &kernel_args was threaded through the init_* declarations. Both onboard headers gained the same parameter on init_chip_swimlane, init_args_dump, init_pmu, init_dep_gen, and init_scope_stats. No @param kernel_args entry was added in either header, and several existing @param entries name parameters that the signatures do not have.

  • src/a2a3/platform/onboard/host/device_runner.h#L305-L305: add @param kernel_args to all five init_* blocks, and remove the @param runtime entry from the init_chip_swimlane block, which takes no Runtime.
  • src/a5/platform/onboard/host/device_runner.h#L250-L250: add @param kernel_args to all five init_* blocks, remove the @param runtime entry from the init_chip_swimlane block and the @param num_aicore entry from the init_args_dump block, and move the PMU block at lines 262-268 directly above init_pmu at line 276 so Doxygen stops attaching it to enable_dep_gen_.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/a2a3/platform/onboard/host/device_runner.h` at line 305, Update the
Doxygen blocks for init_chip_swimlane, init_args_dump, init_pmu, init_dep_gen,
and init_scope_stats in src/a2a3/platform/onboard/host/device_runner.h: add
`@param` kernel_args to each and remove `@param` runtime from init_chip_swimlane.
Apply the same updates in src/a5/platform/onboard/host/device_runner.h, also
remove `@param` num_aicore from init_args_dump and move the PMU documentation
block directly above init_pmu so it documents that declaration rather than
enable_dep_gen_.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/a5/platform/onboard/host/device_runner.cpp`:
- Around line 148-150: Update the rollback guard in device-runner execution
preparation so a prepared successor does not invoke the active run’s drain
cleanup. Ensure prepare_execution() failures leave shared collector state and
the predecessor’s RunPollState intact, while finalized collector teardown and
the Drained transition remain owned by the drain path used by poll_execution().

In `@src/common/platform/onboard/host/c_api_shared.cpp`:
- Around line 727-732: Move the assignment that clears
state->runner_resources_owned to after the rc != 0 failure check in the
prepare_execution flow. Keep the flag set when prepare_execution fails so
cleanup_failed_prepare can release resources, and transfer ownership only after
a successful preparation.

In `@src/common/platform/onboard/host/device_runner_base.cpp`:
- Around line 1285-1315: The reset-failure path in
DeviceRunnerBase::arm_device_wall_buffer currently frees device_wall_dev_ptr_
after it has been published through KernelArgsHelper, which can race with other
runs still reading it. Update this flow to avoid freeing the shared device-wall
buffer in place: either defer release until prior runs are complete or switch
the per-run setup to a separate buffer before clearing
kernel_args.args.device_wall_data_base. Keep the existing copy_to_device failure
handling and pointer reset behavior, but remove the immediate free of the shared
buffer from this method.

In `@src/common/worker/native_run_execution.h`:
- Around line 134-139: Update the exception handler in the Aicore submission
flow around AicoreSubmit so a thrown submission sets result.progress to Partial
while retaining the failure return code. Preserve the existing nonzero-result
handling, allowing callers to use the poison-and-retain path instead of
classifying the execution as NotStarted.

---

Outside diff comments:
In `@src/a2a3/platform/onboard/host/device_runner.cpp`:
- Around line 291-294: Update the LOG_ERROR message in the block_dim validation
guard to identify the current caller/context instead of the removed enqueue_run
and avoid claiming activate_launch_shape must run first, since it executes later
within launch_run. Keep the existing error condition and return behavior
unchanged.

In `@src/common/platform/onboard/host/c_api_shared.cpp`:
- Around line 609-623: Preserve the failure recorded by
abandon_prepared_execution in the surrounding cleanup flow: update the
runner_resources_owned block to assign its abandon_native_run_resources failure
only when resources_rc has not already failed, rather than unconditionally
overwriting resources_rc.

---

Nitpick comments:
In `@src/a2a3/platform/onboard/host/device_runner.h`:
- Line 305: Update the Doxygen blocks for init_chip_swimlane, init_args_dump,
init_pmu, init_dep_gen, and init_scope_stats in
src/a2a3/platform/onboard/host/device_runner.h: add `@param` kernel_args to each
and remove `@param` runtime from init_chip_swimlane. Apply the same updates in
src/a5/platform/onboard/host/device_runner.h, also remove `@param` num_aicore from
init_args_dump and move the PMU documentation block directly above init_pmu so
it documents that declaration rather than enable_dep_gen_.

In `@src/a2a3/platform/sim/host/device_runner.cpp`:
- Around line 524-574: Move platform-base publication, collector startup, and
run_completion_.reset from the pre-transaction setup into the AICore submission
callback in src/a2a3/platform/sim/host/device_runner.cpp lines 524-574, and
apply the same change in src/a5/platform/sim/host/device_runner.cpp lines
450-499. Ensure these side effects occur only after exact_launch_transaction
accepts the permit, while preserving their existing ordering relative to
submission.

In `@src/a5/platform/onboard/host/device_runner.cpp`:
- Line 147: Remove the unused local selected_pipeline_slot in the device_runner
flow and keep using pipeline_slot directly where the value is needed. Update the
nearby logic in the same function to reference the existing pipeline_slot symbol
so the assignment is eliminated without changing behavior.

In `@src/common/platform/onboard/host/device_runner_base.h`:
- Around line 578-602: Update the documentation comments around
prepare_execution, poll_execution, and drain_execution to reference only the
current method names: replace poll_run() with poll_execution(), drain_run() with
drain_execution(), and enqueue_run() with the
prepare_execution()/launch_execution() flow. Do not alter method signatures or
behavior.

In `@tests/ut/cpp/common/test_native_run_execution.cpp`:
- Around line 112-129: Extend NativeRunExecutionTest.PermitIsOneShot after the
successful exact_launch_transaction call by submitting the already-consumed
first permit again, then assert the second transaction is rejected and does not
report LaunchProgress::Complete. Preserve the existing validity checks and use
the test’s established transaction helper and result type.
- Around line 29-50: Extend NativeRunExecutionTest identity-mismatch coverage to
validate every NativeRunIdentity field. Replace or augment the existing
run_epoch and generation cases with a test such as EveryIdentityFieldIsBound
that exercises mismatches in run_epoch, generation, dispatch_id, and
pipeline_slot, asserting each transaction remains NotStarted, performs no
submissions, and returns an invalid receipt.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f9781c42-a7fc-483a-99f1-cf5fb968fc09

📥 Commits

Reviewing files that changed from the base of the PR and between b535fa2 and 1183755.

📒 Files selected for processing (22)
  • src/a2a3/platform/onboard/host/device_runner.cpp
  • src/a2a3/platform/onboard/host/device_runner.h
  • src/a2a3/platform/sim/host/device_runner.cpp
  • src/a2a3/platform/sim/host/device_runner.h
  • src/a5/platform/onboard/host/device_runner.cpp
  • src/a5/platform/onboard/host/device_runner.h
  • src/a5/platform/sim/host/device_runner.cpp
  • src/a5/platform/sim/host/device_runner.h
  • src/common/platform/onboard/host/c_api_shared.cpp
  • src/common/platform/onboard/host/device_runner_base.cpp
  • src/common/platform/onboard/host/device_runner_base.h
  • src/common/platform/onboard/host/device_runner_helpers.h
  • src/common/platform/sim/host/c_api_shared.cpp
  • src/common/platform/sim/host/device_runner_base.cpp
  • src/common/platform/sim/host/device_runner_base.h
  • src/common/worker/native_run_context.h
  • src/common/worker/native_run_execution.h
  • src/common/worker/native_run_launch_signal.h
  • tests/ut/cpp/CMakeLists.txt
  • tests/ut/cpp/common/native_run_execution_test_peer.h
  • tests/ut/cpp/common/test_native_run_execution.cpp
  • tests/ut/cpp/common/test_native_run_launch_signal.cpp

Comment thread src/a5/platform/onboard/host/device_runner.cpp
Comment thread src/common/platform/onboard/host/c_api_shared.cpp
Comment thread src/common/platform/onboard/host/device_runner_base.cpp
Comment thread src/common/worker/native_run_execution.h
@Crane-Liu
Crane-Liu force-pushed the codex/worker-async-b6b-exact-launch branch 2 times, most recently from 7e5227a to 92006cf Compare August 6, 2026 03:00
@ChaoWao

ChaoWao commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Review — B6b exact launch seam

Reviewed dddeed6e (+ the CI budget commit 92006cf5) against
worker-async-pipeline/implementation-plan.md §B6b and contracts.md
§PlatformRunAdapter.

The shape is right, and two changes in particular solve the real problem rather
than wrapping it:

  • Receipt-only acceptance. Acceptance used to be published from inside
    DeviceRunnerBase::launch_aicpu_kernel()'s if (rc == 0) — any caller of that
    helper published. Now the only publisher is a LaunchReceipt that exists only
    when both submissions succeeded, and the runner's raw native_launch_signal_
    pointer is gone.
  • activate_launch_shape() moved into the launch transaction, with prepare
    deriving block_dim locally from runtime.get_worker_count(). worker_count_
    / block_dim_ are runner-global; as long as prepare wrote them, a successor's
    preparation would trample an executing predecessor's geometry. This is the load-
    bearing change that makes overlapped prepare actually safe.

The onboard launch transaction now contains only what the plan's list allows
(permit check, handshake reset, collector arm, submitted publication, the two
submissions, receipt). Below is what I think should change before merge.

Must fix

1. attach_current_thread failure leaks the whole PreparedExecution
src/common/platform/onboard/host/c_api_shared.cpp

if (attach_rc == 0) { ... entered_run = true; launch_execution(std::move(state->prepared_execution), ...); }
else { rc = attach_rc; }
...
if (!entered_run && state->runner_resources_owned) { abandon_native_run_resources(...); }

runner_resources_owned is now unconditionally cleared at the end of
simpler_prepare_run, so that branch can never fire. state->prepared_execution
is still non-null (never moved), and simpler_finalize_run gates its abandon on
!launched — but phase was set to Launching before the thread was created, so
launched == true and the abandon is skipped. PreparedExecution has no
destructor-side cleanup, so regs, pmu_reg_addrs, runtime_args, the device
KernelArgs copy and (a2a3) the run AICore stream all leak.

Before this PR the same path was covered: runner_resources_owned was still true,
so abandon_native_run_resources retired the stream, and there was no per-run
memory to lose.

Sim is worse: abandon_prepared_execution is what resets active_run_. Skip it
and active_run_ stays non-null forever, so every subsequent prepare_execution
returns -1 ("prepare_execution called while another simulated run still owns execution state") — the runner is permanently wedged.

Suggest gating finalize on state->prepared_execution != nullptr rather than
!launched (cleanup_execution is already idempotent via resources_owned), or
abandoning the prepared execution in the !entered_run branch.

2. A device_wall reset failure now kills the run — a2a3:588, a5:342

if (arm_device_wall_buffer(prepared.kernel_args) != 0) return -1;   // → rc=-1, whole run fails

The old code treated this as explicitly non-fatal: warn, clear the base, keep
running with capture disabled. The doc comment on ensure_device_wall_buffer is
unchanged and still says so:

Allocation or reset failure is non-fatal; the base stays null and timing reads as 0.

and arm_device_wall_buffer's own log still says "disabling phase capture this run" immediately before returning the code that aborts the run. Control flow, log
text and doc comment now disagree three ways. Either return 0 after clearing the
base, or update both the comment and the log and say why it became fatal.

(Only reachable under SIMPLER_DEVICE_STRACE_ENABLE + TIMING logging, so CI will
not catch it.)

3. Partial-launch poison is not wired on onboard

LaunchProgress::Partial is the PR's central new state, but:

  • only the two sim runners call if (result.poisoned()) poison_launch();;
  • poison_launch() / launch_poisoned_ exist only on SimDeviceRunnerBase;
  • LaunchOutcome::poisoned() has zero callers on the onboard side;
  • ActiveExecution::progress is stored and never read anywhere in the tree.

Onboard poisons today only incidentally: a non-zero AICPU launch rc makes the
lambda call recover_device_or_mark_unusable(), which sets device_unusable_.
But ExactLaunchTransaction deliberately wraps each submit in catch (...) — if
the AICPU submission throws, rc = -1 and progress = Partial while
recover_device_or_mark_unusable() never runs, so admission is not poisoned
even though the AICore kernel is already in flight. That is exactly the cell
contracts.md requires to poison.

implementation-plan.md lists "partial-launch poison" as required B6b evidence;
onboard does not implement it. Suggest mirroring sim explicitly:
if (transaction.poisoned()) recover_device_or_mark_unusable(transaction.rc);

4. apply_call_config in prepare_execution defeats the concurrency guard

c_api_shared.cpp carries an explicit guard whose comment states the invariant:

// Diagnostic binding reads runner-global collector configuration. It
// is depth-one, while concurrent HBG preparation must leave the active
// run's configuration untouched until launch.
if (!overlaps_active_run) runner->apply_call_config(state->config);

Two lines later prepare_execution calls apply_call_config(config)
unconditionally (a2a3:261, a5:153). That call used to live in enqueue_run, which
ran inside the exclusive launch window after the predecessor had finished — so the
guard held. Moving the body into prepare puts it in exactly the overlappable
window the guard exists to protect, making the guard dead code.

It does not misbehave today, because overlap requires !diagnostics_any() on both
runs so the same zeros get written — but set_output_prefix() still writes the
runner's std::string from another thread, and the runtime_maker comment says the
gate is meant to widen ("until their state is per-epoch"). The guard and its callee
now contradict each other; one of them should go.

5. a2a3/a5 divergence plus a dead runner member

This PR removes kernel_args_.args.scope_stats_data_base = 0; from a2a3's
finalize_collectors() (correct — those args now die with the PreparedExecution)
but leaves the identical line in a5
(src/a5/platform/onboard/host/device_runner.cpp:831), where it now writes to
DeviceRunnerBase::kernel_args_ — a runner member that no longer feeds any launch.
Grepping the onboard tree, that line plus the declaration
(device_runner_base.h:1104) and one doc comment (:849) are the only remaining
references. The member should be deleted and a5's line removed with it.
codestyle.md rule 10 calls out this exact four-tree divergence.

6. Renamed methods still referenced in ~20 places (doc-consistency §1)

enqueue_run / drain_run / poll_run no longer exist:

  • docs/dynamic-linking.md — the SO-lifetime table (252-253) and both
    call-flow diagrams (318/323/327 and 380/386/389) still name them.
  • device_runner_base.h — class docblock 31-32, plus 244, 581, 594, 600, 628.
  • a2a3/device_runner.h 218, 261; a5/device_runner.h 201;
    sim/device_runner_base.h 294.
  • Log strings: a2a3 sim:630 / a5 sim:556 "drain_run called without an enqueued simulated run"; a5:458 "drain_run slot mismatch"; a2a3 sim:828 comment.

a2a3:292 / a5:186 is wrong twice over:

LOG_ERROR("enqueue_run reached with unresolved block_dim; activate_launch_shape must run first");

The function is no longer enqueue_run, and activate_launch_shape now runs
after this check (inside launch) — the block_dim here comes from
prepare_launch_shape writing worker_count into the Runtime. As written this
message sends the next person debugging it the wrong way.

Should fix

7. PreparedExecution is a token, not an owner, on sim.
abandon_prepared_execution(PreparedExecution&), poll_execution(const ActiveExecution&) and drain_execution(ActiveExecution&) all ignore their
argument entirely; the real state stays in the runner's active_run_ /
kernel_args_. contracts.md says PreparedExecution "is a move-only owner of
every per-run value needed for launch and cleanup" — that does not hold on sim.
Capacity is one there so nothing breaks today, but the type promises something it
does not deliver.

Relatedly, sim's launch_execution calls set_platform_regs_func_() and starts
collector threads before exact_launch_transaction — i.e. side effects land
before the permit is validated. Onboard puts these inside the lambda (after
consume). contracts.md wants launch to "consume it, validate its identity, and
perform only the bounded device-exclusive transition"; the two platforms differ.

8. drain_execution never reads active.progress. Its RAII guard runs
cleanup_execution(retire_aicore=true) unconditionally, including when reap_run
times out — i.e. resources are retired without quiescence having been proven, where
contracts.md says a poisoned run "and every resource it may own stay retained until
a later drain or whole-context teardown proves reclamation." The drain shape
predates this PR, but B6b introduces progress and then never consults it.

Also worth stating in the PR body: on a partial launch the old code returned from
run() without draining (rollback retired the stream); the new code routes
Partial into drain_executionreap_runsync_stream_pair, so it now waits
out a stream-sync timeout on the orphaned AICore. Per contracts that is the correct
direction (drain proves quiescence), but it is not free and should be called out.

9. Dead surface added.

  • PreparedExecution's move constructor — and the KernelArgsHelper move
    constructor added to support it — are never invoked; the object only ever travels
    inside a unique_ptr, and operator=(&&) is deleted.
  • PreparedExecution::config is read by none of the four backends, and it is a

    1 KB struct (char output_prefix[1024]) copied per run.

  • Sim's PreparedExecution gained virtual ~; onboard's did not; neither has a
    derived class.
  • class NativeRunLaunchSignal; is now a dead forward declaration in both base
    headers.

10. The runner_resources_owned handoff is implicit.
state->runner_resources_owned = false; sits before the rc check, relying on
prepare_execution's internal prepare_rollback to have retired the stream. But
the first early return (if (prepared == nullptr || *prepared != nullptr) return -1;) is before that guard is armed, so nothing retires there. Unreachable today,
but the "caller lets go, callee promises to catch" coupling needs a comment at
minimum. See also finding 1: both remaining runner_resources_owned checks after
prepare are now dead branches.

11. PreparedExecution / ActiveExecution / LaunchOutcome are defined twice,
in the onboard and sim base headers, both under src/common/, differing by three
fields. contracts.md: "Platform variation lives behind one internal seam shared
by onboard and simulation adapters."

12. Signatures diverge from contracts.md — documented as PrepareOutcome prepare(NativeRunContext&, PreparePermit), PollOutcome poll(const ActiveExecution&) noexcept, DrainOutcome drain(ActiveExecution&, Deadline);
implemented as int prepare_execution(..., unique_ptr<PreparedExecution>*), a
non-noexcept poll_execution, and a drain_execution with no deadline. Probably
deliberate staging (Deadline belongs to B6c), but since .docs is the acceptance
criterion, either reconcile the docs or add the B6b entry to
implementation-record.md recording the deviation — there is no B6b entry there yet.

13. Test coverage does not reach B6b's headline claim. The six new unit tests
are clean, but they all sit at the pure-function layer. Of the plan's required
evidence:

  • partial-launch poison — only result.poisoned() == true is asserted; nothing
    checks that the runner's can_accept_run() goes false (finding 3 is exactly that
    gap). sim's newly added can_accept_run() gate in simpler_prepare_run is also
    untested.
  • no execution overlap — not tested.
  • "for an eligible successor, all material preparation overlaps its predecessor;
    the serialized predecessor-terminal-to-receipt interval contains only the bounded
    launch transaction"
    — the central claim of the stage, with no measurement or
    assertion anywhere.

CI

The previous head's st-onboard-a2a3 failed with exit 124 — the pytest 600 s
session wall, fired after every case reported PASSED (100%). Pulling the logs,
#1695's run of the same job used 587 s of the same 600 s budget, so this lane was
already running against the wall; it is not a B6b slowdown.

92006cf5 raising --pto-session-timeout 600 → 1200 (matching a5) is the right
call. Two notes: it touches .github/workflows/_st-npu-a2a3.yml, which is CI
implementation and correctly outside NON_CODE, so it forces the full matrix — no
action needed there. But bundling a CI budget change into a refactor invites the
next reviewer to read it as "this PR got slower"; either split it out or state the
587 s/600 s evidence in the PR body.

Summary

The direction is right and the two load-bearing changes (receipt-only acceptance,
activate_launch_shape moved into the transaction) genuinely fix the ownership
problem rather than papering over it. Findings 1, 2, 3, 5 and 6 should land before
merge; finding 4 needs at least an explanation, since the guard and its callee
currently contradict each other.

Materialize platform execution state during native prepare, consume an
identity-bound launch permit at the two-submission boundary, and publish
acceptance only from a completed launch receipt.

Retain and poison uncertain partial launches, isolate prepared cleanup from
active-run state, and release resources on every launch failure path. Keep
device-wall capture failures non-fatal and place simulation side effects
inside the exact launch transaction.

Extend the main a2a3 scene session budget to match the expanded corpus while
retaining the shorter limits for isolated smoke and SDMA steps.
@Crane-Liu
Crane-Liu force-pushed the codex/worker-async-b6b-exact-launch branch from 92006cf to e401177 Compare August 6, 2026 03:47
@Crane-Liu

Copy link
Copy Markdown
Contributor Author

@ChaoWao Addressed the B6b review:

  • Prepared execution is reclaimed when the executor thread cannot attach; finalization no longer skips it merely because launch entered the Launching phase.
  • Prepare ownership transfers only on success, cleanup preserves the first failure, and prepared-successor rollback cannot finalize an active predecessor collector or publish Drained.
  • Device-wall reset failure is non-fatal and no longer frees a buffer another run may still reference.
  • Uncertain AICore/AICPU exceptions produce Partial; onboard now poisons admission explicitly.
  • Runner-global call config is no longer reapplied from overlappable prepare, the dead onboard kernel_args_ member is removed, and stale lifecycle names are updated.
  • Simulation platform publication, collector arming, phase setup, and completion reset now occur only after the launch permit is consumed.
  • Direct tests cover both AICore and AICPU exception outcomes.

The progress-aware bounded drain/deadline work remains in the following B6c PR. Consolidating the onboard/sim execution type definitions and the larger PlatformRunAdapter signature cleanup also remain separate so this PR stays on the exact-launch boundary.

@ChaoWao

ChaoWao commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Re-review — e4011771

Re-read the whole diff against the previous head. Most of the round-one findings
are properly closed, and two of the fixes are better than what I suggested
(splitting retire_aicore ownership between prepare_rollback and
cleanup_failed_prepare instead of relying on an implicit handoff; gating a5's
run_poll_state_ = Drained on launched). One of the new fixes introduces a
regression, though — details below.

Closed

# Finding Status
1 attach_current_thread failure leaked the PreparedExecution fixed — !entered_run now abandons it on both onboard and sim, and simpler_finalize_run gates on prepared_execution != nullptr instead of !launched
2 device-wall reset failure became fatal fixed — (void)arm_device_wall_buffer(...), non-fatal again, matching the doc comment
3 partial-launch poison not wired on onboard fixed — if (transaction.poisoned()) recover_device_or_mark_unusable(transaction.rc); on both arches, per-lambda recover removed for the AICPU step; plus the transaction now treats an AICore throw as Partial
4 apply_call_config in prepare_execution defeated the overlap guard fixed — removed from both prepare_executions; the c_api's guarded call is now the only one
5 a5 kept kernel_args_.args.scope_stats_data_base = 0 / dead kernel_args_ member fixed — line removed, KernelArgsHelper kernel_args_ deleted from DeviceRunnerBase
6 ~20 stale enqueue_run / drain_run / poll_run references fixed — repo-wide grep is clean, docs/dynamic-linking.md call-flow diagrams updated, and the two misleading LOG_ERRORs replaced with "prepare_execution computed block_dim < 1 from worker_count=%d"
10 implicit runner_resources_owned handoff fixed — cleared only after the rc check, and prepare_rollback now passes retire_aicore=false so the stream stays the c_api's until prepare succeeds. Clean split
7 (partial) sim side effects landed before the permit was validated fixed — all the set_platform_*_func_ calls, collector starts, phase-base publication and run_completion_.reset moved inside the AICore closure

Also: st-onboard-a2a3 is green at 8m36s on the new head, which supports the
earlier read that the 600 s trip was budget-margin rather than a B6b slowdown.

New — blocking

finalize_collectors() is now skipped on every non-launched cleanup path.

void DeviceRunner::cleanup_execution(PreparedExecution &prepared, bool launched, bool retire_aicore) noexcept {
    if (!prepared.resources_owned) return;
    if (launched) finalize_collectors();          // ← a2a3:501, a5 equivalent

abandon_prepared_execution() passes launched=false, and so does
prepare_rollback. But the collectors are initialized during prepare
(init_chip_swimlane / init_args_dump / init_pmu / init_dep_gen /
init_scope_stats, each under its enable_*_ flag), and on a2a3
start_shared_collectors_for_run() + dep_gen_collector_.start() run inside the
AICore closure, before launch_aicore_kernel
. So three reachable paths now leak:

  1. prepare fails after some init_* succeeded → rollback skips teardown → device
    shm + host registration leak, and is_initialized() stays true so the next run's
    init_* hits an already-initialized collector.
  2. prepared but never launched (finalize without launch, or the new
    !entered_run abandon after an attach_current_thread failure) → same.
  3. launch returns NotStarted after the closure already started the collectors
    mark_submitted failure, or launch_aicore_kernel failure (i.e. the 207001 path
    this file has a whole recovery routine for). The mgmt/poll threads are already
    running
    and nothing stops them.

Before this revision all three called finalize_collectors().

The stated rationale — "Prepared-only cleanup must not touch collector state owned
by an active predecessor"
— does not hold, because an overlapping successor can
only exist when neither run has diagnostics:

  • allow_prepared_successor = concurrent_native_prepare_supported_impl() && !config->diagnostics_any() (the successor's own config), and
  • try_reserve_native_run additionally requires existing->permits_prepared_successor, which is the predecessor's allow_prepared_successor.

So whenever an overlap is in flight, no collector is initialized on either side, and
finalize_collectors() — whose every branch is guarded by is_initialized() — is
already a no-op. The gate cannot prevent a cross-run teardown that can't happen; it
only suppresses this run's own teardown.

Suggest dropping the launched parameter and restoring the unconditional call. If a
real overlap hazard does turn up later, the correct predicate is "an active
predecessor exists" (native_run_active()), not "this run launched". (The a5
run_poll_state_ = Drained gate is a different question and is correctly
launched-conditioned — keep that one.)

New — should fix

A pre-submission throw now poisons the device.

} catch (...) {
    result.rc = -1;
    // Submission callbacks may throw after starting work. Without a
    // receipt, the caller must retain resources and poison admission.
    result.progress = LaunchProgress::Partial;
    return result;
}

The intent is right for the AICPU step and for the actual submission call. But the
AICore closure is not only a submission — on a2a3 it also runs
activate_launch_shape, arm_device_wall_buffer, start_shared_collectors_for_run()
(thread creation), dep_gen_collector_.start() (thread creation), and
std::vector<CoreType> core_types(num_aicore); sim's closure is larger still. A
std::bad_alloc from that vector or a std::system_error from create_thread is
provably before the first stream submission, yet it now lands as Partial
recover_device_or_mark_unusable()device_unusable_ → force-reset at finalize
(and poison_launch() on sim).

contracts.md is explicit: "Failure before the first stream submission is
FAILED_SAFE."
Cheapest fix that keeps the conservative default where it belongs:
wrap the arming prologue inside the closure in its own try { ... } catch (...) { return -1; } so only a throw from launch_aicore_kernel (and the sim thread spawns
that stand in for it) maps to Partial.

Still open from round one (non-blocking, your call)

  • 8drain_execution never reads active.progress; its RAII guard retires
    resources even when reap_run timed out, i.e. without quiescence proven.
  • 9 — dead surface: PreparedExecution's move ctor (and the KernelArgsHelper
    move ctor added for it) is never invoked; PreparedExecution::config still has no
    reader in any of the four backends; sim's PreparedExecution still has a
    virtual ~ with no derived class while onboard's does not; class NativeRunLaunchSignal; is still a dead forward declaration in both base headers.
  • 7 (rest) — sim's poll_execution / drain_execution /
    abandon_prepared_execution still ignore their arguments and drive the runner's
    active_run_, so PreparedExecution remains a token rather than an owner there.
  • 11PreparedExecution / ActiveExecution / LaunchOutcome still defined
    twice under src/common/.
  • 12 — signatures still diverge from contracts.md; no B6b entry in
    implementation-record.md recording the deviation.
  • 13 — the two new transaction tests (AICore throw, AICPU throw) are good
    additions, but the plan's partial-launch poison evidence still stops at
    result.poisoned(); nothing asserts the runner's can_accept_run() goes false, and
    sim's new can_accept_run() gate in simpler_prepare_run remains untested. The
    stage's headline claim — material preparation overlapping the predecessor, with only
    the bounded transaction serialized — still has no measurement.

Summary

Six of six must-fixes from round one are closed, and the ownership split around
retire_aicore came out cleaner than what I proposed. The collector-teardown gate is
the one thing I'd hold the merge on — it turns a hazard that cannot occur into three
leaks that can, including on the AICore-launch-failure path. The throw→Partial
widening is worth narrowing in the same pass since it is two lines.

@ChaoWao
ChaoWao merged commit 3c64da8 into hw-native-sys:main Aug 6, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants