Skip to content

Fix: avoid remote teardown after fatal AICore errors #1664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sunkaixuan2018 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix: avoid remote teardown after fatal AICore errors #1664

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions docs/investigations/2026-07-a2a3-sdma-fault-teardown.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,63 @@
risk and ordinary Workers are unaffected; full recovery inside an SDMA-enabled
Worker after a fault is deferred pending a CANN runtime-and-driver fix

## 2026-08 package-specific follow-up

PR #1664 adds a narrower containment for the installed CANN 9.0.0 and driver
26.0.rc1 combination. Only run streams belonging to a Worker that actually
provisioned the SDMA workspace use stop-on-failure mode. Fatal close on such a
Worker then waits for `max(10 seconds, configured op timeout + 5 seconds)`,
attempts one force reset, and forgets the failed generation's host handles
without issuing per-resource destroy/free calls.

This is an empirically validated workaround, not an SDMA retirement contract.
CANN still exposes neither a completion fence nor a portable upper bound for
the final CP-process stream release. Ordinary a2a3 Workers and all a5 Workers
therefore keep normal stream mode, their existing error/diagnostic behavior,
and no added handoff delay. The hardware regression test keeps the
package-specific workaround visible by covering real SDMA provisioning followed
by an AICore fault and a bounded `Worker.close()`.

### Reset attempts are budgeted per stream population, not globally

`force_reset_device()` drains the card before resetting it and returns 0 only
when its post-reset probe confirms a usable generation, so a second call runs
against a settled card and can recover a poison the first could not. Ordinary
poison therefore keeps a bounded three-attempt budget on both a2a3 and a5.

A Worker holding the 48 CP-process SDMA streams is the exception and gets a
single attempt: there a reset that does not confirm has already blocked on the
driver's 150/300-second remote-event timeout, so a retry multiplies that wait
without adding a completion condition.

Collapsing both populations onto a single attempt regresses ordinary
fault-injection recovery — the poisoned card stays poisoned for the next
process that lands on it, which surfaces as unrelated tests failing with
507018/507046 on the devices a fault-injection test just used.

### Emergency shutdown broadcasts in parallel but still joins the cores

The AICore exit acknowledgement is what leaves a card usable for the next
process: only after a core confirms it stopped does the AICPU quiesce that
core's register block, putting the dispatch register back to idle and closing
the fast path. Returning from emergency shutdown while cores are still running
leaves the card poisoned past the host's device reset, and the next process on
that device fails at launch.

So the fatal path signals every handshake'd core first and joins them second,
rather than signalling and waiting one core at a time. Cores drain
concurrently and the per-core quiesce is preserved.

The join takes **one deadline for the whole group**, not one timeout per core.
That distinction is load-bearing on the fatal path, where every core is
typically dead: the onboard deinit timeout is 1 second, so a per-core deadline
costs a second per unresponsive core and pushes an AICore-timeout run past a
10-second budget, while a shared deadline caps the whole group at one second.

The wait is an on-device poll of the core's `COND` register, so it costs no
host or remote operation and is unrelated to the CANN SDMA teardown problem
above.

## Question

Can simpler keep PTO-ISA async SDMA available while avoiding the roughly
Expand Down
24 changes: 24 additions & 0 deletions src/a2a3/platform/include/aicpu/platform_regs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,30 @@ inline void write_reg(uint64_t reg_base_addr, RegId reg, uint64_t value) {
*/
void platform_init_aicore_regs(uint64_t reg_addr);

/** Send the AICore exit signal without waiting for an acknowledgement. */
void platform_signal_aicore_exit(uint64_t reg_addr);

/**
* Absolute sys-cnt value one deinit timeout from now. Share a single deadline
* across a group of cores so the whole group costs one timeout rather than one
* per unresponsive core.
*/
uint64_t platform_aicore_exit_deadline();

/**
* Wait for a signalled AICore to acknowledge exit, then quiesce its register
* block (dispatch register back to idle, fast path closed).
*
* Pairs with platform_signal_aicore_exit when stopping several cores: signal
* them all, take one deadline from platform_aicore_exit_deadline(), then finish
* each core against that shared deadline.
*
* @param deadline absolute sys-cnt value to give up at.
* @return 0 once the core acknowledges, -1 on timeout — an unresponsive core is
* left for the host's device reset to clear.
*/
int32_t platform_finish_aicore_exit(uint64_t reg_addr, uint64_t deadline);

/**
* Deinitialize AICore registers before termination
*
Expand Down
205 changes: 146 additions & 59 deletions src/a2a3/platform/onboard/host/device_runner.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,14 @@

#include <dlfcn.h>

#include <algorithm>
#include <cassert>
#include <chrono>
#include <cstddef>
#include <cstring>
#include <iostream>
#include <string>
#include <thread>
#include <vector>
#include "acl/acl.h"
#include "host/acl_error_log.h"
Expand All @@ -43,6 +46,7 @@
#include "utils/elf_build_id.h"
#include "host/host_regs.h" // Register address retrieval
#include "host/raii_scope_guard.h"
#include "utils/fatal_shutdown_latch.h"

// dep_gen has two shapes, one per orchestration site, and each runtime provides
// the strong symbols for the one it uses:
Expand Down Expand Up @@ -525,26 +529,70 @@ int DeviceRunner::drain_run(uint32_t pipeline_slot) {
void DeviceRunner::cleanup_active_run(bool retire_aicore) noexcept {
if (!active_run_.owns_resources) return;

// A poisoned card cannot retire per-resource frees or stream destroys, and
// issuing them can block in the driver. Drop host-side ownership instead;
// finalize()'s force reset invalidates the whole device generation.
const bool abandon = device_unusable_.load(std::memory_order_acquire);

// Collectors must stop before their backing arguments are released; the
// per-run stream retires last. Each cleanup operation is idempotent.
finalize_collectors();
(void)kernel_args_.finalize_device_kernel_args();
(void)kernel_args_.finalize_runtime_args();
finalize_collectors(abandon);
if (abandon) {
kernel_args_.abandon_after_device_failure();
} else {
(void)kernel_args_.finalize_device_kernel_args();
(void)kernel_args_.finalize_runtime_args();
}
if (kernel_args_.args.pmu_reg_addrs != 0) {
(void)mem_alloc_.free(reinterpret_cast<void *>(kernel_args_.args.pmu_reg_addrs));
if (!abandon) {
(void)mem_alloc_.free(reinterpret_cast<void *>(kernel_args_.args.pmu_reg_addrs));
}
kernel_args_.args.pmu_reg_addrs = 0;
}
if (kernel_args_.args.regs != 0) {
(void)mem_alloc_.free(reinterpret_cast<void *>(kernel_args_.args.regs));
if (!abandon) {
(void)mem_alloc_.free(reinterpret_cast<void *>(kernel_args_.args.regs));
}
kernel_args_.args.regs = 0;
}
if (retire_aicore && !active_run_.aicore_retirement_attempted) {
if (retire_aicore && !abandon && !active_run_.aicore_retirement_attempted) {
active_run_.aicore_retirement_attempted = true;
(void)retire_run_aicore_stream(active_run_.slot, RunStreamSlots::CompletionStatus::Unproven);
}
active_run_.owns_resources = false;
}

int DeviceRunner::create_run_stream(void **out) {
if (out == nullptr) return -1;
*out = nullptr;

rtStream_t stream = nullptr;
int rc = rtStreamCreate(&stream, 0);
if (rc != 0) {
LOG_ERROR("rtStreamCreate (run stream) failed: %d", rc);
ACL_LOG_ERROR_DETAIL(rc);
return rc;
}

// CANN 9.0.0 + driver 26.0.rc1 needs the host to observe an SDMA-generation
// AICore fault before the device reaches DEV_RUNNING_DOWN; otherwise reset
// walks the 48 CP-process streams and blocks in a 300-second remote event.
// Keep this workaround strictly on Workers that actually provisioned SDMA.
// Ordinary a2a3 and every a5 stream retain their normal error/diagnostic
// contract (for example 507046 rather than the early 507015).
if (dma_workspace_handle_ != nullptr) {
aclError acl_rc = aclrtSetStreamFailureMode(stream, ACL_STOP_ON_FAILURE);
if (acl_rc != ACL_SUCCESS) {
LOG_ERROR("aclrtSetStreamFailureMode (SDMA run stream) failed: %d", static_cast<int>(acl_rc));
(void)rtStreamDestroy(stream);
return static_cast<int>(acl_rc);
}
}

*out = stream;
return 0;
}

int DeviceRunner::ensure_run_stream_set(unsigned slot) {
int rc = run_stream_slots_.acquire(slot);
if (rc != 0) {
Expand Down Expand Up @@ -948,6 +996,86 @@ int DeviceRunner::finalize() {
return 0;
}

// Fatal path: the ordinary stream completion/error boundary has already
// reaped the submitted run. Stop host collector threads locally, drain and
// force-reset the card, then forget old handles without per-resource
// RTS/HAL calls.
if (device_unusable_.load(std::memory_order_acquire)) {
// A Worker that provisioned SDMA holds 48 CP-process streams, which is
// what makes both the handoff delay and the single reset attempt below
// necessary. Read before abandon_common_after_device_failure() clears
// the handle.
const bool sdma_provisioned = dma_workspace_handle_ != nullptr;

// CANN exposes no retirement fence for its CP-process SDMA streams.
// On the package named above, repeated hardware A/B runs established
// this handoff delay as a containment workaround, not a portable
// runtime guarantee. Do not delay ordinary (non-SDMA) failures.
if (sdma_provisioned) {
constexpr uint64_t kCANN900FatalHandoffMs = 10000;
constexpr uint64_t kDeviceDownSettleMarginMs = 5000;
const uint64_t op_timeout_ms = (timeout_config_.op_execute_timeout_us + 999) / 1000;
const uint64_t configured_handoff_ms = op_timeout_ms + kDeviceDownSettleMarginMs;
const uint64_t reset_delay_ms = std::max(kCANN900FatalHandoffMs, configured_handoff_ms);
LOG_WARN(
"SDMA fatal teardown workaround (CANN 9.0.0/driver 26.0.rc1): "
"waiting %llu ms before force reset; no runtime retirement fence is available",
static_cast<unsigned long long>(reset_delay_ms)
);
std::this_thread::sleep_for(std::chrono::milliseconds(reset_delay_ms));
}

finalize_collectors(true);

// force_reset_device() drains before it resets and returns 0 only when
// its post-reset probe confirms the card, so a second pass runs against
// a settled card and can recover a poison the first pass could not
// (verified on a2a3). An SDMA-provisioned card gets a single attempt:
// there a non-confirming reset already blocks on the driver's
// remote-event timeout, which a retry only multiplies.
constexpr int kFatalResetAttempts = 3;
int reset_rc = attempt_fatal_reset(
[this]() {
return force_reset_device();
},
sdma_provisioned ? 1 : kFatalResetAttempts
);
const bool reset_confirmed = reset_rc == 0;
if (!reset_confirmed) {
LOG_ERROR(
"Fatal teardown: force reset of device %d did not confirm clean (rc=%d); "
"quarantining old handles without per-resource RTS calls",
device_id_, reset_rc
);
}

run_stream_slots_.abandon_all();
int abandon_rc = abandon_common_after_device_failure();

// Only finalize the ACL owner after force reset established a clean
// generation. On reset failure, aclFinalize may itself walk poisoned
// stream resources and enter the same remote-event timeout.
if (acl_ready_) {
if (reset_confirmed) {
int finalize_rc = aclFinalize();
if (finalize_rc != 0) {
LOG_ERROR("aclFinalize failed during fatal finalize: %d", finalize_rc);
if (abandon_rc == 0) abandon_rc = finalize_rc;
}
} else {
LOG_WARN("Fatal teardown: skipping aclFinalize because device reset was not confirmed");
}
acl_ready_ = false;
}

device_id_ = -1;
if (reset_confirmed) {
device_unusable_.store(false, std::memory_order_release);
}
LOG_WARN("DeviceRunner finalized after fatal device failure");
return abandon_rc != 0 ? abandon_rc : reset_rc;
}

int rc = attach_current_thread(device_id_);
if (rc != 0) {
LOG_ERROR("Failed to attach finalize thread to device %d: %d", device_id_, rc);
Expand Down Expand Up @@ -1007,57 +1135,11 @@ int DeviceRunner::finalize() {
}
}

// On the poison path the soft reset above does NOT clear the op-timeout
// sticky-error — a fresh in-process Worker.init then fails at rtStreamCreate
// 507899. A FORCE reset clears it, so the next Worker on this card inits
// clean in the SAME process and the remaining tests run instead of cascading
// / being skipped. Only reached on the (rare) device-poison path; onboard
// work always holds an exclusive task-submit lock on the card (enforced by
// .claude/rules/running-onboard.md), and the reset scopes to this card alone,
// so it cannot disturb other devices/users.
int reset_rc = 0;
if (device_unusable_.load(std::memory_order_acquire)) {
// Bounded retry: a single force reset normally clears the op-timeout
// sticky-error (verified 5/5 on a2a3), but the poison occasionally needs
// a drain-then-reset cycle, so retry up to kMaxResetAttempts.
// force_reset_device() drains (best-effort) and resets inside its own
// ACL/bound scope, and returns 0 only when its post-reset probe confirms
// the card is actually clean.
constexpr int kMaxResetAttempts = 3;
for (int attempt = 1; attempt <= kMaxResetAttempts; ++attempt) {
reset_rc = force_reset_device();
if (reset_rc == 0) {
if (attempt > 1) {
LOG_WARN(
"DeviceRunner finalize: device %d recovered on force-reset attempt %d/%d", device_id_, attempt,
kMaxResetAttempts
);
}
break;
}
LOG_ERROR(
"DeviceRunner finalize: force-reset attempt %d/%d of device %d did not confirm clean (rc=%d)", attempt,
kMaxResetAttempts, device_id_, reset_rc
);
}
if (reset_rc != 0) {
LOG_ERROR(
"DeviceRunner finalize: device %d still poisoned after %d force-reset attempts; leaving it marked "
"unusable so the layer above (st_worker poison-skip + dispatcher retry) recovers it.",
device_id_, kMaxResetAttempts
);
}
}

// Only the healthy path reaches here: a poisoned card returned from the
// fatal branch at the top of finalize(), which owns the force reset.
device_id_ = -1;
// Clear the poison flag only if the force reset actually recovered the card,
// so a still-poisoned card stays flagged: a reused DeviceRunner then fails
// admission instead of being treated as clean. On the normal (not
// unusable) path reset_rc stays 0 and the flag is already false.
if (reset_rc == 0) {
device_unusable_.store(false, std::memory_order_release);
}
return rc != 0 ? rc : reset_rc;
device_unusable_.store(false, std::memory_order_release);
return rc;
}

// `launch_aicpu_kernel` and `launch_aicore_kernel` live on `DeviceRunnerBase`.
Expand Down Expand Up @@ -1230,15 +1312,20 @@ int DeviceRunner::init_scope_stats(int num_threads, int device_id) {
return 0;
}

void DeviceRunner::finalize_collectors() {
auto unregister_cb = [](void *dev_ptr, int device_id) -> int {
void DeviceRunner::finalize_collectors(bool abandon_device_resources) {
auto healthy_unregister_cb = [](void *dev_ptr, int device_id) -> int {
HalHostUnregisterFn fn = get_halHostUnregister();
if (fn != nullptr) {
return fn(dev_ptr, device_id);
}
return 0;
};
auto free_cb = [this](void *dev_ptr) -> int {
auto no_op_unregister_cb = [](void *, int) -> int {
return 0;
};
auto unregister_cb = abandon_device_resources ? no_op_unregister_cb : healthy_unregister_cb;
auto free_cb = [this, abandon_device_resources](void *dev_ptr) -> int {
if (abandon_device_resources) return 0;
return mem_alloc_.free(dev_ptr);
};

Expand Down
7 changes: 4 additions & 3 deletions src/a2a3/platform/onboard/host/device_runner.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,13 +232,14 @@ class DeviceRunner : public DeviceRunnerBase {
// fresh AICore stream per run, handle kept when a destroy fails — is
// testable without a device.
RunStreamSlots run_stream_slots_{
[](void **out) {
return rtStreamCreate(reinterpret_cast<rtStream_t *>(out), 0);
[this](void **out) {
return create_run_stream(out);
},
[](void *stream) {
return rtStreamDestroy(static_cast<rtStream_t>(stream));
}
};
int create_run_stream(void **out);
int ensure_run_stream_set(unsigned slot);
// Destroys this run's AICore stream. Returns the driver's error and KEEPS
// the handle when the destroy fails: the stream may still hold the previous
Expand Down Expand Up @@ -359,7 +360,7 @@ class DeviceRunner : public DeviceRunnerBase {
* collectors in a pristine, re-initializable state) and from finalize()
* as a backstop before mem_alloc_.finalize().
*/
void finalize_collectors();
void finalize_collectors(bool abandon_device_resources = false);
// Shared enable flags (`enable_chip_swimlane_`, `enable_dump_args_`,
// `enable_pmu_`, `enable_scope_stats_`, `chip_swimlane_level_`,
// `pmu_event_type_`, `output_prefix_`) live on `DeviceRunnerBase`.
Expand Down
Loading
Loading