Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,6 @@ changing simpler's own internals.
| Document | What it covers |
| -------- | -------------- |
| [Communication Domains](comm-domain.md) | Dynamic `CommDomain` allocation and the symmetric window |
| [a5 SDMA Workspace Overlay](a5-sdma-overlay.md) | a5 SDMA overlay isolation status and the CANN-version gates |
| [L3-L2 Orchestrator Communication](l3-l2-orch-comm.md) | Host-side L3 talking directly to the L2 AICPU orchestrator |
| [L3-L2 Message Queue](l3-l2-message-queue.md) | The queue channel between an L3 host and L2 |
| [Directed NEXT_LEVEL Scheduling](directed-next-level-scheduling.md) | Targeting a specific next-level child instead of any free one |
Expand Down
126 changes: 0 additions & 126 deletions docs/a5-sdma-overlay.md

This file was deleted.

6 changes: 2 additions & 4 deletions docs/capability-survey.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ rather than restating them:
- Level model → [hierarchical-level-runtime.md](hierarchical-level-runtime.md)
- Three-program model → [chip-level-arch.md](chip-level-arch.md)
- AICPU launch mechanics → [aicpu-kernel-launch-mechanisms.md](aicpu-kernel-launch-mechanisms.md)
- Comm domains / overlays → [comm-domain.md](comm-domain.md), [a5-sdma-overlay.md](a5-sdma-overlay.md)
- Comm domains / overlays → [comm-domain.md](comm-domain.md)
- Remote L3 / L4 → [remote-l3-worker-design.md](remote-l3-worker-design.md)

Status claims rot faster than architecture. Re-derive rather than trust this
Expand Down Expand Up @@ -272,8 +272,7 @@ Unresolved after this survey, in rough order of how much they block:
investigation entry, or code comment says whether they are reserved slots or
leftovers from a dropped design.
2. **Has a5 URMA ever run on silicon?** No CI run, test artifact, or
investigation attests to it, and `a5-sdma-overlay.md` has no URMA re-enable
checklist analogous to its SDMA one.
investigation attests to it.
3. **Which CANN mitigation closed issue #822, and is Path B usable on CANN
9.0.0?** The doc says "CANN-side mitigation landed" without naming it, and
nobody re-ran the repro.
Expand All @@ -299,7 +298,6 @@ re-check before editing.
| `dynamic-linking.md:355-361` | AICPU launches before AICore; call is `rtKernelLaunch` | AICore launches first; the call is `rtKernelLaunchWithHandleV2` |
| `comm-domain.md:111` | window is VMM + shareable-handle import with `aclrtDeviceEnablePeerAccess` | a2a3 prefers Fabric V2 (`comm_hccl.cpp:762`); the doc never mentions Fabric |
| `comm-domain.md:262-264` | a producer `CoreCallable` declares the SDMA workspace requirement | that API was removed by PR #1406 |
| `a5-sdma-overlay.md:24` | the a2a3 SDMA path "is always on" | build macro is ON; runtime provisioning defaults to `False` |
| `investigations/2026-07-a2a3-sdma-fault-teardown.md:153` | `sdma_async_completion_demo` is "unaffected by this change" | that demo sets `enable_sdma=True` (test:134), as CI's own comment states |
| `src/common/worker/pto_runtime_c_api.h:259` | "`config` carries block_dim (0 = auto)" | `CallConfig` has no such field — "There is no block_dim knob" (`src/common/task_interface/call_config.h:22`) |
| `src/common/platform/sim/host/device_runner_base.h:62-64` | "an explicit block_dim is still honoured" | same as above |
Expand Down
8 changes: 4 additions & 4 deletions docs/comm-domain.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ symmetric window is realized:
| Window memory | POSIX shm + `ftruncate`, mmap'd per rank | a2a3: Fabric V2 handle exchange (`ACL_MEM_SHARE_HANDLE_TYPE_FABRIC`), falling back to VMM + shareable-handle IPC where Fabric is unsupported. a5: VMM shareable handles only. Cross-card P2P via `aclrtDeviceEnablePeerAccess` on both |
| Subset barrier | shm-header atomic, `allocation_id`-scoped | file barriers, `allocation_id`-scoped |
| Window init | window zeroed before the subset barrier (`memset`) | window zeroed before the handle is announced (`aclrtMemset`) |
| Async-DMA workspace | n/a | a2a3: opt-in per Worker (`enable_sdma`); a5: optional communication overlay, gated off by default |
| Async-DMA workspace | n/a | a2a3: opt-in per Worker (`enable_sdma`); a5: SDMA by default, URMA as an opt-in alternative |

The window is zero-initialized on both backends so scratch/signal protocols see
a known starting state (matching the historical static-path contract).
Expand Down Expand Up @@ -139,9 +139,9 @@ so ordinary workloads are unaffected — see
[docs/investigations/2026-07-a2a3-sdma-fault-teardown.md](investigations/2026-07-a2a3-sdma-fault-teardown.md)
and issue #1425. `enable_sdma` is currently honored only by the a2a3 onboard
`tensormap_and_ringbuffer` runtime; host-build-graph, simulation, a5, and
provider-disabled builds fail Worker init fast when it is set. The a5
communication overlay remains isolated behind its default-off gate; see
[a5-sdma-overlay.md](a5-sdma-overlay.md).
provider-disabled builds fail Worker init fast when it is set. A5 provisions
its communication-context SDMA workspace by default; this is separate from
the callable-declared workspace mechanism controlled by `enable_sdma`.

---

Expand Down
2 changes: 1 addition & 1 deletion examples/a5/tensormap_and_ringbuffer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ rather than on task end. All need two dies.

| Example | Mechanism |
| ------- | --------- |
| [`sdma_async_completion_demo/`](sdma_async_completion_demo/) | `TGET_ASYNC` from a peer's window slot over SDMA, completion registered via `defer_pto_async_event`. Needs `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON` at build **and** run time; skipped otherwise. |
| [`sdma_async_completion_demo/`](sdma_async_completion_demo/) | `TGET_ASYNC` from a peer's window slot over SDMA, completion registered via `defer_pto_async_event`. Enabled by default on a5 onboard. |
| [`urma_deferred_completion_demo/`](urma_deferred_completion_demo/) | The same protocol over **URMA** — `kernel_consumer.cpp` is byte-identical to the SDMA demo's, so the transport is the only variable. The two overlays are **mutually exclusive in one build**, so comparing them means rebuilding. |
| [`async_notify_demo/`](async_notify_demo/) | Notification counters alongside deferred completion. |
| [`deferred_notify_demo/`](deferred_notify_demo/) | The same shape on the simulator. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,35 +18,27 @@ The remote address is plain symmetric-window arithmetic — take the local
pointer's offset from `windowsIn[rankId]` and add it to `windowsIn[peer_rank]`.
Every rank's window is laid out identically, so an offset is rank-independent.

## Gated behind an overlay that is off by default
## Requirements

Unlike its a2a3 namesake, the a5 demo needs the PTO async-SDMA workspace
compiled into the host runtime:
The a5 host runtime includes the PTO async-SDMA workspace by default:

| Gate | Effect |
| ---- | ------ |
| `@pytest.mark.platforms(["a5"])` | deselected on any other `--platform` |
| `@pytest.mark.device_count(2)` | needs two dies |
| `@pytest.mark.skipif(...)` | skipped unless `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE` is `1` / `ON` / `TRUE` / `YES` |
| `@pytest.mark.skipif(_urma_workspace_enabled())` | skipped when `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` selects the URMA backend |

The CMake option defaults `OFF`, so a stock build skips this test even on a5
hardware — **a green CI run says nothing about SDMA completion on a5.**
URMA replaces SDMA in a URMA build. Rebuild without
`SIMPLER_ENABLE_PTO_URMA_WORKSPACE` before running this demo.

```bash
SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON pip install --no-build-isolation -e .
SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON \
pytest examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo \
pytest examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo \
--platform a5 --device 0-1
```
Comment thread
doraemonmj marked this conversation as resolved.

The variable is read twice: `simpler_setup/runtime_builder.py` forwards it to
CMake so the overlay is compiled in, and the test reads it from the environment
to decide whether to skip.

Wrap the hardware run in `task-submit` on a shared box.

## Compare with

- [`../urma_deferred_completion_demo/`](../urma_deferred_completion_demo/) — the same protocol over URMA. `kernel_consumer.cpp` is byte-identical; only the transfer kernel, its completion header, and the build flag differ. **The two overlays are mutually exclusive in one build**, so comparing them means rebuilding — that README has the detail.
- [`examples/a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/`](../../../a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/) — the a2a3 port of this demo, which needs no overlay flag.
- [`docs/a5-sdma-overlay.md`](../../../../docs/a5-sdma-overlay.md) — why the overlays are gated off and the checklist for re-enabling them.
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,7 @@ extern "C" __aicore__ __attribute__((always_inline)) void kernel_entry(__gm__ in

int rank = static_cast<int>(comm_ctx->rankId);
int nranks = static_cast<int>(comm_ctx->rankNum);
// workSpace == 0 means the SDMA overlay is not built in
// (SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=OFF, see docs/a5-sdma-overlay.md
// #1315): self-skip rather than dereferencing a null workspace.
// A null workspace indicates that host-side SDMA provisioning failed.
if (nranks != 2 || comm_ctx->workSpace == 0) {
pipe_barrier(PIPE_ALL);
return;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,21 @@
HERE = os.path.dirname(os.path.abspath(__file__))
N = 128 * 128
DTYPE_NBYTES = 4
_URMA_WORKSPACE_ENV = "SIMPLER_ENABLE_PTO_URMA_WORKSPACE"
_WORKSPACE_TRUTHY = {"1", "ON", "TRUE", "YES"}


def _urma_workspace_enabled() -> bool:
return os.environ.get(_URMA_WORKSPACE_ENV, "").upper() in _WORKSPACE_TRUTHY


def _require_sdma_workspace() -> None:
if not _urma_workspace_enabled():
return
raise RuntimeError(
"sdma_async_completion_demo requires the default SDMA backend; "
f"unset {_URMA_WORKSPACE_ENV} and rebuild simpler."
)


def parse_device_range(spec: str) -> list[int]:
Expand Down Expand Up @@ -103,6 +118,7 @@ def run(
platform: str = "a5",
device_ids: list[int] | None = None,
) -> int:
_require_sdma_workspace()
if device_ids is None:
device_ids = [0, 1]
nranks = len(device_ids)
Expand Down Expand Up @@ -192,9 +208,8 @@ def orch_fn(orch, _args, cfg):
@pytest.mark.runtime("tensormap_and_ringbuffer")
@pytest.mark.device_count(2)
@pytest.mark.skipif(
os.environ.get("SIMPLER_ENABLE_PTO_SDMA_WORKSPACE", "").upper() not in {"1", "ON", "TRUE", "YES"},
reason="SDMA workspace overlay not enabled (set SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON to run). "
"See docs/a5-sdma-overlay.md (#1315).",
_urma_workspace_enabled(),
reason="SDMA demo requires the default SDMA backend; unset SIMPLER_ENABLE_PTO_URMA_WORKSPACE and rebuild simpler.",
)
def test_sdma_async_completion_demo(st_device_ids, st_platform) -> None:
assert run(st_platform, [int(d) for d in st_device_ids]) == 0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,21 +24,21 @@ directories. Only the transfer kernel and the orchestration change:
| ---- | ---- | ---- |
| Transfer kernel | `kernel_urma_tget_async.cpp` | `kernel_sdma_tget_async.cpp` |
| Completion header | `backend/urma/urma_completion_kernel.h` | `backend/sdma/sdma_completion_kernel.h` |
| Build flag | `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` | `SIMPLER_ENABLE_PTO_SDMA_WORKSPACE` |
| Build selection | `SIMPLER_ENABLE_PTO_URMA_WORKSPACE=ON` | default |

Read them side by side and the transport is the only variable — which is
exactly what you want when deciding which one a workload should use.

## Both overlays are off by default, and they are mutually exclusive
## SDMA is the default; URMA is the opt-in alternative

This is the part to know before trying to run either.

```cmake
option(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE "..." OFF)
option(SIMPLER_ENABLE_PTO_URMA_WORKSPACE "..." OFF)
if(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE AND SIMPLER_ENABLE_PTO_URMA_WORKSPACE)
message(FATAL_ERROR "... mutually exclusive because CommContext exposes a
single workSpace/workSpaceSize pair")
if(SIMPLER_ENABLE_PTO_URMA_WORKSPACE)
set(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE OFF)
else()
set(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ON)
endif()
```

Expand All @@ -59,7 +59,7 @@ was not built in.
| `@pytest.mark.skipif(not _urma_workspace_enabled())` | skipped unless `SIMPLER_ENABLE_PTO_URMA_WORKSPACE` is one of `1` / `ON` / `TRUE` / `YES` in the environment |
| `run()` raises | if `platform != "a5"` or the device count is not 2, and re-checks the env var before doing any work |

Since the CMake option defaults `OFF`, a stock build skips this test even on a5
Since the URMA CMake option defaults `OFF`, a stock build skips this test even on a5
hardware. **A green CI run says nothing about URMA.** Treat it as a manual
bring-up check, not as coverage.

Expand All @@ -84,6 +84,5 @@ Wrap the hardware run in `task-submit` on a shared box.

## See also

[`docs/a5-sdma-overlay.md`](../../../../docs/a5-sdma-overlay.md) — why the
overlays are gated off, what is gated, and the checklist for re-enabling them
(issue [#1315](https://github.com/hw-native-sys/simpler/issues/1315)).
[`../sdma_async_completion_demo/`](../sdma_async_completion_demo/) — the
default SDMA variant of the same protocol.
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,8 @@ extern "C" __aicore__ __attribute__((always_inline)) void kernel_entry(__gm__ in
__gm__ Tensor *out_tensor = reinterpret_cast<__gm__ Tensor *>(args[1]);
__gm__ CommContext *comm_ctx = reinterpret_cast<__gm__ CommContext *>(args[2]);

// workSpace == 0 means the URMA overlay is not built in
// (SIMPLER_ENABLE_PTO_URMA_WORKSPACE=OFF, see docs/a5-sdma-overlay.md
// #1315): self-skip rather than dereferencing a null workspace.
// A null workspace means the host runtime was not built with the URMA
// backend; self-skip rather than dereferencing it.
if (comm_ctx == nullptr || comm_ctx->rankNum != 2 || comm_ctx->rankId >= comm_ctx->rankNum ||
comm_ctx->workSpace == 0 || comm_ctx->windowsIn[comm_ctx->rankId] == 0) {
pipe_barrier(PIPE_ALL);
Expand Down
Loading
Loading