diff --git a/docs/en/user/00-getting_started.md b/docs/en/user/00-getting_started.md index a7687eb450..3e9d45feaf 100644 --- a/docs/en/user/00-getting_started.md +++ b/docs/en/user/00-getting_started.md @@ -145,6 +145,10 @@ and read `l2_swimlane_records.json`. ### Benchmarking (`benchmark`) +The full benchmarking guide is at [performance/00-methodology.md](performance/00-methodology.md) +— it covers the programmatic `benchmark()` API, L3 distributed timing, +bus-bandwidth formulas, and common caveats. + For the register-once + rounds pattern, `pypto.runtime.benchmark` owns the loop and aggregation: it registers *compiled* once and dispatches `rounds` cheap launches (no per-round register/load), reads each launch's `[STRACE]` markers, @@ -229,6 +233,61 @@ per-rank / `union` views are empty. ### Distributed (L3+) programs +The complete distributed programming model — from `alloc_window_buffer` through +collectives like `allreduce`, `barrier`, and `broadcast` — is covered in the +[Distributed Programming](distributed/00-model.md). Here's the minimal mesh allreduce +Hello World: + +```python +import pypto.language as pl +import pypto.language.distributed as pld + +NR = pl.dynamic("NR") + +@pl.program +class HelloAllReduce: + @pl.function(type=pl.FunctionType.InCore) + def reduce_step( + self, + inp: pl.Tensor[[1, 256], pl.FP32], + out: pl.Out[pl.Tensor[[1, 256], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, 256], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, 256], pl.FP32]: + ctx = pld.get_comm_ctx(data) + my_rank = pld.rank(ctx) + nranks = pld.nranks(ctx) + + # 1. Stage-in: copy local input into this rank's window slice. + data = pl.store(pl.load(inp, [0, 0], [1, 256]), [0, 0], data) + + # 2. Barrier: notify every peer, then wait on every peer. + for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify(signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd) + for src in pl.range(nranks): + if src != my_rank: + pld.system.wait(signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge) + + # 3. Compute: load own slice, remote-load every peer, accumulate. + acc = pl.load(data, [0, 0], [1, 256]) + for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, 256]) + acc = pl.add(acc, peer_tile) + + # 4. Stage-out: store accumulator to output. + out = pl.store(acc, [0, 0], out) + return out +``` + +The guide includes a **line-by-line walkthrough, ring allreduce trade-offs, +notify/wait handshake patterns, and a debugging table**. The full chapter is +at [distributed/index.md](distributed/index.md). + L3+ distributed programs returned by `ir.compile` (a `DistributedCompiledProgram`) accept `DeviceTensor` arguments the same way as `CompiledProgram`: pass a worker-resident buffer in place of a `torch.Tensor` and the runtime skips H2D/D2H diff --git a/docs/en/user/02-operation_reference.md b/docs/en/user/02-operation_reference.md index 86a7ddcf8f..37231d0d01 100644 --- a/docs/en/user/02-operation_reference.md +++ b/docs/en/user/02-operation_reference.md @@ -277,3 +277,26 @@ scratch tile to materialize numeric results on A2/A3. | `create_tensor` | `(shape: Sequence[IntLike], dtype: DataType, layout: TensorLayout = None, init_value: int \| float \| None = None) -> Tensor` | Create tensor (promoted from `pl.tensor`; `init_value` AICPU-pre-fills the buffer — `0` zeroes any dtype, non-zero needs an int / 32-bit-or-wider float dtype) | | `max` | `(lhs: Scalar \| int \| Expr, rhs: Scalar \| int \| Expr) -> Scalar` | Max of two **scalars** (not a tile reduction — use `pl.tile.row_max` / `pl.tile.col_max`) | | `min` | `(lhs: Scalar \| int \| Expr, rhs: Scalar \| int \| Expr) -> Scalar` | Min of two **scalars** (not a tile reduction — use `pl.tile.row_min` / `pl.tile.col_min`) | + +## Distributed / Collective Operations (`pld.*`) + +Distributed and collective ops are in the `pld` namespace (`import pypto.language.distributed as pld`). +Full reference at [distributed/index.md](distributed/index.md); +tutorial at [distributed/00-model.md](distributed/00-model.md). + +### Capability Matrix + +| Operation | API | Modes | ReduceOp | Atomic | Supported dtypes | Notes | +| --------- | --- | ----- | -------- | ------ | ---------------- | ----- | +| AllReduce | `pld.tensor.allreduce` | `mesh` (InCore + HOST), `ring` (InCore only) | `Sum`, `Max`, `Min`, `Prod` | — | FP16, FP32 — a hard compile-time check (`allreduce.cpp`), both InCore and HOST | Mesh: O(N) remote traffic. Ring: O(N/P) traffic, 2(P-1) steps; currently InCore-only. | +| AllGather | `pld.tensor.allgather` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Push-based. Input and target must be different buffers. | +| ReduceScatter | `pld.tensor.reduce_scatter` | — | `Sum` only | — | FP32 only (HOST builtin); any GM dtype (InCore) | Every rank stages all NR chunks before the call. | +| Broadcast | `pld.tensor.broadcast` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Root stages data before the call. | +| All-to-All | `pld.tensor.all_to_all` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Personalized exchange. Input and target must be different buffers. | +| Barrier | `pld.tensor.barrier` | — | — | — | — | Signal is INT32, single-shot per call. | +| Put | `pld.tensor.put` | — | — | `None_` / `Add` | All GM dtypes | `dst` must be window-bound. Supports chunked + pipelined staging. | +| Get | `pld.tensor.get` | — | — | — | All GM dtypes | `src` must be window-bound. Supports chunked + pipelined staging. | +| Notify | `pld.system.notify` | `AtomicAdd` / `Set` | — | — | — | Side-effect-only signal deposit. | +| Wait | `pld.system.wait` | `Eq` / `Ge` | — | — | — | Side-effect-only signal block. | +| Remote Load | `pld.tile.remote_load` | — | — | — | Any (tile) | Tile-level cross-rank load. | +| Remote Store | `pld.tile.remote_store` | — | — | — | Any (tile) | Tile-level cross-rank store. | diff --git a/docs/en/user/distributed/00-model.md b/docs/en/user/distributed/00-model.md new file mode 100644 index 0000000000..25b39c2a15 --- /dev/null +++ b/docs/en/user/distributed/00-model.md @@ -0,0 +1,193 @@ +# Distributed Programming Model + +> **Prerequisites:** [Getting Started](../00-getting_started.md) — basic PyPTO +> tensor/tile model. This guide uses the `pld` namespace +> (`import pypto.language.distributed as pld`). + +## Quickstart: 2-Rank AllReduce + +The simplest distributed program — two ranks sum their data, both see the same result. + +```python +import pypto.language as pl +import pypto.language.distributed as pld + +NR = pl.dynamic("NR") +SIZE = 256 + +@pl.program +class HelloAllReduce: + @pl.function(type=pl.FunctionType.InCore) + def reduce_step( + self, + inp: pl.Tensor[[1, SIZE], pl.FP32], + out: pl.Out[pl.Tensor[[1, SIZE], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, SIZE], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, SIZE], pl.FP32]: + ctx = pld.get_comm_ctx(data) + my_rank = pld.rank(ctx) + nranks = pld.nranks(ctx) + + # 1. Stage-in: copy local input into this rank's window slice. + local = pl.load(inp, [0, 0], [1, SIZE]) + data = pl.store(local, [0, 0], data) + + # 2. Barrier: notify every peer, then wait on every peer. + for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify( + signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd, + ) + for src in pl.range(nranks): + if src != my_rank: + pld.system.wait( + signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge, + ) + + # 3. Compute: accumulate every peer's slice. + acc = pl.load(data, [0, 0], [1, SIZE]) + for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, SIZE] + ) + acc = pl.add(acc, peer_tile) + + # 4. Stage-out: store the accumulator to local output. + return pl.store(acc, [0, 0], out) + + @pl.function(type=pl.FunctionType.Orchestration) + def chip_orch( + self, + inp: pl.Tensor[[1, SIZE], pl.FP32], + out: pl.Out[pl.Tensor[[1, SIZE], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, SIZE], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, SIZE], pl.FP32]: + # Per-device orchestration wrapper — HOST dispatches this, not the + # InCore kernel directly. + return self.reduce_step(inp, out, data, signal) + + @pl.function(level=pl.Level.HOST, role=pl.Role.Orchestrator) + def orchestrator( + self, + inputs: pl.Tensor[[NR, 1, SIZE], pl.FP32], + outputs: pl.Out[pl.Tensor[[NR, 1, SIZE], pl.FP32]], + ) -> pl.Tensor[[NR, 1, SIZE], pl.FP32]: + data_buf = pld.alloc_window_buffer(SIZE * pl.FP32.get_byte()) + signal_buf = pld.alloc_window_buffer(pld.world_size() * pl.INT32.get_byte()) + + for r in pl.range(pld.world_size()): + data = pld.window(data_buf, [1, SIZE], dtype=pl.FP32) + signal = pld.window(signal_buf, [pld.world_size(), 1], dtype=pl.INT32) + self.chip_orch(inputs[r], outputs[r], data, signal, device=r) + return outputs +``` + +### Expected Output + +`outputs[r] == sum(inputs[*])` for every rank `r`. For 2 ranks with inputs +`[[1, 2, 3]]` and `[[10, 20, 30]]`, both ranks see `[[11, 22, 33]]`. + +### Launch Command + +```bash +python script.py +``` + +No multi-process launcher is involved. `DistributedConfig(device_ids=[...])` +tells the compiled program which devices to use; the runtime forks one worker +process per device from this single Python process. + +## What Is Distributed Programming in PyPTO? + +PyPTO's distributed model is **symmetric-memory + signals**. Each rank has a +per-rank **window buffer** with symmetric address spaces across peers. +Communication happens through one-sided `put`/`get`/`remote_load` plus +**signal synchronisation** (`notify`/`wait`). A **comm domain** is a subset +of ranks sharing a symmetric window pool; the full world is the default +domain. + +Every allreduce, broadcast, and barrier the compiler lowers is a composition +of these same primitives — the `pld.tensor.*` collectives (`allreduce`, +`barrier`, etc.) are syntactic sugar over them, not a separate library. The +full API is at [01-collectives](01-collectives.md). + +## The Model + +### HOST Orchestrator + +The HOST function allocates window buffers, dispatches kernels, and manages +the control plane: + +- Declared with `@pl.function(level=pl.Level.HOST, role=pl.Role.Orchestrator)` + or `@pl.jit.host` +- Calls `alloc_window_buffer`, `window()`, and per-rank dispatch via `device=r` +- Runs once per process — not on the NPU +- Dispatches a per-device `@pl.function(type=pl.FunctionType.Orchestration)` + wrapper (not the `InCore` kernel directly) — see Per-Rank Dispatch below + +### InCore Kernel + +The InCore function runs on the NPU device: + +- Declared with `@pl.function(type=pl.FunctionType.InCore)` +- Receives window-bound `DistributedTensor` arguments +- Uses `notify`/`wait` for cross-rank sync, `remote_load`/`remote_store` for RMA +- Never calls `alloc_window_buffer` or `world_size()` + +### Per-Rank Dispatch + +HOST never dispatches an `InCore` function directly. It dispatches a per-device +`@pl.function(type=pl.FunctionType.Orchestration)` wrapper by setting `device=r` +in the function call; that wrapper then calls the `InCore` kernel with no +`device=` argument. Each rank sees its own view of the symmetric window +buffers through `CommContext`. + +### Window Buffer Lifetime + +`alloc_window_buffer(size)` creates per-rank buffers. `window(buf, shape, dtype)` +creates typed views. Buffers live for the duration of the host orchestrator call; +there is no persistent IPC between orchestrator invocations. + +### Control Plane vs Execution Plane + +```text +HOST orchestrator (@pl.function(level=HOST, role=Orchestrator)) + ├── alloc_window_buffer(...) ← control plane: declare layout + ├── window(buf, shape, dtype) ← control plane: create typed view + └── for r in ranks: ← dispatch loop + self.chip_orch(..., device=r) ← bridges to the per-device wrapper + +Orchestration wrapper (@pl.function(type=Orchestration)) + └── self.reduce_step(...) ← calls the InCore kernel, no device= + +InCore kernel (@pl.function(type=InCore)) + ├── notify / wait ← execution plane: cross-rank sync + ├── remote_load ← execution plane: read peer data + └── store ← execution plane: write local output +``` + +## Line-by-Line Walkthrough + +| What | Why | +| ---- | --- | +| `NR = pl.dynamic("NR")` | The world size is not known at build time. `pl.dynamic` defers the dimension to runtime dispatch — the host binds it from `len(device_ids)`. | +| `pl.InOut[pld.DistributedTensor[...]]` | `data` and `signal` are window-bound: every rank shares the same address space layout. `InOut` means the kernel both reads and writes them. | +| `pld.get_comm_ctx(data)` | Lifts the window-bound tensor into a comm-domain handle. Every rank gets its own `ctx`, from which `rank()` and `nranks()` read per-rank values. | +| `pld.system.notify(..., op=AtomicAdd)` | Each rank atomically adds 1 to every peer's signal slot. `AtomicAdd` is correct here because N ranks write the same slot (it's a global barrier). For a 1:1 handshake, use `Set` instead. | +| `pld.system.wait(..., cmp=Ge, expected=1)` | Blocks until the local signal slot reaches at least 1 — meaning all peer notifies have landed. | +| `pld.tile.remote_load(...)` | Reads a **remote** slice of a `DistributedTensor` into a local tile. This is the tile-level cross-rank equivalent of `pl.tile.load`. | +| `pl.add(acc, peer_tile)` | The local add loop sums all peer contributions. After the loop, `acc` holds `sum(inputs[*])`. | +| `chip_orch` (`@pl.function(type=Orchestration)`) | HOST dispatches this per-device wrapper via `device=r`, not the `InCore` kernel directly. It then calls `reduce_step` with no `device=` argument. | +| `inputs[r]` / `outputs[r]` | Indexing drops the leading rank dimension, giving `reduce_step` the rank-2 `[1, SIZE]` shape it declares — `pl.slice` with an explicit shape would keep the dimension and produce a rank mismatch. | + +## See Also + +- [01-collectives](01-collectives.md) — Built-in collectives and their semantics +- [02-primitives](02-primitives.md) — The substrate beneath the collectives +- [03-execution](03-execution.md) — DistributedWorker lifecycle and production patterns +- [04-debugging](04-debugging.md) — Common failure patterns and diagnostic flags diff --git a/docs/en/user/distributed/01-collectives.md b/docs/en/user/distributed/01-collectives.md new file mode 100644 index 0000000000..c688262038 --- /dev/null +++ b/docs/en/user/distributed/01-collectives.md @@ -0,0 +1,178 @@ +# Collectives + +This page covers the five built-in collectives and when to use each algorithm. +All collectives are **synchronous** across ranks — every rank must call the same +collective with identically shaped signal tensors, or the program hangs or +silently corrupts data. + +## AllReduce + +Every rank contributes its local data; every rank receives the summed +result. + +```python +# Host orchestrator — simplest form (compiler synthesizes signal). +data = pld.tensor.allreduce(data, op=pld.ReduceOp.Sum) # mesh mode, in-place + +# InCore kernel — explicit signal. +data = pld.tensor.allreduce(data, signal, op=pld.ReduceOp.Sum, mode="mesh") +data = pld.tensor.allreduce(data, signal, op=pld.ReduceOp.Sum, mode="ring") +``` + +### Mesh Mode + +- O(N) remote traffic per step — every rank reads every peer +- One global barrier per call (AtomicAdd/Ge on `[NR, 1]` signal) +- Works with `pl.dynamic("NR")` +- Best for small messages and low latency + +### Ring Mode + +- 2(P-1) steps: reduce-scatter + allgather +- O(N/P) remote traffic per step — each rank reads one neighbour +- Signal shape: `[2 × (NR − 1), NR]` +- Requires compile-time-known NR — use a factory function pattern +- Best for large messages (>16 KiB) and high bandwidth + +| Aspect | Mesh | Ring | +| ------ | ---- | ---- | +| Remote traffic per step | O(N) — every rank reads every peer | O(N/P) — each rank reads one neighbour | +| Barrier rounds | 1 (global AtomicAdd/Ge) | 2(P-1) — reduce-scatter + allgather phases | +| Signal shape | `[NR, 1]` | `[2 × (NR − 1), NR]` | +| Best for | Small messages, low latency | Large messages, high bandwidth | + +**Rule of thumb:** Use the default `mode="mesh"`. Switch to `mode="ring"` when +your payload exceeds ~16 KiB and you see mesh bandwidth plateau. + +The host orchestrator form (`signal` omitted) is syntactic sugar — the compiler +synthesizes a signal of `[world_size(), 1]` (mesh only). + +### Mutation + +`target: InOut` — data is both read (as the reduction input) and written (as +the reduced result). All ranks must pass identically shaped `target` tensors. + +### Supported ReduceOp + +All four — `Sum`, `Max`, `Min`, `Prod` — on both the InCore composite and the +HOST builtin path. `target`'s dtype must be `FP16` or `FP32`; this is a hard +compile-time check, not just a storage-width constraint. Every rank must +agree on the same `ReduceOp` and `mode`, on top of the identically-shaped +signal tensors required of every collective. + +## Barrier + +Cross-rank barrier — blocks until all ranks arrive. + +```python +# signal: pld.DistributedTensor[[NR, 1], pl.INT32], freshly allocated. +signal = pld.tensor.barrier(signal) +``` + +Uses `Set(1)` + `Ge(1)` on the signal. Single-shot; allocate a fresh buffer +before the next barrier. + +## Broadcast + +Broadcast root rank's data to all ranks. + +```python +# Root stages data before the call. +if my_rank == ROOT_RANK: + data = pl.store(local, [0, 0], data) +data = pld.tensor.broadcast(data, signal, root=ROOT_RANK) +# Every rank now holds root's data in data[0, 0:SIZE]. +``` + +Root must stage data before the call; non-root slots are ignored on input. +After the call, every rank holds root's data. + +## AllGather + +Push-based all-gather — every rank pushes its local chunk, every rank receives +the full gathered matrix. + +```python +# Stage buffer: this rank's [1, SIZE] chunk (push source). +stage_buf = pld.alloc_window_buffer(SIZE * pl.FP32.get_byte()) +stage = pld.window(stage_buf, [1, SIZE], dtype=pl.FP32) +stage = pl.store(local_input, [0, 0], stage) + +# Result buffer: gathered [NR, SIZE] (push target). +data_buf = pld.alloc_window_buffer(NR * SIZE * pl.FP32.get_byte()) +data = pld.window(data_buf, [NR, SIZE], dtype=pl.FP32) +sig_buf = pld.alloc_window_buffer(NR * pl.INT32.get_byte()) +sig = pld.window(sig_buf, [NR], dtype=pl.INT32) + +data = pld.tensor.allgather(stage, data, sig) +# data[src, :] now holds rank src's chunk for every src. +``` + +`local_data` and `target` **must be different** window buffers. The stage buffer +is the per-rank push source; the target buffer receives the gathered `[NR, SIZE]` +result. + +## ReduceScatter + +Reduce-scatter: every rank stages all NR chunks, receives its own reduced chunk. + +```python +# Signal for the barrier (1-D for host builtins). +sig_buf = pld.alloc_window_buffer(NR * pl.INT32.get_byte()) +sig = pld.window(sig_buf, [NR], dtype=pl.INT32) + +# Stage all NR chunks into data[NR, SIZE]. +for j in pl.range(nranks): + data = pl.store(chunk_j, [j, 0], data) +data = pld.tensor.reduce_scatter(data, sig, op=pld.ReduceOp.Sum) +# data[my_rank, 0:SIZE] holds this rank's reduced chunk. +``` + +## AllToAll + +Personalized all-to-all exchange — every rank sends a distinct chunk to every +peer and receives a distinct chunk from every peer. + +```python +# Stage buffer: push source, [NR, SIZE] with per-destination chunks. +stage_buf = pld.alloc_window_buffer(NR * SIZE * pl.FP32.get_byte()) +stage = pld.window(stage_buf, [NR, SIZE], dtype=pl.FP32) +for dest in pl.range(nranks): + stage = pl.store(chunk_for_dest, [dest, 0], stage) + +# Result buffer: push target, [NR, SIZE]. +data_buf = pld.alloc_window_buffer(NR * SIZE * pl.FP32.get_byte()) +data = pld.window(data_buf, [NR, SIZE], dtype=pl.FP32) +sig_buf = pld.alloc_window_buffer(NR * pl.INT32.get_byte()) +sig = pld.window(sig_buf, [NR], dtype=pl.INT32) + +data = pld.tensor.all_to_all(stage, data, sig) +# data[src, :] holds the chunk received from rank src. +``` + +`input` and `target` must be **separate** window buffers. + +## InCore vs Host-Level Collectives + +PyPTO has three ways to run a collective — pick based on where your code +runs and whether you need `mode="ring"`: + +| Aspect | InCore Hand-Rolled | InCore Composite | HOST Builtin | +| ------ | ------------------ | ---------------- | ------------ | +| **Where** | `@pl.function(type=InCore)` | `@pl.function(type=InCore)` | `@pl.function(level=HOST, role=Orchestrator)` | +| **How** | Manual `notify`/`wait` + `remote_load` loops | `pld.tensor.allreduce(data, sig, ...)` called directly | `pld.tensor.allreduce(data, [sig,] ...)` called directly | +| **Lowering** | You write the primitives | `LowerCompositeOps` | `LowerHostTensorCollectives` | +| **Modes** | Whatever you implement | `mesh` and `ring` | `mesh` only | +| **Signal shape** | Whatever you allocate | `[nranks, 1]` for mesh (rank count may be dynamic); `[2×(NR−1), NR]` for ring (`NR` must be a compile-time constant) | Rank-1 `[world_size]` or rank-2 `[world_size, 1]` — the compiler-synthesized signal is rank-2 | +| **When** | Learning, custom protocols | Need `ring` mode, or already inside an InCore kernel | Day-to-day host-orchestrated collectives | + +Prefer HOST builtins for day-to-day host-orchestrated code — they handle +signal allocation, barrier orchestration, and chunking automatically. Reach +for the InCore composite specifically when you need `mode="ring"`, since the +HOST builtin path only lowers `mesh`. + +## See Also + +- [00-model](00-model.md) — Quickstart and model vocabulary +- [02-primitives](02-primitives.md) — The substrate beneath the collectives +- [04-debugging](04-debugging.md) — Common failure patterns diff --git a/docs/en/user/distributed/02-primitives.md b/docs/en/user/distributed/02-primitives.md new file mode 100644 index 0000000000..6596f0a188 --- /dev/null +++ b/docs/en/user/distributed/02-primitives.md @@ -0,0 +1,205 @@ +# Primitives + +Most users call `pld.tensor.*` collectives directly — reach for these +lower-level primitives only when building a custom protocol. + +## Types and Enums + +| Name | Values | Description | +| ---- | ------ | ----------- | +| `NotifyOp` | `AtomicAdd`, `Set` | Signal deposit mode. `AtomicAdd`: atomically increment the peer's signal slot (use for multi-rank barriers). `Set`: overwrite the peer's signal slot (use for 1:1 handshakes). | +| `WaitCmp` | `Eq`, `Ge` | Wait predicate. `Eq`: block until signal slot equals expected value. `Ge`: block until signal slot >= expected value. | +| `ReduceOp` | `Sum`, `Max`, `Min`, `Prod` | Reduction operator for collective operations. Support is per-operation: `allreduce` accepts all four; `reduce_scatter` accepts only `Sum` and rejects the rest at the deducer. | +| `AtomicType` | `None_`, `Add` | Remote-store combine mode. `None_`: plain store. `Add`: atomically accumulate into peer's destination. | +| `DistributedTensor` | — | A tensor view bound to a comm-domain window buffer. Every collective and RMA op requires this type on the window side. | +| `CommCtx` | — | Communication context handle. Produced by `get_comm_ctx()`; consumed by `rank()` and `nranks()`. | + +## System Substrate (`pld.system.*`) + +These are the lowest-level distributed primitives. Host-only queries are valid only outside +InCore scopes; the remaining ops work in both host orchestrator and InCore kernel code. + +| Name | Signature | Description | +| ---- | --------- | ----------- | +| `world_size` | `() -> Scalar` | **Host-only.** Number of ranks in the distributed execution. Returns `INT64`. | +| `get_comm_ctx` | `(dist_tensor: DT) -> Ctx` | Lift a `DistributedTensor` to its `CommCtx` handle. The verifier rejects plain `pl.Tensor`. | +| `rank` | `(ctx: Ctx) -> Scalar` | Local rank index (`INT32`). Lowers to a load of `CommContext::rankId`. | +| `nranks` | `(ctx: Ctx) -> Scalar` | Number of ranks in this comm group (`INT32`). Lowers to a load of `CommContext::rankNum`. | +| `notify` | `(target: DT, peer: IntLike, offsets: Sequence[IntLike], value: IntLike, *, op: NotifyOp) -> Call` | Cross-rank signal deposit. **Side-effect-only** — no return value. Lowers to `TNOTIFY`. | +| `wait` | `(signal: DT, offsets: Sequence[IntLike], expected: IntLike, *, cmp: WaitCmp) -> Call` | Cross-rank wait. **Side-effect-only** — blocks until the local signal slot satisfies `cmp(expected)`. Lowers to `TWAIT`. | + +## Window Buffer Management (`pld.tensor.*`) + +`window` and `alloc_window_buffer` live in `pld.tensor.*`, not `pld.system.*`, +even though they are as foundational as the substrate above. + +| Name | Signature | Description | +| ---- | --------- | ----------- | +| `window` | `(buf: Ptr, shape: Sequence[IntLike], *, dtype: DataType) -> DT` | Materialise a window-buffer `Ptr` as a `DistributedTensor` view. `buf` comes from `alloc_window_buffer`. | +| `alloc_window_buffer` | `(size: IntLike, *, name: str = "") -> Ptr` | Allocate a per-rank HCCL window buffer. **Size is in bytes.** The `name` kwarg is injected by the parser from the LHS assignment — never pass it explicitly. | +| `alloc_window_buffer` | `(shape: Sequence[IntLike], *, dtype: DataType, name: str = "") -> Ptr` | Convenience overload. `size = prod(shape) x dtype.get_byte()` computed automatically. | + +## Notify & Wait: The Signal Handshake + +The lowest-level synchronisation primitive. Each rank writes to a peer's +signal cell, then blocks until its own cell has been written. + +```python +@pl.program +class SignalHandshake: + @pl.function(type=pl.FunctionType.InCore) + def handshake_step( + self, + out: pl.Out[pl.Tensor[[1, 1], pl.INT32]], + signal: pl.InOut[pld.DistributedTensor[[1, 1], pl.INT32]], + peer: pl.Scalar[pl.INT32], + tag: pl.Scalar[pl.INT32], + ) -> pl.Tensor[[1, 1], pl.INT32]: + # 1. Write our tag into the peer's signal cell. + pld.system.notify( + signal, peer=peer, offsets=[0, 0], + value=tag, op=pld.NotifyOp.Set, + ) + + # 2. Wait until our own cell has been written. + pld.system.wait( + signal=signal, offsets=[0, 0], + expected=1, cmp=pld.WaitCmp.Ge, + ) + + # 3. Read the received tag back out. + received = pl.load(signal, [0, 0], [1, 1]) + out = pl.store(received, [0, 0], out) + return out +``` + +> The `wait` uses `Ge` with `expected=1`, which means the peer's `tag` +> **must be >= 1**. Passing `tag=0` will cause a permanent hang. + +### Choosing NotifyOp and WaitCmp + +| Scenario | NotifyOp | WaitCmp | Why | +| -------- | -------- | ------- | --- | +| 1:1 exchange (one writer per slot) | `Set` | `Eq` or `Ge` | Atomic increment not needed — overwrite is clear and fast. | +| N-to-1 barrier (many writers, one slot) | `AtomicAdd` | `Ge` | Every writer atomically adds its contribution. The sum increments monotonically; wait for the expected total. | +| Multi-round protocol | `AtomicAdd` | `Ge` | The counter advances across rounds without reset — each round uses a fresh row or the caller re-allocates the buffer. | + +**Expected output** for 2 ranks: rank 0 writes tag=2, waits for tag 1 from rank 1: +`outputs[0] == 1`. Rank 1 writes tag=1, waits for tag 2 from rank 0: +`outputs[1] == 2`. Result: `outputs == [[1], [2]]`. + +> **Buffer re-use safety:** Signal cells are zero-initialised by +> `alloc_window_buffer`. After `notify`, the signal cell holds the written +> value; after `wait` returns, the caller has observed the barrier. Do not +> reuse the same signal buffer across back-to-back collectives — the protocol +> uses monotonic counters that do not self-reset. Allocate a fresh buffer. + +## Tile-Level RMA (`pld.tile.*`) + +Low-level cross-rank remote memory access. These are tile-level primitives used to build +collectives; most users call `pld.tensor.*` collectives instead. + +| Name | Signature | Description | +| ---- | --------- | ----------- | +| `remote_load` | `(target: DT, peer: IntLike, offsets: Sequence[IntLike], shape: Sequence[IntLike], valid_shape=None) -> Tile` | Load a region of peer rank's `DT` into a local tile. `shape` defines the tile dimensions. `valid_shape` keeps the physical tile fixed-size while a ragged tail reads only real data. Offsets must match what the peer stored — a 1-element misalignment causes silent corruption. | +| `remote_store` | `(src_tile: Tile, target: DT, peer: IntLike, offsets: Sequence[IntLike]) -> Call` | Write a local tile into peer rank's `DT`. Side-effect-only. | + +## Put and Get + +One-sided bulk transfer — rank A writes to or reads from rank B's window +without rank B participating in the transfer (beyond the signal barrier). + +### Put (Write to Peer) + +| Name | Signature | Mutation | Description | +| ---- | --------- | -------- | ----------- | +| `put` | `(dst: DT, peer: IntLike, src: DT \| Tensor, dst_offsets=None, src_offsets=None, shape=None, *, atomic=AtomicType.None_, chunk_rows=0, chunk_cols=0, pipeline=False) -> Call` | `dst: InOut`, `src: In` | Write local `src` into peer rank's `dst`. `dst` **must** be window-bound; `src` may be plain `Tensor`. With no offsets/shape, writes the full local slice. `atomic=Add` accumulates instead of overwriting. | + +### Get (Read from Peer) + +| Name | Signature | Mutation | Description | +| ---- | --------- | -------- | ----------- | +| `get` | `(dst: DT \| Tensor, peer: IntLike, src: DT, dst_offsets=None, src_offsets=None, shape=None, *, chunk_rows=0, chunk_cols=0, pipeline=False) -> Call` | `dst: Out`, `src: In` | Read peer rank's `src` into local `dst`. `src` **must** be window-bound; `dst` may be plain `Tensor`. | + +### Chunking and Pipelining Constraints + +`chunk_rows`/`chunk_cols` (`0` = full extent) shrink the staging tile so a +transfer larger than the on-chip staging budget still moves in one call, +sliding through the smaller stage automatically. + +> **Fatal pitfall:** `pipeline=True` **requires both `chunk_rows > 0` and +> `chunk_cols > 0`** — the double-buffering benefit only exists when the +> transfer is actually chunked. Passing `pipeline=True` with either chunk +> dimension left at `0` raises a `ValueError` before dispatch. + +A **dynamic** transfer extent (a runtime-sized `shape`, or a full-slice +transfer where `dst`/`src`'s own dims are dynamic) must be bounded by a +matching static chunk: a dynamic innermost dimension requires `chunk_cols` +to be set, and a dynamic leading dimension requires `chunk_rows` to be set — +the staging tile is allocated statically and can't size itself from a +runtime value. + +## Writing Your Own Collective + +Every built-in collective is a composition of lower-level primitives. The +mesh allreduce is: stage-in -> barrier -> remote-accumulate -> stage-out. + +### The Barrier in Isolation + +```python +# signal: pld.DistributedTensor[[NR, 1], pl.INT32] +for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify( + signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd, + ) +for src in pl.range(nranks): + if src != my_rank: + pld.system.wait( + signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge, + ) +``` + +`AtomicAdd` is used because N ranks write the same signal cell — each adds +1, so after all notifies land the cell reaches N-1 and every `WaitGe(1)` +unblocks. With `Set` the last writer would overwrite earlier contributions. + +### Remote Accumulate + +```python +acc = pl.load(data, [0, 0], [1, SIZE]) +for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, SIZE] + ) + acc = pl.add(acc, peer_tile) +``` + +`remote_load` reads the peer's window slice into a local tile. The offset +and shape must match what the peer stored — a mismatch reads garbage. + +## 2-Segment vs 3-Segment Namespace + +| Short form (`pld.*`) | Full path | +| -------------------- | --------- | +| `pld.world_size()` | `pld.system.world_size()` | +| `pld.rank(ctx)` | `pld.system.rank(ctx)` | +| `pld.nranks(ctx)` | `pld.system.nranks(ctx)` | +| `pld.get_comm_ctx(dt)` | `pld.system.get_comm_ctx(dt)` | +| `pld.alloc_window_buffer(...)` | `pld.tensor.alloc_window_buffer(...)` | +| `pld.window(...)` | `pld.tensor.window(...)` | +| `pld.remote_load(...)` | `pld.tile.remote_load(...)` | +| `pld.remote_store(...)` | `pld.tile.remote_store(...)` | + +**No short form:** `pld.notify(...)`, `pld.wait(...)`, `pld.put(...)`, +`pld.get(...)`, `pld.allreduce(...)`, and all other collective ops — these +require the full 3-segment namespace. + +## See Also + +- [01-collectives](01-collectives.md) — The collectives built on these primitives +- [03-execution](03-execution.md) — DistributedWorker lifecycle and environment setup +- [04-debugging](04-debugging.md) — Common failure patterns diff --git a/docs/en/user/distributed/03-execution.md b/docs/en/user/distributed/03-execution.md new file mode 100644 index 0000000000..77982cc8a9 --- /dev/null +++ b/docs/en/user/distributed/03-execution.md @@ -0,0 +1,166 @@ +# Execution + +A one-off compile-and-dispatch call is enough for a quick test. Production +code instead amortizes setup — forking chip processes, assembling kernels — +across many dispatches on a reusable `DistributedWorker`. + +## DistributedWorker + +Obtained via `compiled.prepare()`. Setup (fork, comm bootstrap, kernel assembly) +happens once; dispatch happens many times. + +```python +from pypto.runtime import DistributedWorker + +with DistributedWorker(compiled) as rt: + rt(host_x, host_out) + # ... more dispatches ... +# rt.close() runs on exit — releases buffers and shuts down workers. +``` + +### Methods + +| Method | Description | +| ------ | ----------- | +| `compiled.prepare(config=None, callbacks=None)` | Create worker, fork chip processes, return `DistributedWorker`. Use as context manager. | +| `rt(x, y, z)` | Single dispatch — coerces args, calls host_orch. | +| `rt.run(compiled, x, y, z)` | Multi-program dispatch — selects the target program. | +| `rt.alloc_tensor(shape, dtype, *, init=None)` | Allocate a worker-resident `DeviceTensor`. `init` copies from host (one-time H2D). | +| `rt.free_tensor(tensor)` | Release a `DeviceTensor`. | +| `rt.alloc_stacked_tensor(host_w)` | Shard host_w along dim 0 — shard `i` uploaded to card `i`. Returns `StackedDeviceTensor`. | +| `rt.free_stacked_tensor(stacked)` | Release all shards of a `StackedDeviceTensor`. | +| `rt.copy_stacked_from(stacked, host_out)` | D2H read-back of every shard into `host_out` (shared-memory, allocated before `prepare()`). | +| `rt.release_inherited_host_tensor_refs()` | Drop runtime-held host references in the parent process after fork. | +| `rt.close()` | Release buffers, shut down chip workers. Called automatically as context manager. | + +## DeviceTensor + +A worker-resident buffer that lives on the device across dispatches. +When a `DeviceTensor` is passed as an argument to a compiled program, +the runtime skips H2D/D2H copies — the device already has the data. + +```python +import torch +from pypto.runtime import DeviceTensor + +with compiled.prepare() as rt: + weight = rt.alloc_tensor((1024, 4096), torch.float16, init=host_weight) + rt(x, weight, out) # dispatch via worker — no H2D/D2H +``` + +## StackedDeviceTensor + +Sharded across devices — obtained via `rt.alloc_stacked_tensor()`: + +```python +# Host tensor sharded along dim 0 — shard[i] lives on card i. +host_weights = torch.randn(4, 1024, 4096).share_memory_() # 4 shards +with compiled.prepare() as rt: + stacked = rt.alloc_stacked_tensor(host_weights) + rt(x, stacked, out) +``` + +> **Fatal pitfall:** `host_weights` must call `.share_memory_()` *before* +> `prepare()`. The upload runs inside the already-forked chip worker, which +> can only read host memory it inherited at fork — a plain `torch.Tensor` +> raises `ValueError` at `alloc_stacked_tensor()`. + +## One-Shot vs Persistent Worker + +### One-Shot + +```python +import torch +from pypto.ir.distributed_compiled_program import DistributedConfig +from pypto import ir + +dc = DistributedConfig(device_ids=[0, 1, 2, 3]) +compiled = ir.compile(HelloAllReduce, platform="a2a3", distributed_config=dc) + +inputs = torch.randn(4, 1, 256) +outputs = torch.zeros_like(inputs) +compiled(inputs, outputs) # blocks until all ranks finish +``` + +### Persistent Worker (Repeated Dispatch) + +```python +host_x = torch.zeros((4, 1, 256), dtype=torch.float32).share_memory_() +host_out = torch.zeros_like(host_x).share_memory_() + +with DistributedWorker(compiled) as rt: + for step in steps: + host_x.copy_(next_input(step)) + rt(host_x, host_out) + consume(host_out) +``` + +> **Fatal pitfall:** IO buffers passed to `DistributedWorker` must call +> `.share_memory_()` before `prepare()`. If you forget, the runtime rejects +> the buffer at dispatch time — the child processes cannot access the +> parent's private memory. + +## Several Programs on One Worker + +A single `DistributedWorker` can dispatch multiple compiled programs: + +```python +compiled_a = ir.compile(ProgramA, platform="a2a3", distributed_config=dc) +compiled_b = ir.compile(ProgramB, platform="a2a3", distributed_config=dc) + +with compiled_a.prepare(extra_compiled=[compiled_b]) as rt: + rt.run(compiled_a, host_x, host_out) # dispatch ProgramA + rt.run(compiled_b, host_x, host_out) # dispatch ProgramB +``` + +The worker reuses its chip processes and comm setup — no fork penalty. +`compiled_b` must be passed via `extra_compiled=` for `rt.run(compiled_b, ...)` +to find it; passing an unregistered program raises `ValueError`. Preparing +more than one program also puts the worker in multi-program mode, where the +`rt(*args)` shortcut is ambiguous and raises `TypeError` — dispatch every +program explicitly through `rt.run(...)`, including the primary one. + +## CLI Launch + +A distributed program launches the same way as a single-device one — plain +`python script.py`. `DistributedConfig(device_ids=[...])` picks the rank +count and devices; the runtime forks one worker process per device from this +one Python process, so there is no separate multi-process launcher to invoke. + +```bash +python script.py +``` + +## Environment Variables + +### Compile-Time Macros + +These are C preprocessor `#define` macros in `profiling_config.h`, **not environment variables**. +They default to `1` (enabled) and are set at build time via CMake flags. Setting them as shell +env vars has no effect. + +| Macro | Default | Effect | +| ----- | ------- | ------ | +| `SIMPLER_HOST_STRACE` | `1` (on) | Required at build time for `benchmark()` timing markers. Without it, `benchmark()` raises `RuntimeError`. | +| `SIMPLER_DFX` | `1` (on) | Umbrella gate for device-side profiling (orchestrator/scheduler metrics, PMU counters, scope stats, swimlane trace). Sub-tier flags require this to be `1`. | + +### Runtime Environment Variable + +| Variable | Default | Effect | +| -------- | ------- | ------ | +| `SIMPLER_DEVICE_STRACE_ENABLE` | on (unset or non-`"0"`) | Runtime toggle for device-domain `[STRACE]` markers. Set to `0` to suppress device markers while keeping host markers. | + +### Benchmark Env Vars + +The `pypto-lib` golden benchmark harness reads `PYPTO_BENCH` / +`PYPTO_BENCH_ROUNDS` / `PYPTO_BENCH_WARMUP` / `PYPTO_BENCH_RAW` — these are +not defined or consumed anywhere in this repository. See `pypto-lib`'s own +documentation for current defaults. `pypto.runtime.benchmark()` (this +repo's own harness) is documented in [Performance](../performance/index.md). + +## See Also + +- [00-model](00-model.md) — Quickstart and model vocabulary +- [04-debugging](04-debugging.md) — Common failure patterns +- [Performance](../performance/index.md) — Benchmarking and tuning +- [Getting Started](../00-getting_started.md) — Runtime setup diff --git a/docs/en/user/distributed/04-debugging.md b/docs/en/user/distributed/04-debugging.md new file mode 100644 index 0000000000..dd37731a12 --- /dev/null +++ b/docs/en/user/distributed/04-debugging.md @@ -0,0 +1,64 @@ +# Debugging and Pitfalls + +Distributed bugs rarely leave a local stack trace — the symptom shows up on +one rank while the cause is on another. + +## Common Failure Patterns + +| Symptom | Likely Cause | Fix | +| ------- | ------------ | --- | +| **All ranks hang** | Notify/wait ordering — a rank is waiting on a peer that hasn't notified yet | Ensure every rank calls `notify` before any rank calls `wait`. The notify loop should precede the wait loop. | +| **Silent data corruption** | `remote_load` offsets or shape don't match what the peer stored | Verify offsets align with the peer's store offsets. A 1-element shift introduces a full row of garbage. | +| **Signal cell never reaches expected value** | Wrong `NotifyOp`: used `Set` instead of `AtomicAdd` for a multi-participant barrier | Use `AtomicAdd` when N ranks contribute to the same slot; use `Set` for 1:1 exchanges. | +| **Shape mismatch at compile time** | `NR` (world size) used in type annotations without `pl.dynamic` | Wrap runtime-resolved dims in `pl.dynamic("NR")`. The compiler needs the name to bind the runtime value. | +| **`TypeError` raised at dispatch** | IO buffer not `.share_memory_()` before `prepare()` — the child processes cannot see a buffer allocated after the fork | Call `.share_memory_()` on every host tensor passed to the worker, before `prepare()`. | +| **Allreduce rejected inside loop** | Signal protocol can't inject a fresh buffer per iteration | Allocate a fresh signal buffer for each allreduce call outside loops; allreduce inside `for`/`while` is currently rejected. | + +## Fatal Pitfalls + +> **Missing `.share_memory_()`:** IO buffers passed to `DistributedWorker` must +> call `.share_memory_()` before `prepare()`. If you forget, the runtime raises +> a `TypeError` at dispatch time — the child processes cannot access the parent's +> private memory. +> +> **`alloc_window_buffer` given a rank count instead of bytes:** The `size` +> argument to `alloc_window_buffer` is **in bytes**, not elements. Calling +> `alloc_window_buffer(NR)` allocates `NR` bytes, not `NR * sizeof(element)`. +> Use the shape+dtype overload: `alloc_window_buffer([NR, SIZE], dtype=pl.FP32)`. +> +> **`device_ids` disagreeing with `device=`:** `DistributedConfig.device_ids` +> must match the device IDs used in the orchestrator's per-rank dispatch. A +> mismatch — e.g. `device_ids=[0, 1]` but dispatching with `device=r` where +> `r` iterates over `range(4)` — causes undefined behaviour. + +## Diagnostic Flags + +`SIMPLER_HOST_STRACE` and `SIMPLER_DFX` are **compile-time C preprocessor macros** +(`#define` in `profiling_config.h`), not environment variables. Setting them as +shell env vars (e.g. `SIMPLER_DFX=1 python script.py`) has **no effect** — they +are baked in at build time. They default to `1` (enabled). Flipping them is a +`simpler` runtime build-configuration change, not something set via a bare +`cmake -D...` cache variable — see the `simpler` runtime's own build +documentation for the current mechanism. + +Runtime environment variables: + +```bash +# Toggle device-domain [STRACE] markers at runtime: +SIMPLER_DEVICE_STRACE_ENABLE=0 python script.py +``` + +### Distributed DFX Entry Points + +- **L2 swimlane:** `RunConfig(enable_l2_swimlane=True)` — enables per-task timing + inside the worker, propagates through L3 orchestration. Output in span-tree. +- **Scope stats:** `RunConfig(enable_scope_stats=True)` — writes + `dfx_outputs/scope_stats/scope_stats.jsonl` with task_window, heap, and tensormap watermarks. +- **Dependency graph:** `RunConfig(enable_dep_gen=True)` — exports the task dependency + graph for scheduler analysis. + +## See Also + +- [00-model](00-model.md) — Quickstart and model vocabulary +- [02-primitives](02-primitives.md) — The substrate beneath the collectives +- [Performance](../performance/index.md) — Benchmarking and measurement tools diff --git a/docs/en/user/distributed/index.md b/docs/en/user/distributed/index.md new file mode 100644 index 0000000000..8e7e640e76 --- /dev/null +++ b/docs/en/user/distributed/index.md @@ -0,0 +1,58 @@ +# Distributed Programming + +PyPTO's distributed model is built on **symmetric memory and signals**: +every rank sees the same window-buffer address across peers, reaches other +ranks through one-sided `put`/`get`/`remote_load`, and coordinates through +**signal synchronisation** (`notify`/`wait`). A **comm domain** is a subset +of ranks sharing a symmetric window pool; the full world is the default +domain. + +Every allreduce, broadcast, and barrier the compiler lowers is a composition +of these same primitives — the `pld.tensor.*` collectives (`allreduce`, +`barrier`, etc.) are syntactic sugar over them, not a separate library. + +## L2 vs L3 + +| Layer | Scope | API namespace | +| ----- | ----- | ------------- | +| L2 | Single-device (one NPU chip) | `pl.*` | +| L3 | Cross-rank (multiple NPUs or processes) | `pld.*` | + +> **PyPTO's L2/L3 vs simpler's L0–L6:** these two tiers are PyPTO's own +> user-facing vocabulary, not simpler's numbering. Simpler uses a finer +> seven-level hierarchy (L0 core → L1 die → L2 chip → L3 host → L4 pod → +> L5 super-node → L6 cluster); PyPTO's "L2" spans simpler's L0–L2 (everything +> on one chip), and PyPTO's "L3" spans simpler's L3 and up (everything across +> chips). See simpler's +> [Hierarchical Level Runtime](https://hw-native-sys.github.io/simpler/hierarchical-level-runtime/) +> for the full model. + +The distributed chapter covers L3. L2 is covered in the +[Language Guide](../01-language_guide.md). + +## Glossary + +| Term | Definition | +| ---- | ---------- | +| **Rank** | A single process or chip participating in a distributed program. Each rank has a unique rank index assigned at launch time. | +| **Device** | One Ascend NPU chip (or die), identified by a `device_id`. One rank maps to one device. | +| **Node** | A physical machine hosting one or more devices. | +| **Window buffer** | A symmetric per-rank HCCL buffer. Ranks see peers through `CommContext.windowsIn[peer]`/`windowsOut[peer]`. | +| **Comm domain** | A subset of ranks sharing a symmetric window pool. Default: the full world. | +| **Signal** | A cross-rank synchronisation primitive. Notify/wait counters coordinate access to window buffers. | +| **Orchestrator** | The HOST function that allocates window buffers and dispatches kernels to devices. | +| **InCore kernel** | The device-side function that executes on the NPU. | + +## Reading Path + +1. **[00-model](00-model.md)** — Quickstart-first: run a 2-rank program, then the model vocabulary +2. **[01-collectives](01-collectives.md)** — AllReduce, barrier, broadcast, allgather, reduce_scatter, all-to-all +3. **[02-primitives](02-primitives.md)** — notify/wait, remote_load/remote_store, put/get, CommCtx +4. **[03-execution](03-execution.md)** — DistributedWorker lifecycle, DeviceTensor, multi-program, env vars +5. **[04-debugging](04-debugging.md)** — Common failure patterns and diagnostic flags + +## See Also + +- [Getting Started](../00-getting_started.md) — `ir.compile()`, `CompiledProgram`, `DeviceTensor`, `RunConfig` +- [Performance](../performance/index.md) — Benchmarking and tuning distributed programs +- [Simpler Runtime](https://hw-native-sys.github.io/simpler/) — Runtime internals (scheduler, graph building, tensormap) diff --git a/docs/en/user/index.md b/docs/en/user/index.md index 7d018844d2..ba9aaebd16 100644 --- a/docs/en/user/index.md +++ b/docs/en/user/index.md @@ -4,7 +4,7 @@ How to write, compile, run, and debug PyPTO programs. ## Reading paths -Pick the path that matches what you are trying to do. All three assume +Pick the path that matches what you are trying to do. All four assume [Installation](01-installation.md) is done. ### I want to write my first kernel @@ -28,13 +28,22 @@ constrain execution order. ### I have a kernel and it is slow -[Programming Model § memory hierarchy](03-programming-model.md#memory-hierarchy) → -[Diagnostics](../dev/passes/92-diagnostics.md) → -[Runtime DFX](../dev/03-runtime-dfx.md) +[Performance](performance/index.md) → +[Programming Model § memory hierarchy](03-programming-model.md#memory-hierarchy) Check `report/perf_hints.log` from your compile output before measuring anything — the -compiler may already have told you. The dedicated performance chapter is not written yet; -see the table below for where its material currently lives. +compiler may already have told you. The performance chapter covers the full +measure → locate → optimize → verify loop for both single-node and distributed +programs. + +### I want to run across multiple devices + +[Distributed Programming](distributed/index.md) → [Performance § distributed](performance/02-distributed.md) + +Get a single-device kernel running first — distributed programs compose the same +`pl.*` kernels behind `pld.*` collectives and a HOST orchestrator. Once it runs +correctly, the distributed performance page covers ring vs. mesh cost and +cross-rank overlap. ## Contents @@ -47,6 +56,8 @@ see the table below for where its material currently lives. | [Operation Reference](02-operation_reference.md) | The operator surface across the `pl.*`, `pl.tensor.*`, and `pl.tile.*` namespaces | | [Running on Device](00-getting_started.md) | Resident device tensors, explicit dispatch, benchmarking, distributed execution | | [Torch Codegen Debug Guide](03-torch_codegen_debug.md) | Generating a PyTorch reference implementation from the IR to isolate accuracy problems | +| [Distributed Programming](distributed/index.md) | Symmetric-memory model, collectives, primitives, execution, and debugging for cross-rank programs | +| [Performance](performance/index.md) | Measurement methodology, single-node and distributed optimization techniques, and worked cases | ## What PyPTO gives you @@ -59,7 +70,8 @@ see the table below for where its material currently lives. | The full `@pl.jit` family (`.incore`, `.inline`, `.opaque`, `.host`) | [Quickstart](02-quickstart.md), [Language Guide](01-language_guide.md) | | Hand-written C++ kernel integration | [External Kernels](../dev/language/01-external-kernels.md) | | Device-resident tensors, explicit dispatch, benchmarking | [Running on Device](00-getting_started.md) | -| Distributed (multi-card) programs and collectives | [Distributed Operators](../dev/distributed_ops.md) | +| Distributed (multi-card) programs and collectives | [Distributed Programming](distributed/index.md) | +| Performance measurement and tuning, single-node and distributed | [Performance](performance/index.md) | | Accuracy debugging against a PyTorch reference | [Torch Codegen Debug Guide](03-torch_codegen_debug.md) | | Compile-time diagnostics and performance hints | [Diagnostics](../dev/passes/92-diagnostics.md) | | Runtime DFX: swimlane, PMU, dependency graph, scope stats | [Runtime DFX](../dev/03-runtime-dfx.md) | @@ -67,16 +79,14 @@ see the table below for where its material currently lives. ## What is not here yet -This manual is being expanded into a full chaptered structure — tutorials, distributed -programming, performance optimization, and accuracy debugging each get their own -chapter. Until those land, the corresponding material lives in the -[developer documentation](../dev/index.md): +This manual is being expanded into a full chaptered structure — tutorials and +accuracy debugging each get their own chapter. Until those land, the +corresponding material lives in the [developer documentation](../dev/index.md): | Topic | Current location | | ----- | ---------------- | | Tasks, dependencies, `manual_scope` / `submit` | [Python IR Syntax Specification](../dev/language/00-python_syntax.md), [AutoDeriveTaskDependencies](../dev/passes/36-auto_derive_task_dependencies.md) | | Mixed kernels (AIC + AIV in one function) | [LowerAutoVectorSplit](../dev/passes/19-lower_auto_vector_split.md), [ExpandMixedKernel](../dev/passes/20-expand_mixed_kernel.md), [TPUSH/TPOP](../reference/pto-isa/01-tpush_tpop.md) | -| Distributed DSL and collectives | [Distributed Operators](../dev/distributed_ops.md) | | Performance hints and diagnostics | [Diagnostics](../dev/passes/92-diagnostics.md), [Compile Profiling](../dev/01-compile-profiling.md) | | Runtime DFX flags, ring sizing, memory map | [Runtime DFX](../dev/03-runtime-dfx.md), [Per-Task Ring Sizing](../dev/05-runtime-ring-sizing.md), [Memory Map](../dev/07-memory-map.md) | | External C++ kernels | [Integrating Hand-Written C++ Kernels](../dev/language/01-external-kernels.md) | diff --git a/docs/en/user/performance/00-methodology.md b/docs/en/user/performance/00-methodology.md new file mode 100644 index 0000000000..3c24964bff --- /dev/null +++ b/docs/en/user/performance/00-methodology.md @@ -0,0 +1,162 @@ +# Performance Methodology + +The single-node and distributed tracks share the measurement loop below — +pick the tool for the layer you suspect, then confirm the fix moved the +number that mattered. + +## Decision Tree + +```text +Performance below expectation +├─ 1. Did the compiler already hint? → report/perf_hints.log (PH001 TileInnermostDimGranularity, ...) +├─ 2. Which end-to-end segment? → benchmark span tree (print_mean_tree), host vs device domain +├─ 3. Inside a single kernel? → in-core msprof op-simulator, instruction / cycle level +├─ 4. Resources saturated or wasted? → memory map HTML, scope stats (heap / task_window / tensormap) +└─ 5. Is scheduling serialized? → dependency graph (enable_dep_gen), L2 swimlane +``` + +## Quick Start + +The `pypto-lib` golden harness offers an environment-variable-driven quick +start (`PYPTO_BENCH=1 python my_kernel.py`) — see `pypto-lib`'s own +documentation for that harness's env vars and defaults; they are not defined +in this repository. This repo's own benchmarking contract is the +programmatic API below. + +## Programmatic Benchmark API + +```python +from pypto.runtime import benchmark + +compiled = ir.compile(MyProgram) +stats = benchmark(compiled, args=(x, out), rounds=100, warmup=3) + +print(f"median: {stats.device_us_median:.1f} us") +print(f"min: {stats.device_us_min:.1f} us") +print(f"max: {stats.device_us_max:.1f} us") +print(f"mean: {stats.device_us_mean:.1f} us") +print(f"stdev: {stats.device_us_stdev:.1f} us") +``` + +### API Signature + +```python +benchmark( + compiled, # CompiledProgram (L2) or DistributedCompiledProgram (L3) + args, # tuple of dispatch arguments + *, # all remaining args are keyword-only + rounds: int = 100, + warmup: int = 3, + platform: str | None = None, # target platform (L2 only) + device_id: int | None = None, # NPU device index (L2 only) + config: RunConfig | None = None, + persistent: bool = False, # retain CommDomains across dispatches (L3) + reset_persistent_windows: bool | None = None, # zero retained windows before reuse +) -> BenchmarkStats +``` + +### BenchmarkStats Fields + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `device_wall_us` | `list[float]` | Per-round on-NPU device wall (us). L2: one per launch. L3: per-round max across ranks. | +| `host_wall_us` | `list[float]` | Per-round host wall (us). Includes arg coercion and H2D overhead. | +| `rounds` | `int` | Number of measured launches (warmup excluded). | +| `warmup` | `int` | Number of leading launches discarded. | +| `all_zero_device` | `bool` | `True` when every `device_wall_us` is `0` — typical on `*sim` platforms. | + +### Aggregates + +| Property | Description | +| -------- | ----------- | +| `stats.device_us_median` | Median `device_wall_us` (us). | +| `stats.device_us_min` | Minimum `device_wall_us` (us). | +| `stats.device_us_max` | Maximum `device_wall_us` (us). | +| `stats.device_us_mean` | Arithmetic mean (us). | +| `stats.device_us_stdev` | Standard deviation (us). | + +### Span Tree Rendering + +```python +stats.print_tree() # render one per-launch span tree to stdout +stats.print_mean_tree() # render mean-duration tree annotated with +/-stdev +``` + +Expected output for a single-launch L2 kernel: + +```text +launch[0] (pid=1 inv=1 hid=0): +simpler_run 10875.1us +|- bind 29.6us +|- runner_run 10786.3us +| `- device_wall [dev] 9981.6us +| |- orch [dev] 4.0us +| |- sched [dev] 563.4us +| `- post_orch [dev] 0.5us +`- validate 5.5us +``` + +The **primary metric** is `device_wall [dev]` — the total on-NPU time. + +### Complete Example + +```python +import torch +import pypto.language as pl +from pypto import ir +from pypto.runtime import benchmark + +ROWS = COLS = 128 + +@pl.program +class MatAdd: + @pl.function(type=pl.FunctionType.InCore) + def add_kernel( + self, + a: pl.Tensor[[ROWS, COLS], pl.FP32], + b: pl.Tensor[[ROWS, COLS], pl.FP32], + c: pl.Out[pl.Tensor[[ROWS, COLS], pl.FP32]], + ) -> pl.Tensor[[ROWS, COLS], pl.FP32]: + ta = pl.load(a, [0, 0], [ROWS, COLS]) + tb = pl.load(b, [0, 0], [ROWS, COLS]) + return pl.store(pl.add(ta, tb), [0, 0], c) + + @pl.function(type=pl.FunctionType.Orchestration) + def chip_orch(self, a, b, c): + return self.add_kernel(a, b, c) + +compiled = ir.compile(MatAdd, platform="a2a3sim") +a = torch.full((ROWS, COLS), 2.0) +b = torch.full((ROWS, COLS), 3.0) +c = torch.zeros((ROWS, COLS)) + +stats = benchmark(compiled, args=(a, b, c), rounds=20, warmup=5, platform="a2a3sim") + +print(f"median: {stats.device_us_median:.1f} us") +if stats.all_zero_device: + print("[sim] device_wall_us is all-zero — expected on simulator builds") +``` + +### Interpreting Variance + +- **Prefer median over mean** — median is robust to cold-start outliers +- **Increase `warmup`** to absorb one-time setup (5-10 typical on hardware) +- **Check `all_zero_device`** before interpreting results on simulator builds + +## Important Caveats + +> **`SIMPLER_HOST_STRACE` must be compiled into the runtime.** If the runtime +> was built without this compile-time macro, `benchmark()` raises `RuntimeError`. +> +> **`*sim` platforms report 0 for device wall.** Check `stats.all_zero_device` +> to detect this — if `True`, all `device_wall_us` samples are 0. +> +> **L3 requires shared-memory IO tensors.** Every host `torch.Tensor` passed +> to `benchmark()` for a distributed program must call `.share_memory_()` before +> the call. + +## See Also + +- [01-single-node](01-single-node.md) — Single-node performance techniques +- [02-distributed](02-distributed.md) — Distributed performance and bus bandwidth +- [03-cases](03-cases.md) — End-to-end worked examples diff --git a/docs/en/user/performance/01-single-node.md b/docs/en/user/performance/01-single-node.md new file mode 100644 index 0000000000..b45a4e7a9f --- /dev/null +++ b/docs/en/user/performance/01-single-node.md @@ -0,0 +1,216 @@ +# Single-Node Performance + +Every technique below states **when it applies, what it costs, how to +enable it, and how to verify it took effect**. + +## Partitioning and Parallelism + +### `pl.split(SplitMode)` + +Split cross-core data transfer geometrically within a `CORE_GROUP` region — +not a standalone call; it's passed into `pl.at(..., optimizations=[...])`. + +- **When:** A `CORE_GROUP` region's data needs partitioning across its cores +- **Cost:** Requires split-compatible operations +- **How:** `pl.at(level=pl.Level.CORE_GROUP, optimizations=[pl.split(pl.SplitMode.UP_DOWN)])` — modes are `NONE`, `UP_DOWN` (height halved), `LEFT_RIGHT` (width halved) +- **Verify:** Compare against non-split baseline via benchmark + +### `pl.split_aiv` + +Split computation across AIV (vector) cores specifically. + +- **When:** Vector-heavy workloads benefit from AIV parallelism +- **Cost:** AIV-only — incompatible with AIC units +- **Verify:** Check that AIV cores are utilized in the memory map + +### `pl.spmd(N)` + +Single-program-multiple-data parallelism — launch N copies of the same kernel +with different data shards. + +- **When:** Embarrassingly parallel workloads +- **Cost:** N x memory footprint; workspace serialization for >1 +- **How:** `with pl.spmd(n):` or `for i in pl.spmd(n):` +- **Verify:** Check benchmark scaling — near-linear speedup expected + +### `pl.cluster()` + +Group co-scheduled AIC (Cube) and AIV (Vector) kernels sharing physical +cluster resources. + +- **When:** Workloads benefit from concurrent AIC + AIV execution +- **Cost:** Requires cluster-compatible kernel pairs +- **Verify:** L2 swimlane shows concurrent AIC/AIV execution + +### `pl.at(level=)` + +Mark a region of code for execution at a specific point in the hierarchy +(`pl.Level.AIV`, `.AIC`, `.CORE_GROUP`, `.CHIP_DIE`, `.CHIP`, `.HOST`, ...) — +this is where the region *executes*, not a memory tier. + +- **When:** Coordinating co-scheduled AIC/AIV work, or applying `pl.split` +- **Cost:** The chosen level determines which `optimizations=` entries apply +- **Verify:** L2 swimlane shows the region executing at the specified level + +## Pipelining and Unrolling + +### `pl.pipeline` + +Software pipeline a loop body to overlap compute across iterations. + +- **When:** Loop bodies with independent iterations +- **Cost:** Increased register and buffer pressure +- **How:** Wrap the loop body with `pl.pipeline` +- **Verify:** Benchmark shows reduced per-iteration latency + +### `pl.unroll` + +Fully unroll a compile-time-known loop. + +- **When:** Small loop trip counts known at compile time +- **Cost:** Larger binary — code size grows with unroll factor +- **Verify:** Inspect unrolled code in the compiled artifact + +### Cross-Core Software Pipelining + +Use `pl.cross_core_slot(slot_num=)` to pipeline data across AICore +computing units. + +- **When:** Ring-buffer or producer-consumer patterns across cores +- **Cost:** Ring depth determines buffer size +- **How:** Set `slot_num` to the ring depth +- **Verify:** Swimlane shows pipelined execution with no idle gaps + +## Matmul Path + +### AutoTileMatmulL0 + +The auto-tiling pass selects L0 matmul tile sizes. + +- **When:** All matmul operations (enabled by default) +- **Cost:** Compile-time analysis — no runtime overhead +- **Verify:** IR dump shows tiled matmul dimensions + +### `enable_pypto_l0c_double_buffer` + +Enable double-buffering for L0C output buffer. + +- **When:** Matmul-bound workloads where output write-back is a bottleneck +- **Cost:** 2x L0C buffer allocation +- **How:** `ir.compile(..., enable_pypto_l0c_double_buffer=True)` +- **Verify:** Benchmark shows reduced sched time in span tree + +### `a_trans`/`b_trans` + +Transpose matmul operands in-place. + +- **When:** Memory layout doesn't match the matmul instruction's expected order +- **Cost:** Potential additional transpose overhead +- **Verify:** Benchmark both transposed and non-transposed + +### Split-K + Atomic Add + +Split the K-dimension of a matmul across multiple units with atomic +accumulation. + +- **When:** Very large K dimensions exceed single-unit capacity +- **Cost:** Atomic add introduces non-determinism (order of accumulation varies) +- **Verify:** Accuracy check against golden; benchmark for throughput + +## Memory + +### `target_memory` + +Select the on-chip memory space for a tile allocation (`MemorySpace.DDR` +for off-chip, or `.Vec` / `.Mat` / `.Left` / `.Right` / `.Acc` for on-chip +buffers). + +- **When:** Data placement optimization +- **Cost:** On-chip spaces are faster but much smaller than DDR +- **How:** `pl.load(a, [0, 0], [rows, cols], target_memory=pl.MemorySpace.Vec)` +- **Verify:** Memory map confirms allocation at the specified space + +### MemoryReuse vs `memory_planner=PTOAS` + +Memory planning strategies for allocating buffers. + +- **When:** MemoryReuse (default) — reuse buffers when live ranges don't overlap +- **When:** `memory_planner=PTOAS` — delegate to the PTOAS memory planner +- **Cost:** PTOAS planner is more aggressive but may increase compile time +- **Verify:** Memory map shows buffer allocation and reuse + +### Persistent L3 + +Keep L3 buffers resident across dispatches. + +- **When:** Multi-dispatch workloads with large working sets +- **Cost:** Reduced L3 availability for other uses +- **Verify:** Scope stats show persistent allocation + +## Scheduling + +### `predicate=` + +Skip tasks dynamically at the dispatch point. + +- **When:** Conditional execution paths +- **Cost:** Negligible — predicate check at dispatch +- **Verify:** Dependency graph shows skipped edges + +### `no_dep` + +Drop automatic dependency inference for a call site. + +- **When:** Two ops appear to overlap but are actually independent +- **Cost:** Incorrect use causes race conditions +- **Verify:** Dependency graph confirms no edge between the ops + +### `allow_early_resolve` + +Allow the scheduler to resolve a task early. + +- **When:** Output is produced before the task's full completion +- **Cost:** Weaker ordering guarantees +- **Verify:** Swimlane trace confirms early resolution + +### `manual_scope` + +Turn off automatic dependency tracking for a region. + +- **When:** Manual control over every dependency edge +- **How:** `with pl.scope(mode=pl.ScopeMode.MANUAL):` or `with pl.manual_scope():` +- **Cost:** All edges must be declared explicitly via `deps=` +- **Verify:** Dependency graph matches the declared edges exactly + +### `task_dummy` + +Insert a no-op task as a fan-in point. + +- **When:** Multiple producers feed a single consumer with no data dependency +- **How:** `barrier = pl.system.task_dummy(deps=[task_a, task_b])` — `deps` is required and keyword-only +- **Verify:** Dependency graph shows the dummy task as a convergence node + +### Ring Sizing + +Tune ring-task window and heap sizes for the L2 scheduler. + +- **When:** Task-heavy workloads benefit from larger ring buffers +- **How:** `RunConfig(ring_task_window=N, ring_heap=M)` +- **Verify:** Scope stats show ring buffer utilization + +## Data Residency + +### `DeviceTensor` + +Keep tensors resident on the device across dispatches. + +- **When:** Weights, lookup tables, or other reusable data +- **Cost:** Reduced device memory for other allocations +- **How:** `rt.alloc_tensor(shape, dtype, init=host_data)` +- **Verify:** H2D/D2H spans absent from second dispatch onward + +## See Also + +- [00-methodology](00-methodology.md) — Measurement loop and tools +- [02-distributed](02-distributed.md) — Distributed performance techniques +- [03-cases](03-cases.md) — End-to-end worked examples diff --git a/docs/en/user/performance/02-distributed.md b/docs/en/user/performance/02-distributed.md new file mode 100644 index 0000000000..ddb5fe01d4 --- /dev/null +++ b/docs/en/user/performance/02-distributed.md @@ -0,0 +1,188 @@ +# Distributed Performance + +Distributed (L3) programs add cross-rank concerns — bus bandwidth, collective +choice, start skew — on top of everything in single-node performance. + +## L3 Distributed Benchmarking + +Distributed programs (`DistributedCompiledProgram`) use the same `benchmark()` +API but have important differences in timing and preparation. + +### Preparation + +```python +import torch +from pypto.runtime import benchmark + +# Shared-memory host tensors — MUST call .share_memory_() before benchmark(). +host_x = torch.zeros((4, 1, 256), dtype=torch.float32).share_memory_() +host_out = torch.zeros_like(host_x).share_memory_() + +stats = benchmark(compiled, (host_x, host_out), rounds=100, warmup=3) +``` + +### L3 Metrics + +| Metric | `per_round("...")` key | Description | +| ------ | ---------------------- | ----------- | +| device | `"device"` | Per-round max across ranks of each rank's summed dispatch device walls (us). | +| host | `"host"` | Per-round max across ranks of host wall (us). | +| effective | `"effective"` | Per-round max effective window (orch/sched union, us). | +| union | `"union"` | Cross-rank host-timeline union: `max(host-end) - min(host-start)` across all ranks' dispatches (us). Captures overlap and start skew. L3 only. | + +```python +# Per-rank breakdown (L3 only). +ranks = stats.per_rank("device") # {pid: [round0_us, round1_us, ...]} + +# Per-round aggregation. +device = stats.per_round("device") # list[float], length = rounds +union = stats.per_round("union") # list[float], L3 only +``` + +### DFX Flag Availability + +`RunConfig(enable_l2_swimlane=True)` enables per-task timing inside the worker +and propagates through L3 orchestration — swimlane traces appear in span-tree +output even for distributed jobs. + +## Understanding Bus Bandwidth + +Bus bandwidth (`busbw`) is the standard metric for evaluating collective +communication performance, adopted from the nccl-tests benchmark suite. +It corrects for the fact that algorithms like AllReduce transfer different +amounts of data across the interconnect than the algorithmic data size. + +### Formulas + +```text +algbw = data_size / time (algorithmic bandwidth) +busbw = algbw x correction_factor (interconnect-corrected bandwidth) +``` + +### Correction Factors + +| Operation | Correction Factor | Notes | +| --------- | ----------------- | ----- | +| AllReduce | `2(n-1)/n` | Two-way traffic (reduce + broadcast) scaled by rank count. Approaches 2 for large n. | +| AllGather | `(n-1)/n` | Each rank receives n-1 chunks. | +| ReduceScatter | `(n-1)/n` | Each rank sends n-1 chunks. | +| All-to-All | `(n-1)/n` | Personalized exchange — same factor as AllGather/ReduceScatter. | +| Broadcast | `1` | Root sends to n-1 ranks, but the link carries data once. | +| Reduce | `1` | n-1 ranks send to root. | +| Point-to-Point | `1` | Single-link transfer. | + +### Worked Example + +Hypothetical numbers to illustrate the formula (per v6 §8.1 item 5, real +measured figures are backfilled once data accumulates) — 8 ranks, 1 GB +AllReduce completing in 100 ms: + +```text +algbw = 1 GB / 0.100 s = 10 GB/s +busbw = 10 x 2(8-1)/8 = 10 x 14/8 = 17.5 GB/s +``` + +If your interconnect has a theoretical ceiling of 25 GB/s, then 17.5 GB/s is +**70% utilization** — good for an initial implementation. 90%+ means you're +near the hardware bandwidth ceiling. + +## Collective Algorithm Choice + +### Mesh vs Ring Trade-off + +| Aspect | Mesh | Ring | +| ------ | ---- | ---- | +| Remote traffic per step | O(N) | O(N/P) | +| Barrier rounds | 1 | 2(P-1) | +| Signal shape | `[NR, 1]` | `[2(NR-1), NR]` | +| NR support | `pl.dynamic("NR")` | Compile-time static only | +| Best for | Small messages | Large messages (>16 KiB) | + +**Rule of thumb:** Use mesh for small messages and low latency; switch to +ring when bandwidth plateaus. + +### Overlapping Communication with Compute + +PyPTO's signal model allows overlapping communication and computation +through pipelined notify/wait patterns: + +- Use non-blocking `notify` early in a loop iteration +- Schedule compute work between `notify` and `wait` +- The `wait` blocks only when the result is needed + +### Cross-Rank Start Skew + +The `union` metric captures cross-rank start skew — the difference between +the earliest rank's dispatch start and the latest rank's dispatch end. +High `union` relative to `device` indicates poor rank synchronisation. + +## Resident Shards + +Use `alloc_stacked_tensor` to keep model weights resident on each device: + +```python +host_weights = torch.randn(4, 1024, 4096).share_memory_() +with compiled.prepare() as rt: + stacked = rt.alloc_stacked_tensor(host_weights) + rt(x, stacked, out) + # Weights stay resident — no H2D on subsequent dispatches. +``` + +## Ring Sizing and Prewarm + +Tune ring-task window and heap sizes via `RunConfig`, passed to both +`prepare()` and every dispatch call: + +```python +from pypto.runtime import RunConfig + +compiled = ir.compile(MyRingProgram, platform="a2a3", distributed_config=dc) +ring_config = RunConfig(ring_task_window=256, ring_heap=1024) +worker = compiled.prepare(config=ring_config) +worker(host_x, host_out, config=ring_config) # same config on every dispatch +``` + +`prepare(config=...)` only prewarms the runtime-arena cache with that sizing +so the **first** dispatch skips the ~800ms cold build — the config is not +stored on the worker. Every dispatch must pass its own `config=` with the +same `ring_task_window` / `ring_heap`, or the arena rebuilds (the cache is +single-slot, so alternating sizings rebuilds on every switch). + +`ring_task_window` must be a power of 2 `>= 4` (or a 4-element list/tuple +sizing rings 0..3 independently); `ring_heap` (in bytes) must be a power of +2 `>= 1024`. + +## Sharing One Worker Across Programs + +A single `DistributedWorker` can dispatch multiple compiled programs, +reusing its chip processes and comm setup: + +```python +with compiled_a.prepare(extra_compiled=[compiled_b]) as rt: + rt.run(compiled_a, host_x, host_out) # dispatch ProgramA + rt.run(compiled_b, host_x, host_out) # dispatch ProgramB +``` + +Preparing more than one program puts the worker in multi-program mode, where +the `rt(*args)` shortcut is ambiguous and raises `TypeError` — dispatch every +program explicitly through `rt.run(...)`, including the primary one. + +## Important Caveats + +> **`*sim` platforms report 0 for device wall.** Simulator builds emit host +> `[STRACE]` markers but not device-domain spans. Check +> `stats.all_zero_device` to detect this. +> +> **L3 requires shared-memory IO tensors.** Every host `torch.Tensor` passed +> to `benchmark()` for a distributed program must call `.share_memory_()` before +> the call. Forgetting this causes a `TypeError` at dispatch time. +> +> **DFX flags are plumbed through L3.** `RunConfig(enable_l2_swimlane=True)` +> enables per-task timing and swimlane traces propagate through L3 orchestration. + +## See Also + +- [00-methodology](00-methodology.md) — Measurement loop and tools +- [01-single-node](01-single-node.md) — Single-node performance techniques +- [03-cases](03-cases.md) — End-to-end worked examples +- [Distributed programming](../distributed/index.md) — Writing distributed programs diff --git a/docs/en/user/performance/03-cases.md b/docs/en/user/performance/03-cases.md new file mode 100644 index 0000000000..350aed2f05 --- /dev/null +++ b/docs/en/user/performance/03-cases.md @@ -0,0 +1,27 @@ +# Performance Cases + +Each case below follows the same pattern: **baseline → investigation → +change → effect → verification**. + +> **Status:** Performance cases are planned but not yet written. Real measured +> data across representative workloads has not yet been accumulated. The user +> manual plan ([USER_MANUAL_PLAN_EN §8.1 item 5](https://github.com/hw-native-sys/pypto/issues/2120)) +> targets this page for methodology and relative trends in the first version, +> with measured values backfilled once data accumulates. +> +> The single-node performance techniques in [01-single-node](01-single-node.md) +> and distributed performance guidance in [02-distributed](02-distributed.md) +> are available today. +> +> Planned cases: +> +> - **Single-node:** Tile dimension granularity, auto-tiling diagnostics, false +> task dependencies, ring-sizing arena rebuilds. +> - **Distributed:** Mesh-to-ring collectives transition, resident shards vs +> H2D per-dispatch. + +## See Also + +- [00-methodology](00-methodology.md) — Measurement loop and tools +- [01-single-node](01-single-node.md) — Single-node performance techniques +- [02-distributed](02-distributed.md) — Distributed performance and bus bandwidth diff --git a/docs/en/user/performance/index.md b/docs/en/user/performance/index.md new file mode 100644 index 0000000000..4f6910cb4e --- /dev/null +++ b/docs/en/user/performance/index.md @@ -0,0 +1,32 @@ +# Performance + +PyPTO's performance work follows a **measure → locate → optimize → verify** +loop, split into two tracks: + +| Track | Scope | Page | +| ----- | ----- | ---- | +| Shared methodology | Tools and measurement loop (both tracks) | [00-methodology](00-methodology.md) | +| Single-node | Kernel, tile, pipelining, matmul, memory, scheduling | [01-single-node](01-single-node.md) | +| Distributed | Collective cost, ring vs mesh, cross-rank skew, bus bandwidth | [02-distributed](02-distributed.md) | +| Cases | End-to-end worked examples | [03-cases](03-cases.md) | + +> **Prerequisites:** [Distributed programming](../distributed/00-model.md) for +> the distributed track; [Getting Started](../00-getting_started.md) for a +> kernel-authoring baseline. + +## Tool Matrix + +| Tool | Observes | Entry point | +| ---- | -------- | ----------- | +| Compile-time perf hints | Code patterns | `report/perf_hints.log` | +| Benchmark span tree | End-to-end segmentation | `pypto.runtime.benchmark` → `stats.print_mean_tree(spread=...)` | +| In-core msprof | Per-kernel cycles | op-simulator + Insight traces | +| Memory map | On-chip buffers | `pypto.tools.memory_map` → HTML | +| Scope stats | Runtime watermarks | `RunConfig(enable_scope_stats=True)` | +| L2 swimlane / PMU / dep gen | Task scheduling | `RunConfig(enable_l2_swimlane / enable_pmu / enable_dep_gen)` | + +## See Also + +- [Distributed](../distributed/index.md) — Writing distributed programs +- [Getting Started](../00-getting_started.md) — `ir.compile()` and `RunConfig` +- [Simpler Runtime](https://hw-native-sys.github.io/simpler/) — Scheduler internals diff --git a/docs/zh/user/00-getting_started.md b/docs/zh/user/00-getting_started.md index b662d7c958..97e62aba09 100644 --- a/docs/zh/user/00-getting_started.md +++ b/docs/zh/user/00-getting_started.md @@ -130,6 +130,9 @@ PR #1177，在 `SIMPLER_DFX` 下默认开启）；用 simpler 的 `strace_timing ### 性能基准（`benchmark`） +完整的基准测试指南见 [performance/00-methodology.md](performance/00-methodology.md)—— +涵盖程序化 `benchmark()` API、L3 分布式计时、bus-bandwidth 公式以及常见注意事项。 + 对于 register-once + 多轮（rounds）模式，`pypto.runtime.benchmark` 封装了循环 与聚合：它注册 *compiled* 一次并发起 `rounds` 次廉价 launch（不再每轮重付 register/load），读取每次 launch 的 `[STRACE]` 标记并返回 `BenchmarkStats`： @@ -204,6 +207,60 @@ dispatch 的量。当某 rank 每轮恰好只有 1 次 dispatch 时，求和即 ### 分布式（L3+）程序 +完整的分布式编程模型——从 `alloc_window_buffer` 到 `allreduce`、`barrier`、 +`broadcast` 等集合通信——见 [分布式编程](distributed/00-model.md)。下面展示 +InCore kernel（执行平面）的 mesh allreduce Hello World 示例；完整的可运行程序 +还需包含 host 编排器、`ir.compile` 及分布式 worker 设置，详见上述指南： + +```python +import pypto.language as pl +import pypto.language.distributed as pld + +NR = pl.dynamic("NR") + +@pl.program +class HelloAllReduce: + @pl.function(type=pl.FunctionType.InCore) + def reduce_step( + self, + inp: pl.Tensor[[1, 256], pl.FP32], + out: pl.Out[pl.Tensor[[1, 256], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, 256], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, 256], pl.FP32]: + ctx = pld.get_comm_ctx(data) + my_rank = pld.rank(ctx) + nranks = pld.nranks(ctx) + + # 1. Stage-in：将本地输入复制到本 rank 的 window 分片。 + data = pl.store(pl.load(inp, [0, 0], [1, 256]), [0, 0], data) + + # 2. Barrier：通知每个对端，然后等待每个对端。 + for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify(signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd) + for src in pl.range(nranks): + if src != my_rank: + pld.system.wait(signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge) + + # 3. 计算：加载自身分片，remote-load 每个对端，累加。 + acc = pl.load(data, [0, 0], [1, 256]) + for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, 256]) + acc = pl.add(acc, peer_tile) + + # 4. Stage-out：将累加器写入本地输出。 + out = pl.store(acc, [0, 0], out) + return out +``` + +指南包含**逐行解读、ring allreduce 权衡、notify/wait 握手模式以及调试表格**。 +完整章节见 [distributed/index.md](distributed/index.md)。 + `ir.compile` 对 L3+ 分布式程序返回的 `DistributedCompiledProgram` 与 `CompiledProgram` 一样接受 `DeviceTensor` 入参：用 worker 常驻 buffer 替代 `torch.Tensor`，runtime 即对该 参数跳过 H2D/D2H。这是在 generate 循环的多次 dispatch 之间保持大块静态权重常驻的推荐做法。 diff --git a/docs/zh/user/02-operation_reference.md b/docs/zh/user/02-operation_reference.md index f5296617f1..d93ff47e59 100644 --- a/docs/zh/user/02-operation_reference.md +++ b/docs/zh/user/02-operation_reference.md @@ -277,3 +277,26 @@ packed predicate mask；A2/A3 上如需得到数值结果，请配合 `sel` 和 | `create_tensor` | `(shape: Sequence[IntLike], dtype: DataType, layout: TensorLayout = None, init_value: int \| float \| None = None) -> Tensor` | 创建张量（从 `pl.tensor` 提升；`init_value` 由 AICPU 预填充缓冲区——`0` 对任意 dtype 清零，非零值需整型或 ≥32 位浮点 dtype） | | `max` | `(lhs: Scalar \| int \| Expr, rhs: Scalar \| int \| Expr) -> Scalar` | 两个**标量**取最大值（不是 tile 规约 —— 请用 `pl.tile.row_max` / `pl.tile.col_max`） | | `min` | `(lhs: Scalar \| int \| Expr, rhs: Scalar \| int \| Expr) -> Scalar` | 两个**标量**取最小值（不是 tile 规约 —— 请用 `pl.tile.row_min` / `pl.tile.col_min`） | + +## 分布式 / 集合通信操作 (`pld.*`) + +分布式和集合通信操作位于 `pld` 命名空间（`import pypto.language.distributed as pld`）。 +完整参考见 [distributed/index.md](distributed/index.md)； +教程见 [distributed/00-model.md](distributed/00-model.md)。 + +### 功能矩阵 + +| 操作 | API | 模式 | ReduceOp | Atomic | 支持的 dtype | 说明 | +| ---- | --- | ---- | -------- | ------ | ------------ | ---- | +| AllReduce | `pld.tensor.allreduce` | `mesh`（InCore + HOST），`ring`（仅 InCore） | `Sum`、`Max`、`Min`、`Prod` | — | FP16、FP32——编译期硬性检查（`allreduce.cpp`），InCore 和 HOST 均适用 | Mesh: O(N) 远程流量。Ring: O(N/P) 流量，2(P-1) 步；目前仅限 InCore。 | +| AllGather | `pld.tensor.allgather` | — | — | — | 仅 FP32（HOST builtin）；任意 GM dtype（InCore） | 推式。输入和 target 必须是不同的 buffer。 | +| ReduceScatter | `pld.tensor.reduce_scatter` | — | 仅 `Sum` | — | 仅 FP32（HOST builtin）；任意 GM dtype（InCore） | 每个 rank 在调用前将全部 NR 个数据块写入。 | +| Broadcast | `pld.tensor.broadcast` | — | — | — | 仅 FP32（HOST builtin）；任意 GM dtype（InCore） | Root 在调用前将数据写入。 | +| All-to-All | `pld.tensor.all_to_all` | — | — | — | 仅 FP32（HOST builtin）；任意 GM dtype（InCore） | 个性化交换。输入和 target 必须是不同的 buffer。 | +| Barrier | `pld.tensor.barrier` | — | — | — | — | Signal 为 INT32，每次调用单次使用。 | +| Put | `pld.tensor.put` | — | — | `None_` / `Add` | 所有 GM dtype | `dst` 必须是 window-bound。支持分块和流水线 staging。 | +| Get | `pld.tensor.get` | — | — | — | 所有 GM dtype | `src` 必须是 window-bound。支持分块和流水线 staging。 | +| Notify | `pld.system.notify` | `AtomicAdd` / `Set` | — | — | — | 仅副作用的信号投递。 | +| Wait | `pld.system.wait` | `Eq` / `Ge` | — | — | — | 仅副作用的信号阻塞。 | +| Remote Load | `pld.tile.remote_load` | — | — | — | 任意（tile） | Tile 级跨 rank 加载。 | +| Remote Store | `pld.tile.remote_store` | — | — | — | 任意（tile） | Tile 级跨 rank 写入。 | diff --git a/docs/zh/user/distributed/00-model.md b/docs/zh/user/distributed/00-model.md new file mode 100644 index 0000000000..80499291fc --- /dev/null +++ b/docs/zh/user/distributed/00-model.md @@ -0,0 +1,184 @@ +# 分布式编程模型 + +> **前置知识：** [快速入门](../00-getting_started.md) — 基本 PyPTO tensor/tile 模型。 +> 本指南使用 `pld` 命名空间（`import pypto.language.distributed as pld`）。 + +## 快速开始：2-Rank AllReduce + +最简单的分布式程序——两个 rank 对各自数据求和，结果相同。 + +```python +import pypto.language as pl +import pypto.language.distributed as pld + +NR = pl.dynamic("NR") +SIZE = 256 + +@pl.program +class HelloAllReduce: + @pl.function(type=pl.FunctionType.InCore) + def reduce_step( + self, + inp: pl.Tensor[[1, SIZE], pl.FP32], + out: pl.Out[pl.Tensor[[1, SIZE], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, SIZE], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, SIZE], pl.FP32]: + ctx = pld.get_comm_ctx(data) + my_rank = pld.rank(ctx) + nranks = pld.nranks(ctx) + + # 1. Stage-in：将本地输入数据复制到本 rank 的 window 分片。 + local = pl.load(inp, [0, 0], [1, SIZE]) + data = pl.store(local, [0, 0], data) + + # 2. Barrier：通知每个对端，然后等待每个对端。 + for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify( + signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd, + ) + for src in pl.range(nranks): + if src != my_rank: + pld.system.wait( + signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge, + ) + + # 3. 计算：累加每个对端的分片。 + acc = pl.load(data, [0, 0], [1, SIZE]) + for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, SIZE] + ) + acc = pl.add(acc, peer_tile) + + # 4. Stage-out：将累加器写入本地输出。 + return pl.store(acc, [0, 0], out) + + @pl.function(type=pl.FunctionType.Orchestration) + def chip_orch( + self, + inp: pl.Tensor[[1, SIZE], pl.FP32], + out: pl.Out[pl.Tensor[[1, SIZE], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, SIZE], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, SIZE], pl.FP32]: + # 逐设备的编排包装函数——HOST 派发的是它，而不是直接派发 InCore kernel。 + return self.reduce_step(inp, out, data, signal) + + @pl.function(level=pl.Level.HOST, role=pl.Role.Orchestrator) + def orchestrator( + self, + inputs: pl.Tensor[[NR, 1, SIZE], pl.FP32], + outputs: pl.Out[pl.Tensor[[NR, 1, SIZE], pl.FP32]], + ) -> pl.Tensor[[NR, 1, SIZE], pl.FP32]: + data_buf = pld.alloc_window_buffer(SIZE * pl.FP32.get_byte()) + signal_buf = pld.alloc_window_buffer(pld.world_size() * pl.INT32.get_byte()) + + for r in pl.range(pld.world_size()): + data = pld.window(data_buf, [1, SIZE], dtype=pl.FP32) + signal = pld.window(signal_buf, [pld.world_size(), 1], dtype=pl.INT32) + self.chip_orch(inputs[r], outputs[r], data, signal, device=r) + return outputs +``` + +### 预期输出 + +每个 rank `r` 的 `outputs[r] == sum(inputs[*])`。2 个 rank 输入分别为 +`[[1, 2, 3]]` 和 `[[10, 20, 30]]`，两个 rank 均看到 `[[11, 22, 33]]`。 + +### 启动命令 + +```bash +python script.py +``` + +不涉及任何多进程启动器。`DistributedConfig(device_ids=[...])` 告诉编译后的 +程序使用哪些设备；运行时会从这一个 Python 进程中为每个设备 fork 出一个 +worker 进程。 + +## PyPTO 中的分布式编程是什么？ + +PyPTO 的分布式模型采用**对称内存 + 信号**范式。每个 rank 有一个 per-rank +**window buffer**，各对端的地址空间是对称的。通信通过单边 +`put`/`get`/`remote_load` 加**信号同步**（`notify`/`wait`）来完成。 +**通信域** 是共享对称 window pool 的 rank 子集；整个 world 为默认通信域。 + +编译器 lowering 出的每个 allreduce、broadcast、barrier 都是这些相同原语的 +组合——`pld.tensor.*` 集合通信（`allreduce`、`barrier` 等）是它们的语法糖， +而非另一套独立的库。完整 API 见 [01-collectives](01-collectives.md)。 + +## 模型 + +### HOST 编排器 + +HOST 函数分配 window buffer、分发 kernel 并管理控制平面： + +- 声明为 `@pl.function(level=pl.Level.HOST, role=pl.Role.Orchestrator)` 或 `@pl.jit.host` +- 调用 `alloc_window_buffer`、`window()`，通过 `device=r` 进行 per-rank 分发 +- 每个进程运行一次——不在 NPU 上执行 +- 派发的是逐设备的 `@pl.function(type=pl.FunctionType.Orchestration)` 包装函数 + （而非直接派发 `InCore` kernel）——见下方 Per-Rank 分发 + +### InCore Kernel + +InCore 函数在 NPU 设备上运行： + +- 声明为 `@pl.function(type=pl.FunctionType.InCore)` +- 接收 window-bound 的 `DistributedTensor` 参数 +- 使用 `notify`/`wait` 进行跨 rank 同步，`remote_load`/`remote_store` 进行 RMA +- 不调用 `alloc_window_buffer` 或 `world_size()` + +### Per-Rank 分发 + +HOST 从不直接派发 `InCore` 函数。它通过设置 `device=r` 派发一个逐设备的 +`@pl.function(type=pl.FunctionType.Orchestration)` 包装函数；该包装函数再调用 +`InCore` kernel，且不带 `device=` 参数。每个 rank 通过 `CommContext` 看到 +自己的对称 window buffer 视图。 + +### Window Buffer 生命周期 + +`alloc_window_buffer(size)` 创建 per-rank buffer。`window(buf, shape, dtype)` +创建类型化视图。Buffer 在 host 编排器调用期间存活。 + +### 控制平面 vs 执行平面 + +```text +HOST 编排器 (@pl.function(level=HOST, role=Orchestrator)) + ├── alloc_window_buffer(...) ← 控制平面：声明布局 + ├── window(buf, shape, dtype) ← 控制平面：创建类型化视图 + └── for r in ranks: ← 分发循环 + self.chip_orch(..., device=r) ← 桥接到逐设备包装函数 + +编排包装函数 (@pl.function(type=Orchestration)) + └── self.reduce_step(...) ← 调用 InCore kernel，不带 device= + +InCore kernel (@pl.function(type=InCore)) + ├── notify / wait ← 执行平面：跨 rank 同步 + ├── remote_load ← 执行平面：读取对端数据 + └── store ← 执行平面：写入本地输出 +``` + +## 逐行解读 + +| 代码 | 说明 | +| ---- | ---- | +| `NR = pl.dynamic("NR")` | world size 在构建时未知。`pl.dynamic` 将维度推迟到运行时分发——host 从 `len(device_ids)` 绑定。 | +| `pl.InOut[pld.DistributedTensor[...]]` | `data` 和 `signal` 是 window-bound：每个 rank 共享相同的地址空间布局。`InOut` 表示 kernel 既读又写。 | +| `pld.get_comm_ctx(data)` | 将 window-bound tensor 提升为通信域句柄。每个 rank 获得自己的 `ctx`，从中读取 `rank()` 和 `nranks()`。 | +| `pld.system.notify(..., op=AtomicAdd)` | 每个 rank 原子地加 1 到每个对端的 signal slot。`AtomicAdd` 在此正确，因为 N 个 rank 写入同一个 slot（全局屏障）。1:1 握手请用 `Set`。 | +| `pld.system.wait(..., cmp=Ge, expected=1)` | 阻塞直到本地 signal slot 达到至少 1——表示所有对端的 notify 均已到达。 | +| `pld.tile.remote_load(...)` | 读取对端 `DistributedTensor` 的远程分片到本地 tile。这是 `pl.tile.load` 的 tile 级别跨 rank 版本。 | +| `pl.add(acc, peer_tile)` | 本地加法循环累加所有对端贡献。循环结束后 `acc` 持有 `sum(inputs[*])`。 | +| `chip_orch`（`@pl.function(type=Orchestration)`） | HOST 通过 `device=r` 派发这个逐设备包装函数，而不是直接派发 `InCore` kernel；它再以不带 `device=` 的方式调用 `reduce_step`。 | +| `inputs[r]` / `outputs[r]` | 下标索引会去掉最前面的 rank 维度，得到 `reduce_step` 声明的二维 `[1, SIZE]` 形状——若改用带显式 shape 的 `pl.slice`，会保留该维度，导致形状不匹配。 | + +## 相关链接 + +- [01-collectives](01-collectives.md) — 内置集合通信及其语义 +- [02-primitives](02-primitives.md) — 集合通信的底层基础 +- [03-execution](03-execution.md) — DistributedWorker 生命周期和生产模式 +- [04-debugging](04-debugging.md) — 常见故障模式和诊断标志 diff --git a/docs/zh/user/distributed/01-collectives.md b/docs/zh/user/distributed/01-collectives.md new file mode 100644 index 0000000000..021b0e7960 --- /dev/null +++ b/docs/zh/user/distributed/01-collectives.md @@ -0,0 +1,154 @@ +# 集合通信 + +本页介绍五种内置集合通信及算法选择。所有集合通信在各 rank 间**同步执行**—— +每个 rank 必须以相同形状的 signal tensor 调用同一集合通信，否则程序会挂起 +或静默数据损坏。 + +## AllReduce + +每个 rank 提交其本地数据；每个 rank 接收求和结果。 + +```python +# Host 编排器——最简形式（编译器合成 signal）。 +data = pld.tensor.allreduce(data, op=pld.ReduceOp.Sum) # mesh 模式，就地 + +# InCore kernel——显式 signal。 +data = pld.tensor.allreduce(data, signal, op=pld.ReduceOp.Sum, mode="mesh") +data = pld.tensor.allreduce(data, signal, op=pld.ReduceOp.Sum, mode="ring") +``` + +### Mesh 模式 + +- 每步 O(N) 远程流量——每个 rank 读取所有对端 +- 每次调用一个全局屏障（AtomicAdd/Ge 在 `[NR, 1]` signal 上） +- 支持 `pl.dynamic("NR")` +- 最适合小消息和低延迟 + +### Ring 模式 + +- 2(P-1) 步：reduce-scatter + allgather +- 每步 O(N/P) 远程流量——每个 rank 读取一个邻居 +- Signal 形状：`[2 × (NR − 1), NR]` +- 要求编译时已知 NR——使用工厂函数模式 +- 最适合大消息（>16 KiB）和高带宽 + +| 方面 | Mesh | Ring | +| ---- | ---- | ---- | +| 每步远程流量 | O(N) | O(N/P) | +| 屏障轮次 | 1 | 2(P-1) | +| Signal 形状 | `[NR, 1]` | `[2 × (NR − 1), NR]` | +| 最适合 | 小消息，低延迟 | 大消息，高带宽 | + +**经验法则：** 默认使用 `mode="mesh"`。当负载超过约 16 KiB 且 mesh 带宽达到平台期时 +切换到 `mode="ring"`。 + +Host 编排器形式（省略 `signal`）是语法糖——编译器合成 `[world_size(), 1]` 的 +signal（仅限 mesh）。 + +### 变更 + +`target: InOut` — 数据既被读取（作为规约输入）又被写入（作为规约结果）。所有 rank +必须传入形状相同的 `target` tensor。 + +### 支持的 ReduceOp + +全部四种——`Sum`、`Max`、`Min`、`Prod`——InCore 组合调用和 Host 内置路径均 +支持。`target` 的 dtype 必须是 `FP16` 或 `FP32`；这是编译期硬性检查，而非 +仅存储位宽的限制。除了本页开头要求的形状相同的 signal tensor 外，所有 +rank 还必须使用相同的 `ReduceOp` 和 `mode`。 + +## Barrier + +跨 rank 屏障——阻塞直到所有 rank 到达。 + +```python +# signal: pld.DistributedTensor[[NR, 1], pl.INT32]，新分配的。 +signal = pld.tensor.barrier(signal) +``` + +在 signal 上使用 `Set(1)` + `Ge(1)`。单次使用；下一次 barrier 前需分配新 +buffer。 + +## Broadcast + +将 root rank 的数据广播到所有 rank。 + +```python +if my_rank == ROOT_RANK: + data = pl.store(local, [0, 0], data) +data = pld.tensor.broadcast(data, signal, root=ROOT_RANK) +``` + +## AllGather + +推式 allgather。 + +```python +stage_buf = pld.alloc_window_buffer(SIZE * pl.FP32.get_byte()) +stage = pld.window(stage_buf, [1, SIZE], dtype=pl.FP32) +stage = pl.store(local_input, [0, 0], stage) + +data_buf = pld.alloc_window_buffer(NR * SIZE * pl.FP32.get_byte()) +data = pld.window(data_buf, [NR, SIZE], dtype=pl.FP32) +sig_buf = pld.alloc_window_buffer(NR * pl.INT32.get_byte()) +sig = pld.window(sig_buf, [NR], dtype=pl.INT32) + +data = pld.tensor.allgather(stage, data, sig) +``` + +`local_data` 和 `target` **必须是不同的** window buffer。 + +## ReduceScatter + +```python +sig_buf = pld.alloc_window_buffer(NR * pl.INT32.get_byte()) +sig = pld.window(sig_buf, [NR], dtype=pl.INT32) + +for j in pl.range(nranks): + data = pl.store(chunk_j, [j, 0], data) +data = pld.tensor.reduce_scatter(data, sig, op=pld.ReduceOp.Sum) +``` + +## AllToAll + +个性化 all-to-all 交换。 + +```python +stage_buf = pld.alloc_window_buffer(NR * SIZE * pl.FP32.get_byte()) +stage = pld.window(stage_buf, [NR, SIZE], dtype=pl.FP32) +for dest in pl.range(nranks): + stage = pl.store(chunk_for_dest, [dest, 0], stage) + +data_buf = pld.alloc_window_buffer(NR * SIZE * pl.FP32.get_byte()) +data = pld.window(data_buf, [NR, SIZE], dtype=pl.FP32) +sig_buf = pld.alloc_window_buffer(NR * pl.INT32.get_byte()) +sig = pld.window(sig_buf, [NR], dtype=pl.INT32) + +data = pld.tensor.all_to_all(stage, data, sig) +``` + +`input` 和 `target` 必须是**不同的** window buffer。 + +## InCore 手写 vs Host 级别内置集合通信 + +PyPTO 有三种方式运行集合通信——根据代码运行的位置以及是否需要 +`mode="ring"` 来选择： + +| 方面 | InCore 手写 | InCore 组合调用 | Host 级别内置 | +| ---- | ----------- | --------------- | ------------- | +| **位置** | `@pl.function(type=InCore)` | `@pl.function(type=InCore)` | `@pl.function(level=HOST, role=Orchestrator)` | +| **实现** | 手写 `notify`/`wait` + `remote_load` 循环 | 直接调用 `pld.tensor.allreduce(data, sig, ...)` | 直接调用 `pld.tensor.allreduce(data, [sig,] ...)` | +| **Lowering** | 自行实现原语 | `LowerCompositeOps` | `LowerHostTensorCollectives` | +| **支持的模式** | 取决于自己的实现 | `mesh` 和 `ring` | 仅 `mesh` | +| **Signal 形状** | 取决于自己的分配 | mesh 为 `[nranks, 1]`（rank 数量可为动态）；ring 为 `[2×(NR−1), NR]`（`NR` 必须是编译期常量） | 一维 `[world_size]` 或二维 `[world_size, 1]`——编译器合成的 signal 为二维 | +| **适用场景** | 学习、自定义协议 | 需要 `ring` 模式，或已身处 InCore kernel 内部 | 日常的 host 编排集合通信 | + +日常 host 编排代码优先使用 Host 级别内置——它们自动处理 signal 分配、 +屏障编排和分块。当需要 `mode="ring"` 时改用 InCore 组合调用，因为 Host +内置路径只 lowering `mesh`。 + +## 相关链接 + +- [00-model](00-model.md) — 快速开始和模型词汇 +- [02-primitives](02-primitives.md) — 集合通信的底层基础 +- [04-debugging](04-debugging.md) — 常见故障模式 diff --git a/docs/zh/user/distributed/02-primitives.md b/docs/zh/user/distributed/02-primitives.md new file mode 100644 index 0000000000..351885acd6 --- /dev/null +++ b/docs/zh/user/distributed/02-primitives.md @@ -0,0 +1,172 @@ +# 原语 + +大多数用户应直接调用 `pld.tensor.*` 集合通信——只有在构建自定义协议时 +才需要这些更底层的原语。 + +## 类型与枚举 + +| 名称 | 取值 | 描述 | +| ---- | ---- | ---- | +| `NotifyOp` | `AtomicAdd`, `Set` | 信号投递模式。`AtomicAdd`：原子递增对端信号槽（多 rank 屏障）。`Set`：覆盖对端信号槽（1:1 握手）。 | +| `WaitCmp` | `Eq`, `Ge` | 等待谓词。`Eq`：等于时解除阻塞。`Ge`：大于等于时解除阻塞。 | +| `ReduceOp` | `Sum`, `Max`, `Min`, `Prod` | 集合通信的规约算子。支持情况按操作而定：`allreduce` 支持全部四种；`reduce_scatter` 仅支持 `Sum`，其余在 deducer 阶段被拒绝。 | +| `AtomicType` | `None_`, `Add` | 远程存储合并模式。`None_`：普通存储。`Add`：原子累加。 | +| `DistributedTensor` | — | 绑定到通信域 window buffer 的 tensor 视图。 | +| `CommCtx` | — | 通信上下文句柄。 | + +## 系统基础设施 (`pld.system.*`) + +| 名称 | 签名 | 描述 | +| ---- | ---- | ---- | +| `world_size` | `() -> Scalar` | **仅限 host。** 分布式执行中的 rank 数量。 | +| `get_comm_ctx` | `(dist_tensor: DT) -> Ctx` | 提升为 `CommCtx` 句柄。 | +| `rank` | `(ctx: Ctx) -> Scalar` | 本地 rank 索引。 | +| `nranks` | `(ctx: Ctx) -> Scalar` | 通信组中 rank 数量。 | +| `notify` | `(target, peer, offsets, value, *, op) -> Call` | 跨 rank 信号投递。仅副作用。 | +| `wait` | `(signal, offsets, expected, *, cmp) -> Call` | 跨 rank 等待。仅副作用。 | + +## Window Buffer 管理 (`pld.tensor.*`) + +`window` 和 `alloc_window_buffer` 属于 `pld.tensor.*`，而非 `pld.system.*`， +尽管它们和上面的基础设施同样底层。 + +| 名称 | 签名 | 描述 | +| ---- | ---- | ---- | +| `window` | `(buf: Ptr, shape, *, dtype) -> DT` | 物化为 `DistributedTensor` 视图。 | +| `alloc_window_buffer` | `(size, *, name="") -> Ptr` | 分配 HCCL window buffer。**size 以字节为单位。** | + +## Notify & Wait：信号握手 + +最底层的同步原语。每个 rank 写入对端信号槽，然后阻塞直到自己的槽被写入。 + +```python +@pl.program +class SignalHandshake: + @pl.function(type=pl.FunctionType.InCore) + def handshake_step( + self, + out: pl.Out[pl.Tensor[[1, 1], pl.INT32]], + signal: pl.InOut[pld.DistributedTensor[[1, 1], pl.INT32]], + peer: pl.Scalar[pl.INT32], + tag: pl.Scalar[pl.INT32], + ) -> pl.Tensor[[1, 1], pl.INT32]: + pld.system.notify( + signal, peer=peer, offsets=[0, 0], + value=tag, op=pld.NotifyOp.Set, + ) + pld.system.wait( + signal=signal, offsets=[0, 0], + expected=1, cmp=pld.WaitCmp.Ge, + ) + received = pl.load(signal, [0, 0], [1, 1]) + out = pl.store(received, [0, 0], out) + return out +``` + +> `wait` 使用 `Ge` 且 `expected=1`，对端的 `tag` **必须 >= 1**。传入 `tag=0` +> 会导致永久挂起。 + +### 选择 NotifyOp 和 WaitCmp + +| 场景 | NotifyOp | WaitCmp | 原因 | +| ---- | -------- | ------- | ---- | +| 1:1 交换（每个槽一个写者） | `Set` | `Eq` 或 `Ge` | 不需要原子递增 | +| N-to-1 屏障（多个写者一个槽） | `AtomicAdd` | `Ge` | 每个写者原子累加，等待总量 | +| 多轮协议 | `AtomicAdd` | `Ge` | 计数跨轮推进 | + +> **Buffer 重用安全：** Signal 使用单调计数器且不会自重置。不要在背靠背集合通信中 +> 重用同一 signal buffer。每次调用分配新 buffer。 + +## Tile 级 RMA (`pld.tile.*`) + +| 名称 | 签名 | 描述 | +| ---- | ---- | ---- | +| `remote_load` | `(target, peer, offsets, shape, valid_shape=None) -> Tile` | 加载对端区域到本地 tile。 | +| `remote_store` | `(src_tile, target, peer, offsets) -> Call` | 写入本地 tile 到对端。 | + +## Put 和 Get + +单边批量传输。 + +### Put（写入对端） + +```python +# dst 必须为 window-bound +pld.tensor.put(dst, peer=1, src=local_chunk, atomic=pld.AtomicType.Add) +``` + +### Get（从对端读取） + +```python +# src 必须为 window-bound +pld.tensor.get(dst, peer=1, src=peer_data) +``` + +### 分块与流水线约束 + +`chunk_rows`/`chunk_cols`（`0` 表示完整范围）会缩小 staging tile，让超过 +片上 staging 预算的传输仍能一次调用完成，自动滑动通过较小的 stage。 + +> **致命陷阱：** `pipeline=True` **要求 `chunk_rows > 0` 且 `chunk_cols > 0` +> 同时成立**——双缓冲的收益只有在传输确实被分块时才存在。若 `pipeline=True` +> 而任一 chunk 维度仍为 `0`，会在派发前抛出 `ValueError`。 + +**动态**传输范围（运行时确定的 `shape`，或 `dst`/`src` 自身维度为动态的 +整片传输）必须由匹配的静态 chunk 限定：动态的最内层维度要求设置 +`chunk_cols`，动态的最外层维度要求设置 `chunk_rows`——staging tile 是静态 +分配的，无法按运行时值确定大小。 + +## 编写自己的集合通信 + +每个内置集合通信都是底层原语的组合。mesh allreduce 的模式为： +stage-in → barrier → remote-accumulate → stage-out。 + +### 隔离的 Barrier + +```python +for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify( + signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd, + ) +for src in pl.range(nranks): + if src != my_rank: + pld.system.wait( + signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge, + ) +``` + +### 远程累加 + +```python +acc = pl.load(data, [0, 0], [1, SIZE]) +for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, SIZE] + ) + acc = pl.add(acc, peer_tile) +``` + +## 2 段 vs 3 段命名空间 + +| 短格式 (`pld.*`) | 完整路径 | +| ---------------- | -------- | +| `pld.world_size()` | `pld.system.world_size()` | +| `pld.rank(ctx)` | `pld.system.rank(ctx)` | +| `pld.nranks(ctx)` | `pld.system.nranks(ctx)` | +| `pld.alloc_window_buffer(...)` | `pld.tensor.alloc_window_buffer(...)` | +| `pld.window(...)` | `pld.tensor.window(...)` | +| `pld.remote_load(...)` | `pld.tile.remote_load(...)` | +| `pld.remote_store(...)` | `pld.tile.remote_store(...)` | + +**无短格式：** `pld.notify(...)`、`pld.wait(...)`、`pld.allreduce(...)` 等—— +这些需要完整的 3 段命名空间。 + +## 相关链接 + +- [01-collectives](01-collectives.md) — 基于这些原语构建的集合通信 +- [03-execution](03-execution.md) — DistributedWorker 生命周期 +- [04-debugging](04-debugging.md) — 常见故障模式 diff --git a/docs/zh/user/distributed/03-execution.md b/docs/zh/user/distributed/03-execution.md new file mode 100644 index 0000000000..ccd8d3fa11 --- /dev/null +++ b/docs/zh/user/distributed/03-execution.md @@ -0,0 +1,107 @@ +# 执行 + +一次性的编译并派发调用足以应付快速测试。生产代码则会将设置成本—— +fork chip 进程、组装 kernel——分摊到可复用的 `DistributedWorker` 上的多次 +派发中。 + +## DistributedWorker + +通过 `compiled.prepare()` 获得。设置（fork、通信引导、kernel 组装）仅执行一次； +分发可执行多次。 + +```python +from pypto.runtime import DistributedWorker + +with DistributedWorker(compiled) as rt: + rt(host_x, host_out) +``` + +### 方法 + +| 方法 | 描述 | +| ---- | ---- | +| `compiled.prepare(config=None, callbacks=None)` | 创建 worker、fork 芯片进程，返回 `DistributedWorker`。 | +| `rt(x, y, z)` | 单次分发。 | +| `rt.run(compiled, x, y, z)` | 多程序分发。 | +| `rt.alloc_tensor(shape, dtype, *, init=None)` | 分配设备驻留 `DeviceTensor`。 | +| `rt.free_tensor(tensor)` | 释放 `DeviceTensor`。 | +| `rt.alloc_stacked_tensor(host_w)` | 沿 dim 0 分片 host_w。 | +| `rt.free_stacked_tensor(stacked)` | 释放所有分片。 | +| `rt.copy_stacked_from(stacked, host_out)` | D2H 读回每个分片。 | +| `rt.close()` | 释放 buffer，关闭芯片 worker。 | + +## DeviceTensor + +设备驻留 buffer，跨分发存活。 + +```python +with compiled.prepare() as rt: + weight = rt.alloc_tensor((1024, 4096), torch.float16, init=host_weight) + rt(x, weight, out) # 无 H2D/D2H +``` + +## One-Shot vs 持久 Worker + +### One-Shot + +```python +from pypto.ir.distributed_compiled_program import DistributedConfig +from pypto import ir + +dc = DistributedConfig(device_ids=[0, 1, 2, 3]) +compiled = ir.compile(HelloAllReduce, platform="a2a3", distributed_config=dc) + +inputs = torch.randn(4, 1, 256) +outputs = torch.zeros_like(inputs) +compiled(inputs, outputs) +``` + +### 持久 Worker + +```python +host_x = torch.zeros((4, 1, 256), dtype=torch.float32).share_memory_() +host_out = torch.zeros_like(host_x).share_memory_() + +with DistributedWorker(compiled) as rt: + for step in steps: + host_x.copy_(next_input(step)) + rt(host_x, host_out) + consume(host_out) +``` + +> **致命陷阱：** 传入 `DistributedWorker` 的 IO buffer 在 `prepare()` 前必须调用 +> `.share_memory_()`。若忘记，运行时在分发时拒绝该 buffer。 + +## CLI 启动 + +分布式程序的启动方式与单设备程序完全一样——直接 `python script.py`。 +`DistributedConfig(device_ids=[...])` 决定 rank 数量和使用的设备；运行时 +会从这一个 Python 进程中为每个设备 fork 出一个 worker 进程，因此不需要 +调用单独的多进程启动器。 + +```bash +python script.py +``` + +## 环境变量 + +### 编译时宏 + +这些是 C 预处理器 `#define` 宏，**不是环境变量**。通过 CMake 标记设置。 + +| 宏 | 默认值 | 效果 | +| -- | ------ | ---- | +| `SIMPLER_HOST_STRACE` | `1`（开） | `benchmark()` 计时标记必需。 | +| `SIMPLER_DFX` | `1`（开） | 设备端分析总开关。 | + +### 运行时环境变量 + +| 变量 | 默认值 | 效果 | +| ---- | ------ | ---- | +| `SIMPLER_DEVICE_STRACE_ENABLE` | 开 | 运行时切换设备域 `[STRACE]` 标记。 | + +## 相关链接 + +- [00-model](00-model.md) — 快速开始和模型词汇 +- [04-debugging](04-debugging.md) — 常见故障模式 +- [性能](../performance/index.md) — 基准测试和调优 diff --git a/docs/zh/user/distributed/04-debugging.md b/docs/zh/user/distributed/04-debugging.md new file mode 100644 index 0000000000..88b329b46a --- /dev/null +++ b/docs/zh/user/distributed/04-debugging.md @@ -0,0 +1,54 @@ +# 调试与陷阱 + +分布式 bug 很少留下本地堆栈——症状出现在某个 rank 上，而原因却在另一个 +rank 上。 + +## 常见故障模式 + +| 症状 | 可能原因 | 修复 | +| ---- | -------- | ---- | +| **所有 rank 挂起** | notify/wait 顺序错误 | 确保 notify 循环在 wait 循环之前。 | +| **静默数据损坏** | `remote_load` offsets 或 shape 不匹配 | 验证 offsets 与对端的 store offsets 对齐。 | +| **Signal cell 永不达到期望值** | 错误 `NotifyOp` | 多参与者屏障用 `AtomicAdd`；1:1 交换用 `Set`。 | +| **编译时形状不匹配** | `NR` 未使用 `pl.dynamic` | 将运行时维度包裹在 `pl.dynamic("NR")` 中。 | +| **派发时抛出 `TypeError`** | IO buffer 在 `prepare()` 前未调用 `.share_memory_()`——fork 出的子进程看不到 fork 之后分配的 buffer | 在 `prepare()` 之前对每个传给 worker 的 host tensor 调用 `.share_memory_()`。 | +| **循环内 allreduce 被拒绝** | Signal 协议无法每轮注入新 buffer | 在循环外每次调用分配新 signal buffer。 | + +## 致命陷阱 + +> **缺少 `.share_memory_()`：** 传入 `DistributedWorker` 的 IO buffer 在 +> `prepare()` 前必须调用 `.share_memory_()`。若忘记调用，运行时会在派发时 +> 抛出 `TypeError`——fork 出的子进程无法访问父进程私有的内存。 +> +> **`alloc_window_buffer` 传入 rank 数量而非字节数：** `size` 参数以**字节** +> 为单位。使用 shape+dtype 重载。 +> +> **`device_ids` 与 `device=` 不匹配：** `DistributedConfig.device_ids` 必须 +> 与编排器中 per-rank 分发使用的 device ID 一致。例如 `device_ids=[0, 1]` +> 却用 `device=r`（`r` 遍历 `range(4)`）派发，会导致未定义行为。 + +## 诊断标志 + +`SIMPLER_HOST_STRACE` 和 `SIMPLER_DFX` 是**编译时 C 预处理器宏**，设置为 shell +环境变量**无效**——它们在编译期就已固定。默认开启。切换它们属于 `simpler` +运行时的构建配置变更，而非一个简单的 `cmake -D...` 缓存变量——具体机制见 +`simpler` 运行时自己的构建文档。 + +运行时环境变量： + +```bash +# 切换设备域 [STRACE] 标记： +SIMPLER_DEVICE_STRACE_ENABLE=0 python script.py +``` + +### 分布式 DFX 入口点 + +- **L2 swimlane：** `RunConfig(enable_l2_swimlane=True)` +- **Scope 统计：** `RunConfig(enable_scope_stats=True)` +- **依赖图：** `RunConfig(enable_dep_gen=True)` + +## 相关链接 + +- [00-model](00-model.md) — 快速开始和模型词汇 +- [02-primitives](02-primitives.md) — 集合通信的底层基础 +- [性能](../performance/index.md) — 基准测试和测量工具 diff --git a/docs/zh/user/distributed/index.md b/docs/zh/user/distributed/index.md new file mode 100644 index 0000000000..e08f11998b --- /dev/null +++ b/docs/zh/user/distributed/index.md @@ -0,0 +1,53 @@ +# 分布式编程 + +PyPTO 的分布式模型建立在**对称内存与信号**之上：每个 rank 在所有对端看到 +相同的 window buffer 地址，通过单边 `put`/`get`/`remote_load` 访问其他 +rank，并通过**信号同步**（`notify`/`wait`）进行协调。**通信域**是共享对称 +window pool 的 rank 子集；整个 world 为默认通信域。 + +编译器 lowering 出的每个 allreduce、broadcast、barrier 都是这些相同原语的 +组合——`pld.tensor.*` 集合通信（`allreduce`、`barrier` 等）是它们的语法糖， +而非另一套独立的库。 + +## L2 vs L3 + +| 层级 | 范围 | API 命名空间 | +| ---- | ---- | ------------ | +| L2 | 单设备（一个 NPU 芯片） | `pl.*` | +| L3 | 跨 rank（多个 NPU 或进程） | `pld.*` | + +> **PyPTO 的 L2/L3 与 simpler 的 L0–L6：** 这两层是 PyPTO 自己的用户侧词汇， +> 并非 simpler 的编号体系。simpler 使用更细的七层体系（L0 核心 → L1 die → +> L2 芯片 → L3 主机 → L4 pod → L5 超节点 → L6 集群）；PyPTO 的 "L2" 对应 +> simpler 的 L0–L2（单芯片内的一切），PyPTO 的 "L3" 对应 simpler 的 L3 +> 及以上（跨芯片的一切）。完整模型见 simpler 的 +> [层级化 Level Runtime](https://hw-native-sys.github.io/simpler/hierarchical-level-runtime/)。 + +分布式章节涵盖 L3。L2 内容见[语言指南](../01-language_guide.md)。 + +## 术语表 + +| 术语 | 定义 | +| ---- | ---- | +| **Rank** | 参与分布式程序的单个进程或芯片。每个 rank 在启动时分配唯一索引。 | +| **Device** | 一个 Ascend NPU 芯片（或 die），由 `device_id` 标识。一个 rank 对应一个 device。 | +| **Node** | 托管一个或多个设备的物理机器。 | +| **Window buffer** | 对称 per-rank HCCL 缓冲区。Rank 通过对等端 `CommContext.windowsIn[peer]`/`windowsOut[peer]` 查看对等端。 | +| **通信域** | 共享对称 window pool 的 rank 子集。默认：整个 world。 | +| **信号** | 跨 rank 同步原语。notify/wait 计数器协调对 window buffer 的访问。 | +| **编排器** | 分配 window buffer 并将 kernel 分发到设备的 HOST 函数。 | +| **InCore kernel** | 在 NPU 上执行的设备端函数。 | + +## 阅读路径 + +1. **[00-model](00-model.md)** — 快速开始优先：运行 2-rank 程序，然后了解模型词汇 +2. **[01-collectives](01-collectives.md)** — AllReduce、barrier、broadcast、allgather、reduce_scatter、all-to-all +3. **[02-primitives](02-primitives.md)** — notify/wait、remote_load/remote_store、put/get、CommCtx +4. **[03-execution](03-execution.md)** — DistributedWorker 生命周期、DeviceTensor、多程序、环境变量 +5. **[04-debugging](04-debugging.md)** — 常见故障模式和诊断标志 + +## 相关链接 + +- [入门指南](../00-getting_started.md) — `ir.compile()`、`CompiledProgram`、`DeviceTensor`、`RunConfig` +- [性能](../performance/index.md) — 分布式程序基准测试与调优 +- [Simpler 运行时](https://hw-native-sys.github.io/simpler/) — 运行时内部机制（调度器、图构建、tensormap） diff --git a/docs/zh/user/index.md b/docs/zh/user/index.md index fd0ad9514b..397874e3f7 100644 --- a/docs/zh/user/index.md +++ b/docs/zh/user/index.md @@ -4,7 +4,7 @@ ## 阅读路径 -按你当前要做的事挑一条。三条路径都假设[安装](01-installation.md)已完成。 +按你当前要做的事挑一条。四条路径都假设[安装](01-installation.md)已完成。 ### 我要写第一个 kernel @@ -24,12 +24,19 @@ ### 我有 kernel，但太慢 -[编程模型 § 内存层次](03-programming-model.md) → -[诊断](../dev/passes/92-diagnostics.md) → -[运行时 DFX](../dev/03-runtime-dfx.md) +[性能](performance/index.md) → +[编程模型 § 内存层次](03-programming-model.md) 在动手测量之前，先看编译产物里的 `report/perf_hints.log` —— 编译器可能已经告诉你了。 -性能专章尚未编写，其内容当前的位置见下表。 +性能专章覆盖单节点和分布式程序完整的测量 → 定位 → 优化 → 验证循环。 + +### 我想跨多个设备运行 + +[分布式编程](distributed/index.md) → [性能 § 分布式](performance/02-distributed.md) + +先让单设备 kernel 跑通 —— 分布式程序是在 `pld.*` 集合通信和 HOST 编排器之上 +组合同样的 `pl.*` kernel。跑通之后，分布式性能页面覆盖 ring 与 mesh 的 +开销取舍以及跨 rank 重叠。 ## 目录 @@ -42,6 +49,8 @@ | [操作参考](02-operation_reference.md) | `pl.*`、`pl.tensor.*`、`pl.tile.*` 三个命名空间的算子全貌 | | [在设备上运行](00-getting_started.md) | 常驻设备张量、显式派发、性能基准、分布式执行 | | [Torch Codegen 调试指南](03-torch_codegen_debug.md) | 从 IR 生成 PyTorch 参考实现，用于定位精度问题 | +| [分布式编程](distributed/index.md) | 跨 rank 程序的对称内存模型、集合通信、底层原语、执行与调试 | +| [性能](performance/index.md) | 测量方法论、单节点与分布式优化技巧、以及实际案例 | ## PyPTO 提供了什么 @@ -54,7 +63,8 @@ | `@pl.jit` 全家族（`.incore`、`.inline`、`.opaque`、`.host`） | [快速上手](02-quickstart.md)、[语言指南](01-language_guide.md) | | 手写 C++ kernel 接入 | [外部 Kernel](../dev/language/01-external-kernels.md) | | 设备常驻张量、显式派发、性能基准 | [在设备上运行](00-getting_started.md) | -| 分布式（多卡）程序与集合通信 | [分布式算子](../dev/distributed_ops.md) | +| 分布式（多卡）程序与集合通信 | [分布式编程](distributed/index.md) | +| 单节点与分布式的性能测量与调优 | [性能](performance/index.md) | | 对照 PyTorch 参考实现做精度定位 | [Torch Codegen 调试指南](03-torch_codegen_debug.md) | | 编译期诊断与性能提示 | [诊断](../dev/passes/92-diagnostics.md) | | 运行时 DFX：swimlane、PMU、依赖图、scope stats | [运行时 DFX](../dev/03-runtime-dfx.md) | @@ -62,14 +72,13 @@ ## 尚未收录的内容 -本手册正在扩展为完整的分章结构 —— 教程、分布式编程、性能优化、精度定位各自成章。 +本手册正在扩展为完整的分章结构 —— 教程、精度定位各自成章。 在这些章节落地之前，相应内容位于[开发者文档](../dev/index.md)： | 主题 | 当前位置 | | ---- | -------- | | 任务与依赖、`manual_scope` / `submit` | [Python IR 语法规范](../dev/language/00-python_syntax.md)、[AutoDeriveTaskDependencies](../dev/passes/36-auto_derive_task_dependencies.md) | | 混合 kernel（AIC + AIV 同一函数） | [LowerAutoVectorSplit](../dev/passes/19-lower_auto_vector_split.md)、[ExpandMixedKernel](../dev/passes/20-expand_mixed_kernel.md)、[TPUSH/TPOP](../reference/pto-isa/01-tpush_tpop.md) | -| 分布式 DSL 与集合通信 | [分布式算子](../dev/distributed_ops.md) | | 性能提示与诊断 | [诊断](../dev/passes/92-diagnostics.md)、[编译性能剖析](../dev/01-compile-profiling.md) | | 运行时 DFX 开关、ring sizing、memory map | [运行时 DFX](../dev/03-runtime-dfx.md)、[逐任务 Ring Sizing](../dev/05-runtime-ring-sizing.md)、[内存图](../dev/07-memory-map.md) | | 外部 C++ kernel | [集成手写 C++ Kernel](../dev/language/01-external-kernels.md) | diff --git a/docs/zh/user/performance/00-methodology.md b/docs/zh/user/performance/00-methodology.md new file mode 100644 index 0000000000..f0597f7b5d --- /dev/null +++ b/docs/zh/user/performance/00-methodology.md @@ -0,0 +1,120 @@ +# 性能方法论 + +单节点和分布式两条轨道共享下面的测量循环——先选择怀疑瓶颈所在层级对应的 +工具，再确认改动确实改变了那个关键数字。 + +## 决策树 + +```text +性能低于预期 +├─ 1. 编译器是否已提示？ → report/perf_hints.log +├─ 2. 哪个端到端分段？ → benchmark span tree +├─ 3. 单个 kernel 内部？ → in-core msprof op-simulator +├─ 4. 资源饱和或浪费？ → memory map HTML, scope stats +└─ 5. 调度被序列化？ → dependency graph, L2 swimlane +``` + +## 快速开始 + +`pypto-lib` golden 框架提供一个环境变量驱动的快速开始方式 +（`PYPTO_BENCH=1 python my_kernel.py`）——该框架的环境变量和默认值 +见 `pypto-lib` 自己的文档，本仓库中未定义它们。本仓库自身的基准测试 +契约是下面的程序化 API。 + +## 程序化 Benchmark API + +```python +from pypto.runtime import benchmark + +compiled = ir.compile(MyProgram) +stats = benchmark(compiled, args=(x, out), rounds=100, warmup=3) + +print(f"median: {stats.device_us_median:.1f} us") +``` + +### API 签名 + +```python +benchmark( + compiled, # CompiledProgram 或 DistributedCompiledProgram + args, + *, + rounds: int = 100, + warmup: int = 3, + platform: str | None = None, # 仅 L2 + device_id: int | None = None, # 仅 L2 + config: RunConfig | None = None, + persistent: bool = False, # 跨分发保留 CommDomain（L3） + reset_persistent_windows: bool | None = None, # 复用前是否清零保留的 window +) -> BenchmarkStats +``` + +### BenchmarkStats 字段 + +| 字段 | 类型 | 描述 | +| ---- | ---- | ---- | +| `device_wall_us` | `list[float]` | 每轮 NPU 端设备墙钟（µs）。 | +| `host_wall_us` | `list[float]` | 每轮 host 端墙钟（µs）。 | +| `rounds` | `int` | 测量轮次（不含预热）。 | +| `warmup` | `int` | 舍弃的前置轮次。 | +| `all_zero_device` | `bool` | 所有采样为 0 时为 True。 | + +### 聚合值 + +| 属性 | 描述 | +| ---- | ---- | +| `stats.device_us_median` | 中位数（µs）。 | +| `stats.device_us_min` | 最小值（µs）。 | +| `stats.device_us_max` | 最大值（µs）。 | +| `stats.device_us_mean` | 算术平均（µs）。 | +| `stats.device_us_stdev` | 标准差（µs）。 | + +### Span Tree 渲染 + +```python +stats.print_tree() +stats.print_mean_tree() +``` + +### 完整示例 + +```python +import torch +import pypto.language as pl +from pypto import ir +from pypto.runtime import benchmark + +ROWS = COLS = 128 + +@pl.program +class MatAdd: + @pl.function(type=pl.FunctionType.InCore) + def add_kernel(self, a, b, c): + ta = pl.load(a, [0, 0], [ROWS, COLS]) + tb = pl.load(b, [0, 0], [ROWS, COLS]) + return pl.store(pl.add(ta, tb), [0, 0], c) + + @pl.function(type=pl.FunctionType.Orchestration) + def chip_orch(self, a, b, c): + return self.add_kernel(a, b, c) + +compiled = ir.compile(MatAdd, platform="a2a3sim") +a = torch.full((ROWS, COLS), 2.0) +b = torch.full((ROWS, COLS), 3.0) +c = torch.zeros((ROWS, COLS)) + +stats = benchmark(compiled, args=(a, b, c), rounds=20, warmup=5, platform="a2a3sim") +print(f"median: {stats.device_us_median:.1f} us") +``` + +## 重要注意事项 + +> **`SIMPLER_HOST_STRACE` 必须编译到运行时中。** +> **`*sim` 平台设备墙钟为 0。** +> **L3 需要共享内存 IO tensor。** + +## 相关链接 + +- [01-single-node](01-single-node.md) — 单节点性能技术 +- [02-distributed](02-distributed.md) — 分布式性能和总线带宽 +- [03-cases](03-cases.md) — 端到端工作示例 diff --git a/docs/zh/user/performance/01-single-node.md b/docs/zh/user/performance/01-single-node.md new file mode 100644 index 0000000000..163bca09f5 --- /dev/null +++ b/docs/zh/user/performance/01-single-node.md @@ -0,0 +1,141 @@ +# 单节点性能 + +下面每项技术都说明**适用场景、成本、启用方式和验证方式**。 + +## 分区与并行 + +### `pl.split(SplitMode)` + +在 `CORE_GROUP` 区域内对跨核数据传输做几何切分——并非独立调用，而是 +传入 `pl.at(..., optimizations=[...])`。 + +- **适用场景：** `CORE_GROUP` 区域的数据需要在其核心间切分 +- **成本：** 需要 split 兼容操作 +- **启用方式：** `pl.at(level=pl.Level.CORE_GROUP, optimizations=[pl.split(pl.SplitMode.UP_DOWN)])`——可选模式为 `NONE`、`UP_DOWN`（高度对半分）、`LEFT_RIGHT`（宽度对半分） +- **验证方式：** 与非 split 基线 benchmark 对比 + +### `pl.split_aiv` + +将计算分配到 AIV 核心。 + +- **适用场景：** 向量密集型工作负载 +- **成本：** 仅 AIV +- **验证方式：** 内存映射中确认 AIV 核心被使用 + +### `pl.spmd(N)` + +SPMD 并行——启动 N 个相同 kernel 副本。 + +- **适用场景：** 天然可并行的工作负载 +- **成本：** N 倍内存占用 +- **启用方式：** `with pl.spmd(n):` 或 `for i in pl.spmd(n):` +- **验证方式：** Benchmark 显示近线性加速 + +### `pl.cluster()` + +协同调度 AIC + AIV 共享物理集群资源。 + +- **适用场景：** 受益于并发 AIC + AIV 执行 +- **成本：** 需要集群兼容的 kernel 对 +- **验证方式：** L2 swimlane 显示并发 AIC/AIV 执行 + +## 流水线与展开 + +### `pl.pipeline` + +跨迭代重叠计算的软件流水线。 + +- **适用场景：** 迭代独立的循环体 +- **成本：** 增加寄存器和 buffer 压力 +- **启用方式：** 用 `pl.pipeline` 包裹循环体 +- **验证方式：** Benchmark 显示减少的每迭代延迟 + +### `pl.unroll` + +完全展开编译时已知的循环。 + +- **适用场景：** 编译时已知的小循环次数 +- **成本：** 更大的二进制文件 +- **验证方式：** 检查编译产物中的展开代码 + +## Matmul 路径 + +### AutoTileMatmulL0 + +自动 tiling pass 选择 L0 matmul tile 大小。 + +- **启用方式：** 默认开启 +- **成本：** 仅有编译时分析开销 +- **验证方式：** IR dump 显示 tiled matmul 维度 + +### `enable_pypto_l0c_double_buffer` + +为 L0C 输出 buffer 启用双缓冲。 + +- **适用场景：** Matmul 密集型工作负载 +- **成本：** 2x L0C buffer 分配 +- **启用方式：** `ir.compile(..., enable_pypto_l0c_double_buffer=True)` +- **验证方式：** Benchmark 显示 sched 时间减少 + +## 内存 + +### `target_memory` + +为 tile 分配选择片上内存空间（`MemorySpace.DDR` 为片外；`.Vec`/`.Mat`/ +`.Left`/`.Right`/`.Acc` 为片上 buffer）。 + +- **适用场景：** 数据放置优化 +- **成本：** 片上空间更快但远小于 DDR +- **启用方式：** `pl.load(a, [0, 0], [rows, cols], target_memory=pl.MemorySpace.Vec)` +- **验证方式：** 内存映射确认分配到指定空间 + +### MemoryReuse vs `memory_planner=PTOAS` + +内存规划策略。 + +- **MemoryReuse：** 默认——不重叠的生命周期重用 buffer +- **PTOAS：** 更积极的规划，可能导致编译时间增加 +- **验证方式：** 内存映射显示分配和重用 + +## 调度 + +### `predicate=` + +在分发点动态跳过任务。 + +- **适用场景：** 条件执行 +- **成本：** 可忽略 +- **验证方式：** 依赖图显示跳过的边 + +### `no_dep` + +对调用点放弃自动依赖推断。 + +- **适用场景：** 表面重叠但实际独立的操作 +- **成本：** 错误使用导致竞态条件 +- **验证方式：** 依赖图确认操作间无边 + +### `manual_scope` + +关闭自动依赖跟踪。 + +- **启用方式：** `with pl.scope(mode=pl.ScopeMode.MANUAL):` +- **成本：** 必须显式声明所有边 +- **验证方式：** 依赖图精确匹配声明的边 + +## 数据驻留 + +### `DeviceTensor` + +在分发之间保持 tensor 驻留在设备上。 + +- **适用场景：** 权重、查找表等可重用数据 +- **成本：** 减少设备内存 +- **启用方式：** `rt.alloc_tensor(shape, dtype, init=host_data)` +- **验证方式：** 第二次分发起不存在 H2D/D2H span + +## 相关链接 + +- [00-methodology](00-methodology.md) — 测量循环和工具 +- [02-distributed](02-distributed.md) — 分布式性能技术 +- [03-cases](03-cases.md) — 端到端工作示例 diff --git a/docs/zh/user/performance/02-distributed.md b/docs/zh/user/performance/02-distributed.md new file mode 100644 index 0000000000..0c67f5b9f1 --- /dev/null +++ b/docs/zh/user/performance/02-distributed.md @@ -0,0 +1,139 @@ +# 分布式性能 + +分布式（L3）程序在单节点性能的基础上，还要处理跨 rank 的问题——总线带宽、 +集合通信选择、启动偏差。 + +## L3 分布式基准测试 + +分布式程序使用相同的 `benchmark()` API，但在计时和准备方面有重要差异。 + +### 准备 + +```python +import torch +from pypto.runtime import benchmark + +host_x = torch.zeros((4, 1, 256), dtype=torch.float32).share_memory_() +host_out = torch.zeros_like(host_x).share_memory_() + +stats = benchmark(compiled, (host_x, host_out), rounds=100, warmup=3) +``` + +### L3 指标 + +| 指标 | `per_round("...")` 键 | 描述 | +| ---- | --------------------- | ---- | +| device | `"device"` | 每轮跨 rank 最大设备墙钟（µs）。 | +| host | `"host"` | 每轮跨 rank 最大 host 墙钟（µs）。 | +| union | `"union"` | 跨 rank host 时间线并集（µs）。捕获重叠和启动偏差。 | + +```python +ranks = stats.per_rank("device") # {pid: [round0_us, ...]} +device = stats.per_round("device") +union = stats.per_round("union") +``` + +## 理解总线带宽 + +总线带宽（`busbw`）是评估集合通信性能的标准指标，源自 nccl-tests 基准测试套件。 + +### 公式 + +```text +algbw = data_size / time +busbw = algbw x correction_factor +``` + +### 修正因子 + +| 操作 | 修正因子 | 说明 | +| ---- | -------- | ---- | +| AllReduce | `2(n-1)/n` | 双向流量（reduce + broadcast），大 n 时趋近 2。 | +| AllGather | `(n-1)/n` | 每个 rank 接收 n-1 个 chunk。 | +| ReduceScatter | `(n-1)/n` | 每个 rank 发送 n-1 个 chunk。 | +| All-to-All | `(n-1)/n` | 个性化交换。 | +| Broadcast | `1` | Root 发送给 n-1 个 rank。 | + +### 计算示例 + +用于说明公式的假设数值（依据 v6 §8.1 第 5 项，真实测量数据会在积累后回填）—— +8 个 rank，1 GB AllReduce 在 100 ms 内完成： + +```text +algbw = 1 GB / 0.100 s = 10 GB/s +busbw = 10 x 2(8-1)/8 = 17.5 GB/s +``` + +## 集合通信算法选择 + +### Mesh vs Ring 权衡 + +| 方面 | Mesh | Ring | +| ---- | ---- | ---- | +| 每步远程流量 | O(N) | O(N/P) | +| 屏障轮次 | 1 | 2(P-1) | +| NR 支持 | `pl.dynamic("NR")` | 编译时静态 | +| 最适合 | 小消息 | 大消息（>16 KiB） | + +### 通信与计算重叠 + +PyPTO 的信号模型通过流水线化的 notify/wait 模式支持通信与计算重叠。 + +### 跨 Rank 启动偏差 + +`union` 指标捕获跨 rank 启动偏差。高 `union` 相对 `device` 表示 rank 同步不佳。 + +## 驻留分片 + +```python +host_weights = torch.randn(4, 1024, 4096).share_memory_() +with compiled.prepare() as rt: + stacked = rt.alloc_stacked_tensor(host_weights) + rt(x, stacked, out) +``` + +## Ring 大小与预热 + +通过 `RunConfig` 调整 ring-task window 和 heap 大小，需要同时传给 +`prepare()` 和每次派发调用： + +```python +from pypto.runtime import RunConfig + +compiled = ir.compile(MyRingProgram, platform="a2a3", distributed_config=dc) +ring_config = RunConfig(ring_task_window=256, ring_heap=1024) +worker = compiled.prepare(config=ring_config) +worker(host_x, host_out, config=ring_config) # 每次派发都传相同的 config +``` + +`prepare(config=...)` 仅用该配置预热运行时 arena 缓存，使**第一次**派发 +跳过约 800ms 的冷启动构建——该配置不会存储在 worker 上。每次派发都必须 +携带自己的 `config=`（使用相同的 `ring_task_window` / `ring_heap`），否则 +arena 会重建（缓存是单槽位的，交替使用不同大小会导致每次切换都重建）。 + +`ring_task_window` 必须是 `>= 4` 的 2 的幂（或长度为 4 的 list/tuple，分别 +设置 ring 0..3）；`ring_heap`（以字节为单位）必须是 `>= 1024` 的 2 的幂。 + +## 共享 Worker + +```python +with compiled_a.prepare(extra_compiled=[compiled_b]) as rt: + rt.run(compiled_a, host_x, host_out) + rt.run(compiled_b, host_x, host_out) +``` + +准备多个程序会让 worker 进入多程序模式，此时 `rt(*args)` 快捷方式含义 +不明确会抛出 `TypeError`——包括主程序在内的每个程序都必须显式通过 +`rt.run(...)` 派发。 + +## 重要注意事项 + +> **`*sim` 平台设备墙钟为 0。** +> **L3 需要共享内存 IO tensor。** +> **DFX 标志通过 L3 管道传递。** + +## 相关链接 + +- [00-methodology](00-methodology.md) — 测量循环和工具 +- [01-single-node](01-single-node.md) — 单节点性能技术 +- [03-cases](03-cases.md) — 端到端工作示例 diff --git a/docs/zh/user/performance/03-cases.md b/docs/zh/user/performance/03-cases.md new file mode 100644 index 0000000000..a993049ceb --- /dev/null +++ b/docs/zh/user/performance/03-cases.md @@ -0,0 +1,22 @@ +# 性能案例 + +下面每个案例都遵循相同的模式：**基线 → 调查 → 变更 → 效果 → 验证**。 + +> **状态：** 性能案例已规划但尚未编写。代表性工作负载的真实测量数据尚 +> 未积累。用户手册计划 ([USER_MANUAL_PLAN_EN §8.1 item 5](https://github.com/hw-native-sys/pypto/issues/2120)) +> 目标是在第一版中提供方法论和相对趋势，待数据积累后回填测量值。 +> +> 单节点性能技术（[01-single-node](01-single-node.md)）和分布式性能指南 +> （[02-distributed](02-distributed.md)）现已可用。 +> +> 计划案例： +> +> - **单节点：** Tile 维度粒度、自动 Tiling 诊断、假任务依赖、Ring sizing +> 竞技场重建。 +> - **分布式：** Mesh-to-ring 集合通信转换、驻留分片 vs 每分发 H2D。 + +## 相关链接 + +- [00-methodology](00-methodology.md) — 测量循环和工具 +- [01-single-node](01-single-node.md) — 单节点性能技术 +- [02-distributed](02-distributed.md) — 分布式性能和总线带宽 diff --git a/docs/zh/user/performance/index.md b/docs/zh/user/performance/index.md new file mode 100644 index 0000000000..08e88a293f --- /dev/null +++ b/docs/zh/user/performance/index.md @@ -0,0 +1,30 @@ +# 性能 + +PyPTO 的性能工作遵循 **测量 → 定位 → 优化 → 验证** 循环，分为两个轨道： + +| 轨道 | 范围 | 页面 | +| ---- | ---- | ---- | +| 共享方法论 | 工具和测量循环（两个轨道通用） | [00-methodology](00-methodology.md) | +| 单节点 | Kernel、tile、流水线、matmul、内存、调度 | [01-single-node](01-single-node.md) | +| 分布式 | 集合通信成本、ring vs mesh、跨 rank 偏差、总线带宽 | [02-distributed](02-distributed.md) | +| 案例 | 端到端工作示例 | [03-cases](03-cases.md) | + +> **前置知识：** 分布式轨道需要 [分布式编程](../distributed/00-model.md)； +> kernel 编写基础见[入门指南](../00-getting_started.md)。 + +## 工具矩阵 + +| 工具 | 观测对象 | 入口点 | +| ---- | -------- | ------ | +| 编译时性能提示 | 代码模式 | `report/perf_hints.log` | +| 基准测试 span tree | 端到端分段 | `pypto.runtime.benchmark` → `stats.print_mean_tree(spread=...)` | +| In-core msprof | 每个 kernel 的周期数 | op-simulator + Insight traces | +| 内存映射 | 片上缓冲区 | `pypto.tools.memory_map` → HTML | +| Scope 统计 | 运行时水位 | `RunConfig(enable_scope_stats=True)` | +| L2 swimlane / PMU / dep gen | 任务调度 | `RunConfig(enable_l2_swimlane / enable_pmu / enable_dep_gen)` | + +## 相关链接 + +- [分布式](../distributed/index.md) — 编写分布式程序 +- [入门指南](../00-getting_started.md) — `ir.compile()` 和 `RunConfig` +- [Simpler 运行时](https://hw-native-sys.github.io/simpler/) — 调度器内部机制 diff --git a/mkdocs.yml b/mkdocs.yml index 33fd5d3db9..68db6c9883 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -109,6 +109,8 @@ plugins: Code Generation: 代码生成 Backend: 后端 Debug: 调试 + Distributed: 分布式 + Performance: 性能 # Renders `pl.*` / `pypto.runtime.*` API pages from source docstrings. No # generated pages are wired into the nav yet -- that lands with the Reference # chapter (see issue #2120); the handler is configured here so the contract @@ -170,6 +172,19 @@ nav: - en/user/02-operation_reference.md - en/user/00-getting_started.md - en/user/03-torch_codegen_debug.md + - Distributed: + - en/user/distributed/index.md + - en/user/distributed/00-model.md + - en/user/distributed/01-collectives.md + - en/user/distributed/02-primitives.md + - en/user/distributed/03-execution.md + - en/user/distributed/04-debugging.md + - Performance: + - en/user/performance/index.md + - en/user/performance/00-methodology.md + - en/user/performance/01-single-node.md + - en/user/performance/02-distributed.md + - en/user/performance/03-cases.md - Reference: - en/reference/index.md - PTO ISA: diff --git a/python/pypto/ir/distributed_compiled_program.py b/python/pypto/ir/distributed_compiled_program.py index 37795552c9..29f944d290 100644 --- a/python/pypto/ir/distributed_compiled_program.py +++ b/python/pypto/ir/distributed_compiled_program.py @@ -58,8 +58,17 @@ class DistributedConfig: """Configuration for L3 distributed execution. - ``aicpu_thread_num=4`` matches the ``tensormap_and_ringbuffer`` runtime's - 3-scheduler-plus-1-dispatcher layout. + Fields: + device_ids: List of NPU device indices to use. Defaults to ``[0]`` + (single-card). Pass ``[0, 1, 2, 3]`` for a 4-card ring. Must be + a non-empty list of distinct ints. + num_sub_workers: Number of sub-workers per chip process. ``0`` (default) + means one sub-worker per chip process. Increase for multi-slice or + internal pipelining scenarios. + runtime: Simpler runtime flavour. ``"tensormap_and_ringbuffer"`` (default) + enables the tensor-map helpers and the ring-buffer DMA driver. + aicpu_thread_num: Number of aiCPU threads allocated to the simpler + runtime (3 schedulers + 1 dispatcher = 4 by default). Must be ≥ 1. """ device_ids: list[int] = field(default_factory=lambda: [0]) @@ -83,6 +92,18 @@ class DistributedCompiledProgram: **Return** (program has a return value):: c = compiled(a, b) + + **One-shot dispatch**:: + + compiled = ir.compile(MyProgram, platform="a2a3", distributed_config=dc) + compiled(inputs, outputs) # blocks until all ranks finish + + **Persistent dispatch** (repeated launches without re-registering):: + + with DistributedWorker(compiled) as rt: + for step in steps: + rt(host_x, host_out) + # rt.close() on exit — releases buffers and shuts down workers """ __test__ = False diff --git a/python/pypto/language/distributed/op/system_ops.py b/python/pypto/language/distributed/op/system_ops.py index 0bfb16b94c..93f99e3ec7 100644 --- a/python/pypto/language/distributed/op/system_ops.py +++ b/python/pypto/language/distributed/op/system_ops.py @@ -53,6 +53,18 @@ def world_size() -> Scalar: return wrapping lets call sites compose naturally with Python operators (``pld.world_size() * 4``, ``pl.range(pld.world_size())``), which the parser's ``invoke_dsl`` unwraps back to the underlying Call. + + .. warning:: + + This function is callable **only** inside HOST-level orchestration + (``level=pl.Level.HOST, role=pl.Role.Orchestrator``). Calling it + inside InCore (``type=pl.FunctionType.InCore``) raises a parser error. + + .. seealso:: + + :func:`rank` and :func:`nranks` — the per-rank equivalents for InCore + kernels, called on a :class:`CommCtx` obtained via + :func:`get_comm_ctx`. """ return Scalar(expr=_ir_system.world_size()) @@ -128,6 +140,14 @@ def notify( parser; ``op`` stays keyword-only because it lowers to an IR attr (printed as ``op=``), mirroring ``pld.tensor.window``'s ``dtype``. + .. note:: + + The first parameter is named ``target`` for legacy reasons (inherited + from early signal protocol drafts). The companion :func:`wait` names + the same logical operand ``signal``. Both refer to the signal + :class:`DistributedTensor` — always use positional args: + ``notify(signal, peer=..., ...)``. + Args: target: Window-bound :class:`pld.DistributedTensor` signal matrix. The C++ verifier refuses a plain :class:`pl.Tensor`. diff --git a/python/pypto/language/distributed/op/tensor_ops.py b/python/pypto/language/distributed/op/tensor_ops.py index 3aca862433..1887b04fbc 100644 --- a/python/pypto/language/distributed/op/tensor_ops.py +++ b/python/pypto/language/distributed/op/tensor_ops.py @@ -193,7 +193,7 @@ def alloc_window_buffer( # type: ignore[no-redef] * **Shape+dtype convenience overload:** ``alloc_window_buffer(shape, *, dtype=..., name=...)`` — ``shape`` is a list / tuple of per-rank dimensions. The byte size is computed - automatically as ``product(shape) × dtype.get_byte()`` and the call + automatically as ``product(shape) x dtype.get_byte()`` and the call normalizes to the canonical byte form. ``dtype`` is required when ``shape`` is a sequence and rejected otherwise. @@ -220,6 +220,17 @@ def alloc_window_buffer( # type: ignore[no-redef] through :func:`window` materialises a :class:`DistributedTensor` view. + .. note:: + + This function is callable only inside HOST-level orchestration + (``level=pl.Level.HOST, role=pl.Role.Orchestrator``). Calling it + inside InCore (``type=pl.FunctionType.InCore``) raises a parser error. + + .. seealso:: + + :func:`window` for creating typed DistributedTensor views over an + allocated buffer. + Raises: ValueError: If ``name`` is empty (the parser must have injected it). ValueError: If ``shape`` is a sequence but ``dtype`` is not provided. @@ -274,6 +285,11 @@ def window( Returns: A :class:`DistributedTensor` view of the given shape and dtype. + + .. seealso:: + + :func:`alloc_window_buffer` for the two-phase ``alloc → window`` pattern. + """ buf_expr = _unwrap(buf) if not isinstance(buf_expr, Expr): @@ -519,10 +535,33 @@ def allreduce( pub = pld.tensor.allreduce(pub, sig, op=pld.ReduceOp.Sum) pub = pld.tensor.allreduce(pub, sig, op=pld.ReduceOp.Sum, mode="ring") + # Mesh mode on the host orchestrator can omit the signal argument: + pub = pld.tensor.allreduce(pub, op=pld.ReduceOp.Sum) + + **Signal shape:** host builtins accept rank-1 ``[world_size]`` or rank-2 + ``[world_size, 1]`` (the compiler-synthesized signal is rank-2). InCore + composites take rank-2 ``[nranks, 1]`` for mesh -- the rank count may be + dynamic -- and ``[2*(NR-1), NR]`` for ring, where ``NR`` must be a + compile-time constant. Both are single-shot per call. + + **Do not reuse the same signal buffer for a back-to-back allreduce** + — allocate a fresh signal buffer (``alloc_window_buffer`` + ``window``) + for each allreduce call. All allreduce calls in ``for`` and ``while`` + loops are rejected because the current signal protocol cannot provide a + fresh signal for every dynamic iteration. A self-resetting variant is + blocked on a runtime fix — PTOAS issue #797. + + .. seealso:: + + :func:`alloc_window_buffer` and :func:`window` for buffer allocation + and view creation. + + .. rubric:: Implementation Notes + LowerCompositeOps expands the explicit-signal InCore form into either: (a) a notify/wait ready barrier followed by UB-sized remote_load+accumulate/store chunks for ``mode="mesh"`` (default); or - (b) the NCCL-style 2(P−1)-step + (b) the NCCL-style 2(P-1)-step chunked reduce-scatter + allgather ring schedule for ``mode="ring"``. In both modes the kernel sees only the lowered primitives. Host-orchestrator code can omit ``signal`` @@ -531,26 +570,11 @@ def allreduce( (mesh mode only — ring mode on the HOST rail is delivered by a subsequent host builtin). - Mesh signal shape is ``[NR, 1]``; ring signal shape is - ``[2 * (NR − 1), NR]`` (one row per ring round). Both are single-shot - per call. - Fully-valid packed mesh targets are viewed as one logical 1D stream and processed in chunks of at most 16 KiB. For statically known smaller targets, the physical chunk width shrinks to the smallest 32-byte-aligned width that covers the target. The final chunk uses ``valid_shape`` so arbitrary element - counts do not read or store past the end. Mesh lowering - also preserves a packed ND target ``TensorView.valid_shape`` when - its valid box can be represented by collapsing leading dimensions to one - 2D rectangle and fits within one 16-KiB chunk, and reduces only that - rectangle with the established single-rectangle path. Oversized partial - rectangles, strided targets, DN partial views, and - non-representable partial boxes are rejected explicitly. - Any symbolic target or partial-valid extent that survives lowering must be - runtime-bound by a kernel scalar, loop variable, or physical tensor-shape - parameter; a symbol that appears only in type metadata is rejected during - PTO codegen. A fully dynamic physical target dimension is bound from that - tensor parameter. + counts do not read or store past the end. **Mesh barrier protocol:** ``AtomicAdd(1) → WaitGe(1)`` is the ready wave. Every reduced chunk then performs ``AtomicAdd(1)`` and waits for the @@ -633,14 +657,17 @@ def barrier( """Cross-rank barrier synchronisation. Blocks until all ranks in the comm group have reached the barrier. - Uses a window-bound INT32 ``signal`` matrix for cross-rank - synchronisation (one slot per rank). LowerCompositeOps expands this + Uses a window-bound INT32 ``signal`` tensor for cross-rank + synchronisation. LowerCompositeOps expands this into a notify-all / wait-all sequence. .. code-block:: python sig = pld.tensor.barrier(sig) + **Signal shape:** host builtins require rank-1 ``[world_size]``. InCore + composites take rank-2 ``[nranks, 1]`` -- the rank count may be dynamic. + **Signal buffer is single-shot per call.** The lowering uses ``Set(1)`` + ``Ge(1)`` — cells go from 0 to 1. Do not reuse the same signal buffer for back-to-back barriers without reallocation. @@ -668,9 +695,12 @@ def broadcast( """Broadcast root rank's data to all ranks. After this call returns, every rank's slice of ``target`` holds - root's data. Uses a window-bound INT32 ``signal`` matrix for the + root's data. Uses a window-bound INT32 ``signal`` tensor for the cross-rank barrier. + **Signal shape:** host builtins require rank-1 ``[world_size]``. InCore + composites take rank-2 ``[nranks, 1]`` -- the rank count may be dynamic. + .. code-block:: python # Root stages data; non-root skip. @@ -716,6 +746,10 @@ def allgather( plain :class:`pl.Tensor` ``[1, SIZE]`` — both are accepted. HOST vs InCore is a function-context property resolved by the lowering passes. + **Signal shape:** host builtins accept rank-1 ``[world_size]`` or rank-2 + ``[world_size, 1]``. InCore composites take rank-2 ``[nranks, 1]`` -- the + rank count may be dynamic. + Args: local_data: This rank's single chunk — ``[1, SIZE]`` :class:`pl.Tensor` (InCore) or ``[1, SIZE]`` :class:`pld.DistributedTensor` staging @@ -752,6 +786,9 @@ def reduce_scatter( data = pld.tensor.reduce_scatter(data, sig, op=pld.ReduceOp.Sum) # data[my_rank, 0:SIZE] now holds this rank's reduced chunk. + **Signal shape:** host builtins require rank-1 ``[world_size]``. InCore + composites take rank-2 ``[nranks, 1]`` -- the rank count may be dynamic. + Args: target: Window-bound :class:`pld.DistributedTensor` of shape [NR, SIZE]. Each rank stages all NR chunks, one per row. @@ -790,6 +827,10 @@ def all_to_all( by an earlier InCore step) rather than the InCore composite's plain :class:`pl.Tensor` — both are accepted. + **Signal shape:** host builtins accept rank-1 ``[world_size]`` or rank-2 + ``[world_size, 1]``. InCore composites take rank-2 ``[nranks, 1]`` -- the + rank count may be dynamic. + Args: input: [NR, SIZE] Tensor or DistributedTensor with per-destination chunks, distinct from ``target``. ``input[dest, :]`` is the @@ -797,7 +838,9 @@ def all_to_all( target: :class:`pld.DistributedTensor` [NR, SIZE] window that receives the result in-place. After the call, ``target[src, :]`` holds the chunk received from rank ``src``. - signal: :class:`pld.DistributedTensor` [NR, 1] INT32 barrier. + signal: Window-bound INT32 :class:`pld.DistributedTensor` barrier + tensor. Rank-1 ``[world_size]`` or rank-2 ``[world_size, 1]`` + for host builtins; rank-2 ``[nranks, 1]`` for InCore composites. Returns: The ``target`` :class:`pld.DistributedTensor` (window-as-result). diff --git a/python/pypto/language/distributed/op/tile_ops.py b/python/pypto/language/distributed/op/tile_ops.py index 0cca948fe8..92b42b52a8 100644 --- a/python/pypto/language/distributed/op/tile_ops.py +++ b/python/pypto/language/distributed/op/tile_ops.py @@ -53,6 +53,13 @@ def remote_load( is a *remote* slice of a window-bound :class:`pld.DistributedTensor`. Address translation happens at codegen via ``CommRemoteOffset`` + addptr + make_tensor_view. + .. code-block:: python + + # Barrier example — after notify/wait, all windows are visible. + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, SIZE] + ) + All arguments are positional-or-keyword (mirroring :func:`pl.tile.load`), so the printed IR — which emits them positionally — round-trips through the parser. Callers may still pass them by keyword for readability. @@ -136,6 +143,11 @@ def remote_store( :class:`pld.DistributedTensor`. Address translation happens at codegen via ``CommRemoteOffset`` + addptr + make_tensor_view. + .. code-block:: python + + # Write a computed tile into peer rank 1's signal cell. + pld.tile.remote_store(tile, signal, peer=1, offsets=[0, 0]) + All arguments are positional-or-keyword (mirroring :func:`pl.tile.store`), so the printed IR — which emits them positionally — round-trips through the parser. Callers may still pass them by keyword for readability. diff --git a/python/pypto/language/dsl_api.py b/python/pypto/language/dsl_api.py index 16cfa18aad..13e5ce203a 100644 --- a/python/pypto/language/dsl_api.py +++ b/python/pypto/language/dsl_api.py @@ -712,6 +712,19 @@ def spmd( ``range(n)``. Loop start is fixed at 0 and step at 1; each block gets an index ``i`` in ``[0, core_num)``. + .. rubric:: Three usage forms at a glance + + ============================== =================================================== + Form Description + ============================== =================================================== + ``with pl.spmd(n):`` Dispatch or inline block (no captured TaskId). + ``for i in pl.spmd(n):`` Loop-style; ``i`` = per-block index, body + is auto-outlined to InCore. + ``with pl.spmd(n) as tid:`` Same body shapes as form 1, plus captured + producer TaskId in ``tid``. Optionally pass + ``deps=[...]`` after ``n``. + ============================== =================================================== + Usage forms: 1. ``with pl.spmd(n):`` — body is either a *dispatch* body calling a diff --git a/python/pypto/pypto_core/ir.pyi b/python/pypto/pypto_core/ir.pyi index 9419767a9d..e54d27a759 100644 --- a/python/pypto/pypto_core/ir.pyi +++ b/python/pypto/pypto_core/ir.pyi @@ -2252,7 +2252,15 @@ class ReduceOp(enum.IntEnum): """Reduction operator for collective reductions — ``pld.tensor.allreduce`` and friends. Stored as ``int`` in op kwargs; the C++ deducer validates the int falls - within this enum's range. + within this enum's range. Support is per-operation, not uniform across + the enum: + + .. note:: + + **Per-operation support:** ``pld.tensor.allreduce`` (both the InCore + composite and the HOST builtin) accepts all four values. Other + reducing collectives are narrower — ``pld.tensor.reduce_scatter`` + accepts only :attr:`Sum` and rejects the rest at the deducer. """ Sum = 0 diff --git a/python/pypto/runtime/bench.py b/python/pypto/runtime/bench.py index 849b7826f2..b2dc42c216 100644 --- a/python/pypto/runtime/bench.py +++ b/python/pypto/runtime/bench.py @@ -303,6 +303,23 @@ def _span_names() -> dict[str, str]: class BenchmarkStats: """Aggregated per-launch timing from :func:`benchmark`. + .. rubric:: Quick-reference + + ============================== ========================================================= + Accessor Description + ============================== ========================================================= + ``stats.device_us_median`` Median device wall (µs). + ``stats.device_us_min`` Minimum device wall (µs). + ``stats.device_us_max`` Maximum device wall (µs). + ``stats.device_us_mean`` Arithmetic mean device wall (µs). + ``stats.device_us_stdev`` Std-dev of device wall (µs). + ``stats.all_zero_device`` True if every sample is 0 (e.g. sim builds). + ``stats.samples`` Alias for ``device_wall_us`` (the raw list). + ``stats.per_round("device")`` List[float]: device wall per round. + ``stats.per_rank("device")`` Dict[int, List[float]]: per-rank breakdown (L3). + ``stats.print_tree()`` Render per-dispatch span tree to stdout. + ============================== ========================================================= + The min / median / mean / max / stdev helpers operate on ``device_wall_us`` — the on-NPU metric. ``host_wall_us`` samples are kept for context, but they include per-launch arg coercion + H2D and so are not