diff --git a/docs/en/user/00-getting_started.md b/docs/en/user/00-getting_started.md index 256b51868..57a15f80a 100644 --- a/docs/en/user/00-getting_started.md +++ b/docs/en/user/00-getting_started.md @@ -264,6 +264,63 @@ per-rank / `union` views are empty. ### Distributed (L3+) programs +The complete distributed programming model — from `alloc_window_buffer` through +collectives like `allreduce`, `barrier`, and `broadcast` — is covered in the +[Distributed Programming](distributed/00-model.md). Here is the mesh allreduce +Hello World, shown as the InCore kernel (execution plane); a fully runnable +program also needs the host orchestrator, `ir.compile`, and distributed worker +setup — see the guide above: + +```python +import pypto.language as pl +import pypto.language.distributed as pld + +NR = pl.dynamic("NR") + +@pl.program +class HelloAllReduce: + @pl.function(type=pl.FunctionType.InCore) + def reduce_step( + self, + inp: pl.Tensor[[1, 256], pl.FP32], + out: pl.Out[pl.Tensor[[1, 256], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, 256], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, 256], pl.FP32]: + ctx = pld.get_comm_ctx(data) + my_rank = pld.rank(ctx) + nranks = pld.nranks(ctx) + + # 1. Stage-in: copy local input into this rank's window slice. + data = pl.store(pl.load(inp, [0, 0], [1, 256]), [0, 0], data) + + # 2. Barrier: notify every peer, then wait on every peer. + for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify(signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd) + for src in pl.range(nranks): + if src != my_rank: + pld.system.wait(signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge) + + # 3. Compute: load own slice, remote-load every peer, accumulate. + acc = pl.load(data, [0, 0], [1, 256]) + for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, 256]) + acc = pl.add(acc, peer_tile) + + # 4. Stage-out: store accumulator to output. + out = pl.store(acc, [0, 0], out) + return out +``` + +The guide includes a **line-by-line walkthrough, ring allreduce trade-offs, +notify/wait handshake patterns, and a debugging table**. The full chapter is +at [distributed/index.md](distributed/index.md). + L3+ distributed programs returned by `ir.compile` (a `DistributedCompiledProgram`) accept `DeviceTensor` arguments the same way as `CompiledProgram`: pass a worker-resident buffer in place of a `torch.Tensor` and the runtime skips H2D/D2H diff --git a/docs/en/user/distributed/01-collectives.md b/docs/en/user/distributed/01-collectives.md index bd10c1dcd..de2024dcb 100644 --- a/docs/en/user/distributed/01-collectives.md +++ b/docs/en/user/distributed/01-collectives.md @@ -67,11 +67,12 @@ the reduced result). All ranks must pass identically shaped `target` tensors. ### Supported ReduceOp -All four — `Sum`, `Max`, `Min`, `Prod` — on both the InCore composite and the -HOST builtin path. `target`'s dtype must be `FP16` or `FP32`; this is a hard -compile-time check, not just a storage-width constraint. Every rank must -agree on the same `ReduceOp` and `mode`, on top of the identically-shaped -signal tensors required of every collective. +All four — `Sum`, `Max`, `Min`, `Prod` — on the InCore composite and the HOST +builtin mesh path. The HOST builtin ring path (`builtin.tensor.allreduce_ring`) +is narrower: `Sum` only, with a 4-byte `FP32` target (a compile-time check). +`mesh` targets must be `FP16` or `FP32`; the ring path is `FP32`-only. Every +rank must agree on the same `ReduceOp` and `mode`, on top of the +identically-shaped signal tensors required of every collective. ## Barrier @@ -175,17 +176,19 @@ runs and whether you need `mode="ring"`: | **Where** | `@pl.jit.incore` | `@pl.jit.incore` | `@pl.jit.host` | | **How** | Manual `notify`/`wait` + `remote_load` loops | `pld.tensor.allreduce(data, sig, ...)` called directly | `pld.tensor.allreduce(data, [sig,] ...)` called directly | | **Lowering** | You write the primitives | `LowerCompositeOps` | `LowerHostTensorCollectives` | -| **Modes** | Whatever you implement | `mesh` and `ring` | `mesh` only | -| **Signal shape** | Whatever you allocate | `[nranks, 1]` for mesh (rank count may be dynamic); `[2×(NR−1), NR]` for ring (`NR` must be a compile-time constant) | Rank-1 `[world_size]` or rank-2 `[world_size, 1]` — the compiler-synthesized signal is rank-2 | -| **When** | Learning, custom protocols | Need `ring` mode, or already inside an InCore kernel | Day-to-day host-orchestrated collectives | +| **Modes** | Whatever you implement | `mesh` and `ring` | `mesh` and `ring` (ring: `Sum` + `FP32` only) | +| **Signal shape** | Whatever you allocate | `[nranks, 1]` for mesh (rank count may be dynamic); `[2×(NR−1), NR]` for ring (`NR` must be a compile-time constant) | Mesh: rank-1 `[world_size]` or rank-2 `[world_size, 1]` (the compiler-synthesized signal is rank-2). Ring: `[2*(NR−1)+1, NR]` | +| **When** | Learning, custom protocols | Ring with non-`Sum`/non-`FP32`, or already inside an InCore kernel | Day-to-day host-orchestrated collectives | Prefer HOST builtins for day-to-day host-orchestrated code — they handle barrier orchestration and chunking automatically. Only `allreduce` can also omit the signal argument (the compiler synthesizes one outside loops); the other five collectives (`barrier`, `broadcast`, `allgather`, `reduce_scatter`, `all_to_all`) always take an explicit, caller-allocated -signal. Reach for the InCore composite specifically when you need -`mode="ring"`, since the HOST builtin path only lowers `mesh`. +signal. Both the InCore composite and the HOST builtin lower `mode="ring"`; +reach for the InCore composite when you need ring with a `ReduceOp` other than +`Sum` or a non-`FP32` dtype, since the HOST builtin ring path is `Sum`+`FP32` +only. ## Runnable Examples @@ -195,7 +198,7 @@ Every collective above has a runnable counterpart under | Collective | InCore hand-rolled | InCore composite | HOST builtin | | ---------- | ------------------ | ---------------- | ------------ | | allreduce | `collectives/test_l3_allreduce.py` | `collectives/test_l3_tensor_allreduce_intrinsic.py` | `test_l3_host_tensor_allreduce.py` | -| allreduce (ring) | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | n/a (mesh only) | +| allreduce (ring) | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | `test_l3_host_tensor_allreduce_ring.py` | | barrier | — | `collectives/test_l3_tensor_barrier_intrinsic.py` | `test_l3_host_tensor_barrier.py` | | broadcast | `collectives/test_l3_broadcast.py` | `collectives/test_l3_tensor_broadcast_intrinsic.py` | `test_l3_host_tensor_broadcast.py` | | allgather | `collectives/test_l3_allgather.py` | `collectives/test_l3_tensor_allgather_intrinsic.py` | `test_l3_host_tensor_allgather.py` | diff --git a/docs/en/user/index.md b/docs/en/user/index.md index c2966677b..a3b408f3c 100644 --- a/docs/en/user/index.md +++ b/docs/en/user/index.md @@ -4,7 +4,7 @@ How to write, compile, run, and debug PyPTO programs. ## Reading paths -Pick the path that matches what you are trying to do. All three assume +Pick the path that matches what you are trying to do. All four assume [Installation](01-installation.md) is done. ### I want to write my first kernel @@ -36,6 +36,15 @@ Check `report/perf_hints.log` from your compile output before measuring anything compiler may already have told you. The dedicated performance chapter is not written yet; see the table below for where its material currently lives. +### I want to run across multiple devices + +[Distributed Programming](distributed/index.md) + +Get a single-device kernel running first — distributed programs compose the same +`pl.*` kernels behind `pld.*` collectives and a HOST orchestrator. Once it runs +correctly, the distributed chapter covers ring vs. mesh trade-offs and +cross-rank overlap. + ## Contents | Page | What it covers | @@ -48,6 +57,7 @@ see the table below for where its material currently lives. | [Compiling a Program](01-language_guide.md) | `ir.compile()` and `JITFunction.compile()`, and inspecting the result | | [Running on Device](00-getting_started.md) | Resident device tensors, explicit dispatch, benchmarking, distributed execution | | [Torch Codegen Debug Guide](03-torch_codegen_debug.md) | Generating a PyTorch reference implementation from the IR to isolate accuracy problems | +| [Distributed Programming](distributed/index.md) | Symmetric-memory model, collectives, primitives, execution, and debugging for cross-rank programs | ## What PyPTO gives you @@ -60,7 +70,7 @@ see the table below for where its material currently lives. | The full `@pl.jit` family (`.incore`, `.inline`, `.opaque`, `.host`) | [Quickstart](02-quickstart.md), [Functions and Programs](language/01-functions.md) | | Hand-written C++ kernel integration | [External Kernels](../dev/language/01-external-kernels.md) | | Device-resident tensors, explicit dispatch, benchmarking | [Running on Device](00-getting_started.md) | -| Distributed (multi-card) programs and collectives | [Distributed Operators](../dev/distributed_ops.md) | +| Distributed (multi-card) programs and collectives | [Distributed Programming](distributed/index.md) | | Accuracy debugging against a PyTorch reference | [Torch Codegen Debug Guide](03-torch_codegen_debug.md) | | Compile-time diagnostics and performance hints | [Diagnostics](../dev/passes/92-diagnostics.md) | | Runtime DFX: swimlane, PMU, dependency graph, scope stats | [Runtime DFX](../dev/03-runtime-dfx.md) | @@ -68,15 +78,14 @@ see the table below for where its material currently lives. ## What is not here yet -This manual is being expanded into a full chaptered structure — tutorials, distributed -programming, performance optimization, and accuracy debugging each get their own -chapter. Until those land, the corresponding material lives in the -[developer documentation](../dev/index.md): +This manual is being expanded into a full chaptered structure — tutorials, +performance optimization, and accuracy debugging each get their own chapter. +Until those land, the +corresponding material lives in the [developer documentation](../dev/index.md): | Topic | Current location | | ----- | ---------------- | | Mixed kernels (AIC + AIV in one function) | [LowerAutoVectorSplit](../dev/passes/20-lower_auto_vector_split.md), [ExpandMixedKernel](../dev/passes/21-expand_mixed_kernel.md), [TPUSH/TPOP](../reference/pto-isa/01-tpush_tpop.md) | -| Distributed DSL and collectives | [Distributed Operators](../dev/distributed_ops.md) | | Performance hints and diagnostics | [Diagnostics](../dev/passes/92-diagnostics.md), [Compile Profiling](../dev/01-compile-profiling.md) | | Runtime DFX flags, ring sizing, memory map | [Runtime DFX](../dev/03-runtime-dfx.md), [Per-Task Ring Sizing](../dev/05-runtime-ring-sizing.md), [Memory Map](../dev/07-memory-map.md) | | External C++ kernels | [Integrating Hand-Written C++ Kernels](../dev/language/01-external-kernels.md) | diff --git a/docs/en/user/ops/01-catalog.md b/docs/en/user/ops/01-catalog.md index bef79c635..03d59364f 100644 --- a/docs/en/user/ops/01-catalog.md +++ b/docs/en/user/ops/01-catalog.md @@ -193,8 +193,25 @@ See [Scopes and Placement](../language/04-scopes.md). ## Distributed `pypto.language.distributed` (conventionally `pld`) carries the collectives and remote -primitives. It gets its own chapter, which is not written yet — until then see -[Distributed Operators](../../dev/distributed_ops.md). +primitives. Full reference at [Distributed Programming](../distributed/index.md); +tutorial at [distributed/00-model.md](../distributed/00-model.md). + +### Capability Matrix + +| Operation | API | Modes | ReduceOp | Atomic | Supported dtypes | Notes | +| --------- | --- | ----- | -------- | ------ | ---------------- | ----- | +| AllReduce | `pld.tensor.allreduce` | `mesh` (InCore + HOST), `ring` (InCore + HOST) | `Sum`, `Max`, `Min`, `Prod` (mesh); `Sum` only (HOST ring) | — | FP16, FP32 (mesh; hard compile-time check); HOST ring: FP32 only (4-byte) | Mesh: O(N) remote traffic per step. Ring: O(N/P) remote traffic per step, 2(P-1) steps. | +| AllGather | `pld.tensor.allgather` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Push-based. Input and target must be different buffers. | +| ReduceScatter | `pld.tensor.reduce_scatter` | — | `Sum` only | — | FP32 only (HOST builtin); any GM dtype (InCore) | Every rank stages all NR chunks before the call. | +| Broadcast | `pld.tensor.broadcast` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Root stages data before the call. | +| All-to-All | `pld.tensor.all_to_all` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Personalized exchange. Input and target must be different buffers. | +| Barrier | `pld.tensor.barrier` | — | — | — | — | Signal is INT32, single-shot per call. | +| Put | `pld.tensor.put` | — | — | `None_` / `Add` | All GM dtypes | `dst` must be window-bound. Supports chunked + pipelined staging. | +| Get | `pld.tensor.get` | — | — | — | All GM dtypes | `src` must be window-bound. Supports chunked + pipelined staging. | +| Notify | `pld.system.notify` | `AtomicAdd` / `Set` | — | — | — | Side-effect-only signal deposit. | +| Wait | `pld.system.wait` | `Eq` / `Ge` | — | — | — | Side-effect-only signal block. | +| Remote Load | `pld.tile.remote_load` | — | — | — | Any (tile) | Tile-level cross-rank load. | +| Remote Store | `pld.tile.remote_store` | — | — | — | Any (tile) | Tile-level cross-rank store. | ## See Also diff --git a/docs/zh/user/00-getting_started.md b/docs/zh/user/00-getting_started.md index 60ce4ffe6..c56830160 100644 --- a/docs/zh/user/00-getting_started.md +++ b/docs/zh/user/00-getting_started.md @@ -230,6 +230,60 @@ dispatch。`format_tree()` 的 launch 表头也带上 `round=` / `slot=`。`mean ### 分布式(L3+)程序 +完整的分布式编程模型——从 `alloc_window_buffer` 到 `allreduce`、`barrier`、 +`broadcast` 等集合通信——见 [分布式编程](distributed/00-model.md)。下面展示 +InCore kernel(执行平面)的 mesh allreduce Hello World 示例;完整的可运行程序 +还需包含 host 编排器、`ir.compile` 及分布式 worker 设置,详见上述指南: + +```python +import pypto.language as pl +import pypto.language.distributed as pld + +NR = pl.dynamic("NR") + +@pl.program +class HelloAllReduce: + @pl.function(type=pl.FunctionType.InCore) + def reduce_step( + self, + inp: pl.Tensor[[1, 256], pl.FP32], + out: pl.Out[pl.Tensor[[1, 256], pl.FP32]], + data: pl.InOut[pld.DistributedTensor[[1, 256], pl.FP32]], + signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]], + ) -> pl.Tensor[[1, 256], pl.FP32]: + ctx = pld.get_comm_ctx(data) + my_rank = pld.rank(ctx) + nranks = pld.nranks(ctx) + + # 1. Stage-in:将本地输入复制到本 rank 的 window 分片。 + data = pl.store(pl.load(inp, [0, 0], [1, 256]), [0, 0], data) + + # 2. Barrier:通知每个对端,然后等待每个对端。 + for peer in pl.range(nranks): + if peer != my_rank: + pld.system.notify(signal, peer=peer, offsets=[my_rank, 0], + value=1, op=pld.NotifyOp.AtomicAdd) + for src in pl.range(nranks): + if src != my_rank: + pld.system.wait(signal, offsets=[src, 0], + expected=1, cmp=pld.WaitCmp.Ge) + + # 3. 计算:加载自身分片,remote-load 每个对端,累加。 + acc = pl.load(data, [0, 0], [1, 256]) + for peer in pl.range(nranks): + if peer != my_rank: + peer_tile = pld.tile.remote_load( + data, peer=peer, offsets=[0, 0], shape=[1, 256]) + acc = pl.add(acc, peer_tile) + + # 4. Stage-out:将累加器写入本地输出。 + out = pl.store(acc, [0, 0], out) + return out +``` + +指南包含**逐行解读、ring allreduce 权衡、notify/wait 握手模式以及调试表格**。 +完整章节见 [distributed/index.md](distributed/index.md)。 + `ir.compile` 对 L3+ 分布式程序返回的 `DistributedCompiledProgram` 与 `CompiledProgram` 一样接受 `DeviceTensor` 入参:用 worker 常驻 buffer 替代 `torch.Tensor`,runtime 即对该 参数跳过 H2D/D2H。这是在 generate 循环的多次 dispatch 之间保持大块静态权重常驻的推荐做法。 diff --git a/docs/zh/user/distributed/01-collectives.md b/docs/zh/user/distributed/01-collectives.md index 26ba9750b..3f7fa143f 100644 --- a/docs/zh/user/distributed/01-collectives.md +++ b/docs/zh/user/distributed/01-collectives.md @@ -62,10 +62,12 @@ signal(仅限 mesh)。 ### 支持的 ReduceOp -全部四种——`Sum`、`Max`、`Min`、`Prod`——InCore 组合调用和 Host 内置路径均 -支持。`target` 的 dtype 必须是 `FP16` 或 `FP32`;这是编译期硬性检查,而非 -仅存储位宽的限制。除了本页开头要求的形状相同的 signal tensor 外,所有 -rank 还必须使用相同的 `ReduceOp` 和 `mode`。 +全部四种——`Sum`、`Max`、`Min`、`Prod`——InCore 组合调用和 Host 内置的 +mesh 路径均支持。Host 内置的 ring 路径(`builtin.tensor.allreduce_ring`) +更窄:仅 `Sum`,且 target 须为 4 字节的 `FP32`(编译期检查)。mesh 路径的 +`target` dtype 必须是 `FP16` 或 `FP32`;ring 路径仅 `FP32`。除了本页开头 +要求的形状相同的 signal tensor 外,所有 rank 还必须使用相同的 `ReduceOp` +和 `mode`。 ## Barrier @@ -168,15 +170,16 @@ PyPTO 有三种方式运行集合通信——根据代码运行的位置以及 | **位置** | `@pl.jit.incore` | `@pl.jit.incore` | `@pl.jit.host` | | **实现** | 手写 `notify`/`wait` + `remote_load` 循环 | 直接调用 `pld.tensor.allreduce(data, sig, ...)` | 直接调用 `pld.tensor.allreduce(data, [sig,] ...)` | | **Lowering** | 自行实现原语 | `LowerCompositeOps` | `LowerHostTensorCollectives` | -| **支持的模式** | 取决于自己的实现 | `mesh` 和 `ring` | 仅 `mesh` | -| **Signal 形状** | 取决于自己的分配 | mesh 为 `[nranks, 1]`(rank 数量可为动态);ring 为 `[2×(NR−1), NR]`(`NR` 必须是编译期常量) | 一维 `[world_size]` 或二维 `[world_size, 1]`——编译器合成的 signal 为二维 | -| **适用场景** | 学习、自定义协议 | 需要 `ring` 模式,或已身处 InCore kernel 内部 | 日常的 host 编排集合通信 | +| **支持的模式** | 取决于自己的实现 | `mesh` 和 `ring` | `mesh` 和 `ring`(ring:仅 `Sum` + `FP32`) | +| **Signal 形状** | 取决于自己的分配 | mesh 为 `[nranks, 1]`(rank 数量可为动态);ring 为 `[2×(NR−1), NR]`(`NR` 必须是编译期常量) | mesh:一维 `[world_size]` 或二维 `[world_size, 1]`(编译器合成的 signal 为二维);ring:`[2*(NR−1)+1, NR]` | +| **适用场景** | 学习、自定义协议 | ring 需要非 `Sum`/非 `FP32`,或已身处 InCore kernel 内部 | 日常的 host 编排集合通信 | 日常 host 编排代码优先使用 Host 级别内置——它们自动处理屏障编排和分块。 只有 `allreduce` 可以省略 signal 参数(编译器会在循环外自动合成一个); 其余五种集合通信(`barrier`、`broadcast`、`allgather`、`reduce_scatter`、 -`all_to_all`)始终需要调用方显式分配并传入 signal。当需要 `mode="ring"` -时改用 InCore 组合调用,因为 Host 内置路径只 lowering `mesh`。 +`all_to_all`)始终需要调用方显式分配并传入 signal。InCore 组合调用与 Host +内置均支持 `mode="ring"`;当 ring 需要 `Sum` 以外的 `ReduceOp` 或非 `FP32` +的 dtype 时改用 InCore 组合调用,因为 Host 内置的 ring 路径仅支持 `Sum` + `FP32`。 ## 可运行示例 @@ -186,7 +189,7 @@ PyPTO 有三种方式运行集合通信——根据代码运行的位置以及 | 集合通信 | InCore 手写 | InCore 组合调用 | HOST 内置 | | -------- | ----------- | --------------- | --------- | | allreduce | `collectives/test_l3_allreduce.py` | `collectives/test_l3_tensor_allreduce_intrinsic.py` | `test_l3_host_tensor_allreduce.py` | -| allreduce(ring) | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | 无(仅 mesh) | +| allreduce(ring) | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | `test_l3_host_tensor_allreduce_ring.py` | | barrier | — | `collectives/test_l3_tensor_barrier_intrinsic.py` | `test_l3_host_tensor_barrier.py` | | broadcast | `collectives/test_l3_broadcast.py` | `collectives/test_l3_tensor_broadcast_intrinsic.py` | `test_l3_host_tensor_broadcast.py` | | allgather | `collectives/test_l3_allgather.py` | `collectives/test_l3_tensor_allgather_intrinsic.py` | `test_l3_host_tensor_allgather.py` | diff --git a/docs/zh/user/index.md b/docs/zh/user/index.md index 4a3d55931..f07d8b086 100644 --- a/docs/zh/user/index.md +++ b/docs/zh/user/index.md @@ -4,7 +4,7 @@ ## 阅读路径 -按你当前要做的事挑一条。三条路径都假设[安装](01-installation.md)已完成。 +按你当前要做的事挑一条。四条路径都假设[安装](01-installation.md)已完成。 ### 我要写第一个 kernel @@ -31,6 +31,14 @@ 在动手测量之前,先看编译产物里的 `report/perf_hints.log` —— 编译器可能已经告诉你了。 性能专章尚未编写,其内容当前的位置见下表。 +### 我想跨多个设备运行 + +[分布式编程](distributed/index.md) + +先让单设备 kernel 跑通 —— 分布式程序是在 `pld.*` 集合通信和 HOST 编排器之上 +组合同样的 `pl.*` kernel。跑通之后,分布式章节覆盖 ring 与 mesh 的 +开销取舍以及跨 rank 重叠。 + ## 目录 | 页面 | 内容 | @@ -43,6 +51,7 @@ | [编译程序](01-language_guide.md) | `ir.compile()` 与 `JITFunction.compile()`,以及检视结果 | | [在设备上运行](00-getting_started.md) | 常驻设备张量、显式派发、性能基准、分布式执行 | | [Torch Codegen 调试指南](03-torch_codegen_debug.md) | 从 IR 生成 PyTorch 参考实现,用于定位精度问题 | +| [分布式编程](distributed/index.md) | 跨 rank 程序的对称内存模型、集合通信、底层原语、执行与调试 | ## PyPTO 提供了什么 @@ -55,7 +64,7 @@ | `@pl.jit` 全家族(`.incore`、`.inline`、`.opaque`、`.host`) | [快速上手](02-quickstart.md)、[函数与程序](language/01-functions.md) | | 手写 C++ kernel 接入 | [外部 Kernel](../dev/language/01-external-kernels.md) | | 设备常驻张量、显式派发、性能基准 | [在设备上运行](00-getting_started.md) | -| 分布式(多卡)程序与集合通信 | [分布式算子](../dev/distributed_ops.md) | +| 分布式(多卡)程序与集合通信 | [分布式编程](distributed/index.md) | | 对照 PyTorch 参考实现做精度定位 | [Torch Codegen 调试指南](03-torch_codegen_debug.md) | | 编译期诊断与性能提示 | [诊断](../dev/passes/92-diagnostics.md) | | 运行时 DFX:swimlane、PMU、依赖图、scope stats | [运行时 DFX](../dev/03-runtime-dfx.md) | @@ -63,13 +72,12 @@ ## 尚未收录的内容 -本手册正在扩展为完整的分章结构 —— 教程、分布式编程、性能优化、精度定位各自成章。 +本手册正在扩展为完整的分章结构 —— 教程、性能优化、精度定位各自成章。 在这些章节落地之前,相应内容位于[开发者文档](../dev/index.md): | 主题 | 当前位置 | | ---- | -------- | | 混合 kernel(AIC + AIV 同一函数) | [LowerAutoVectorSplit](../dev/passes/20-lower_auto_vector_split.md)、[ExpandMixedKernel](../dev/passes/21-expand_mixed_kernel.md)、[TPUSH/TPOP](../reference/pto-isa/01-tpush_tpop.md) | -| 分布式 DSL 与集合通信 | [分布式算子](../dev/distributed_ops.md) | | 性能提示与诊断 | [诊断](../dev/passes/92-diagnostics.md)、[编译性能剖析](../dev/01-compile-profiling.md) | | 运行时 DFX 开关、ring sizing、memory map | [运行时 DFX](../dev/03-runtime-dfx.md)、[逐任务 Ring Sizing](../dev/05-runtime-ring-sizing.md)、[内存图](../dev/07-memory-map.md) | | 外部 C++ kernel | [集成手写 C++ Kernel](../dev/language/01-external-kernels.md) | diff --git a/docs/zh/user/ops/01-catalog.md b/docs/zh/user/ops/01-catalog.md index f90674e61..c72bb1f48 100644 --- a/docs/zh/user/ops/01-catalog.md +++ b/docs/zh/user/ops/01-catalog.md @@ -181,7 +181,24 @@ push 与 pop 必须**配对**,且每次 pop 都必须有对应的 `tfree`。 ## 分布式 -`pypto.language.distributed`(约定简写 `pld`)承载集合通信与远程原语。它有独立的一章,但尚未编写 —— 在此之前见 [分布式算子](../../dev/distributed_ops.md)。 +`pypto.language.distributed`(约定简写 `pld`)承载集合通信与远程原语。完整参考见[分布式编程](../distributed/index.md);教程见[distributed/00-model.md](../distributed/00-model.md)。 + +### 功能矩阵 + +| 操作 | API | 模式 | ReduceOp | Atomic | 支持的 dtype | 说明 | +| ---- | --- | ---- | -------- | ------ | ------------ | ---- | +| AllReduce | `pld.tensor.allreduce` | `mesh`(InCore + HOST),`ring`(InCore + HOST) | `Sum`、`Max`、`Min`、`Prod`(mesh);仅 `Sum`(HOST ring) | — | FP16、FP32(mesh;编译期硬性检查);HOST ring:仅 FP32(4 字节) | Mesh: 每步 O(N) 远程流量。Ring: 每步 O(N/P) 远程流量,2(P-1) 步。 | +| AllGather | `pld.tensor.allgather` | — | — | — | 仅 FP32(HOST builtin);任意 GM dtype(InCore) | 推式。输入和 target 必须是不同的 buffer。 | +| ReduceScatter | `pld.tensor.reduce_scatter` | — | 仅 `Sum` | — | 仅 FP32(HOST builtin);任意 GM dtype(InCore) | 每个 rank 在调用前将全部 NR 个数据块写入。 | +| Broadcast | `pld.tensor.broadcast` | — | — | — | 仅 FP32(HOST builtin);任意 GM dtype(InCore) | Root 在调用前将数据写入。 | +| All-to-All | `pld.tensor.all_to_all` | — | — | — | 仅 FP32(HOST builtin);任意 GM dtype(InCore) | 个性化交换。输入和 target 必须是不同的 buffer。 | +| Barrier | `pld.tensor.barrier` | — | — | — | — | Signal 为 INT32,每次调用单次使用。 | +| Put | `pld.tensor.put` | — | — | `None_` / `Add` | 所有 GM dtype | `dst` 必须是 window-bound。支持分块和流水线 staging。 | +| Get | `pld.tensor.get` | — | — | — | 所有 GM dtype | `src` 必须是 window-bound。支持分块和流水线 staging。 | +| Notify | `pld.system.notify` | `AtomicAdd` / `Set` | — | — | — | 仅副作用的信号投递。 | +| Wait | `pld.system.wait` | `Eq` / `Ge` | — | — | — | 仅副作用的信号阻塞。 | +| Remote Load | `pld.tile.remote_load` | — | — | — | 任意(tile) | Tile 级跨 rank 加载。 | +| Remote Store | `pld.tile.remote_store` | — | — | — | 任意(tile) | Tile 级跨 rank 写入。 | ## See Also