docs(user): wire the Distributed Programming chapter into the manual

docs/en/user/00-getting_started.md

@@ -264,6 +264,63 @@ per-rank / `union` views are empty.
 
 ### Distributed (L3+) programs
 
+The complete distributed programming model — from `alloc_window_buffer` through
+collectives like `allreduce`, `barrier`, and `broadcast` — is covered in the
+[Distributed Programming](distributed/00-model.md). Here is the mesh allreduce
+Hello World, shown as the InCore kernel (execution plane); a fully runnable
+program also needs the host orchestrator, `ir.compile`, and distributed worker
+setup — see the guide above:
+
+```python
+import pypto.language as pl
+import pypto.language.distributed as pld
+
+NR = pl.dynamic("NR")
+
+@pl.program
+class HelloAllReduce:
+    @pl.function(type=pl.FunctionType.InCore)
+    def reduce_step(
+        self,
+        inp: pl.Tensor[[1, 256], pl.FP32],
+        out: pl.Out[pl.Tensor[[1, 256], pl.FP32]],
+        data: pl.InOut[pld.DistributedTensor[[1, 256], pl.FP32]],
+        signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]],
+    ) -> pl.Tensor[[1, 256], pl.FP32]:
+        ctx = pld.get_comm_ctx(data)
+        my_rank = pld.rank(ctx)
+        nranks = pld.nranks(ctx)
+
+        # 1. Stage-in: copy local input into this rank's window slice.
+        data = pl.store(pl.load(inp, [0, 0], [1, 256]), [0, 0], data)
+
+        # 2. Barrier: notify every peer, then wait on every peer.
+        for peer in pl.range(nranks):
+            if peer != my_rank:
+                pld.system.notify(signal, peer=peer, offsets=[my_rank, 0],
+                                  value=1, op=pld.NotifyOp.AtomicAdd)
+        for src in pl.range(nranks):
+            if src != my_rank:
+                pld.system.wait(signal, offsets=[src, 0],
+                                expected=1, cmp=pld.WaitCmp.Ge)
+
+        # 3. Compute: load own slice, remote-load every peer, accumulate.
+        acc = pl.load(data, [0, 0], [1, 256])
+        for peer in pl.range(nranks):
+            if peer != my_rank:
+                peer_tile = pld.tile.remote_load(
+                    data, peer=peer, offsets=[0, 0], shape=[1, 256])
+                acc = pl.add(acc, peer_tile)
+
+        # 4. Stage-out: store accumulator to output.
+        out = pl.store(acc, [0, 0], out)
+        return out
+```
+
+The guide includes a **line-by-line walkthrough, ring allreduce trade-offs,
+notify/wait handshake patterns, and a debugging table**. The full chapter is
+at [distributed/index.md](distributed/index.md).
+
 L3+ distributed programs returned by `ir.compile` (a `DistributedCompiledProgram`)
 accept `DeviceTensor` arguments the same way as `CompiledProgram`: pass a
 worker-resident buffer in place of a `torch.Tensor` and the runtime skips H2D/D2H

docs/en/user/distributed/01-collectives.md

@@ -67,11 +67,12 @@ the reduced result). All ranks must pass identically shaped `target` tensors.
 
 ### Supported ReduceOp
 
-All four — `Sum`, `Max`, `Min`, `Prod` — on both the InCore composite and the
-HOST builtin path. `target`'s dtype must be `FP16` or `FP32`; this is a hard
-compile-time check, not just a storage-width constraint. Every rank must
-agree on the same `ReduceOp` and `mode`, on top of the identically-shaped
-signal tensors required of every collective.
+All four — `Sum`, `Max`, `Min`, `Prod` — on the InCore composite and the HOST
+builtin mesh path. The HOST builtin ring path (`builtin.tensor.allreduce_ring`)
+is narrower: `Sum` only, with a 4-byte `FP32` target (a compile-time check).
+`mesh` targets must be `FP16` or `FP32`; the ring path is `FP32`-only. Every
+rank must agree on the same `ReduceOp` and `mode`, on top of the
+identically-shaped signal tensors required of every collective.
 
 ## Barrier
 
@@ -175,17 +176,19 @@ runs and whether you need `mode="ring"`:
 | **Where** | `@pl.jit.incore` | `@pl.jit.incore` | `@pl.jit.host` |
 | **How** | Manual `notify`/`wait` + `remote_load` loops | `pld.tensor.allreduce(data, sig, ...)` called directly | `pld.tensor.allreduce(data, [sig,] ...)` called directly |
 | **Lowering** | You write the primitives | `LowerCompositeOps` | `LowerHostTensorCollectives` |
-| **Modes** | Whatever you implement | `mesh` and `ring` | `mesh` only |
-| **Signal shape** | Whatever you allocate | `[nranks, 1]` for mesh (rank count may be dynamic); `[2×(NR−1), NR]` for ring (`NR` must be a compile-time constant) | Rank-1 `[world_size]` or rank-2 `[world_size, 1]` — the compiler-synthesized signal is rank-2 |
-| **When** | Learning, custom protocols | Need `ring` mode, or already inside an InCore kernel | Day-to-day host-orchestrated collectives |
+| **Modes** | Whatever you implement | `mesh` and `ring` | `mesh` and `ring` (ring: `Sum` + `FP32` only) |
+| **Signal shape** | Whatever you allocate | `[nranks, 1]` for mesh (rank count may be dynamic); `[2×(NR−1), NR]` for ring (`NR` must be a compile-time constant) | Mesh: rank-1 `[world_size]` or rank-2 `[world_size, 1]` (the compiler-synthesized signal is rank-2). Ring: `[2*(NR−1)+1, NR]` |
+| **When** | Learning, custom protocols | Ring with non-`Sum`/non-`FP32`, or already inside an InCore kernel | Day-to-day host-orchestrated collectives |
 
 Prefer HOST builtins for day-to-day host-orchestrated code — they handle
 barrier orchestration and chunking automatically. Only `allreduce` can also
 omit the signal argument (the compiler synthesizes one outside loops); the
 other five collectives (`barrier`, `broadcast`, `allgather`,
 `reduce_scatter`, `all_to_all`) always take an explicit, caller-allocated
-signal. Reach for the InCore composite specifically when you need
-`mode="ring"`, since the HOST builtin path only lowers `mesh`.
+signal. Both the InCore composite and the HOST builtin lower `mode="ring"`;
+reach for the InCore composite when you need ring with a `ReduceOp` other than
+`Sum` or a non-`FP32` dtype, since the HOST builtin ring path is `Sum`+`FP32`
+only.
 
 ## Runnable Examples
 
@@ -195,7 +198,7 @@ Every collective above has a runnable counterpart under
 | Collective | InCore hand-rolled | InCore composite | HOST builtin |
 | ---------- | ------------------ | ---------------- | ------------ |
 | allreduce | `collectives/test_l3_allreduce.py` | `collectives/test_l3_tensor_allreduce_intrinsic.py` | `test_l3_host_tensor_allreduce.py` |
-| allreduce (ring) | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | n/a (mesh only) |
+| allreduce (ring) | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | `test_l3_host_tensor_allreduce_ring.py` |
 | barrier | — | `collectives/test_l3_tensor_barrier_intrinsic.py` | `test_l3_host_tensor_barrier.py` |
 | broadcast | `collectives/test_l3_broadcast.py` | `collectives/test_l3_tensor_broadcast_intrinsic.py` | `test_l3_host_tensor_broadcast.py` |
 | allgather | `collectives/test_l3_allgather.py` | `collectives/test_l3_tensor_allgather_intrinsic.py` | `test_l3_host_tensor_allgather.py` |

docs/en/user/index.md

@@ -4,7 +4,7 @@ How to write, compile, run, and debug PyPTO programs.
 
 ## Reading paths
 
-Pick the path that matches what you are trying to do. All three assume
+Pick the path that matches what you are trying to do. All four assume
 [Installation](01-installation.md) is done.
 
 ### I want to write my first kernel
@@ -36,6 +36,15 @@ Check `report/perf_hints.log` from your compile output before measuring anything
 compiler may already have told you. The dedicated performance chapter is not written yet;
 see the table below for where its material currently lives.
 
+### I want to run across multiple devices
+
+[Distributed Programming](distributed/index.md)
+
+Get a single-device kernel running first — distributed programs compose the same
+`pl.*` kernels behind `pld.*` collectives and a HOST orchestrator. Once it runs
+correctly, the distributed chapter covers ring vs. mesh trade-offs and
+cross-rank overlap.
+
 ## Contents
 
 | Page | What it covers |
@@ -48,6 +57,7 @@ see the table below for where its material currently lives.
 | [Compiling a Program](01-language_guide.md) | `ir.compile()` and `JITFunction.compile()`, and inspecting the result |
 | [Running on Device](00-getting_started.md) | Resident device tensors, explicit dispatch, benchmarking, distributed execution |
 | [Torch Codegen Debug Guide](03-torch_codegen_debug.md) | Generating a PyTorch reference implementation from the IR to isolate accuracy problems |
+| [Distributed Programming](distributed/index.md) | Symmetric-memory model, collectives, primitives, execution, and debugging for cross-rank programs |
 
 ## What PyPTO gives you
 
@@ -60,23 +70,22 @@ see the table below for where its material currently lives.
 | The full `@pl.jit` family (`.incore`, `.inline`, `.opaque`, `.host`) | [Quickstart](02-quickstart.md), [Functions and Programs](language/01-functions.md) |
 | Hand-written C++ kernel integration | [External Kernels](../dev/language/01-external-kernels.md) |
 | Device-resident tensors, explicit dispatch, benchmarking | [Running on Device](00-getting_started.md) |
-| Distributed (multi-card) programs and collectives | [Distributed Operators](../dev/distributed_ops.md) |
+| Distributed (multi-card) programs and collectives | [Distributed Programming](distributed/index.md) |
 | Accuracy debugging against a PyTorch reference | [Torch Codegen Debug Guide](03-torch_codegen_debug.md) |
 | Compile-time diagnostics and performance hints | [Diagnostics](../dev/passes/92-diagnostics.md) |
 | Runtime DFX: swimlane, PMU, dependency graph, scope stats | [Runtime DFX](../dev/03-runtime-dfx.md) |
 | On-chip memory map visualization | [Memory Map](../dev/07-memory-map.md) |
 
 ## What is not here yet
 
-This manual is being expanded into a full chaptered structure — tutorials, distributed
-programming, performance optimization, and accuracy debugging each get their own
-chapter. Until those land, the corresponding material lives in the
-[developer documentation](../dev/index.md):
+This manual is being expanded into a full chaptered structure — tutorials,
+performance optimization, and accuracy debugging each get their own chapter.
+Until those land, the
+corresponding material lives in the [developer documentation](../dev/index.md):
 
 | Topic | Current location |
 | ----- | ---------------- |
 | Mixed kernels (AIC + AIV in one function) | [LowerAutoVectorSplit](../dev/passes/20-lower_auto_vector_split.md), [ExpandMixedKernel](../dev/passes/21-expand_mixed_kernel.md), [TPUSH/TPOP](../reference/pto-isa/01-tpush_tpop.md) |
-| Distributed DSL and collectives | [Distributed Operators](../dev/distributed_ops.md) |
 | Performance hints and diagnostics | [Diagnostics](../dev/passes/92-diagnostics.md), [Compile Profiling](../dev/01-compile-profiling.md) |
 | Runtime DFX flags, ring sizing, memory map | [Runtime DFX](../dev/03-runtime-dfx.md), [Per-Task Ring Sizing](../dev/05-runtime-ring-sizing.md), [Memory Map](../dev/07-memory-map.md) |
 | External C++ kernels | [Integrating Hand-Written C++ Kernels](../dev/language/01-external-kernels.md) |

docs/en/user/ops/01-catalog.md

@@ -193,8 +193,25 @@ See [Scopes and Placement](../language/04-scopes.md).
 ## Distributed
 
 `pypto.language.distributed` (conventionally `pld`) carries the collectives and remote
-primitives. It gets its own chapter, which is not written yet — until then see
-[Distributed Operators](../../dev/distributed_ops.md).
+primitives. Full reference at [Distributed Programming](../distributed/index.md);
+tutorial at [distributed/00-model.md](../distributed/00-model.md).
+
+### Capability Matrix
+
+| Operation | API | Modes | ReduceOp | Atomic | Supported dtypes | Notes |
+| --------- | --- | ----- | -------- | ------ | ---------------- | ----- |
+| AllReduce | `pld.tensor.allreduce` | `mesh` (InCore + HOST), `ring` (InCore + HOST) | `Sum`, `Max`, `Min`, `Prod` (mesh); `Sum` only (HOST ring) | — | FP16, FP32 (mesh; hard compile-time check); HOST ring: FP32 only (4-byte) | Mesh: O(N) remote traffic per step. Ring: O(N/P) remote traffic per step, 2(P-1) steps. |
+| AllGather | `pld.tensor.allgather` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Push-based. Input and target must be different buffers. |
+| ReduceScatter | `pld.tensor.reduce_scatter` | — | `Sum` only | — | FP32 only (HOST builtin); any GM dtype (InCore) | Every rank stages all NR chunks before the call. |
+| Broadcast | `pld.tensor.broadcast` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Root stages data before the call. |
+| All-to-All | `pld.tensor.all_to_all` | — | — | — | FP32 only (HOST builtin); any GM dtype (InCore) | Personalized exchange. Input and target must be different buffers. |
+| Barrier | `pld.tensor.barrier` | — | — | — | — | Signal is INT32, single-shot per call. |
+| Put | `pld.tensor.put` | — | — | `None_` / `Add` | All GM dtypes | `dst` must be window-bound. Supports chunked + pipelined staging. |
+| Get | `pld.tensor.get` | — | — | — | All GM dtypes | `src` must be window-bound. Supports chunked + pipelined staging. |
+| Notify | `pld.system.notify` | `AtomicAdd` / `Set` | — | — | — | Side-effect-only signal deposit. |
+| Wait | `pld.system.wait` | `Eq` / `Ge` | — | — | — | Side-effect-only signal block. |
+| Remote Load | `pld.tile.remote_load` | — | — | — | Any (tile) | Tile-level cross-rank load. |
+| Remote Store | `pld.tile.remote_store` | — | — | — | Any (tile) | Tile-level cross-rank store. |
 
 ## See Also
 

docs/zh/user/00-getting_started.md

@@ -230,6 +230,60 @@ dispatch。`format_tree()` 的 launch 表头也带上 `round=` / `slot=`。`mean
 
 ### 分布式（L3+）程序
 
+完整的分布式编程模型——从 `alloc_window_buffer` 到 `allreduce`、`barrier`、
+`broadcast` 等集合通信——见 [分布式编程](distributed/00-model.md)。下面展示
+InCore kernel（执行平面）的 mesh allreduce Hello World 示例；完整的可运行程序
+还需包含 host 编排器、`ir.compile` 及分布式 worker 设置，详见上述指南：
+
+```python
+import pypto.language as pl
+import pypto.language.distributed as pld
+
+NR = pl.dynamic("NR")
+
+@pl.program
+class HelloAllReduce:
+    @pl.function(type=pl.FunctionType.InCore)
+    def reduce_step(
+        self,
+        inp: pl.Tensor[[1, 256], pl.FP32],
+        out: pl.Out[pl.Tensor[[1, 256], pl.FP32]],
+        data: pl.InOut[pld.DistributedTensor[[1, 256], pl.FP32]],
+        signal: pl.InOut[pld.DistributedTensor[[NR, 1], pl.INT32]],
+    ) -> pl.Tensor[[1, 256], pl.FP32]:
+        ctx = pld.get_comm_ctx(data)
+        my_rank = pld.rank(ctx)
+        nranks = pld.nranks(ctx)
+
+        # 1. Stage-in：将本地输入复制到本 rank 的 window 分片。
+        data = pl.store(pl.load(inp, [0, 0], [1, 256]), [0, 0], data)
+
+        # 2. Barrier：通知每个对端，然后等待每个对端。
+        for peer in pl.range(nranks):
+            if peer != my_rank:
+                pld.system.notify(signal, peer=peer, offsets=[my_rank, 0],
+                                  value=1, op=pld.NotifyOp.AtomicAdd)
+        for src in pl.range(nranks):
+            if src != my_rank:
+                pld.system.wait(signal, offsets=[src, 0],
+                                expected=1, cmp=pld.WaitCmp.Ge)
+
+        # 3. 计算：加载自身分片，remote-load 每个对端，累加。
+        acc = pl.load(data, [0, 0], [1, 256])
+        for peer in pl.range(nranks):
+            if peer != my_rank:
+                peer_tile = pld.tile.remote_load(
+                    data, peer=peer, offsets=[0, 0], shape=[1, 256])
+                acc = pl.add(acc, peer_tile)
+
+        # 4. Stage-out：将累加器写入本地输出。
+        out = pl.store(acc, [0, 0], out)
+        return out
+```
+
+指南包含**逐行解读、ring allreduce 权衡、notify/wait 握手模式以及调试表格**。
+完整章节见 [distributed/index.md](distributed/index.md)。
+
 `ir.compile` 对 L3+ 分布式程序返回的 `DistributedCompiledProgram` 与 `CompiledProgram`
 一样接受 `DeviceTensor` 入参：用 worker 常驻 buffer 替代 `torch.Tensor`，runtime 即对该
 参数跳过 H2D/D2H。这是在 generate 循环的多次 dispatch 之间保持大块静态权重常驻的推荐做法。

docs/zh/user/distributed/01-collectives.md

@@ -62,10 +62,12 @@ signal（仅限 mesh）。
 
 ### 支持的 ReduceOp
 
-全部四种——`Sum`、`Max`、`Min`、`Prod`——InCore 组合调用和 Host 内置路径均
-支持。`target` 的 dtype 必须是 `FP16` 或 `FP32`；这是编译期硬性检查，而非
-仅存储位宽的限制。除了本页开头要求的形状相同的 signal tensor 外，所有
-rank 还必须使用相同的 `ReduceOp` 和 `mode`。
+全部四种——`Sum`、`Max`、`Min`、`Prod`——InCore 组合调用和 Host 内置的
+mesh 路径均支持。Host 内置的 ring 路径（`builtin.tensor.allreduce_ring`）
+更窄：仅 `Sum`，且 target 须为 4 字节的 `FP32`（编译期检查）。mesh 路径的
+`target` dtype 必须是 `FP16` 或 `FP32`；ring 路径仅 `FP32`。除了本页开头
+要求的形状相同的 signal tensor 外，所有 rank 还必须使用相同的 `ReduceOp`
+和 `mode`。
 
 ## Barrier
 
@@ -168,15 +170,16 @@ PyPTO 有三种方式运行集合通信——根据代码运行的位置以及
 | **位置** | `@pl.jit.incore` | `@pl.jit.incore` | `@pl.jit.host` |
 | **实现** | 手写 `notify`/`wait` + `remote_load` 循环 | 直接调用 `pld.tensor.allreduce(data, sig, ...)` | 直接调用 `pld.tensor.allreduce(data, [sig,] ...)` |
 | **Lowering** | 自行实现原语 | `LowerCompositeOps` | `LowerHostTensorCollectives` |
-| **支持的模式** | 取决于自己的实现 | `mesh` 和 `ring` | 仅 `mesh` |
-| **Signal 形状** | 取决于自己的分配 | mesh 为 `[nranks, 1]`（rank 数量可为动态）；ring 为 `[2×(NR−1), NR]`（`NR` 必须是编译期常量） | 一维 `[world_size]` 或二维 `[world_size, 1]`——编译器合成的 signal 为二维 |
-| **适用场景** | 学习、自定义协议 | 需要 `ring` 模式，或已身处 InCore kernel 内部 | 日常的 host 编排集合通信 |
+| **支持的模式** | 取决于自己的实现 | `mesh` 和 `ring` | `mesh` 和 `ring`（ring：仅 `Sum` + `FP32`） |
+| **Signal 形状** | 取决于自己的分配 | mesh 为 `[nranks, 1]`（rank 数量可为动态）；ring 为 `[2×(NR−1), NR]`（`NR` 必须是编译期常量） | mesh：一维 `[world_size]` 或二维 `[world_size, 1]`（编译器合成的 signal 为二维）；ring：`[2*(NR−1)+1, NR]` |
+| **适用场景** | 学习、自定义协议 | ring 需要非 `Sum`/非 `FP32`，或已身处 InCore kernel 内部 | 日常的 host 编排集合通信 |
 
 日常 host 编排代码优先使用 Host 级别内置——它们自动处理屏障编排和分块。
 只有 `allreduce` 可以省略 signal 参数（编译器会在循环外自动合成一个）；
 其余五种集合通信（`barrier`、`broadcast`、`allgather`、`reduce_scatter`、
-`all_to_all`）始终需要调用方显式分配并传入 signal。当需要 `mode="ring"`
-时改用 InCore 组合调用，因为 Host 内置路径只 lowering `mesh`。
+`all_to_all`）始终需要调用方显式分配并传入 signal。InCore 组合调用与 Host
+内置均支持 `mode="ring"`；当 ring 需要 `Sum` 以外的 `ReduceOp` 或非 `FP32`
+的 dtype 时改用 InCore 组合调用，因为 Host 内置的 ring 路径仅支持 `Sum` + `FP32`。
 
 ## 可运行示例
 
@@ -186,7 +189,7 @@ PyPTO 有三种方式运行集合通信——根据代码运行的位置以及
 | 集合通信 | InCore 手写 | InCore 组合调用 | HOST 内置 |
 | -------- | ----------- | --------------- | --------- |
 | allreduce | `collectives/test_l3_allreduce.py` | `collectives/test_l3_tensor_allreduce_intrinsic.py` | `test_l3_host_tensor_allreduce.py` |
-| allreduce（ring） | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | 无（仅 mesh） |
+| allreduce（ring） | `collectives/test_l3_allreduce_ring.py` | `collectives/test_l3_tensor_allreduce_ring_intrinsic.py` | `test_l3_host_tensor_allreduce_ring.py` |
 | barrier | — | `collectives/test_l3_tensor_barrier_intrinsic.py` | `test_l3_host_tensor_barrier.py` |
 | broadcast | `collectives/test_l3_broadcast.py` | `collectives/test_l3_tensor_broadcast_intrinsic.py` | `test_l3_host_tensor_broadcast.py` |
 | allgather | `collectives/test_l3_allgather.py` | `collectives/test_l3_tensor_allgather_intrinsic.py` | `test_l3_host_tensor_allgather.py` |