docs(user): add the Distributed Programming chapter - #2219
Conversation
Split out of PR hw-native-sys#2162. Adds docs/en/user/distributed/ (+ zh mirror): symmetric-memory-and-signals model, comm domains, the three collective execution paths (InCore hand-rolled, InCore composite, HOST builtin), every collective with worked examples, primitives (notify/wait, window buffers, put/get, tile RMA), DistributedWorker lifecycle, and a debugging/failure-mode reference. Wires the chapter into mkdocs.yml nav. Cross-references to the not-yet-landed Performance chapter are deferred to the PR that lands it.
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds bilingual documentation for PyPTO’s distributed programming model, primitives, collectives, execution runtime, and debugging workflow. The English distributed pages are added to MkDocs navigation. ChangesDistributed documentation
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/en/user/distributed/01-collectives.md`:
- Line 3: Update the introductory collective count in
docs/en/user/distributed/01-collectives.md:3-3 from five to six, and update the
corresponding Chinese count in docs/zh/user/distributed/01-collectives.md:3-3
from 五种 to 六种; no other content needs changing.
- Around line 10-11: Update the AllReduce description in
docs/en/user/distributed/01-collectives.md lines 10-11 to say every rank
receives the reduced result instead of the summed result; make the equivalent
sum-neutral wording change in docs/zh/user/distributed/01-collectives.md line 9,
preserving the existing meaning in both translations.
In `@docs/en/user/distributed/02-primitives.md`:
- Around line 153-167: Correct the barrier explanation around the AtomicAdd
example: clarify that offsets=[my_rank, 0] gives each [src, 0] cell a single
writer, so N ranks do not update one shared cell, and state that this
per-source-slot protocol can use Set instead. Do not retain the incorrect
shared-cell AtomicAdd rationale unless the example is changed to demonstrate a
true shared-slot N-to-1 barrier.
In `@docs/en/user/distributed/04-debugging.md`:
- Around line 51-58: Update the “Distributed DFX Entry Points” documentation to
state the actual output paths: L2 records at
dfx_outputs/l2_swimlane_records.json, scope statistics at
dfx_outputs/scope_stats/scope_stats.jsonl, and dependency data at
dfx_outputs/deps.json. Replace the vague “span-tree” reference and mention any
generated merged files where applicable.
In `@docs/zh/user/distributed/02-primitives.md`:
- Around line 28-36: 完善 docs/zh/user/distributed/02-primitives.md:28-36 的
alloc_window_buffer 文档，补充 shape+dtype 分配重载及 name 命名行为，并在 02-primitives.md:80-117
说明 RMA、Put/Get、分块与流水线规则及动态 shape 限制；完善
docs/zh/user/distributed/01-collectives.md:72-130，补充
Broadcast、AllGather、ReduceScatter、AllToAll 的前置条件和结果语义；完善
docs/zh/user/distributed/04-debugging.md:44-48，写明具体 DFX 产物路径及相关 flag 的行为。
In `@docs/zh/user/distributed/03-execution.md`:
- Around line 19-41: 补齐中文执行文档，使其与英文版本保持完整对应：在现有方法和 DeviceTensor 章节中恢复
StackedDeviceTensor、fork 前共享内存要求、多程序 worker 语义、基准测试环境变量，以及 init 和
release_inherited_host_tensor_refs 等方法细节。保持现有中文术语和示例风格，并确保生产工作流与英文文档涵盖相同内容。
- Around line 37-56: Update the Python examples in the distributed execution
documentation to import torch before any use of torch.float16, torch.randn,
torch.zeros_like, torch.zeros, or torch.float32, including the snippets around
compiled.prepare and the One-Shot example.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: ea522659-4081-446c-8a09-a73af22877b4
📒 Files selected for processing (13)
docs/en/user/distributed/00-model.mddocs/en/user/distributed/01-collectives.mddocs/en/user/distributed/02-primitives.mddocs/en/user/distributed/03-execution.mddocs/en/user/distributed/04-debugging.mddocs/en/user/distributed/index.mddocs/zh/user/distributed/00-model.mddocs/zh/user/distributed/01-collectives.mddocs/zh/user/distributed/02-primitives.mddocs/zh/user/distributed/03-execution.mddocs/zh/user/distributed/04-debugging.mddocs/zh/user/distributed/index.mdmkdocs.yml
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 250e60b104
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
- 01-collectives.md (en+zh): "five" -> "six" collectives; AllReduce description said "summed result" but the op supports Sum/Max/Min/Prod -> "reduced result"; qualified the "HOST builtins handle signal allocation automatically" claim as allreduce-only (the other five collectives always require a caller-allocated signal, verified against tensor_ops.py's overloads). - 02-primitives.md (en): the "Barrier in Isolation" example claimed N ranks write the same signal cell and that's why AtomicAdd is needed; with offsets=[my_rank, 0] each cell actually has exactly one writer, so Set would work identically. Corrected the explanation. Also corrected the System Substrate table: only get_comm_ctx (and world_size) work in HOST orchestrator code -- rank/nranks/notify/wait have codegen support only in InCore kernel code (verified: no orchestration_codegen.cpp handler for any of the four). - 04-debugging.md (en+zh): replaced vague "output in span-tree" wording with the real DFX artifact paths (dfx_outputs/l2_swimlane_records.json, dfx_outputs/deps.json), verified against docs/en/dev/03-runtime-dfx.md. - 02-primitives.md (zh): added the missing alloc_window_buffer shape+dtype convenience overload row. - 03-execution.md (zh): backfilled the sections missing versus the English page -- StackedDeviceTensor, multi-program worker semantics, release_inherited_host_tensor_refs, benchmark env vars paragraph -- and added the missing `import torch` in code examples.
ReviewDocs-only PR, and the quality bar is high. I spot-checked the API claims against the implementation and the STs rather than reading them as prose, and most of them hold:
Findings below, roughly in priority order. A. Link the collectives/primitives examples to the ST reference casesThe snippets in Rather than inflating them into runnable programs, I'd say explicitly that they are sketches and link the runnable ST. The repo already has a clean 1:1 mapping onto the three execution paths your own table describes:
(all under Same for This also closes the dangling "factory function pattern" reference in the Ring Mode section. B.
|
- 01-collectives.md, 02-primitives.md (en+zh): label the illustrative code sketches as such (they omit nranks/my_rank derivation and buffer setup) and add a Runnable Examples table linking each snippet to its tests/st/distributed/ counterpart. Filled in the dangling "factory function pattern" reference in Ring Mode with the actual pattern (outer function derives total_rounds, defines @pl.program in its own body) and a pointer to the ring intrinsic test. - 00-model.md, 03-execution.md (en+zh): added a driver snippet (DistributedConfig -> ir.compile -> dispatch) after the Quickstart class so "python script.py" has a referent, and turned 03-execution's duplicate CLI Launch section into a back-link. - 02-primitives.md (zh): closed content gaps vs. the English page -- Put/Get signature tables, the missing pld.get_comm_ctx namespace row, pld.put/pld.get in the no-short-form list, the handshake's expected- output paragraph, remote_load's misalignment warning, and the AtomicAdd-vs-Set corrective paragraph in the barrier section. Named the namespace on the Put and Get heading (en+zh). - 00-model.md (en+zh): fixed the Line-by-Line Walkthrough's AtomicAdd rationale, which claimed N ranks write the same signal slot -- with offsets=[my_rank, 0] each cell has exactly one writer, matching what 02-primitives.md already said about the same code. - 03-execution.md (en+zh): completed prepare()'s signature (extra_compiled, persistent, reset_persistent_windows, sub_worker_overrides) and added a section explaining what config/persistent actually do, qualified 00-model.md's "no persistent IPC" claim accordingly, and switched the DistributedWorker(compiled) examples to compiled.prepare() to match the documented entry point. - 04-debugging.md (en+zh): rewrote the device_ids/device= pitfall -- device_ids are physical card IDs (need not be contiguous or start at 0); device=r is a logical rank index validated against [0, world); the actual invariant is that the dispatch loop's trip count must equal len(device_ids).
|
@YunjiQin thanks for the thorough review — fixed all seven in
Ran |
Every example previously used the class-based @pl.program/@pl.function authoring API, which requires a two-step ir.compile()+compiled(...) call just to run. Rewrite to the free-function @pl.jit family instead (@pl.jit.incore/@pl.jit/@pl.jit.host), matching the decorator this manual's readers will actually copy-paste and run, and matching the convention already used in examples/runtime/multi_program_kv_cache.py.
| ) | ||
|
|
||
| # 3. Read the received tag back out. | ||
| received = pl.load(signal, [0, 0], [1, 1]) |
There was a problem hiding this comment.
use pl.read/write for single element operation
There was a problem hiding this comment.
Fixed in 5408b3e (en + zh) — the handshake now uses pl.read(signal, [0, 0]) / pl.write(out, [0, 0], received) for the single-element access, matching tests/st/distributed/test_l3_notify_wait.py (scalar read/write avoids the 32-byte tile-alignment constraint on the 4-byte INT32 signal cell). Verified with markdownlint-cli2, pre-commit (en/zh parity + nav + english-only), and the docs build.
Single-element access to the 4-byte INT32 signal cell avoids the 32-byte tile-alignment constraint that a pl.load/pl.store of a [1, 1] tile would hit, matching tests/st/distributed/test_l3_notify_wait.py.
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a new “Distributed Programming” chapter to the user docs (EN + ZH) and wires it into the MkDocs navigation, covering the distributed model, collectives, primitives, execution/worker lifecycle, and debugging guidance.
Changes:
- Added
docs/en/user/distributed/with five new pages (index + 00–04). - Added a mirrored
docs/zh/user/distributed/chapter with the same structure/content. - Updated
mkdocs.ymlnav and i18n label mapping to surface the new chapter.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| mkdocs.yml | Adds “Distributed” to nav and provides zh translation label for the nav item. |
| docs/en/user/distributed/index.md | Introduces the distributed model, L2/L3 vocabulary, and reading path. |
| docs/en/user/distributed/00-model.md | End-to-end quickstart + conceptual model (HOST orchestration, windows, signals). |
| docs/en/user/distributed/01-collectives.md | Documents built-in collectives, modes/constraints, and runnable examples. |
| docs/en/user/distributed/02-primitives.md | Documents low-level primitives (notify/wait, RMA, put/get) and constraints. |
| docs/en/user/distributed/03-execution.md | Documents DistributedWorker, persistence modes, and env vars. |
| docs/en/user/distributed/04-debugging.md | Documents failure patterns, pitfalls, and diagnostic flags. |
| docs/zh/user/distributed/index.md | Chinese mirror of the distributed model overview + reading path. |
| docs/zh/user/distributed/00-model.md | Chinese mirror of the quickstart + conceptual model. |
| docs/zh/user/distributed/01-collectives.md | Chinese mirror of collectives documentation + examples list. |
| docs/zh/user/distributed/02-primitives.md | Chinese mirror of primitives documentation + constraints. |
| docs/zh/user/distributed/03-execution.md | Chinese mirror of DistributedWorker execution/lifecycle + env vars. |
| docs/zh/user/distributed/04-debugging.md | Chinese mirror of debugging/pitfalls + diagnostic flags. |
| | Symptom | Likely Cause | Fix | | ||
| | ------- | ------------ | --- | | ||
| | **All ranks hang** | Notify/wait ordering — a rank is waiting on a peer that hasn't notified yet | Ensure every rank calls `notify` before any rank calls `wait`. The notify loop should precede the wait loop. | | ||
| | **Silent data corruption** | `remote_load` offsets or shape don't match what the peer stored | Verify offsets align with the peer's store offsets. A 1-element shift introduces a full row of garbage. | | ||
| | **Signal cell never reaches expected value** | Wrong `NotifyOp`: used `Set` instead of `AtomicAdd` for a multi-participant barrier | Use `AtomicAdd` when N ranks contribute to the same slot; use `Set` for 1:1 exchanges. | |
| | 症状 | 可能原因 | 修复 | | ||
| | ---- | -------- | ---- | | ||
| | **所有 rank 挂起** | notify/wait 顺序错误 | 确保 notify 循环在 wait 循环之前。 | | ||
| | **静默数据损坏** | `remote_load` offsets 或 shape 不匹配 | 验证 offsets 与对端的 store offsets 对齐。 | | ||
| | **Signal cell 永不达到期望值** | 错误 `NotifyOp` | 多参与者屏障用 `AtomicAdd`；1:1 交换用 `Set`。 | |
Standardize the chapter on the @pl.jit DSL form: the ring-mode text now presents the compile-time-NR factory pattern with @pl.jit (the specializer folds the closure constants), and a DSL-form note frames @pl.program as the class-based equivalent used by older tests under tests/st/distributed/.
|
Quick note on the latest push (
Verified with markdownlint-cli2, |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Suppressed comments (3)
docs/en/user/distributed/index.md:19
- Markdown table rows starting with
||create an empty first column (the first cell is blank), which will render as a misaligned 4-column table in standard Markdown. Use a single leading|instead (and keep the header separator aligned). This|| ...pattern appears across the new distributed docs (EN/ZH), so it should be fixed consistently to avoid broken table rendering.
| Layer | Scope | API namespace |
| ----- | ----- | ------------- |
| L2 | Single-device (one NPU chip) | `pl.*` |
| L3 | Cross-rank (multiple NPUs or processes) | `pld.*` |
docs/en/user/distributed/04-debugging.md:15
- The suggested fix is internally contradictory: it recommends allocating a fresh signal per iteration, but then states allreduce inside
for/whileis rejected. Clarify the actual allowed pattern (e.g., whether only signal-synthesized host allreduce is rejected in loops, or whether all forms are rejected), and update the fix text accordingly. Please mirror the clarification in the zh version as well to keep parity.
| **Allreduce rejected inside loop** | Signal protocol can't inject a fresh buffer per iteration | Allocate a fresh signal buffer for each allreduce call outside loops; allreduce inside `for`/`while` is currently rejected. |
docs/en/user/distributed/01-collectives.md:31
- This statement hard-codes a specific barrier signaling scheme (AtomicAdd/Ge on
[NR, 1]), but elsewhere in the chapter/primitives you describe cases whereSetis sufficient (e.g., dedicated slots/rows per writer). Consider rephrasing to describe the required property (a global sync on a signal tensor of shape[NR, 1]) and mention the op choice depends on whether the protocol uses shared slots vs per-writer slots. Please keep the zh mirror aligned.
- One global barrier per call (AtomicAdd/Ge on `[NR, 1]` signal)
…nual Split out of PR hw-native-sys#2162 — the final piece, wiring the two new chapters (hw-native-sys#2219, hw-native-sys#2220) into the existing manual: - index.md (+ zh): new Contents/capability-table rows for the two chapters, a 4th reading path ("I want to run across multiple devices"), dropped the now-superseded dev-doc pointer for distributed collectives. - 00-getting_started.md (+ zh): short pointer paragraphs into the new chapters from the existing benchmark/distributed sections. - 02-operation_reference.md (+ zh): new "Distributed / Collective Operations" capability matrix cross-linking into the distributed chapter. Depends on hw-native-sys#2219 (Distributed) and hw-native-sys#2220 (Performance) merging first — this branch references pages that only exist once both land, so `mkdocs build --strict` will fail here until then. Verified together with both dependencies via a local integration merge (all green: markdownlint, EN/ZH parity, nav, pre-commit, mkdocs build --strict); opening this now so the full picture of the split is visible.
…nual Split out of PR hw-native-sys#2162 — the final piece, wiring the two new chapters (hw-native-sys#2219, hw-native-sys#2220) into the existing manual: - index.md (+ zh): new Contents/capability-table rows for the two chapters, a 4th reading path ("I want to run across multiple devices"), dropped the now-superseded dev-doc pointer for distributed collectives. - 00-getting_started.md (+ zh): short pointer paragraphs into the new chapters from the existing benchmark/distributed sections. - 02-operation_reference.md (+ zh): new "Distributed / Collective Operations" capability matrix cross-linking into the distributed chapter. Depends on hw-native-sys#2219 (Distributed) and hw-native-sys#2220 (Performance) merging first — this branch references pages that only exist once both land, so `mkdocs build --strict` will fail here until then. Verified together with both dependencies via a local integration merge (all green: markdownlint, EN/ZH parity, nav, pre-commit, mkdocs build --strict); opening this now so the full picture of the split is visible.
Summary
Split out of #2162 (which is being broken into smaller PRs — see that
PR's description). Adds
docs/en/user/distributed/(+ fulldocs/zh/mirror) and wires it into
mkdocs.ymlnav.index.md— PyPTO's distributed model in one page: symmetricmemory + signals, comm domains, and how PyPTO's simplified L2
(single-chip) / L3 (cross-rank) vocabulary relates to the
simplerruntime's own finer L0–L6 hierarchy.
00-model.md— A minimal end-to-endHelloAllReducewalkthrough:host orchestrator, per-chip orchestration function, InCore kernel,
window buffers, and the launch model (a distributed program launches
as plain
python script.py— PyPTO forks the per-chip workersinternally).
01-collectives.md— Every collective (allreduce,barrier,broadcast,allgather,reduce_scatter,all_to_all), the threeexecution paths (InCore hand-rolled, InCore composite, HOST builtin)
and when to reach for each, supported
ReduceOp/dtype per operation,worked examples.
02-primitives.md— Lower-level building blocks:notify/waithandshakes, window buffer allocation, tile-level RMA, one-sided
put/getwith chunking/pipelining constraints, composingprimitives into a hand-rolled collective.
03-execution.md—DistributedWorkerlifecycle,DistributedConfig, dispatch.04-debugging.md— Common failure modes with symptoms and fixes.Note: this chapter cross-references a Performance chapter that is
being split out as a separate PR. Those specific links are omitted
here and will be added back once that PR (and this one) land — see
#2162 for the full picture of how the pieces fit together.
Verification
npx markdownlint-cli2 --config tests/lint/.markdownlint.yamlpython3 tests/lint/check_docs_en_zh_parity.pypython3 tests/lint/check_docs_nav.pypre-commit run --all-filesmkdocs build --strict(clean, no broken links)