diff --git a/docs/README.md b/docs/README.md index bcaefdff48..8d51742fd7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -43,7 +43,7 @@ changing simpler's own internals. | [Orchestrator](orchestrator.md) | DAG submission: TensorMap, Scope, Ring, task state machine | | [Scheduler](scheduler.md) | DAG dispatch: wiring / ready / completion queues, dispatch loop | | [Worker Manager](worker-manager.md) | Worker pool, THREAD/PROCESS modes, fork + mailbox mechanics | -| [hardware/](hardware/) | Hardware substrate: chip architecture, cache coherency, MMIO performance, CANN source references | +| [hardware/](hardware/README.md) | Hardware substrate: chip architecture, cache coherency, MMIO performance, CANN source references | ## Kernels and task authoring @@ -74,7 +74,7 @@ changing simpler's own internals. | [L3-L2 Message Queue](l3-l2-message-queue.md) | The queue channel between an L3 host and L2 | | [Directed NEXT_LEVEL Scheduling](directed-next-level-scheduling.md) | Targeting a specific next-level child instead of any free one | | [Remote L3 Worker Design](remote-l3-worker-design.md) | L4 host-to-host workers — protocol, transports, status | -| [remote-l3-worker-design/](remote-l3-worker-design/) | Full design set: protocol, buffers and transports, implementation plan and record | +| [remote-l3-worker-design/](remote-l3-worker-design/README.md) | Full design set: protocol, buffers and transports, implementation plan and record | ## Profiling, logging, and DFX @@ -97,5 +97,5 @@ changing simpler's own internals. | Document | What it covers | | -------- | -------------- | -| [troubleshooting/](troubleshooting/) | Device error codes, local timeout defaults, AICPU shared-SO faults, sim oversubscription hangs, macOS build issues, cpput ABI issues | +| [troubleshooting/](troubleshooting/README.md) | Device error codes, local timeout defaults, AICPU shared-SO faults, sim oversubscription hangs, macOS build issues, cpput ABI issues | | [investigations/](investigations/README.md) | Considered-and-dropped proposals and measured dead ends. **Check here before proposing an optimization or refactor** | diff --git a/docs/dfx/README.md b/docs/dfx/README.md index 3f9b687787..80bc8111eb 100644 --- a/docs/dfx/README.md +++ b/docs/dfx/README.md @@ -47,4 +47,4 @@ Analysis CLIs that consume these outputs are documented in | Document | What it covers | | -------- | -------------- | | [Log System](../logging.md) | Log levels and sinks — the host/device logging path, distinct from profiling collectors | -| [troubleshooting/](../troubleshooting/) | Device error codes and timeout defaults, for when a run fails rather than runs slowly | +| [troubleshooting/](../troubleshooting/README.md) | Device error codes and timeout defaults, for when a run fails rather than runs slowly | diff --git a/docs/hardware/README.md b/docs/hardware/README.md new file mode 100644 index 0000000000..820bffac6f --- /dev/null +++ b/docs/hardware/README.md @@ -0,0 +1,28 @@ +# Hardware + +Every hardware-level reference lives in this directory. Start with the chip +architecture overview if you need the host/chip model, or jump to the +cache-coherency and MMIO references when working on a data or notification +path. + +## Architecture and memory + +| Document | What it covers | +| -------- | -------------- | +| [Ascend Chip Architecture](chip-architecture.md) | Host, AICPU, and AICore execution tiers; chip generations; task flow; memory hierarchy | +| [Cache Coherency](cache-coherency.md) | GM visibility across AICore, AICPU, DMA, and SDMA paths, including required barriers and cache operations | + +## Control-path investigation + +| Document | What it covers | +| -------- | -------------- | +| [MMIO Performance](mmio-performance.md) | AIC control-register memory attributes, measured access costs, concurrency limits, and notification-channel tradeoffs | +| [CANN Source References](cann-source-references.md) | Upstream driver, HCCL, and HCOMM source locations used to investigate hardware and communication behavior | + +## Related, outside this directory + +| Document | What it covers | +| -------- | -------------- | +| [Chip-Level Architecture](../chip-level-arch.md) | The three-program software model layered onto the host, AICPU, and AICore tiers | +| [Task Flow](../task-flow.md) | End-to-end callable, argument, and task flow across the execution tiers | +| [AICore Kernel Programming](../aicore-kernel-programming.md) | Kernel authoring rules for the compute tier described here | diff --git a/docs/remote-l3-worker-design/README.md b/docs/remote-l3-worker-design/README.md new file mode 100644 index 0000000000..34472b712f --- /dev/null +++ b/docs/remote-l3-worker-design/README.md @@ -0,0 +1,37 @@ +# Remote L3 Worker Design References + +Detailed protocol, data-movement, delivery, and audit references live in this +directory. Start with the +[Remote L3 Worker Design](../remote-l3-worker-design.md) for the architecture +and current implementation status, then use the references below for a +specific layer or delivery stage. + +## Protocol and data movement + +| Document | What it covers | +| -------- | -------------- | +| [Remote L3 Protocol](protocol.md) | Versioned wire frames, payloads, ordering, controls, and bounds validation | +| [Remote L3 Buffers and Transports](buffers-and-transports.md) | Remote buffer ownership and lifetime, tensor sidecars, transport profiles, and simulation behavior | + +## Delivery status + +| Document | What it covers | +| -------- | -------------- | +| [Remote L3 Implementation Plan](implementation-plan.md) | Incremental delivery sequence, required tests, hardware-gated work, and open decisions | +| [Remote L3 Implementation Record](implementation-record.md) | Feature-by-feature implementation status and verification evidence | + +## PR split and audit + +| Document | What it covers | +| -------- | -------------- | +| [PR Split and Audit Plan](pr-split-and-audit-plan.md) | Strategy for splitting and reviewing the remote L3 stack without losing contract coverage | +| [PR Split and Audit Artifacts](pr-split-and-audit-artifacts.md) | Recorded split inputs, compliance matrix, risk register, and verification log | + +## Related, outside this directory + +| Document | What it covers | +| -------- | -------------- | +| [Callable Identity Registration](../callable-identity-registration.md) | Stable callable identity and target-private execution slots used by remote routing | +| [Dynamic Callable Registration over IPC](../callable-ipc-dynamic-register.md) | Registration lifecycle and control semantics shared with hierarchical workers | +| [Python Callable Serialization](../python-callable-serialization.md) | Serialization contract for negotiated remote Python callable payloads | +| [Communication Domains](../comm-domain.md) | Deferred-release communication resources that remote domains build on | diff --git a/docs/troubleshooting/README.md b/docs/troubleshooting/README.md new file mode 100644 index 0000000000..2edf3a62fe --- /dev/null +++ b/docs/troubleshooting/README.md @@ -0,0 +1,30 @@ +# Troubleshooting + +Every troubleshooting reference lives in this directory. Start with the device +error-code reference when a run fails, or jump to the environment-specific +guide when the failure happens during build, test setup, or simulation. + +## Runtime and device failures + +| Document | What it covers | +| -------- | -------------- | +| [Device Error Codes](device-error-codes.md) | Runtime and CANN error classification, first-response triage, minimal reproductions, and links to focused diagnostic guides | +| [Local Runtime Timeouts](local-timeout-defaults.md) | Local and CI timeout defaults, override variables, and the required ordering for onboard watchdogs | +| [a2a3 AICPU Shared-SO Device Fault](a2a3-507899-aicpu-shared-so-fault.md) | Diagnosing mass `507899`/`507018` cascades caused by an AICPU shared-library device fault | +| [Sim CPU Oversubscription](sim-oversubscription-hang.md) | Simulation hangs and false timeouts when AICPU and AICore host threads are CPU-starved | + +## Build and test environments + +| Document | What it covers | +| -------- | -------------- | +| [macOS Build and Lint Toolchain](macos-build.md) | Compiler, SDK, sysroot, and `clang-tidy` behavior in the macOS build pipeline | +| [macOS libomp Collision](macos-libomp-collision.md) | Duplicate OpenMP runtime crashes in single-process simulation CI and their mitigation | +| [C++ Unit-Test gtest ABI Mismatch](ut-cpp-gtest-abi.md) | Link failures caused by a system GoogleTest library built with the old libstdc++ ABI | + +## Related, outside this directory + +| Document | What it covers | +| -------- | -------------- | +| [DFX and Profiling](../dfx/README.md) | Timing, traces, dependencies, and resource-usage evidence for runs that complete or make partial progress | +| [Testing](../testing.md) | Test types, commands, platform selection, and failure-reproduction workflows | +| [Debug a Failed Run](../user/how-to/debug-a-failed-run.md) | User-facing first-response path from the surfaced error to the focused guides in this directory | diff --git a/docs/troubleshooting/device-error-codes/README.md b/docs/troubleshooting/device-error-codes/README.md new file mode 100644 index 0000000000..68cda7a43a --- /dev/null +++ b/docs/troubleshooting/device-error-codes/README.md @@ -0,0 +1,27 @@ +# Device Error Code Diagnostics + +Focused procedures for chasing runtime and device failures live in this +directory. Start with [Device Error Codes](../device-error-codes.md) to identify +the reported code or watchdog signature, then use the matching guide below. + +## Capacity and progress failures + +| Document | What it covers | +| -------- | -------------- | +| [Capacity Codes](capacity.md) | Diagnosing codes 1, 2, and 4 with scope resource peaks and distinguishing structural deadlock from stalled reclaim | +| [Stalls](stall.md) | Diagnosing scheduler timeout code 100 and tensor-wait code 8 by ordering watchdogs and locating the stuck task | + +## Core faults and defensive codes + +| Document | What it covers | +| -------- | -------------- | +| [AICore Faults](aicore-fault.md) | Separating kernel addressing faults from control-flow corruption using device logs and static inspection | +| [Codes Without End-to-End Tests](untested.md) | Why codes 10, 11, 103 and stall classes S4, S5, and unknown cannot be triggered through the public API | + +## Related, outside this directory + +| Document | What it covers | +| -------- | -------------- | +| [Local Runtime Timeouts](../local-timeout-defaults.md) | Default watchdog ordering and local overrides used to expose the most useful failure code | +| [Debug a Failed Run](../../user/how-to/debug-a-failed-run.md) | User-facing first-response workflow for collecting and classifying failure evidence | +| [L0 Swimlane Profiling](../../dfx/l0-swimlane-profiling.md) | Intra-core task timing for kernels that run but are unexpectedly slow | diff --git a/docs/user/how-to/debug-a-failed-run.md b/docs/user/how-to/debug-a-failed-run.md index cf0ab9bc57..ace016877d 100644 --- a/docs/user/how-to/debug-a-failed-run.md +++ b/docs/user/how-to/debug-a-failed-run.md @@ -25,7 +25,7 @@ mechanism before you theorize: 2. **Then look up the code** in [device-error-codes](../../troubleshooting/device-error-codes.md), the repo's own table, with per-mechanism notes in - [`device-error-codes/`](../../troubleshooting/device-error-codes/): + [`device-error-codes/`](../../troubleshooting/device-error-codes/README.md): [aicore-fault](../../troubleshooting/device-error-codes/aicore-fault.md), [capacity](../../troubleshooting/device-error-codes/capacity.md), [stall](../../troubleshooting/device-error-codes/stall.md). diff --git a/mkdocs.yml b/mkdocs.yml index e93ee0ea71..5019f34f40 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -35,6 +35,9 @@ validation: omitted_files: warn not_found: warn links: + # A directory link needs a README.md or index.md landing page. Otherwise + # MkDocs leaves the URL unchanged and the published site serves a 404. + unrecognized_links: warn # Defaults to `info`, which --strict ignores, so a link to a heading that # was renamed or never existed builds clean and 404s in the reader's # browser. Note the two slug dialects differ: GitHub keeps the doubled @@ -106,6 +109,7 @@ nav: - A5 AICPU core selection: design/a5-fg-pg-core-selection.en.md - Worker Manager: worker-manager.md - Hardware: + - Overview: hardware/README.md - Chip architecture: hardware/chip-architecture.md - Cache coherency: hardware/cache-coherency.md - MMIO performance: hardware/mmio-performance.md @@ -127,7 +131,9 @@ nav: - L3-L2 Orchestrator Comm: l3-l2-orch-comm.md - L3-L2 Message Queue: l3-l2-message-queue.md - Directed NEXT_LEVEL Scheduling: directed-next-level-scheduling.md - - Remote L3 Worker Design: remote-l3-worker-design.md + - Remote L3 Worker Design: + - Overview: remote-l3-worker-design.md + - Design document index: remote-l3-worker-design/README.md - Profiling and DFX: - Overview: dfx/README.md - Profiling Framework: dfx/profiling-framework.md @@ -154,7 +160,10 @@ nav: - Sanitizers: sanitizers.md - Sim Multi-Device Isolation: sim-multi-device-isolation.md - Troubleshooting: - - Device error codes: troubleshooting/device-error-codes.md + - Overview: troubleshooting/README.md + - Device error codes: + - Overview and code reference: troubleshooting/device-error-codes.md + - Diagnostic guide index: troubleshooting/device-error-codes/README.md - Local timeout defaults: troubleshooting/local-timeout-defaults.md - a2a3 507899 AICPU shared-SO fault: troubleshooting/a2a3-507899-aicpu-shared-so-fault.md - Sim oversubscription hang: troubleshooting/sim-oversubscription-hang.md