Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ changing simpler's own internals.
| [Orchestrator](orchestrator.md) | DAG submission: TensorMap, Scope, Ring, task state machine |
| [Scheduler](scheduler.md) | DAG dispatch: wiring / ready / completion queues, dispatch loop |
| [Worker Manager](worker-manager.md) | Worker pool, THREAD/PROCESS modes, fork + mailbox mechanics |
| [hardware/](hardware/) | Hardware substrate: chip architecture, cache coherency, MMIO performance, CANN source references |
| [hardware/](hardware/README.md) | Hardware substrate: chip architecture, cache coherency, MMIO performance, CANN source references |

## Kernels and task authoring

Expand Down Expand Up @@ -74,7 +74,7 @@ changing simpler's own internals.
| [L3-L2 Message Queue](l3-l2-message-queue.md) | The queue channel between an L3 host and L2 |
| [Directed NEXT_LEVEL Scheduling](directed-next-level-scheduling.md) | Targeting a specific next-level child instead of any free one |
| [Remote L3 Worker Design](remote-l3-worker-design.md) | L4 host-to-host workers — protocol, transports, status |
| [remote-l3-worker-design/](remote-l3-worker-design/) | Full design set: protocol, buffers and transports, implementation plan and record |
| [remote-l3-worker-design/](remote-l3-worker-design/README.md) | Full design set: protocol, buffers and transports, implementation plan and record |

## Profiling, logging, and DFX

Expand All @@ -97,5 +97,5 @@ changing simpler's own internals.

| Document | What it covers |
| -------- | -------------- |
| [troubleshooting/](troubleshooting/) | Device error codes, local timeout defaults, AICPU shared-SO faults, sim oversubscription hangs, macOS build issues, cpput ABI issues |
| [troubleshooting/](troubleshooting/README.md) | Device error codes, local timeout defaults, AICPU shared-SO faults, sim oversubscription hangs, macOS build issues, cpput ABI issues |
| [investigations/](investigations/README.md) | Considered-and-dropped proposals and measured dead ends. **Check here before proposing an optimization or refactor** |
2 changes: 1 addition & 1 deletion docs/dfx/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,4 @@ Analysis CLIs that consume these outputs are documented in
| Document | What it covers |
| -------- | -------------- |
| [Log System](../logging.md) | Log levels and sinks — the host/device logging path, distinct from profiling collectors |
| [troubleshooting/](../troubleshooting/) | Device error codes and timeout defaults, for when a run fails rather than runs slowly |
| [troubleshooting/](../troubleshooting/README.md) | Device error codes and timeout defaults, for when a run fails rather than runs slowly |
28 changes: 28 additions & 0 deletions docs/hardware/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Hardware

Every hardware-level reference lives in this directory. Start with the chip
architecture overview if you need the host/chip model, or jump to the
cache-coherency and MMIO references when working on a data or notification
path.

## Architecture and memory

| Document | What it covers |
| -------- | -------------- |
| [Ascend Chip Architecture](chip-architecture.md) | Host, AICPU, and AICore execution tiers; chip generations; task flow; memory hierarchy |
| [Cache Coherency](cache-coherency.md) | GM visibility across AICore, AICPU, DMA, and SDMA paths, including required barriers and cache operations |

## Control-path investigation

| Document | What it covers |
| -------- | -------------- |
| [MMIO Performance](mmio-performance.md) | AIC control-register memory attributes, measured access costs, concurrency limits, and notification-channel tradeoffs |
| [CANN Source References](cann-source-references.md) | Upstream driver, HCCL, and HCOMM source locations used to investigate hardware and communication behavior |

## Related, outside this directory

| Document | What it covers |
| -------- | -------------- |
| [Chip-Level Architecture](../chip-level-arch.md) | The three-program software model layered onto the host, AICPU, and AICore tiers |
| [Task Flow](../task-flow.md) | End-to-end callable, argument, and task flow across the execution tiers |
| [AICore Kernel Programming](../aicore-kernel-programming.md) | Kernel authoring rules for the compute tier described here |
37 changes: 37 additions & 0 deletions docs/remote-l3-worker-design/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Remote L3 Worker Design References

Detailed protocol, data-movement, delivery, and audit references live in this
directory. Start with the
[Remote L3 Worker Design](../remote-l3-worker-design.md) for the architecture
and current implementation status, then use the references below for a
specific layer or delivery stage.

## Protocol and data movement

| Document | What it covers |
| -------- | -------------- |
| [Remote L3 Protocol](protocol.md) | Versioned wire frames, payloads, ordering, controls, and bounds validation |
| [Remote L3 Buffers and Transports](buffers-and-transports.md) | Remote buffer ownership and lifetime, tensor sidecars, transport profiles, and simulation behavior |

## Delivery status

| Document | What it covers |
| -------- | -------------- |
| [Remote L3 Implementation Plan](implementation-plan.md) | Incremental delivery sequence, required tests, hardware-gated work, and open decisions |
| [Remote L3 Implementation Record](implementation-record.md) | Feature-by-feature implementation status and verification evidence |

## PR split and audit

| Document | What it covers |
| -------- | -------------- |
| [PR Split and Audit Plan](pr-split-and-audit-plan.md) | Strategy for splitting and reviewing the remote L3 stack without losing contract coverage |
| [PR Split and Audit Artifacts](pr-split-and-audit-artifacts.md) | Recorded split inputs, compliance matrix, risk register, and verification log |

## Related, outside this directory

| Document | What it covers |
| -------- | -------------- |
| [Callable Identity Registration](../callable-identity-registration.md) | Stable callable identity and target-private execution slots used by remote routing |
| [Dynamic Callable Registration over IPC](../callable-ipc-dynamic-register.md) | Registration lifecycle and control semantics shared with hierarchical workers |
| [Python Callable Serialization](../python-callable-serialization.md) | Serialization contract for negotiated remote Python callable payloads |
| [Communication Domains](../comm-domain.md) | Deferred-release communication resources that remote domains build on |
30 changes: 30 additions & 0 deletions docs/troubleshooting/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Troubleshooting

Every troubleshooting reference lives in this directory. Start with the device
error-code reference when a run fails, or jump to the environment-specific
guide when the failure happens during build, test setup, or simulation.

## Runtime and device failures

| Document | What it covers |
| -------- | -------------- |
| [Device Error Codes](device-error-codes.md) | Runtime and CANN error classification, first-response triage, minimal reproductions, and links to focused diagnostic guides |
| [Local Runtime Timeouts](local-timeout-defaults.md) | Local and CI timeout defaults, override variables, and the required ordering for onboard watchdogs |
| [a2a3 AICPU Shared-SO Device Fault](a2a3-507899-aicpu-shared-so-fault.md) | Diagnosing mass `507899`/`507018` cascades caused by an AICPU shared-library device fault |
| [Sim CPU Oversubscription](sim-oversubscription-hang.md) | Simulation hangs and false timeouts when AICPU and AICore host threads are CPU-starved |

## Build and test environments

| Document | What it covers |
| -------- | -------------- |
| [macOS Build and Lint Toolchain](macos-build.md) | Compiler, SDK, sysroot, and `clang-tidy` behavior in the macOS build pipeline |
| [macOS libomp Collision](macos-libomp-collision.md) | Duplicate OpenMP runtime crashes in single-process simulation CI and their mitigation |
| [C++ Unit-Test gtest ABI Mismatch](ut-cpp-gtest-abi.md) | Link failures caused by a system GoogleTest library built with the old libstdc++ ABI |

## Related, outside this directory

| Document | What it covers |
| -------- | -------------- |
| [DFX and Profiling](../dfx/README.md) | Timing, traces, dependencies, and resource-usage evidence for runs that complete or make partial progress |
| [Testing](../testing.md) | Test types, commands, platform selection, and failure-reproduction workflows |
| [Debug a Failed Run](../user/how-to/debug-a-failed-run.md) | User-facing first-response path from the surfaced error to the focused guides in this directory |
27 changes: 27 additions & 0 deletions docs/troubleshooting/device-error-codes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Device Error Code Diagnostics

Focused procedures for chasing runtime and device failures live in this
directory. Start with [Device Error Codes](../device-error-codes.md) to identify
the reported code or watchdog signature, then use the matching guide below.

## Capacity and progress failures

| Document | What it covers |
| -------- | -------------- |
| [Capacity Codes](capacity.md) | Diagnosing codes 1, 2, and 4 with scope resource peaks and distinguishing structural deadlock from stalled reclaim |
| [Stalls](stall.md) | Diagnosing scheduler timeout code 100 and tensor-wait code 8 by ordering watchdogs and locating the stuck task |

## Core faults and defensive codes

| Document | What it covers |
| -------- | -------------- |
| [AICore Faults](aicore-fault.md) | Separating kernel addressing faults from control-flow corruption using device logs and static inspection |
| [Codes Without End-to-End Tests](untested.md) | Why codes 10, 11, 103 and stall classes S4, S5, and unknown cannot be triggered through the public API |

## Related, outside this directory

| Document | What it covers |
| -------- | -------------- |
| [Local Runtime Timeouts](../local-timeout-defaults.md) | Default watchdog ordering and local overrides used to expose the most useful failure code |
| [Debug a Failed Run](../../user/how-to/debug-a-failed-run.md) | User-facing first-response workflow for collecting and classifying failure evidence |
| [L0 Swimlane Profiling](../../dfx/l0-swimlane-profiling.md) | Intra-core task timing for kernels that run but are unexpectedly slow |
2 changes: 1 addition & 1 deletion docs/user/how-to/debug-a-failed-run.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ mechanism before you theorize:
2. **Then look up the code** in
[device-error-codes](../../troubleshooting/device-error-codes.md), the repo's
own table, with per-mechanism notes in
[`device-error-codes/`](../../troubleshooting/device-error-codes/):
[`device-error-codes/`](../../troubleshooting/device-error-codes/README.md):
[aicore-fault](../../troubleshooting/device-error-codes/aicore-fault.md),
[capacity](../../troubleshooting/device-error-codes/capacity.md),
[stall](../../troubleshooting/device-error-codes/stall.md).
Expand Down
13 changes: 11 additions & 2 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ validation:
omitted_files: warn
not_found: warn
links:
# A directory link needs a README.md or index.md landing page. Otherwise
# MkDocs leaves the URL unchanged and the published site serves a 404.
unrecognized_links: warn
# Defaults to `info`, which --strict ignores, so a link to a heading that
# was renamed or never existed builds clean and 404s in the reader's
# browser. Note the two slug dialects differ: GitHub keeps the doubled
Expand Down Expand Up @@ -106,6 +109,7 @@ nav:
- A5 AICPU core selection: design/a5-fg-pg-core-selection.en.md
- Worker Manager: worker-manager.md
- Hardware:
- Overview: hardware/README.md
- Chip architecture: hardware/chip-architecture.md
- Cache coherency: hardware/cache-coherency.md
- MMIO performance: hardware/mmio-performance.md
Expand All @@ -127,7 +131,9 @@ nav:
- L3-L2 Orchestrator Comm: l3-l2-orch-comm.md
- L3-L2 Message Queue: l3-l2-message-queue.md
- Directed NEXT_LEVEL Scheduling: directed-next-level-scheduling.md
- Remote L3 Worker Design: remote-l3-worker-design.md
- Remote L3 Worker Design:
- Overview: remote-l3-worker-design.md
- Design document index: remote-l3-worker-design/README.md
- Profiling and DFX:
- Overview: dfx/README.md
- Profiling Framework: dfx/profiling-framework.md
Expand All @@ -154,7 +160,10 @@ nav:
- Sanitizers: sanitizers.md
- Sim Multi-Device Isolation: sim-multi-device-isolation.md
- Troubleshooting:
- Device error codes: troubleshooting/device-error-codes.md
- Overview: troubleshooting/README.md
- Device error codes:
- Overview and code reference: troubleshooting/device-error-codes.md
- Diagnostic guide index: troubleshooting/device-error-codes/README.md
- Local timeout defaults: troubleshooting/local-timeout-defaults.md
- a2a3 507899 AICPU shared-SO fault: troubleshooting/a2a3-507899-aicpu-shared-so-fault.md
- Sim oversubscription hang: troubleshooting/sim-oversubscription-hang.md
Expand Down
Loading