Fix: make SDMA a declared capability, and drop stale ci.yml line refs - #1609
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe PR introduces the ChangesIsolated device test selection
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/testing/SKILL.md:
- Around line 71-73: Update the testing guide’s instructions around the hardware
scene test command to remove the stale --ignore extraction and grep steps.
Direct readers to use the established pytest marker contract, excluding
isolated_device tests with -m "not isolated_device" so local execution matches
CI.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c5ef5398-7c79-493c-92a1-b2321bb34d77
📒 Files selected for processing (6)
.claude/skills/testing/SKILL.md.github/workflows/ci.ymlconftest.pydocs/capability-survey.mdexamples/a2a3/tensormap_and_ringbuffer/prefetch_async_demo/test_prefetch_async_demo.pyexamples/a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py
94d8dac to
cff90f7
Compare
3cde730 to
764a47a
Compare
The SDMA quarantine was a pair of `--ignore=<path>` arguments. `pytest --ignore` on a path that no longer exists **exits 0 and says nothing** — measured — so moving either demo directory would have dropped the quarantine silently and landed both tests back beside the `aicore_op_timeout` fault-injection test. That collision is the whole hazard: provisioning the SDMA workspace creates 48 device-only STARS streams that sit in the device fault domain, so a later AICore fault on that device costs ~306 s instead of ~0.3 s (hw-native-sys#1425, contained in hw-native-sys#1406). `@pytest.mark.sdma` replaces the path matching, and carries the whole consequence rather than one arbitrary piece of it. One declaration now drives three things: - **The Worker is built with `enable_sdma=True`.** Both pytest construction sites in `conftest.py` and both standalone sites in `scene_test.py` read it, the latter from `cls.pytestmark` so `python test_x.py` behaves like the pytest path. - **The L2 Worker pool stops mixing capabilities.** The key gains the flag and reuse tests it, because an enable_sdma Worker holds its STARS streams for life: handing it to a test that did not ask for them would spread the teardown hazard to every later L2 case on that device. The existing same-device retire loop then performs the swap, so no new teardown path. - **SDMA sorts last.** `sort_key` gains a term, keyed off the marker rather than the class because the fault-injection tests are plain functions with no `_st_level`. This is what will make merging the dedicated CI step back into the sweep safe once hw-native-sys#1425 is fixed; today `-m` already separates them, so it matters for local full runs. With the capability expressible, `prefetch_async_demo` becomes an ordinary L2 `@scene_test` class. It was a hand-rolled Worker only because `CASES` had no channel to `Worker.__init__`, and it paid for that by forfeiting golden comparison, case parametrization, `--rounds`, `--case` and the dispatcher's device allocation. 160 lines become 89, and its verification — `torch.equal(out, src)` — is now a one-line `compute_golden`. The framework's default orchestration includes turned out to be sufficient, so no new passthrough was needed there. The conversion also moves it from the Resource phase to the L2 phase, which runs after it, so it provisions on a device the fault-injection cases have already finished with. `sdma_async_completion_demo` keeps its hand-rolled L3 Worker for now and takes the marker for ordering and CI selection only; converting it means expressing a comm domain through `CASES`, which is a larger change and independent of this one. CI selects with `-m sdma` / `-m "not sdma"`, so the sweep and the dedicated step can no longer drift apart. **The dedicated step stays until hw-native-sys#1425 is fixed**: the two phases run their jobs in parallel across devices, so only separate device pools keep an SDMA provisioning away from a fault injection. `.claude/skills/testing/SKILL.md` moves with the mechanism throughout — not only the mirror-CI command, but the two instructions that told readers to extract `--ignore` sets from `ci.yml` and to grep it when a test passes alone and fails in the sweep. Both now name the marker; following the skill reproduces CI rather than a mechanism that no longer exists. Separately, `docs/capability-survey.md` carried four `ci.yml:<line>` citations, three already invalidated by this session's edits — `:607` lands on a blank line, `:884` on a `cmake --build`, `:628-643` on a dep_gen comment, after the qwen step was deleted in hw-native-sys#1601 and `detect-changes` reworked in hw-native-sys#1589 / hw-native-sys#1590 / hw-native-sys#1607. Line numbers into a file this active cannot be maintained, so all four become greppable anchors. `grep -rIn 'ci\.yml:[0-9]'` is now empty repo-wide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The chip-init path makes direct driver calls that bypass ACL (halMemCtl / halGetDeviceInfo* on a2a3, halResMap / halGetDeviceInfo* / dsmi_get_device_info on a5), so they index the driver-visible space and must translate through acl_to_hal_device_id. Nothing exercised that translation: no CI job sets ASCEND_RT_VISIBLE_DEVICES, so every job ran the identity path and a call site that forgot to translate stayed green. tests/st/vis_isolation runs dummy_task in a subprocess with the granted card exposed as logical 0. init_aicore_register_addresses and probe_aicpu_topology run on every onboard bring-up, so one scene test covers every translated site on both arches. The subprocess is load-bearing: the variable must be set before ACL init, and mutating it in-process would leak into the session's pooled workers. Carrying this as a scene test rather than the two ci.yml steps keeps the selection in the test tree, so the sweep and the check cannot drift apart -- the direction hw-native-sys#1601 and hw-native-sys#1609 took for qwen and SDMA. The onboard jobs already run `pytest examples tests/st`, and the platforms marker keeps it off the sim runners, where a driver-level remap has nothing to verify.
`st-onboard-a2a3` has been two pytest invocations since #1609 — the sweep with `-m "not sdma"` and a later step with `-m sdma`, so that no fault-injection case meets a device that has already provisioned SDMA (#1425). Three places that claim to reproduce that job still issued one flat sweep, which is the invocation `.claude/skills/testing/SKILL.md` warns "is *not* what CI runs and will report failures that CI never sees". - **`test-all-device`** describes itself as running "the full hardware (onboard) CI pipeline" and already reads `--pto-session-timeout` out of the `st-onboard-<platform>` job. It now reads the marker expression from the same place, and runs the second pass after the first. Taking both from `ci.yml` rather than hardcoding them is what keeps the skill from drifting again. - **`test-runtime-device`** likewise. Both quarantined tests are `tensormap_and_ringbuffer`, so the second pass is noted as needed only when that runtime is selected. - **`docs/ci.md`'s "Recommended CI invocation"** block, which #1617 missed while correcting the job table three sections above it. The a5 line is unchanged — `st-onboard-a5` carries no marker filter. Left alone: the recipes that describe running the tests locally rather than mirroring a job (`README.md`, `docs/testing.md`, `docs/user/reference/cli.md`, `examples/README.md`, the `SUBMIT_BY_CLUSTER.md` trio). Whether a local sweep should exclude SDMA by default is a question about the documented default, not about matching CI, and `testing/SKILL.md` already tells anyone who trips over it what happened.
The quarantine failed silently
It was a pair of
--ignore=<path>arguments. Measured:pytest <path>pytest --ignore=<path>So moving either demo directory would have dropped the quarantine without a word, landing both tests back beside the
aicore_op_timeoutfault-injection test. That collision is the hazard: provisioning the SDMA workspace creates 48 device-only STARS streams in the device fault domain, so a later AICore fault there costs ~306 s instead of ~0.3 s (#1425, contained in #1406).@pytest.mark.sdma— one declaration, three consequencesReplacing the path match with a marker is not just a sturdier selector. The capability implies the isolation, so one declaration should carry all of it:
enable_sdma=Trueconftest.py, both standalone sites inscene_test.py(viacls.pytestmark, sopython test_x.pymatches the pytest path)enable_sdmaWorker holds its streams for life, so handing it to a test that never asked would spread the teardown hazard to every later L2 case on that devicesort_keygains a term, keyed off the marker because the fault-injection tests are plain functions with no_st_levelThe ordering is what actually addresses the interaction, rather than merely quarantining it: fault injection now always runs on a device that has never provisioned. Verified:
The pool swap needs no new teardown path — the existing same-device retire loop (
stale_key[1] != dev_id → continue; retire) already does it, and index 1 is stilldev_id.prefetch_async_demobecomes an ordinary L2 scene testIt was a hand-rolled
Workeronly becauseCASEShad no channel toWorker.__init__—Worker(level, **config)takesenable_sdmaas a plain config key, but@scene_testhardcodes the constructor call andCASES["config"]feedsCallConfig(per-run), not the Worker (per-init).It paid for that by forfeiting golden comparison, case parametrization,
--rounds,--caseand the dispatcher's device allocation. 160 lines → 89, and its verification collapses to one line:The framework's default orchestration includes turned out to be sufficient (checked by compiling it with no extras), so no further framework change was needed.
sdma_async_completion_demokeeps its hand-rolled L3 Worker and takes the marker for ordering and CI selection only — converting it means expressing a comm domain throughCASES, a larger and independent change.CI
-m sdma/-m "not sdma", so the sweep and the dedicated step can no longer drift apart.The dedicated step stays until #1425 is fixed. In-session ordering separates the cases, but a fault on a device that has already provisioned still costs minutes, so they must not share a device. Merging it back is the follow-up once the underlying bug is closed.
Also: stale
ci.yml:<line>citationsdocs/capability-survey.mdcarried four; three were already wrong, invalidated by this session's own edits:ci.yml:607ci.yml:884cmake --build buildci.yml:628-643All four become greppable anchors.
grep -rIn 'ci\.yml:[0-9]'is now empty repo-wide.Testing
--ignore=<missing>exit 0 silent;<missing path>exit 4key[1]and are length-agnosticprefetch_async_orch.cppcompiles with the framework's default orchestration includes only — 558 KiB, noextra_include_dirsneededast.parseonconftest.py,scene_test.py, the converted test;yaml.safe_loadonci.ymlgrep -rIn isolated_deviceempty;check_english_only.pyclean;markdownlint-cli2cleanst-onboard-a2a3on this PR is the real check, and the SDMA step is where the conversion proves out