Skip to content

Fix: make SDMA a declared capability, and drop stale ci.yml line refs - #1609

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:fix/sdma-quarantine-and-stale-line-refs
Jul 31, 2026
Merged

Fix: make SDMA a declared capability, and drop stale ci.yml line refs#1609
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:fix/sdma-quarantine-and-stale-line-refs

Conversation

@ChaoWao

@ChaoWao ChaoWao commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

The quarantine failed silently

It was a pair of --ignore=<path> arguments. Measured:

Invocation Path no longer exists
pytest <path> exit 4 — loud
pytest --ignore=<path> exit 0, no output — silent

So moving either demo directory would have dropped the quarantine without a word, landing both tests back beside the aicore_op_timeout fault-injection test. That collision is the hazard: provisioning the SDMA workspace creates 48 device-only STARS streams in the device fault domain, so a later AICore fault there costs ~306 s instead of ~0.3 s (#1425, contained in #1406).

@pytest.mark.sdma — one declaration, three consequences

Replacing the path match with a marker is not just a sturdier selector. The capability implies the isolation, so one declaration should carry all of it:

Consequence Where
Worker built with enable_sdma=True both pytest sites in conftest.py, both standalone sites in scene_test.py (via cls.pytestmark, so python test_x.py matches the pytest path)
L2 Worker pool stops mixing capabilities key gains the flag and reuse tests it — an enable_sdma Worker holds its streams for life, so handing it to a test that never asked would spread the teardown hazard to every later L2 case on that device
SDMA sorts last sort_key gains a term, keyed off the marker because the fault-injection tests are plain functions with no _st_level

The ordering is what actually addresses the interaction, rather than merely quarantining it: fault injection now always runs on a device that has never provisioned. Verified:

b_l3_case            ← L3 still first
aicore_op_timeout    ← fault injection before…
z_l2_case
a_sdma_completion    ← …any SDMA
a_sdma_prefetch

The pool swap needs no new teardown path — the existing same-device retire loop (stale_key[1] != dev_id → continue; retire) already does it, and index 1 is still dev_id.

prefetch_async_demo becomes an ordinary L2 scene test

It was a hand-rolled Worker only because CASES had no channel to Worker.__init__Worker(level, **config) takes enable_sdma as a plain config key, but @scene_test hardcodes the constructor call and CASES["config"] feeds CallConfig (per-run), not the Worker (per-init).

It paid for that by forfeiting golden comparison, case parametrization, --rounds, --case and the dispatcher's device allocation. 160 lines → 89, and its verification collapses to one line:

def compute_golden(self, args, params):
    args.out[:] = args.src

The framework's default orchestration includes turned out to be sufficient (checked by compiling it with no extras), so no further framework change was needed.

sdma_async_completion_demo keeps its hand-rolled L3 Worker and takes the marker for ordering and CI selection only — converting it means expressing a comm domain through CASES, a larger and independent change.

CI

-m sdma / -m "not sdma", so the sweep and the dedicated step can no longer drift apart.

The dedicated step stays until #1425 is fixed. In-session ordering separates the cases, but a fault on a device that has already provisioned still costs minutes, so they must not share a device. Merging it back is the follow-up once the underlying bug is closed.

Also: stale ci.yml:<line> citations

docs/capability-survey.md carried four; three were already wrong, invalidated by this session's own edits:

Citation Lands on today
ci.yml:607 a blank line
ci.yml:884 cmake --build build
ci.yml:628-643 a comment about dep_gen

All four become greppable anchors. grep -rIn 'ci\.yml:[0-9]' is now empty repo-wide.

Testing

  • Both pytest behaviours measured — --ignore=<missing> exit 0 silent; <missing path> exit 4
  • Ordering verified against a simulated item set (output above)
  • Pool-key change audited for tuple-shape assumptions: the reuse loop unpacked a 2-tuple and was fixed; the two other consumers index key[1] and are length-agnostic
  • prefetch_async_orch.cpp compiles with the framework's default orchestration includes only — 558 KiB, no extra_include_dirs needed
  • ast.parse on conftest.py, scene_test.py, the converted test; yaml.safe_load on ci.yml
  • grep -rIn isolated_device empty; check_english_only.py clean; markdownlint-cli2 clean
  • Not run on hardware from this worktree — st-onboard-a2a3 on this PR is the real check, and the SDMA step is where the conversion proves out

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 341488bd-5665-4e72-90be-8197a3cf5463

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR introduces the isolated_device pytest marker, applies it to two a2a3 tests, updates CI selection commands, and aligns testing and capability documentation with marker-based SDMA isolation.

Changes

Isolated device test selection

Layer / File(s) Summary
Marker contract and test usage
conftest.py, examples/a2a3/tensormap_and_ringbuffer/*/test_*.py
Registers the isolated_device marker and applies it to the prefetch and SDMA completion tests.
CI marker selection
.github/workflows/ci.yml
General a2a3 scene-test commands exclude isolated_device tests. The dedicated SDMA step still runs them.
Documentation alignment
.claude/skills/testing/SKILL.md, docs/capability-survey.md
Replaces SDMA_IGNORE references and updates CI job and step references.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Poem

A rabbit marks tests in a neat little row,
So isolated devices know where to go.
CI skips the marked path in the main run,
Then SDMA hops through its dedicated sun.
Old exclusions fade; clear markers now gleam.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title identifies the SDMA quarantine and stale CI reference changes, which are central parts of the pull request.
Description check ✅ Passed The description explains the marker-based SDMA quarantine, CI behavior, stale references, and validation performed.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.claude/skills/testing/SKILL.md:
- Around line 71-73: Update the testing guide’s instructions around the hardware
scene test command to remove the stale --ignore extraction and grep steps.
Direct readers to use the established pytest marker contract, excluding
isolated_device tests with -m "not isolated_device" so local execution matches
CI.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c5ef5398-7c79-493c-92a1-b2321bb34d77

📥 Commits

Reviewing files that changed from the base of the PR and between a07bc90 and 94d8dac.

📒 Files selected for processing (6)
  • .claude/skills/testing/SKILL.md
  • .github/workflows/ci.yml
  • conftest.py
  • docs/capability-survey.md
  • examples/a2a3/tensormap_and_ringbuffer/prefetch_async_demo/test_prefetch_async_demo.py
  • examples/a2a3/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py

Comment thread .claude/skills/testing/SKILL.md Outdated
@ChaoWao
ChaoWao force-pushed the fix/sdma-quarantine-and-stale-line-refs branch from 94d8dac to cff90f7 Compare July 31, 2026 03:48
@ChaoWao ChaoWao changed the title Fix: quarantine the SDMA demos by marker, and drop stale ci.yml line refs Fix: make SDMA a declared capability, and drop stale ci.yml line refs Jul 31, 2026
@ChaoWao
ChaoWao force-pushed the fix/sdma-quarantine-and-stale-line-refs branch 2 times, most recently from 3cde730 to 764a47a Compare July 31, 2026 04:46
The SDMA quarantine was a pair of `--ignore=<path>` arguments. `pytest
--ignore` on a path that no longer exists **exits 0 and says nothing** —
measured — so moving either demo directory would have dropped the quarantine
silently and landed both tests back beside the `aicore_op_timeout`
fault-injection test. That collision is the whole hazard: provisioning the
SDMA workspace creates 48 device-only STARS streams that sit in the device
fault domain, so a later AICore fault on that device costs ~306 s instead of
~0.3 s (hw-native-sys#1425, contained in hw-native-sys#1406).

`@pytest.mark.sdma` replaces the path matching, and carries the whole
consequence rather than one arbitrary piece of it. One declaration now drives
three things:

- **The Worker is built with `enable_sdma=True`.** Both pytest construction
  sites in `conftest.py` and both standalone sites in `scene_test.py` read it,
  the latter from `cls.pytestmark` so `python test_x.py` behaves like the
  pytest path.
- **The L2 Worker pool stops mixing capabilities.** The key gains the flag and
  reuse tests it, because an enable_sdma Worker holds its STARS streams for
  life: handing it to a test that did not ask for them would spread the
  teardown hazard to every later L2 case on that device. The existing
  same-device retire loop then performs the swap, so no new teardown path.
- **SDMA sorts last.** `sort_key` gains a term, keyed off the marker rather
  than the class because the fault-injection tests are plain functions with no
  `_st_level`. This is what will make merging the dedicated CI step back into
  the sweep safe once hw-native-sys#1425 is fixed; today `-m` already separates them, so it
  matters for local full runs.

With the capability expressible, `prefetch_async_demo` becomes an ordinary L2
`@scene_test` class. It was a hand-rolled Worker only because `CASES` had no
channel to `Worker.__init__`, and it paid for that by forfeiting golden
comparison, case parametrization, `--rounds`, `--case` and the dispatcher's
device allocation. 160 lines become 89, and its verification —
`torch.equal(out, src)` — is now a one-line `compute_golden`. The framework's
default orchestration includes turned out to be sufficient, so no new
passthrough was needed there. The conversion also moves it from the Resource
phase to the L2 phase, which runs after it, so it provisions on a device the
fault-injection cases have already finished with.

`sdma_async_completion_demo` keeps its hand-rolled L3 Worker for now and takes
the marker for ordering and CI selection only; converting it means expressing a
comm domain through `CASES`, which is a larger change and independent of this
one.

CI selects with `-m sdma` / `-m "not sdma"`, so the sweep and the dedicated
step can no longer drift apart. **The dedicated step stays until hw-native-sys#1425 is
fixed**: the two phases run their jobs in parallel across devices, so only
separate device pools keep an SDMA provisioning away from a fault injection.

`.claude/skills/testing/SKILL.md` moves with the mechanism throughout — not
only the mirror-CI command, but the two instructions that told readers to
extract `--ignore` sets from `ci.yml` and to grep it when a test passes alone
and fails in the sweep. Both now name the marker; following the skill
reproduces CI rather than a mechanism that no longer exists.

Separately, `docs/capability-survey.md` carried four `ci.yml:<line>` citations,
three already invalidated by this session's edits — `:607` lands on a blank
line, `:884` on a `cmake --build`, `:628-643` on a dep_gen comment, after the
qwen step was deleted in hw-native-sys#1601 and `detect-changes` reworked in hw-native-sys#1589 / hw-native-sys#1590 /
hw-native-sys#1607. Line numbers into a file this active cannot be maintained, so all four
become greppable anchors. `grep -rIn 'ci\.yml:[0-9]'` is now empty repo-wide.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ChaoWao
ChaoWao merged commit 80aa287 into hw-native-sys:main Jul 31, 2026
18 checks passed
@ChaoWao
ChaoWao deleted the fix/sdma-quarantine-and-stale-line-refs branch July 31, 2026 09:23
ChaoZheng109 added a commit to lterrac/simpler that referenced this pull request Aug 1, 2026
The chip-init path makes direct driver calls that bypass ACL (halMemCtl /
halGetDeviceInfo* on a2a3, halResMap / halGetDeviceInfo* / dsmi_get_device_info
on a5), so they index the driver-visible space and must translate through
acl_to_hal_device_id. Nothing exercised that translation: no CI job sets
ASCEND_RT_VISIBLE_DEVICES, so every job ran the identity path and a call site
that forgot to translate stayed green.

tests/st/vis_isolation runs dummy_task in a subprocess with the granted card
exposed as logical 0. init_aicore_register_addresses and probe_aicpu_topology
run on every onboard bring-up, so one scene test covers every translated site
on both arches. The subprocess is load-bearing: the variable must be set before
ACL init, and mutating it in-process would leak into the session's pooled
workers.

Carrying this as a scene test rather than the two ci.yml steps keeps the
selection in the test tree, so the sweep and the check cannot drift apart --
the direction hw-native-sys#1601 and hw-native-sys#1609 took for qwen and SDMA. The onboard jobs already
run `pytest examples tests/st`, and the platforms marker keeps it off the sim
runners, where a driver-level remap has nothing to verify.
ChaoWao added a commit that referenced this pull request Aug 2, 2026
`st-onboard-a2a3` has been two pytest invocations since #1609 — the sweep with
`-m "not sdma"` and a later step with `-m sdma`, so that no fault-injection case
meets a device that has already provisioned SDMA (#1425). Three places that
claim to reproduce that job still issued one flat sweep, which is the invocation
`.claude/skills/testing/SKILL.md` warns "is *not* what CI runs and will report
failures that CI never sees".

- **`test-all-device`** describes itself as running "the full hardware (onboard)
  CI pipeline" and already reads `--pto-session-timeout` out of the
  `st-onboard-<platform>` job. It now reads the marker expression from the same
  place, and runs the second pass after the first. Taking both from `ci.yml`
  rather than hardcoding them is what keeps the skill from drifting again.
- **`test-runtime-device`** likewise. Both quarantined tests are
  `tensormap_and_ringbuffer`, so the second pass is noted as needed only when
  that runtime is selected.
- **`docs/ci.md`'s "Recommended CI invocation"** block, which #1617 missed while
  correcting the job table three sections above it. The a5 line is unchanged —
  `st-onboard-a5` carries no marker filter.

Left alone: the recipes that describe running the tests locally rather than
mirroring a job (`README.md`, `docs/testing.md`, `docs/user/reference/cli.md`,
`examples/README.md`, the `SUBMIT_BY_CLUSTER.md` trio). Whether a local sweep
should exclude SDMA by default is a question about the documented default, not
about matching CI, and `testing/SKILL.md` already tells anyone who trips over it
what happened.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant