From 3eea4ff05d249010fe97a360337dbb20bf829d88 Mon Sep 17 00:00:00 2001 From: Chao Wang <26245345+ChaoWao@users.noreply.github.com> Date: Fri, 31 Jul 2026 06:48:56 -0700 Subject: [PATCH] Docs: give the CI-mirroring recipes both onboard pytest passes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `st-onboard-a2a3` has been two pytest invocations since #1609 — the sweep with `-m "not sdma"` and a later step with `-m sdma`, so that no fault-injection case meets a device that has already provisioned SDMA (#1425). Three places that claim to reproduce that job still issued one flat sweep, which is the invocation `.claude/skills/testing/SKILL.md` warns "is *not* what CI runs and will report failures that CI never sees". - **`test-all-device`** describes itself as running "the full hardware (onboard) CI pipeline" and already reads `--pto-session-timeout` out of the `st-onboard-` job. It now reads the marker expression from the same place, and runs the second pass after the first. Taking both from `ci.yml` rather than hardcoding them is what keeps the skill from drifting again. - **`test-runtime-device`** likewise. Both quarantined tests are `tensormap_and_ringbuffer`, so the second pass is noted as needed only when that runtime is selected. - **`docs/ci.md`'s "Recommended CI invocation"** block, which #1617 missed while correcting the job table three sections above it. The a5 line is unchanged — `st-onboard-a5` carries no marker filter. Left alone: the recipes that describe running the tests locally rather than mirroring a job (`README.md`, `docs/testing.md`, `docs/user/reference/cli.md`, `examples/README.md`, the `SUBMIT_BY_CLUSTER.md` trio). Whether a local sweep should exclude SDMA by default is a question about the documented default, not about matching CI, and `testing/SKILL.md` already tells anyone who trips over it what happened. Co-Authored-By: Claude Opus 5 --- .claude/skills/test-all-device/SKILL.md | 29 +++++++++++++++++---- .claude/skills/test-runtime-device/SKILL.md | 23 +++++++++++++--- docs/ci.md | 8 +++--- 3 files changed, 48 insertions(+), 12 deletions(-) diff --git a/.claude/skills/test-all-device/SKILL.md b/.claude/skills/test-all-device/SKILL.md index 52559a7ff2..dde20bd042 100644 --- a/.claude/skills/test-all-device/SKILL.md +++ b/.claude/skills/test-all-device/SKILL.md @@ -16,15 +16,34 @@ Detection / isolation procedures referenced below live in `--pto-session-timeout`. 4. **Select a device range** (§C, range ≤4) — or, when wrapping in `task-submit`, let it pick via `--device auto --device-num `. -5. **Run through `task-submit`** (§E). The underlying command: +5. **Read the marker selector out of the same job** you took the timeout from. + `st-onboard-a2a3` is two pytest passes, not one: the sweep deselects + `-m "not sdma"` and a later step runs `-m sdma`. `st-onboard-a5` has no + marker filter. Take the expression from `ci.yml` rather than assuming, so + this skill cannot drift from the job it reproduces. +6. **Run through `task-submit`** (§E). The underlying command, on a platform + whose job carries the filter: ```bash - pytest examples tests/st --platform --device \ - --pto-session-timeout -v + pytest examples tests/st -m "not sdma" --platform \ + --device --pto-session-timeout -v ``` + Then the quarantined pass, after it — never at the same time, since the + point of the split is that no fault-injection case meets a device that has + already provisioned SDMA: + + ```bash + pytest examples tests/st -m sdma --platform \ + --device <2 devs or $TASK_DEVICE> --pto-session-timeout -v + ``` + + Dropping either pass changes what you tested: a single flat sweep is not + what CI runs and reports failures CI never sees + ([`../testing/SKILL.md`](../testing/SKILL.md), issue #1425). + Parallelism is auto-driven by `--device`: on hardware, one in-flight subprocess per device (`--max-parallel auto` = `len(--device)`); see `docs/testing.md` for the full reuse hierarchy. -6. Report the results summary (pass/fail counts per task). -7. If any tests fail, show the relevant error output and which device failed. +7. Report the results summary (pass/fail counts per task), across both passes. +8. If any tests fail, show the relevant error output and which device failed. diff --git a/.claude/skills/test-runtime-device/SKILL.md b/.claude/skills/test-runtime-device/SKILL.md index cd6bb55a06..6b3a026ca5 100644 --- a/.claude/skills/test-runtime-device/SKILL.md +++ b/.claude/skills/test-runtime-device/SKILL.md @@ -18,15 +18,30 @@ Detection / isolation procedures referenced below live in `--pto-session-timeout`. 5. **Select a device range** (§C, range ≤4) — or, when wrapping in `task-submit`, let it pick via `--device auto --device-num `. -6. **Run through `task-submit`** (§E). The underlying command: +6. **Read the marker selector out of the same job** you took the timeout from. + `st-onboard-a2a3` is two pytest passes: the sweep deselects `-m "not sdma"` + and a later step runs `-m sdma`. `st-onboard-a5` has no marker filter. Both + quarantined tests are `tensormap_and_ringbuffer`, so the second pass is only + needed when `$ARGUMENTS` names that runtime. +7. **Run through `task-submit`** (§E). The underlying command: ```bash - pytest examples tests/st --platform --runtime $ARGUMENTS \ + pytest examples tests/st -m "not sdma" --platform --runtime $ARGUMENTS \ --device \ --pto-session-timeout -v ``` + Then, for `tensormap_and_ringbuffer` on a platform whose job carries the + filter, the quarantined pass afterwards — never concurrently, since the + split exists so that no fault-injection case meets a device that has already + provisioned SDMA ([`../testing/SKILL.md`](../testing/SKILL.md), issue #1425): + + ```bash + pytest examples tests/st -m sdma --platform --runtime $ARGUMENTS \ + --device <2 devs or $TASK_DEVICE> --pto-session-timeout -v + ``` + Hardware parallelism is auto-driven by `--device` (one subprocess per device); no extra flag needed. -7. Report the results summary (pass/fail counts per task). -8. If any tests fail, show the relevant error output and which device failed. +8. Report the results summary (pass/fail counts per task), across both passes. +9. If any tests fail, show the relevant error output and which device failed. diff --git a/docs/ci.md b/docs/ci.md index 14a916b836..3149386401 100644 --- a/docs/ci.md +++ b/docs/ci.md @@ -82,10 +82,12 @@ benefit — device bin-packing for L3, xdist fanout for L2, and a shared `ChipWorker` per `(runtime, device)`: ```bash -# Recommended CI invocation -pytest examples tests/st --platform a2a3 --device 4-7 -x +# Recommended CI invocation — a2a3 deselects the SDMA marker, as the job does, +# and runs it as a second pass afterwards +pytest examples tests/st -m "not sdma" --platform a2a3 --device 4-7 -x +pytest examples tests/st -m sdma --platform a2a3 --device 4-5 -x -# Same for a5 +# Same for a5, which has no marker filter pytest examples tests/st --platform a5 --device 0-7 -x ```