CI: get the A5 V4-Pro sweep running and into the daily report - #859
Closed
lwDavid wants to merge 2 commits into
Closed
CI: get the A5 V4-Pro sweep running and into the daily report#859lwDavid wants to merge 2 commits into
lwDavid wants to merge 2 commits into
Conversation
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe setup action adds queued NPU probing after direct access fails. The daily workflow allows A5 failures, waits for the A5 job, and renders separate summaries with performance data and missing-artifact warnings. ChangesCI reliability and reporting
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CIWorkflow
participant SetupCIJob
participant task_submit
participant npu_smi
participant SummaryJob
CIWorkflow->>SetupCIJob: Start model-test job
SetupCIJob->>npu_smi: Run direct npu-smi info
SetupCIJob->>task_submit: Retry device probe when direct access fails
task_submit->>npu_smi: Run wrapped npu-smi info
task_submit-->>SetupCIJob: Return probe status or queue timeout
CIWorkflow->>SummaryJob: Wait for simulator, a2a3, and A5 jobs
SummaryJob-->>CIWorkflow: Render result tables and warnings
Possibly related PRs
Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
lwDavid
force-pushed
the
wip/debug-daily-ci
branch
from
July 30, 2026 07:57
c152bf1 to
608fe25
Compare
lwDavid
changed the title
lwDavid
force-pushed
the
wip/debug-daily-ci
branch
from
July 30, 2026 09:58
608fe25 to
f872d9b
Compare
lwDavid
changed the title
model-tests-a5 died in setup on every run, most recently at Check NPU with
npu-smi exiting 187 and "dcmi module initialize failed. ret is -8005".
That is permission denial, not a missing DCMI component. Since the A5 host's
driver upgrade the device nodes are crw-rw---- HwHiAiUser:HwHiAiUser and the CI
user is not in that group, so dcmi_init cannot open the character device —
npu-smi even lists "Failed to open the character device" among its own
candidate causes. task-submit drops privileges into the HwHiAiUser group, which
is how every model run reaches a card in the first place.
Direct invocation is still tried first, so hosts whose device nodes are
world-readable — the 910B runners — behave exactly as before and borrow no
card.
Two details the retry needs, both found by running it:
- The payload ends on `rc=$?; exit $rc` rather than on `npu-smi info`. With
the probe as the bare last command task-submit reports 215 even though
npu-smi returned 0 (reproduced 3/3 on an A5 host); setting the status
explicitly makes the outer code npu-smi's own. Verified to pass 0, 9 and
127 through unchanged, so an absent card still fails the step.
- task-submit leases a real card, so a busy queue makes it give up on the
wait and exit 1 having run nothing. That says nothing about the NPU, and it
failed the job just as the original probe did. The NPUSMI_RC sentinel
separates the two: present means the probe ran and its status is
authoritative, absent means the task never got scheduled, which warns and
continues. Every model run goes through task-submit anyway and fails
loudly when there is no device.
With this the A5 sweep completes for the first time since it was added in #816.
model-tests-a5's results never reached the summary, for two reasons. The summary did not depend on the job, so it was built as soon as the sim and a2a3 sweeps finished. The V4-Pro sweep is the longest of the four — 65 min against their 23-50 — so results-a5 did not exist yet and the A5 section always rendered its "no artifact" placeholder. Add it to `needs`; `always()` still lets the summary run when a sweep fails. Give A5 its own table rather than a column in the shared one. The case sets are disjoint by construction — model-tests-a5 sweeps only models/deepseek/v4-pro and the other three jobs exclude it — so a single matrix makes every row half filler and there is no row worth comparing across both. Each table now carries a per-platform passed count and names any platform whose artifact is missing, so an absent sweep is never mistaken for a set of skipped cases. Mark the A5 job report-only. A measured sweep puts V4-Pro at 13/35: 13 cases die with sync_stream_pair device faults, 2 with rtMalloc 207001, 7 on numerics — pypto/device-side rather than defects in this repo, so gating on them would paint the workflow red nightly, which is exactly what has made model-tests-sim's daily failure invisible. The table is published either way. Rendering checked against the real results of runs 30595513890 and 30599291163, and with the a5, a5sim and all artifacts removed in turn.
lwDavid
force-pushed
the
wip/debug-daily-ci
branch
from
July 31, 2026 06:14
db2dd9b to
639bbd5
Compare
lwDavid
changed the title
lwDavid
marked this pull request as ready for review
July 31, 2026 06:15
Collaborator
Author
|
Closed automatically when the branch was renamed. Continued in the PR that supersedes it; the branch is now |
zhangqi-chen
pushed a commit
that referenced
this pull request
Jul 31, 2026
`model-tests-a5` has not executed a single test since it was added in #816 — every scheduled run died in setup. This gets the DeepSeek V4-Pro sweep running and its results into the daily report. ## Why it never ran The job failed in `setup-ci-job` on every run, and each failure hid the next one behind it. Most were host provisioning and have since been fixed on the `npu-a5-1` runner (`CANN_ROOT` pointed at a CANN that was not installed; the conda env, GCC 15 and ccache were absent). One is a real repo bug and is fixed here. **`Check NPU` ran `npu-smi info` directly, which cannot work on that host.** It exits 187 with `dcmi module initialize failed. ret is -8005`, and the step runs under `bash -e`. That reads like a missing DCMI component, but it is permission denial: since the driver upgrade the device nodes are `crw-rw---- HwHiAiUser:HwHiAiUser` and the CI user is not in that group, so `dcmi_init` cannot open the character device. npu-smi even lists *"Failed to open the character device"* among its own candidate causes. ``` $ ls -l /dev/davinci0 crw-rw---- 1 HwHiAiUser HwHiAiUser 235, 0 /dev/davinci0 $ npu-smi info -> exit 187 $ task-submit --run 'npu-smi info' -> exit 0, lists all 8 NPUs ``` `task-submit` drops privileges into the HwHiAiUser group — which is how every model run reaches a card in the first place — so the check now falls back to it. **Direct invocation is tried first, so the 910B runners behave exactly as before and borrow no card.** Two details the retry needs, both found by running it rather than reasoning about it: - The payload ends on `rc=$?; exit $rc` rather than on `npu-smi info`. With the probe as the bare last command, task-submit reports **215** even though npu-smi returned 0 (reproduced 3/3). Setting the status explicitly makes the outer code npu-smi's own — verified to pass 0, 9 and 127 through unchanged, so an absent card still fails the step. - task-submit leases a real card, so a busy queue makes it give up on the wait and exit 1 having run nothing. That says nothing about the NPU, yet it failed the job just as the original probe did. An `NPUSMI_RC` sentinel separates the two: present means the probe ran and its status is authoritative; absent means the task never got scheduled, which warns and continues. ## Getting the results into the report Even once the sweep ran, its results did not appear. The summary did not `needs` the job, so it was built as soon as the sim and a2a3 sweeps finished — and A5 is the longest of the four: | Job | finished | | --- | -------- | | `model-tests-sim` (both) | 01:32 / 01:33 | | `model-tests-a2a3` | 01:57 | | `Aggregate results summary` | **02:23** ⬅️ built here | | `model-tests-a5` | **02:47** | So `results-a5` never existed in time and the A5 section always rendered its "no artifact" placeholder. Adding it to `needs` fixes that (`always()` still lets the summary run when a sweep fails); the summary now starts three seconds after the A5 job ends. A5 also gets **its own table** rather than a column in the shared one. The case sets are disjoint by construction — `model-tests-a5` sweeps only `models/deepseek/v4-pro` and the other three jobs exclude it — so a single matrix made every row half filler, and no row is worth comparing across both. ``` ## Daily CI Model Test Results ### a2a3 and simulators | Case | a2a3 | a2a3 effective (us) | a2a3sim | a5sim | ... Passed: **a2a3** 48/48, **a2a3sim** 20/41, **a5sim** 17/41 ### a5 (DeepSeek V4-Pro) | Case | a5 | a5 effective (us) | | `models/deepseek/v4-pro/rmsnorm.py` | ✅ | 43.4 | | `models/deepseek/v4-pro/gate.py` | ❌ | 93.2 | ... Passed: **a5** 13/35 ``` Each table carries a per-platform passed count and names any platform whose artifact is missing, so an absent sweep is never mistaken for a set of skipped cases — which is precisely what fooled us here. ## Report-only, for now The A5 job is `continue-on-error: true`. A measured sweep puts V4-Pro at **13/35**: 13 cases die with `sync_stream_pair` device faults, 2 with `rtMalloc failed: 207001`, 7 on numerics. Those are pypto/device-side rather than defects in this repo, so gating on them would paint the workflow red every night — exactly what has made `model-tests-sim`'s daily failure invisible. The table is published either way, so the coverage is real even though the check does not block. Flip it off once the A5 blockers land. ## Verification Exercised end to end on the real `npu-a5-1` runner across several runs: - The full 35-case sweep completes in ~44-58 min against the job's 120-minute budget; slowest case `decode_fwd` at 417s, inside `--max-time 900`, and nothing hit the cap — **no timeout changes needed**. - `13/35` reproduced on three separate runs. - `--device auto --device-num 2` schedules fine here (`got cards: 2,3` in 2s), so the eight `# ci: devices=2` V4-Pro files are safe. - Summary rendering checked against the real results of runs 30595513890 and 30599291163, and with the `a5`, `a5sim`, and all artifacts removed in turn. - `ci.yml` is untouched; the 910B `sim` / `a2a3` jobs ran green through the modified shared action. --- Supersedes #859, which GitHub closed when its branch was renamed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
model-tests-a5has not executed a single test since it was added in #816 —every scheduled run died in setup. This gets the DeepSeek V4-Pro sweep running
and its results into the daily report.
Why it never ran
The job failed in
setup-ci-jobon every run, and each failure hid the nextone behind it. Most were host provisioning and have since been fixed on the
npu-a5-1runner (CANN_ROOTpointed at a CANN that was not installed; theconda env, GCC 15 and ccache were absent). One is a real repo bug and is fixed
here.
Check NPUrannpu-smi infodirectly, which cannot work on that host.It exits 187 with
dcmi module initialize failed. ret is -8005, and the stepruns under
bash -e. That reads like a missing DCMI component, but it ispermission denial: since the driver upgrade the device nodes are
crw-rw---- HwHiAiUser:HwHiAiUserand the CI user is not in that group, sodcmi_initcannot open the character device. npu-smi even lists "Failed toopen the character device" among its own candidate causes.
task-submitdrops privileges into the HwHiAiUser group — which is how everymodel run reaches a card in the first place — so the check now falls back to
it. Direct invocation is tried first, so the 910B runners behave exactly as
before and borrow no card.
Two details the retry needs, both found by running it rather than reasoning
about it:
rc=$?; exit $rcrather than onnpu-smi info. With theprobe as the bare last command, task-submit reports 215 even though
npu-smi returned 0 (reproduced 3/3). Setting the status explicitly makes the
outer code npu-smi's own — verified to pass 0, 9 and 127 through unchanged,
so an absent card still fails the step.
and exit 1 having run nothing. That says nothing about the NPU, yet it failed
the job just as the original probe did. An
NPUSMI_RCsentinel separates thetwo: present means the probe ran and its status is authoritative; absent
means the task never got scheduled, which warns and continues.
Getting the results into the report
Even once the sweep ran, its results did not appear. The summary did not
needsthe job, so it was built as soon as the sim and a2a3 sweeps finished —and A5 is the longest of the four:
model-tests-sim(both)model-tests-a2a3Aggregate results summarymodel-tests-a5So
results-a5never existed in time and the A5 section always rendered its"no artifact" placeholder. Adding it to
needsfixes that (always()stilllets the summary run when a sweep fails); the summary now starts three seconds
after the A5 job ends.
A5 also gets its own table rather than a column in the shared one. The case
sets are disjoint by construction —
model-tests-a5sweeps onlymodels/deepseek/v4-proand the other three jobs exclude it — so a singlematrix made every row half filler, and no row is worth comparing across both.
Each table carries a per-platform passed count and names any platform whose
artifact is missing, so an absent sweep is never mistaken for a set of skipped
cases — which is precisely what fooled us here.
Report-only, for now
The A5 job is
continue-on-error: true. A measured sweep puts V4-Pro at13/35: 13 cases die with
sync_stream_pairdevice faults, 2 withrtMalloc failed: 207001, 7 on numerics. Those are pypto/device-side ratherthan defects in this repo, so gating on them would paint the workflow red every
night — exactly what has made
model-tests-sim's daily failure invisible. Thetable is published either way, so the coverage is real even though the check
does not block. Flip it off once the A5 blockers land.
Verification
Exercised end to end on the real
npu-a5-1runner across several runs:budget; slowest case
decode_fwdat 417s, inside--max-time 900, andnothing hit the cap — no timeout changes needed.
13/35reproduced on three separate runs.--device auto --device-num 2schedules fine here (got cards: 2,3in 2s),so the eight
# ci: devices=2V4-Pro files are safe.30599291163, and with the
a5,a5sim, and all artifacts removed in turn.ci.ymlis untouched; the 910Bsim/a2a3jobs ran green through themodified shared action.