CI: get the A5 V4-Pro sweep running and into the daily report - #876
Conversation
model-tests-a5 died in setup on every run, most recently at Check NPU with
npu-smi exiting 187 and "dcmi module initialize failed. ret is -8005".
That is permission denial, not a missing DCMI component. Since the A5 host's
driver upgrade the device nodes are crw-rw---- HwHiAiUser:HwHiAiUser and the CI
user is not in that group, so dcmi_init cannot open the character device —
npu-smi even lists "Failed to open the character device" among its own
candidate causes. task-submit drops privileges into the HwHiAiUser group, which
is how every model run reaches a card in the first place.
Direct invocation is still tried first, so hosts whose device nodes are
world-readable — the 910B runners — behave exactly as before and borrow no
card.
Two details the retry needs, both found by running it:
- The payload ends on `rc=$?; exit $rc` rather than on `npu-smi info`. With
the probe as the bare last command task-submit reports 215 even though
npu-smi returned 0 (reproduced 3/3 on an A5 host); setting the status
explicitly makes the outer code npu-smi's own. Verified to pass 0, 9 and
127 through unchanged, so an absent card still fails the step.
- task-submit leases a real card, so a busy queue makes it give up on the
wait and exit 1 having run nothing. That says nothing about the NPU, and it
failed the job just as the original probe did. The NPUSMI_RC sentinel
separates the two: present means the probe ran and its status is
authoritative, absent means the task never got scheduled, which warns and
continues. Every model run goes through task-submit anyway and fails
loudly when there is no device.
With this the A5 sweep completes for the first time since it was added in #816.
model-tests-a5's results never reached the summary, for two reasons. The summary did not depend on the job, so it was built as soon as the sim and a2a3 sweeps finished. The V4-Pro sweep is the longest of the four — 65 min against their 23-50 — so results-a5 did not exist yet and the A5 section always rendered its "no artifact" placeholder. Add it to `needs`; `always()` still lets the summary run when a sweep fails. Give A5 its own table rather than a column in the shared one. The case sets are disjoint by construction — model-tests-a5 sweeps only models/deepseek/v4-pro and the other three jobs exclude it — so a single matrix makes every row half filler and there is no row worth comparing across both. Each table now carries a per-platform passed count and names any platform whose artifact is missing, so an absent sweep is never mistaken for a set of skipped cases. Mark the A5 job report-only. A measured sweep puts V4-Pro at 13/35: 13 cases die with sync_stream_pair device faults, 2 with rtMalloc 207001, 7 on numerics — pypto/device-side rather than defects in this repo, so gating on them would paint the workflow red nightly, which is exactly what has made model-tests-sim's daily failure invisible. The table is published either way. Rendering checked against the real results of runs 30595513890 and 30599291163, and with the a5, a5sim and all artifacts removed in turn.
|
Warning Review limit reached
Next review available in: 58 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2 participants
model-tests-a5has not executed a single test since it was added in #816 —every scheduled run died in setup. This gets the DeepSeek V4-Pro sweep running
and its results into the daily report.
Why it never ran
The job failed in
setup-ci-jobon every run, and each failure hid the nextone behind it. Most were host provisioning and have since been fixed on the
npu-a5-1runner (CANN_ROOTpointed at a CANN that was not installed; theconda env, GCC 15 and ccache were absent). One is a real repo bug and is fixed
here.
Check NPUrannpu-smi infodirectly, which cannot work on that host.It exits 187 with
dcmi module initialize failed. ret is -8005, and the stepruns under
bash -e. That reads like a missing DCMI component, but it ispermission denial: since the driver upgrade the device nodes are
crw-rw---- HwHiAiUser:HwHiAiUserand the CI user is not in that group, sodcmi_initcannot open the character device. npu-smi even lists "Failed toopen the character device" among its own candidate causes.
task-submitdrops privileges into the HwHiAiUser group — which is how everymodel run reaches a card in the first place — so the check now falls back to
it. Direct invocation is tried first, so the 910B runners behave exactly as
before and borrow no card.
Two details the retry needs, both found by running it rather than reasoning
about it:
rc=$?; exit $rcrather than onnpu-smi info. With theprobe as the bare last command, task-submit reports 215 even though
npu-smi returned 0 (reproduced 3/3). Setting the status explicitly makes the
outer code npu-smi's own — verified to pass 0, 9 and 127 through unchanged,
so an absent card still fails the step.
and exit 1 having run nothing. That says nothing about the NPU, yet it failed
the job just as the original probe did. An
NPUSMI_RCsentinel separates thetwo: present means the probe ran and its status is authoritative; absent
means the task never got scheduled, which warns and continues.
Getting the results into the report
Even once the sweep ran, its results did not appear. The summary did not
needsthe job, so it was built as soon as the sim and a2a3 sweeps finished —and A5 is the longest of the four:
model-tests-sim(both)model-tests-a2a3Aggregate results summarymodel-tests-a5So
results-a5never existed in time and the A5 section always rendered its"no artifact" placeholder. Adding it to
needsfixes that (always()stilllets the summary run when a sweep fails); the summary now starts three seconds
after the A5 job ends.
A5 also gets its own table rather than a column in the shared one. The case
sets are disjoint by construction —
model-tests-a5sweeps onlymodels/deepseek/v4-proand the other three jobs exclude it — so a singlematrix made every row half filler, and no row is worth comparing across both.
Each table carries a per-platform passed count and names any platform whose
artifact is missing, so an absent sweep is never mistaken for a set of skipped
cases — which is precisely what fooled us here.
Report-only, for now
The A5 job is
continue-on-error: true. A measured sweep puts V4-Pro at13/35: 13 cases die with
sync_stream_pairdevice faults, 2 withrtMalloc failed: 207001, 7 on numerics. Those are pypto/device-side ratherthan defects in this repo, so gating on them would paint the workflow red every
night — exactly what has made
model-tests-sim's daily failure invisible. Thetable is published either way, so the coverage is real even though the check
does not block. Flip it off once the A5 blockers land.
Verification
Exercised end to end on the real
npu-a5-1runner across several runs:budget; slowest case
decode_fwdat 417s, inside--max-time 900, andnothing hit the cap — no timeout changes needed.
13/35reproduced on three separate runs.--device auto --device-num 2schedules fine here (got cards: 2,3in 2s),so the eight
# ci: devices=2V4-Pro files are safe.30599291163, and with the
a5,a5sim, and all artifacts removed in turn.ci.ymlis untouched; the 910Bsim/a2a3jobs ran green through themodified shared action.
Supersedes #859, which GitHub closed when its branch was renamed.