WIP: dsv4-pro Hybrid MXFP8–MXFP4 operator rewrite - #836
Draft
yanghaoran29 wants to merge 8 commits into
Draft
Conversation
Rewrite MoE/qkv/indexer/o_proj/mtp and main KV C8 read/write toward AscendC Hybrid MX; document remaining gaps vs AscendC (W4A4 interim, FP32 KV scale, layer wiring). Co-authored-by: Cursor <cursoragent@cursor.com>
Wire MX LeftScale through GM ND store → mx_a_zz tload, pack B-scale with device-col-0 tiled offsets, and Acc peel/unroll for v4-pro operators. expert_shared golden passes; qkv compiles again pending RoPE golden. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Slice q/kv RoPE gathers into 4-row subtiles so A5 FP32 index-form tgather no longer corrupts every t%8==7 token. Drop redundant FP4 reinterpret_view in expert_routed (mx_quant already returns FP4). qkv_proj_rope decode+prefill golden pass on board. Co-authored-by: Cursor <cursoragent@cursor.com>
Compressed bias vector path and INT32 scratch reads mis-mark slots on A5; align gather/bias with scalar fills, RoPE 4-row tgather, matmul_mx peel, and HCA compare at 5% outlier ratio. Co-authored-by: Cursor <cursoragent@cursor.com>
Prefill compressor packs only 4 trusted RMS rows per 16-box; indexer/score paths use 4-row tgather and scalar ReLU so A5 gather/ReLU bugs no longer corrupt scores. Prefill sparse attn ports decode bias/peel/RoPE-4 fixes. Co-authored-by: Cursor <cursoragent@cursor.com>
Peel matmul_mx to init Acc (flash-style), load tiled MX_B_NN weight scales, and run FP32 e+h in a separate vec CORE_GROUP so mixed AIC/AIV no longer drops the add (was h-only) or stalls with 507018. Co-authored-by: Cursor <cursoragent@cursor.com>
yanghaoran29
force-pushed
the
wip/v4-pro-mxfp8-mxfp4
branch
from
July 27, 2026 01:56
ae29afa to
9e93321
Compare
lwDavid
added a commit
that referenced
this pull request
Aug 5, 2026
## Summary [#847](#847) made `golden_expert_shared` mirror the kernel's int32 accumulate. Its literal sibling in the same file family, `golden_expert_routed`, was left on the old form. This applies the same change. The kernel does an exact INT8 × INT8 → INT32 accumulate for all three matmuls (`pl.matmul(..., out_dtype=pl.INT32)` + `pl.matmul_acc`) and dequantizes **once, after** the reduction, by per-row input scale × per-channel weight scale. The golden pre-dequantized `w1`/`w3`/`w2` to fp32 and ran fp32 matmuls, accumulating rounded products. Integer accumulation is exact and order-independent, so the int32 form is bit-identical to the kernel's tiled accumulate regardless of K-tiling. ## Two details worth reviewing **Parenthesisation is load-bearing.** The w2 stage keeps `(h_sd * w_per_row)` grouped, because the kernel forms `row_scale_blk = pl.mul(h_tile_scale_dq, w_col_blk)` as one fp32 product before applying it through `row_expand_mul`. Ungrouping it changes the rounding. **Host memory.** `dequant_w` materialized all three weights as fp32 tensors held live across the whole expert loop: ``` 3 × 48 × 3072 × 7168 × 4 B = 11.81 GiB (N_LOCAL_EXPERTS=48, MOE_INTER=3072, D=7168) ``` The replacement casts one expert's weights to int32 inside the loop — 84 MiB, freed each iteration. This file has already been the subject of a host-memory fix (#745 on `gen_routed_weight`), so the cost is a known concern here. ## Known limitation, same as #847 The w2 stage still cannot be bit-exact: the kernel computes silu as `recip(add(exp(neg(gate)), 1.0))`, which won't match `F.silu` to the last ULP, and that flips ties in the INT8 requant. Only w1/w3 become exact. ## Scope Golden only. No kernel change, no device-side change, no API change. Note for @yanghaoran29: draft #836 rewrites this golden wholesale for MXFP8 and does not include the int32 alignment — worth picking up there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WIP：DeepSeek V4-Pro 算子按 Hybrid MXFP8–MXFP4 接线（LeftScale
AND2ZZ、tiledMX_B_NN、Acc peel/unroll 等）。依赖尚未合入 main 的 pypto MX DSL 与 simpler FP8/FP4 host dtype。
依赖版本（本 PR 验证环境）
88d2d57e9c84— hw-native-sys/pypto#2117 (issue-1975-mx-dsl-ops)reinterpret_viewruntimesubmodule）d55ad749f1c6— hw-native-sys/simpler#1473 (add-mx-fp8-fp4-dtypes)83d01313d9bfc247c4b7c8bcf969d1019f0d106fruntime/pto_isa.pin（与 simpler 构建一致）PTOAS_ROOT=~/opt/ptoas-x86_64）ptoas --versionbbb5e1cdd760—wip/v4-pro-mxfp8-mxfp4cann-9.1.0-beta.1上板 golden 状态（A5）
主 sweep：
~/board_logs/sweep_20260726_022600/（2026-07-26 02:26）。其后单独复跑：
qkv_proj_rope（RoPE 4-row gather 修复后 decode+prefill 全 PASS）。PASS（整算子 golden 全过）— 7
expert_sharedshgolden PASSqkv_proj_ropeq/kv/qr/qr_scale全 PASS（A5tgather8 行末行损坏 → RoPE gather 拆成 4 行子块）decode_indexer_compressordecode_compressor_ratio4cmp_kv_cachePASSdecode_compressor_ratio128prefill_compressor_ratio4prefill_compressor_ratio128FAIL — 仍过不了
decode_indexeridx_kv_cache/idx_kv_scale/topk_idxsPASS，scoreFAILprefill_indexerscoreFAILprefill_indexer_compressorkvFAILdecode_sparse_attnattn_outgolden mismatchprefill_sparse_attn/mtp_projection507018等expert_routedmoeTest plan
qkv_proj_ropedecode + prefill A5 golden（RoPE fix 后）expert_sharedA5 golden