Skip to content

fix(a2a3): correct TCOLEXPAND pipe from PIPE_V to PIPE_MTE1 - #212

Closed
georgebisbas wants to merge 2 commits into
hw-native-sys:mainfrom
georgebisbas:fix-tcolexpand-pipe-sync
Closed

fix(a2a3): correct TCOLEXPAND pipe from PIPE_V to PIPE_MTE1#212
georgebisbas wants to merge 2 commits into
hw-native-sys:mainfrom
georgebisbas:fix-tcolexpand-pipe-sync

Conversation

@georgebisbas

Copy link
Copy Markdown
Contributor

Summary

TCOLEXPAND (plain broadcast) uses pto_copy_ubuf_to_ubuf (copy engine / PIPE_MTE1) in its a2a3 implementation, not a pure vector-unit op. A pipe_barrier(PIPE_V) after TCOLEXPAND does NOT wait for the copy-engine broadcast to complete, so a subsequent vector read can see stale data.

The race is data-size dependent on real hardware: C=32 fails (consumer wins the race), C=128 passes (copy is slow enough).

Root cause

This is a regression from commit 021789c0 ("refactor: extract Event CRTP base class and replace opPipeList with OpPipeEntry template", 2026-06-23), which tagged all expand-family ops as PIPE_V during the array-to-template conversion. Before that refactoring, TCOLEXPAND was not in the old opPipeList at all and defaulted to PIPE_ALL, so barriers were safe.

What was checked

  • Only TCOLEXPAND uses the copy engine — every other TCOLEXPAND* / TROWEXPAND* variant uses vector instructions (vadd, vdiv, vmul, vector_dup, vbrcb, etc.) and remains correctly on PIPE_V.
  • No other pipeline-critical structures reference expand ops' pipe assignments outside event.hpp.
  • The old opPipeList (pre-refactoring) confirms TCOLEXPAND had no entry.

Test plan

  • All existing TCOLEXPAND tests pass (they use explicit set_flag/wait_flag and are unaffected by the pipe declaration)
  • MFE kernel included in tests/npu/a2a3/src/st/testcase/tcolexpand_pipe_bug_mfe/ for real-hardware regression testing (cannot reproduce on functional simulator)

Related

  • Issue reported by @vloncar: pipe_barrier(PIPE_V) under-synchronization after TCOLEXPAND in GLA chunk_h_prep kernel produces wrong results at C=32 on a2a3 hardware
  • Workaround in existing code: chunk_h_prep uses pipe_barrier(PIPE_ALL) instead

/cc @vloncar

@georgebisbas
georgebisbas force-pushed the fix-tcolexpand-pipe-sync branch 2 times, most recently from 2d36c3a to 0117c76 Compare July 28, 2026 13:57
TCOLEXPAND (plain broadcast) uses pto_copy_ubuf_to_ubuf (copy engine /
PIPE_MTE1) in its a2a3 implementation, not a pure vector-unit op.
A pipe_barrier(PIPE_V) after TCOLEXPAND does NOT wait for the copy-engine
broadcast to complete, so a subsequent vector read can see stale data.

This is a regression from commit 021789c ("refactor: extract Event CRTP
base class and replace opPipeList with OpPipeEntry template"), which
tagged all expand-family ops as PIPE_V. Before that refactoring,
TCOLEXPAND was not in the old opPipeList and defaulted to PIPE_ALL.

All other TCOLEXPAND* / TROWEXPAND* variants use vector instructions
(vadd, vdiv, vmul, vector_dup, vbrcb, etc.) and remain correctly on
PIPE_V.

Co-authored-by: Vladimir Loncar <vloncar@users.noreply.github.com>
@georgebisbas
georgebisbas force-pushed the fix-tcolexpand-pipe-sync branch from 0117c76 to ab06a21 Compare July 28, 2026 14:00
@georgebisbas

Copy link
Copy Markdown
Contributor Author

@Crystal-wzy please have a look wwhen you have available time, thanks

@Crystal-wzy

Copy link
Copy Markdown
Collaborator

@Crystal-wzy please have a look wwhen you have available time, thanks

Sorry for the late reply. The GitHub pto-isa repository no longer accepts merge requests. Please submit your pull request at https://gitcode.com/cann/pto-isa instead. Thank you!

@Crystal-wzy Crystal-wzy closed this Aug 6, 2026
@georgebisbas

Copy link
Copy Markdown
Contributor Author

@Crystal-wzy please have a look wwhen you have available time, thanks

Sorry for the late reply. The GitHub pto-isa repository no longer accepts merge requests. Please submit your pull request at https://gitcode.com/cann/pto-isa instead. Thank you!

Hi we have submitted this already, please have a look when you have time, thanks @Crystal-wzy !
https://gitcode.com/cann/pto-isa/pull/1406

1 similar comment
@georgebisbas

Copy link
Copy Markdown
Contributor Author

@Crystal-wzy please have a look wwhen you have available time, thanks

Sorry for the late reply. The GitHub pto-isa repository no longer accepts merge requests. Please submit your pull request at https://gitcode.com/cann/pto-isa instead. Thank you!

Hi we have submitted this already, please have a look when you have time, thanks @Crystal-wzy !
https://gitcode.com/cann/pto-isa/pull/1406

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants