Skip to content

[WaveTransform] Redesign lane mask contribution mechanism to propagate known subsets of exec - #3804

Open
lalaniket8 wants to merge 5 commits into
amd-feature/wave-transformfrom
amd/dev/lalaniket8/wt-redesign
Open

[WaveTransform] Redesign lane mask contribution mechanism to propagate known subsets of exec#3804
lalaniket8 wants to merge 5 commits into
amd-feature/wave-transformfrom
amd/dev/lalaniket8/wt-redesign

Conversation

@lalaniket8

@lalaniket8 lalaniket8 commented Aug 7, 2026

Copy link
Copy Markdown

Replace the analysis-driven, all-ones–materializing lane-mask merge with an explicit LaneMaskKind classification supplied by the caller during rewrite step. Callers now describe how each contribution relates to EXEC; buildMergeLaneMasks switches on that kind to pick the cheapest merge.

LaneMaskKind (new)

Enum classifying a lane mask vs. EXEC:

  • ZeroReg & EXEC == 0
  • SubsetReg & EXEC == Reg
  • ExecEXEC or -1 (Reg & EXEC == EXEC)
  • NeedsMask — superset/unknown; emit Reg & EXEC

classifyLaneMask() derives it (EXEC → Exec, const ±1/0, subset via LMA, else NeedsMask).

buildMergeLaneMasks

  • Takes LaneMaskKind CurKind instead of AMDGPULaneMaskAnalysis *LMA; branch selection is now a switch.
  • Prev-zero + Zero emits MOV 0; accumulator + Zero elides the merge (copy only when Dst != Prev).

AMDGPULaneMaskUpdater

  • addAvailable() gains a LaneMaskKind Kind argument.
  • Dropped LMA member and setLaneMaskAnalysis().

AMDGPUWaveTransform

  • Removed the RegAllOnes (-1) materialization and its dead-reg cleanup; EXEC/all-lanes contributions use LMC.ExecReg directly with LaneMaskKind::Exec.
  • Each addAvailable() call passes an explicit kind (Exec/Zero/Subset).
  • Added assert(!LaneOrigin.CondIsUndef).

AMDGPUWaitSGPRHazards (fix)

When sinking a wait before a terminator, anchor it before the first terminator (e.g. leading SI_WAVE_CF_EDGE) instead of before the branch, so it is not placed between terminator pseudos.

Effect

  • No S_MOV_B64 -1 emitted for all-lanes contributions.
  • Fewer redundant self-copies in accumulator merges.
  • No functional change to computed EXEC values.

@rocm-cciapp

rocm-cciapp Bot commented Aug 7, 2026

Copy link
Copy Markdown

@lalaniket8
lalaniket8 marked this pull request as ready for review August 7, 2026 09:34

@cdevadas cdevadas left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look reasonable. But delay this patch until PR #3783 gets merged.

// Do not optimize over branches
if (PrevWait && (MI.isCall() || MI.isReturn() || MI.isBranch())) {
PrevWait->moveBefore(&MI);
// Anchor before leading terminator pseudos (e.g. SI_WAVE_CF_EDGE).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I posted PR #3813. Revert these changes later after it gets merged.

@rocm-cciapp

rocm-cciapp Bot commented Aug 7, 2026

Copy link
Copy Markdown

@lalaniket8
lalaniket8 force-pushed the amd/dev/lalaniket8/wt-redesign branch from 53b0f3d to 040d0e6 Compare August 12, 2026 06:18
@rocm-cciapp

rocm-cciapp Bot commented Aug 12, 2026

Copy link
Copy Markdown

Zero, ///< 0 (Reg & EXEC == 0).
Subset, ///< Subset of EXEC (Reg & EXEC == Reg).
Exec, ///< EXEC or -1 (Reg & EXEC == EXEC).
NeedsMask, ///< Superset/unknown; emit Reg & EXEC.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NeedsMask doesn't fit well as a Kind. It could be a None or Unknown for none-of-the-other known kinds. It should go as the first entry and push Zero down.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NeedsMaks cannot be used as else because it represents 2 classes : Superset of exec and cases where no property is known about the register, in which case we conservatively chose to mask it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whether it's a superset or an unknown value, you eventually need a new Kind. NeedsMask doesn't accurately describe this category.
In situations like this, when it's difficult to come up with a concise and meaningful name, None or Unknown are commonly used. They represent the default state for values that don't naturally fit into any of the other defined MaskKinds.

Comment on lines +113 to +114
Subset, ///< Subset of EXEC (Reg & EXEC == Reg).
Exec, ///< EXEC or -1 (Reg & EXEC == EXEC).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related question here;
Why do you have to add a single kind for both EXEC and AllOnes scenarios?
Don't we need a separate AllOnes kind just like Zero?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good question and relates to what earlier happened to the contributed AllOnes in the buildMergeLaneMasks section:
It has a bunch of ifs and elses that decide which lane mask instr is built - if it sees 'AllOnes' contributed, it replaces it with exec.

There is a single point in rewrite step2.1 where AllOnes is contributed. So we can safely contribute exec directly.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If AllOnes is replaced here, let's avoid mentioning -1 in the comment. There's no need to leave traces of intermediate designs as we are nearing the upstream effort.

}

/// Classify \p Reg by its relationship to EXEC; the all-ones constant (-1)
/// maps to \ref LaneMaskKind::Exec since -1 & EXEC == EXEC.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to understand what scenario this -1 & EXEC == EXEC is?
Don't we have other combinations with -1?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants