ci(rccl): add host-only unit test workflow by pvallem · Pull Request #9742 · ROCm/rocm-systems

pvallem · 2026-08-06T00:33:31Z

Summary

Adds a GitHub Actions workflow that builds and runs RCCL's CPU-only host unit-test binary (rccl-HostUnitTests, added in #9320) on every PR touching projects/rccl/**. No GPU or network hardware is required — the binary is compiled with hipcc --offload-host-only and links no HIP/ROCm runtime (~275 tests, <1s).

What it does

Runs in a rocm/dev-ubuntu-22.04:7.2 container on ubuntu-latest:

Configure RCCL and build the hipify_all target to stage hipified sources (prerequisite for test/host; no device-kernel compile).
Configure + build projects/rccl/test/host.
Run rccl-HostUnitTests --gtest_output=xml — non-zero exit fails the job.
Upload the JUnit XML as an artifact.

The workflow's own path is included in the paths: trigger so it self-runs on this PR — you should see the RCCL host unit tests / Host unit tests (CPU-only) check execute here.

Making it blocking (follow-up, admin)

This PR only adds the workflow so the check starts running/reporting. To make a failure block merges, the check must be added as a required status check on the develop ruleset — that can only be done after it has appeared once, and is an admin/settings action, not part of this YAML.

Out of scope

ccache, a slimmed/pre-baked toolchain image, and build/hipify caching are intentionally deferred until there's a production baseline; the dominant per-run cost is the ROCm container pull, not the build (~50s) or hipify (~9s).

JIRA ID: AICOMRCCL-1711

Run projects/rccl/test/host/rccl-HostUnitTests (CPU-only, no GPU/network HW, added in #9320) on every PR touching projects/rccl/**. Stages hipified sources via the hipify_all target, builds test/host with hipcc --offload-host-only, runs the suite, and uploads JUnit results. Self-triggers on this PR via the workflow path in the trigger. JIRA ID: AICOMRCCL-1711

therock-pr-bot · 2026-08-06T00:34:34Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	PR does not contain code files — Unit Test auto-passed
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

The rocm/dev-ubuntu-22.04 image ships hipcc/hipify but not cmake or a host compiler, so 'Stage hipified sources' failed with cmake: not found (exit 127). Add cmake, build-essential, git, python3, rocm-cmake to the deps and put /opt/rocm/bin on PATH. JIRA ID: AICOMRCCL-1711

tee the binary's stdout+stderr (WARN/debug output, per-test lines, any crash backtrace) to host_tests.log and upload it alongside the JUnit XML. pipefail preserves the test exit code so failures still gate. The log is uploaded with if: always(), so it is available precisely when tests fail. JIRA ID: AICOMRCCL-1711

prasanna-amd

Changes look good

Please look at the comment below, not sure if its a major concern.

actions/checkout@08c6903... is labeled # v5.0.0 but checkout v5.0.0 doesn't exist — the current release line is v4.x. That SHA is actually from v3.3.0 (March 2023). This is misleading. If the
intent is to use a recent version, it should be v4.x. If the intent is to pin an old known-good version, the comment should be accurate. Either way, the comment and the SHA disagree, and v3.3.0
lacks Node 20 support (GitHub deprecated Node 16 runners).
actions/upload-artifact@043fb46... is labeled # v7.0.1 — worth verifying this SHA actually corresponds to that tag. upload-artifact v4.x is the current stable line; v7 doesn't exist publicly.

Pipe the test output through moreutils ts to prepend a real-UTC timestamp to every line before tee-ing to host_tests.log, so the downloaded artifact carries its own timeline (the raw binary output has none). pipefail keeps the test exit code so failures still gate. Swap to 'ts -s' for a relative start=0 timeline. JIRA ID: AICOMRCCL-1711

mch

LGTM

pvallem · 2026-08-06T15:27:15Z

Changes look good

Please look at the comment below, not sure if its a major concern.

actions/checkout@08c6903... is labeled # v5.0.0 but checkout v5.0.0 doesn't exist — the current release line is v4.x. That SHA is actually from v3.3.0 (March 2023). This is misleading. If the
intent is to use a recent version, it should be v4.x. If the intent is to pin an old known-good version, the comment should be accurate. Either way, the comment and the SHA disagree, and v3.3.0
lacks Node 20 support (GitHub deprecated Node 16 runners).

actions/upload-artifact@043fb46... is labeled # v7.0.1 — worth verifying this SHA actually corresponds to that tag. upload-artifact v4.x is the current stable line; v7 doesn't exist publicly.

 git ls-remote --tags https://github.com/actions/checkout.git | egrep 'v5.0.0|v7.0.0|v7.0.1'
08c6903cd8c0fde910a37f88322edcfb5dd907a8        refs/tags/v5.0.0
9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0        refs/tags/v7.0.0
3d3c42e5aac5ba805825da76410c181273ba90b1        refs/tags/v7.0.1
====
git ls-remote --tags https://github.com/actions/upload-artifact.git | egrep 'v5.0.0|v7.0.0|v7.0.1'
330a01c490aca151604b8cf639adc76d48f6c5d4        refs/tags/v5.0.0
bbbca2ddaa5d8feaa63e36b76fdaad77386f024f        refs/tags/v7.0.0
043fb46d1a93c77aae656e7c1c64a875d1fc6a0a        refs/tags/v7.0.1

Align both pinned actions on the v7.0.1 release line (checkout 3d3c42e, upload-artifact 043fb46 -- different repos, so different SHAs, same tag). checkout v7.0.1 is already used elsewhere in rocm-systems. JIRA ID: AICOMRCCL-1711

Extract the configure/build/run/log logic from the workflow into test/host/run_host_tests.sh so the same command runs locally and in CI, and so the suite (GTEST_FILTER, extra gtest flags, build type) can be tuned in one place without editing the workflow. The workflow now just invokes the script after staging the hipify tree. Behavior is unchanged: UTC-timestamped log via ts, JUnit XML, pipefail-preserved exit code. JIRA ID: AICOMRCCL-1711

@geomin12

…kout Incorporate @geomin12 review feedback on the host-UT workflow: - restrict pull_request trigger to branches: [develop] - rename job -> rccl-host-unit-tests for clarity - pin runner to ubuntu-24.04 (was ubuntu-latest) - sparse-checkout only projects/rccl (rocm-systems is large) - split RCCL configure vs hipify build into separate steps - run_host_tests.sh gains phase dispatch (configure|build|run|all); the workflow invokes it as three separate steps so a failure is clearly attributable, while keeping build/run logic and knobs in one script - lift GPU_TARGETS=gfx942 to a job-level env (documented: prevents GPU probe) JIRA ID: AICOMRCCL-1711

Centralize every pipeline command in the script (single source of truth) so nothing is scattered in the workflow YAML. Adds rccl-configure and hipify phases (operating on the RCCL root) alongside configure/build/run (test/host); 'all' chains them. The workflow now just invokes each phase as its own step, keeping per-phase failure attribution. GPU_TARGETS is a script knob (default gfx942), still overridable via the workflow's job-level env. JIRA ID: AICOMRCCL-1711

@geomin12

Per review (@geomin12): record intent to migrate this standalone workflow into TheRock's packaged CI once the host-test gate stabilizes. JIRA ID: AICOMRCCL-1711

github-actions · 2026-08-07T18:08:50Z

RCCL Perf-Regression Gate: ✅ PASS

Mode: detect (reference vs candidate)
Thresholds (per-tier, calibrated): small 17.2% · mid 12.3% · large 12.9%
Keys compared: 0 · Confirmed regressions: 0 · Inconclusive: 0

Per-collective breakdown

group	keys	regressions	inconclusive
all_gather_perf-d=bfloat16-default	0	0	0
all_gather_perf-d=float-default	0	0	0
all_reduce_perf-d=bfloat16-default	0	0	0
all_reduce_perf-d=float-default	0	0	0
broadcast_perf-d=bfloat16-default	0	0	0
broadcast_perf-d=float-default	0	0	0
reduce_scatter_perf-d=bfloat16-default	0	0	0
reduce_scatter_perf-d=float-default	0	0	0

geomin12

thanks. lgtm

alex-breslow-amd

LGTM, see my comment about adding -x for improved logging if this fails

pvallem · 2026-08-07T21:31:38Z

Sure will add -x in follow up pr.

…4, TODO, -x Mirror the fixes from PR #9742's review on the rccl formatting workflow: - scope pull_request to the develop branch (geomin12) - pin the runner to ubuntu-24.04 instead of ubuntu-latest (geomin12) - add a TODO to fold per-project *-formatting.yml into TheRock packaging (geomin12) - set -x in the run step for more useful failure logs (alex-breslow) - name the step for rccl clarity (sparse-checkout of projects/rccl was already present.) JIRA ID: AITESTAUTO-6995

MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Adds a GitHub Actions workflow that builds and runs RCCL's CPU-only host unit-test binary (`rccl-HostUnitTests`, added in #9320) on every PR touching `projects/rccl/**`. No GPU or network hardware is required — the binary is compiled with `hipcc --offload-host-only` and links no HIP/ROCm runtime (~275 tests, <1s). ## What it does Runs in a `rocm/dev-ubuntu-22.04:7.2` container on `ubuntu-latest`: 1. Configure RCCL and build the `hipify_all` target to stage hipified sources (prerequisite for `test/host`; no device-kernel compile). 2. Configure + build `projects/rccl/test/host`. 3. Run `rccl-HostUnitTests --gtest_output=xml` — non-zero exit fails the job. 4. Upload the JUnit XML as an artifact. The workflow's own path is included in the `paths:` trigger so it self-runs on this PR — you should see the **RCCL host unit tests / Host unit tests (CPU-only)** check execute here. ## Making it blocking (follow-up, admin) This PR only adds the workflow so the check starts running/reporting. To make a failure block merges, the check must be added as a **required status check** on the `develop` ruleset — that can only be done after it has appeared once, and is an admin/settings action, not part of this YAML. ## Out of scope ccache, a slimmed/pre-baked toolchain image, and `build/hipify` caching are intentionally deferred until there's a production baseline; the dominant per-run cost is the ROCm container pull, not the build (~50s) or hipify (~9s). JIRA ID: AICOMRCCL-1711 [rocm-systems] ROCm/rocm-systems#9742 (commit e484606)

The kernel-count leak guards were registered only via add_test() in test/CMakeLists.txt, but nothing in RCCL CI runs `ctest`, so they never gated (the precheckin test_runner runs the rccl-UnitTests gtest binary via --gtest_filter; install.sh runs the gtest binaries directly). Reviewer alex-breslow-amd flagged exactly this on the add_test block. Wire the guards into the CPU-only host-test pipeline instead, which does run in CI on every PR touching projects/rccl/** (rccl-host-unit-tests.yml, added in #9742): - test/host/run_host_tests.sh: add a `do_guards` phase that runs both generators' baseline checks (python3 only -- no rccl build, no hipify, no GPU). Refactor the gtest suite into do_host_tests and make `do_run` an aggregator (do_host_tests + do_guards) so future CPU-only checks slot in cleanly and both `run` and `all` pick them up. Folding guards into the `run` phase gets them executed by the existing workflow step with no change to the workflow YAML. do_host_tests runs first so the JUnit XML is always produced before a guard can gate. A standalone `guards` phase is also exposed. - test/CMakeLists.txt: drop the two dead add_test() entries and leave a note pointing at the host-test pipeline. The Python guards themselves are unchanged and stay next to generate.py.

pvallem requested a review from a team as a code owner August 6, 2026 00:33

github-actions Bot added the github actions Pull requests that update GitHub Actions code label Aug 6, 2026

systems-assistant Bot added the organization: ROCm label Aug 6, 2026

prasanna-amd approved these changes Aug 6, 2026

View reviewed changes

mch approved these changes Aug 6, 2026

View reviewed changes

Prasannakumar Murugesan added 2 commits August 6, 2026 09:20

ci(rccl): bump actions/checkout to v7.0.1 for consistency

4b033e7

Align both pinned actions on the v7.0.1 release line (checkout 3d3c42e, upload-artifact 043fb46 -- different repos, so different SHAs, same tag). checkout v7.0.1 is already used elsewhere in rocm-systems. JIRA ID: AICOMRCCL-1711

pvallem requested a review from a team as a code owner August 6, 2026 19:43

Merge branch 'develop' into users/pvallem/rccl-host-ut-ci

f3bb439

geomin12 reviewed Aug 6, 2026

View reviewed changes

Prasannakumar Murugesan added 2 commits August 6, 2026 14:21

pvallem requested a review from geomin12 August 6, 2026 21:29

pvallem closed this Aug 6, 2026

pvallem reopened this Aug 6, 2026

geomin12 reviewed Aug 6, 2026

View reviewed changes

Comment thread projects/rccl/test/host/run_host_tests.sh

Comment thread .github/workflows/rccl-host-unit-tests.yml

ci(rccl): add TODO to fold host-UT workflow into TheRock

683d6f1

Per review (@geomin12): record intent to migrate this standalone workflow into TheRock's packaged CI once the host-test gate stabilizes. JIRA ID: AICOMRCCL-1711

github-actions Bot added the project: rccl label Aug 6, 2026

pvallem requested a review from geomin12 August 6, 2026 23:34

pvallem mentioned this pull request Aug 7, 2026

test(rccl): consolidate host-only microtest framework into test/host #9817

Open

geomin12 approved these changes Aug 7, 2026

View reviewed changes

Merge branch 'develop' into users/pvallem/rccl-host-ut-ci

ff5d269

pvallem requested a review from a team August 7, 2026 21:14

alex-breslow-amd reviewed Aug 7, 2026

View reviewed changes

Comment thread projects/rccl/test/host/run_host_tests.sh

alex-breslow-amd approved these changes Aug 7, 2026

View reviewed changes

Merge branch 'develop' into users/pvallem/rccl-host-ut-ci

c0bfccc

pvallem merged commit e484606 into develop Aug 10, 2026
33 of 38 checks passed

pvallem deleted the users/pvallem/rccl-host-ut-ci branch August 10, 2026 18:19

Uh oh!

Conversation

pvallem commented Aug 6, 2026

Summary

What it does

Making it blocking (follow-up, admin)

Out of scope

Uh oh!

therock-pr-bot Bot commented Aug 6, 2026

✅ All Policy Checks Passed

Uh oh!

prasanna-amd left a comment

Choose a reason for hiding this comment

Uh oh!

mch left a comment

Choose a reason for hiding this comment

Uh oh!

pvallem commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026

RCCL Perf-Regression Gate: ✅ PASS

Uh oh!

geomin12 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-breslow-amd left a comment

Choose a reason for hiding this comment

Uh oh!

pvallem commented Aug 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pvallem commented Aug 6, 2026 •

edited

Loading