[Issue]: rocm-libraries multi-arch CI names RPMs with the wheel version — sles16 install test fails on every PR

[AmosLewis]

Update (2026-08-06): the root cause is not the S3/zypper encoding behaviour described below.
rocm-libraries' therock-multi-arch-ci.yml never passes rocm_rpm_package_version, so the ||
fallback in multi_arch_ci_linux.yml puts the wheel version (10.1.0.dev0+<sha>) into the RPM
filenames instead of the intended 10.1.0~<date>g<sha>. TheRock's own CI passes all four install
lanes. See the discussion below; the original text is kept for the reproduction details.

Summary

Linux::release / Test RPM Install - sles16 fails on every PR and has done so since at least
2026-06-20. All 367 RPMs return HTTP 404 under zypper, while rhel8 / rhel10 / ubuntu2404 pass on
the same run. This makes Multi-Arch CI Summary red repo-wide and forces manual overrides
(most recently ROCm/rocm-libraries#10180, which is blocking SGLang).

Root cause

RPM filenames embed a + from the dev version 10.1.0.dev0+<sha>. The S3 REST endpoint decodes a
literal + in the URL path as a space, so the object is only reachable when it is sent as %2B.

B=https://therock-ci-artifacts-external.s3.amazonaws.com/ROCm-rocm-libraries/31063962255-linux/packages/rpm/x86_64
F='amdrocm-10.1.0.dev0+84be6e7935adc8647edbc2f74fdd71bd1a9730c9-31063962255.x86_64.rpm'

curl -o /dev/null -w '%{http_code}\n' "$B/$F"            # 404  literal '+'
curl -o /dev/null -w '%{http_code}\n' "$B/${F//+/%2B}"   # 200  '%2B'
curl -o /dev/null -w '%{http_code}\n' "$B/${F//+/%20}"   # 404  space

The stored key does contain a literal +, so the upload itself is fine:

<Key>.../packages/rpm/x86_64/amdrocm-10.1.0.dev0+84be6e79...-31063962255.x86_64.rpm</Key>

dnf and apt percent-encode + and succeed; zypper sends the literal + and 404s on every
package. Repo metadata (repomd.xml, primary.xml.gz) downloads fine, which is why zypper gets as
far as resolving package names before failing.

Failing log excerpt:

Preloading: amdrocm-ck10.1-...x86_64.rpm [The requested URL returned error: 404]
Installation has completed with error.
[FAIL] Failed to install RPM packages (exit code: 4)
[FAIL] Step 1 (repo setup and install) failed.

Evidence

Reproduced identically against a second run's artifacts (31099145995), so it is not a one-off
upload problem. sles16 is red on every unrelated run sampled, while the other three OS lanes pass:

Run	rhel8	rhel10	ubuntu2404	sles16
31099145995	pass	pass	pass	fail
31096739996	pass	pass	pass	fail
31094598762	pass	pass	pass	fail
31092050233	pass	pass	pass	fail
31058200317 (Aug 5)	pass	pass	pass	fail
30179912967 (Jul 25)	pass	pass	pass	fail
27865212542 (Jun 20)	pass	—	pass	fail

Example failing job:
https://github.com/ROCm/rocm-libraries/actions/runs/31063962255/job/92660497427

Impact

• Every rocm-libraries PR needs a manual override to merge.
• Likely also affects real SLES users installing from any S3-backed ROCm repo, not just CI.

Possible fixes

1. Drop + from the package version (e.g. 10.1.0.dev0.<sha> or ~<sha>). No client needs to
   encode anything. Changes package NEVRA, so it needs packaging-owner sign-off.
2. Percent-encode + as %2B in the repodata <location href> after createrepo_c
   (build_tools/packaging/linux/upload_package_repo.py). Needs validation that libzypp does not
   unescape it back to a literal +.
3. Make the sles16 test download RPMs locally then zypper install ./*.rpm. Fixes CI only, not
   end users.

I could not validate options 2 and 3 locally — pulling registry.suse.com/bci/bci-base:16.0 is
blocked by a TLS-intercepting proxy on my network.

[ScottTodd]

rpm filenames in TheRock don't have this issue, only those in rocm-libraries? Or only those using therock-ci-artifacts-external?

Versions look the same, I have a suspicion that the server-side index page generation is differing between therock-ci-artifacts and therock-ci-artifacts-external

Example:

• https://github.com/ROCm/TheRock/actions/runs/30955580254/job/92147690819

  Loading version from file '/home/runner/work/TheRock/TheRock/version.json'
  Base version  : '10.1.0'
  Package type  : 'wheel'
  Version suffix: '.dev0+7ffefad73cdf0d4a38d2bf2cb4cda9959e3cce7c'
  Full version  : '10.1.0.dev0+7ffefad73cdf0d4a38d2bf2cb4cda9959e3cce7c'
  Loading version from file '/home/runner/work/TheRock/TheRock/version.json'
  Base version  : '10.1.0'
  Package type  : 'deb'
  Version suffix: '~dev20260804'
  Full version  : '10.1.0~dev20260804'
  Loading version from file '/home/runner/work/TheRock/TheRock/version.json'
  Base version  : '10.1.0'
  Package type  : 'rpm'
  Version suffix: '~20260804g7ffefad7'
  Full version  : '10.1.0~20260804g7ffefad7'
  Setting github output:
  {
    "rocm_package_version": "10.1.0.dev0+7ffefad73cdf0d4a38d2bf2cb4cda9959e3cce7c",
    "rocm_deb_package_version": "10.1.0~dev20260804",
    "rocm_rpm_package_version": "10.1.0~20260804g7ffefad7"
  }
  OUTPUT rocm_package_version=10.1.0.dev0+7ffefad73cdf0d4a38d2bf2cb4cda9959e3cce7c
  OUTPUT rocm_deb_package_version=10.1.0~dev20260804
  

• https://therock-ci-artifacts.s3.amazonaws.com/30955580254-linux/packages/rpm/x86_64/index.html
• https://github.com/ROCm/TheRock/actions/runs/30955580254/job/92165809483#step:7:134

  Preloading: amdrocm-fft10.1-gfx1103-10.1.0~20260804g7ffefad7-30955580254.x86_64.rpm [done]
  

Compared to rocm-libraries:

• https://github.com/ROCm/rocm-libraries/actions/runs/31063962255/job/92660524984

  Loading version from file '/home/runner/work/rocm-libraries/rocm-libraries/version.json'
  Base version  : '10.1.0'
  Package type  : 'wheel'
  Version suffix: '.dev0+84be6e7935adc8647edbc2f74fdd71bd1a9730c9'
  Full version  : '10.1.0.dev0+84be6e7935adc8647edbc2f74fdd71bd1a9730c9'
  Loading version from file '/home/runner/work/rocm-libraries/rocm-libraries/version.json'
  Base version  : '10.1.0'
  Package type  : 'deb'
  Version suffix: '~dev20260806'
  Full version  : '10.1.0~dev20260806'
  Loading version from file '/home/runner/work/rocm-libraries/rocm-libraries/version.json'
  Base version  : '10.1.0'
  Package type  : 'rpm'
  Version suffix: '~20260806g84be6e79'
  Full version  : '10.1.0~20260806g84be6e79'
  Setting github output:
  {
    "rocm_package_version": "10.1.0.dev0+84be6e7935adc8647edbc2f74fdd71bd1a9730c9",
    "rocm_deb_package_version": "10.1.0~dev20260806",
    "rocm_rpm_package_version": "10.1.0~20260806g84be6e79"
  }
  OUTPUT rocm_package_version=10.1.0.dev0+84be6e7935adc8647edbc2f74fdd71bd1a9730c9
  OUTPUT rocm_deb_package_version=10.1.0~dev20260806
  OUTPUT rocm_rpm_package_version=10.1.0~20260806g84be6e79
  

• https://therock-ci-artifacts-external.s3.amazonaws.com/ROCm-rocm-libraries/31063962255-linux/packages/rpm/x86_64/index.html

[ScottTodd]

Index page generation seems to be the same in both buckets, I see similar URLs in both? The tests also pass in TheRock for PRs from forks that use therock-ci-artifacts-external like https://github.com/ROCm/TheRock/actions/runs/30424683400?pr=6954

[nunnikri]

This is strange, The versions computed initially is correct.

OUTPUT rocm_package_version=10.1.0.dev0+84be6e7935adc8647edbc2f74fdd71bd1a9730c9
OUTPUT rocm_deb_package_version=10.1.0~dev20260806
OUTPUT rocm_rpm_package_version=10.1.0~20260806g84be6e79

But somewhere in the workflow seems wheel package version is getting passed to deb/rpm. Let me take a look

[ScottTodd]

Ah, in the workflows that were forked to rocm-libraries:

• https://github.com/ROCm/rocm-libraries/blob/24975741a1e0bf3a80a4f09adb04c2cd86b0f870/.github/workflows/therock-multi-arch-ci.yml#L177
• compared to TheRock:

  TheRock/.github/workflows/multi_arch_ci.yml

  Lines 127 to 129 in e94ddfd

  	rocm_package_version: ${{ needs.setup.outputs.rocm_package_version }}
  	rocm_deb_package_version: ${{ needs.setup.outputs.rocm_deb_package_version }}
  	rocm_rpm_package_version: ${{ needs.setup.outputs.rocm_rpm_package_version }}

why does this have an || for rocm_version: ${{ inputs.rocm_rpm_package_version || inputs.rocm_package_version }}?

TheRock/.github/workflows/multi_arch_ci_linux.yml

Lines 168 to 181 in e94ddfd

	build_native_rpm_packages:
	needs: [build_multi_arch_stages]
	name: Build RPM Packages
	if: ${{ !failure() && !cancelled() && fromJSON(inputs.build_config).build_native_linux == true }}
	uses: ./.github/workflows/multi_arch_build_native_linux_packages.yml
	with:
	dist_amdgpu_families: ${{ fromJSON(inputs.build_config).dist_amdgpu_families }}
	rocm_version: ${{ inputs.rocm_rpm_package_version || inputs.rocm_package_version }}
	native_package_type: rpm
	repository: ${{ inputs.repository }}
	ref: ${{ inputs.ref }}
	permissions:
	contents: read
	id-token: write

Bug in #5799