fix(hw-gate): pin lanes by PCI address, not HIP index (the gfx1201 lane was the eGPU) - #716
Conversation
Every gfx1201 hardware lane has been benchmarking the Thunderbolt eGPU.
HIP enumerates in KFD-node order, which is neither PCI bus order nor stable
across hotplug. Read from this host's live topology:
HIP index map: 0=03:00.0 1=c3:00.0 2=e3:00.0 3=4b:00.0 4=13:00.0
The lane matrix said `device: "3"`, intending "the fourth mainboard card".
Index 3 is 4b:00.0 -- the R9700 behind a JHL9480 TB5 bridge on an x4 @ 16GT/s
link, versus x16 @ 32GT/s for the other four. rocm-smi confirmed it live: with
a lane running, GPU[2] (=4b) sat at 99% while the four mainboard cards were at
0%. So the flakiest, narrowest-linked, dock-attached card in the box was
carrying every gate benchmark, which is why lanes died with
`MES(1) failed to respond to msg=REMOVE_QUEUE` on 4b *during* a run, and why
that card was the one fwupd's DP-AUX storm took down.
It also moved: when the eGPU re-enumerated from 7b to 4b after a tunnel cycle,
every index after it shifted, so the same literal `3` pointed at a different
GPU than it had an hour earlier. A hardcoded index cannot express "this card".
Lanes now pin a PCI address (`pci: "0000:c3:00.0"` / `"0000:03:00.0"`) and
resolve it to an index at lane start via scripts/hw-gate/resolve_device.py,
which walks the KFD topology, skips the CPU node (simd_count 0), and unpacks
`location_id` as (bus << 8) | device. A pinned card that is absent is a hard
error naming what IS present -- benchmarking whichever GPU happens to hold an
index is the defect, so falling back to one would reintroduce it.
This does not disqualify the eGPU. It makes the choice explicit, so giving that
card a lane becomes a decision rather than an accident.
Tests: scripts/hw-gate/tests/test_resolve_device.py -- KFD order vs bus order,
the CPU node not consuming an index, the 7b->4b shift, absent-card hard error,
address forms, and both CLI exit paths. It pins `4b:00.0 -> 3` explicitly so
the historical defect cannot be reintroduced as "just use index 3". 115/115
hw-gate tests pass, and the resolver was checked against the live topology
(`--pci 0000:c3:00.0` -> `HW_GATE_DEVICE=1`).
hw-gate sol prelimsummary: Replaces fixed HIP indices in the hardware-lane matrix with PCI-address pins, adds a KFD-topology resolver that maps the selected card to this boot's HIP index and checks its gfx architecture, then exports that index for the existing gate runner. The resolver only reads KFD sysfs and writes one environment assignment; an absent or mismatched card fails closed. run_hardware: true routes:
unavailable_routes: (none) claim_assessment: The author claims the gfx1201 and gfx1100 lanes will resolve stable PCI pins to the intended HIP indices and reject absent, moved, or wrong-architecture cards. The diff and unit fixtures support KFD-order mapping and fail-closed behavior, but proof requires post-merge/base-branch execution on both named runners showing the emitted HIP index map, selected BDF, expected gfx, and successful handoff of HW_GATE_DEVICE. This PR's pull_request_target run uses base-branch gate scripts and therefore does not exercise the new resolver. questions_for_author:
|
…ress
Follow-up to the PCI pinning in this PR, prompted by a bug in the pinning
itself: the hipx lane was pinned to 0000:03:00.0, which is a hiptrx address.
hipx's gfx1100 is 0000:66:00.0 — the two hosts share no PCI addresses, and I
had assumed otherwise. Verified against both topologies:
hiptrx 0=03:00.0(gfx1201) 1=c3:00.0(gfx1201) 2=e3:00.0(gfx1201)
3=4b:00.0(gfx1201) 4=13:00.0(gfx1201)
hipx 0=66:00.0(gfx1100) 1=bf:00.0(gfx1151) 2=6e:00.0(gfx1030)
3=99:00.0(gfx1010)
Left uncaught, that would have failed every hipx lane on the first run after
merge — the same class of silent-wrong-card error the PR set out to remove,
just louder.
So the pin is now self-checking. Lanes pass `--expect-gfx $HW_GATE_GFX` (the
arch already declared in the lane matrix) and the resolver compares it against
the node's `gfx_target_version`, failing with every card listed. A wrong
address, a swapped card, and a re-slotted card are all now hard errors instead
of a benchmark on the wrong GPU.
`gfx_name()` decodes `major * 10000 + minor * 100 + step` with hex digits for
minor and step, verified on both hosts: 120001 -> gfx1201, 110000 -> gfx1100,
110501 -> gfx1151, 100300 -> gfx1030.
Tests: six more cases — the decoder against all four observed values, the
arch assertion accepting the right card and rejecting a swapped one, the
hiptrx-address-on-hipx mistake with the error naming the correct card, and the
CLI's nonzero exit on mismatch. 120/120 hw-gate tests pass, and all three paths
were run live against real topology on both hosts.
|
Found and fixed a bug in this PR before merge, worth recording because it is the same failure class the PR removes. The hipx lane was pinned to Left uncaught it would have failed every hipx lane on the first run after merge — silent-wrong-card, just louder. So the pin is now self-checking. Lanes pass
120/120 hw-gate tests pass, and all three paths were run live on both hosts: hiptrx |
hw-gate evidence — 2 lane(s) — verdict passlane hiptrx (gfx1201)hw-gate evidence
fixturesno fixtures kernelnot run lane hipx (gfx1100)hw-gate evidence
fixturesno fixtures kernelnot run |
hw-gate sol verdict{
"claim_verdict": "not-exercised",
"confidence": 0.98,
"coverage": {
"gaps": [
"The run did not execute the new PCI resolver: hiptrx reported HW_GATE_DEVICE=3, whereas resolving the pinned c3:00.0 topology described by the PR should produce index 1.",
"No trusted run demonstrates --expect-gfx validation or GITHUB_ENV handoff on either runner.",
"Complete PCI domain and function identity is not validated because parse_pci accepts those fields but resolution discards them."
],
"surfaces_evidenced": [
"existing hardware-gate execution"
],
"surfaces_touched": [
"policy",
"ci",
"hardware-selection"
]
},
"decision": "needs-human",
"eyeball": [
"hw-gate.json host records: hiptrx used device 3 and hipx used device 0, showing the base-branch fixed-index workflow ran rather than the new resolver.",
"scripts/hw-gate/resolve_device.py:39 and resolve(): domain and function are accepted syntactically but omitted from the location_id comparison."
],
"phase": "verdict",
"rationale": "The hardware job passed both lanes, but it exercised the base-branch gate plumbing: hiptrx still selected device 3, the eGPU index this PR intends to eliminate, rather than resolving c3:00.0 to device 1. Therefore the central PCI-selection claim has no runtime evidence. The diff appears fail-closed and its configured domain-0/function-0 pins avoid the identified collision concern on the named hosts, so there is no evidenced regression warranting a block. Human review and a trusted post-merge or workflow-dispatch smoke are required because .github/workflows/hw-gate.yml is policy code and the changed selection path was not exercised.",
"regressions": []
}Floor: hard=['policy_paths: .github/workflows/hw-gate.yml,scripts/hw-gate/resolve_device.py,scripts/hw-gate/tests/test_resolve_device.py'] soft=["coverage_gaps: ['The run did not execute the new PCI resolver: hiptrx reported HW_GATE_DEVICE=3, whereas resolving the pinned c3:00.0 topology described by the PR should produce index 1.', 'No trusted run demonstrates --expect-gfx validation or GITHUB_ENV handoff on either runner.', 'Complete PCI domain and function identity is not validated because parse_pci accepts those fields but resolution discards them.']", 'model needs-human'] model_decision=needs-human final=needs-human |
![hipfire-sol[bot]](https://avatars.githubusercontent.com/u/308720658?s=60&v=4){kind=small}
There was a problem hiding this comment.
hw-gate sol verdict needs-human: Both hardware lanes completed successfully at head 12c2dc3, disproving the preliminary bootstrap concern that base/scripts/hw-gate/resolve_device.py could not execute. However, hw-gate.json preserves only HIP indices and reports hiptrx device 3; without the Resolve-step map it cannot establish that the selected device was the pinned c3:00.0 card. The diff also changes CI policy and silently discards PCI domain/function components in scripts/hw-gate/resolve_device.py:40-44. Human inspection of the workflow log and address semantics is therefore required.
There was a problem hiding this comment.
hw-gate sol verdict needs-human: The hardware job passed both lanes, but it exercised the base-branch gate plumbing: hiptrx still selected device 3, the eGPU index this PR intends to eliminate, rather than resolving c3:00.0 to device 1. Therefore the central PCI-selection claim has no runtime evidence. The diff appears fail-closed and its configured domain-0/function-0 pins avoid the identified collision concern on the named hosts, so there is no evidenced regression warranting a block. Human review and a trusted post-merge or workflow-dispatch smoke are required because .github/workflows/hw-gate.yml is policy code and the changed selection path was not exercised.
1 participant
Every gfx1201 hardware lane has been benchmarking the Thunderbolt eGPU, and I only found it because #682's lanes died mid-run tonight.
The defect
HIP enumerates in KFD-node order, which is neither PCI bus order nor stable across hotplug. This host's live map:
The lane matrix said
device: "3", meaning "the fourth mainboard card". Index 3 is4b:00.0— the R9700 behind a JHL9480 TB5 bridge at x4 @ 16GT/s, versus x16 @ 32GT/s for the other four.rocm-smiconfirmed it with a lane running: GPU[2] (=4b) at 99% while all four mainboard cards sat at 0%.So the narrowest-linked, dock-attached card in the box was carrying every gate benchmark. That explains the lane deaths with
MES(1) failed to respond to msg=REMOVE_QUEUEon4bduring a run, and why that card was the one fwupd's DP-AUX storm took down (dce_aux.c:393,REG_WAIT timeout … acquire_engine).It also moved: when the eGPU re-enumerated
7b→4bafter a tunnel cycle, every index after it shifted, so the same literal3pointed at a different GPU than it had an hour earlier. A hardcoded index cannot express "this card".The fix
Lanes pin a PCI address (
pci: "0000:c3:00.0"/"0000:03:00.0") and resolve it to an index at lane start viascripts/hw-gate/resolve_device.py, which walks the KFD topology, skips the CPU node (simd_count 0), and unpackslocation_idas(bus << 8) | device. An absent pinned card is a hard error naming what is present — falling back to an index would reintroduce the exact defect.This does not disqualify the eGPU. It makes the choice explicit, so giving that card a lane becomes a decision instead of an accident.
Tests
scripts/hw-gate/tests/test_resolve_device.py— KFD order vs bus order, CPU node not consuming an index, the7b→4bshift, absent-card hard error, address forms, both CLI exit paths. It pins4b:00.0 → 3explicitly so nobody reintroduces this as "just use index 3". 115/115 hw-gate tests pass, and the resolver was checked against the live topology (--pci 0000:c3:00.0→HW_GATE_DEVICE=1).Policy-floor by construction (
.github/,scripts/hw-gate/**), so it cannot self-merge.