[Code Health] Pod L4 examples are enumerated by hand instead of collected by the ST framework

[ChaoWao]

Category

Technical Debt (cleanup, refactor)

Component

Tests

Description

The pod job registers each L4 two-machine example as its own hand-written workflow step, while every other worker example is collected by pytest and scheduled by the ST framework in conftest.py. The asymmetry is not cosmetic: it means the L4 examples get none of what that framework provides, and adding one costs four edits to _st-pod.yml instead of zero.

How the L2/L3 examples are wired. Each one ships a thin test_*.py next to its main.py that declares its resource needs as markers and delegates to main.run(...), e.g. examples/workers/l3/allreduce/test_allreduce.py:

@pytest.mark.platforms(["a2a3sim", "a2a3"])
@pytest.mark.runtime("tensormap_and_ringbuffer")
@pytest.mark.device_count(2)
def test_allreduce(st_platform, st_device_ids):
    rc = run([int(d) for d in st_device_ids], platform=st_platform)
    assert rc == 0

conftest.py::_collect_resource_jobs picks these up as kind="standalone" jobs (any non-class function carrying device_count + runtime), bin-packs them with the L3 SceneTestCase classes through parallel_scheduler.run_jobs, and hands each one its devices via the st_device_ids fixture. One pytest examples tests/st invocation in _st-npu-a2a3.yml collects all of them; --platform filtering, device allocation, marker-based skipping and quarantine (-m "not sdma") all come for free.

Counting test_*.py under examples/workers/: l2 = 5, l3 = 10, l4 = 0. The three L4 examples have only main.py + run_parent.sh.

What the L4 examples get instead. Three near-identical 8-line steps in _st-pod.yml, each with a hand-assigned id:, plus a matching line in the Report pod example results shell script, because continue-on-error: true rewrites a failed step's conclusion to success and only outcome carries the truth.

Consequences:

1. Registration is manual and duplicated in two places. Adding a fourth example means: a new step, a unique id:, a new env-prefix, and a new line in the summary script. Forgetting the summary line is silent — the example runs, fails, and the job still reports green, which is the exact failure mode .claude/rules/ci-change-detection.md §7 warns about ("a gating bug shows up as a green check").
2. No platform declaration. The L2/L3 examples state @pytest.mark.platforms([...]). The L4 examples encode their target platform as a shell default inside run_parent.sh (SIMPLER_..._PLATFORM:=a2a3), so nothing prevents the pod job from invoking one against a platform it was never written for, and nothing skips it when the pod machines don't match.
3. conftest.py has no concept of a second machine at all. Grepping it for remote / peer / node_count / pod returns nothing. So this is a genuine framework gap, not an example that forgot to opt in — the st_device_ids fixture allocates local devices from a local pool, and there is no fixture that could hand a test the peer's endpoint.
4. The two suites can drift. A conftest.py change to device allocation, platform filtering or quarantine applies to 15 examples and silently skips 3.

Location

- `.github/workflows/_st-pod.yml` — the three enumerated `pod-run-example` steps and the `Report pod example results` script that mirrors them
- `conftest.py::_collect_resource_jobs` — collects `kind="standalone"` jobs from `device_count` + `runtime` markers; no multi-machine notion
- `conftest.py::st_platform`, `conftest.py::st_device_ids` — allocate from a local device pool only
- `examples/workers/l3/allreduce/test_allreduce.py` — the pattern the L4 examples do not follow
- `examples/workers/l4/{vector_add,global_tload,compute_then_tload}_mixed_l3/` — `main.py` + `run_parent.sh`, no `test_*.py`

Proposed Fix

The blocker is that a pod test needs the peer endpoint, and no fixture can currently supply it. Two options, in increasing scope:

(a) Collapse the enumeration without touching conftest.py. Keep the shell driver, but make the step list data instead of copy-paste — a matrix over (example, env-prefix, parent-script), or a single step looping over the examples and accumulating failures. This removes the summary-script duplication (consequence 1) and nothing else. Small, and worth doing regardless.

(b) Give the framework a pod fixture and let pytest collect the L4 examples too. Add a peer_endpoint fixture that reads the pod .env values the workflow already resolves (POD_REMOTE_HOST, POD_L3_DAEMON_PORT, POD_REMOTE_DEVICES) and skips when they're absent, plus a marker for "needs a peer machine" so _collect_resource_jobs can route it. Then each L4 example gets the same thin test_*.py its L2/L3 siblings have, and _st-pod.yml shrinks to one pytest invocation with a peer-selecting marker. This addresses all four consequences, and starting the daemon stays in the action (it must outlive the step).

Note that (b) subsumes (a) but is meaningfully larger, since the daemon lifecycle currently lives in .github/actions/pod-run-example and would need to be reachable from a fixture or hoisted to a once-per-job step. Suggest doing (a) now and treating (b) as the real fix.

Whichever path: the summary-script mirroring should go away, since a registration that can be half-completed and still report green is the part most likely to bite.

Priority

Medium (minor risk, should fix in next few releases)

[ccyywwen]

Implemented in #1758.

Summary:

• Replaced the hand-written L4 pod example enumeration in _st-pod.yml with a single pytest sweep:
  pytest examples tests/st -m pod --platform a2a3 --device ... --max-parallel 1
• Added pod pytest support in conftest.py:
  • @pytest.mark.pod
  • @pytest.mark.pod_remote_device_count(n)
  • st_pod_peer
  • st_pod_remote_device_ids
  • st_pod_logs
• Added thin pytest wrappers for the existing L4 pod examples. Adding a new pod example now means adding a test_*.py wrapper with @pytest.mark.pod, not editing _st-pod.yml.
• Added a tests/st sentinel pod test to verify pod CI collects pod-marked tests from both examples/ and tests/st/.
• Removed the old per-example pod-run-example action and the separate summary-step mirroring.
• Kept the peer daemon lifecycle in the CI action, where it can outlive the pytest invocation.
• Explicitly excluded pod tests from ordinary onboard ST lanes:
  • a2a3 main sweep: not sdma and not pod
  • a2a3 legacy SDMA ignore sweep: not pod
  • a5 ARM64: not pod
  • a5 x86_64: not sdma and not pod
• Left sim lanes without -m "not pod"; pod tests are filtered by platform compatibility there.
• Updated CI/testing/example docs to describe the marker-based pod path.
• Missing pod runner environment now skips pod tests instead of failing.
• The remote-device fixture documents the current --max-parallel 1 serialization constraint.
• The pod sweep timeout was raised for the full pytest sweep.
• L4 example cleanup now covers setup-failure paths before the local L3 is attached.

This implements the larger option (b) from the issue: pod fixtures plus pytest collection for L4 examples. The smaller option (a) is superseded because the workflow no longer carries per-example
registration or duplicated summary logic.