Add: an L4 example that spans two machines, and the CI job that runs it - #1688
Merged
Merged
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe PR adds host TCP remote L3 support, a two-machine mixed-L3 vector-add example, and reusable pod CI infrastructure. It updates runtime buffer handling, worker callable registration, transport tests, documentation, composite actions, workflow orchestration, cleanup, and log collection. ChangesHost TCP runtime
L4 mixed vector example
Multi-machine pod CI
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Nitpick comments (7)
examples/workers/l4/vector_add_mixed_l3/run_parent.sh (1)
22-23: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winFail with a clear message when
.venvis absent.If
.venv/bin/activatedoes not exist,set -eaborts with a bare "No such file or directory". The pod CI creates the virtual environment in a separate step, so a staging failure surfaces here first. Add an explicit check and a Shellcheck source directive.♻️ Proposed change
cd "${ROOT_DIR}" -source .venv/bin/activate +if [[ ! -f .venv/bin/activate ]]; then + echo "error: ${ROOT_DIR}/.venv/bin/activate not found; create the virtual environment first" >&2 + exit 1 +fi +# shellcheck source=/dev/null +source .venv/bin/activate🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/workers/l4/vector_add_mixed_l3/run_parent.sh` around lines 22 - 23, Update the setup flow around `source .venv/bin/activate` to add the ShellCheck source directive and explicitly verify that `.venv/bin/activate` exists before sourcing it. If missing, exit with a clear, actionable error message while preserving the existing activation behavior when present.Source: Linters/SAST tools
examples/workers/l4/vector_add_mixed_l3/main.py (1)
335-342: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLog remote_free failures and drop the dead reference assignments.
Two issues in the teardown block:
- Line 338 swallows every exception from
remote_free. The pod CI diagnoses failures from stdout only, so a leaked remote buffer leaves no trace.- Lines 341-342 assign to
output_arrayandoutput_map, which are loop variables from the validation loop. The assignments have no effect on cleanup.♻️ Proposed change
for handle in reversed(remote_buffers): try: worker.remote_free(handle) - except Exception: # noqa: BLE001 - pass + except Exception as exc: # noqa: BLE001 + print(f"[vector-add-mixed-l3] remote_free failed: {exc}") worker.close() - output_array = None - output_map = None local_outputs.clear()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/workers/l4/vector_add_mixed_l3/main.py` around lines 335 - 342, Update the teardown block using reversed(remote_buffers) to log remote_free failures instead of silently swallowing them, including the exception details in the stdout-visible message. Remove the output_array and output_map assignments after worker.close(), since they are unrelated loop-variable references and do not affect cleanup.Source: Linters/SAST tools
examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cpp (1)
43-47: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAll three AIV kernel doc blocks document an
args[3] = sizeentry that no kernel reads. Every kernel derives its element count from the fixed128 x 128tile configuration instead of from an argument. The shared root cause is a copied argument-layout comment. State the fixed tile size in each doc block so a reader does not expect a dynamic size.
examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cpp#L43-L47: remove theargs[3] = sizeline and note that the kernel processes a fixed128 x 128float tile.examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add_scalar.cpp#L45-L49: remove theargs[3] = sizeline, keepargs[2] = scalar, and note that the fourth scalar passed by the orchestration DAG is ignored.examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_mul.cpp#L43-L47: remove theargs[3] = sizeline and note the fixed128 x 128float tile.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cpp` around lines 43 - 47, The kernel documentation incorrectly advertises an unused size argument. In examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cpp#L43-L47 and kernel_mul.cpp#L43-L47, remove args[3] = size and document that the kernel processes a fixed 128 x 128 float tile. In examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add_scalar.cpp#L45-L49, remove the same entry, retain args[2] = scalar, and note that the fourth scalar passed by the orchestration DAG is ignored.docs/ci.md (1)
61-61: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the pod job to the job matrix too.
Line 18 states the matrix at lines 20-23 lists every job that exists in
ci.yml. The st × a2a3 cell still names onlyst-onboard-a2a3, so the matrix and this new row disagree. Addst-pod-onboard-a2a3to that cell, or note there that pod jobs are listed separately in the new "Multi-machine pod jobs" subsection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/ci.md` at line 61, Update the st × a2a3 entry in the CI job matrix to include st-pod-onboard-a2a3, keeping it consistent with the documented jobs in ci.yml and the new multi-machine pod jobs subsection..github/workflows/_st-pod.yml (1)
29-33: 🩺 Stability & Availability | 🔵 TrivialAdd a pod-pair concurrency guard to avoid concurrent L3 daemons.
st-pod-onboard-a2a3uses a reusable workflow with no workflow-levelconcurrency; concurrent PR/ref runs that both target ana2a3podrunner can occupy the same pod pair.docs/ci.mdsays the runner is the parent and drives the peer over ssh, and.envvalues such asPOD_L3_DAEMON_PORTare taken from that config. Use pod-pair-specificgroupandcancel-in-progresswhere two jobs would compete for the same machines, or queue them.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/_st-pod.yml around lines 29 - 33, Add a concurrency guard to the run job in the reusable workflow, using a group key derived from the targeted pod pair (including the platform or equivalent runner-pair identifier) so competing runs share the same group. Configure cancel-in-progress according to the workflow’s desired queueing behavior, ensuring concurrent runs targeting the same a2a3pod machines cannot execute together..github/actions/pod-stage/action.yml (1)
79-96: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winRun the remote script under bash explicitly.
ssh hostname "command"runs the command via the peer’s default non-login shell. Sincepod_sshdoes not pin the remote interpreter, move to a non-bash shell such as dash/ash can makeset -o pipefailfail before staging, andpod_ssh bash -c "..."is required for the/dev/tcpproxy probe.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/actions/pod-stage/action.yml around lines 79 - 96, Update the remote staging command invoked through pod_ssh to execute the script explicitly with bash, such as by passing bash -c around the existing command string. Preserve the current script contents, including set -o pipefail and the /dev/tcp proxy probe, while ensuring they run under bash rather than the remote host’s default shell..github/actions/pod-run-example/action.yml (1)
89-103: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winStop polling once the daemon ssh process is gone.
_serve_looponly exits on a listener-closeOSError; the bare connect probe will only get stale results while a failed daemon ssh imports or crashes after accepting. Check the PID file from the background ssh before each connection attempt and fail immediately with a reference todaemon.ssh.logif it is gone.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/actions/pod-run-example/action.yml around lines 89 - 103, Update the Python polling block in the pod-run example action to check the background daemon SSH PID file before each connection attempt, and exit immediately if the process is no longer present. Include a clear failure message referencing daemon.ssh.log, while preserving the existing connection polling and timeout behavior for a running process.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/actions/pod-stage/action.yml:
- Around line 26-35: Update the proxy parsing and reachability flow around
POD_REMOTE_HTTP_PROXY to also derive and validate POD_REMOTE_HTTPS_PROXY,
ensuring the remote HTTPS proxy used by pip is probed and produces the named
failure. Reuse the existing proxy_reachable logic or extract the shared
parsing/probe implementation into one script under .github, preserving the port
fallback consistently for both proxies.
In @.github/actions/setup-venv/action.yml:
- Around line 60-61: Update the proxy export step in the setup-venv action so
each of http_proxy/HTTP_PROXY, https_proxy/HTTPS_PROXY, and no_proxy/NO_PROXY is
exported only when its corresponding IN_* input is non-empty; otherwise leave
the inherited runner environment unchanged.
In `@docs/capability-survey.md`:
- Line 37: Update the L4 availability documentation around the stale statement
near the referenced lines to reflect the PR’s shipped remote host_tcp support:
replace the claim that no L4 example or daemon-starting CI job exists with the
mixed-local/remote L3 vector-add example and pod CI coverage, while preserving
the accurate L3 and local L4 information.
In `@docs/remote-l3-worker-design.md`:
- Around line 590-599: Clarify the remaining “sim” terminology after transport
migration: in docs/remote-l3-worker-design.md lines 590-599, qualify “simulation
session runner” as the a2a3sim platform or host_tcp transport; in
docs/remote-l3-worker-design/implementation-record.md lines 12-20, update the
earlier sim health-lane label to identify the verified transport.
In
`@examples/workers/l4/vector_add_mixed_l3/kernels/orchestration/vector_add_mixed_l3_orchestration.cpp`:
- Around line 60-64: Update the orchestration flow around SIZE and
TensorCreateInfo to validate that the runtime tensor size equals 128u * 128u
before allocating intermediates or launching kernels. Log an error and return
immediately when SIZE differs from the fixed kernel tile, while preserving the
existing path for the valid size.
In `@examples/workers/l4/vector_add_mixed_l3/main.py`:
- Around line 288-298: Update the zip call in the loop over remote_handles and
the hardcoded array tuple to pass strict=True, ensuring mismatched handle and
array counts raise an error instead of silently truncating inputs.
- Around line 245-257: Add the requested pytest module beside main.py, covering
the worker example while deselecting tests at runtime when the two-machine or
platform requirements are unavailable; do not rely on collection filters.
Preserve the README’s existing direct-launch instructions and keep the test
focused on the symbols exposed by main.py.
In `@examples/workers/l4/vector_add_mixed_l3/README.md`:
- Around line 41-46: Update the two-machine command examples in the README to
make 192.0.2.20 explicitly a replaceable peer-address placeholder, preferably by
defining and reusing a PEER_HOST variable in both commands; preserve the
existing worker startup and environment setup.
In `@examples/workers/README.md`:
- Around line 45-59: Align the host-role conventions in
examples/workers/README.md lines 45-59 and
examples/workers/l4/vector_add_mixed_l3/README.md lines 3-9 by updating both
topology diagrams and command descriptions consistently, or replace machine A/B
labels with shared role-based labels; ensure the parent and remote daemon roles
identify the same hosts in both documents.
In `@python/simpler/remote_l3_session.py`:
- Around line 667-675: Replace the local shared-memory export contract across
python/simpler/remote_l3_session.py:667-675, 773-790, and 810-811. In the
device-backed creation path around create_host_buffer and _RemoteBufferEntry,
retain owner-side state that can serve exported bytes through the host TCP
transport; update the export logic to stop emitting entry.shm_name, and make
import use the owner-mediated network transfer or another network-visible
backing store. Add an export/import test covering separate hosts with a
device-backed remote L3.
---
Nitpick comments:
In @.github/actions/pod-run-example/action.yml:
- Around line 89-103: Update the Python polling block in the pod-run example
action to check the background daemon SSH PID file before each connection
attempt, and exit immediately if the process is no longer present. Include a
clear failure message referencing daemon.ssh.log, while preserving the existing
connection polling and timeout behavior for a running process.
In @.github/actions/pod-stage/action.yml:
- Around line 79-96: Update the remote staging command invoked through pod_ssh
to execute the script explicitly with bash, such as by passing bash -c around
the existing command string. Preserve the current script contents, including set
-o pipefail and the /dev/tcp proxy probe, while ensuring they run under bash
rather than the remote host’s default shell.
In @.github/workflows/_st-pod.yml:
- Around line 29-33: Add a concurrency guard to the run job in the reusable
workflow, using a group key derived from the targeted pod pair (including the
platform or equivalent runner-pair identifier) so competing runs share the same
group. Configure cancel-in-progress according to the workflow’s desired queueing
behavior, ensuring concurrent runs targeting the same a2a3pod machines cannot
execute together.
In `@docs/ci.md`:
- Line 61: Update the st × a2a3 entry in the CI job matrix to include
st-pod-onboard-a2a3, keeping it consistent with the documented jobs in ci.yml
and the new multi-machine pod jobs subsection.
In `@examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cpp`:
- Around line 43-47: The kernel documentation incorrectly advertises an unused
size argument. In
examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cpp#L43-L47 and
kernel_mul.cpp#L43-L47, remove args[3] = size and document that the kernel
processes a fixed 128 x 128 float tile. In
examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add_scalar.cpp#L45-L49,
remove the same entry, retain args[2] = scalar, and note that the fourth scalar
passed by the orchestration DAG is ignored.
In `@examples/workers/l4/vector_add_mixed_l3/main.py`:
- Around line 335-342: Update the teardown block using reversed(remote_buffers)
to log remote_free failures instead of silently swallowing them, including the
exception details in the stdout-visible message. Remove the output_array and
output_map assignments after worker.close(), since they are unrelated
loop-variable references and do not affect cleanup.
In `@examples/workers/l4/vector_add_mixed_l3/run_parent.sh`:
- Around line 22-23: Update the setup flow around `source .venv/bin/activate` to
add the ShellCheck source directive and explicitly verify that
`.venv/bin/activate` exists before sourcing it. If missing, exit with a clear,
actionable error message while preserving the existing activation behavior when
present.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c07ae536-d033-405a-bae6-9d1f03c635a5
📒 Files selected for processing (30)
.github/actions/pod-run-example/action.yml.github/actions/pod-stage/action.yml.github/actions/pod-teardown/action.yml.github/actions/setup-venv/action.yml.github/workflows/_st-pod.yml.github/workflows/ci.ymldocs/capability-survey.mddocs/ci.mddocs/remote-l3-worker-design.mddocs/remote-l3-worker-design/implementation-plan.mddocs/remote-l3-worker-design/implementation-record.mddocs/remote-l3-worker-design/pr-split-and-audit-artifacts.mddocs/remote-l3-worker-design/pr-split-and-audit-plan.mddocs/user/how-to/run-on-multiple-chips.mdexamples/workers/README.mdexamples/workers/l4/vector_add_mixed_l3/README.mdexamples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cppexamples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add_scalar.cppexamples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_mul.cppexamples/workers/l4/vector_add_mixed_l3/kernels/orchestration/vector_add_mixed_l3_orchestration.cppexamples/workers/l4/vector_add_mixed_l3/main.pyexamples/workers/l4/vector_add_mixed_l3/run_parent.shpython/simpler/remote_l3_protocol.pypython/simpler/remote_l3_session.pypython/simpler/remote_l3_worker.pypython/simpler/task_interface.pypython/simpler/worker.pytests/ut/py/test_callable_identity.pytests/ut/py/test_remote_l3_lifecycle.pytests/ut/py/test_worker/test_remote_startup_budget.py
luohuan19
force-pushed
the
ci-multiserver-validation
branch
3 times, most recently
from
August 5, 2026 02:45
8be59e6 to
64ffc9c
Compare
Contributor
Author
|
@coderabbitai Addressed the review summary's 7 nitpicks, plus a CI failure the review predates. Nitpicks — all applied:
Also in this push: Each inline thread has its own reply. Two are skipped with reasons on the thread (#6 pytest sibling, #9 host-role labels), and #10 is confirmed real but deferred — see that thread. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '=== Changed paths relevant to the follow-up ==='
git diff --name-only HEAD^ HEAD -- \
.github/actions/pod-run-example/action.yml \
.github/actions/pod-stage/action.yml \
.github/workflows/_st-pod.yml \
docs/ci.md \
examples/workers/l4/vector_add_mixed_l3/run_parent.sh \
examples/workers/l4/vector_add_mixed_l3/main.py \
examples/workers/l4/vector_add_mixed_l3/kernels/aiv \
examples/workers/l4/vector_add_mixed_l3/kernels/orchestration/vector_add_mixed_l3_orchestration.cpp
printf '%s\n' '=== Relevant diff ==='
git diff HEAD^ HEAD -- \
.github/actions/pod-run-example/action.yml \
.github/actions/pod-stage/action.yml \
.github/workflows/_st-pod.yml \
docs/ci.md \
examples/workers/l4/vector_add_mixed_l3/run_parent.sh \
examples/workers/l4/vector_add_mixed_l3/main.py \
examples/workers/l4/vector_add_mixed_l3/kernels/aiv \
examples/workers/l4/vector_add_mixed_l3/kernels/orchestration/vector_add_mixed_l3_orchestration.cpp
printf '%s\n' '=== Current targeted lines ==='
for file in \
.github/actions/pod-run-example/action.yml \
.github/actions/pod-stage/action.yml \
.github/workflows/_st-pod.yml \
docs/ci.md \
examples/workers/l4/vector_add_mixed_l3/run_parent.sh \
examples/workers/l4/vector_add_mixed_l3/main.py \
examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add.cpp \
examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_add_scalar.cpp \
examples/workers/l4/vector_add_mixed_l3/kernels/aiv/kernel_mul.cpp \
examples/workers/l4/vector_add_mixed_l3/kernels/orchestration/vector_add_mixed_l3_orchestration.cpp
do
if [ -f "$file" ]; then
printf '\n--- %s ---\n' "$file"
rg -n -C 3 \
'activate|shellcheck|remote_free|output_array|output_map|128|args\[3\]|ChipTensor|Tensor\b|concurrency|cancel-in-progress|st-pod-onboard-a2a3|bash -s|daemon\.ssh\.log|POD_DAEMON_WAIT_S|kill -0|State:' \
"$file" || true
fi
doneLength of output: 84154
I also confirmed the I will keep review item 🐇 ✅ If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! You are interacting with an AI system. |
luohuan19
force-pushed
the
ci-multiserver-validation
branch
2 times, most recently
from
August 5, 2026 03:09
a9b9bb9 to
960c0a1
Compare
A `Worker(level=4)` can hold a forked local L3 and a TCP-attached remote one at the same time, but nothing exercised that: every example and every job here fits on one host. `examples/workers/l4/vector_add_mixed_l3` dispatches to both in a single run and checks the golden on both sides, staging local inputs through fork-inherited shared memory and remote ones through remote_malloc / remote_copy_to. Running it needs a pair of machines, so `_st-pod.yml` drives one: the runner it lands on becomes the parent and reaches its peer entirely over ssh, and the peer runs no workflow code at all. ci.yml gains only a caller, gated exactly as st-onboard-a2a3 is — this is arch-specific and by far the most expensive thing in the file, and a job holding two machines must never sit behind a weaker signal than a cheaper one. Nothing identifying a machine is in the repo. Addresses, the device split, ports, the staging root and the proxies come from a .env the runner carries, which the first step parses, defaults, validates and publishes; only POD_* keys are accepted, since the runner service reads that same file. The body then splits by what is per-run and what is per-example: staging the tree on the peer and building it there is the job's whole cost and every example shares the result, so pod-stage runs once and only pod-run-example repeats. Examples run with continue-on-error and a summary step decides the result, because one round holds two machines and stopping at the first failure wastes the rest. That makes the summary the only step whose result means anything — continue-on-error rewrites a failed step's conclusion to success and leaves the truth in outcome. Both sides' device logs upload as one artifact: a device-side failure names its reason only there, and without it the run page shows a host traceback saying the peer's scheduler gave up and nothing about why. Three guards exist because a runner claim covers only the machine the job landed on. The job takes a pod-pair concurrency group with no ref in it, so two PRs queue for the peer instead of sharing its devices and its daemon port, and it does not cancel in progress — a cancelled job is SIGKILLed before teardown can clear the peer. Both of the peer's proxies are probed from the peer, not just the http one, since pip reaches its index over https and an unreachable proxy otherwise surfaces as five retries ending in an opaque ProxyError. And the readiness wait watches the daemon's ssh process as well as the port, so a daemon that dies during import is reported immediately, naming the log that holds why, rather than after the full timeout. Three supporting changes. Eligibility now walks the frozen topology for a chip target instead of testing the worker's own device_ids and a non-empty child list: an L4 parent has neither, and reaches its chips through a next-level child or a remote spec, so it was eligible only by the accident of having any child at all and a purely remote one was rejected outright. And simpler-remote-worker accepts only the host_tcp transport now, so the remote L3 unit tests move off the profile it no longer takes — they were failing on that check before reaching what they meant to assert. And setup-venv gains proxy inputs for the pod job's sake, but exports each one only when it is set: every other caller leaves them empty, and on a self-hosted runner an empty export would erase the machine's own working proxy setting. Co-authored-by: ccyywwen <75376396+ccyywwen@users.noreply.github.com>
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Aug 6, 2026
`simpler-remote-worker` rejects anything but `host_tcp` (hw-native-sys#1011), and hw-native-sys#1688 narrowed the unit tests it knew about onto that profile. The zero-residual acceptance tests landed from hw-native-sys#1692 in the same window still asked for `sim`, so the merge of two independently green PRs left every case in that file failing its own setup with `only host_tcp transport is accepted by simpler-remote-worker` — the rollback each one exists to observe never ran. Unrelated to the Buffer ABI; it rides here because it is one line and it is what keeps `ut` red on every PR that touches nothing near it.
7 tasks
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
Worker(level=4)can hold a forked local L3 and a TCP-attached remote one atthe same time, but nothing exercised that — every example and every job here
fits on one host.
examples/workers/l4/vector_add_mixed_l3dispatches to both L3s in onerun and checks the golden on both sides. Local inputs are staged through
fork-inherited shared memory, remote ones through
remote_malloc/remote_copy_to._st-pod.ymlruns it across a pair of machines. The runner it lands onbecomes the parent and reaches its peer entirely over ssh; the peer runs no
workflow code.
ci.ymlgains only a caller.pod-*composite actions split the body by what is per-run(staging the tree on the peer and building it there — the job's whole cost,
shared by every example) and what is per-example (daemon + parent).
Points worth a reviewer's attention
One behaviour change, in
_eligible_target_need. Chip-target eligibilitynow walks the frozen topology instead of testing this worker's own
device_idsand a non-empty child list. An L4 parent has neither and reaches its chips
through a next-level child or a remote spec, so under the old check it passed
only by the accident of having any child, and a purely remote L4 was rejected
outright. The rejection message is deliberately unchanged —
RemoteWorkerSpecnames its devices
device_idstoo, so the old wording stays accurate and theassertions pinning it keep passing.
26 unit tests were already failing on this branch and are fixed here, not
broken by it.
simpler-remote-workeraccepts only thehost_tcptransport,and the remote L3 tests still built manifests with the profile it no longer
takes, so they failed on that check before reaching what they meant to assert.
The pod job's gate matches
st-onboard-a2a3exactly — arch-specific, andby far the most expensive thing in
ci.yml. Per.claude/rules/ci-change-detection.mda job holding two machines must neversit behind a weaker signal than a cheaper one.
setup-venvgainscann-envand three proxy inputs, all defaulting totoday's behaviour, because the pod runners keep both in their
.env.Runner prerequisites
The pod machines need a
.envat the runner root naming the pair (addresses,device split, ports, staging root, proxies); nothing identifying a machine is
in the repo. Only a machine hosting a runner needs one — the peer is driven
over ssh. The pool label is
a2a3pod; a job whoseruns_onnames a label norunner carries queues until timeout rather than failing, so that is the first
thing to check if this job never starts.
Testing
pytest tests/ut -m "not requires_hardware"— 1072 passed, 13 skipped