Add: support mpirun-launched multi-host L3 workers

docs/comm-domain.md

@@ -48,6 +48,71 @@ allocation: if `sum(b.nbytes) > window_size`, `allocate_domain` raises
 Kernels read peer windows through `device_ctx` (which holds every rank's
 window base, local + imported peer); `buffer_ptrs[name]` is the local slice.
 
+### Global CommDomain across local and remote L3 nodes
+
+An L4 worker can build the same `CommContext` shape across any combination of
+forked local L3 workers (`add_worker`) and TCP-connected L3 workers
+(`add_remote_worker`) without `mpirun`:
+
+```python
+with orch.allocate_global_domain(
+    name="tp",
+    members=[(node0_worker_id, 0), (node1_worker_id, 0)],
+    window_size=4096,
+    buffers=[CommBufferSpec("payload", "uint8", 4096, 4096)],
+) as domain:
+    ...
+```
+
+Each member is `(l3_worker_id, local_l2_worker_id)`. The order defines dense
+domain ranks. A remote node reads `comm_profile` and `global_device_ranks`
+from `RemoteWorkerSpec`; a local L3 reads the same fields from its `Worker`
+configuration. All participating nodes must use the same profile.
+
+An MPI-launched group registered with `add_mpirun_worker_group` uses the same
+member contract. Rank 0 writes the group manifest before launch, every MPI rank
+must publish READY before the L4 parent exposes the returned worker ids, and
+the `MpiL3GroupSpec.hosts` order defines node ranks. Global CommDomain members
+must include the complete returned group; their order still defines dense
+domain ranks.
+
+Global CommDomain capability follows the backend that the node actually
+loads: a platform ending in `sim` supports the `sim` profile, and a real
+`a2a3` platform supports `a3-fabric-v1`. Real A5 and any other
+platform/profile combination currently reject allocation before `PREPARE`.
+Each local or remote L3 repeats the same check during `COMM_INIT`, so an
+unsupported backend never advertises a usable descriptor capability.
+
+The control flow is:
+
+1. L4 sends `COMM_INIT` with cluster, node, global-device, and domain-rank
+   identities.
+2. Each L3 asks its participating L2 children to create a local window and
+   export a transport descriptor.
+3. L4 validates and assembles one complete rank-ordered descriptor table.
+4. L4 returns that table to every L3, which forwards it to each L2 for import.
+5. L4 commits only after all imports succeed. Any earlier failure sends
+   `ABORT` and releases every prepared local window.
+
+The descriptor reports the backend's actual mapped size. A3 Fabric may align
+the requested size to its VMM granularity; bounds checks against the mapped
+window use the returned mapping size, while named-buffer offsets and limits use
+each buffer's `nbytes`. The L4 `GlobalCommDomainHandle` exposes only topology
+and buffer metadata. L3-local `ChipDomainContext` objects retain device context
+and pointers for kernel submission. Remote orchestration code calls
+`orch.get_global_domain(domain_id)` to obtain only its committed L3-local
+contexts.
+
+`copy_to_global_domain` and `copy_from_global_domain` provide bounded
+control-plane staging and smoke checks. Normal communication still runs in
+L2 kernels through the imported `CommContext`.
+
+By default a live Global CommDomain is swept after the current `Worker.run`
+drains. Set `retain_after_run=True` when a communication kernel writes results
+into the window and a second L4 run must inspect them. The later run should
+call `domain.release()` after copying the results; `Worker.close()` is the
+final safety net.
+
 ---
 
 ## 2. Lifetime model

docs/remote-l3-worker-design.md

@@ -67,11 +67,18 @@ Implemented:
   imported-handle scheduling eligibility, and deferred owner free.
 - Registry-scope-aware remote callable manifest/control install for dispatcher
   `PYTHON_IMPORT`, inner `PYTHON_IMPORT`, and inner inline `CHIP_CALLABLE`.
+  Pre-init `ChipCallable` registrations on an L4 worker are serialized into
+  each remote session manifest and installed on that L3's L2 children.
+- A no-`mpirun` A3 TCP smoke that keeps a Global CommDomain across two L4
+  runs, executes peer `TLOAD` from each remote L2, and verifies the reduced
+  values before release.
+- Two-server hardware validation covers L4-brokered peer `TLOAD`, one L2
+  compute followed by cross-machine communication, and two-NPU-per-node
+  remote L3 group compute.
 
 Still pending:
 
 - A2 RoCE, A3 HCCS, and A5 UB HCOMM profiles.
-- Remote `CommDomain` allocation/import and hardware-gated validation.
 - Negotiated `PYTHON_SERIALIZED` remote callable payloads and staged
   `CHIP_CALLABLE` blob adapters.
 
@@ -449,10 +456,9 @@ Session execution rules:
   the current one-`WorkerThread`-per-child local scheduling model and keeps
   ordering, buffer lifetime, and callable visibility simple.
 - State-changing CONTROL frames such as register, unregister, buffer free,
-  copy, export/import, and import release serialize with TASK execution on the
-  ordered command lane. They are not applied concurrently with a running TASK
-  on the same endpoint. Future Remote CommDomain controls follow the same
-  ordering rule when they enter scope.
+  copy, export/import, import release, and Global CommDomain transactions
+  serialize with TASK execution on the ordered command lane. They are not
+  applied concurrently with a running TASK on the same endpoint.
 - Bulk data movement may use a separate data plane, but the state change that
   makes staged bytes, callable payloads, or imported handles visible is ordered
   by the command lane.

docs/remote-l3-worker-design/implementation-record.md

@@ -16,7 +16,7 @@ It is updated as each documented feature is completed and verified.
 | 5 | Versioned remote frame codec | In progress | TASK/COMPLETION/CONTROL_REPLY/HELLO/CONTROL/HEALTH exist; core fuzz/bounds coverage is present, with more exhaustive corpus testing still possible. |
 | 6 | Remote callable registry | In progress | Dispatcher `PYTHON_IMPORT`, inner manifest/control `PYTHON_IMPORT`, and inner manifest/control inline `CHIP_CALLABLE` are implemented; serialized payloads and staged chip blobs remain negotiated extensions. |
 | 7 | Fork-safe simulation session runner | In progress | Daemon/session bootstrap and HELLO READY barrier are implemented for sim transport. |
-| 8 | Remote control-plane parity | In progress | Registry, alloc/free/copy, export/import/release-import controls are implemented for sim; Remote CommDomain controls are reserved/unsupported. |
+| 8 | Remote control-plane parity | In progress | Registry, remote buffers, and Global CommDomain prepare/import/commit/release/copy controls are implemented. |
 | 9 | Remote buffer registry | In progress | Sim owner/imported buffers, TASK materialization, public memory API, opaque handles, slot/import-ref capture, and deferred free/release-import are implemented. |
 | 10 | A2 RoCE HCOMM profile | Pending | Hardware-gated profile. |
 | 11 | A3 HCCS HCOMM profile | Pending | Hardware-gated profile. |
@@ -88,13 +88,27 @@ It is updated as each documented feature is completed and verified.
   Imports use shared-memory backed mappings in the session runner, imported
   handles remain opaque on the parent, and owner frees wait for live imports
   and slot refs to drain.
+- Added L4-brokered Global CommDomain setup without MPI. L2 export
+  descriptors are collected by L3, assembled by L4, returned to every L3/L2
+  for import, and released after the L4 DAG drain by default. Domains created
+  with `retain_after_run=True` remain live for a later run until explicitly
+  released or the Worker closes. Sim shm and A3 Fabric V2 use the same
+  descriptor ABI.
+- Added startup-manifest delivery for pre-registered inner `CHIP_CALLABLE`
+  payloads, allowing remote sessions to resolve installed chip callables
+  before task dispatch.
+- Remote buffers use L3-owned child-visible host buffers whenever the L3 has
+  forked chip children, while childless sim sessions keep the shared-memory
+  fallback.
 - Documented the v1 remote registry target/kind matrix, inner
   `INNER_L3_WORKER` visibility rules, remote `CHIP_CALLABLE` staged/inline
   payload contract, partial-register cleanup outcomes, and health-expiry
   scheduling behavior.
 
 ## Verification
 
+- Global CommDomain codec/validation tests and the Linux two-daemon sim
+  transaction test live in `tests/ut/py/test_global_comm_domain.py`.
 - Python focused sidecar/callable tests:
   `tests/ut/py/test_task_interface.py tests/ut/py/test_callable_identity.py`
   passed with `145 passed`.

docs/remote-l3-worker-design/protocol.md

@@ -296,15 +296,30 @@ Required remote controls:
 - `IMPORT_BUFFER`
 - `RELEASE_IMPORT`
 
-Reserved future controls for Remote CommDomain:
+Required Global CommDomain controls:
 
 - `COMM_INIT`
 - `ALLOC_DOMAIN`
 - `RELEASE_DOMAIN`
-
-The first Remote L3 task-dispatch cut rejects the reserved domain controls
-with an unsupported-control reply. They become required only when Remote
-CommDomain enters scope.
+- `COPY_TO_DOMAIN`
+- `COPY_FROM_DOMAIN`
+
+`COMM_INIT` validates the cluster id, node identity, communication profile,
+global device ranks, and dense domain-rank table. `ALLOC_DOMAIN` is a
+transaction with `PREPARE_EXPORT`, `IMPORT`, `COMMIT`, and `ABORT` phases.
+Each L2 exports its local transport descriptor during prepare. L4 assembles
+the complete rank-ordered table and sends it to every L3; each L3 forwards it
+to its L2 children for import. No domain becomes visible to a remote task
+before every node acknowledges `COMMIT`.
+
+`RELEASE_DOMAIN` is idempotent. The handle becomes released when the caller
+requests release; physical backend teardown runs after the owning L4 DAG
+drains. An allocation marked `retain_after_run` may remain live for a later L4
+run that reads kernel results. Explicit release or session shutdown requests
+the same fence-ordered teardown.
+`COPY_TO_DOMAIN` and `COPY_FROM_DOMAIN` are bounded smoke/control data
+operations for a committed local window. They do not replace kernel data
+movement through `CommContext`.
 
 The register-family controls are registry-scope-aware.
 `PREPARE_REGISTER_CALLABLE` carries:

python/bindings/CMakeLists.txt

@@ -61,6 +61,7 @@ target_include_directories(_task_interface PRIVATE
     ${CMAKE_SOURCE_DIR}/src/common/task_interface
     ${CMAKE_SOURCE_DIR}/src/common/worker
     ${CMAKE_SOURCE_DIR}/src/common/hierarchical
+    ${CMAKE_SOURCE_DIR}/src/common/platform/include
     ${CMAKE_SOURCE_DIR}/src/common/platform/include/common
     ${CMAKE_SOURCE_DIR}/src/common/platform/include/host
     ${CMAKE_CURRENT_SOURCE_DIR}

python/bindings/task_interface.cpp

@@ -1585,6 +1585,36 @@ NB_MODULE(_task_interface, m) {
             nb::arg("allocation_id"), nb::arg("rank_count"), nb::arg("domain_rank"),
             "Pair to comm_alloc_domain_windows: collectively release the per-rank pool."
         )
+        .def(
+            "comm_global_domain_prepare",
+            [](ChipWorker &self, uint64_t domain_id, uint32_t domain_rank, uint32_t rank_count, size_t window_size,
+               uint32_t profile) {
+                auto [descriptor, local_window_base, actual_window_size] =
+                    self.comm_global_domain_prepare(domain_id, domain_rank, rank_count, window_size, profile);
+                return nb::make_tuple(
+                    nb::bytes(reinterpret_cast<const char *>(descriptor.data()), descriptor.size()), local_window_base,
+                    actual_window_size
+                );
+            },
+            nb::arg("domain_id"), nb::arg("domain_rank"), nb::arg("rank_count"), nb::arg("window_size"),
+            nb::arg("profile"), "Create a Global CommDomain local window and return its transport descriptor."
+        )
+        .def(
+            "comm_global_domain_import",
+            [](ChipWorker &self, uint64_t domain_id, nb::bytes descriptors) {
+                std::vector<uint8_t> descriptor_bytes(
+                    reinterpret_cast<const uint8_t *>(descriptors.c_str()),
+                    reinterpret_cast<const uint8_t *>(descriptors.c_str()) + descriptors.size()
+                );
+                return self.comm_global_domain_import(domain_id, descriptor_bytes);
+            },
+            nb::arg("domain_id"), nb::arg("descriptors"),
+            "Import a rank-ordered Global CommDomain descriptor table and return the device context."
+        )
+        .def(
+            "comm_global_domain_release", &ChipWorker::comm_global_domain_release, nb::arg("domain_id"),
+            "Release a prepared or imported Global CommDomain."
+        )
         .def("comm_barrier", &ChipWorker::comm_barrier, nb::arg("comm_handle"), "Synchronize all ranks.")
         .def(
             "comm_destroy", &ChipWorker::comm_destroy, nb::arg("comm_handle"),

python/bindings/worker_bind.h

@@ -701,6 +701,28 @@ inline void bind_worker(nb::module_ &m) {
             nb::arg("importer_worker_id"), nb::arg("owner_worker_id"), nb::arg("buffer_id"), nb::arg("generation"),
             nb::arg("import_id"), "Release an imported remote buffer mapping."
         )
+        .def(
+            "remote_domain_control",
+            [](Worker &self, int worker_id, uint32_t control_name, nb::bytes command) {
+                if (!remote_l3::valid_control_name(control_name)) {
+                    throw nb::value_error("control_name is not supported");
+                }
+                std::vector<uint8_t> command_bytes(
+                    reinterpret_cast<const uint8_t *>(command.c_str()),
+                    reinterpret_cast<const uint8_t *>(command.c_str()) + command.size()
+                );
+                std::vector<uint8_t> result;
+                {
+                    nb::gil_scoped_release release;
+                    result = self.control_remote_domain(
+                        worker_id, static_cast<remote_l3::ControlName>(control_name), command_bytes
+                    );
+                }
+                return nb::bytes(reinterpret_cast<const char *>(result.data()), result.size());
+            },
+            nb::arg("worker_id"), nb::arg("control_name"), nb::arg("command"),
+            "Send one Global CommDomain control to a remote L3 endpoint."
+        )
         .def(
             "broadcast_unregister_all",
             [](Worker &self, nb::object digest) {
@@ -742,6 +764,25 @@ inline void bind_worker(nb::module_ &m) {
             "If payload is a Python buffer, C++ stages it in POSIX shm and writes the shm name "
             "into the mailbox. Returns per-child ControlResult entries."
         )
+        .def(
+            "control_payload",
+            [](Worker &self, WorkerType worker_type, int worker_id, uint64_t sub_cmd, nb::object payload,
+               nb::object timeout_s) {
+                std::string payload_bytes = buffer_to_string(payload, "payload");
+                double timeout_val = timeout_s.is_none() ? -1.0 : nb::cast<double>(timeout_s);
+                std::vector<uint8_t> result;
+                {
+                    nb::gil_scoped_release release;
+                    result = self.control_payload(
+                        worker_type, worker_id, sub_cmd, payload_bytes.data(), payload_bytes.size(), timeout_val
+                    );
+                }
+                return nb::bytes(reinterpret_cast<const char *>(result.data()), result.size());
+            },
+            nb::arg("worker_type"), nb::arg("worker_id"), nb::arg("sub_cmd"), nb::arg("payload"),
+            nb::arg("timeout_s") = nb::none(),
+            "Drive one local worker control with a mutable staged payload and return its final bytes."
+        )
         .def(
             "control_alloc_domain", &Worker::control_alloc_domain, nb::arg("worker_id"), nb::arg("request_shm_name"),
             nb::arg("reply_shm_name"), nb::call_guard<nb::gil_scoped_release>(),