Add: dispatch MPI worker groups through mailbox

docs/comm-domain.md

@@ -48,6 +48,71 @@ allocation: if `sum(b.nbytes) > window_size`, `allocate_domain` raises
 Kernels read peer windows through `device_ctx` (which holds every rank's
 window base, local + imported peer); `buffer_ptrs[name]` is the local slice.
 
+### Global CommDomain across local and remote L3 nodes
+
+An L4 worker can build the same `CommContext` shape across any combination of
+forked local L3 workers (`add_worker`) and TCP-connected L3 workers
+(`add_remote_worker`) without `mpirun`:
+
+```python
+with orch.allocate_global_domain(
+    name="tp",
+    members=[(node0_worker_id, 0), (node1_worker_id, 0)],
+    window_size=4096,
+    buffers=[CommBufferSpec("payload", "uint8", 4096, 4096)],
+) as domain:
+    ...
+```
+
+Each member is `(l3_worker_id, local_l2_worker_id)`. The order defines dense
+domain ranks. A remote node reads `comm_profile` and `global_device_ranks`
+from `RemoteWorkerSpec`; a local L3 reads the same fields from its `Worker`
+configuration. All participating nodes must use the same profile.
+
+An MPI-launched group registered with `add_mpirun_worker_group` uses the same
+member contract. Rank 0 writes the group manifest before launch, every MPI rank
+must publish READY before the L4 parent exposes the returned worker ids, and
+the `MpiL3GroupSpec.hosts` order defines node ranks. Global CommDomain members
+must include the complete returned group; their order still defines dense
+domain ranks.
+
+Global CommDomain capability follows the backend that the node actually
+loads: a platform ending in `sim` supports the `sim` profile, and a real
+`a2a3` platform supports `a3-fabric-v1`. Real A5 and any other
+platform/profile combination currently reject allocation before `PREPARE`.
+Each local or remote L3 repeats the same check during `COMM_INIT`, so an
+unsupported backend never advertises a usable descriptor capability.
+
+The control flow is:
+
+1. L4 sends `COMM_INIT` with cluster, node, global-device, and domain-rank
+   identities.
+2. Each L3 asks its participating L2 children to create a local window and
+   export a transport descriptor.
+3. L4 validates and assembles one complete rank-ordered descriptor table.
+4. L4 returns that table to every L3, which forwards it to each L2 for import.
+5. L4 commits only after all imports succeed. Any earlier failure sends
+   `ABORT` and releases every prepared local window.
+
+The descriptor reports the backend's actual mapped size. A3 Fabric may align
+the requested size to its VMM granularity; bounds checks against the mapped
+window use the returned mapping size, while named-buffer offsets and limits use
+each buffer's `nbytes`. The L4 `GlobalCommDomainHandle` exposes only topology
+and buffer metadata. L3-local `ChipDomainContext` objects retain device context
+and pointers for kernel submission. Remote orchestration code calls
+`orch.get_global_domain(domain_id)` to obtain only its committed L3-local
+contexts.
+
+`copy_to_global_domain` and `copy_from_global_domain` provide bounded
+control-plane staging and smoke checks. Normal communication still runs in
+L2 kernels through the imported `CommContext`.
+
+By default a live Global CommDomain is swept after the current `Worker.run`
+drains. Set `retain_after_run=True` when a communication kernel writes results
+into the window and a second L4 run must inspect them. The later run should
+call `domain.release()` after copying the results; `Worker.close()` is the
+final safety net.
+
 ---
 
 ## 2. Lifetime model

docs/mpi-l3-mailbox.md

@@ -0,0 +1,132 @@
+# MPI L3 group mailbox protocol
+
+An MPI L3 group has one L4-owned, named shared-memory mailbox. Only local MPI
+rank 0 opens that mailbox. All ranks participate in the same ordered
+`dispatch_comm` collectives, while Global CommDomain descriptor exchange uses
+a separate `domain_comm`.
+
+```text
+L4 / MpiGroupMailboxEndpoint
+          |
+          | named SharedMemory (one request lane)
+          v
+local MPI rank 0 / L3
+          |
+          | dispatch_comm Bcast + Gather
+          v
+all MPI ranks / L3 -> each rank's local L2 workers
+```
+
+MPI groups never start or connect Simpler command/health TCP sockets. Ordinary
+`RemoteWorkerSpec` workers still use `RemoteL3SocketTransport` and keep their
+existing command and health lanes.
+
+## Startup and shutdown
+
+1. L4 creates the mailbox and writes its name, protocol version, size, and
+   world size into the group manifest.
+2. L4 starts `mpirun` as a new process group and monitors that direct child.
+3. Each rank creates and initializes its own L3 `Worker`.
+4. All ranks complete a readiness `allgather`.
+5. Rank 0 reopens the mailbox by name and publishes `READY`. Other ranks never
+   map it.
+6. L4 attaches every stable MPI worker id to the same
+   `MpiGroupMailboxChannel`; it does not create `RemoteL3Endpoint` sockets.
+7. Shutdown is a mailbox `SHUTDOWN` request, followed by one MPI broadcast.
+   Each rank closes its inner worker, communicators are freed, `mpirun` exits,
+   and L4 unlinks the mailbox and manifest directory.
+
+If startup, a collective, the mailbox, or `mpirun` fails, the group becomes
+terminal. Runtime timeout also kills the complete `mpirun` process group.
+There is no TCP fallback.
+
+## Envelope and state
+
+Protocol version 1 has a fixed 256-byte header and two 16 MiB payload regions.
+The header contains:
+
+- magic `SMPIBOX\0`
+- protocol version and layout size
+- MPI world size
+- group state: `INITIALIZING`, `READY`, `TERMINAL`, or `CLOSED`
+- request state: `IDLE`, `REQUEST_READY`, `TASK_ACCEPTED`, `TASK_DONE`,
+  `TASK_FAILED`, `SHUTDOWN_READY`, or `SHUTDOWN_DONE`
+- monotonic mailbox `sequence_id`
+- opcode: `TASK`, `CONTROL`, `PING`, or `SHUTDOWN`
+- target: `GROUP`, `RANK`, or `PER_RANK`
+- target rank, payload count, and byte lengths
+
+Rank 0 copies the complete request to private memory before publishing
+`TASK_ACCEPTED`. It publishes `TASK_DONE` only after gathering every rank's
+status. Duplicate or decreasing sequence ids make the group terminal.
+
+Every gathered error contains `rank`, `error_type`, and `message`. Any target
+rank failure fails the group operation. A broken command processor or
+collective is terminal; an ordinary task/control application error is returned
+to L4 and the communicator may be reused.
+
+## Target and API semantics
+
+- `orch.submit_next_level(..., worker=id)` remains a directed rank operation.
+  Every MPI rank receives the envelope in collective order, but only the
+  selected rank executes it.
+- `orch.submit_next_level_group(args_list, workers=...)` remains one DAG node.
+  When `workers` is the complete MPI group, C++ batches all members into one
+  `PER_RANK` mailbox request. Rank `workers[i]` uses `args_list[i]`.
+- A subset group remains supported as ordered directed requests. It is not
+  silently widened to the complete MPI group.
+- Group-wide controls use one `GROUP` mailbox request.
+
+The existing remote task codec is reused. It serializes scalar values, tensor
+metadata, inline host payloads, and `RemoteTensorRef` descriptors. Bare host or
+child virtual addresses without a valid remote sidecar are rejected before
+execution; a pointer value is never forwarded as if it were meaningful on
+another rank. `PYTHON_SERIALIZED` callable payloads remain unsupported by the
+underlying Remote L3 protocol; `PYTHON_IMPORT` and inline `CHIP_CALLABLE`
+registration are supported.
+
+## Remote protocol audit and MPI mapping
+
+The wire `FrameType` values remain unchanged:
+
+| Existing frame | MPI mailbox mapping |
+| -------------- | ------------------- |
+| `HELLO` / ready | rank-local initialization, readiness `allgather`, then rank 0 publishes mailbox `READY` |
+| `TASK` | `TASK`; directed `RANK`, or one full-group `PER_RANK` vector |
+| `CONTROL` / `CONTROL_REPLY` | `CONTROL`; directed except the group-wide controls below |
+| `COMPLETION` | gathered per-rank status; selected/per-rank replies returned to L4 |
+| `HEALTH` | `PING` to `GROUP`, gathered before success |
+| `SHUTDOWN` | `SHUTDOWN` to `GROUP`, gathered before `SHUTDOWN_DONE` |
+
+All existing remote controls use the mailbox path:
+
+| Number | Control | MPI target |
+| -----: | ------- | ---------- |
+| 1 | `UNREGISTER_CALLABLE` | directed rank |
+| 2 | `PREPARE_REGISTER_CALLABLE` | directed rank |
+| 3 | `COMMIT_REGISTER_CALLABLE` | directed rank |
+| 4 | `ABORT_REGISTER_CALLABLE` | directed rank |
+| 5 | `PREPARE_CALLABLE` | directed rank |
+| 6 | `ALLOC_REMOTE_BUFFER` | directed rank |
+| 7 | `FREE_REMOTE_BUFFER` | directed rank |
+| 8 | `COPY_TO_REMOTE` | directed rank |
+| 9 | `COPY_FROM_REMOTE` | directed rank |
+| 10 | `EXPORT_BUFFER` | directed rank |
+| 11 | `IMPORT_BUFFER` | directed rank |
+| 12 | `RELEASE_IMPORT` | directed rank |
+| 13 | `COMM_INIT` | directed rank |
+| 14 | `ALLOC_DOMAIN` prepare/import/commit/abort | one group request; descriptor work uses `domain_comm` |
+| 15 | `RELEASE_DOMAIN` | one group request |
+| 16 | `COPY_TO_DOMAIN` | directed rank |
+| 17 | `COPY_FROM_DOMAIN` | directed rank |
+
+Remote control number 18 is intentionally not assigned. The local hierarchical
+protocol keeps number 18 for committed-device-memory control.
+
+## Threading
+
+Only the main dispatcher thread calls MPI. The existing command processor runs
+on a rank-local thread over an in-memory, socket-shaped queue. Global
+CommDomain operations cross back to the dispatcher through a queue and
+`threading.Event`, so they use `domain_comm` on the MPI-owning thread. This
+design does not require `MPI_THREAD_MULTIPLE`.

docs/remote-l3-worker-design.md

@@ -9,6 +9,7 @@ Detailed protocol, buffer, transport, and rollout notes live in:
 
 - [protocol.md](remote-l3-worker-design/protocol.md)
 - [buffers-and-transports.md](remote-l3-worker-design/buffers-and-transports.md)
+- [MPI L3 group mailbox](mpi-l3-mailbox.md)
 - [implementation-plan.md](remote-l3-worker-design/implementation-plan.md)
 - [pr-split-and-audit-plan.md][split-audit-plan]
 
@@ -63,15 +64,26 @@ Implemented:
 - Socket-backed simulation remote sessions via `simpler-remote-worker` and
   `simpler-remote-l3-session`, including `HELLO READY`, TASK/COMPLETION,
   CONTROL/CONTROL_REPLY, SHUTDOWN, and an independent health lane.
+- MPI L3 groups use one rank-0 named shared-memory mailbox plus ordered MPI
+  collectives for task, control, health, Global CommDomain, error, and shutdown
+  handling. They do not create Simpler command or health TCP sockets; ordinary
+  non-MPI Remote L3 sessions retain the socket transport.
 - Simulation remote buffer allocation, copy, export, import, release-import,
   imported-handle scheduling eligibility, and deferred owner free.
 - Registry-scope-aware remote callable manifest/control install for dispatcher
   `PYTHON_IMPORT`, inner `PYTHON_IMPORT`, and inner inline `CHIP_CALLABLE`.
+  Pre-init `ChipCallable` registrations on an L4 worker are serialized into
+  each remote session manifest and installed on that L3's L2 children.
+- A no-`mpirun` A3 TCP smoke that keeps a Global CommDomain across two L4
+  runs, executes peer `TLOAD` from each remote L2, and verifies the reduced
+  values before release.
+- Two-server hardware validation covers L4-brokered peer `TLOAD`, one L2
+  compute followed by cross-machine communication, and two-NPU-per-node
+  remote L3 group compute.
 
 Still pending:
 
 - A2 RoCE, A3 HCCS, and A5 UB HCOMM profiles.
-- Remote `CommDomain` allocation/import and hardware-gated validation.
 - Negotiated `PYTHON_SERIALIZED` remote callable payloads and staged
   `CHIP_CALLABLE` blob adapters.
 
@@ -449,10 +461,9 @@ Session execution rules:
   the current one-`WorkerThread`-per-child local scheduling model and keeps
   ordering, buffer lifetime, and callable visibility simple.
 - State-changing CONTROL frames such as register, unregister, buffer free,
-  copy, export/import, and import release serialize with TASK execution on the
-  ordered command lane. They are not applied concurrently with a running TASK
-  on the same endpoint. Future Remote CommDomain controls follow the same
-  ordering rule when they enter scope.
+  copy, export/import, import release, and Global CommDomain transactions
+  serialize with TASK execution on the ordered command lane. They are not
+  applied concurrently with a running TASK on the same endpoint.
 - Bulk data movement may use a separate data plane, but the state change that
   makes staged bytes, callable payloads, or imported handles visible is ordered
   by the command lane.

docs/remote-l3-worker-design/implementation-record.md

@@ -16,7 +16,7 @@ It is updated as each documented feature is completed and verified.
 | 5 | Versioned remote frame codec | In progress | TASK/COMPLETION/CONTROL_REPLY/HELLO/CONTROL/HEALTH exist; core fuzz/bounds coverage is present, with more exhaustive corpus testing still possible. |
 | 6 | Remote callable registry | In progress | Dispatcher `PYTHON_IMPORT`, inner manifest/control `PYTHON_IMPORT`, and inner manifest/control inline `CHIP_CALLABLE` are implemented; serialized payloads and staged chip blobs remain negotiated extensions. |
 | 7 | Fork-safe simulation session runner | In progress | Daemon/session bootstrap and HELLO READY barrier are implemented for sim transport. |
-| 8 | Remote control-plane parity | In progress | Registry, alloc/free/copy, export/import/release-import controls are implemented for sim; Remote CommDomain controls are reserved/unsupported. |
+| 8 | Remote control-plane parity | In progress | Registry, remote buffers, and Global CommDomain prepare/import/commit/release/copy controls are implemented. |
 | 9 | Remote buffer registry | In progress | Sim owner/imported buffers, TASK materialization, public memory API, opaque handles, slot/import-ref capture, and deferred free/release-import are implemented. |
 | 10 | A2 RoCE HCOMM profile | Pending | Hardware-gated profile. |
 | 11 | A3 HCCS HCOMM profile | Pending | Hardware-gated profile. |
@@ -88,13 +88,27 @@ It is updated as each documented feature is completed and verified.
   Imports use shared-memory backed mappings in the session runner, imported
   handles remain opaque on the parent, and owner frees wait for live imports
   and slot refs to drain.
+- Added L4-brokered Global CommDomain setup without MPI. L2 export
+  descriptors are collected by L3, assembled by L4, returned to every L3/L2
+  for import, and released after the L4 DAG drain by default. Domains created
+  with `retain_after_run=True` remain live for a later run until explicitly
+  released or the Worker closes. The sim shm and `a3-fabric-v1` profiles use
+  the same descriptor ABI.
+- Added startup-manifest delivery for pre-registered inner `CHIP_CALLABLE`
+  payloads, allowing remote sessions to resolve installed chip callables
+  before task dispatch.
+- Remote buffers use L3-owned child-visible host buffers whenever the L3 has
+  forked chip children, while childless sim sessions keep the shared-memory
+  fallback.
 - Documented the v1 remote registry target/kind matrix, inner
   `INNER_L3_WORKER` visibility rules, remote `CHIP_CALLABLE` staged/inline
   payload contract, partial-register cleanup outcomes, and health-expiry
   scheduling behavior.
 
 ## Verification
 
+- Global CommDomain codec/validation tests and the Linux two-daemon sim
+  transaction test live in `tests/ut/py/test_global_comm_domain.py`.
 - Python focused sidecar/callable tests:
   `tests/ut/py/test_task_interface.py tests/ut/py/test_callable_identity.py`
   passed with `145 passed`.

docs/remote-l3-worker-design/protocol.md

@@ -296,15 +296,30 @@ Required remote controls:
 - `IMPORT_BUFFER`
 - `RELEASE_IMPORT`
 
-Reserved future controls for Remote CommDomain:
+Required Global CommDomain controls:
 
 - `COMM_INIT`
 - `ALLOC_DOMAIN`
 - `RELEASE_DOMAIN`
-
-The first Remote L3 task-dispatch cut rejects the reserved domain controls
-with an unsupported-control reply. They become required only when Remote
-CommDomain enters scope.
+- `COPY_TO_DOMAIN`
+- `COPY_FROM_DOMAIN`
+
+`COMM_INIT` validates the cluster id, node identity, communication profile,
+global device ranks, and dense domain-rank table. `ALLOC_DOMAIN` is a
+transaction with `PREPARE_EXPORT`, `IMPORT`, `COMMIT`, and `ABORT` phases.
+Each L2 exports its local transport descriptor during prepare. L4 assembles
+the complete rank-ordered table and sends it to every L3; each L3 forwards it
+to its L2 children for import. No domain becomes visible to a remote task
+before every node acknowledges `COMMIT`.
+
+`RELEASE_DOMAIN` is idempotent. The handle becomes released when the caller
+requests release; physical backend teardown runs after the owning L4 DAG
+drains. An allocation marked `retain_after_run` may remain live for a later L4
+run that reads kernel results. Explicit release or session shutdown requests
+the same fence-ordered teardown.
+`COPY_TO_DOMAIN` and `COPY_FROM_DOMAIN` are bounded smoke/control data
+operations for a committed local window. They do not replace kernel data
+movement through `CommContext`.
 
 The register-family controls are registry-scope-aware.
 `PREPARE_REGISTER_CALLABLE` carries:

mkdocs.yml

@@ -129,6 +129,7 @@ nav:
       - L3-L2 Message Queue: l3-l2-message-queue.md
       - Directed NEXT_LEVEL Scheduling: directed-next-level-scheduling.md
       - Remote L3 Worker Design: remote-l3-worker-design.md
+      - MPI L3 group mailbox: mpi-l3-mailbox.md
   - Profiling and DFX:
       - Overview: dfx/README.md
       - Profiling Framework: dfx/profiling-framework.md

python/bindings/CMakeLists.txt

@@ -61,6 +61,7 @@ target_include_directories(_task_interface PRIVATE
     ${CMAKE_SOURCE_DIR}/src/common/task_interface
     ${CMAKE_SOURCE_DIR}/src/common/worker
     ${CMAKE_SOURCE_DIR}/src/common/hierarchical
+    ${CMAKE_SOURCE_DIR}/src/common/platform/include
     ${CMAKE_SOURCE_DIR}/src/common/platform/include/common
     ${CMAKE_SOURCE_DIR}/src/common/platform/include/host
     ${CMAKE_CURRENT_SOURCE_DIR}

python/bindings/task_interface.cpp

@@ -1585,6 +1585,36 @@ NB_MODULE(_task_interface, m) {
             nb::arg("allocation_id"), nb::arg("rank_count"), nb::arg("domain_rank"),
             "Pair to comm_alloc_domain_windows: collectively release the per-rank pool."
         )
+        .def(
+            "comm_global_domain_prepare",
+            [](ChipWorker &self, uint64_t domain_id, uint32_t domain_rank, uint32_t rank_count, size_t window_size,
+               uint32_t profile) {
+                auto [descriptor, local_window_base, actual_window_size] =
+                    self.comm_global_domain_prepare(domain_id, domain_rank, rank_count, window_size, profile);
+                return nb::make_tuple(
+                    nb::bytes(reinterpret_cast<const char *>(descriptor.data()), descriptor.size()), local_window_base,
+                    actual_window_size
+                );
+            },
+            nb::arg("domain_id"), nb::arg("domain_rank"), nb::arg("rank_count"), nb::arg("window_size"),
+            nb::arg("profile"), "Create a Global CommDomain local window and return its transport descriptor."
+        )
+        .def(
+            "comm_global_domain_import",
+            [](ChipWorker &self, uint64_t domain_id, nb::bytes descriptors) {
+                std::vector<uint8_t> descriptor_bytes(
+                    reinterpret_cast<const uint8_t *>(descriptors.c_str()),
+                    reinterpret_cast<const uint8_t *>(descriptors.c_str()) + descriptors.size()
+                );
+                return self.comm_global_domain_import(domain_id, descriptor_bytes);
+            },
+            nb::arg("domain_id"), nb::arg("descriptors"),
+            "Import a rank-ordered Global CommDomain descriptor table and return the device context."
+        )
+        .def(
+            "comm_global_domain_release", &ChipWorker::comm_global_domain_release, nb::arg("domain_id"),
+            "Release a prepared or imported Global CommDomain."
+        )
         .def("comm_barrier", &ChipWorker::comm_barrier, nb::arg("comm_handle"), "Synchronize all ranks.")
         .def(
             "comm_destroy", &ChipWorker::comm_destroy, nb::arg("comm_handle"),