Summary
Raise the chip-level tensor argument capacity from 128 to at least 192, or
replace the fixed cap with scalable/configurable storage, so that realistic
composed L2 callables can be dispatched without artificial splitting.
This is a follow-up to #786 / #788, which raised the previous limit from 64 to
128.
Motivation
While composing the DeepSeek V4 MTP decode path into a single L2 callable, the
combined entry needs 164 tensor parameters:
- main-model decode;
- device-side MTP verification and token packing;
- MTP draft decode.
The generated callable fails during startup with:
ValueError: make_callable: sig_count exceeds MaxSig
The current limit is:
inline constexpr int CHIP_MAX_TENSOR_ARGS = 128;
Because 164 exceeds this limit, the downstream implementation had to retain two
L2 submissions:
- L2 1: main decode + MTP verification/packing;
- L2 2: MTP draft decode.
Downstream references:
The validated 3-to-2 L2 reduction lowered steady critical-rank host decode time
from 45.159 ms to 42.980 ms. Supporting a single L2 would allow us to evaluate
removing the remaining dispatch, bind, runner and output-validation boundary.
No additional latency improvement is claimed until a comparable 2-to-1
benchmark is available.
Requested change
Please consider one of the following:
- raise
CHIP_MAX_TENSOR_ARGS to at least 192, leaving some headroom above the
current 164-argument reproducer; or
- make chip-level argument storage scalable or configurable so smaller models
do not pay the maximum fixed-storage cost.
The change should cover all storage and transport points derived from the cap,
including:
ChipCallable::signature_;
ChipStorageTaskArgs;
- runtime
L2TaskArgs;
- Python
make_callable bindings;
- hierarchical-worker mailbox sizing and its capacity
static_assert;
- relevant a2a3/a5 and sim/onboard paths.
Please also add a regression test that accepts a 164-tensor callable and rejects
CHIP_MAX_TENSOR_ARGS + 1 with an error containing both the requested and
supported tensor counts.
Capacity considerations
Tensor is currently 128 bytes, so raising the fixed capacity from 128 to 192
adds approximately 8 KiB to each fixed tensor-argument storage object.
The hierarchical-worker mailbox is currently 32 KiB and is statically sized to
hold the maximum accepted TaskArgs blob. A 192-tensor capacity may still fit,
but the complete layout including CallConfig, scalar arguments, headers,
alignment and the error region should be verified. A 256-tensor choice would
likely require a larger mailbox or out-of-line argument storage.
Expected behavior
A composed callable with 164 tensor parameters should compile, upload and
dispatch successfully on supported a2a3/a5 sim and onboard runtimes, without
requiring a model-specific argument-packing workaround or an extra L2 boundary.
Summary
Raise the chip-level tensor argument capacity from 128 to at least 192, or
replace the fixed cap with scalable/configurable storage, so that realistic
composed L2 callables can be dispatched without artificial splitting.
This is a follow-up to #786 / #788, which raised the previous limit from 64 to
128.
Motivation
While composing the DeepSeek V4 MTP decode path into a single L2 callable, the
combined entry needs 164 tensor parameters:
The generated callable fails during startup with:
The current limit is:
Because 164 exceeds this limit, the downstream implementation had to retain two
L2 submissions:
Downstream references:
The validated 3-to-2 L2 reduction lowered steady critical-rank host decode time
from 45.159 ms to 42.980 ms. Supporting a single L2 would allow us to evaluate
removing the remaining dispatch, bind, runner and output-validation boundary.
No additional latency improvement is claimed until a comparable 2-to-1
benchmark is available.
Requested change
Please consider one of the following:
CHIP_MAX_TENSOR_ARGSto at least 192, leaving some headroom above thecurrent 164-argument reproducer; or
do not pay the maximum fixed-storage cost.
The change should cover all storage and transport points derived from the cap,
including:
ChipCallable::signature_;ChipStorageTaskArgs;L2TaskArgs;make_callablebindings;static_assert;Please also add a regression test that accepts a 164-tensor callable and rejects
CHIP_MAX_TENSOR_ARGS + 1with an error containing both the requested andsupported tensor counts.
Capacity considerations
Tensoris currently 128 bytes, so raising the fixed capacity from 128 to 192adds approximately 8 KiB to each fixed tensor-argument storage object.
The hierarchical-worker mailbox is currently 32 KiB and is statically sized to
hold the maximum accepted
TaskArgsblob. A 192-tensor capacity may still fit,but the complete layout including
CallConfig, scalar arguments, headers,alignment and the error region should be verified. A 256-tensor choice would
likely require a larger mailbox or out-of-line argument storage.
Expected behavior
A composed callable with 164 tensor parameters should compile, upload and
dispatch successfully on supported a2a3/a5 sim and onboard runtimes, without
requiring a model-specific argument-packing workaround or an extra L2 boundary.