Skip to content

[Feature] Raise CHIP_MAX_TENSOR_ARGS above 128 for composed L2 callables #1645

Description

@high-cloud

Summary

Raise the chip-level tensor argument capacity from 128 to at least 192, or
replace the fixed cap with scalable/configurable storage, so that realistic
composed L2 callables can be dispatched without artificial splitting.

This is a follow-up to #786 / #788, which raised the previous limit from 64 to
128.

Motivation

While composing the DeepSeek V4 MTP decode path into a single L2 callable, the
combined entry needs 164 tensor parameters:

  1. main-model decode;
  2. device-side MTP verification and token packing;
  3. MTP draft decode.

The generated callable fails during startup with:

ValueError: make_callable: sig_count exceeds MaxSig

The current limit is:

inline constexpr int CHIP_MAX_TENSOR_ARGS = 128;

Because 164 exceeds this limit, the downstream implementation had to retain two
L2 submissions:

  • L2 1: main decode + MTP verification/packing;
  • L2 2: MTP draft decode.

Downstream references:

The validated 3-to-2 L2 reduction lowered steady critical-rank host decode time
from 45.159 ms to 42.980 ms. Supporting a single L2 would allow us to evaluate
removing the remaining dispatch, bind, runner and output-validation boundary.
No additional latency improvement is claimed until a comparable 2-to-1
benchmark is available.

Requested change

Please consider one of the following:

  1. raise CHIP_MAX_TENSOR_ARGS to at least 192, leaving some headroom above the
    current 164-argument reproducer; or
  2. make chip-level argument storage scalable or configurable so smaller models
    do not pay the maximum fixed-storage cost.

The change should cover all storage and transport points derived from the cap,
including:

  • ChipCallable::signature_;
  • ChipStorageTaskArgs;
  • runtime L2TaskArgs;
  • Python make_callable bindings;
  • hierarchical-worker mailbox sizing and its capacity static_assert;
  • relevant a2a3/a5 and sim/onboard paths.

Please also add a regression test that accepts a 164-tensor callable and rejects
CHIP_MAX_TENSOR_ARGS + 1 with an error containing both the requested and
supported tensor counts.

Capacity considerations

Tensor is currently 128 bytes, so raising the fixed capacity from 128 to 192
adds approximately 8 KiB to each fixed tensor-argument storage object.

The hierarchical-worker mailbox is currently 32 KiB and is statically sized to
hold the maximum accepted TaskArgs blob. A 192-tensor capacity may still fit,
but the complete layout including CallConfig, scalar arguments, headers,
alignment and the error region should be verified. A 256-tensor choice would
likely require a larger mailbox or out-of-line argument storage.

Expected behavior

A composed callable with 164 tensor parameters should compile, upload and
dispatch successfully on supported a2a3/a5 sim and onboard runtimes, without
requiring a model-specific argument-packing workaround or an extra L2 boundary.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions