Skip to content

feat: Qwen3.6 target tensor parallelism - #532

Merged
davide221 merged 7 commits into
Luce-Org:mainfrom
jkyamog:feat-qwen35-target-tensor-parallel
Jul 30, 2026
Merged

feat: Qwen3.6 target tensor parallelism#532
davide221 merged 7 commits into
Luce-Org:mainfrom
jkyamog:feat-qwen35-target-tensor-parallel

Conversation

@jkyamog

@jkyamog jkyamog commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Add target-only tensor parallelism for dense Qwen3.5/Qwen3.6 models through
the GGML meta backend.

The public interface is:

--target-devices cuda:1,cuda:2 --target-split-mode tensor

layer remains the default. Tensor mode validates homogeneous local CUDA
devices and rejects incompatible weighted placement, IPC, PFlash, and draft
configurations. The implementation splits attention, fused projections,
DeltaNet, FFN, KV cache, and recurrent state tensors. The LM head remains
mirrored because Lucebox performs argmax inside the target graph.

Verification

  • Full native build passes.
  • All 15 CTest targets pass.
  • Qwen3.6 split-layout unit tests cover dimensions, segments, reconstruction,
    and placement validation.
  • Target-only streaming and non-streaming generation pass.
  • Q5 and Q6 long-context runs pass at 98,304 context with Q8 KV.
  • The branch explicitly rejects --draft in tensor mode; DFlash support is in
    the stacked follow-up.

Sustained stock-power results

Each configuration processed the same 26,758-token OpenClaw payload and
generated all 1,024 requested tokens with Q8 KV, no cache, and 98,304 context.
The RTX 5090 used its 600 W limit and each RTX 3090 used its stock 350 W limit.

Model / configuration PP TG Prefill Decode Total
Q5, RTX 5090 2,816 tok/s 59.5 tok/s 9.5 s 17.2 s 26.7 s
Q5, RTX 3090 980 tok/s 32.4 tok/s 27.3 s 31.6 s 58.9 s
Q5, 2x3090 layer split 611 tok/s 20.6 tok/s 43.8 s 49.7 s 94.2 s
Q5, 2x3090 TP + NVLink 1,423 tok/s 40.4 tok/s 18.8 s 25.4 s 44.2 s
Q5, 2x3090 TP, P2P disabled 977 tok/s 38.9 tok/s 27.4 s 26.3 s 53.7 s
Q6, RTX 5090 2,478 tok/s 48.7 tok/s 10.8 s 21.0 s 31.9 s
Q6, 2x3090 layer split 591 tok/s 17.7 tok/s 45.3 s 58.0 s 103.9 s
Q6, 2x3090 TP + NVLink 1,372 tok/s 34.8 tok/s 19.5 s 29.4 s 48.9 s
Q6, 2x3090 TP, P2P disabled 966 tok/s 34.3 tok/s 27.7 s 29.9 s 57.6 s

These are single sustained runs per configuration. Peak sampled power was
552.9 W on the RTX 5090 and 349.8 W on the single RTX 3090. Q6 on the 5090
peaked at 31,441 MiB VRAM.

This requires this PR: Luce-Org/lucebox-ggml#39

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 22 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/deps/llama.cpp/ggml/src/ggml-backend-meta.cpp Outdated
Comment thread server/src/qwen35/qwen35_tensor_parallel.h
Comment thread server/deps/llama.cpp/ggml/src/ggml-backend-meta.cpp Outdated
Comment thread server/src/placement/placement_config.h
Comment thread server/src/qwen35/qwen35_tensor_parallel.cpp
Comment thread server/deps/llama.cpp/ggml/src/ggml-backend-meta.cpp
@davide221
davide221 force-pushed the feat-qwen35-target-tensor-parallel branch from 82db48c to d538e6c Compare July 30, 2026 18:03
@davide221

Copy link
Copy Markdown
Contributor

Rebased onto current main and updated this branch at d538e6c. Additional fixes preserve centralized placement validation (2-16 unique, local, homogeneous CUDA devices; dense Qwen3.6; no weights, remote shard, PFlash, or draft), document the vendored lucebox-ggml#39 source, and extend test_cuda_comm_api with an opt-in real selected-device all-reduce check via DFLASH_TP_TEST_DEVICES. Validation on lucebox3: CUDA build plus 336/336 relevant tests; HIP build plus 324/324 relevant tests. lucebox3 has one NVIDIA GPU and this CUDA build has no NCCL, so the real two-NVIDIA reduction path compiled but could not be executed there.

@davide221
davide221 merged commit bc881af into Luce-Org:main Jul 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants