Skip to content

Fix MTP speculative decoding on multi-GPU splits (fixes #260) - #262

Closed
dushyant30suthar wants to merge 1 commit into
turboderp-org:masterfrom
dushyant30suthar:fix/multi-gpu-mtp
Closed

Fix MTP speculative decoding on multi-GPU splits (fixes #260)#262
dushyant30suthar wants to merge 1 commit into
turboderp-org:masterfrom
dushyant30suthar:fix/multi-gpu-mtp

Conversation

@dushyant30suthar

Copy link
Copy Markdown

Both fixes from #260, as offered there.

1. Group batched rewind jobs by each layer's actual device (modules/gated_delta_net.py)

_collect_rewind_jobs assumed all of a cache's GDN layers live on one device and launched every rewind job under the first layer's index. With layer-split loading the layers span every device in the split, so batched_conv_rewind/batched_state_rewind dereferenced foreign pointers — CUDA illegal memory access on the first draft rejection. Jobs are now grouped per device and dispatched under the right guard/stream. The TP path is unaffected: within a worker all layers share a device, so the grouping degenerates to the previous single-entry behavior.

2. Completion wins over requeue (generator/generator.py)

In the drafting branch of iterate_gen the requeue check ran before the EOS check. A job producing its final token while a requeue condition held was requeued instead of completed; the replacement carried a zero-or-negative remaining token budget, could never reach EOS, requeued itself indefinitely, and each orphan held a recurrent state slot — with the default pool of 4, the fifth distinct request failed on Cannot create new state: no available slots and the generator wedged. rq and not eos restores the intended precedence; non-EOS requeues (output chunking, rollback recovery on long generations) are untouched.

Validation: 2× RTX 5060 Ti (sm_120), native Linux, CUDA 12.8 and 13.3, Qwen3.6-27B EXL3 5.0bpw with embedded MTP tensors, via TabbyAPI and via the bare generator API. 30/30 requests across the failure patterns from #260 (max_tokens=1 series, distinct prompts, growing multi-turn conversation), plus full 4k–163k depth sweeps with drafting active on both layer split and tensor_parallel — no illegal memory access, no slot exhaustion, draft acceptance in the expected range (~2.5–3.1× accept-speedup on code/agentic prompts).

Two fixes, both for hybrid GDN models with a draft under layer-split
loading across multiple devices. Details and traces in turboderp-org#260.

1. Group batched rewind jobs by each layer's actual device.
   _collect_rewind_jobs assumed all of a cache's GDN layers share one
   device and launched every job under the first layer's index; with
   layer-split loading the layers span every device in the split, so the
   rewind kernels dereferenced foreign pointers — CUDA illegal memory
   access on the first draft rejection (gdn.cu batched_conv_rewind /
   batched_state_rewind). The TP path is unaffected: within a worker all
   layers share a device, so the grouping degenerates to the previous
   behavior.

2. Completion wins over requeue.
   In the drafting branch of iterate_gen the requeue check ran before
   the EOS check, so a job producing its final token while a requeue
   condition held was requeued instead of completed. The replacement job
   carried a zero-or-negative remaining token budget, could never reach
   EOS, requeued itself indefinitely, and each orphan held a recurrent
   state slot — with the default pool of 4, the fifth distinct request
   died on "Cannot create new state: no available slots". Non-EOS
   requeues (output chunking, rollback recovery) behave as before.

Validated on 2x RTX 5060 Ti (sm_120), native Linux, Qwen3.6-27B EXL3
5.0bpw with embedded MTP: 30/30 requests across the previous failure
patterns, full 4k-163k depth sweeps with drafting on layer split and on
tensor_parallel, draft acceptance normal.
@dushyant30suthar

Copy link
Copy Markdown
Author

Both fixes have landed in dev (ce77f0a and 1048003) — thanks for taking them, and the receive_sample() placement for the requeue/EOS one is the better call since it covers the non-draft path too.

Closing this in favour of those commits. I've been running dev equivalents on the 2× RTX 5060 Ti setup with tensor_parallel + MTP for a full day of real agentic traffic since: 787 requests, ~181k tokens generated, 97.6% prefix cache hit rate, no slot exhaustion and no rewind crashes.

@dushyant30suthar
dushyant30suthar deleted the fix/multi-gpu-mtp branch July 29, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant