Fix MTP speculative decoding on multi-GPU splits (fixes #260) - #262
Closed
dushyant30suthar wants to merge 1 commit into
Closed
Fix MTP speculative decoding on multi-GPU splits (fixes #260)#262dushyant30suthar wants to merge 1 commit into
dushyant30suthar wants to merge 1 commit into
Conversation
Two fixes, both for hybrid GDN models with a draft under layer-split loading across multiple devices. Details and traces in turboderp-org#260. 1. Group batched rewind jobs by each layer's actual device. _collect_rewind_jobs assumed all of a cache's GDN layers share one device and launched every job under the first layer's index; with layer-split loading the layers span every device in the split, so the rewind kernels dereferenced foreign pointers — CUDA illegal memory access on the first draft rejection (gdn.cu batched_conv_rewind / batched_state_rewind). The TP path is unaffected: within a worker all layers share a device, so the grouping degenerates to the previous behavior. 2. Completion wins over requeue. In the drafting branch of iterate_gen the requeue check ran before the EOS check, so a job producing its final token while a requeue condition held was requeued instead of completed. The replacement job carried a zero-or-negative remaining token budget, could never reach EOS, requeued itself indefinitely, and each orphan held a recurrent state slot — with the default pool of 4, the fifth distinct request died on "Cannot create new state: no available slots". Non-EOS requeues (output chunking, rollback recovery) behave as before. Validated on 2x RTX 5060 Ti (sm_120), native Linux, Qwen3.6-27B EXL3 5.0bpw with embedded MTP: 30/30 requests across the previous failure patterns, full 4k-163k depth sweeps with drafting on layer split and on tensor_parallel, draft acceptance normal.
Author
|
Both fixes have landed in Closing this in favour of those commits. I've been running |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both fixes from #260, as offered there.
1. Group batched rewind jobs by each layer's actual device (
modules/gated_delta_net.py)_collect_rewind_jobsassumed all of a cache's GDN layers live on one device and launched every rewind job under the first layer's index. With layer-split loading the layers span every device in the split, sobatched_conv_rewind/batched_state_rewinddereferenced foreign pointers — CUDA illegal memory access on the first draft rejection. Jobs are now grouped per device and dispatched under the right guard/stream. The TP path is unaffected: within a worker all layers share a device, so the grouping degenerates to the previous single-entry behavior.2. Completion wins over requeue (
generator/generator.py)In the drafting branch of
iterate_genthe requeue check ran before the EOS check. A job producing its final token while a requeue condition held was requeued instead of completed; the replacement carried a zero-or-negative remaining token budget, could never reach EOS, requeued itself indefinitely, and each orphan held a recurrent state slot — with the default pool of 4, the fifth distinct request failed onCannot create new state: no available slotsand the generator wedged.rq and not eosrestores the intended precedence; non-EOS requeues (output chunking, rollback recovery on long generations) are untouched.Validation: 2× RTX 5060 Ti (sm_120), native Linux, CUDA 12.8 and 13.3, Qwen3.6-27B EXL3 5.0bpw with embedded MTP tensors, via TabbyAPI and via the bare generator API. 30/30 requests across the failure patterns from #260 (max_tokens=1 series, distinct prompts, growing multi-turn conversation), plus full 4k–163k depth sweeps with drafting active on both layer split and
tensor_parallel— no illegal memory access, no slot exhaustion, draft acceptance in the expected range (~2.5–3.1× accept-speedup on code/agentic prompts).