Does this project allow me to run Qwen3.6-35B-A3B Q4_K_XL faster than llama.cpp on my GTX 1080 Ti (11GB VRAM) because of MoE aware GPU caching?
I am looking for this optimization:
llama.cpp partitions a MoE model vertically, layer by layer, across CPU RAM and GPU VRAM. As a result, no token is processed entirely on the GPU, placing the CPU on the critical path for every token and creating a substantial performance bottleneck.
The idea: if a small subset of experts of the whole network is only active for most token, we should slice vertically so that the hot (active) experts are cached on GPU and cold ones on CPU. This would allow most tokens to be processed solely on the GPU, leading to massive speedups.
asking because of this post: https://www.lucebox.com/blog/spark
Does this project allow me to run Qwen3.6-35B-A3B Q4_K_XL faster than llama.cpp on my GTX 1080 Ti (11GB VRAM) because of MoE aware GPU caching?
I am looking for this optimization:
llama.cpp partitions a MoE model vertically, layer by layer, across CPU RAM and GPU VRAM. As a result, no token is processed entirely on the GPU, placing the CPU on the critical path for every token and creating a substantial performance bottleneck.
The idea: if a small subset of experts of the whole network is only active for most token, we should slice vertically so that the hot (active) experts are cached on GPU and cold ones on CPU. This would allow most tokens to be processed solely on the GPU, leading to massive speedups.
asking because of this post: https://www.lucebox.com/blog/spark