TurboQuant v1.9.0 — DeepSeek-V4-Flash FP4 on 2× DGX Spark: Tensor-Parallel + Multi-Slot + MTP

🆕 v1.9.0 — DeepSeek-V4-Flash FP4 on 2× DGX Spark: Tensor-Parallel + Multi-Slot + MTP

The full serving stack on native FP4, all at once: 2-node tensor parallelism across two DGX Sparks, concurrent multi-slot batching, and MTP self-speculative decoding — on the FP4/FP8-native checkpoint with no requantization.

Environment: 2× NVIDIA DGX Spark (GB10, 128GB) over RoCE · DeepSeek-V4-Flash-FP4-FP8-native (nsparks lineage: F8_E4M3 dense + MXFP4 experts, 146GB) · ctx=8192.

Config (single stream)	gen t/s
FP4 PLAIN (2-node TP)	~15
FP4 + MTP + verify-reuse	16.6 (beats no-MTP)

Concurrent multi-slot requests batch on top of this for ~2× aggregate throughput.

What landed:

• A new GGML_TYPE_F8_E4M3_B128 quant type, end to end. Full machinery for the FP8 dense weights: the block struct (128× E4M3 values + one E8M0 power-of-two block scale), a bit-exact CPU codec (E4M3FN decode with subnormals/NaN/±448 saturation + E8M0 2^(e−127) scale), CUDA dequant (to_fp16 for the cuBLAS GEMM path) and a CUDA MMVQ GEMV kernel. MMVQ defaults to a __shared__-staged 256-entry LUT (bit-identical to the scalar dequant); an int8 dp4a approximation is available but env-gated OFF (GGML_CUDA_F8_APPROX_DP4A, lossy). The CPU↔CUDA↔LUT decode tables were verified identical across all 256 codes. To free the type slot, GGML_TYPE_TBQ3_0 was relocated 42 → 65 (with the range-guards that depend on it fixed — see audit below).
• FP4/FP8-native loads with no requantization, via a load-time adapter. The 146GB nsparks checkpoint (F8_E4M3 dense + MXFP4 experts, ftype 41) is served as-is. A loader adapter — gated strictly on ftype 41 so upstream models keep every key required — translates ~21 nsparks tensor names to our deepseek4 schema (compressor→compress, attn_kv→attn_kv_latent, output_hc→hc_head, dropped .weight suffixes) and fills the metadata keys nsparks omits (output_lora_rank, n_hc, hc_sinkhorn, the 43-entry compress_ratios, compress_rope_freq_base, …) with the architecture's defaults.
• MTP baked into the FP4 file (not a side-shard). The standalone MTP head (arch deepseek4_mtp_support, mtp.0.*) has no tok_embd/output, so it can't load as a draft on its own. turboquant/ds4_fp4_bake_mtp.py streams (via mmap — no 150GB in RAM) one combined GGUF = every FP4 tensor byte-identical + the MTP head renamed to blk.43.nextn.* + nextn_predict_layers=1, keeping file_type 41 so the FP4 adapter still applies. Then --spec-type draft-mtp consumes it directly.
• DSV4_VERIFY_REUSE is the unlock — an honest reversal from v1.8.0. In v1.8.0 this verify-graph-reuse infra shipped but OFF because graph rebuild overlapped the async GPU and it gained ~0 t/s. On the FP4 + MTP stack the situation inverts: MTP first measured 2.27 t/s with graphs reused = 0 — every speculative round rebuilds the full DSV4 graph (~1 s/round), and that rebuild is now the bottleneck. Turning DSV4_VERIFY_REUSE on restores reuse (0 → 100+/req) → 16.6 t/s, beating the no-MTP baseline (~15). It is perplexity-sensitive (the 256-padded view perturbs flash-attn fp-accumulation order), so it stays gated and was verified lossless here (greedy France→Paris / 수도→서울, plus a Hangul-jamo-decomposition multi-turn repro, stable across both boxes).
• 2-node TP + multi-slot + MTP, composed. SPMD tensor parallelism across two Sparks over RoCE (mirrored output, NCCL all-reduce), the multi-slot decode-batching path (DSV4_MULTISLOT, concurrent requests → ~2× aggregate), and MTP all run together. The enabling fix is a graph-scoped metadata rebuild in the meta-backend so the multi-slot + MTP verify graph no longer corrupts across TP splits.
• Long-context graph-arena crash fixed. The DSV4 chunk compressor built O(context) graph objects on long multi-turn prefills, exhausting the metadata arena (the "괭"-after-N-turns crash). DSV4_BATCHED_COMPRESSOR rewrites the chunk compressor to a fixed O(1) op count (carry-in / bulk-pool / carry-out, numerically equal to the unrolled recurrence); sched-context decoupling (max_splits cap, 21GB → ~1.3GB metadata) and -ub 256 keep every arena bounded.
• Pre-push code audit (4 parallel deep reviews + manual verify). Caught and fixed two critical latent bugs before release: (1) the new batched compressor's carry-out read a negative tensor offset when the carry-in block was the only completed block (n_full==0, e.g. a non-aligned ratio==128 prefill chunk); (2) the TBQ3_0 42→65 relocation silently broke three >=TBQ3_0 && <=TBQP4_4 range-guards (65..61 = always false) that gate the GB10 SoC-freeze protections for TBQ KV caches — restored to ==TBQ3_0 || (TBQ4_0..TBQP4_4). Plus a per-layer-per-token stderr debug flood removed from the launch scripts. F8 codec / sched cap / loader alias / bake tool audited clean.

Performance characterization

Single-stream decode is ~16.6 t/s (MTP+reuse) vs ~15 plain — FP4 by itself is not a decode-speed lever over Q4 (decode is memory-bound GEMV; native FP4 tensor cores help GEMM/prefill, not per-token GEMV), so the win here is MTP made viable by graph reuse, not the quant. The structural ceiling is the same one v1.8.0 documented: DSV4 decode is LPDDR-bandwidth-bound (matmuls ~52%, data-shuffle ~19%, attention ~2.3%), and MoE speculation is capped by distinct-expert byte growth — so MTP's edge over plain is modest by design, and the real result is that the full TP + multi-slot + MTP stack runs stably and losslessly on the native FP4 checkpoint at parity-or-better throughput. Multi-slot does not batch a single MTP stream (pure_decode=0, recurrent-state rollback ⇒ split_seq); its ~2× is aggregate, under concurrent load.

Usage

# 1) Bake the MTP head into the FP4 checkpoint → one combined GGUF (~5 min, no requant)
python3 turboquant/ds4_fp4_bake_mtp.py \
  DeepSeek-V4-Flash-FP4-FP8-native.gguf \
  DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf \
  DeepSeek-V4-Flash-FP4-FP8-native-MTP.gguf      # add --dry-run to validate metadata only

# 2) Mirror the combined GGUF to the second Spark (4-NIC parallel copy; edit INTERFACES for your cluster)
python3 turboquant/scp_dgx_spark.py ~/Models/DeepSeek-V4-Flash-GGUF/FP4

# 3) Serve the 2-node TP + multi-slot + MTP stack
bash fp4ctl.sh start                 # start | stop | restart | status  (pid-only kill, GPU-reclaim wait)

# raw two-box launch (master on box A 10.0.1.1, slave on box B 10.0.1.2):
#   bash run_tp_MASTER_DSV4-FP4-MTP.sh   # serves http://0.0.0.0:8080
#   bash run_tp_SLAVE_DSV4-FP4-MTP.sh    # follower, no HTTP

Key env baked into the launch scripts: DSV4_MULTISLOT=1 (concurrent slot batching), DSV4_VERIFY_REUSE=1 (MTP verify-graph reuse — the speed key), DSV4_BATCHED_COMPRESSOR=1 (long-context safety), with --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.0. Master and slave must run identical env (SPMD).

🛠️ Development effort. This was not a weekend project. The FP4 + 2-node TP + multi-slot + MTP stack landed in a ~5-day near-continuous sprint (Jun 18–22 2026, ~46 commits), the tail end of a ~2-week DeepSeek-V4-Flash marathon (~140 DSV4-related commits, near-daily Jun 9–22). One developer, two DGX Sparks, very little sleep — the type port, the loader adapter, the bake pipeline, the long-context crash hunt, the graphs-reused-0 → verify-reuse breakthrough, and a full pre-push audit, all by hand.

TurboQuant v1.8.0 — DeepSeek-V4-Flash + MTP Self-Speculative Decoding

TurboQuant v1.8.0 — DeepSeek-V4-Flash Full CUDA Port + MTP Self-Speculative Decoding

DeepSeek-V4-Flash (deepseek4) runs end-to-end on the TurboQuant fork — CSA/HCA compressed attention, hyper-connections (sinkhorn), the DSA lightning indexer, the 256-expert IQ2_XS MoE, and a phase-uniform decode graph with CUDA-graph capture. On top of that:

• MTP self-speculative decoding (--spec-type draft-mtp) from antirez's side GGUF. The MTP head ships as a separate third split shard (no requantization of the 82GB main shards); the --spec-draft-p-min 0.75 gate is mandatory.
• -ctk tbq3 -ctv tbq3 on DSV4 — global (ratio==0) layers get TBQ3_0 @ head_dim 512; SWA + compressed side caches stay f16 by quality policy.

Performance (GB10, production tbq3 + MTP)

-ctk tbq3 -ctv tbq3 --spec-type draft-mtp --spec-draft-p-min 0.75 --spec-draft-n-max 2

• ~16–20 tok/s decode, acceptance-driven (96–100% draft accept). The jitter is the nature of speculative decoding: round time is ~constant (~25 ms, target verify GPU pass), tokens-per-round swing 1→3 with text predictability.
• DSV4_KERNEL_PROF shows decode is memory-bound: matmul (IQ2_XS MoE + Q8_0 projections) ~52% near the LPDDR ceiling, data-shuffle ~19%, flash-attention (D=512 compressed KV) ~2.3%.

A verify-graph phase-uniform reuse infra also landed, OFF by default (DSV4_VERIFY_REUSE) — it's correct but off the critical path (graph build overlaps async GPU compute) and must be perplexity-gated.

Environment: NVIDIA DGX Spark (GB10, 128GB) · DeepSeek-V4-Flash-IQ2_XS-XL (82GB, antirez lineage) · ctx=16384 · greedy.

See README.md / README_KO.md for full details.

TurboQuant v1.7.0 — TriAttention + attn_rot_k Cleanup

TurboQuant v1.7.0 — TriAttention + attn_rot_k Duplicate Rotation Cleanup

TriAttention token pruning on AMX3_1 hybrid K cache — dequant-free pre-RoPE polar scoring + physical eviction. All TBQ/TBQP/AMX encoders freed from external attn_rot_k dependency (redundant Hadamard gone).

⚠️ Breaking change

• TBQX3_1 (tbqx3) removed. Its polar (r, φ) idea now lives inside AMX3_1's Part B, paired with a WHT Part A. Scripts calling --cache-type-k tbqx3 must switch to --cache-type-k amx3.

The compression story (looks like 2.37×, actually more)

Axis	Before	After	Gain
Raw block size (128 elements)	f16 → 256 B	AMX3_1 → 108 B	2.37×
Live tokens per slot (budget=128, 50% retention)	all alive	~128 alive, ~2370 evicted	~2×
Effective token-level compression	1×	—	~4.74× at attention-equivalent quality

Physical KV allocation still matches your -c flag. What shrinks is the fraction of it attention has to touch each decode step.

Quick start — TriAttention (Qwen3-14B, head_dim=128)

Step 1. Build calibration data (one-time, needs the bf16 HuggingFace weights for your model):

# Clone the reference scripts
git clone https://github.com/domvox/triattention-ggml ~/triattention-ref
cd ~/triattention-ref

mkdir -p ~/triattention-stats
CUDA_VISIBLE_DEVICES=0 python3 calibrate_ref.py \
    --model /path/to/Qwen3-14B \
    --input /path/to/wiki.test.raw \
    --output ~/triattention-stats/qwen3_14b.bin \
    --max-length 4096 \
    --device cuda

This produces a compact TRIA v2 binary (a few hundred KB to a few MB) — per-head Q statistics. Only needs to be built once per model.

Step 2. Run the server with TriAttention:

./llama-server \
    -m /path/to/qwen3-14b-q4_k_m.gguf \
    -c 40960 -ngl 999 --flash-attn on \
    --cache-type-k amx3 --cache-type-v amxv3 \
    --triattention ~/triattention-stats/qwen3_14b.bin \
    --tri-budget 128 \
    --tri-interval 128 \
    --tri-keep-first 32

Flag	Recommended	Meaning
--triattention FILE	calibration .bin	TRIA v2 stats (from Step 1)
--tri-budget N	128	Per-layer Top-B slots kept after each trigger
--tri-interval N	128	Trigger every N decoded tokens (paper β)
--tri-keep-first N	32	Attention sink — first N slots always kept. Critical: use 32 to protect prompt header; lower values cause repetition loops.

Non-TriAttention models (head_dim ≠ 128)

attn_rot_k cleanup applies universally — no flag needed, works on every head_dim:

# Just use the standard tbq3 shorthand; auto-resolves by head_dim
./llama-server -m MODEL.gguf --cache-type-k tbq3 --cache-type-v tbq3 ...

Auto-resolution map:

• head_dim=64 K → TBQP3_3 (double WHT per-head, D=64 specific)
• head_dim=128 → TBQ3_1
• head_dim=256/512 → TBQ3_0
• head_dim=576 (MLA) → TBQ3_4 (512-WHT + rope 64 passthrough)

Verified on

head_dim	Model	Result
64	gpt-oss-20b	✅ Korean poem natural, attn_rot_k=0
128	Qwen3-14B Q4_K_M	✅ needle test "다람쥐7429" recalled, 20.7 tok/s decode
256	Qwen3.5-9B Q8_0	✅ Korean reasoning natural, attn_rot_k=0
576 (MLA)	GLM-4.7-Flash	✅ Korean poem natural, attn_rot_k=0

Key design decisions

1. Dequant-free scoring — AMX3_1 Part B decodes straight into pre-RoPE (kxc, kyc) pairs. No shadow fp16 K buffer, no D2H copy.
2. Physical eviction with permanence — evicted slots zero both d_wht (FA attention) and d_r (future scoring). Zeroing only d_wht lets ghost slots re-enter Top-B next trigger.
3. Attention sink is mandatory — --tri-keep-first 0 collapses into "다름, 다름…" token-repetition loop. StreamingLLM-style fix: first N slots always kept.
4. attn_rot_k was duplicate rotation — every TBQ/TBQP/AMX encoder already carries internal WHT; external Hadamard cancelled via Parseval but cost one matmul per token. Now disabled across all 23 internally-WHT types.

Bottleneck we removed

The domvox Python reference needs a GPU→CPU roundtrip per scoring trigger: K gets dequantized, pulled back to host memory, scored in Python, and a mask gets pushed back. At β=128 and 40 layers this is a real throughput hit.

The AMX3_1 hybrid block was designed specifically to make this copy unnecessary: Part B stores (r, φ) in exactly the layout the scoring formula wants, so our three CUDA kernels (raw · z-norm · aggregate) read quantized blocks in place and never materialize a float K tensor. The mask produced by the histogram Top-B kernel is consumed by the eviction kernel on the same stream. Nothing crosses the PCIe bus during a trigger except the current n_kv position array (a few KB of ints). Result: TriAttention scoring is essentially free at the token-throughput scale (~2% of a 128-token decode window), instead of the seconds-per-trigger the host-roundtrip path would cost at 14B-model scale.

Credits

• TriAttention algorithm — Mao et al., "Tri-attention: Tail-token saliency via trigonometric scoring on pre-RoPE keys", arxiv 2604.04921 (2026).
• Python reference port — domvox/triattention-ggml. The TRIA v2 binary calibration format, the scoring math, and the per-head statistics extraction were adapted from that reference. The CUDA port, the AMX3_1 hybrid K cache integration, and the physical eviction + attention-sink wiring in llama-kv-cache / llama-context / fattn-vec are new in this release.

A small behind-the-scenes note

Back in early v1.6.0 work we sketched out "polar derotation" from scratch — storing K as (r, φ_content) with position peeled off algebraically — thinking it was our own little idea. It shipped, we wrote a paper, we pulled the paper when the theoretical framing didn't fully hold. Then while reading through TriAttention we realized the same polar decomposition was already sitting inside someone else's score formula, used for a different purpose entirely — token importance, not storage. The overlap turned out to be a gift: the AMX3_1 hybrid block slotted into TriAttention's scoring math almost unchanged, because Part B was the pre-RoPE polar pair the paper wanted. Independent rediscovery is humbling, but it made this integration much shorter than it had any right to be.

---

Binaries will attach automatically once the release workflow completes.

TurboQuant v1.6.0 — Polar Derotate + Tangent Residual (Qwen3-14B)

TurboQuant v1.6.0

📄 Paper

• Polar Derotation: Breaking the 3-Bit KV Cache Quality Cliff on Coupled-RoPE Language Models — theoretical foundation and evaluation of TBQX3_1
• 한국어판

Key Changes

• Polar Derotation on coupled-RoPE: first end-to-end implementation of content/position separation on an already-trained KV cache. K is stored as position-invariant polar coordinates (r, φ_content); position is restored algebraically at attention-read time.
• Tangent Residual: 1-bit per-pair analytical half-cell correction. Magnitude r·π/16 is a geometric constant, direction reuses dequant cos/sin. Cuts φ error in half (22.5° → 11.25°) with 2 extra FMAs per pair.
• TBQX3_1 block format: 58 B / 3.625 bpw — d_r (half) + qr[24] + qphi[24] + qtan[8].
• Beats f16 on math at 3.625 bpw (Qwen3-14B, temp=0, seed 1234).
• Removes failed variants (TBQXP3_1 Direct Signs, TBQX2_1 2-bit quadrant).

Math Accuracy (Qwen3-14B, ctx=40960, 35 problems, temp=0)

Config	K	V	Math (/35)	vs f16
tbqx3/tbq3	tbqx3_1	tbq3_1	13/35 (37.1%)	+8%
f16/f16	f16	f16	12/35 (34.3%)	—
tbq3/tbq3	tbq3_1	tbq3_1	10/35 (28.6%)	−17%

Seed-42 run: tbqx3/tbq3 17/35 vs f16 11/35 (+55%). Seed-52: tie. Average across 5 seeds: ~+20% vs f16, ~+35% vs legacy tbq3.

Memory & Speed (ctx=40960)

Config	KV buffer	Compression	tokens/s	vs f16
f16/f16	6400 MiB	1.00x	24–25	1.00x
tbqx3/tbq3	1350 MiB	4.74x	21–22	0.87x
tbq3/tbq3	1250 MiB	5.12x	21–22	0.87x

Recommended Settings

Tested:

Model	head_dim	K cache	V cache	Status
Qwen3-14B	128	tbqx3	tbq3	✅ Tested
Gemma 4 26B (MoE)	256/512	tbqp3	tbq3	✅ v1.5.2
GLM-4.7-Flash (MLA)	576/512	tbqp3	tbq3	✅ v1.5.2
GPT-OSS 120B	64	tbq4	tbq3	✅ v1.5.3

Should work (not tested by us):

Model	head_dim	K cache	V cache
Llama 3.x	128	tbqx3	tbq3
Qwen 2.5 / Qwen 3	128	tbqx3	tbq3
Mistral	128	tbqx3	tbq3
Yi	128	tbqx3	tbq3

For head_dim=128 coupled-RoPE models, tbqx3 (v1.6.0) is now the recommended K cache. Measured improvement over legacy tbq3 in math reasoning (+35% on Qwen3-14B), and token drift on rare-name transliterations is eliminated by the Tangent Residual correction. Recommendations for other head_dims (64, 256, 512, 576) are unchanged.

Run Options

--flash-attn on --n-gpu-layers 999

# Qwen3-14B / Llama 3.x / Mistral / Yi (head_dim=128, v1.6.0 new)
--cache-type-k tbqx3 --cache-type-v tbq3

# Gemma 4 26B (head_dim=256/512)
--cache-type-k tbqp3 --cache-type-v tbq3

# GLM-4.7-Flash (head_dim=576/512 MLA)
--cache-type-k tbqp3 --cache-type-v tbq3

# GPT-OSS 120B (head_dim=64)
--cache-type-k tbq4 --cache-type-v tbq3

⚠️ Requirements: CUDA only. Flash attention ON. All layers on GPU. No CPU offloading.

⚠️ Known limitation — VEC kernel only. TBQX3_1 is currently implemented on the fattn-vec path only. Decode throughput on Qwen3-14B is ~0.87× f16 as a result. Porting to the fattn-mma (Tensor Core) kernel would restore full speed but is not yet implemented — the main open item for a future release.

Target GPUs

sm_70 (V100), sm_80 (A100), sm_86 (3090 Ti), sm_89 (4090), sm_90 (H100), sm_100/120 (5090/DGX Spark)

Build from source

git clone https://github.com/AmesianX/TurboQuant.git
cd TurboQuant
mkdir build && cd build

cmake -DCMAKE_BUILD_TYPE=Release \
      -DBUILD_SHARED_LIBS=ON \
      -DGGML_CUDA=ON \
      -DGGML_BLAS=ON \
      -DGGML_F16C=ON \
      -DGGML_FMA=ON \
      -DLLAMA_BUILD_TESTS=OFF \
      -DLLAMA_BUILD_EXAMPLES=OFF \
      -DLLAMA_BUILD_SERVER=ON \
      -DGGML_CUDA_GRAPHS=ON \
      ..
make -j$(nproc)

Full changelog: v1.5.3...v1.6.0

TurboQuant v1.5.3 — Double WHT Per-Head for D=64

TurboQuant v1.5.3

Key Changes

• Double WHT per-head (D=64): Cross-head WHT abandoned. S1→WHT64→S2→WHT64 per-head.
• QJL re-enabled for D=64: 1-bit correction critical for multi-turn (9+ turns verified).

Recommended Settings

Tested:

Model	head_dim	K cache	V cache	Status
Gemma 4 26B (MoE)	256/512	tbqp3	tbq3	✅ Tested
GLM-4.7-Flash (MLA)	576(K)/512(V)	tbqp3	tbq3	✅ Tested
GPT-OSS 120B	64	tbq4	tbq3	✅ Tested (35/35)

Should work (not tested by us):

Model	head_dim	K cache	V cache
Llama 3.x	128	tbqp3	tbq3
Qwen 2.5 / Qwen 3	128	tbqp3	tbq3
Mistral	128	tbqp3	tbq3

head_dim=64 모델: 3-bit K(tbqp3)는 한국어 대화/멀티턴은 지원하나 행렬 연산 정밀도 부족. 4-bit K(tbq4) 사용 권장.

Math Accuracy (GPT-OSS 120B, head_dim=64, 35 problems, temp=0)

Config	K	V	Math	Korean	Multi-turn
f16/f16	f16	f16	35/35	✅	✅
tbq4/tbq3	tbq4_2	tbq3_2	35/35	✅	✅
tbqp3/tbq3	tbqp3_3	tbq3_2	❌ (matrix)	✅	✅ (9+ turns)

Run Options

# Required for ALL TurboQuant configurations:
--flash-attn on --n-gpu-layers 999

# Gemma 4 26B
--cache-type-k tbqp3 --cache-type-v tbq3

# GLM-4.7-Flash
--cache-type-k tbqp3 --cache-type-v tbq3

# GPT-OSS 120B (head_dim=64)
--cache-type-k tbq4 --cache-type-v tbq3

# Llama 3.x / Qwen / Mistral (head_dim=128, not tested by us)
--cache-type-k tbqp3 --cache-type-v tbq3

⚠️ Requirements: CUDA only (no ROCm/Metal/CPU). Flash attention ON. All layers on GPU. No MoE CPU offloading.

Target GPUs

sm_70 (V100), sm_80 (A100), sm_86 (3090 Ti), sm_89 (4090), sm_90 (H100), sm_100/120 (5090/DGX Spark)

Build from source

git clone https://github.com/AmesianX/TurboQuant.git
cd TurboQuant
mkdir build && cd build

Full build (all TBQ instances — slower compile, supports all K/V combinations):

cmake -DCMAKE_BUILD_TYPE=Release \
      -DBUILD_SHARED_LIBS=ON \
      -DGGML_CUDA=ON \
      -DGGML_BLAS=ON \
      -DGGML_F16C=ON \
      -DGGML_FMA=ON \
      -DLLAMA_BUILD_TESTS=OFF \
      -DLLAMA_BUILD_EXAMPLES=OFF \
      -DLLAMA_BUILD_SERVER=ON \
      -DGGML_CUDA_GRAPHS=ON \
      ..
make -j$(nproc)

TBQ tuning build (essential instances only — faster compile, recommended for development):

cmake -DCMAKE_BUILD_TYPE=Release \
      -DBUILD_SHARED_LIBS=ON \
      -DGGML_CUDA=ON \
      -DGGML_BLAS=ON \
      -DGGML_F16C=ON \
      -DGGML_FMA=ON \
      -DLLAMA_BUILD_TESTS=OFF \
      -DLLAMA_BUILD_EXAMPLES=OFF \
      -DLLAMA_BUILD_SERVER=ON \
      -DGGML_CUDA_GRAPHS=ON \
      -DGGML_CUDA_FA_TBQ_TUNING=ON \
      ..
make -j$(nproc)

Full changelog: v1.5.2...v1.5.3

TurboQuant v1.5.2 — PPL 21%→6%, Precision Fix, Deterministic Kernel

v1.5.2 — PPL + Math Bench Dual Improvement

3-bit KV cache now achieves f16-equivalent quality on both PPL and math benchmarks.

PPL (wikitext-2, ctx=2048, Gemma 4 26B MoE)

Config	PPL	vs f16
f16/f16	419.8	1.00x
tbqp3/tbq3	454.7	1.08x

v1.5.1 was 1.21x → now 1.08x (21% gap → 8% gap).

Math Accuracy (35 problems × 10 runs, 262K ctx, temp=0)

Config	Average	Peak
tbqp3/tbq3	19.1/35	23/35
f16/f16	20.1/35	21/35

Key Changes

1. Attention Sharpening (α = 1 + 1/(2×SQNR)): Compensates softmax flattening from quantization noise. TBQP3 α=1.036, TBQ3 α=1.016. Derived from MMSE theory.
2. V Rotation Bugfix: attn_rot_v was enabled but IWHT decode has no inverse rotation — V output was corrupted.
3. Per-block Norm for TBQ3 D=512 encode.
4. 1.15x V compensation removed (replaced by principled sharpening).
5. tbq4_0 D=512 OOB read fix.

4.2x compression, zero quality loss.

TurboQuant v1.5.1 — Exceeds f16 Quality (4.2x compression)

TurboQuant v1.5.1

Highlights

• 3-bit KV cache EXCEEDS f16 quality — tbqp3/tbq3 avg 37.4/65 vs f16 36.6/65
• SWA f16 bypass — auto-upgrade SWA K+V to f16 (the hidden quality bottleneck)
• V 512-WHT — V cache now uses same 512-point WHT as K
• QJL D=512 restored — works correctly with SWA f16 bypass
• 4.2× compression — 5120 MiB → 1290 MiB

Benchmark (Gemma 4 26B-A4B MoE, DGX Spark GB10, 262K ctx, temp=0)

Config	K	V	Global KV	SWA KV	Math Accuracy (10 runs)	Avg	Compression
tbqp3/tbq3	tbqp3	tbq3	990 MiB	300 MiB(f16)	37,38,40,38,38,36,37,36,37,37	37.4	4.2x
tbq3/tbq3	tbq3	tbq3	980 MiB	300 MiB(f16)	39,39,37,37,38,35,35,39,35,36	37.0	4.2x
f16/f16	f16	f16	5120 MiB	300 MiB(f16)	37,36,36,36,36,38,36,38,37,36	36.6	1.0x

Key Techniques

1. SWA KV f16 Bypass: SWA cache is small (~300 MiB) but has 25 layers dominating quality. Auto-upgrade eliminates SWA quantization noise.
2. V 512-WHT + 512-IWHT: V cache uses same encode path as K (512-point WHT + global norm).
3. QJL D=512: Previously removed as "ineffective" — SWA noise was masking the improvement. Now restored with attn_rot auto-disabled for TBQP.
4. Recommended: --cache-type-k tbqp3 --cache-type-v tbq3 (SWA auto-upgraded to f16)

Compatibility

• All v1.5.0 features preserved
• Upstream llama.cpp synced
• Existing models work without changes

TurboQuant v1.5.0 — Upstream Rebase + Gemma 4 Support

TurboQuant v1.5.0

Highlights

• Full upstream rebase on latest llama.cpp (b7ad48ebd) — future upstream sync via git merge upstream/master
• Gemma 4 support — first TBQ implementation for hybrid SWA architecture (head_dim=512 global + 256 SWA)
• All existing features preserved: MMA tensor core, QJL scalar correction, MLA asymmetric (GLM-4.7-Flash)

Gemma 4 Key Techniques

1. SWA cache type auto-remapping: Automatically assigns correct TBQ sub-type when global and SWA head_dim differ
2. Variable GQA support: Per-layer head_count_kv now works with WHT rotation
3. D=512 single-pass WHT + global norm: K quantization applies 512-point WHT in one pass (9-stage butterfly). Both 256-blocks share global norm for cross-block scale consistency. Q preprocessing also uses 512-point WHT. V IWHT uses 256-block independent processing
4. head_dim via op_params: head_dim passed to set_rows kernel for correct D=512 vs D=256 dispatch

Benchmark: Gemma 4 31B-it Dense (UD-Q4_K_XL, DGX Spark GB10, 262K ctx)

Cache	GPU Memory	Compression	PP t/s	TG t/s	PPL (wiki, 2K)	Math Accuracy	Pauli
f16/f16	41,500 MiB	1.0x	152.9	10.1	309.7	42/65 (64.6%)	PASS
tbq3/tbq3	23,215 MiB	1.8x	112.9	9.0	212.1	32/65 (49.2%)	PASS

Benchmark: Gemma 4 26B-A4B MoE (UD-Q4_K_XL, DGX Spark GB10, 262K ctx)

Cache	GPU Memory	Compression	TG t/s	Math Accuracy	Pauli
f16/f16	5,720 MiB	1.0x	56.4	37/65 (56.9%)	PASS
tbq3/tbq3	1,106 MiB	5.2x	41.1	30/65 (46.2%)	PASS

D=512 limitation: QJL 1-bit correction (TBQP) does not work at D=512. All 8 QJL variants tested degraded quality. Only TBQ (MSE-only) is supported for head_dim=512. TBQP works normally for head_dim<=256.

Benchmark: Qwen3-14B (Q4_K_M, head_dim=128)

Cache	Math Accuracy	KV Memory
f16/f16	22/65 (33.8%)	20,480 MiB
tbqp3/tbq3	22/65 (33.8%)	8,160 MiB

Identical accuracy with 2.5x less memory. f16 wins 9, TBQ wins 9 -- perfectly balanced.

Benchmark: GLM-4.7-Flash (MLA, K=576/V=512) -- unchanged from v1.4.2

Cache	KV MiB	Compression	TG t/s
f16/f16	10,469	1.0x	67.5
tbq3/tbq3	2,944	3.6x	49.7
tbqp3/tbq3	2,981	3.5x	42.8

Supported Architectures

Architecture	head_dim	Status
Standard (Llama, Qwen, Mistral)	128	Full support (TBQ + TBQP)
GQA (Qwen3-14B, etc.)	128/256	Full support (TBQ + TBQP)
MoE (Qwen3-30B-A3B, etc.)	128	Full support (TBQ + TBQP)
MLA (GLM-4.7-Flash, DeepSeek)	576/512	MMA tensor core (TBQ + TBQP)
Gemma 4 hybrid SWA (Dense + MoE)	512/256	New — TBQ only (QJL N/A at D=512)
Gemma 2/3 SWA	256	Full support (TBQ + TBQP)

Build

cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON \
      -DGGML_CUDA=ON -DGGML_BLAS=ON -DGGML_CCACHE=OFF \
      -DCMAKE_EXE_LINKER_FLAGS="-lpthread -lm" \
      -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF \
      -DLLAMA_BUILD_SERVER=ON -DCMAKE_VERBOSE_MAKEFILE=ON ..
make -j12

Usage

# head_dim<=256: use tbqp3 (QJL) for K
./llama-server -m your-model.gguf \
    --flash-attn on --cache-type-k tbqp3 --cache-type-v tbq3 -ngl 999 ...

# Gemma 4 (head_dim=512): use tbq3 for K (QJL not supported at D=512)
./llama-server -m gemma4-model.gguf \
    --flash-attn on --cache-type-k tbq3 --cache-type-v tbq3 -ngl 999 ...

Full Changelog: v1.4.2...v1.5.0

TurboQuant v1.4.2 — MMA Tensor Core Acceleration

TurboQuant v1.4.2 — MMA Tensor Core Acceleration + QJL Scalar Correction

Speed Improvements (GLM-4.7-Flash, DGX Spark GB10)

Cache	TG t/s (v1.4.1)	TG t/s (v1.4.2)	Improvement
tbq3/tbq3	32.0	49.7	+55%
tbqp3/tbq3	31.5	42.8	+36%
tbq4/tbq4	33.4	~49	+47%
tbqp4/tbq4	28.9	~42	+45%

What's New

• MMA tensor core attention: TBQ/TBQP KV cache types now use MMA (tensor core) for QK and softmaxV
• QJL scalar correction: TBQP 1-bit correction via lightweight scalar ops
• Spatial K/V: K dequanted to spatial domain, V = K view. Output IWHT eliminated
• Warp shuffle IWHT: syncthreads reduced 8 to 4 per sub-block
• Proper kernel signature: fattn_kernel_t extended with raw_K_data, Q_wht2_data
• Fused Q WHT12: Single kernel, no cudaMemcpy
• Dead code cleanup: 213 lines removed

TurboQuant v1.4.1 — GLM-4.7-Flash (MLA) Asymmetric K/V Support

GLM-4.7-Flash / DeepSeek-V2/V3 (MLA) TurboQuant Support

MLA architecture: K=concat(latent[512], rope[64])=576, V=latent[512].

Key Techniques

• D_V template parameter: Separates K/Q dim (576) from V dim (512)
• RoPE f16 passthrough: Sub-block 3 (rope 64) stored as raw f16 — rope norm ~80x larger than latent makes any quantization catastrophic
• MLA V-as-K-view: V dequantize uses MSE only (no QJL — QJL is K·Q dot product correction only)

Benchmark (GLM-4.7-Flash UD-Q4_K_XL, DGX Spark GB10)

Cache	KV MiB	Compress	PP t/s	TG t/s	PPL	Pauli
f16/f16	10,469	1.0x	73.0	60.3	5.998	PASS
tbqp3/tbq3	2,981	3.5x	66.8	31.5	6.586	PASS
tbq3/tbq3	2,944	3.6x	68.2	32.0	6.836	PASS

Bugs Fixed (vs v1.4.0)

• Dispatch ordering: ASYM before symmetric 576 cases
• RoPE quantization → garbage: f16 passthrough
• TBQP V dequant QJL → garbage: MSE only for V
• Q WHT sub-block race condition: added __syncthreads

⚠️ v1.4.0 has been deleted — it had critical bugs causing garbage output on GLM models.

🤖 Generated with Claude Code