TurboQuant+

Getting Started Guide | Configuration Recommendations | llama.cpp Fork | Swift MLX Fork | vllm-swift | Commercial Support

🍎 On Apple Silicon and want the fastest path? Use ekryski/mlx-swift-lm — Eric Kryski's Swift MLX implementation that I've been actively collaborating on. Native Swift, ~2.5x faster decode than Python mlx-lm, full TurboQuant+ support including turbo4v2 (4-bit K + 2-bit V). 144 tok/s on Qwen3.5-35B-A3B MoE at 4K on M5 Max. For OpenAI-compatible serving, use vllm-swift — a native Swift/Metal backend for vLLM built on mlx-swift-lm. No Python in the inference hot path, works with Hermes, OpenCode, and any OpenAI client. This llama.cpp repo is for cross-platform deployment (CUDA, ROCm, CPU, Metal).

Prebuilt Binaries

Download ready-to-run builds from Releases. No build tools needed.

| Platform | Download | |----------|----------| | Mac (Apple Silicon, Metal) | turboquant-plus-*-macos-arm64-metal.tar.gz | | Windows (CUDA 12.4) | turboquant-plus-*-windows-x64-cuda12.4.zip |

Unpack and run. Includes llama-server, llama-bench, llama-cli, and all tools.

---

Implementation of TurboQuant (ICLR 2026) with implementation work, experiments, and follow-on findings beyond the base paper. KV cache compression for local LLM inference.

Note

This repository is an experimental integration and research workspace for TurboQuant-related work targeting llama.cpp. The goal is to make it easier to compare approaches, collect reproducible benchmark and quality data, and share implementation details across hardware and backends. It is not intended as a separate long-term fork or a proposal to merge the branch as a whole.

If individual pieces prove useful and stable, the intent is to upstream them incrementally as small, reviewable patches in line with llama.cpp's normal contribution process.

What's In This Branch

• Experimental TurboQuant-related integrations for llama.cpp
• Benchmark and quality validation across models, contexts, and hardware
• Backend-specific implementation work and performance experiments
• Documentation and writeups intended to make testing and reproduction easier
• Candidate ideas that may be worth upstreaming individually if they prove stable

Current Findings

Three follow-on findings in this branch have been independently validated by multiple researchers across different hardware and backends:

1. V compression is free. Compressing the value cache (even down to 2 bits) has zero measurable effect on attention quality when key precision is maintained. Confirmed on Metal (M5 Max), CUDA RTX 4090 (@sztlink), and CUDA RTX 3090 (@HyperionMS2040). See asymmetric K/V paper.
2. All quality degradation comes from K compression. This is why asymmetric configs (q8_0-K + turbo-V) rescue models where symmetric fails. Validated across Qwen, Llama, Mistral, and Command-R+ families. See M5 Max stress test.
3. Boundary layers are disproportionately sensitive. Protecting the first 2 + last 2 layers at higher precision recovers 37-91% of the quality gap. See Boundary V paper.

Additional experiments and writeups: Sparse V dequant (+22.8% decode), block size optimization (5.12x compression), turbo4 resurrection (QJL hurts, PolarQuant works), EDEN optimal-S response (rotation is first-order, scale is second-order).

Compresses transformer KV cache 3.8-6.4x using PolarQuant + Walsh-Hadamard rotation. Near q8_0 prefill speed and ~0.9x decode throughput at long context (Apple Silicon). Full format family: turbo2 (2-bit, 6.4x), turbo3 (3-bit, 4.6-5.1x), turbo4 (4-bit, 3.8x). turbo3 compression depends on storage block size; see block size study.

Sparse V: Attention-gated KV cache decoding that skips low-weight V positions during inference. Up to +22.8% decode speed at 32K context, validated on wikitext-103 (50 chunks, CI +/-0.021) with no measurable PPL change. Not TurboQuant-specific; validated across q8_0, q4_0, and turbo3 KV formats. ~1% perplexity increase vs q8_0 from compression; Sparse V itself introduces no additional degradation (ON/OFF delta = 0.000).

Validated end-to-end from 1.5B to 104B on M5 Max via llama.cpp Metal. 104B at 128K context on a MacBook with turbo3 (PPL 4.024, 74 GB peak memory).

Status: v1 Complete, Speed Optimized, Community-Tested

• 511+ Python tests, 100% code coverage on diagnostics
• C port integrated into llama.cpp with Metal GPU kernels
• --cache-type-k turbo3 --cache-type-v turbo3 works on Apple Silicon (turbo2/turbo3/turbo4 all supported)
• turbo2 Metal support: 2-bit, 6.4x compression, +6.48% PPL — for extreme memory pressure or asymmetric K/V
• q8_0 prefill speed parity achieved (2747 vs 2694 tok/s)
• Norm correction: PPL beats q8_0 on CUDA (-1.17%), +1.1% on Metal (ported from @spiritbuun)
• 4-mag LUT: auto-detected on M1/M2/M3/M4, +38-45% decode at long context
• Layer-adaptive mode 2: q8_0 quality at 3.5x compression (last 8 layers at q8_0)
• Temporal decay: 30-34% memory savings at long context (experiment branch)
• NIAH retrieval: 9/9 single needle with sparse V (vs 7/9 baseline), 100% multi-key through 32K. 30/30 on Llama-70B, 10/10 on Command-R+ 104B
• 14 decode approaches tested on M2 Pro — comprehensive hardware analysis
• Stress tested up to 104B: Command-R+ 104B Q4_K_M at 128K context (PPL 4.024). Llama-70B Q4_K_M at 48K (PPL 4.019). turbo3 prefill faster than q8_0 at 32K on both models
• Community: 30+ testers across M1/M2/M3/M5 Mac, RTX 3080 Ti/3090/4090/5090, AMD 6800 XT/9070 XT
• Rotation Gaussianization validated on real Qwen3 KV tensors (kurtosis 900 → 2.9)

---

Quality and Speed (M5 Max 128GB)

Top-of-Tree Results

| Cache Type | Bits/val | Compression | PPL (wikitext-2, 512c) | vs q8_0 | |------------|----------|-------------|----------------------|---------| | f16 | 16.0 | 1.0x | 6.121 | -0.16% | | q8_0 | 8.5 | 1.9x | 6.111 | baseline | | turbo4 | 4.25 | 3.8x | 6.125 | +0.23% | | q4_0 | 4.5 | 3.6x | 6.142 | +0.52% | | turbo3 | 3.5† | 4.6x† | 6.176 | +1.06% | | turbo2 | 2.5 | 6.4x | 6.507 | +6.48% |

turbo4 (4-bit PolarQuant) has the best quality after q8_0 — closer to q8_0 than q4_0, at better compression. turbo3 trades quality for maximum compression. turbo2 (2-bit) trades more quality for extreme compression — best used asymmetrically.

†turbo3 at default block_size=32. At block_size=128, turbo3 achieves 3.125 bits/val and 5.12x compression with identical PPL, validated on Metal across 3 model architectures, 3 context lengths (512–32K), and 2 Apple Silicon platforms. Tested on both asymmetric (q8_0-K + turbo3-V) and symmetric (turbo3/turbo3) paths. On the tested M2 Pro setup (Qwen2.5-1.5B, q8_0-K + turbo3-V), block_size=128 also improved decode by 3–7%; this gain was not observed on M5 Max. Earlier turbo3 figures (4.6x) reflect the block_size=32 default. CUDA not yet validated. See block size study.

Important: choosing the right config for your model. TurboQuant quality depends on your base weight quantization. Models with Q8_0+ weights work well with symmetric turbo (e.g., -ctk turbo3 -ctv turbo3). Some low-bit models with Q4_K_M weights may benefit from asymmetric K/V: use -ctk q8_0 -ctv turbo4 to keep K precision high while compressing V (tested on Qwen2.5-7B Q4_K_M). K precision is the dominant quality factor because it controls attention routing via softmax. Note: not all Q4_K_M models are sensitive — Mistral-24B, Llama-70B, and Command-R+ 104B all handle symmetric turbo fine. Bigger models absorb quantization stacking better (104B: +3.6% vs 70B: +11.4% for turbo3). Validate on your specific model. See Configuration Recommendations for the full tested matrix and practical guidance.

Validated on Metal (Apple Silicon). CUDA mixed q8_0 × turbo parity is not yet verified.

Asymmetric K/V (NEW)

TurboQuant supports independent K and V cache types. In current testing, keeping K at q8_0 while compressing V with turbo rescues quality on low-bit models where symmetric turbo degrades:

| Model (weights) | K | V | PPL | vs q8_0 | |-----------------|---|---|------|---------| | Qwen2.5-7B (Q4_K_M) | q8_0 | turbo4 | 6.64 | +1.0% | | Qwen2.5-7B (Q4_K_M) | q8_0 | turbo3 | 6.71 | +2.0% | | Qwen2.5-7B (Q4_K_M) | turbo3 | turbo3 | 3556 | catastrophic |

# Validated starting point for low-bit models
# (tested on Qwen2.5-7B Q4_K_M; not all Q4_K_M models need this)
llama-server -m model-Q4_K_M.gguf -ctk q8_0 -ctv turbo4 -fa 1

Boundary V (Layer-Aware V Compression)

Not all V layers need the same precision. Boundary V protects the first 2 + last 2 layers with q8_0-V while compressing all remaining layers with turbo2-V. 15 lines of code, no speed penalty.

| Model | turbo2 PPL | Boundary V PPL | turbo3 PPL | Quality recovered | |-------|-----------|---------------|-----------|-------------------| | phi-4-Q8_0 (40L) | 4.835 | 4.784 | 4.742 | 55% | | Qwen2.5-7B Q4_K_M (28L) | 6.911 | 6.835 | 6.707 | 37% | | Qwen3.5-35B MoE (64L) | 5.257 | 5.148 | 5.137 | 91% | | Qwen3.5-27B Dense (36L) | 6.534 | 6.423 | 6.273 | 42% |

Validated at 512 and 8K context. NIAH retrieval passed. Benefit scales with model depth (91% on 64-layer MoE). Independently validated by @Corianas_ on NanoGPT.

Enabled by default. Activate manually on older builds with TURBO_LAYER_ADAPTIVE=7 env var.

# Boundary V — boundary layers q8_0-V, rest turbo2-V
llama-server -m model.gguf -ctk q8_0 -ctv turbo2 -fa 1

See full paper.

Prefill Context Scaling (Verified 2K-32K)

| Context | turbo4 tok/s | turbo3 tok/s | q8_0 tok/s | turbo4/q8_0 | turbo3/q8_0 | |---------|-------------|-------------|-----------|------------|------------| | 2K | 2682 | 2708 | 2665 | 1.01x | 1.02x | | 4K | 2370 | 2289 | 2255 | 1.05x | 1.01x | | 8K | 2041 | 2054 | 2002 | 1.02x | 1.03x | | 16K | 1621 | 1698 | 1605 | 1.01x | 1.06x | | 32K | 1141 | 1204 | 1098 | 1.04x | 1.10x |

Prefill: both turbo3 and turbo4 match or exceed q8_0 speed. Compressed cache uses less bandwidth.

Decode Speed — MoE (M5 Max 128GB, Qwen3.5-35B-A3B, Sparse V)

| Config | Short (tg128) | pp32768+tg128 | Short vs q8_0 | |--------|--------------|---------------|--------------| | q8_0 | 85.71 tok/s | 1173.91 tok/s | — | | turbo4 | 79.87 tok/s | 1060.12 tok/s | 0.93x | | turbo3 | 76.84 tok/s | 1141.74 tok/s | 0.90x |

turbo4 decode is faster than turbo3 due to simpler nibble packing and direct-extract dequant.

Real-world server benchmark (70-page PDF, ~24K context):

| Config | Prefill tok/s | Decode tok/s | Decode vs q8_0 | |--------|-------------|-------------|---------------| | q8_0 | 1449.9 | 68.2 | — | | turbo4 | 1405.9 | 63.7 | 0.93x | | turbo3 | 1417.8 | 53.3 | 0.78x |

NIAH Retrieval (turbo4)

| Test | q8_0 | turbo4 | turbo3 + sparse V | |------|------|--------|-------------------| | Single needle (33 positions) | 30/33 (90.9%) | 31/33 (93.9%) | 9/9 (3-pos) |

turbo4 beats q8_0 on retrieval (31/33 vs 30/33). Shared failure at 8K/100% is a model weakness, not quantization. See turbo4 resurrection for the full investigation.

Large Model Stress Tests (M5 Max 128GB)

| Model | Params | Weights | Config | PPL | vs q8_0 | Max Context | NIAH | |-------|--------|---------|--------|-----|---------|-------------|------| | Llama-3.1-70B | 70B | Q4_K_M | turbo4/turbo4 | 3.461 | +6.3% | 48K | 30/30 | | Llama-3.1-70B | 70B | Q4_K_M | turbo3/turbo3 | 3.629 | +11.4% | 48K | 30/30 | | Command-R+ 104B | 104B | Q4_K_M | turbo4/turbo4 | 6.312 | +1.9% | 128K | 10/10 | | Command-R+ 104B | 104B | Q4_K_M | turbo3/turbo3 | 6.415 | +3.6% | 128K | 10/10 |

turbo3 prefill is faster than q8_0 at 32K on both models (70B: 80.8 vs 75.2 t/s, 104B: 64.5 vs 62.3 t/s). Smaller KV cache = less memory bandwidth during attention.

104B at 128K requires raising macOS GPU memory cap: sudo sysctl iogpu.wired_limit_mb=117964 (90% of 128GB). Without this, Metal stalls at ~49K context on 70B+ models. See Getting Started Guide for per-RAM values.

See M5 Max stress test for the full data.

KL Divergence vs f16

| Cache | Mean KLD | Δp RMS | Same top-p % | |-------|----------|--------|-------------| | q8_0 | 0.001549 | 1.23% | 98.43% | | turbo4 | 0.009633 | 2.71% | 95.98% | | q4_0 | 0.008091 | 2.75% | 95.83% | | turbo3 | 0.016145 | 4.09% | 94.31% |

turbo4 KLD is 40% lower than turbo3. Same top-p agreement matches q4_0.

Decode Speed — Dense (M5 Max 128GB, Qwen3.5-27B, Sparse V)

| Test | With sparse V | Without | Delta | |------|-------------|---------|-------| | Short (tg128) | 16.73 | 16.61 | +0.7% | | 8K (pp8192+tg128) | 298.27 | 294.52 | +1.3% | | 16K (pp16384+tg128) | 316.98 | 311.24 | +1.8% |

Dense models see smaller gains (attention is <5% of decode — FFN dominates). No regressions. Safe to enable by default.

Sparse V dequant skips V dequantization for positions where softmax attention weight < 1e-6. At long context, most attention weights are negligible — this saves approximately half the total dequant cost. +22.8% decode at 32K vs turbo3 without sparse V, pushing the ratio from 0.76x to 0.93x of q8_0. Sparse V introduces no additional PPL degradation beyond the underlying compression (validated at 32K with 50 chunks on wikitext-103, CI ±0.021). Benefit scales with context length. This is implemented as a minimal kernel modification.

Sparse V is not TurboQuant-specific: on q8_0 KV cache it yields a +5% decode speedup with identical PPL and NIAH, confirming this is a general attention-aware optimization rather than a compression-specific trick. See the full paper.

On M2/M1 (pre-M5), the auto-detected 4-mag LUT gives an additional +38-45% decode improvement at long context, and is additive with sparse V. See Decode Speed Hardware Analysis for the full 14-approach experiment log, and Context Scaling Deep Dive for the M5 Max optimization journey.

Community Hardware: CUDA (RTX 3090)

Tested by @jaker86 on RTX 3090. Model: Qwen3.5-9B Q4_K_M. Build from signalnine's CUDA fork PR #24.

| Config | K | V | PPL (wikitext-2) | vs q8_0 | Decode t/s | Prefill t/s | |--------|---|---|-----------------|---------|-----------|------------| | q8_0 | q8_0 | q8_0 | 8.2018 | — | 102.69 | 3774 | | turbo3 | turbo3 | turbo3 | 8.3124 | +1.3% | 98.68 | 3707 | | turbo4 | turbo4 | turbo4 | 8.3012 | +1.2% | 95.87 | 3628 | | turbo2 | turbo2 | turbo2 | 8.6639 | +5.6% | 98.05 | 3680 | | mixed | turbo3 | turbo2 | 8.5312 | +4.0% | 97.32 | 3524 | | mixed | turbo2 | turbo3 | 8.4356 | +2.9% | 96.61 | 3608 |

CUDA decode within 4-7% of q8_0 across all configs. Prefill within 4-7%. Mixed K/V configs working correctly after PR #24 fix (prefill was 329 t/s before fix, now 3500+).

Community Hardware: M1 Max 64GB

Tested by @mariotomich. Model: Qwen3.5-35B-A3B Q8_0, Sparse V ON. Real prompt: 38,596 tokens (70-pages.md), llama-cli with Qwen chat template.

| KV | Prefill t/s | Decode t/s | vs q8_0 | |----|------------|-----------|---------| | q8_0 | 399.0 | 12.4 | — | | turbo2 | 406.2 | 10.8 | -12.9% | | turbo3 | 370.4 | 7.7 | -37.9% | | turbo4 | 365.0 | 16.6 | +33.9% |

turbo4 decode beats q8_0 by +33.9% at long context on M1 Max. At 38K tokens, KV bandwidth savings outweigh dequant cost. Sparse V amplifies the gain. turbo3 decode regression (-37.9%) is the known M1 L2 cache wall — turbo3 dequant complexity causes cache eviction on pre-M5 hardware.

Asymmetric q8_0-K + turbo4-V (recommended for pre-M5):

Synthetic (llama-bench):

| KV | pp512 t/s | tg128 t/s | pp65536+tg128 t/s | |----|-----------|-----------|-------------------| | q8_0 | 876.1 | 39.55 | 275.0 | | q8_0-K + turbo4-V | 894.9 (+2.2%) | 38.64 (-2.3%) | 271.0 (-1.5%) |

Asymmetric avoids the turbo3 decode regression (-37.9%) on pre-M5 hardware.

KV cache memory at 262K context:

| KV | Cache MiB | Saved | Compression | |----|-----------|-------|-------------| | q8_0 | 2782 | — | baseline | | turbo4 | 1422 | 1360 MiB | 1.96x | | q8_0-K + turbo4-V | 2102 | 680 MiB | 1.32x |

PPL on real document (70-pages.md, ctx=512, 20 chunks): q8_0 16.29, turbo4 16.44 (+0.93%), turbo3 16.42 (+0.76%), turbo2 17.22 (+5.69%).

Community Hardware: AMD RX 9070 XT (RDNA 4, gfx1201, Windows 11)

First AMD GPU validation. First attempt — no debugging, no analysis, just raw testing out of the box. Qwen2.5-7B Q4_K_M on HIP SDK 7.1. gfx1201 detected natively — no HSA_OVERRIDE_GFX_VERSION needed.

| K | V | PPL (wikitext-2) | vs q8_0 | Prefill t/s | Decode t/s | Status | |---|---|-----------------|---------|-------------|-----------|--------| | q8_0 | q8_0 | 7.794 | baseline | 589.5 | 84.7 | OK | | q8_0 | turbo4 | 7.876 | +1.0% | 588.4 | 86.8 | recommended | | q8_0 | turbo3 | NaN | catastrophic | 605.1 | 87.8 | broken (HIP-specific) | | turbo4 | turbo4 | 401.4 | catastrophic | 556.4 | 84.0 | broken (Q4_K_M) | | turbo3 | turbo3 | 81,277 | catastrophic | 580.3 | 86.0 | broken (Q4_K_M) |

Key findings:

• q8_0-K + turbo4-V confirmed on AMD — +1.0% PPL, no speed penalty, 25% KV memory savings
• Symmetric turbo catastrophic on Q4_K_M, consistent with Metal/CUDA results
• q8_0/turbo3 produces NaN on this model (Metal gets +2.0%) — HIP-specific, under investigation
• Speed flat across configs (~85 t/s decode, ~590 t/s prefill at pp512)
• Context scaling: 0.96-0.99x vs q8_0 at pp2048-8192

See Windows RDNA 4 Setup Guide for build instructions and 9 gotchas.

Speed Optimization Journey

| Optimization | Prefill tok/s | vs q8_0 | |-------------|--------------|---------| | turbo3 fp32 WHT (initial) | 739 | 0.27x | | + fp16 WHT | 1074 | 0.40x | | + half4 vectorized butterfly | 1411 | 0.52x | | + graph-side WHT rotation | 2095 | 0.78x | | + block-32 storage | 2747 | 1.02x | | + optimized dequant | 2524 | 0.98x |

The final number (2524 at 4K) is lower than the peak (2747 at 512) because longer context is naturally slower. The key metric is the ratio vs q8_0, which stays flat at 0.99x. See Speed Experiments for the full journey.

Compression Quality (Python Prototype)

| Config | Compression | Cosine Sim | MSE | |--------|-------------|------------|-----| | TurboQuant 2-bit | 7.1× | 0.79 | 0.0047 | | TurboQuant 2.5-bit (outlier) | 4.9× | 0.86 | 0.0029 | | TurboQuant 3-bit | 4.9× | 0.91 | 0.0018 | | TurboQuant 3.5-bit (outlier) | 3.8× | 0.95 | 0.0009 | | TurboQuant 4-bit | 3.8× | 0.96 | 0.0007 |

Needle-In-A-Haystack (NIAH) Retrieval

Tested using Kamradt and NVIDIA RULER methodology. Qwen3.5-35B-A3B on M5 Max 128GB.

Single Needle Retrieval (with sparse V dequant):

| Test | q8_0 | turbo3 | turbo3 + sparse V | |------|------|--------|-------------------| | Single needle (9 positions) | 7/9 | 7/9 | 9/9 (100%) |

turbo3 + sparse V achieves 9/9 in this setup (vs 7/9 baseline), suggesting a potential denoising effect from removing low-weight quantization noise. Needle positions have meaningful attention weights (well above the 1e-6 threshold) and are never skipped.

Sparse V shows no measurable impact on perplexity across all tested contexts and datasets. Observed improvements in retrieval tasks (e.g., NIAH) are treated as secondary signals and may reflect reduced quantization noise rather than fundamental model quality changes.

Single Needle — Depth (0-100%) x Context Length (pre-sparse-V):

| Depth | 4K | 8K | 16K | 32K | |-------|----|----|-----|-----| | q8_0 | 5/5 | 4/5 | 4/5 | 4/5 | | turbo3 | 5/5 | 4/5 | 5/5 | 3/5 |

Pre-sparse-V aggregate: q8_0 85% (17/20), turbo3 80% (16/20). No systematic degradation from compression. N=10 needles remarkably stable (9-10/10 at every depth).

Multi-Key with 3 Distractors (RULER MK-NIAH):

| Cache Type | 4K | 8K | 16K | 32K | |------------|----|----|-----|-----| | q8_0 | 1/1 | 1/1 | 1/1 | 1/1 | | turbo3 | 1/1 | 1/1 | 1/1 | 1/1 |

100% retrieval accuracy with distractors through 32K. turbo3 correctly ignores distractor needles at all context depths.

Long-Context Perplexity (Primary Quality Metric)

50-chunk wikitext-103 at 32K context (strongest validation, CI ±0.021):

| Config | PPL | vs q8_0 | Sparse V Δ | |--------|-----|---------|------------| | q8_0 (8-bit KV) | 7.0638 | — | — | | q4_0 (4-bit KV) | 7.0857 | +0.31% | — | | turbo3 WITHOUT sparse V (3.5-bit) | 7.1796 | +1.64% | — | | turbo3 WITH sparse V (3.5-bit) | 7.1796 | +1.64% | 0.0000 |

Note: q4_0 is included as a reference baseline. No optimization was applied to q4_0 in this work. Development focused on q8_0 and turbo3 paths.

Key Validation

Real Qwen3-1.7B KV tensor rotation Gaussianization:

Raw kurtosis:       900.4  → After rotation: 2.9  (Gaussian = 3.0)
Std after rotation:  0.088388
Expected (1/√d):     0.088388
Ratio:               1.000 exactly

---

Getting Started

Prerequisites

• Python >= 3.10
• NumPy >= 1.24, SciPy >= 1.10
• cmake + C/C++ compiler (for llama.cpp build)
• Xcode Command Line Tools (macOS Metal build)
• Optional: torch, transformers, accelerate (~4GB download, for real model validation)

Install the Python Prototype

git clone https://github.com/TheTom/turboquant_plus.git
cd turboquant_plus
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Verify — should print "141 passed"
python3 -m pytest tests/ -v

Run the Demo

# Quick compression demo (no model needed)
python3 benchmarks/demo.py

# Validate on real model KV tensors (downloads Qwen3-1.7B, ~4GB)
pip install transformers torch accelerate
python3 benchmarks/validate_real_model.py

Build llama.cpp with TurboQuant

The llama.cpp port adds two new KV cache types: turbo3 (3.25 bits, 4.9× compression) and turbo4 (4.25 bits, 3.8× compression).

# Clone the llama.cpp fork with TurboQuant support
git clone https://github.com/TheTom/llama-cpp-turboquant.git
cd llama-cpp-turboquant
git checkout feature/turboquant-kv-cache

# Build with Metal (Apple Silicon)
cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# Build with CUDA (NVIDIA) — community tested on RTX 3080 Ti/3090/4090/5090
# cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
# cmake --build build -j

# Build with HIP (AMD) — tested on RX 9070 XT (RDNA 4, gfx1201)
# See docs/windows-rdna4-setup.md for Windows gotchas
# cmake -S . -B build -G Ninja -DGPU_TARGETS=gfx1201 -DGGML_HIP=ON \
#   -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_C_COMPILER=clang \
#   -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Release
# cmake --build build --config Release

# Verify turbo types are available
./build/bin/llama-server --help | grep turbo
# Expected output includes: turbo3, turbo4

The fork modifies these files from upstream llama.cpp:

• ggml/include/ggml.h — new type enum entries
• ggml/src/ggml-common.h — block structures
• ggml/src/ggml-quants.h — function declarations
• ggml/src/ggml-turbo-quant.c — C quantize/dequantize (new file)
• ggml/src/ggml.c — type traits registration
• ggml/src/CMakeLists.txt — build config
• ggml/src/ggml-metal/ggml-metal.metal — Metal GPU kernels
• ggml/src/ggml-metal/ggml-metal-device.m — Metal device validation
• common/arg.cpp — CLI arg parsing

Run Inference with TurboQuant KV Cache

# Server mode (for Hermes Agent, Claude Code, OpenCode, etc.)
./build/bin/llama-server \
  -m models/your-model.gguf \
  --alias "model-turbo" \
  --jinja -ngl 99 -c 262144 -fa on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  -np 1 --metrics --host 0.0.0.0 --port 8080

# CLI mode (quick test)
./build/bin/llama-cli \
  -m models/your-model.gguf \
  -ngl 99 -c 2048 -fa on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  -n 100 -p "Hello world" --jinja

Cache Type Reference

| Flag | Bits/val | Compression vs fp16 | Description | |------|----------|--------------------:|-------------| | turbo3 | 3.5† | 4.6x† | 3-bit PolarQuant + WHT rotation. Best compression, q8_0 speed. | | turbo4 | 4.25 | 3.8x | 4-bit PolarQuant (16 centroids). Best quality. | | q8_0 | 8 | 2.0x | llama.cpp default quantized cache. | | q4_0 | 4 | 4.0x | llama.cpp 4-bit cache. |

---

Architecture

Input: KV cache vector x ∈ R^d (one attention head)
    │
    ├── Extract norm: γ = ||x||, x̂ = x/γ
    │
    ├── Random rotation: WHT + random sign flips
    │   coordinates ~ N(0, 1/d) after rotation
    │
    ├── Optimal scalar quantization (Lloyd-Max)
    │   turbo4: 16 centroids (4-bit), turbo3: 8 centroids (3-bit), turbo2: 4 centroids (2-bit)
    │
    └── Output: quantized indices + norm per block
        Compression: 3.8x (turbo4), 5.1x (turbo3), 7.5x (turbo2)

Note on QJL: reference only, not used in production.

The original TurboQuant paper (Zandieh et al. 2024, arXiv 2406.03482) includes a 1-bit QJL error-correction stage. The Python qjl.py here implements it for paper reproducibility.

Production drops QJL on both K and V. TheTom/llama-cpp-turboquant recommended config is --cache-type-k q8_0 --cache-type-v turbo3. No QJL on either side.

QJL eliminates reconstruction bias but amplifies variance, which softmax turns into attention noise. Five independent groups confirmed (buun, scos-lab, Arclabs001, +2). See turbo4-resurrection.md for the full ablation and mechanism.

If you're building on this repo: use TurboQuantMSE (V cache), or implement straight 4 to 8 bit PolarQuant on K. Only enable the QJL / TurboQuant (with QJL) classes if you are reproducing the original paper or doing K-side research below 8-bit. Even then, validate at your target context length. Historical evidence is that QJL noise accumulates past ~16K context.

Project Structure

turboquant/
├── rotation.py        # Walsh-Hadamard Transform + random sign flips
├── codebook.py        # Lloyd-Max optimal centroid computation
├── polar_quant.py     # PolarQuant — norm extraction + WHT rotation + scalar quantization
├── qjl.py            # QJL 1-bit quantizer (paper-faithful reference, see README §QJL). Not used by TheTom/llama-cpp-turboquant in production.
├── turboquant.py      # Full TurboQuant pipeline
├── kv_cache.py        # KV cache integration layer
├── outlier.py         # Outlier channel strategy (2.5-bit, 3.5-bit)
├── lloyd_max.py       # Lloyd-Max quantizer implementation
├── utils.py           # Bit packing, memory measurement
├── isoquant.py        # IsoQuant (quaternion SO(4)) experimental comparison
└── rotorquant.py      # RotorQuant experimental comparison

tests/                 # 14 test files, 500+ tests
benchmarks/
├── demo.py                       # Quick compression demo
├── run_benchmark.py              # Server-based benchmark runner
├── benchmark_results.md          # Full benchmark report
├── benchmark_llama.sh            # llama.cpp benchmark script
├── benchmark_norm_correction.py  # Norm correction validation
├── benchmark_ppl_tq_vs_rq.py    # TurboQuant vs RotorQuant PPL comparison
├── temporal_decay_prototype.py   # Temporal decay experiment
├── test_with_llama.py            # Integration test at Qwen 3.5 dimensions
├── test_outlier_comparison.py    # Outlier strategy comparison
└── validate_real_model.py        # Real model KV tensor validation

docs/
├── turboquant-recommendations.md # Configuration guide (tested matrix)
├── windows-rdna4-setup.md        # Windows + AMD RDNA 4 build guide
├── papers/
│   ├── turbo4-resurrection.md    # turbo4 bug hunt (PPL 679 → 6.125)
│   ├── sparse-v-dequant.md       # Sparse V attention-gated optimization
│   ├── layer-aware-v-compression.md  # Boundary V (layer-adaptive V precision)
│   └── block-size-experiment.md  # Block size 32→128 (12% compression win)
└── (25+ engineering docs, investigations, experiment logs)

Roadmap

| Phase | Status | Details | |-------|--------|---------| | Core algorithms (NumPy) | ✅ | 500+ tests across 14 test files | | Distortion validation | ✅ | Matches paper bounds (Table 2) | | Real model validation | ✅ | Rotation validated on Qwen3 KV tensors (kurtosis 900→2.9) | | llama.cpp C port | ✅ | Metal GPU inference working on M1 through M5 | | Metal shader optimization | ✅ | q8_0 speed parity: prefill matches or beats q8_0 | | CUDA backend | ✅ | Community-tested on RTX 3080 Ti/3090/4090/5090, DGX Spark Blackwell | | HIP/AMD backend | ✅ | RX 9070 XT (RDNA 4) validated, gfx1201 native | | Asymmetric K/V | ✅ | q8_0-K + turbo-V rescues Q4_K_M models | | Boundary V | ✅ | Layer-aware V compression, 37-91% quality recovery | | Sparse V | ✅ | Attention-gated dequant skip, +22.8% decode on MoE. Upstream PR #21119 | | Block size optimization | ✅ | 32→128, 12% better compression, zero quality cost | | Upstream coordination | 🔄 | llama.cpp PR preparation (#27) | | TurboQuant+ extensions | ⏳ | Adaptive bits, temporal decay, MoE-aware compression | | MLX Swift port | 🔄 | Active collaboration with @ekryski on mlx-swift-lm — turbo4v2 working, Gemma4 fixes in progress |

Paper Reference

• TurboQuant: arXiv 2504.19874 (ICLR 2026)
• PolarQuant: arXiv 2502.02617 (AISTATS 2026)
• QJL: arXiv 2406.03482
• Google Research Blog: TurboQuant: Redefining AI Efficiency

Engineering Docs

Detailed debugging logs, gotchas, and benchmarks from the llama.cpp port:

• Quality Benchmarks — perplexity validation, bisection log, top-of-tree quality+speed table
• Speed Investigation — Metal gotchas, fp16 WHT results, optimization history
• Speed Experiments — the full 739 → 2747 tok/s optimization journey (5 experiments)
• Context Scaling Deep Dive — why turbo3 degraded at long context, how we fixed it (every failed approach documented)
• Pre-Rotate-Queries Investigation — why graph-side WHT failed initially, how we fixed it
• Quality + Speed Gate — pre-push script checking PPL AND context scaling ratio (required before merge)

MLX Framework Port (Experimental)

TurboQuant KV cache compression is being ported to Apple's MLX framework for native Python/Swift inference on Apple Silicon.

Fork: TheTom/mlx feature/turboquant-plus

Results (M5 Max)

Qwen2.5-3B 4bit — delegated KVCache (5-run avg, 500 decode tokens, 7ad7500):

| Config | Decode tok/s | vs Baseline | PPL | PPL Delta | |--------|-------------|-------------|-----|-----------| | Baseline (f16 KV) | 172.6 | 100% | 1.8764 | — | | Sym turbo4 | 171.2 | 99.2% | 1.9083 | +1.70% | | Asym (K=FP16, V=turbo4) | 171.0 | 99.0% | 1.8859 | +0.51% |

Quality: Output text indistinguishable from baseline. KL divergence < 0.001, cosine similarity > 0.989.

35B MoE (Qwen3.5-35B-A3B 8bit):

| Config | Prefill | Decode | vs Baseline | |--------|---------|--------|-------------| | Baseline | 11.4 | 95.7 | 100% | | turbo4 fused + boundary | 132.7 | 94.2 | 96% |

Qwen3.5-27B Dense 8bit (16/64 KV layers):

| Config | PPL | PPL Delta | Decode | vs Baseline | |--------|-----|-----------|--------|-------------| | Baseline | 1.4800 | — | 17.9 | 100% | | turbo4 asymmetric | 1.5082 | +1.91% | 15.5 | 87% | | turbo4 symmetric | 1.5219 | +2.83% | 15.4 | 86% |

Quality Validation (Qwen2.5-7B 8bit, dense, all 28 layers KV):

| Test | Symmetric turbo4 | Asymmetric (K=FP16) | |------|-----------------|---------------------| | KLD | 6.86 (broken) | 0.003 | | Top-1 match | 10.5% (broken) | 98.1% | | NIAH | 0/15 FAIL | 15/15 PASS |

[!warning] Symmetric turbo is catastrophic on dense models. All K layers compressed → softmax error compounds across 28 layers. Asymmetric (K=FP16, V=turbo4) is mandatory for dense architectures. Hybrid models (Qwen3.5) with delta net layers are accidentally safe because only a fraction of layers use KV cache.

Dense models (short context, deferred compression):

| Model | Baseline Decode | turbo4 asym Decode | PPL Delta | |-------|----------------|-------------------|-----------| | Qwen2.5-7B 8bit | 64.2 | 64.1 | 0.00% | | phi-4 8bit | 32.9 | 32.7 | 0.00% |

M2 Pro — Qwen2.5-1.5B 8bit (dense, 28/28 KV layers, asymmetric):

| Test | Result | |------|--------| | KLD | 0.004 | | Top-1 match | 96.8% | | NIAH | 30/30 PASS |

| Context | Baseline Decode | Turbo Asymmetric | vs Baseline | |---------|----------------|-----------------|-------------| | 128 | 34.8 | 35.2 | 101% | | 4096 | 46.9 | 21.6 | 46% |

M2 Pro shows more decode regression at long context — lower memory bandwidth amplifies turbo overhead.

M5 Max Context Scaling (Qwen2.5-7B 8bit, delegated KVCache, 7ad7500):

| Context | Baseline | Sym turbo4 | vs Baseline | Asym (K=FP16) | vs Baseline | |---------|----------|-----------|-------------|---------------|-------------| | 512 | 63.6 | 63.6 | 100% | 64.0 | 101% | | 1K | 63.1 | 62.8 | 100% | 62.6 | 99% | | 2K | 62.7 | 61.8 | 98% | 62.2 | 99% | | 4K | 61.0 | 60.2 | 99% | 61.0 | 100% | | 8K | 58.2 | 56.9 | 98% | 57.7 | 99% | | 16K | 54.6 | 53.0 | 97% | 53.8 | 99% |

Previous numbers (61-83%) were measured before the delegated KVCache optimization (7ad7500). Root cause was mx.concatenate allocating new arrays every decode step × n_layers. Fixed by delegating FP16 storage to an internal KVCache with pre-allocated buffers.

MLX Python vs llama.cpp (Qwen2.5-7B, M5 Max):

| Framework | Prefill (400 tok) | Decode | Memory | |-----------|------------------|--------|--------| | llama.cpp (Q8_0) | 387 | 20.9 | 7.5 GB | | MLX (8bit) | 243 | 21.2 | 8.5 GB |

MLX decode matches llama.cpp. Prefill 37% slower (lazy graph vs pre-compiled).

MLX Python vs llama.cpp (M2 Pro, Qwen2.5-7B):

| Framework | Prefill (400 tok) | Decode | |-----------|------------------|--------| | llama.cpp | 387 | 20.9 | | MLX | 243 | 21.3 |

Note: Future benchmark logs should record Apple Silicon power mode (Low / Auto / High) when known, as it can materially affect throughput.

Quick Start (MLX Python)

import mlx_lm
from mlx.nn.layers.turbo_kv_cache import TurboKVCache

model, tokenizer = mlx_lm.load("mlx-community/Qwen2.5-7B-Instruct-8bit")
n_layers = len(model.model.layers)
cache = [TurboKVCache(bits=4, key_bits=4) for _ in range(n_layers)]
text = mlx_lm.generate(model, tokenizer, prompt="Hello!",
                        max_tokens=200, prompt_cache=cache, verbose=True)

pip install git+https://github.com/TheTom/mlx.git@feature/turboquant-plus
pip install mlx-lm

How it works

TurboKVCache is a drop-in replacement for mlx-lm's KVCache that adds TurboQuant 4-bit K+V compression. Compatible with mlx-lm and mlx-vlm — no framework changes needed.

Delegated KVCache architecture (7ad7500): During prefill, stores raw FP16. On first decode step, compresses to packed TurboQuant storage and seeds an internal KVCache with decoded FP16. Subsequent decode tokens go through the native KVCache (pre-allocated buffers, zero-alloc slice-assign). Packed storage updated in background via periodic batch recompression on CPU stream.

• 97–100% baseline decode speed across 512–16K context (Qwen2.5-7B, M5 Max)
• +0.51% PPL (asymmetric), +1.70% PPL (symmetric)
• 99% answer agreement with baseline (520 multimodal samples)
• Works with stock mlx-lm and mlx-vlm, no fork needed
• All TurboQuant+ papers applied (beta centroids, dual SRHT signs, boundary layers)

Quick Start — mlx-vlm (multimodal)

from mlx_vlm import load
from mlx_vlm.models.cache import make_prompt_cache
from mlx_lm.models.cache import KVCache
from mlx.nn.layers.turbo_kv_cache import TurboKVCacheLite, compact_turbo_cache

model, processor = load("mlx-community/gemma-4-26b-a4b-it-bf16")

# Wrap KV layers with TurboKVCacheLite
cache = make_prompt_cache(model.language_model)
kv_indices = [i for i, c in enumerate(cache) if isinstance(c, KVCache)]
for idx in kv_indices:
    cache[idx] = TurboKVCacheLite(cache[idx], bits=4, key_bits=4)

# Generate as normal — prefill stores FP16
from mlx_vlm import generate
generate(model, processor, prompt="...", max_tokens=1, prompt_cache=cache)

# Compact: compress K+V to 4-bit TurboQuant
compact_turbo_cache(cache)

# Continue generating — native SDPA at full speed
generate(model, processor, prompt="Continue.", max_tokens=200, prompt_cache=cache)

pip install git+https://github.com/TheTom/mlx.git@feature/turboquant-plus
pip install mlx-vlm

Quick Start — mlx-lm (text)

import mlx_lm
from mlx.nn.layers.turbo_kv_cache import make_turbo_cache, compact_turbo_cache

model, tokenizer = mlx_lm.load("mlx-community/Qwen2.5-7B-Instruct-8bit")
cache = make_turbo_cache(model, bits=4)
mlx_lm.generate(model, tokenizer, prompt="Hello!", max_tokens=1, prompt_cache=cache)
compact_turbo_cache(cache)
mlx_lm.generate(model, tokenizer, prompt="Continue.", max_tokens=200,
                 prompt_cache=cache, verbose=True)

MM-NIAH Multimodal Benchmark (520 samples)

gemma-4-26b-a4b-it · BF16 · 4-bit TQ+ Compact · MM-NIAH (val) · M5 Max 128GB

| Bucket | BL Acc | TQ+ Acc | Agree | BL Decode | TQ+ Decode | Speedup | BL KV | TQ+ KV | KV saved | |--------|--------|---------|-------|-----------|-----------|---------|-------|--------|----------| | ~1K | 85% | 84% | 99% | 55.1 | 54.7 | 0.99x | 0.21G | 0.19G | 10% | | ~3K | 81% | 79% | 99% | 55.2 | 54.1 | 0.98x | 0.27G | 0.21G | 22% | | ~7K | 80% | 81% | 99% | 54.1 | 51.7 | 0.96x | 0.37G | 0.24G | 35% | | ~15K | 76% | 76% | 100% | 52.0 | 47.8 | 0.92x | 0.53G | 0.28G | 47% | | ~30K | 77% | 75% | 98% | 46.9 | 40.2 | 0.86x | 0.87G | 0.36G | 59% | | ~60K | 75% | 76% | 99% | 42.6 | 33.7 | 0.79x | 1.30G | 0.47G | 64% | | Total | 79% | 78% | 99% | 51.1 | 47.2 | 0.92x | 0.58G | 0.29G | 50% |

99% answer agreement with baseline across all context lengths — zero systematic quality degradation. KV savings of 10–64% where TQ+ is active. Decode speedup scales from 0.99x at ~1K to 0.79x at ~60K (dequant-once overhead on longer prefills).

---

Contributing

Issues and PRs welcome. The main areas where help is needed:

1. Upstream PR — prepare llama.cpp contribution (CONTRIBUTING.md requirements)
2. CUDA kernel optimization — fused FA kernels, decode speed parity
3. MLX memory recovery — implement FP16 KV drop + compressed-only attention for memory-constrained long context
4. Quality metrics — multi-run statistics, additional task benchmarks (GSM8K, code gen, reasoning)
5. Long context validation — 64K+ testing across architectures

Support

If you find this work useful, you can support it via GitHub Sponsors or BTC:

BTC: bc1qsfaaf6mkz2yxx2vavg2n0zgsf3qj25uh94t83rwuq7de67dey05sc3tgjx

Commercial support: For inference optimization and KV cache tuning engagements, DM @no_stp_on_snek on X.

License

Apache License 2.0 — see LICENSE.

Copyright 2026 Tom Turney.

Based on Google Research's TurboQuant paper (arXiv 2504.19874).