TurboQuant+ squeezed LLM KV caches 3.8-6.4x and got merged into vLLM, llama.cpp, MLX

In early 2026 the memory cost of KV caches was the wall for long-context LLM inference, so Tom Turney created TheTom/turboquant_plus on 2026-03-25 as an implementation of TurboQuant, Google Research's ICLR 2026 KV-cache compression scheme (arXiv 2504.19874): extract each vector's norm, rotate the residual with a Walsh-Hadamard transform, then quantize to 2-4 bits per value. He targeted 3.8-6.4x cache compression at near q8_0 prefill speed and ~0.9x decode throughput on long context, validated end-to-end from 1.5B to 104B models at 128K context on a MacBook.

The bet was distribution, not product: rather than build a competing engine, the repo was a research home and reference implementation whose win condition was upstream adoption. Experiments found value-cache compression nearly free, key precision dominant for quality, and boundary layers disproportionately sensitive, yielding asymmetric q8_0-key/turbo-value configs. The project also dropped the paper's QJL error-correction stage after five independent groups confirmed it amplified attention noise.

By September 2026 the strategy had landed. vLLM merged a TurboQuant KV-cache backend in April 2026 (PR #38479, turboquant_k8v4 and friends), llama.cpp merged Hadamard KV-cache rotation (PR #21038) with WHT kernels on CPU, CUDA and Vulkan, and Apple merged the asymmetric scheme family into mlx-swift-lm on 2026-07-20 (PR #232). The repo reached 7,018 stars and 927 forks; community ports and a production llama.cpp fork orbit it, funded by GitHub Sponsors and commercial support rather than venture capital.