Google Research has announced the launch of a new and innovative compression algorithm called “TurboQuant,” which represents a qualitative leap in large language model acceleration technologies. This algorithm is characterized by its superior ability to reduce the storage space consumed by the key-value cache—technically known in the field of language models—by at least 6 times. More importantly, this massive compression is accompanied by an up to 8-fold acceleration in attention calculation operations, all achieved without the need to retrain the base model or perform any fine-tuning, making it a tool ready for immediate deployment.
Overcoming the bottleneck in language model memory
With the evolution of large language models and their increasing capacity to process longer texts and more complex information, the problem of memory consumption has emerged as a major obstacle for developers. These models rely on storing conversation context and past information in the key-value cache, and as the volume of input data increases, this cache swells massively, causing inference and text generation operations to slow down. The “TurboQuant” algorithm provides a radical solution to this problem, having been designed specifically to handle high memory requirements, opening up new horizons for artificial intelligence applications that require fast and efficient processing in actual production environments.
A dual approach inspired by Shannon’s theory
The credit for developing this revolutionary algorithm goes to a distinguished team of researchers including Amir Zandieh, Majid Daliri, Majid Hadian, and Wahab Mirrokni. The team built the algorithm based on a sophisticated two-stage approach derived from Claude Shannon’s source coding theory. The first stage, which researchers dubbed “Polar Quant,” focuses on randomly rotating the input vectors and then rewriting coordinate pairs as length and angle. This method produces a compact representation of the data that avoids reliance on the exact constants for each block, which typically lead to bloated storage sizes in traditional quantization operations.
The second stage of the algorithm applies a mathematical transformation known as “Quantized Johnson-Lindenstrauss Transform” with a 1-bit system on the residual error from the first stage. This precise step corrects the hidden bias in the inner product, which would otherwise degrade the accuracy of attention scores in language models if left unaddressed. A technical summary of the project aptly encapsulated this process: “The first stage stores the core shape of the memory, while the second stage stores a tiny, almost free correction note.”
Exceptional performance and full compatibility
Experiments have proven that combining both stages compresses the cache to approximately 3 bits per channel only. The research report noted achieving “absolute quality neutrality” at the 3.5-bit level, while showing only very minor, negligible drops at the 2.5-bit level. During laboratory tests conducted using powerful Nvidia H100 graphics processing units, the 4-bit version of the algorithm achieved an 8-fold speedup in attention score calculations compared to the standard uncompressed 32-bit version.
The efficiency of the algorithm was verified using rigorous long-text evaluation benchmarks such as “LongBench” and “ZeroScrolls,” and it was successfully applied to advanced models like “Gemma” and “Mistral.” The benefits of this technology are not limited to inference in language models alone; it also outperformed current quantization techniques in nearest-neighbor search tasks, improving retrieval rates and reducing indexing time to near zero.
Readiness for work and looking to the future
Because this algorithm is data-agnostic and requires no custom tuning for specific datasets, it can be seamlessly inserted as a ready-made layer within existing models. Google Research confirmed in its announcement that this method is ideal for inference workloads in live production environments, as well as large-scale semantic search tasks—areas where cache traffic has become the primary bottleneck.
It is worth noting that the research paper on this innovation was first published on arXiv in April 2025, and its technical details are scheduled to be officially presented and explored in depth during the International Conference on Learning Representations in 2026, underlining its importance and standing in shaping the future of AI infrastructure.
Frequently Asked Questions
What is the TurboQuant algorithm launched by Google?
It is an innovative, training-free compression algorithm designed to reduce memory consumption in large language models by sixfold and accelerate attention calculations eightfold without the need for fine-tuning the base models.
How does this algorithm work technically?
It relies on a two-stage approach inspired by Shannon’s theory; the first stage converts data into length and angle to reduce storage space, and the second stage uses a 1-bit mathematical transformation to correct any potential errors in model calculations.
Does the new algorithm affect the quality of language models?
Tests have proven that the algorithm maintains absolute quality neutrality at the 3.5-bit level, with an extremely minor and negligible drop when compression is increased to 2.5-bit, meaning it preserves the model’s efficiency almost entirely.
What hardware and models was the algorithm tested on?
It was tested on Nvidia H100 graphics processing units, and its efficiency was verified using prominent language models like Gemma and Mistral within benchmarks tailored for long texts.
