Grouped Value Attention cuts KV cache by ~45%
Grouped Value Attention and DeepSeek-V4.1-Flash slash transformer KV memory by over 45% to 890 bytes per token while maintaining benchmark accuracy.
Stock photo for illustration only, not from the actual event
- Grouped Value Attention cuts KV cache by around 45% without losing benchmark accuracy
- Achieves 44.35 average score on a 350M parameter model, close to baseline GQA
- DeepSeek-V4.1-Flash shrinks global HBM footprint to just 890 bytes per token
Artificial intelligence architectures are seeing a major memory efficiency boost with a new technique known as Grouped Value Attention, which slashes transformer KV memory by roughly 45 percent without hurting benchmark scores. By storing only grouped values and dynamically reconstructing keys on the fly during decoding, the method bypasses the need to materialize a full key matrix and drastically cuts persistent cache scalars.
Previously, long-context decoding relied heavily on full key-value caches that scaled linearly with sequence length. Even widely used approaches like grouped-query attention still required maintaining a key for every single token, making memory pressure the primary bottleneck during autoregressive inference.
Benchmarking on a 350-million-parameter model demonstrated that GVA reduces persistent cache scalars by about 46 percent while keeping task accuracy within 0.01 points of standard GQA. The 16-dimensional variant reached an average accuracy of 44.35 across five tasks, compared to 44.36 for GQA and 43.88 for MLA.
"this representation reduces persistent cache scalars by approximately 45–47 % relative to matched GQA"
Study Authors
Furthermore, DeepSeek-V4.1-Flash advances KV compression even further by shrinking the global HBM footprint to 890 bytes per token—roughly one quarter of its predecessor. Utilizing a dedicated optimization called SWA Bounded Replay, it further reduces the persistent KV cache footprint to approximately one-eighth of DeepSeek-V4.1-Flash while delivering superior performance over the baseline.
KV cache allocation has long been the primary limiting factor for scaling context lengths in decoder-only language models. Reducing memory overhead by nearly half without sacrificing output quality opens up significant possibilities for running long-context models on less expensive or more modest consumer hardware.
Despite these promising figures, both research papers leave open how these memory savings translate into real-world latency reductions on commodity GPUs. Because GVA's custom decoding kernels remain under evaluation, concrete throughput gains in production environments are yet to be fully determined.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment