DeepSeek-V4.1-Flash Released with 1M Context & FP4 KV Cache
DeepSeek AI has released DeepSeek-V4.1-Flash featuring a 1-million token context window, FP4 KV cache compression, and open-weight availability.
Stock photo for illustration only, not from the actual event
- DeepSeek-V4.1-Flash supports up to a 1-million token context window.
- Utilizes FP4 KV Cache to nearly halve memory storage requirements.
- Features a 40-layer backbone split into 20 encoder and 20 decoder layers.
- Available as open weights under an MIT license on Hugging Face.
DeepSeek AI has officially released its latest artificial intelligence model, DeepSeek-V4.1-Flash, engineered to support an expansive context window of up to 1 million tokens while optimizing inference efficiency for developers.
The model is released under an MIT license with open weights, providing integration paths via vLLM, SGLang, and Transformers on Hugging Face. Additionally, a public API has been introduced featuring low, high, and max reasoning tiers to accommodate various operational demands.
Under the hood, the model utilizes a 40-layer backbone split evenly into a 20-layer causal encoder and a 20-layer decoder. Inspired by YOCO architecture, the decoder derives projection weights from the final encoder hidden state rather than computing its own global KV, nearly halving prefill compute overhead.
The research team also implemented Decoder SWA Bounded Replay to handle sliding-window attention (SWA) with a 128-token window, allowing the system to rebuild decoder SWA states by replaying only the final 128 tokens. Furthermore, the main KV cache is quantized to FP4, nearly cutting storage size in half compared to the FP8 cache of previous iterations.
The introduction of DeepSeek-V4.1-Flash highlights ongoing engineering efforts to tackle the severe memory bottlenecks associated with processing massive context windows in large language models. By combining aggressive cache compression techniques like FP4 quantization with decoupled encoder-decoder structures, the model achieves high efficiency without sacrificing foundational world knowledge or coding capabilities.
Pre-training encompassed 45 trillion multimodal tokens with a 7:1 text-to-multimodal ratio. Sparse attention was trained from scratch at a 64K sequence length before extending the context to 1 million tokens. The base model matches DeepSeek-V4-Pro-Base performance while utilizing only one-third of total parameters and one-fourth of activated parameters.
Source: MarkTechPost
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment