DeepSeek-V4: Hybrid Sparse-Attention Architecture Enables Efficient Million-Token Context Inference
By
Andrew Lukyanenko
The bagel they save for the regulars. Don't skim, savour.
Summary
DeepSeek-V4 introduces a hybrid sparse-attention architecture combined with on-policy distillation across domain specialists, enabling 1M-token inference at frontier quality while significantly reducing FLOPs and KV cache compared to its predecessor. The paper argues that long context windows are only useful if the model can efficiently attend over the context during inference, tool use, and reasoning trajectories, shifting focus from simply expanding context windows to optimizing attention mechanisms.
Key quotes
· 3 pulledDeepSeek V4 pairs a hybrid sparse-attention stack with on-policy distillation across domain specialists to bring 1M-token inference to frontier quality at a fraction of the FLOPs and KV cache of its predecessor.
a long context window is only useful if the model can actually afford to attend over it during inference, tool use, and long reasoning trajectories
DeepSeek-V4 changes the focus
You might also wanna read
DeepSeek-V4 Series Preview: Million-Token Context MoE Models with 1.6T Parameters
DeepSeek introduces the V4 series of Mixture-of-Experts (MoE) language models, including DeepSeek-V4-Pro (1.6T parameters, 49B activated) an
DeepSeek-V3.1: Open-Source Language Model with Hybrid Inference for Advanced Reasoning and Coding
DeepSeek-V3.1 is an open-source large language model that introduces hybrid inference with both 'Think' and 'Non-Think' modes, optimized for
Product Hunt·9mo ago
DeepSeek-V3.1 Released with Hybrid Inference and Enhanced Agent Capabilities
DeepSeek has released DeepSeek-V3.1, featuring hybrid inference with both 'Think' and 'Non-Think' modes in a single model. The new version o
api-docs.deepseek.com·9mo ago
How New Open-Weight LLMs Are Reducing Long-Context Costs: KV Sharing, Attention Budgeting, and Compressed Attention
The article analyzes recent developments in open-weight LLM architectures, focusing on how newer models like Gemma 4 and DeepSeek V4 are imp
magazine.sebastianraschka.com·12d ago
DeepSeek Releases V4 Preview with Open-Source Models Featuring 1M Context Length
DeepSeek has officially released and open-sourced the preview version of DeepSeek-V4, featuring two models: DeepSeek-V4-Pro (1.6T total / 49
api-docs.deepseek.com·1mo ago
DeepSeek's mHC Architecture: Transforming Transformer Design with Multiple Residual Streams
The article discusses DeepSeek's novel mHC (multi-head connection) architecture that fundamentally changes transformer design by introducing
taylorkolasinski.com·4mo ago