Revolutionizing Self-Attention: DeltaNet and Kimi Delta Lead the Charge
A deep dive into the performance and efficiency of self-attention models, highlighting the strengths of DeltaNet and Kimi Delta Attention. These architectures show promise in tackling the quadratic…
Read the full articleYou might also wanna read
New Method Enables Constant-Cost Self-Attention Computation for Transformers
The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attenti
arxiv.org·5mo ago
Kimi Linear: Hybrid Linear Attention Architecture for Efficient AI Models
Contribute to MoonshotAI/Kimi-Linear development by creating an account on GitHub.
github.com·8mo agoTauformer: A Topological Transformer Architecture Using Laplacian-Derived Scalar Attention
Tauformer is a topological transformer (see paper) that replaces dot‑product attention with a Laplacian-derived scalar (taumode) per token/h
δ-mem: A Compact Online Memory Mechanism for Efficient Long-Context LLM Processing
Large language models increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply exp
arxiv.org·1mo ago
Why LLMs Are So Expensive: The Quadratic Cost of Dense Attention — and How Subquadratic Claims to Fix It
Dense attention's quadratic compute scaling has been the hidden cost driver behind enterprise AI since 2017. Subquadratic's SubQ model posts
shashi.co·11d ago
How New Open-Weight LLMs Are Reducing Long-Context Costs: KV Sharing, Attention Budgeting, and Compressed Attention
From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs
magazine.sebastianraschka.com·1mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.