Google's Titans Architecture: Neural Long-Term Memory Achieves 2M+ Token Context with O(n) Complexity
By
washedup
A respectable bake. You'd come back tomorrow for another.
Summary
Google's Titans architecture introduces neural long-term memory that learns during inference with 'surprise-based' updates, achieving 2M+ token context windows with O(n) complexity instead of O(n²). Benchmarks show 98.8% needle-in-haystack accuracy compared to Mamba-2's 31%, outperforming GPT-4 on BABILong. However, there's skepticism due to lack of official code, ambiguous implementation details, and a follow-up paper finding that chunking degrades performance.
Key quotes
· 5 pulledGoogle's Titans introduces neural long-term memory that learns during inference via 'surprise-based' updates — 2M+ token context with O(n) complexity instead of O(n²)
Benchmarks show 98.8% needle-in-haystack accuracy vs Mamba-2's 31%
But no official code exists, implementation details are ambiguous, and a follow-up paper found chunking degrades performance
Google Titans learns to memorize at test time with 2M+ token context
Impressive innovation, but wait for independent reproduction
You might also wanna read
Parametric Memory Law: A Quantitative Framework for Understanding LoRA Memory Capacity in LLMs
This research paper introduces the Parametric Memory Law, a quantitative framework for understanding how Low-Rank Adaptation (LoRA) enables
arxiv.org·1d ago
Bridge-Garden Theory Explains Why Mixing Hard and Soft Labels Improves Knowledge Distillation for LLMs
This research paper investigates knowledge distillation (KD) for language models, specifically why mixing hard labels (sampled tokens) and s
arxiv.org·4d ago
Researchers Develop Method to Predict Real-Time Progress in Reasoning Language Models
This research paper investigates whether real-time progress prediction is feasible for reasoning language models that use long latent chains
arxiv.org·4d ago
AI systems achieve 50% pass rate in standard three-party Turing test, study finds
This paper demonstrates that three current AI systems (when suitably prompted) achieve a pass rate of at least 50% in a standard three-party
pnas.org·4d ago
RICP: A Teacher-Student Framework for Retrieved In-Context Principles from Mistakes in LLMs
This paper introduces Retrieved In-Context Principles (RICP), a novel teacher-student framework for improving Large Language Models (LLMs) t
arxiv.org·5d ago
HSIR: New Method Improves Self-Improvement Training for Large Reasoning Models
This research paper identifies two key problems in self-improvement training for Large Reasoning Models (LRMs): data imbalance (too many sim
arxiv.org·5d ago