Study Reveals Convergent Evolution in How Language Models Learn Number Representations
Language models trained on natural text learn to represent numbers using periodic features with dominant periods at $T=2, 5, 10$. In this paper, we identify a two-tiered hierarchy of these features…
Read the full articleYou might also wanna read
Theoretical Analysis Reveals Why Linear RNNs Are More Parallelizable Than Nonlinear RNNs
The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. W
arxiv.org·1mo ago
Neural Collapse Is Forbidden: Information Floors in Language Models
arXiv:2607.09487v1 Announce Type: new Abstract: Within-class variance in language-model representations is commonly read as incomplete neura
Study Finds Larger Language Models Delay But Don't Prevent Plasticity Loss During Training
The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental
arxiv.org·21d agoHow are linear representations learned? Exact solutions to the dynamics of abstraction
arXiv:2607.08843v1 Announce Type: new Abstract: In artificial and biological neural networks, concepts are often encoded as consistent linea
Breaking the Temporal Barrier: The Future of Point-in-Time Language Models
A new approach in training large language models narrows the performance gap while maintaining temporal validity. But will it hold up under
Emergent Hebbian Dynamics in Regularized Learning: A Theoretical Analysis
Hebbian and anti-Hebbian plasticity are widely observed in the brain and are classically modeled as mechanistic, local homosynaptic rules st
arxiv.org·1mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.