Looped Transformers Require Stronger Residual Scaling: 1/N Outperforms 1/√N for Weight-Tied Architectures
Looped (weight-tied) Transformers apply a shared residual block $N$ times ($h \leftarrow h + \varepsilon\,f(h)$, same $f$ at each step), increasing effective depth without adding parameters. Prior…
Read the full articleYou might also wanna read
Systematic Study Shows Transformers Can Drop One or More QKV Projections Without Quality Loss
Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a ce
arxiv.org·1mo agoSystematic Study Shows Transformers Can Drop One or More QKV Projections Without Quality Loss
Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a ce
arxiv.org·1mo agoParcae: Doing more with fewer parameters using stable looped models
Parcae is a stable looped language model that matches the quality of a Transformer twice its size — a 770M model reaching 1.3B-level perform
Theoretical Analysis Reveals Why Linear RNNs Are More Parallelizable Than Nonlinear RNNs
The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. W
arxiv.org·1mo ago
Study Finds Single Transformer Layer Can Match Full-Parameter RL Training in LLMs
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how
arxiv.org·14d agoStudy Finds Single Transformer Layer Can Match Full-Parameter RL Training in LLMs
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how
arxiv.org·14d ago
Binary normalized neural networks achieve 32x memory reduction with single-bit parameters while maintaining performance
The increasing size of large neural network models, specifically language models and foundational image models, poses deployment challenges,
arxiv.org·9mo ago
Transformers Can Learn to Predict Permuted Congruential Generator Sequences Through Curriculum Learning and Scaling Laws
We study the ability of Transformer models to learn sequences generated by Permuted Congruential Generators (PCGs), a widely used family of
arXiv.org·2mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.