How Transformers Process Information: A Deep Dive into Complexity
Exploring how transformers handle information reveals complexities in compositionality. Our analysis shows surprising gaps in their zero-shot performance.
Read the full articleYou might also wanna read
Analyzing Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper pres
arxiv.org·1y ago
Understanding Transformer Circuits: A Mechanistic Interpretability Perspective
A mental model for addressing the residual stream
connorjdavis.com·3mo ago
Transformer LMs encode some relational knowledge linearly, but not all
Much of the knowledge encoded in transformer language models (LMs) may be expressed in terms of relations: relations between words and their
arxiv.org·3d ago
Research Proves Transformer Language Models Are Injective and Invertible
Transformer components such as non-linear activations and normalization are inherently non-injective, suggesting that different inputs could
arxiv.org·8mo ago
New Method Enables Constant-Cost Self-Attention Computation for Transformers
The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attenti
arxiv.org·5mo ago
Examining the Limitations of Transformer Models and the Gap to Human-Level AI
February 14, 2026
dlants.me·4mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.