Transformer LMs encode some relational knowledge linearly, but not all
Much of the knowledge encoded in transformer language models (LMs) may be expressed in terms of relations: relations between words and their synonyms, entities and their attributes, etc. We show…
Read the full articleYou might also wanna read
Research Proves Transformer Language Models Are Injective and Invertible
Transformer components such as non-linear activations and normalization are inherently non-injective, suggesting that different inputs could
arxiv.org·8mo ago
Understanding Linear Representations and Superposition in Large Language Model Interpretability
As LLMs become larger, more capable, and more ubiquitous, the field of mechanistic interpretability -- that is, understanding the inner wor.
ternarysearch.blogspot.com·5mo ago
How Transformers Process Information: A Deep Dive into Complexity
Exploring how transformers handle information reveals complexities in compositionality. Our analysis shows surprising gaps in their zero-sho
Study Finds Single Transformer Layer Can Match Full-Parameter RL Training in LLMs
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how
arxiv.org·15d agoStudy Finds Single Transformer Layer Can Match Full-Parameter RL Training in LLMs
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how
arxiv.org·15d agoAnalyzing Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper pres
arxiv.org·1y ago
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
arXiv:2607.08393v1 Announce Type: cross Abstract: Fine-tuning LLMs to inject new knowledge faces a critical challenge: LLMs can quickly memo
Comments
Sign in to join the conversation.
No comments yet. Be the first.