Spectral Preprocessing: The Secret Sauce for Better Attention in Language Models
Spectral preprocessing using FFT is redefining transformer attention in character-level language models. Key improvements are seen with fixed random spectral filters and learned frequencies.
Read the full articleYou might also wanna read
Reverse-Engineering Transformer Attention Heads Using Program Synthesis
A longstanding goal of research on interpretable deep learning is to replace opaque neural computations with human-meaningful symbolic descr
arxiv.org·10d ago
Study Reveals Convergent Evolution in How Language Models Learn Number Representations
Language models trained on natural text learn to represent numbers using periodic features with dominant periods at $T=2, 5, 10$. In this pa
arxiv.org·2mo ago
PHOTON: Hierarchical Autoregressive Model for Efficient Language Generation
Transformers operate as horizontal token-by-token scanners; at each generation step, the model attends to an ever-growing sequence of token-
arxiv.org·6mo agoAnalyzing Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper pres
arxiv.org·1y ago
New Method Enables Constant-Cost Self-Attention Computation for Transformers
The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attenti
arxiv.org·5mo agoTauformer: A Topological Transformer Architecture Using Laplacian-Derived Scalar Attention
Tauformer is a topological transformer (see paper) that replaces dot‑product attention with a Laplacian-derived scalar (taumode) per token/h
Comments
Sign in to join the conversation.
No comments yet. Be the first.