Study Reveals Convergent Evolution in How Language Models Learn Number Representations

Summary

This research paper investigates how different language models (Transformers, Linear RNNs, LSTMs, and classical word embeddings) learn to represent numbers using periodic features with dominant periods at T=2, 5, and 10. The authors identify a two-tiered hierarchy: while all models learn features with period-T spikes in the Fourier domain, only some learn geometrically separable features for mod-T classification. They prove that Fourier domain sparsity is necessary but not sufficient for geometric separability. The study finds that data, architecture, optimizer, and tokenizer all influence whether models acquire geometrically separable features, which can come from complementary co-occurrence signals in language data or from multi-token addition problems. The paper highlights convergent evolution in feature learning across diverse model architectures.