Emerging Scientific Theory of Deep Learning: The Case for "Learning Mechanics"

Summary

This academic paper argues that a scientific theory of deep learning is emerging, which the authors call "learning mechanics." They identify five growing bodies of research that point toward such a theory: solvable idealized settings, tractable limits, simple mathematical laws, theories of hyperparameters, and universal behaviors. The emerging theory focuses on training dynamics, coarse aggregate statistics, and falsifiable quantitative predictions. The authors distinguish this mechanics perspective from statistical and information-theoretic approaches, and anticipate a symbiotic relationship with mechanistic interpretability. They also address common arguments against the possibility or importance of fundamental theory in deep learning.