Proposal: Overparameterized Neural Networks with High Learning Rates Could Bridge AI-Human Intelligence Gap

Summary

This speculative article proposes a major shift in deep learning scaling paradigms by suggesting that the key difference between artificial neural networks (particularly LLMs) and human brains lies in a bias-variance tradeoff. The author argues that LLMs minimize variance while human brains minimize bias, and that human brains achieve this through deep double descent-style overparameterization combined with extremely high learning rates. The proposal suggests that training overparameterized neural networks with high learning rates and regularization could trigger "catapulting" or "grokking" phenomena, potentially leading to artificial neural networks with human-like performance and true generalization capabilities.