Emergent Hebbian Dynamics in Regularized Learning: A Theoretical Analysis
By
[Submitted on 23 May 2025 (v1), last revised 28 May 2026 (this version, v3)]
A weekday bagel. Dependable, satisfying, no fuss.
Summary
This research paper investigates whether observed Hebbian/anti-Hebbian plasticity in synaptic updates necessarily implies an underlying Hebbian computation mechanism. The authors identify an alternative, emergent route: near stationarity, L2 weight decay can drive the learning-signal component of many update rules to align with a Hebbian direction, with alignment increasing with decay strength. This Hebbian-like signature can arise even for non-learning or random update rules, and stochastic noise can induce anti-Hebbian alignment. The findings suggest that emergent Hebbian signatures can coexist with genuine Hebbian plasticity, complicating the interpretation of synaptic measurements and motivating new experiments to distinguish mechanistic from emergent Hebbian computation.
Key quotes
· 4 pulledWe identify an alternative, emergent route: near stationarity, L2 weight decay generically drives the learning-signal component of many update rules to align with a Hebbian direction, with alignment increasing monotonically with decay strength.
This Hebbian-like signature is not specific to SGD and can arise even for non-learning or random update rules long before learning has ceased.
Stochastic noise in the learning signal can induce anti-Hebbian alignment, yielding a simple tradeoff with weight decay and a phase boundary in regression settings.
These mechanisms do not replace standard Hebbian theory; they can coexist with genuine Hebbian plasticity and complicate the interpretation of synaptic measurements.
You might also wanna read
Emergence of Diffusion Models from Associative Memory
Study Reveals Convergent Evolution in How Language Models Learn Number Representations
This research paper investigates how different language models (Transformers, Linear RNNs, LSTMs, and classical word embeddings) learn to re
arxiv.org·1mo ago
Emerging Scientific Theory of Deep Learning: The Case for "Learning Mechanics"
This academic paper argues that a scientific theory of deep learning is emerging, which the authors call "learning mechanics." They identify
arxiv.org·1mo ago
Research on Hallucination-Associated Neurons in Large Language Models: Identification, Impact, and Origins
This research paper investigates hallucination-associated neurons (H-Neurons) in large language models, examining their identification, beha
arxiv.org·5mo ago
Nested Learning: A New Machine Learning Paradigm for Continual Learning Inspired by Human Neuroplasticity
The article introduces "Nested Learning," a new machine learning paradigm for continual learning that addresses the challenge of models acqu
research.google·6mo ago
The Mathematics of Random Walks in High-Dimensional Spaces and Their Role in Deep Learning
The article explores the mathematics and physics of random walks in high-dimensional spaces, explaining how this concept underpins modern dy
galileo-unbound.blog·9mo ago