L2 Regularization and Batch Norm
This blog post is about an interesting detail about machine learning that I came across as a researcher at Jane Street - that of the interaction between L2 regularization, also known as weight decay…
Read the full articleYou might also wanna read
Study Shows Weight Decay During Pretraining Improves Language Model Adaptability After Fine-Tuning
Large language models are typically trained in two broad phases: pretraining to produce a base model, followed by further training to improv
arxiv.org·1mo ago
Demystifying the Training of Two-Layer Neural Networks
Exploring the mysteries of two-layer neural networks, a new method reveals how back-propagation untangles the 'black box.' But is it truly g
Emergent Hebbian Dynamics in Regularized Learning: A Theoretical Analysis
Hebbian and anti-Hebbian plasticity are widely observed in the brain and are classically modeled as mechanistic, local homosynaptic rules st
arxiv.org·1mo ago
Understanding Continuous Batching in Large Language Models: From Attention Mechanisms to Throughput Optimization
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
The Underestimated Impact of L0 in Sparse Autoencoder Training
Sparse Autoencoders rely heavily on a hyperparameter, L0, which if misconfigured, can compromise their effectiveness. Recent findings highli
Benign Overfitting: A New Era for Linear Classifiers
Deep learning's peculiar ability to overfit yet generalize well is stirring up the world of linear classifiers. There's more than meets the
Comments
Sign in to join the conversation.
No comments yet. Be the first.