Does batch size matter?
This post is aimed at readers who are already familiar with stochastic gradient descent (SGD) and terms like “batch size”. For an introduction to these ideas, I recommend Goodfellow et al.’s Deep…
Read the full articleYou might also wanna read
Understanding Continuous Batching in Large Language Models: From Attention Mechanisms to Throughput Optimization
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Batch vs Online Learning - Choosing the Right AI Training
Batch vs online learning explained for AI engineers. Explore core concepts, real-world applications, trade-offs, and practical selection str
Why Gradient Descent's Evolution Matters for AI
Gradient-based optimization isn't just a buzzword. It's the backbone of deep neural network training. A new analysis sheds light on its evol
Understanding Scaling Laws in Deep Learning: A Framework for Optimal Compute Allocation
Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decr
lilianweng.github.io·14d agoUnderstanding Scaling Laws in Deep Learning: A Framework for Optimal Compute Allocation
Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decr
lilianweng.github.io·14d agoUnderstanding Scaling Laws in Deep Learning: A Framework for Optimal Compute Allocation
Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decr
lilianweng.github.io·14d ago
Reflecting on Optimisation: A Personal Take on Theory vs. Modern Practice
This is nothing to be proud of, but I have never really studied optimisation in depth. Oh sure, I know my Adam from my AdaGrad and I even us
Unveiling the Hidden Dynamics of Score-based Generative Models
Exploring the optimization dynamics in Score-based Generative Models, researchers bring new insights into SGD's role, offering guidance for
Comments
Sign in to join the conversation.
No comments yet. Be the first.