Reflecting on Optimisation: A Personal Take on Theory vs. Modern Practice
Summary
Magnus Ross reflects on his lack of deep study in optimisation, admitting he knows the basics (Adam, AdaGrad, L-BFGS) but zones out when discussions get theoretical. He questions the relevance of learning classical constrained convex optimisation in an era dominated by gradient descent and pre-trained models, drawing a humorous parallel to making the cheapest possible diet. The article appears to be a personal, reflective piece on the perceived gap between classical optimisation theory and modern machine learning practice.
Source
Twitter / XReflecting on Optimisation: A Personal Take on Theory vs. Modern Practicemagnusross.github.ioKey quotes
· 4 pulledI have never really studied optimisation in depth.
Oh sure, I know my Adam from my AdaGrad and I even used L-BFGS one time, but when people start talking about dual spaces and convergence for L∞ continuous functions, I tend to glaze over a bit.
Why do I need to learn about optimising constrained convex functions in the gradient-descent-for-everything-wait-actually-just-use-a-pre-trained-model era?
Isn't that what they used to, like, make the cheapest possible diet? Boring!
You might also wanna read
Optimizing Deep Learning Performance Through First-Principles Reasoning
The article discusses improving deep learning model performance by reasoning from first principles rather than relying on ad-hoc tricks and
Learning from Redundant Optimization: The asin() Function Case Study
The article discusses the author's realization that a faster implementation of the asin() (arcsine) function was already available in standa
16bpp.net·3mo agoThe Significance of Generalization in AI Systems and the Quest for Consciousness
The blog post discusses the importance of generalization in building AI systems with deep learning, emphasizing the significance of diverse
The Decline of Decision Theory in AI: How ImageNet and Academic Incentives Shifted the Field
The article explores why decision theory and mathematical approaches to AI stopped being widely used despite their effectiveness. The author
gfrm.in·3mo agoDarwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
Examining the Limitations of Transformer Models and the Gap to Human-Level AI
The article presents a skeptical perspective on claims about imminent Artificial General Intelligence (AGI), arguing that current transforme
dlants.me·4mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.