All Topics
All Topics
Technology
Technology
AI
AI
Business
Business
Entertainment
Entertainment
News
News
Programming
Programming
Science
Science
Design
Design
Environment
Environment
Finance
Finance
Crypto
Crypto
Politics
Politics
Sports
Sports
Education
Education
Gaming
Gaming
Art
Art
Music
Music
Health
Health
Security
Security
Books
Books
Food
Food
Travel
Travel
Personal
Personal
Bluesky
Twitter

Fragmented References to ML Optimization Algorithms (Grafting, Shampoo, AdamW)

10d ago· 1 min readenInsight
scienceprogramming

Summary

This is a fragmented, incomplete excerpt listing references to optimization algorithms in machine learning research. It mentions techniques like grafting (Agarwal et al.), Shampoo optimizer with per-mode preconditioners (Gupta et al.), scalable Shampoo with iterative inverse-root methods (Anil et al.), and AdamW (Loshchilov and Hutter). The content is essentially a list of citations with brief descriptions, cut off mid-sentence.

Source

Twitter / XFragmented References to ML Optimization Algorithms (Grafting, Shampoo, AdamW)rohan-anil.github.io

Key quotes

· 4 pulled
Agarwal, Anil, Hazan, Koren, Zhang: grafting decouples update magnitude from update direction; layer-wise grafting applies this per parameter group.
Gupta, Koren, Singer: Shampoo maintains per-mode preconditioners and gives stochastic convex guarantees for tensor optimization.
Anil, Gupta, Koren, Regan, Singer: scalable Shampoo replaces expensive spectral decompositions with iterative inverse-root methods and pipelines stale preconditioners.
Loshchilov and Hutter: AdamW dec
Snippet from the RSS feed
Best viewed on a desktop computer Keep the feedback coming.

You might also wanna read

Systematic Evaluation of Deep Learning Optimizers Reveals Limited Speedup Over AdamW in Language Model Pretraining

This research paper systematically evaluates ten deep learning optimizers for language model pretraining, challenging previous claims of 1.4

arxiv.org·10mo ago

Learning from Redundant Optimization: The asin() Function Case Study

The article discusses the author's realization that a faster implementation of the asin() (arcsine) function was already available in standa

16bpp.net·3mo ago

Gaussian Processes for Machine Learning (2006) [pdf]

gaussianprocess.org·10mo ago

Developing a BLAS Library for the AMD AI Engine [pdf]

uni.tlaan.nl·6mo ago

Flow Map Learning via Nongradient Vector Flow [pdf]

openreview.net·2mo ago

Advanced Scheme Techniques (2004) [pdf]

people.csail.mit.edu·9mo ago

Comments

Sign in to join the conversation.

No comments yet. Be the first.