Adaptive Polyak Step Sizes for Sharpness-Aware Minimization: Theory-Grounded Learning Rate Schedulers
This paper introduces adaptive learning rate schedulers for Sharpness-Aware Minimization (SAM), a popular optimizer for training machine learning models. The authors derive Polyak-type step sizes tailored to SAM updates, creating novel adaptive algorithms for both deterministic and stochastic settings. They prove linear convergence for strongly convex objectives and O(1/T) convergence for convex objectives in the deterministic case, with analogous guarantees up to a neighborhood of the optimum in the stochastic setting. Numerical experiments show the proposed schedulers match or exceed carefully tuned SAM baselines while reducing the need for learning-rate tuning.
Key quotes
Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models.
By explicitly minimizing the sharpness of the loss landscape, SAM often improves generalization while delivering strong empirical performance.
We derive Polyak schedulers tailored to SAM-style updates, yielding novel adaptive algorithms in both deterministic and stochastic settings.
Numerical experiments demonstrate that the proposed Polyak schedulers achieve performance comparable to or better than carefully tuned SAM baselines, while substantially reducing the need for learning-rate tuning.
From the article
You might also wanna read
Optimal Prediction-Correction Algorithm Using Sparse Linear Extrapolation for Time-Varying Optimization
Sparse Bayesian Learning Algorithms Revisited: From Learning Majorizers to Structured Algorithmic Learning Using Neural Networks
Asynchronous Distributed Learning Based on Gradient Coding
Structure-Aware Matrix Pencil for Estimating the Number of Damped Complex Exponentials
Understanding Reinforcement Learning for Model Training, and future directions with GRAPE
arxiv.org·9mo ago
Ultrafast FPGA-based inference and online learning using Kolmogorov-Arnold Networks
This post explains the author's Master's thesis on designing hardware architectures for ultrafast inference and online learning using Kolmog
aarushgupta.io·29d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.