Systematic Evaluation of Deep Learning Optimizers Reveals Limited Speedup Over AdamW in Language Model Pretraining
AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings…
Read the full articleYou might also wanna read
Is the Softmax Bottleneck Holding Back AI Progress?
Neural language models face a hidden bottleneck limiting both expressivity and optimization. Is it time for a rethink on LM design?
Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers
arXiv:2607.14427v1 Announce Type: new Abstract: A depth-recurrent transformer applies a weight-tied core a variable number of times, and pri
Flash-MSA Method Aims to Speed Up AI Training on Million-Token Sequences
Researchers have introduced Flash-MSA, a technique designed to accelerate the training of large language models on very long sequences of up
AI: Deep-Thinking Tokens Outpace Length in Language Models
Language models shine when prioritizing deep-thinking tokens over sheer length. Think@n optimizes this approach, enhancing accuracy and cost
LK Losses: A New Training Objective to Optimize Acceptance Rate in Speculative Decoding for LLMs
Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to propose candidate
arxiv.org·1mo agoAccelerating Large-Scale LLM Inference on AMD Instinct MI350X/MI355X with Eagle3 and AMD Quark
Large language model (LLM) inference is increasingly constrained by autoregressive decoding. Even when prefill is highly optimized, the deco
Comments
Sign in to join the conversation.
No comments yet. Be the first.