Study Shows Weight Decay During Pretraining Improves Language Model Adaptability After Fine-Tuning

Summary

This research paper investigates how weight decay during pretraining of large language models affects their downstream adaptability (plasticity). Through systematic experiments, the authors demonstrate that larger weight decay increases model plasticity, leading to better performance after fine-tuning—even when base models show worse pretraining loss. This creates counterintuitive trade-offs where worse-performing base models can become better after additional training. The mechanistic analysis reveals weight decay encourages linearly separable representations, regularizes attention matrices, and reduces overfitting. The findings challenge using cross-entropy loss as the sole metric for hyperparameter optimization and highlight the importance of considering downstream adaptability during pretraining.