ICLR 2026Oral

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining

Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu^†, Wenguang Chen^†

^† Co-corresponding authors

Learning-rate decay weakens late-arriving high-quality data, while moderate decay and curriculum model averaging recover its training benefits.

Constant, WSD and cosine learning rates reveal different effects of ascending-quality curricula. — Curriculum × learning rate · Fig. 1

Oral Presentation (Top 1.2%), ICLR 2026
OpenReview PDF
arXiv

This work studies curriculum-based LLM pretraining under mixed-quality data and identifies a key incompatibility between ascending-quality curricula and decaying learning rates. It shows that moderate learning-rate decay together with model averaging can recover the gains of curriculum ordering.

Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu, and Wenguang Chen. (2026). "How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining." ICLR 2026.