ICLR 2025

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules

Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun, Kaifeng Lyu^†, Wenguang Chen^†

^† Co-corresponding authors

A multi-power law predicts loss under unseen learning-rate schedules, turning schedule selection from trial and error into a guided search.

Optimized learning-rate schedules and their measured loss curves compared with cosine and WSD. — Predicted schedules, measured gains · Fig. 1

arXiv

This paper introduces an empirical law to predict the pretraining loss of large language models under various learning-rate schedules. The proposed multi-power law accurately predicts loss curves for unseen schedules and helps identify schedules that outperform commonly used ones such as cosine decay.

Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun, Kaifeng Lyu, and Wenguang Chen. (2025). "A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules." ICLR 2025.