Publications

On data, optimization, and making pretraining more accessible. Google Scholar

Conference papers

Evaluation of the related Kaiyuan-2B model, whose data recipe uses quantile benchmarking. — Open pretraining · Kaiyuan-2B

NeurIPS 2026 Spotlight Data quality

Open LLM Pretraining Guided by Quantile Benchmarking Heterogeneous Web Corpora

Kairong Luo^*, Zhenbo Sun^*, Xinyu Shi, Shengqi Chen, Bowen Yu, Yunyi Chen, Chenyi Dang, Hengtao Tao, Hui Wang, Fangming Liu, Kaifeng Lyu, Wenguang Chen

^* Co-first authors

Comparing web corpora across quality quantiles helps choose better data mixtures for open LLM pretraining.

Paper Model Data Code

The RoBSA workflow showing RoPE score calculation and blockwise token selection. — RoBSA workflow · Fig. 6

ACL 2026 Efficient inference

RoBSA: RoPE-based Blockwise Sparse Multi-head Latent Attention

Xinyu Shi, Kairong Luo, Zhen Zheng, Wenguang Chen

Training-free blockwise sparse attention for MLA reduces long-context decoding latency with minimal accuracy loss.

ACL Anthology

Constant, WSD and cosine learning rates reveal different effects of ascending-quality curricula. — Curriculum × learning rate · Fig. 1

ICLR 2026 Oral Training dynamics

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining

Kairong Luo, Zhenbo Sun, Haodong Wen, Xinyu Shi, Jiarui Cui, Chenyi Dang, Kaifeng Lyu^†, Wenguang Chen^†

^† Co-corresponding authors

Learning-rate decay weakens late-arriving high-quality data, while moderate decay and curriculum model averaging recover its training benefits.

Paper arXiv

LLM experiments show larger datasets sustain more useful epochs before saturation. — Data reuse in LLM experiments · Fig. 1

ICLR 2026 Data reuse

Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression

Tingkai Yan^*, Haodong Wen^*, Binghui Li^*, Kairong Luo, Wenguang Chen, Kaifeng Lyu

^* Co-first authors

Theory and LLM experiments show why larger datasets can be repeated more: useful reuse depends on both dataset size and distribution.

OpenReview arXiv

Optimized learning-rate schedules and their measured loss curves compared with cosine and WSD. — Predicted schedules, measured gains · Fig. 1

ICLR 2025 Scaling laws

A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules

Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun, Kaifeng Lyu^†, Wenguang Chen^†

^† Co-corresponding authors

A multi-power law predicts loss under unseen learning-rate schedules, turning schedule selection from trial and error into a guided search.

arXiv

Preprints & technical reports

Puro-2B performance versus reproduction cost compared with open language models. — Cost vs. capability · Fig. 1

Technical Report · 2026 Open pretraining

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Kairong Luo^*, Jiarui Cui^*, Yaorui Yin^*, Shengqi Chen^*, Yiming Yang, Linxiang Gao, Yanmohan Wang, Chengxia Li, Mingzhe Zhang, Kaifeng Lyu^†, Wenguang Chen^†

^* Co-first authors ^† Co-corresponding authors

Training a 2B language model on consumer GPUs becomes practical when data, optimization, and FP8 systems are designed together.

arXiv Models & recipe

Kaiyuan-2B performance compared with fully open and open-weight language models. — Model performance · Fig. 1

Technical Report · 2025 Open pretraining

PCMind-2.1-Kaiyuan-2B Technical Report

Kairong Luo, Zhenbo Sun, Xinyu Shi, Shengqi Chen, Bowen Yu, Yunyi Chen, Chenyi Dang, Hengtao Tao, Hui Wang, Fangming Liu, Kaifeng Lyu, Wenguang Chen

An end-to-end open 2B pretraining stack releases the data, processing tools, training code, and model weights needed for reproducible research.

Technical Report Related paper Model Dataset Kaiyuan-Spark kaiyuan-mindformers

Preprint · 2023 Reinforcement learning

DreamFuser: Value-guided Diffusion Policy for Offline Reinforcement Learning

Kairong Luo^*, Caiwei Xiao^*, Zhiao Huang, Zhan Ling, Yunhao Fang, Hao Su

^* Co-first authors

A trajectory-based value optimization approach that combines diffusion policy learning with efficient Q-function learning for offline reinforcement learning.

OpenReview