NeurIPS 2026Spotlight

Open LLM Pretraining Guided by Quantile Benchmarking Heterogeneous Web Corpora

Kairong Luo^*, Zhenbo Sun^*, Xinyu Shi, Shengqi Chen, Bowen Yu, Yunyi Chen, Chenyi Dang, Hengtao Tao, Hui Wang, Fangming Liu, Kaifeng Lyu, Wenguang Chen

^* Co-first authors

Paper Model Data Code

Comparing web corpora across quality quantiles helps choose better data mixtures for open LLM pretraining.

Evaluation of the related Kaiyuan-2B model, whose data recipe uses quantile benchmarking. — Open pretraining · Kaiyuan-2B

NeurIPS 2026 Spotlight. Previously titled Quantile Benchmarking Heterogeneous Web Corpora in Open LLM Pretraining.

Web corpora contain a wide range of data quality, making corpus-level comparisons difficult to interpret. Quantile benchmarking compares their training effectiveness across quality levels and uses the results to guide data selection and mixing.

I am a co-first author, listed first. This paper is related to the PCMind-2.1-Kaiyuan-2B project, which releases the model, data, and pretraining code. The figure above is the model evaluation from that project’s technical report.

Kairong Luo, Zhenbo Sun, Xinyu Shi, Shengqi Chen, Bowen Yu, Yunyi Chen, Chenyi Dang, Hengtao Tao, Hui Wang, Fangming Liu, Kaifeng Lyu, and Wenguang Chen. (2026). "Open LLM Pretraining Guided by Quantile Benchmarking Heterogeneous Web Corpora." NeurIPS 2026 Spotlight.