Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data

Feiyu Duan; Jingang Wang; Liangyu Xu; Rongxiang Weng; Sirui Wang; Xuemiao Zhang; Xunliang Cai; Yongwei Zhou

REVIEW 3 major objections 5 minor 1 cited by

Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that LLM pretraining should follow the model's evolving data preferences, and proposes a curriculum that reorders an existing corpus offline by a new Perplexity Difference (PD) metric.

desk verdict A promising offline data-ordering method whose qualitative gains look real, but the headline 3B numbers rest on tuned, single-run estimates and a partly assumed preference mechanism. read the letter →

arxiv 2501.13126 v2 pith:7RIMH6J2 submitted 2025-01-21 cs.CL cs.AI

Xuemiao Zhang , Liangyu Xu , Feiyu Duan , Yongwei Zhou , Sirui Wang , Rongxiang Weng , Jingang Wang , Xunliang Cai This is my paper · ORCID

classification cs.CLcs.AI

keywords LLMpretrainingcurriculumlearningperplexitydifferencedataorderingpreferencefunctionefficiencyMMLUCMMLU

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that during pretraining, a language model's preferred data changes as the model grows stronger, so training should gradually shift from easy, low-PD samples to hard, high-PD samples. It introduces Perplexity Difference (PD), the normalized gap between a weak model's perplexity and a strong model's perplexity on a sample, and uses it to partition the corpus. An S-shaped preference function then decides, at each training step, what fraction of the batch should come from the low-PD versus high-PD partition, all arranged offline so pretraining never pauses. In experiments with 1.3B and 3B models, this reordering alone—with no extra data or compute—beats uniform sampling and several curriculum baselines; the 3B model trained on 1T tokens gains 8.1% average accuracy on MMLU and CMMLU. If correct, the result identifies a cheap, orthogonal axis for improving LLM pretraining: the order in which existing data is consumed.

What carries the argument

The load-bearing mechanism is the Perplexity Difference (PD) metric combined with an S-shaped preference function. PD is computed from two reference models of different sizes trained on a subset of the corpus, approximating an early and a late checkpoint of the experimental model. The S-shaped function $f(p) = 1/(1+\exp(a(p-0.5)))$ prescribes the concentration of low-PD data at each training step, ensuring low-PD data dominates early and high-PD data dominates late, while the integral constraint keeps the total volumes of the two partitions equal.

What would settle it

Train a 3B model on 1T tokens with the reverse schedule (high-PD data first and low-PD data last) and check whether it matches or exceeds PDPC; or compute PD from the actual early and late checkpoints of the experimental model and check whether the reference-model PD ordering agrees with it—if the reverse schedule wins or the orderings diverge, the curriculum's mechanism is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that high-PD samples—texts a strong model fits far better than a weak model—are exactly the samples a training model should encounter late, while low-PD samples should appear early. The paper defines $\mathrm{PD}(x) = (\mathrm{PPL}_{M_w}(x) - \mathrm{PPL}_{M_s}(x))/\mathrm{PPL}_{M_w}(x)$ and partitions the corpus into two equal halves, low-PD and high-PD. A preference function $f(p)$ maps training progress $p = k/K$ to the proportion of low-PD data in the current batch, and the paper shows empirically that an S-shaped $f(p) = 1/(1+\exp(a(p-0.5)))$ outperforms linear, Z-shaped, and uniform schedules. With this schedule, test loss drops faster in the later half of training, gradient norms stabilize, and downstream accuracy improves most on benchmarks like MMLU, CMMLU, and CEVAL; the reported 3B result is 4.1% average gain across all benchmarks and 8.1% across MMLU and CMMLU.

Load-bearing premise

The load-bearing premise is that the data preference of the actual 1.3B or 3B model during training is well approximated by perplexity differences between two smaller reference models trained on a subset of the same corpus.

Editorial extensions

If this is right

Data ordering alone can yield substantial downstream gains at a fixed token budget and fixed model size.
Because the entire arrangement is performed offline, PDPC can be added as a final data preprocessing step without interrupting pretraining.
PD is stable across reference-model sizes, so a cheap 100M/700M model pair suffices to compute the ordering.
The observed gains concentrate in the later half of training, suggesting that the high-PD portion of the corpus is what unlocks the largest accuracy increases.
The S-shape preference function, rather than full sample-level sorting, preserves batch diversity and avoids the homogeneity that hurts sequential curricula.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The same 'learnability gap' logic could be applied to other domains—vision, speech, multimodal data—wherever a weak and strong reference model can be trained on a sample of the corpus.
Editorial inference: The 3B/1T result may not transfer to frontier-scale training, and the paper's own limitation section notes that only $n=2$ partitions were tested; larger $n$ might improve or change the picture.
Editorial inference: Iteratively refitting the preference function from new checkpoints, as sketched in the appendix, could compound gains but adds compute; the paper does not claim this is already validated.
Editorial inference: PD is a difficulty signal that appears orthogonal to quality ratings, so combining PDPC with data selection or quality filtering is a natural next test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Main thing to know: this paper proposes an offline data-curriculum method that reorders pretraining data by a perplexity-difference (PD) metric between a weak and a strong reference model, then feeds low-PD data first and high-PD later following an S-shaped schedule. Reported gains are real in shape — consistent across 1.3B/100B and 3B/1T runs, several ablations, and against Random and curriculum baselines. The 3B result claims +8.1 average on MMLU/CMMLU, and if that held at that precision it would be a strong no-extra-data, no-extra-compute improvement. What's actually new: PD as a difficulty signal and the offline preference-function scheduler; prior work either interrupts training (MATES) or scores quality rather than difficulty (QuRating). That is a fair contribution.

What the paper does well: the design is sensible, the ablations on RM sizes and function shapes support the qualitative claim, and the limitations section openly admits n=2, single runs, and unexplored iteration. The case study on data source and quality distributions is a nice sanity check.

Where it is soft, in proportion: the headline magnitude is not pinned down. The S-shape steepness a=10 is chosen by searching on the evaluation benchmarks; all runs are single runs with no variance; the 3B comparison has only Random as baseline; and the mechanism story relies on reference-model PD standing in for the 3B model's preferences, which is never checked against the actual 3B model on the benchmarks that carry the gain. The Figure 4 ablation at 1.3B is relevant but does not cover MMLU/CMMLU, and Appendix B's annealing validation is described but not actually run. So the stress-test note is right: the 'preference' is in part a fitted schedule, and the 8.1% number should be read as an existence proof, not a measured property of the model's preference.

That said, the central claim — low-to-high PD reordering helps pretraining — holds up in shape. The soft spots are addressable, not fatal. A serious referee should see it: ask for multiple seeds, a fixed schedule chosen before evaluation, a direct comparison of RM-based PD against the actual model's early and late checkpoints, and code or data release. I would take it for review and expect heavy revision; I would not desk-reject.

Referee Report

3 major / 5 minor

Summary. The paper proposes PDPC, a curriculum pretraining framework that orders data by Perplexity Difference (PD), defined as the relative perplexity gap between a weak and a strong reference model. The data are partitioned into low-PD and high-PD halves, and an S-shape preference function (Eq. 8) schedules the mixing ratio of low-PD data over training progress, gradually shifting from low-PD to high-PD data. The authors report consistent gains over random sampling at 1.3B scale on 100B tokens, and a headline 3B/1T result with +8.1% average accuracy on MMLU and CMMLU (Table 2). The data ordering is computed offline with small reference models, so pretraining runs without interruption.

Significance. If the 3B result is reliable, the paper offers a practical and surprisingly effective data-reordering curriculum that requires no extra data, no extra compute, and no training interruption. The 1.3B controlled comparisons in Tables 1 and 3 are a genuine strength: they span multiple metrics (PD, PPL, QuRating) and multiple schedule families, and the PD-based S-shape curve consistently wins. The reference-model approximation for PD is a sensible cost-reduction idea, and the distributional analyses in Figures 6-9 help build intuition. However, the central mechanism the paper advertises in its title and abstract---that the method 'always perceives and uses the data preferred by LLMs'---is not empirically grounded. The preference curve is fitted to downstream benchmark accuracy, and the reference-model PD is validated only at 1.3B on a subset of benchmarks that exclude MMLU and CMMLU. The authors honestly acknowledge the functional-exploration origin of the curve in Section 6, but that acknowledgment also marks the load-bearing gap: the title claim is currently a hypothesis, not a measured property.

major comments (3)

[§3.2, Table 2; §3.3, Figure 4] The 3B/1T headline gain of +8.1% on MMLU/CMMLU rests on PD computed from 100M and 1.3B reference models, yet the only direct validation of this approximation is at 1.3B scale and on ARC-E/ARC-C/SciQ/HellaSwag/PIQA (Figure 4). The paper never checks whether RM-PD agrees with the 3B model's actual preferences---for example, by computing PD from early and late checkpoints of a randomly trained 3B model and comparing it with the RM-based PD, or by reporting any 3B-scale correlation. Moreover, the validation benchmarks exclude exactly the two benchmarks (MMLU, CMMLU) that produce the headline improvement. Because the entire method is a reordering based on this preference signal, the 3B result is currently unexplained by the proposed mechanism; it could be an artifact of the specific schedule rather than of the model's true data preference.
[§2.3, Eq. (8); Table 3; §6; Appendix B] The S-shape preference function f(p)=1/(1+exp(a(p-0.5))) with a=10 is selected by maximizing downstream accuracy on the evaluation benchmarks themselves (Table 3), and the same benchmarks are used to report the main results. The paper's own limitation section states that the curve was 'determined through functional exploration' rather than measured, and the annealing-based measurement described in Appendix B is not actually executed---no experimental results are reported there. Thus the 'preference function' is a fitted hyperparameterized schedule, not an empirical measurement of what the model prefers. The central claim that the method 'always perceives and uses the data preferred by LLMs' therefore overstates what is demonstrated. The authors should either run the annealing experiments and show that the fitted S-shape matches measured preferences, or reframe the contribution as a tuned curriculum schedule and discuss the risk of benchmark overfitting.
[§3.1-3.2, Tables 1-2] All reported results are single runs with no variance, confidence intervals, or significance tests. For a claim as large as +8.1% on MMLU/CMMLU, a single run is insufficient evidence of a real effect. Furthermore, the 3B comparison (Table 2) includes only Random as a baseline; the 1.3B table shows that Sequential-PD-Low2High and Preference CL-PPL are markedly worse than PDPC, but at 3B we do not know whether PDPC beats these alternatives or merely beats random sampling. The abstract's claim that PDPC 'significantly surpasses baselines' is not substantiated for the 3B scale without at least one strong baseline (e.g., Sequential-PD-Low2High or Preference CL-PPL) and preferably a small number of seeds to gauge run-to-run variation.

minor comments (5)

[§3.1, Table 2] The benchmark name 'CEV AL' in the text and Table 2 header contains a spurious space; it should be 'CEVAL' for consistency with the Appendix figures.
[Figure 3 caption] The legend notation '(PDPC, PD, Sig.)' is ambiguous; the text elsewhere uses 'S.' for S-shape. Please use a consistent abbreviation such as '(PDPC, PD, S.)'.
[§3.5, Figure 9] The semantic-properties analysis uses GPT-4o annotations but reports no sample size, no inter-rater agreement, and no statistical test. The absolute differences between high-PD and low-PD proportions are often small (e.g., 0.83 vs 0.82 for Rule 1), so the claim that 'PD is independent of other linguistic features' is not supported without additional quantification.
[Appendix B] The text of Appendix B is written in the present tense as if the annealing experiments were carried out ('We conduct experiments on checkpoints...'), but no results are reported. Since Section 6 lists this as a future direction, Appendix B should be clearly labeled as a proposed method or preliminary sketch, not as an executed experiment.
[Figure 5 caption] The caption says 'during model training of 1B model,' but the experiment described in the text is the 1.3B model; please correct the label.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the PD ordering and S-shape schedule are empirically selected and validated against external benchmarks, with the 3B headline result an out-of-sample application.

full rationale

The paper's derivation chain is self-contained and empirical rather than circular. PD is defined directly from two reference-model perplexities (Eq. 2) and the low-to-high ordering is a stated curriculum assumption, not an equation that forces the reported gains. The S-shape preference function is selected by explicit function search over three candidate families and a hyperparameter grid (Section 2.3, Table 3), then applied to train 1.3B and 3B models; the 3B result on MMLU/CMMLU/CEVAL is an out-of-sample application because those benchmarks were not used in the preference-function selection, which used ARC-E, ARC-C, SciQ, HellaSwag, and PIQA on the 1.3B setting. Figure 4 independently compares RM-based PD against early/end checkpoints of a randomly trained 1.3B model and finds consistent behavior, supporting the reference-model approximation rather than assuming it. The only caveat is interpretational: the paper labels the fitted S-shape schedule as the model's 'preference' without directly measuring that preference, and Section 6 concedes the curve was determined by functional exploration. That is an unfalsifiable-label or validity concern, not a circular reduction, because the reported accuracy gains are measured against Random and other baselines on held-out benchmarks and do not reduce to the fitting procedure by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The core mechanism rests on an assumed link from PD to model preference, an assumed reference-model proxy, and a tuned sigmoid schedule. None of these are derived from first principles.

free parameters (4)

S-shape steepness a = a = 10
Grid-searched over a in {2.5, 5.0, 7.5, 10.0} and chosen by best average downstream score in Table 3.
Preference function family = S-shape (sigmoid)
Selected from Linear, Z-shape, and S-shape candidates based on downstream performance (Table 3); not derived from measured preference.
Number of PD partitions n = n = 2
Chosen for tractability; the paper states that n > 2 is left to future work.
Reference model size pair = 100M/700M for 1.3B runs; 100M/1.3B for 3B runs
Selected via ablation (Figure 4) as slightly better than alternatives; no theoretical justification is given.

assumptions (4)

domain assumption High-PD samples are harder for weak models and should be scheduled later in pretraining.
Stated in Section 2.3 as the CL principle; no proof is given that later placement is optimal beyond the empirical gains.
domain assumption A model's data preference across training can be approximated by perplexity differences between two smaller reference models trained on a subset of the same corpus.
Section 3.1 introduces reference models to approximate early and final checkpoints based on trained FLOPs; ablation in Figure 4 supports but does not prove this.
ad hoc to paper The S-shape preference function with a=10 predicts the proportion of low-PD data the model prefers at each training step.
Appendix B attempts to measure preferences by annealing, but the main experiments use the searched S-shape function, so the preference prediction is assumed rather than measured.
domain assumption Equal-volume low/high PD partition (median split) preserves enough diversity while ordering difficulty.
Section 2.3 argues that extreme sorting harms diversity, motivating n=2 mixing; no formal criterion for the split point is offered.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data." pith.science (2026). https://pith.science/paper/7RIMH6J2

@misc{pith2026250113126,
  author       = {Pith},
  title        = {Pith review of: Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RIMH6J2}},
  note         = {Machine review of arXiv:2501.13126}
}

read the original abstract

Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model's capability improves, it is intuitive that its data preferences dynamically change, indicating the need for pretraining with different data at various training stages. To achieve it, we propose the Perplexity Difference (PD) based Preference Curriculum learning (PDPC) framework, which always perceives and uses the data preferred by LLMs to train and boost them. First, we introduce the PD metric to quantify the difference in how challenging a sample is for weak versus strong models. Samples with high PD are more challenging for weak models to learn and are more suitable to be arranged in the later stage of pretraining. Second, we propose the preference function to approximate and predict the data preference of the LLM at any training step, so as to complete the arrangement of the dataset offline and ensure continuous training without interruption. Experimental results on 1.3B and 3B models demonstrate that PDPC significantly surpasses baselines. Notably, the 3B model trained on 1T tokens achieves an increased average accuracy of over 8.1% across MMLU and CMMLU.

Figures

Figures reproduced from arXiv: 2501.13126 by the authors.

**Figure 2.** Left: Preference Functions and Right: a com [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Few-shot downstream performance with re [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Ablation on PD calculation methods. Z-shape function, which uses only low-PD data initially and high-PD data later, slightly outperforms Random. The linear, Z-shape, and all S-shape parameter settings outperform Random, confirming the robustness of our framework. 3.4…

**Figure 6.** Figure 6: Data distribution across different sources. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 9.** Figure 9: Semantic properties differences of low-PD [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 10.** Figure 10: PPL distribution of low-PD and high-PD data. PD (100M-700M) PD (100M-1.3B) PD (700M-1.3B) PD (100M-700M) PD (100M-1.3B) PD (700M-1.3B) 1.00 0.95 0.41 0.95 1.00 0.66 0.41 0.66 1.00 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

**Figure 11.** Figure 11: Spearman correlation coefficient of PDs from [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

**Figure 12.** Figure 12: PD distribution across different domains. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

**Figure 13.** Figure 13: Few-shot downstream performance on various benchmarks with respect to pretraining iterations for [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Estimating the Effects of Sample Training Orders for Large Language Models without Retraining
cs.LG 2025-05 reject novelty 6.0 of 10

A framework using Taylor expansions and random projections estimates LLM performance under arbitrary training batch orders from one reference run.

Reference graph

Works this paper leans on

13 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

Does the text contain polysemous words? Polysemous words may make understanding more difficult
[2]

Does the text use specialized terminology? Specialized terminology may require specific domain knowledge to understand
[3]

Does understanding the text require specific cultural background knowledge? Cultural background dependence may increase the complexity of understanding
[4]

Does the text require logical reasoning to understand? Logical reasoning adds depth to understanding
[5]

Does the text contain elements of humor? Humor may affect the way the text is understood
[6]

Does the text explore ethical or moral issues? This may increase the depth of thought
[7]

Does the text use complex sentence structures? Complex sentence structures may increase the difficulty of understanding
[8]

Does the text contain scientific or technical concepts? These concepts may require specific knowledge to understand

Show all 13 references

[9]

Does the text express obvious emotional tones? Emotional tones may affect the understanding of the text
[10]

I was used to weekly massage and monthly chiropractic care. That was gone,

Does understanding the text require additional background knowledge? Background knowledge requirements may affect the comprehensibility of the text. 16 D Case Study Table 4: Samples are divided into 10 PD quantiles, with two samples representing each quantile. 0-10% Sample 1: ...

1915
[2019]

arXiv preprint arXiv:1903.09848

Competence-based curriculum learning for neural machine translation. arXiv preprint arXiv:1903.09848. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling ...

1903 arXiv
[2020]

arXiv preprint arXiv:2009.03300

Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-...

2009 arXiv
[2024]

Because..., my answer is 'X'

How to train data-efficient llms. arXiv preprint arXiv:2402.09668. Eva Sharma, Chen Li, and Lu Wang. 2019. Bigpatent: A large-scale dataset for abstractive and coherent summarization. arXiv preprint arXiv:1906.03741. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGres...

2019 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Does the text contain polysemous words? Polysemous words may make understanding more difficult

[2] [2]

Does the text use specialized terminology? Specialized terminology may require specific domain knowledge to understand

[3] [3]

Does understanding the text require specific cultural background knowledge? Cultural background dependence may increase the complexity of understanding

[4] [4]

Does the text require logical reasoning to understand? Logical reasoning adds depth to understanding

[5] [5]

Does the text contain elements of humor? Humor may affect the way the text is understood

[6] [6]

Does the text explore ethical or moral issues? This may increase the depth of thought

[7] [7]

Does the text use complex sentence structures? Complex sentence structures may increase the difficulty of understanding

[8] [8]

Does the text contain scientific or technical concepts? These concepts may require specific knowledge to understand