{"id":"e1e792cd-ea72-45b7-a187-de106bce9bdc","arxiv_id":"2411.16035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Finetuning small models shifts the point where capability emerges, and extrapolating this shift to the low-data limit predicts few-shot emergence up to about 4x the compute in advance.","lead":"By finetuning today's smaller language models on a task, the authors can predict the scale at which much larger models will suddenly show non-trivial ability on that task. The method fits an 'emergence law' that maps finetuning data amount to the emergence point, then extrapolates to the few-shot setting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All validation is within one OpenLLaMA V1 run; the central claim's transfer to independently trained future models rests on the untested equivalence stated in Section 5.3.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing condition: intermediate checkpoints from a single OpenLLaMA V1 run, indexed by pretraining loss, may not be a valid proxy for independently trained models of different scale, and the finetuning-induced shift may not transfer across architectures and training distributions. This is the condition that must hold for the paper's headline contribution to be practically meaningful. The paper is internally consistent and the empirical observation of a finetuning-induced shift is well supported, but all quantitative validation of the emergence law is within one run family. The V2 case study provides partial cross-run evidence for the same architecture and a different data mixture, but it does not test a substantially different pretraining setup, and the APPS case study has no verifiable ground truth. The reader's CONDITIONAL verdict already makes a second-family validation a condition, so my read does not change the verdict; it sharpens why that condition is load-bearing and proposes a concrete test that would settle it.","tokens_in":30503,"tokens_out":5032,"duration_ms":49032,"concrete_test":"Run the full emergence-law pipeline on a second, independently trained model family with multiple intermediate checkpoints, such as Pythia or OLMo, using a common held-out loss metric like C4 validation loss (as in Section 7.1). Fit only on small pre-emergence checkpoints from that family, then compare the predicted few-shot emergence point to the true few-shot emergence point measured on larger checkpoints of the same family. If the absolute prediction error exceeds 0.1 nats on any benchmark, the central claim must be scoped to same-run predictions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that finetuning small pre-emergence models predicts the few-shot emergence point of larger future models. That claim requires pretraining loss to be a portable capability index across training runs, so the emergence law fit on one run transfers to another. In this paper, every within-0.1-nat validation (Section 6, Figures 4 and 7) uses intermediate checkpoints from a single OpenLLaMA V1 run: the 'future models' are later checkpoints of the same optimizer trajectory, and the FLOPS-in-advance numbers (Section 6.4) are computed by holding out later checkpoints from that same run. Section 5.3 asserts that intermediate checkpoints from one run are 'just as effective as using models from separate pretraining runs of varying scale,' but the supporting evidence (Figure 3 left) only shows consistency across 3B/7B/13B checkpoints within the same run family at matched pretraining losses. The V2 experiment (Section 7.1) is a second data mixture of the same architecture, and the APPS experiment (Section 7.2) has no ground truth. Section 8 explicitly concedes that it is not understood whether models with different architectures or training schemes show the same downstream capabilities at a given pretraining loss. If the loss-capability mapping is run-specific, the 0.1-nat accuracy is a within-run extrapolation, not a prediction about GPT-N+1.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper poses the problem of emergence prediction: given access to pre-emergence LLM checkpoints (models at or below random-chance few-shot accuracy on a task), can one predict the point in scaling at which few-shot performance will emerge? The authors observe that task-specific finetuning shifts the emergence elbow toward weaker models, and that the magnitude of the shift grows with the amount of finetuning data. They formalize this with an \"emergence law\" E_θ(D) = k·log(D)^α + C, fit jointly with a ReLU model of downstream accuracy as a function of pretraining loss, and then extrapolate to a low-data limit D_0 to predict the few-shot emergence point. Validation on MMLU, GSM8K, CommonsenseQA, and CoLA using OpenLLaMA V1 intermediate checkpoints shows MLE predictions within 0.1 nats of a ReLU-fit ground truth on all four tasks, with the ability to predict up to roughly 4x the pretraining FLOPS in advance on MMLU and GSM8K when later checkpoints are held out. The paper also presents two case studies: using the method to compare pretraining data quality between OpenLLaMA V1 and V2, and a proof-of-concept prediction of APPS emergence in LLaMA 2 at approximately 325B parameters.","tokens_in":30772,"tokens_out":6202,"duration_ms":56150,"significance":"If the method generalizes beyond the specific run family studied, it would be a valuable and inexpensive tool for forecasting emergent capabilities, with direct implications for safety preparedness, model development, and data-quality decisions. The paper's strengths include a clearly posed problem, a simple and reproducible method, and an honest discussion of limitations. The \"No Few-shot\" ablation in Section 6.3 rules out the most direct form of circularity, since accurate predictions are obtained even when the few-shot target data are excluded from fitting. The central empirical phenomenon—finetuning shifts the emergence elbow in a data-dependent way—is convincingly demonstrated across four benchmarks and three model sizes, and the paper makes its code and checkpoints available. The main weakness is that all primary validation is within a single pretraining run family (OpenLLaMA V1), which limits the strength of the claim that the method predicts emergence in future, independently trained models.","major_comments":[{"comment":"The central claim as framed in the abstract and introduction is that the method can predict whether future models (GPT-N+1) will emerge. However, the validation in Section 6 is entirely within a single OpenLLaMA V1 run: the \"future models\" are later checkpoints of the same optimizer trajectory, and the 4x-FLOPS-in-advance numbers in Section 6.4 are computed by holding out later checkpoints from that same run. Section 5.3 asserts that intermediate checkpoints from one run are \"just as effective as using models from separate pretraining runs of varying scale,\" but the supporting evidence (Figure 3 left) only shows consistency across 3B/7B/13B checkpoints within the same run family at matched pretraining losses. Section 8 explicitly concedes that it is not understood whether models with different architectures or training schemes show the same downstream capabilities at a given pretraining loss. The V2 experiment in Section 7.1 is a second data mixture of the same architecture, and the APPS experiment has no ground truth. This is a load-bearing limitation: if the loss-capability mapping is run-specific, the 0.1-nat accuracy is a within-run extrapolation rather than a prediction about independently trained future models. The authors should either reframe the central claim as within-run emergence prediction (which is well supported) or provide evidence across independent pretraining runs, for example by fitting on one open model family and validating on another.","section":"Section 5.3 / Section 8"},{"comment":"The ground-truth emergence point is itself estimated by fitting a ReLU to the few-shot checkpoints, and the success criterion is the distance between the predicted elbow and this fitted elbow. This is a reasonable operationalization, but the paper does not report uncertainty in the ground-truth fit. A noisy ground-truth fit can make the 0.1-nat threshold either too lenient or too strict, and the MCMC prediction intervals in Table 3 are often wider than the MLE error (e.g., GSM8K full-data interval [0.004, 0.170] nats). Please add a sensitivity analysis of the ground-truth ReLU fit—for example, bootstrap resampling of the few-shot checkpoints or an alternative definition of the emergence point—and report the resulting confidence interval alongside the prediction error. This would clarify whether the reported successes are robust to the choice of ground-truth model.","section":"Section 6.2 / Equation 1"},{"comment":"The headline numbers \"4.3x and 3.9x FLOPS in advance\" are computed by counting the earliest successful held-out checkpoint, but the paper itself notes (footnote 4) that GSM8K has a failed prediction between two successes. With only a handful of held-out checkpoints, the earliest-success criterion is sensitive to noise; the failed prediction is just outside the 0.1-nat threshold, but it is still a failure under the stated criterion. The full sequence of held-out predictions (including failures) appears only in Appendix A.6, and Figure 7 compresses them to single points. Please present the full sequence in the main text or figure, and if the \"4x\" claim is retained, state the criterion exactly as \"earliest successful prediction\" with the caveat about intermediate failures. The current presentation risks overstating the reliability of the advance-prediction capability.","section":"Section 6.4 / Figure 7"}],"minor_comments":[{"comment":"In the sentence \"as we increase the amount of finetuning data, the point of point of emergence is shifted further,\" the phrase \"point of point of\" should be corrected to \"point of emergence.\"","section":"Section 4.2"},{"comment":"The text contains a typo: \"OpeLLaMA\" should be \"OpenLLaMA,\" and \"corpa\" should be \"corpora\" (or \"corpora\" in the singular, \"corpus\").","section":"Section 7.1"},{"comment":"The symbol table would be more complete if it included the optional parameter Δ from Section 5.4, which is subsequently used in the model equation but is absent from Table 1.","section":"Section 5.1 / Table 1"},{"comment":"The caption states that error bars represent the 5th and 95th percentile errors from MCMC posterior sampling; it would be clearer to note explicitly that these are percentiles of prediction error, not a confidence interval for the emergence point itself, although the main text does explain this.","section":"Figure 6 caption"},{"comment":"The MCMC temperature sweep (selecting the greatest temperature for which the posterior mode is centered on the MLE) is a somewhat ad hoc calibration step that is only described in the appendix. A brief summary of this procedure in Section 6.1 would help readers assess the claimed uncertainty calibration.","section":"Appendix A.7 / Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published at COLM 2024; the arXiv version appears to be a post-conference report. For a journal submission, the main value is the emergence-law formulation and the empirical demonstration of finetuning-induced emergence shifts. The most serious concern is the gap between the GPT-N+1 framing and the within-run validation; I would encourage the editor to ask for either a scoped title/abstract or additional cross-run evidence. The ground-truth circularity is adequately addressed by the No Few-shot ablation, so I do not consider that a rejection-level issue. The evaluation robustness concerns (ground-truth uncertainty and the FLOPS-in-advance criterion) are fixable with additional analysis and presentation changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. The paper's core observation holds up: finetuning shifts the ReLU emergence elbow toward weaker checkpoints, and the amount of finetuning data controls the shift. That is shown consistently on four tasks and three model sizes, and the emergence-law extrapolation predicts the few-shot emergence point within 0.1 nats. The No Few-shot ablation is important—it shows the prediction is not just fitting the target value by construction. The MCMC uncertainty intervals are a real plus, and the paper is honest about its limitations in Section 8.\n\nThe soft spot is exactly what the stress-test note says. All validation comes from intermediate checkpoints of a single OpenLLaMA V1 run. The 'future models' in Section 6 are later checkpoints of the same optimizer trajectory, and the FLOPS-in-advance numbers are computed by holding out those checkpoints. Section 5.3 asserts that intermediate checkpoints from one run are 'just as effective' as separate pretraining runs, but the supporting evidence only shows consistency across 3B/7B/13B checkpoints within that same run family. So the within-run 0.1-nat accuracy is real, but the transfer to independently trained future models is an untested assumption. The V1/V2 experiment is suggestive, and the APPS case study is honest about having no ground truth, but neither fixes the core gap.\n\nSome minor flags: finetuning on the MMLU test set is a legitimate choice given the benchmark's structure, but it should be more prominent; the ground-truth emergence point is itself a ReLU fit, which biases the reported errors; and the 4x-in-advance headline is the optimistic bound from the earliest successful holdout, with some closer holdouts failing, as Table 2 shows. None of these are fatal.\n\nThe citation pattern is fine. They engage with the emergence-is-a-mirage debate and related scaling-law work rather than ignoring it. The math is simple but appropriate for the question; the fitting procedure follows Hoffmann et al. and the uncertainty estimation is careful.\n\nWho is this for: anyone working on capability forecasting, scaling laws, or safety-relevant prediction of emergent abilities. It deserves a serious referee—I would send it out. The review should push for validation on a second pretraining family or independent runs, and for a cleaner statement of what the method can and cannot claim about GPT-N+1.","headline":"Genuinely new emergence-prediction result, well-executed within a single OpenLLaMA V1 run; the cross-run transfer that would make it a true GPT-N+1 predictor is the one thing not yet demonstrated.","tokens_in":31337,"tokens_out":1765,"would_cite":true,"duration_ms":18162,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fitting an emergence law to finetuned small models predicts few-shot emergence within 0.1 nats and up to 4.3x the FLOPS in advance.","keywords":["emergent capabilities","emergence prediction","scaling laws","finetuning","pretraining loss","language models","emergence law","few-shot prompting"],"falsifier":"Take a model family with a substantially different architecture or data mix, fit an emergence law using only its small pre-emergence checkpoints, and compare the predicted pretraining-loss elbow with the observed few-shot emergence of its larger checkpoints; if the prediction misses by more than 0.1 nats on several tasks, the transfer assumption fails.","tokens_in":30227,"feed_emoji":"📈","tokens_out":8958,"duration_ms":74719,"temperature":0.7,"pith_summary":"Emergent capabilities are abilities that appear suddenly once a language model passes some scale threshold, and pre-emergence models look random on the task, so predicting the threshold has been open. This paper tries to predict that threshold using only small, pre-emergence models. Its central finding is that task-specific finetuning shifts the emergence point toward weaker models, and the amount of finetuning data controls how far it shifts. Fitting a parametric \"emergence law\" to those shifts and extrapolating to the few-shot limit predicts the true few-shot emergence point within 0.1 nats of pretraining loss on all four benchmarks studied, and in the best cases up to 4.3x the pretraining FLOPS before the emergence point is reached. If this transfers to future model families, capability forecasts could be made without training the large models first.","feed_headline":"Small finetuned models predict when capabilities emerge","feed_subtitle":"Extrapolate the finetuning shift to few-shot prompting: MMLU and GSM8K predicted 3.9-4.3x FLOPS before emergence.","key_machinery":"The load-bearing object is the emergence law, a parametric function $E_\\theta(D)=k\\log(D)^\\alpha + C$ that gives the ReLU elbow (the point of emergence, measured in pretraining loss) as a function of finetuning data amount $D$. It is embedded in a ReLU performance model $\\mathrm{Perf}=A\\max(E_\\theta(D)-L(M),0)+B+\\Delta\\mathbf{1}_{\\mathrm{finetuned}}$ and fit to raw accuracy of checkpoints at pretraining loss $L(M)$; the low-data limit $D_0$ near the few-shot count converts finetuning measurements into a few-shot prediction. Pretraining loss acts as the capability index, so intermediate checkpoints from one run stand in for independently trained models of different scales, and MCMC sampling turns the fit into a calibrated posterior over the emergence point.","core_discovery":"Finetuning a pre-emergence model on a task does not just improve that model: it reveals where the task's few-shot emergence elbow would sit in a larger model. Concretely, the paper shows that when intermediate checkpoints from a single 3B pretraining run are finetuned, the accuracy-versus-pretraining-loss curve keeps its ReLU shape but the elbow moves to higher loss (weaker models), and more finetuning data moves it further. The paper encodes this as an emergence law $E_\\theta(D) = k \\log(D)^\\alpha + C$, fits $A, B, \\theta$ jointly to raw finetuning accuracies through $\\mathrm{Perf} = A\\max(E_\\theta(D)-L(M),0)+B+\\Delta\\mathbf{1}_{\\mathrm{finetuned}}$, then evaluates at the low-data limit $D_0$ equal to the number of few-shot examples. On MMLU, GSM8K, CommonsenseQA, and CoLA the maximum-likelihood prediction lands within 0.1 nats of the true ReLU elbow, and holdout analysis shows MMLU and GSM8K can be predicted 4.3x and 3.9x the FLOPS before the first emerged checkpoint. A case study on the APPS coding benchmark with LLaMA 2 predicts emergence near 325B parameters, offered as a proof-of-concept for frontier-relevant capabilities.","pith_inferences":["The paper leaves implicit that a minimal-rank adapter (LoRA rank 1 already shifts emergence) could serve as an even cheaper probe than full finetuning; testing whether adapter rank and data amount trade off in the same emergence law would make the method more economical.","If pretraining loss is a universal capability index, the emergence law fit on one model family should transfer to another family trained on the same data distribution; a direct cross-family transfer experiment would settle this and is the most natural next step.","The APPS prediction is testable: if a model trained on LLaMA 2-style data is scaled past roughly 250-500B parameters and evaluated on APPS, the first non-trivial pass@1 accuracy should appear in that window; a strong miss would refute the extrapolation.","Emergence laws could be inverted into a design tool: instead of only predicting when a task will emerge, a developer could select a finetuning data budget that pulls a desired capability down to a target scale, turning prediction into controlled elicitation."],"forward_implications":["Model developers can use emergence laws to compare pretraining data quality without training large models: on MMLU the method predicts that OpenLLaMA V2 emerges before V1, matching the observed order.","Safety and policy teams could estimate when a capability will emerge from small checkpoints alone, giving lead time before the large model that exhibits the capability is trained.","Predictions are insensitive to the exact low-data extrapolation limit within a reasonable range, so the method does not require a finely chosen $D_0$.","Accurate predictions are sometimes possible with only a handful of checkpoints and the smallest finetuning subsets, so data collection cost can be far below training a large model.","The same finetuning-induced shift appears on the harder APPS coding task, indicating the approach can extend beyond simple benchmarks toward capabilities closer to the frontier."],"supporting_citations":[{"why":"Establishes pretraining loss as a predictable power law of compute and scale, the foundation for using loss as the independent variable throughout.","marker":"Kaplan et al., 2020"},{"why":"Provides the grid-search-plus-L-BFGS fitting procedure the paper adapts for the emergence law.","marker":"Hoffmann et al., 2022"},{"why":"Defines emergent abilities in large language models, the phenomenon the paper aims to predict.","marker":"Wei et al., 2022a"},{"why":"Raises the 'emergence is a mirage' objection that the paper counters by showing emergence under continuous metrics.","marker":"Schaeffer et al., 2023"},{"why":"Supports modeling downstream accuracy as a function of pretraining loss, the core capability index.","marker":"Gadre et al., 2024"},{"why":"Provides loss-perspective evidence that models at equal pretraining loss behave similarly, justifying intermediate checkpoints as proxies for scale.","marker":"Du et al., 2024"},{"why":"Supplies the OpenLLaMA V1 and V2 intermediate checkpoints used to fit and validate the emergence laws.","marker":"Geng & Liu, 2023"},{"why":"Supplies the LLaMA 2 models and the scaling law used for the APPS case-study prediction at ~325B parameters.","marker":"Touvron et al., 2023b"}],"fun_headline_variants":["Finetuning reveals where LLM capabilities emerge","Predict emergence 4x before it happens via finetuning","Small finetuned models forecast large-model emergence","Emergence laws: finetuning predicts GPT-N+1 skills","See emergent capabilities coming with finetuning shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that pretraining loss is a universal capability index: checkpoints from one pretraining run stand in for independently trained models of different sizes, and the finetuning shift observed on that run transfers to other architectures, data distributions, and dataset difficulties.","fun_headline_variants_meta":{"raw":{"variants":["Finetuning reveals where LLM capabilities emerge","Predict emergence 4x before it happens via finetuning","Small finetuned models forecast large-model emergence","Emergence laws: finetuning predicts GPT-N+1 skills","See emergent capabilities coming with finetuning shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1424,"prompt_tokens":1094,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":252}},"tokens_in":710,"tokens_out":330,"duration_ms":3628,"temperature":1.0,"reasoning_tokens":252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:37:13.274782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model family with a substantially different architecture or data mix, fit an emergence law using only its small pre-emergence checkpoints, and compare the predicted pretraining-loss elbow with the observed few-shot emergence of its larger checkpoints; if the prediction misses by more than 0.1 nats on several tasks, the transfer assumption fails.","supporting_citations":[{"cited_title":"Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt","cited_arxiv_id":null,"evidence_quote":"Supports modeling downstream accuracy as a function of pretraining loss, the core capability index."}],"review_version":1}