{"id":"f1023b22-c8cd-43d1-8ff2-31c947acd549","arxiv_id":"2505.06150","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The claimed scaling law Accuracy = A(N·L)^β M^γ + E is a re-parameterization of total-token scaling, and the paper reports no fitted coefficients, errors, or code to test it.","lead":"This paper proposes to model LLM fine-tuning accuracy with Accuracy = A (N times L) raised to beta times model size raised to gamma plus E, where N is example count and L is average token length. It reports small accuracy differences across data-composition strategies, but it withholds the fitted parameters and code needed to check the law.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proposed scaling law cannot represent the very data-composition effect it claims to measure: V=N·L is total tokens, so matched-V rows imply identical accuracy, while Table 1 reports accuracy differences at matched V.","rationale":"The reader's REJECT verdict identified missing parameters, lack of error bars, and internal inconsistencies such as the copied MMLU volume column. My stress-test converges on rejection but for a different, more fundamental reason: the proposed functional form cannot express the paper's own data-composition claim. Because V=N·L is exactly total tokens, any monotone scaling law in V predicts no accuracy difference for the matched-V rows in Table 1. The paper nevertheless treats those differences as evidence that composition matters, and the appendix further undermines the claim by admitting strategy-dependent exponents and ablation results that are inconsistent with a single V-only law. This is not a question of noise or withheld numerical values; it is a logical mismatch between the proposed equation and the empirical phenomenon it purports to explain. The reader's weakest assumption about run-to-run noise is plausible but secondary: even if every accuracy difference were statistically significant, the central formula would still be unable to account for it. Therefore the verdict should remain REJECT, and the strongest concrete check is to evaluate the fitted law on the matched-V rows of Table 1, which should expose the inconsistency directly once parameters and data are available.","tokens_in":4304,"tokens_out":3899,"duration_ms":41610,"concrete_test":"Obtain the fitted A, β, γ, and E values from the authors' code or supplementary material, or re-fit them from the released data using the stated linear-regression procedure. Then evaluate the fitted law at the V and M values in the three rows of Table 1. If a single global law was fit, the three predictions will be nearly identical because the V values are within 0.3% of each other, while the observed accuracies span 0.278–0.294; that discrepancy is the load-bearing failure. If instead separate parameters were fit per strategy, the central 'single scaling law' claim is already false. A complementary check is to fit the alternative model Accuracy = A N^{β_N} L^{β_L} M^γ + E to the same data and test whether β_N ≠ β_L. A statistically significant difference would demonstrate that composition effects require separate exponents for N and L, which the product V=N·L cannot capture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is internally inconsistent, not merely under-evidenced. The paper defines V=N·L and states 'Although V=N·L is mathematically equivalent to total tokens,' then proposes Accuracy=A V^β M^γ+E. Under this law, any two training sets with the same V and same model size have identical predicted accuracy, regardless of how tokens are split into examples. Table 1 reports three strategies with nearly identical V values—11558, 11570, 11545—but mean accuracies of 0.278, 0.294, and 0.294. The paper reads this spread as evidence that data composition significantly affects token efficiency, yet the functional form in the abstract and Section 2 cannot produce such a spread. The appendix then states that the exponents β and γ 'vary systematically with the subsampling strategy' and that ablation studies 'vary N and L independently while keeping V constant, confirming that both components contribute significantly.' If the parameters vary by strategy, there is no single predictive scaling law of the claimed form; if the parameters are global, the observed composition effect is unexplained by V. Either way, the headline claim that incorporating dataset volume V=N·L 'yields a scaling law that predicts fine-tuning performance' is not supported by the presented model. This concern is independent of missing error bars or withheld parameter values: even with perfect data and full code, the claimed law is mathematically unable to distinguish the three Table 1 rows.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a scaling law for LLM fine-tuning under fixed compute budgets, Accuracy = A V^β M^γ + E, where V = N·L is the product of the number of fine-tuning examples and their average token length, and M is model size. The authors argue that this 'dataset volume' formulation captures data-composition effects that total-token scaling misses. They report experiments on the BRICC dataset and MMLU subsets with three subsampling strategies (few long, many short, balanced) and four model sizes, concluding that data composition significantly affects token efficiency and that larger models use tokens more effectively. The appendix describes parameter tuning via grid search over E and linear regression on log-transformed variables, but it does not report fitted parameter values, fit quality, or error bars.","tokens_in":4570,"tokens_out":2987,"duration_ms":29131,"significance":"If the proposed scaling law were valid, it would offer a practically useful prediction tool for fine-tuning under limited compute. However, the manuscript as written does not make a checkable numerical claim: the fitted constants A, β, γ, E are not reported, no fit quality measures are given, and no error bars or significance tests accompany the accuracy comparisons. More fundamentally, the functional form Accuracy = A V^β M^γ + E depends on V only through total tokens, so it cannot represent the very data-composition effects the paper uses as its main evidence. The manuscript also contains an apparent table error in the MMLU appendix. These issues jointly undermine the central claim, so the paper cannot be accepted in its current form.","major_comments":[{"comment":"The proposed law Accuracy = A V^β M^γ + E depends on V = N·L only, so any two training sets with the same V and same M are predicted to have identical accuracy regardless of how tokens are split into examples. Table 1 lists V values of 11558, 11570, and 11545 for the three strategies—nearly identical—yet reports mean accuracies of 0.278, 0.294, and 0.294. The text interprets this spread as evidence for a data-composition effect, but the functional form in the abstract and Section 2 cannot produce such a spread. This is an internal inconsistency in the paper's central claim, independent of missing error bars.","section":"Section 2 and Table 1"},{"comment":"The appendix states that 'the exponents β and γ vary systematically with the subsampling strategy.' If the exponents are strategy-dependent, there is no single predictive scaling law of the form Accuracy = A V^β M^γ + E; the model is a per-strategy fit that cannot predict performance for unseen data compositions. This contradicts the abstract's claim that the formulation 'yields a scaling law that predicts fine-tuning performance under fixed compute budgets' and also undermines the claimed baseline comparison against total-token scaling.","section":"Appendix, Parameter Tuning Procedure"},{"comment":"The central equation is never numerically instantiated: the paper says 'the internal parameter values are not reported here' and defers details to an appendix that also omits them. A scaling law without its fitted constants, fitting residuals, or goodness-of-fit measures is not a testable or usable claim. The abstract's equation cannot be checked against the experimental data, and the claimed 'better prediction errors' from baseline comparisons are never quantified.","section":"Appendix, Additional Experimental Details"},{"comment":"The MMLU table reports V values identical to the BRICC table (11558, 11570, 11545) even though the N and L columns are different. For example, 77.91 × 30.17 does not equal 11558. This indicates the V column was likely copied from the BRICC table, making the MMLU results unreliable as reported.","section":"Appendix, Table 2"},{"comment":"The paper reports mean-accuracy differences of 0.278 vs. 0.294 on BRICC and 0.33 vs. 0.34 on MMLU as evidence that subsampling strategy matters, but no error bars, number of seeds, or significance tests are provided. Without run-to-run variance information, these differences could be within noise, and the claim that 'data composition significantly affects token efficiency' is not statistically supported.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The text says 'Figure 1 illustrates the distribution of accuracy across subsampling strategies,' but the Figure 1 caption describes normalized token efficiency, and Figure 2's caption describes box plots of accuracy. The figure references appear to be swapped or mislabeled.","section":"Section 3"},{"comment":"The appendix says 'Figure 3 shows the normalized token efficiency' and later says 'Figure 3 presents box plots of accuracy'; the two statements cannot both refer to the same figure, and the figure numbering is inconsistent.","section":"Appendix"},{"comment":"The table is titled 'Representative performance on BRIMI' but the dataset is called BRICC throughout the rest of the paper; this appears to be a typo.","section":"Table 1 title"},{"comment":"The phrase 'Although V=N·L is mathematically equivalent to total tokens' is itself an admission that the proposed volume measure is not a new data-composition variable; the paper would benefit from a clearer explanation of why relabeling total tokens changes scaling behavior.","section":"Section 2"}],"recommendation":"reject","confidential_remarks":"The central issue is not simply missing details but a logical inconsistency: the proposed law cannot produce the composition-dependent accuracy differences the paper reports, because V=N·L is exactly total tokens. The appendix's statement that the exponents vary per subsampling strategy further means no unified scaling law is actually presented. Even with the missing values and code supplied, the functional form would need to be changed (e.g., separate exponents for N and L, or a composition term) before the core claim could be assessed. This is beyond the scope of a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2505.06150. The paper asks a good question—whether fine-tuning performance depends on how a fixed token budget is split between example count and average length—but the central quantitative claim is internally inconsistent. The authors define V=N·L, acknowledge this is mathematically identical to total tokens, then propose Accuracy=A V^β M^γ+E. Under that law, any two runs with the same V and same model size must have the same predicted accuracy. Table 1 reports V values of 11558, 11570, and 11545 with mean accuracies of 0.278, 0.294, and 0.294. The paper interprets that spread as evidence that data composition affects token efficiency, yet its own equation cannot produce any spread. That is a load-bearing flaw, not a missing robustness check.\n\nThe qualitative hypothesis is worth testing, and the experimental design—three subsampling strategies under a fixed compute budget across several small models—is a reasonable first cut. Using BRICC and MMLU subsets is fine, and the authors are transparent that the tuning procedure follows Zhang et al. (2024). That said, the evidence is thin. The accuracy differences have no error bars, seeds, or significance tests; the fitted parameters A, β, γ, E are never given, so the claimed law is uncheckable; and the appendix says β and γ vary systematically with the subsampling strategy, which contradicts the idea of a single predictive law. The normalized efficiency η_norm uses E and γ fit to the same data, so the trend that larger models are more token-efficient is partly circular. The MMLU table also repeats the BRICC V column verbatim despite different N and L, and the figure captions describe different content than the figures, which further undermines trust in the reporting.\n\nIf the authors instead proposed Accuracy=A N^α L^δ M^γ+E and tested whether α and δ differ, with reported parameters and uncertainty, that would be a real contribution. As written, the law cannot distinguish the rows of its own main table, so the central conclusion does not follow. This is a desk reject for me. The research question is worth pursuing, but this version doesn't give referees anything checkable. I'd suggest telling the authors to reformulate and resubmit with full artifacts.","headline":"The paper asks a good question but its central law, defining V=N·L, cannot explain the composition effects it reports; with parameters withheld and no error bars, the claims are not checkable—desk reject.","tokens_in":5129,"tokens_out":3269,"would_cite":false,"duration_ms":31194,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning accuracy under fixed compute follows a power law in dataset volume V = N·L—example count times average token length—not raw token count alone.","keywords":["scaling laws","LLM fine-tuning","data composition","token efficiency","compute budgets","dataset volume","subsampling strategies"],"falsifier":"Re-run the three subsampling strategies on the same datasets and models with at least ten seeds per condition; if the mean accuracy gaps fall within the seed-to-seed spread, the composition effect vanishes. In parallel, publication of the fitted $A$, $\\beta$, $\\gamma$, and $E$ values would permit a direct numerical test of whether the $V$-based law outperforms a total-token baseline on both BRICC and MMLU.","tokens_in":4087,"feed_emoji":"📈","tokens_out":8562,"duration_ms":72072,"temperature":0.7,"pith_summary":"This paper claims that fine-tuning a large language model under a fixed compute budget cannot be predicted from total token count alone; what matters is how those tokens are divided into examples. It proposes a scaling law, $\\text{Accuracy} = A V^{\\beta} M^{\\gamma} + E$, where $V$ is the dataset volume—the number of examples $N$ times their average token length $L$—and $M$ is model size. Experiments on the BRICC and MMLU datasets with three subsampling strategies (few long, many short, balanced) show that for similar total tokens, mean accuracy differs, for example 0.278 versus 0.294 on BRICC. The authors report that volume-based scaling beats conventional total-token baselines in prediction error. If correct, the law turns data composition into a tunable variable for practitioners with limited compute.","feed_headline":"Fine-tuning accuracy follows dataset volume, not token count","feed_subtitle":"The new law Accuracy = A·V^β·M^γ + E makes example count and length tunable levers.","key_machinery":"The central object is the dataset volume $V = N \\cdot L$, the product of the number of examples and their average token length; although mathematically equal to total tokens, it reorganizes the data so that composition is explicit. It carries the argument by being the variable whose exponent $\\beta$ in the power law $\\text{Accuracy} = A V^{\\beta} M^{\\gamma} + E$ absorbs the effect of subsampling strategy. The fitting procedure linearizes the law as $\\ln(\\text{Accuracy} - E) = \\ln A + \\beta \\ln V + \\gamma \\ln M$, then performs a regression over a grid of the offset $E$, following established fine-tuning scaling practice. The normalized token efficiency $\\eta = (\\text{Accuracy} - E)/(V M^{\\gamma})$ is plotted against model size to show that larger models convert additional tokens into accuracy more effectively.","core_discovery":"On its own terms, the paper establishes a claim about data composition in fine-tuning: holding total tokens fixed, the split into example count and average token length changes final accuracy. The proposed relationship is $\\text{Accuracy} = A V^{\\beta} M^{\\gamma} + E$, a power law in dataset volume $V = N \\cdot L$ and model size $M$, with fitted constants $A$, $\\beta$, $\\gamma$, and $E$. The authors report that the fitted exponents vary systematically with the subsampling strategy, and that a volume-based law outperforms conventional total-token scaling in prediction error. The concrete evidence is mean accuracy differences across strategies: on BRICC, few-long yields 0.278 while many-short and balanced yield 0.294; on MMLU, 0.33 versus 0.34.","pith_inferences":["A testable extension left implicit: if the fitted exponents transfer across datasets, $V = N \\cdot L$ becomes a universal coordinate for comparing data shapes in fine-tuning, independent of a specific benchmark.","The $V$ columns in the BRICC and MMLU appendix tables are identical to the unit, inviting a check of whether the MMLU composition was recomputed or carried over; a corrected table would distinguish a genuine replication from a repeated number.","The law's functional form suggests a design rule for fixed compute: maximize $V^{\\beta}$ for a given model size and budget, which is checkable without knowing $A$ and $E$."],"forward_implications":["Practitioners under fixed compute can optimize fine-tuning by choosing a subsampling strategy rather than by maximizing raw token count.","The reported variation of $\\beta$ and $\\gamma$ with strategy means data composition changes how much accuracy an additional token or an additional parameter buys.","The rising normalized token efficiency with model size supports choosing the largest model that fits the compute cap, since larger models use extra tokens more effectively.","Volume-based scaling's lower prediction error, if it holds across datasets, gives a practical forecasting tool for small-data, GPU-limited fine-tuning."],"supporting_citations":[{"why":"Supplies the scaling-law form and tuning procedure used to fit $A$, $\\beta$, $\\gamma$, $E$, and the baseline that volume-based scaling is compared against.","marker":"Zhang et al. (2024)"},{"why":"Provides the MMLU benchmark and its subsets used in the cross-dataset experiments.","marker":"Hendrycks et al. (2021)"},{"why":"Provides the BRICC dataset, the domain-specific low-resource testbed for the main experiments.","marker":"Salavati et al. (2024)"},{"why":"Establishes compute-optimal pretraining scaling that motivates the fixed-compute, total-token baseline the paper contrasts.","marker":"Hoffmann et al. (2022)"},{"why":"Supplies scaling laws for transfer that the paper extends from pretraining to fine-tuning.","marker":"Hernandez et al. (2021)"},{"why":"Used to select the four model sizes from the leaderboard for the experiments.","marker":"Beeching et al. (2023)"}],"fun_headline_variants":["Fine-tune smarter: dataset volume beats token count","Example count and length matter as much as tokens","New scaling law: fine-tuning accuracy depends on dataset volume","Dataset volume, not token count, drives fine-tuning accuracy","Equal tokens, different accuracy: it's about example composition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the mean accuracy gaps between subsampling strategies—0.016 on BRICC and 0.01 on MMLU—are genuine effects rather than run-to-run noise, and that the fitted parameters are trustworthy; the paper reports no error bars, no seeds, and no fitted values for $A$, $\\beta$, $\\gamma$, or $E$.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tune smarter: dataset volume beats token count","Example count and length matter as much as tokens","New scaling law: fine-tuning accuracy depends on dataset volume","Dataset volume, not token count, drives fine-tuning accuracy","Equal tokens, different accuracy: it's about example composition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3650,"prompt_tokens":841,"completion_tokens":2809,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2731}},"tokens_in":457,"tokens_out":2809,"duration_ms":18901,"temperature":1.0,"reasoning_tokens":2731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:46:39.679647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three subsampling strategies on the same datasets and models with at least ten seeds per condition; if the mean accuracy gaps fall within the seed-to-seed spread, the composition effect vanishes. In parallel, publication of the fitted $A$, $\\beta$, $\\gamma$, and $E$ values would permit a direct numerical test of whether the $V$-based law outperforms a total-token baseline on both BRICC and MMLU.","supporting_citations":[{"cited_title":"Reducing biases towards minoritized populations in medical curricular content via artificial intelligence for fairer health outcomes","cited_arxiv_id":null,"evidence_quote":"Provides the BRICC dataset, the domain-specific low-resource testbed for the main experiments."}],"review_version":1}