A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

Aidan Kierans; Avijit Ghosh; Ryan Lagasse; Shiri Dori-Hacohen

REVIEW 5 major objections 4 minor 8 references

A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning accuracy under fixed compute follows a power law in dataset volume V = N·L—example count times average token length—not raw token count alone.

desk verdict The paper asks a good question but its central law, defining V=N·L, cannot explain the composition effects it reports; with parameters withheld and no error bars, the claims are not checkable—desk reject. read the letter →

arxiv 2505.06150 v2 pith:QTJGAUGH submitted 2025-05-09 cs.CL cs.AI

Ryan Lagasse , Aidan Kierans , Avijit Ghosh , Shiri Dori-Hacohen This is my paper · ORCID

classification cs.CLcs.AI

keywords scalinglawsLLMfine-tuningdatacompositiontokenefficiencycomputebudgetsdatasetvolumesubsamplingstrategies

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that fine-tuning a large language model under a fixed compute budget cannot be predicted from total token count alone; what matters is how those tokens are divided into examples. It proposes a scaling law, $\text{Accuracy} = A V^{\beta} M^{\gamma} + E$, where $V$ is the dataset volume—the number of examples $N$ times their average token length $L$—and $M$ is model size. Experiments on the BRICC and MMLU datasets with three subsampling strategies (few long, many short, balanced) show that for similar total tokens, mean accuracy differs, for example 0.278 versus 0.294 on BRICC. The authors report that volume-based scaling beats conventional total-token baselines in prediction error. If correct, the law turns data composition into a tunable variable for practitioners with limited compute.

What carries the argument

The central object is the dataset volume $V = N \cdot L$, the product of the number of examples and their average token length; although mathematically equal to total tokens, it reorganizes the data so that composition is explicit. It carries the argument by being the variable whose exponent $\beta$ in the power law $\text{Accuracy} = A V^{\beta} M^{\gamma} + E$ absorbs the effect of subsampling strategy. The fitting procedure linearizes the law as $\ln(\text{Accuracy} - E) = \ln A + \beta \ln V + \gamma \ln M$, then performs a regression over a grid of the offset $E$, following established fine-tuning scaling practice. The normalized token efficiency $\eta = (\text{Accuracy} - E)/(V M^{\gamma})$ is plotted against model size to show that larger models convert additional tokens into accuracy more effectively.

What would settle it

Re-run the three subsampling strategies on the same datasets and models with at least ten seeds per condition; if the mean accuracy gaps fall within the seed-to-seed spread, the composition effect vanishes. In parallel, publication of the fitted $A$, $\beta$, $\gamma$, and $E$ values would permit a direct numerical test of whether the $V$-based law outperforms a total-token baseline on both BRICC and MMLU.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a claim about data composition in fine-tuning: holding total tokens fixed, the split into example count and average token length changes final accuracy. The proposed relationship is $\text{Accuracy} = A V^{\beta} M^{\gamma} + E$, a power law in dataset volume $V = N \cdot L$ and model size $M$, with fitted constants $A$, $\beta$, $\gamma$, and $E$. The authors report that the fitted exponents vary systematically with the subsampling strategy, and that a volume-based law outperforms conventional total-token scaling in prediction error. The concrete evidence is mean accuracy differences across strategies: on BRICC, few-long yields 0.278 while many-short and balanced yield 0.294; on MMLU, 0.33 versus 0.34.

Load-bearing premise

The load-bearing premise is that the mean accuracy gaps between subsampling strategies—0.016 on BRICC and 0.01 on MMLU—are genuine effects rather than run-to-run noise, and that the fitted parameters are trustworthy; the paper reports no error bars, no seeds, and no fitted values for $A$, $\beta$, $\gamma$, or $E$.

Editorial extensions

If this is right

Practitioners under fixed compute can optimize fine-tuning by choosing a subsampling strategy rather than by maximizing raw token count.
The reported variation of $\beta$ and $\gamma$ with strategy means data composition changes how much accuracy an additional token or an additional parameter buys.
The rising normalized token efficiency with model size supports choosing the largest model that fits the compute cap, since larger models use extra tokens more effectively.
Volume-based scaling's lower prediction error, if it holds across datasets, gives a practical forecasting tool for small-data, GPU-limited fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension left implicit: if the fitted exponents transfer across datasets, $V = N \cdot L$ becomes a universal coordinate for comparing data shapes in fine-tuning, independent of a specific benchmark.
The $V$ columns in the BRICC and MMLU appendix tables are identical to the unit, inviting a check of whether the MMLU composition was recomputed or carried over; a corrected table would distinguish a genuine replication from a repeated number.
The law's functional form suggests a design rule for fixed compute: maximize $V^{\beta}$ for a given model size and budget, which is checkable without knowing $A$ and $E$.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take on arXiv:2505.06150. The paper asks a good question—whether fine-tuning performance depends on how a fixed token budget is split between example count and average length—but the central quantitative claim is internally inconsistent. The authors define V=N·L, acknowledge this is mathematically identical to total tokens, then propose Accuracy=A V^β M^γ+E. Under that law, any two runs with the same V and same model size must have the same predicted accuracy. Table 1 reports V values of 11558, 11570, and 11545 with mean accuracies of 0.278, 0.294, and 0.294. The paper interprets that spread as evidence that data composition affects token efficiency, yet its own equation cannot produce any spread. That is a load-bearing flaw, not a missing robustness check.

The qualitative hypothesis is worth testing, and the experimental design—three subsampling strategies under a fixed compute budget across several small models—is a reasonable first cut. Using BRICC and MMLU subsets is fine, and the authors are transparent that the tuning procedure follows Zhang et al. (2024). That said, the evidence is thin. The accuracy differences have no error bars, seeds, or significance tests; the fitted parameters A, β, γ, E are never given, so the claimed law is uncheckable; and the appendix says β and γ vary systematically with the subsampling strategy, which contradicts the idea of a single predictive law. The normalized efficiency η_norm uses E and γ fit to the same data, so the trend that larger models are more token-efficient is partly circular. The MMLU table also repeats the BRICC V column verbatim despite different N and L, and the figure captions describe different content than the figures, which further undermines trust in the reporting.

If the authors instead proposed Accuracy=A N^α L^δ M^γ+E and tested whether α and δ differ, with reported parameters and uncertainty, that would be a real contribution. As written, the law cannot distinguish the rows of its own main table, so the central conclusion does not follow. This is a desk reject for me. The research question is worth pursuing, but this version doesn't give referees anything checkable. I'd suggest telling the authors to reformulate and resubmit with full artifacts.

Referee Report

5 major / 4 minor

Summary. The paper proposes a scaling law for LLM fine-tuning under fixed compute budgets, Accuracy = A V^β M^γ + E, where V = N·L is the product of the number of fine-tuning examples and their average token length, and M is model size. The authors argue that this 'dataset volume' formulation captures data-composition effects that total-token scaling misses. They report experiments on the BRICC dataset and MMLU subsets with three subsampling strategies (few long, many short, balanced) and four model sizes, concluding that data composition significantly affects token efficiency and that larger models use tokens more effectively. The appendix describes parameter tuning via grid search over E and linear regression on log-transformed variables, but it does not report fitted parameter values, fit quality, or error bars.

Significance. If the proposed scaling law were valid, it would offer a practically useful prediction tool for fine-tuning under limited compute. However, the manuscript as written does not make a checkable numerical claim: the fitted constants A, β, γ, E are not reported, no fit quality measures are given, and no error bars or significance tests accompany the accuracy comparisons. More fundamentally, the functional form Accuracy = A V^β M^γ + E depends on V only through total tokens, so it cannot represent the very data-composition effects the paper uses as its main evidence. The manuscript also contains an apparent table error in the MMLU appendix. These issues jointly undermine the central claim, so the paper cannot be accepted in its current form.

major comments (5)

[Section 2 and Table 1] The proposed law Accuracy = A V^β M^γ + E depends on V = N·L only, so any two training sets with the same V and same M are predicted to have identical accuracy regardless of how tokens are split into examples. Table 1 lists V values of 11558, 11570, and 11545 for the three strategies—nearly identical—yet reports mean accuracies of 0.278, 0.294, and 0.294. The text interprets this spread as evidence for a data-composition effect, but the functional form in the abstract and Section 2 cannot produce such a spread. This is an internal inconsistency in the paper's central claim, independent of missing error bars.
[Appendix, Parameter Tuning Procedure] The appendix states that 'the exponents β and γ vary systematically with the subsampling strategy.' If the exponents are strategy-dependent, there is no single predictive scaling law of the form Accuracy = A V^β M^γ + E; the model is a per-strategy fit that cannot predict performance for unseen data compositions. This contradicts the abstract's claim that the formulation 'yields a scaling law that predicts fine-tuning performance under fixed compute budgets' and also undermines the claimed baseline comparison against total-token scaling.
[Appendix, Additional Experimental Details] The central equation is never numerically instantiated: the paper says 'the internal parameter values are not reported here' and defers details to an appendix that also omits them. A scaling law without its fitted constants, fitting residuals, or goodness-of-fit measures is not a testable or usable claim. The abstract's equation cannot be checked against the experimental data, and the claimed 'better prediction errors' from baseline comparisons are never quantified.
[Appendix, Table 2] The MMLU table reports V values identical to the BRICC table (11558, 11570, 11545) even though the N and L columns are different. For example, 77.91 × 30.17 does not equal 11558. This indicates the V column was likely copied from the BRICC table, making the MMLU results unreliable as reported.
[Tables 1 and 2] The paper reports mean-accuracy differences of 0.278 vs. 0.294 on BRICC and 0.33 vs. 0.34 on MMLU as evidence that subsampling strategy matters, but no error bars, number of seeds, or significance tests are provided. Without run-to-run variance information, these differences could be within noise, and the claim that 'data composition significantly affects token efficiency' is not statistically supported.

minor comments (4)

[Section 3] The text says 'Figure 1 illustrates the distribution of accuracy across subsampling strategies,' but the Figure 1 caption describes normalized token efficiency, and Figure 2's caption describes box plots of accuracy. The figure references appear to be swapped or mislabeled.
[Appendix] The appendix says 'Figure 3 shows the normalized token efficiency' and later says 'Figure 3 presents box plots of accuracy'; the two statements cannot both refer to the same figure, and the figure numbering is inconsistent.
[Table 1 title] The table is titled 'Representative performance on BRIMI' but the dataset is called BRICC throughout the rest of the paper; this appears to be a typo.
[Section 2] The phrase 'Although V=N·L is mathematically equivalent to total tokens' is itself an admission that the proposed volume measure is not a new data-composition variable; the paper would benefit from a clearer explanation of why relabeling total tokens changes scaling behavior.

Circularity Check

2 steps flagged · score 8.0 of 10

Dataset volume is a rename of total tokens, and the normalized-efficiency evidence is built from the same fitted parameters.

renaming known result [Section 2, Methodology (equation defining V and the scaling law)]
"Although V=N·L is mathematically equivalent to total tokens, it explicitly emphasizes data composition. We hypothesize that fine-tuning accuracy scales as Accuracy=AV βM γ +E."

By defining V=N·L, the paper defines V to be exactly the total token count, which is the conventional input of the very scaling laws it claims to supersede. Substituting V into Accuracy=AV^β M^γ+E yields the standard total-token scaling law with the same functional form; no compositional information enters the equation. The claim that this new 'dataset volume' accounts for data composition is therefore a renaming of total tokens, not an independent derivation. The paper's own Table 1 reports different mean accuracies (0.278, 0.294, 0.294) at nearly identical V values (11558, 11570, 11545), a spread the single-V functional form cannot produce, confirming that the composition effect is outside the law rather than predicted by it.
fitted input called prediction [Section 3 (Figure 2) and Appendix 'Parameter Tuning Procedure']
"Figure 2 presents the normalized token efficiency, defined as ηnorm = Accuracy−E V Mγ , which demonstrates that, when properly normalized, larger models leverage additional tokens more effectively."

The parameters E and γ in ηnorm are not independent external values; they are fit to the same accuracy data in the Appendix: E is chosen by grid search to keep Accuracy−E positive, and γ is estimated by linear regression of ln(Accuracy−E) on ln(V) and ln(M). Plotting (Accuracy−E)/(V M^γ) and reading off a size-dependent trend is therefore reading the fitted parameters back as evidence. Under the fitted law, ηnorm equals A V^{β−1}, which is independent of M, so any apparent trend is a residual of the fit, not a newly measured effect. The Appendix also states that β and γ vary with subsampling strategy, so the composition conclusion is encoded in the per-strategy fit, not predicted by a single scaling law.

full rationale

The central 'dataset volume' V=N·L is, by the paper's own sentence, mathematically equivalent to total tokens. The scaling law Accuracy=A V^β M^γ+E is therefore the conventional total-token scaling law with a new name for T, and the abstract's claim that the formulation 'explicitly accounts for data composition' is not supported by the equation. The second circular step is the normalized token efficiency: E and γ are fitted to the same accuracy data, then the resulting ratio is presented as evidence that larger models use tokens more efficiently. That is fitted input being called a prediction. The paper's own report that β and γ change with subsampling strategy means there is no single predictive law of the claimed form; the composition effect is inserted through the per-strategy fit, not derived from V. The only author-overlapping citation (Salavati et al. for BRICC) is a dataset source and is not load-bearing for the scaling-law derivation. Missing error bars and unreported parameter values are additional reliability concerns, but the circularity finding rests on the paper's own definitions and fitting procedure.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The scaling law rests on four fitted parameters whose values are not disclosed, plus a domain-level assumption that V=N·L captures composition. No new physical entity is introduced; the only new construct, dataset volume, reduces by definition to total tokens. The grid search over E makes the residual positive and is a post-hoc fitting choice.

free parameters (4)

A = not reported
Multiplicative scale fit by linear regression of ln(Accuracy - E) on ln V and ln M after E is chosen.
β = not reported
Volume exponent; the paper claims it varies by subsampling strategy but gives no values.
γ = not reported
Model-size exponent; used in η_norm normalization and claimed to show larger models are more token-efficient.
E = grid-searched 0.20 to 0.30, exact values not reported
Irreducible accuracy offset chosen per strategy to keep Accuracy - E positive, a post-hoc selection.

assumptions (4)

domain assumption Accuracy follows a power law A V^β M^γ + E with an irreducible offset
The functional form is assumed in Section 2, citing Zhang et al., not derived.
domain assumption Compute budget can be considered fixed while varying N, L, and model size across families
Experiments use different model families and subsampling strategies but no compute measurement or control is reported.
domain assumption A single product V=N·L captures all data-composition effects
The model has no term for the shape of the length distribution beyond the mean, so composition enters only through N and L.
standard math Huber-loss linear regression is an appropriate robust fitting procedure for ln(Accuracy-E)
Standard regression method, but applied to residual values made positive by hand-tuning E.

invented entities (1)

Dataset volume V = N·L
purpose: Reparameterizes total token count into example count times mean length to emphasize data composition.
V is mathematically identical to total tokens, so it is not an independent measurable quantity; the claim of added predictive value rests entirely on the unreported fits.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets." pith.science (2026). https://pith.science/paper/QTJGAUGH

@misc{pith2026250506150,
  author       = {Pith},
  title        = {Pith review of: A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QTJGAUGH}},
  note         = {Machine review of arXiv:2505.06150}
}

read the original abstract

We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets that explicitly accounts for data composition. Conventional approaches measure training data solely by total tokens, yet the number of examples and their average token length -- what we term \emph{dataset volume} -- play a decisive role in model performance. Our formulation is tuned following established procedures. Experiments on the BRICC dataset \cite{salavati2024reducing} and subsets of the MMLU dataset \cite{hendrycks2021measuringmassivemultitasklanguage}, evaluated under multiple subsampling strategies, reveal that data composition significantly affects token efficiency. These results motivate refined scaling laws for practical LLM fine-tuning in resource-constrained settings.

Figures

Figures reproduced from arXiv: 2505.06150 by the authors.

**Figure 2.** Box plots of accuracy for each subsampling strategy, demonstrating the influence of data composition on performance. The distinct medians indicate that subsampling strategy is a critical factor in fine-tuning outcomes. 4 DISCUSSION AND CONCLUSION Our results clearly demonstrate that incorporating data composition via the dataset volume V = N · L yields a scaling law that predicts fine-tuning performance under fixe… view at source ↗

**Figure 4.** Box plots of accuracy for each subsampling strategy showing the same trend as BRICC. We tested many of the other MMLU tracks and found this is consistent [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 1 canonical work pages

[1]

Open llm leaderboard

Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023

2023
[2]

Measuring massive multitask language understanding, 2021

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

arXiv 2021
[3]

Scaling laws for transfer, 2021

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL https://arxiv.org/abs/2102.01293

arXiv 2021
[4]

Rae, Oriol Vinyals, and Laurent Sifre

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

arXiv 2022
[5]

Unveiling the secret recipe: A guide for supervised fine-tuning small llms, 2024

Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, and Akash Srivastava. Unveiling the secret recipe: A guide for supervised fine-tuning small llms, 2024. URL https://arxiv.org/abs/2412.13337

arXiv 2024
[6]

Reducing biases towards minoritized populations in medical curricular content via artificial intelligence for fairer health outcomes

Chiman Salavati, Shannon Song, Willmar Sosa Diaz, Scott A Hale, Roberto E Montenegro, Fabricio Murai, and Shiri Dori-Hacohen. Reducing biases towards minoritized populations in medical curricular content via artificial intelligence for fairer health outcomes. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pp.\ 1269--1280, 2024

work page 2024
[7]

When scaling meets llm finetuning: The effect of data, model and finetuning method, 2024

Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method, 2024. URL https://arxiv.org/abs/2402.17193

arXiv 2024
[8]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Open llm leaderboard

Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023

2023

[2] [2]

Measuring massive multitask language understanding, 2021

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

arXiv 2021

[3] [3]

Scaling laws for transfer, 2021

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021. URL https://arxiv.org/abs/2102.01293

arXiv 2021

[4] [4]

Rae, Oriol Vinyals, and Laurent Sifre

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

arXiv 2022

[5] [5]

Unveiling the secret recipe: A guide for supervised fine-tuning small llms, 2024

Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, and Akash Srivastava. Unveiling the secret recipe: A guide for supervised fine-tuning small llms, 2024. URL https://arxiv.org/abs/2412.13337

arXiv 2024

[6] [6]

Reducing biases towards minoritized populations in medical curricular content via artificial intelligence for fairer health outcomes

Chiman Salavati, Shannon Song, Willmar Sosa Diaz, Scott A Hale, Roberto E Montenegro, Fabricio Murai, and Shiri Dori-Hacohen. Reducing biases towards minoritized populations in medical curricular content via artificial intelligence for fairer health outcomes. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pp.\ 1269--1280, 2024

work page 2024

[7] [7]

When scaling meets llm finetuning: The effect of data, model and finetuning method, 2024

Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method, 2024. URL https://arxiv.org/abs/2402.17193

arXiv 2024

[8] [8]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...