Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that weighting token-level losses by learned importance scores makes downstream benchmark accuracy a predictable sigmoid of validation loss.

desk verdict A clever weighted-loss idea with a solid open-source held-out check, but the headline cross-distribution transfer experiment leaks early checkpoints from the test series into the fitting, so that claim is not actually demonstrated. read the letter →

arxiv 2506.13216 v1 pith:2U2O6KMD submitted 2025-06-16 cs.CL

Qiming Ge , Shuhao Xing , Songyang Gao , Yunhua Zhou , Yicheng Zou , Songyang Zhang , Zhi Chen , Hang Yan
show 3 more authors
Qi Zhang Qipeng Guo Kai Chen
This is my paper · ORCID
classification cs.CL
keywords scalinglawsdownstreamtaskpredictiontoken-levellossCapabilitySalienceVectorlanguagemodelevaluationvalidationbenchmarksigmoidallaw
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes aim at a known gap: validation loss trends smoothly with compute, but equal loss does not mean equal downstream performance. Its diagnosis is that average token loss treats every token as equally informative, while different tokens demand different skills, or meta-capabilities. To close the gap, the paper introduces the Capability Salience Vector (CSV), a set of learned per-token importance weights that turn raw cross-entropy loss into a scalar capability score. It then fits a sigmoid from that score to benchmark accuracy and reports that the fitted curve predicts held-out models across model families and across checkpoints trained on different data distributions, with prediction error around $10^{-3}$ on open models and $10^{-4}$ during training. If the claim holds, model builders could forecast how a partially trained model will score on tasks such as MMLU, BBH, or GSM8K using only weighted loss on a small mixed validation set, without running the benchmark.

What carries the argument

The central object is the Capability Salience Vector $W=\{w_{s,i}\}$, a per-token importance weight produced by a linear scoring head $f_\theta$ on top of a frozen language model's hidden states. It is used to form the capability score $C_m$ in Eq. (1), and the downstream scaling law is the sigmoid $A_{t,m}=\gamma+\frac{1-\gamma}{1+\exp(-\alpha(C_m-\beta))}$ in Eq. (2). The load-bearing mechanism is the alternating optimization: fix $f_\theta$ and fit $\alpha,\beta$ with Levenberg-Marquardt, then fix $\alpha,\beta$ and update $\theta$ by SGD on the prediction MSE, iterating until the fit stabilizes. The vector transfers across models because the same scoring head is applied to all of them, so the weighted loss behaves as a common scale on which different models' capabilities can be compared.

What would settle it

Take a model family with a markedly different tokenizer, architecture, or data mixture from the scoring model, compute the CSV capability scores on its checkpoints using the shared weights, and check whether the sigmoid fitted on the training models predicts held-out checkpoints within the reported ~$10^{-3}$ MSE; a family that requires re-estimating $\theta$, or a held-out checkpoint that lands far off the curve, would refute the transfer claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that the relationship between language-model cross-entropy loss and downstream task accuracy becomes a clean, predictable sigmoid once loss is computed with the Capability Salience Vector instead of averaged uniformly. Given a validation text, a scoring head assigns each token a weight, and the capability score $C_m = -\frac{1}{N_c}\sum_s\sum_i w_{s,i}\log p(x_i|x_{<i})$ replaces raw loss; task accuracy is then modeled as $A_{t,m}=\gamma+\frac{1-\gamma}{1+\exp(-\alpha(C_m-\beta))}$. The paper reports that this weighted score aligns model families onto a shared curve, that the fitted sigmoid predicts held-out larger models and held-out training checkpoints, and that the learned weights are interpretable, such as highlighting reasoning-demanding tokens when the target is BBH.

Load-bearing premise

The load-bearing premise is that one scoring head, trained on a single small model's hidden states, assigns the same token-level salience weights to every other model, so a single CSV can align loss with accuracy across model families and data distributions.

Editorial extensions

If this is right

  • Downstream benchmark accuracy can be predicted from token loss alone, so a model's performance on a task can be forecast before the full evaluation is run.
  • Because the learned weights transfer across model families, a CSV fit on one set of open models can predict checkpoints from a different series trained on a different data mix.
  • Raw FLOPs and average token loss are not sufficient predictors of downstream performance; the same compute or same average loss can correspond to different accuracies.
  • The sigmoidal form means capability growth in weighted loss, not compute, is the quantity that obeys a scaling law.
  • The learned weights are inspectable, so CSV doubles as a diagnostic: for a reasoning benchmark the heavy weights fall on tokens that require combining context and step-by-step inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If token salience is truly shared across models, a CSV computed once for a benchmark becomes a reusable probe: any new checkpoint's capability score can be read off its weighted loss without any task-specific evaluation or additional optimization.
  • The same weighted-loss construction could be applied to other labels beyond accuracy, such as instruction-following, safety, or style metrics, whenever a small set of scored checkpoints is available to fit the sigmoid.
  • A natural stress test is tokenizer shift: the paper maps losses between tokenizers at character level before weighting, and whether CSV weights remain valid for very differently tokenized or non-English-heavy models is a direct open question.
  • The convergence behavior in Figure 6 suggests the optimization is doing more than curve-fitting: it is finding a projection of the loss vector on which model families order identically, which could be used to compare data mixtures before training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes the Capability Salience Vector (CSV), a learned linear weighting of token-level cross-entropy losses intended to better align validation loss with downstream task accuracy. The method trains a small linear scoring head on top of a frozen 1.8B language model, fits a sigmoidal function between the resulting capability score and task accuracy, and then applies this score to predict held-out models' performance. Experiments on six benchmarks (MMLU, BBH, GSM8K, HellaSwag, CMMLU, C-Eval) claim improved prediction mean squared error relative to unweighted token loss and label-token loss, both for larger open-source models in the same families and for checkpoints from two internally trained model series with different data mixes.

Significance. If the transfer claims held, the idea of learning token-salience weights from a small model and applying them across model families and data distributions would be a practical and low-cost way to predict downstream capabilities during training. The paper's strengths are that it attempts out-of-sample prediction on held-out model sizes and on checkpoints from different training runs, and it uses a simple, interpretable weighting scheme. However, the most distinctive claim—transfer to new data distributions—is not actually supported by the closed-source experiment as designed, and the open-source experiment is closer to within-family size extrapolation than to cross-distribution generalization. The paper also lacks uncertainty quantification and contains a per-task result that contradicts its own 'consistent' improvement claim.

major comments (4)
  1. [Section 4.2, Closed-source Model Evaluation] The closed-source experiment uses early checkpoints (the first 50k steps) from both series during CSV optimization, then uses one series as validation and the other as test. Since early checkpoints from the test series are included in the optimization, the test points are later checkpoints of the same series, making this a within-series trajectory extrapolation rather than a transfer to an unseen data distribution. This is load-bearing because the paper's central claim is that CSV transfers to checkpoints trained on different data distributions. The experiment should be re-run with the test series entirely excluded from optimization, or the claim should be downgraded to within-series extrapolation and a separate held-out-distribution experiment should be provided.
  2. [Table 1 and Section 4.2, Main Results] On HellaSwag, the all-token baseline achieves MSE 3.15e-3, which is lower than CSV's 4.49e-3. This contradicts the abstract and Section 4.2's claim that CSV 'consistently achieves a high correlation' and 'significantly improves predictability' across benchmarks. The paper should report per-task comparisons with confidence intervals or significance tests and explicitly discuss this exception rather than presenting Table 1 without qualification.
  3. [Section 3.3, Eqs. (4)-(5) and Algorithm 2; Table 2] The capability weights theta and the sigmoid parameters alpha, beta are optimized to minimize MSE on the training models' observed accuracies. The open-source evaluation then tests only larger models from families already present in training (e.g., Llama-3-8B to 70B, Qwen2 to 72B, Gemma-2-2/9B to 27B, Yi-1.5 to 34B, InternLM2.5 to 20B). This demonstrates interpolation or modest extrapolation along model size within seen families, not generalization to unseen model families or unseen data distributions. An experiment that holds out entire families or distributions is needed to support the paper's generalization claims.
  4. [Section 3.3, Eq. (3)] The scoring head f_theta is computed from the frozen hidden states of InternLM2.5-1.8B and then applied to all evaluated models without any argument that token salience is model-independent. This is a correctness-risk concern: if token salience is specific to a model family or training distribution, the shared weight vector may not align loss with downstream accuracy across models. A concrete test would be to compute weights from hidden states of several different models and verify that the resulting capability scores are stable, or to train the scoring head on multiple models' hidden states.
minor comments (6)
  1. [Abstract and Table 1] The abstract states that the predicted mean squared error was 'maintained within the range of 1e-3', but Table 1 reports 6.32e-3 for gsm8k; this should be reworded or qualified.
  2. [Section 3.3, Algorithm 2] The selection criterion for 'the parameters that achieve the best downstream performance prediction' is not specified; please state which validation set is used for early stopping or model selection to avoid overfitting concerns.
  3. [Section 4.1, Evaluation Details] The paper does not release code or data, and it does not report random seeds, the number of optimization iterations, or convergence criteria; these details are needed for reproducibility.
  4. [Section 2 and 4] No comparison is made with existing downstream-scaling-law prediction methods such as Observational Scaling Law (Ruan et al., 2024) or the approach of Gadre et al. (2024); a discussion or baseline comparison would help position the contribution.
  5. [Throughout] There are several typos and presentational issues, including 'untritial' in the abstract, 'sa data distributions' in Section 3.1, 'Inspired of' in Section 3.2, 'Aafter' in Section 3.3, and inconsistent model names such as 'internlm2_5-20b' versus 'internlm2.5-20B'.
  6. [Figures 4-8] The rendered figures appear garbled in the manuscript text (e.g., unicode substitutions); the captions should be corrected so that axis labels and legends are readable.

Circularity Check

1 steps flagged · score 5.0 of 10

Closed-source transfer test leaks early checkpoints of the test series into CSV/sigmoid fitting, so the cross-distribution prediction is partially in-sample.

  1. fitted input called prediction [Section 4.1 (Closed-source Model Evaluation) and Section 4.2 (Downstream Task Prediction for Closed-source Models)]
    "For Capability Salience Vector optimization, we use all the open-source models along with early checkpoints (the first 50k steps) from both series. One series of our own model is used as the validation set, while the other serves as the test set."

    The claimed result is that CSV transfers to models trained on different data distributions. But the closed-source experiment places early checkpoints of both series, including the series later designated as the test set, inside the optimization set for theta, alpha, and beta (Eqs. 3-5). Consequently the fitted predictor has already observed the loss-accuracy relationship of the test series' data distribution; the 'test' checkpoints are later steps along the same training trajectory. The reported prediction is therefore within-series temporal extrapolation, not transfer to an unseen distribution, so this headline experiment partially reduces to fitting on the distribution it claims to predict.

full rationale

The core CSV construction is not circular in the definitional sense: C_m in Eq. 1 is a weighted token loss, the weights w_{s,i}=f_theta (Eq. 3) and sigmoid parameters alpha,beta are explicitly optimized against training-model accuracies (Eqs. 4-5), and the open-source experiments evaluate held-out larger models (Table 2), so those predictions are genuinely out-of-sample. The main circularity concern is confined to the closed-source transfer experiment, where early checkpoints from the test series are included in optimization; that makes the cross-distribution transfer claim partially in-sample. There is no load-bearing self-citation chain and no renaming of a known result. Overall score reflects one partially leaked prediction experiment rather than a definitional collapse.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a supervised fitting procedure: the token weights and sigmoid parameters are optimized against the training models' accuracies. The main unproven assumption is that these weights transfer across model families and data distributions. The 550-example validation set and the sigmoid functional form are additional choices made by the authors.

free parameters (4)
  • Scoring head parameters theta
    Optimized to minimize MSE between predicted and observed downstream performance on the training model set (Eq. 5). These weights define the Capability Salience Vector.
  • Sigmoid parameters alpha, beta
    Fitted per task via Levenberg-Marquardt to map capability score to accuracy (Eq. 4).
  • Validation set composition = 550 examples (100 per task plus 50 CoT)
    The choice of which examples are used to compute token losses is a design choice that affects the capability score; not derived from theory.
  • Optimization iterations and early stopping
    The paper selects parameters with the best downstream performance prediction, which is a model selection choice.
assumptions (4)
  • domain assumption Character-level loss mapping (Algorithm 1) makes token losses comparable across different tokenizers.
    Losses from different vocabularies are averaged over characters and re-tokenized to a common space; this assumes the re-tokenized character losses preserve the signal relevant to capabilities.
  • ad hoc to paper The relationship between capability score and downstream accuracy follows a sigmoid with a known random-guess floor gamma (Eq. 2).
    The sigmoid form is assumed, not derived, and gamma is set to the task's random-guess score.
  • ad hoc to paper A single linear scoring head on a frozen 1.8B model can produce transferable token importance weights.
    This is the core transferability assumption; no theoretical justification is provided.
  • domain assumption The 550 mixed validation examples sufficiently cover the meta-capabilities needed for the six benchmarks.
    The paper samples 100 examples per task (with CMMLU and CEvAL sharing one sample) plus 50 CoT examples; the adequacy of this set is empirical.
invented entities (1)
  • Capability Salience Vector
    purpose: A set of token weights that defines a weighted cross-entropy score intended to measure a specific meta-capability relevant to a downstream task.
    The construct exists only within this paper's fitting procedure; its predictive success on the paper's own held-out models is internal evidence, not an external handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law." pith.science (2026). https://pith.science/paper/2U2O6KMD

@misc{pith2026250613216,
  author       = {Pith},
  title        = {Pith review of: Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2U2O6KMD}},
  note         = {Machine review of arXiv:2506.13216}
}
read the original abstract

Scaling law builds the relationship between training computation and validation loss, enabling researchers to effectively predict the loss trending of models across different levels of computation. However, a gap still remains between validation loss and the model's downstream capabilities, making it untrivial to apply scaling law to direct performance prediction for downstream tasks. The loss typically represents a cumulative penalty for predicted tokens, which are implicitly considered to have equal importance. Nevertheless, our studies have shown evidence that when considering different training data distributions, we cannot directly model the relationship between downstream capability and computation or token loss. To bridge the gap between validation loss and downstream task capabilities, in this work, we introduce Capability Salience Vector, which decomposes the overall loss and assigns different importance weights to tokens to assess a specific meta-capability, aligning the validation loss with downstream task performance in terms of the model's capabilities. Experiments on various popular benchmarks demonstrate that our proposed Capability Salience Vector could significantly improve the predictability of language model performance on downstream tasks.

Figures

Figures reproduced from arXiv: 2506.13216 by the authors.

Figure 1
Figure 1. Relationship between computation (FLOPs) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The relationship between average token loss [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Capability salience vector establishes the relationship between cross-entropy loss and downstream task [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Downstream task prediction results of open-source models using capability score derived from Capability [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Downstream task prediction results of closed-source models using capability score derived from Capability [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Relationship between capability scores and downstream task performance across different steps of the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualization of prediction results of our [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Visualization of the importance weights as [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 8
Figure 8. Figure 8: Impact of varying validation set distributions [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages

  1. [1]

    Sanjeev Arora and Anirudh Goyal. 2023. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936

    arXiv 2023

  2. [2]

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2024. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121

    work page 2024

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  4. [4]

    Frank B Baker. 2001. The basics of item response theory. ERIC

    2001

  5. [5]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

    arXiv 2024

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  7. [7]

    LMDeploy Contributors. 2023 a . Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy

    2023

  8. [8]

    OpenCompass Contributors. 2023 b . Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    2023

Show all 40 references
  1. [9]

    Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. 2024. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796

    2024 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. 2024. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540

    2024 arXiv

  4. [12]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  5. [13]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    2022 arXiv

  6. [14]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36

    2024

  7. [15]

    Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. 2024. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177

    2024

  8. [16]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  9. [17]

    Alex Kipnis, Konstantinos Voudouris, Luca M Schulze Buschoff, and Eric Schulz. 2024. metabench --a sparse benchmark to measure general ability in large language models. arXiv preprint arXiv:2407.12844

    2024 arXiv

  10. [18]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212

    2023 arXiv

  11. [19]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2022 arXiv

  12. [20]

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. 2024. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965

    2024 arXiv

  13. [21]

    Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. 2023. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, pages 22188--22214. PMLR

    2023

  14. [22]

    Frederic M Lord and Melvin R Novick. 2008. Statistical theories of mental test scores. IAP

    2008

  15. [23]

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    David Owen. 2024. How predictable is language model benchmark performance? arXiv preprint arXiv:2401.04757

    2024 arXiv

  17. [25]

    Lorenzo Pacchiardi, Lucy G Cheke, and Jos \'e Hern \'a ndez-Orallo. 2024. 100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances. arXiv preprint arXiv:2409.03563

    2024 arXiv

  18. [26]

    Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. 2023. Efficient benchmarking (of language models). arXiv preprint arXiv:2308.11696

    2023 arXiv

  19. [27]

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992

    2024 arXiv

  20. [28]

    Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. 2024. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938

    2024 arXiv

  21. [29]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

    2022 arXiv

  22. [30]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  23. [31]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  24. [32]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  25. [33]

    Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2022. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09803

    2022 arXiv

  26. [34]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  27. [35]

    Qinyuan Ye, Harvey Yiyun Fu, Xiang Ren, and Robin Jia. 2023. How predictable are large language model capabilities? a case study on big-bench. arXiv preprint arXiv:2305.14947

    2023 arXiv

  28. [36]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    2024 arXiv

  29. [37]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  30. [38]

    Qiyuan Zhang, Fuyuan Lyu, Xue Liu, and Chen Ma. 2024. Collaborative performance prediction for large language models. arXiv preprint arXiv:2407.01300

    2024 arXiv

  31. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.