COSMOS: Predictable and Cost-Effective Adaptation of LLMs

Aws Albarghouthi; Frederic Sala; Jiayu Wang

REVIEW 4 major objections 6 minor 42 references

COSMOS: Predictable and Cost-Effective Adaptation of LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read COSMOS claims that the performance and cost of QLoRA fine-tuning and retrieval-augmented in-context learning can be predicted cheaply enough to replace exhaustive strategy search, with an average prediction error of 1.09 percentage points…

desk verdict Useful framework and broad experiments, but the headline QLoRA accuracy and cost claims rest on a test-set selection protocol and underspecified calibration that need fixing before I'd trust them. read the letter →

arxiv 2505.01449 v1 pith:ZGKGLTFH submitted 2025-04-30 cs.LG cs.AI

Jiayu Wang , Aws Albarghouthi , Frederic Sala This is my paper · ORCID

classification cs.LGcs.AI

keywords LLMadaptationstrategyselectionperformancepredictioncostQLoRAfine-tuningretrieval-augmentedin-contextlearningscalinglawsproxymodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models can be adapted to a task in many ways, and choosing the best model–strategy combination normally means running expensive trials. This paper asks whether both the performance and the cost of adaptation can be predicted instead, and it answers yes with COSMOS, a framework that couples cheap, strategy-specific predictors. For QLoRA fine-tuning, a linear probe on frozen embeddings is calibrated to real fine-tuning accuracy; for retrieval-augmented in-context learning, an exponential saturation curve is fit from as few as two shot-count measurements. Across 55 strategy combinations on eight benchmarks, the predicted best strategy misses the actual best by 1.09 percentage points on average while cutting the cost of selection by 92.72% (up to 98.71%). If the prediction mechanism transfers beyond these benchmarks, model and strategy choice becomes a cheap data-driven decision rather than an exhaustive search.

What carries the argument

The load-bearing object is the predictor pair $(P_{j,k}, C_{j,k})$ inside a strategy navigator: a performance predictor and a cost predictor for each model–strategy pair. For fine-tuning, the performance predictor is an embedding-augmented linear probe trained on frozen bidirectional representations and calibrated by $\hat{\pi} = a\,\pi_{\phi''}+b$, where the coefficients are learned on a small validation subset. For in-context learning, the performance predictor is the exponential saturation curve $\hat{\pi}(d)=\alpha(1-e^{-\beta d})+\pi_0$, fit to sparse shot-count measurements. These predictors are what let the framework score all configurations without running them; the cost model makes the comparison concrete by converting GPU time and token use into dollars.

What would settle it

Take a task outside the eight benchmarks, learn the linear rescaling coefficients on a 10% validation subset, predict QLoRA accuracy for all data portions from 0.1 to 1.0 and all training iterations from 4 to 8, then run those configurations; if the predictions miss by more than roughly two percentage points on average, or if coefficients learned on one task do not reduce error on another task, the central transfer assumption fails. Likewise, the exponential curve fit from one-shot and eight-shot measurements should predict four-shot and sixteen-shot performance; a persistent bias there would sink the in-context-learning predictor.

Watch

Extended reading notes

Core claim

COSMOS formalizes strategy selection as choosing a model, an adaptation strategy, and a configuration to maximize a score that trades off performance against cost, and it replaces exhaustive evaluation with a pair of predictors for each model–strategy combination. For QLoRA, the performance predictor turns a causal language model into a bidirectional encoder, trains a linear projector on frozen embeddings, and calibrates the projector's accuracy to true fine-tuning accuracy through an affine map $\hat{\pi} = a\pi_{\phi''}+b$ with coefficients learned on a small validation subset. For retrieval-augmented in-context learning, the predictor fits the exponential saturation law $\hat{\pi}(d)=\alpha(1-e^{-\beta d})+\pi_0$ to one-shot and eight-shot measurements and then extrapolates to other shot counts. The cost predictor accounts for adaptation, evaluation, and prediction costs, so both strategies are compared on a common dollar axis. The experiments show that the predicted optimal strategy matches the actual optimal strategy with 1.09 percentage points mean absolute error while spending about 7% of the exhaustive-evaluation cost.

Load-bearing premise

The whole prediction pipeline rests on the assumption that real fine-tuning accuracy tracks a cheap linear probe on frozen word embeddings through one fixed linear rescaling, and that the rescaling learned from a small validation set keeps working across every data size, training length, and task; the paper demonstrates this only on the eight tasks it evaluates.

Editorial extensions

If this is right

A practitioner can select a near-optimal model–strategy configuration by running only lightweight proxy and validation experiments, not the full grid of fine-tuning and prompting trials.
Fine-tuning and in-context learning can be compared on one performance-cost axis, so the choice between them becomes a resource-level decision rather than a default preference.
Expanding model routing to include adaptation strategies moves the Pareto frontier, meaning a router can offer better accuracy at the same cost or lower cost at the same accuracy.
Cost savings grow with task scale and with the number of configurations searched, so the method becomes more valuable as the strategy pool grows.
The linear calibration and exponential saturation predictors together give an end-to-end estimate of total deployment cost, including prediction, adaptation, and evaluation, before any expensive run is started.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An untested extension: the same frozen-embedding calibration could be tried on full fine-tuning, unquantized LoRA, or adapter methods; if the linear transfer holds there, one predictor would cover most training-time strategies.
Another extension the paper leaves implicit: the exponential saturation law is fixed in form and fit to only two points, so a small learned prior over tasks could allow ICL gains to be predicted before running any few-shot measurements at all.
The headline cost reduction is computed under specific GPU-hour and token prices, so real-world savings scale with those prices and with the size of the configuration grid; the 92.72% figure is an illustration of the mechanism, not a universal constant.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Let me give you the short version. COSMOS is a sensible engineering package: linear probes on frozen embeddings for QLoRA, an exponential saturation fit for retrieval-augmented ICL, and a clean formalization of model-strategy selection under cost. The empirical study is broad—eight tasks, 55 strategy combinations, an expanded model pool—and the ICL results look genuinely solid. The 1.09% MAE and 92.72% cost reduction, however, are not trustworthy out of the box because the QLoRA predictor is selected on test labels, and the calibration cost is not auditable.

What's actually new: the unified framework that lets you compare training-time and test-time adaptation under one cost model. The individual predictors are known components, but the packaging and the cost analysis are a real contribution. The ICL scaling-law predictor is honest and effective: fitting an exponential to 1-shot and 8-shot actual runs and predicting 2, 4, 16 is a legitimate extrapolation check, and the held-out errors are small.

Now the soft spots, in order of severity. First, Appendix C.1 says the contrastive linear projector is selected on 'peak test accuracy over 300 iterations.' That is using the same test labels that later define actual fine-tuning accuracy, so the reported MAE and near-oracle selection are not unbiased. The authors need to rerun with checkpoints chosen on validation only. Second, the calibration of a and b is underspecified: how many actual QLoRA runs are required, on which data portions, and are those runs included in the reported 'Ours Cost'? Without that, the 92.72% CRR cannot be verified. Third, results are averaged over three seeds but no error bars are shown; that's a minor fix.

The paper's formalization and empirical scope make it worth a serious referee, but the QLoRA prediction claim needs a clean re-evaluation before I would trust it. This is a paper for practitioners who need a cheap way to choose between fine-tuning and ICL under a budget, and for researchers working on cheap model and strategy selection. I would not cite it in its current form, but I would bring it to reading group as a case study in evaluation pitfalls. My recommendation: send to peer review with a required fix for the checkpoint selection and a full disclosure of calibration costs and code.

Referee Report

4 major / 6 minor

Summary. The paper formalizes the model--strategy selection problem for LLMs and proposes COSMOS, a unified prediction framework that estimates both performance and cost of adaptation strategies. For QLoRA fine-tuning, COSMOS trains a lightweight linear probe on frozen bidirectional embeddings and calibrates probe accuracy to actual fine-tuning accuracy through a linear map (Section 4.2). For retrieval-augmented ICL, it fits an exponential saturation curve to performance at a small number of shot counts (Section 4.3). Experiments across eight benchmarks with 55 QLoRA/ICL configurations report a mean absolute error of 1.09% and an average cost reduction of 92.72% (Table 1), with appendices extending the evaluation to five model families, limited data access, and a GPT-4o fine-tuning cost case study.

Significance. If the reported protocol is clean, the paper makes a practically valuable contribution: a cheap proxy-plus-calibration method for comparing training-time and test-time adaptation strategies, accompanied by a transparent cost model. The empirical scope is substantial, covering eight tasks, two strategy families, and up to 275 configurations, and the limited-data and expanded-model appendices are useful robustness checks. The two prediction assumptions---linear proxy-to-fine-tuning calibration and exponential ICL saturation---are simple and falsifiable. However, as written, the manuscript does not establish that the headline numbers come from an unbiased evaluation protocol: checkpoint selection on test accuracy and the inclusion of calibration points in the ICL evaluation undermine the central accuracy and cost-savings claims until corrected.

major comments (4)

[Appendix C.1 / Section 4.2] Appendix C.1 states that for the four general-domain tasks the contrastive linear projector is selected by 'peak test accuracy over 300 iterations,' while Appendix B states that all adapted models are evaluated on the test set. The projector accuracy pi_phi'' is the input to the calibration pi_hat = a*pi_phi'' + b in Section 4.2, and the actual fine-tuning accuracies to which it is calibrated are test-set accuracies. Choosing the projector checkpoint with the same test labels therefore constitutes selection on the test set: it can only improve the apparent predicted-actual agreement in Table 1 and Figures 6-8 relative to an honest validation-based protocol. The main claims of a 1.09% MAE and near-oracle strategy selection are not clean evidence of transfer to a new task. Please rerun the pipeline with checkpoints selected on the validation split only; if 'test accuracy' was intended to mean the 10% validation split, this must be stated explicitly and the implications for the reported protocol must be addressed.
[Section 4.3 / Appendix C.2] The ICL predictor fixes pi0 as the lower of zero-shot and 1-shot performance and then fits alpha and beta to the measured 1-shot and 8-shot points (Appendix C.2). With two free parameters and two calibration points, the fitted curve passes exactly through the 1-shot and 8-shot measurements, so predictions at those shot counts are trivially accurate. Because the 55-configuration grid in Section 5 includes 1-, 2-, 4-, 8-, and 16-shot configurations, the reported aggregate MAE and selection results include configurations whose 'prediction' is a restatement of measured data. Please report held-out shot counts (e.g., 2, 4, 16) or use a leave-one-out protocol for the curve fit.
[Section 4.2 / Appendix C.1] The calibration of a and b is underspecified. Learning two parameters requires paired actual fine-tuning outcomes, but the manuscript does not state how many QLoRA runs were performed for calibration, on which data portions and iteration counts they were run, or whether the same a and b are applied to all configurations. Since these calibration runs are genuine fine-tuning runs, their cost must be included in 'Ours Cost' in Tables 1 and 3; otherwise the reported 92.72% cost reduction is overstated. Please specify the calibration budget explicitly and account for it in the cost model, or justify why the calibration cost is negligible.
[Section 5 / Tables 1 and 3] All headline numbers are averages over three seeds, but no error bars, standard deviations, or confidence intervals are reported for the MAE and CRR values in Tables 1 and 3. Figure 4 shows substantial seed variance in low-cost regimes, so it is not possible to assess whether the 1.09% MAE or the per-task CRR differences are stable. Please report variability for the main metrics.

minor comments (6)

[Section 3.2] There is a typesetting error in the cost-efficiency condition: 'cpredict(Pj,k,Cj,k) much less than cadapt(T_j^omega, f_k)' is missing the subscript on the adaptation cost; the claim should read cpredict(Pj,k,Cj,k) << cadapt(T_j^omega, f_k).
[Table 4 / Tables 1 and 3] Table 4 uses fractional accuracy values (e.g., 0.944) while Tables 1 and 3 use percentages (e.g., 94.4); please unify the units across all tables.
[Appendix D.1] The terms 'Range Avg Acc' and 'Range Avg Cost' are introduced without definition; please define them in the caption or text so the reader can interpret the baseline comparison.
[Figure 5] Figure 5 appears to be a schematic; please state explicitly whether the old and new Pareto frontiers are computed from the experimental data or are illustrative.
[Appendix C.1] The calibration set size is described as '200 examples or 10% of the training set, whichever is larger,' but the manuscript does not specify which of these was used for each task; please list the calibration set size per task in the appendix.
[Section 5.2] The statement that FPB achieves 'MAE of 0.007 for both performance and cost predictions' should specify the unit for the cost MAE (dollars) to avoid ambiguity.

Circularity Check

1 steps flagged · score 4.0 of 10

One partial circularity: the ICL exponential fit is evaluated on its own fitting points (1-shot and 8-shot), so the headline MAE is partly in-sample; the held-out ICL and QLoRA predictions remain independent.

fitted input called prediction [Appendix C.2; Section 5 Setup; Appendix E.3]
"Building on this finding, we fit an exponential saturation function (Eq. 5) using performance measurements from 1-shot and 8-shot settings. ... For ICL, we vary the number of retrieved demonstrations∈ {1, 2, 4, 8, 16} ... This results in 55 distinct transformation combinations."

Equation (5) has two free shape parameters α and β, with π0 fixed as the lower of zero-shot and 1-shot accuracy. The curve is fit to the measured 1-shot and 8-shot accuracies, so those two points are in-sample: with two parameters and two constraints the fit interpolates them exactly, making their prediction error zero by construction. The MAE evaluation averages over shot counts {1,2,4,8,16}, so d=1 and d=8 are included in the reported ICL MAE (0.85%) and in the aggregate 1.09% MAE. The headline metrics therefore mix fit quality on the fitting inputs with genuine held-out prediction at d=2,4,16, which are not separated.

full rationale

COSMOS's derivation chain is otherwise self-contained. The QLoRA predictor is a frozen-embedding linear probe with an affine calibration (π_hat = aπ_φ'' + b) learned from a small validation set; the evaluation compares predicted and actual outcomes across data portions and iterations, so held-out configurations provide independent evidence. The strategy-selection protocol (predict scores, then compare actual performance of the predicted-best strategy to the true best) is a legitimate decision-rule evaluation. The only demonstrated circularity is the ICL evaluation's inclusion of its own fit points, which inflates the MAE and cost-reduction headline numbers. Two non-circular but important protocol concerns remain: Appendix C.1 selects the linear projector checkpoint on 'peak test accuracy over 300 iterations,' which uses test labels and can bias the reported MAE favorably; and the paper does not state how many actual QLoRA runs are used to fit a and b, leaving the calibration cost and the possible overlap between calibration configurations and evaluated configurations underspecified. These do not reduce the core claim to a definitional identity, but they mean the headline numbers should be re-reported on strictly held-out points.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework's claimed prediction accuracy and cost reduction rest on a small number of fitted parameters (calibration and scaling-law coefficients) and on strong transfer and linearity assumptions. These are not derived from first principles or validated on held-out tasks. The paper introduces a framework named COSMOS but no new physical entities, forces, or conserved quantities, so the invented-entities ledger is empty.

free parameters (5)

a, b (QLoRA calibration coefficients) = not reported
Linear mapping from frozen-embedding linear probe accuracy to predicted QLoRA fine-tuning accuracy; learned per task on a small validation set (Section 4.2). The core performance predictor for fine-tuning relies on these fitted values.
alpha, beta, pi0 (ICL exponential saturation parameters) = not reported
Parameters of Eq. 5 fitted to 1-shot and 8-shot retrieval-augmented ICL accuracy, with pi0 taken as the min of zero-shot and 1-shot accuracy (Appendix C.2). Used to predict ICL accuracy at other shot counts.
Calibration data size (200 examples or 10% of training set) = 200 or 10%
Chosen per task to calibrate the QLoRA predictor; affects both prediction accuracy and prediction cost (Appendix C.1).
Per-step processing times (1.09 s per fine-tuning step, 0.0009 s per data point per epoch) = 1.09 s; 0.0009 s
Assumed average step times on uncontested A100 GPUs, used to convert GPU usage into dollar costs in the cost model (Appendix B). The cost reduction numbers are sensitive to these assumptions.
Hourly GPU rate and token prices = $1/h A100; $0.2/$0.1 per M tokens (Llama 3 8B/Gemma 2B)
External pricing assumptions used in cost calculations and the GPT-4o case study (Appendix B, G).

assumptions (5)

ad hoc to paper Actual QLoRA fine-tuning accuracy is a linear function of frozen-embedding linear probe accuracy, with task-specific coefficients.
Section 4.2; this linearity is assumed and is what makes the calibration extrapolate across data portions and training iterations.
ad hoc to paper Retrieval-augmented ICL accuracy follows the exponential saturation curve alpha*(1-exp(-beta*d))+pi0.
Equation 5; no derivation from first principles, chosen to fit sparse measurements.
domain assumption Calibration on a small validation subset transfers to the full data distribution and to the test set.
Sections 4.2 and C.1; the coefficients a,b are learned on a small subset, then applied to predict performance for all configurations evaluated on the test set.
domain assumption The fixed cost model (step times, GPU rates, token prices) accurately reflects real adaptation costs.
Section 3.4 and Appendix B; the 92.72% cost reduction and related CRR numbers are computed within this model.
domain assumption LLM2Vec-style bidirectional frozen embeddings contain enough task information for the linear probe to rank fine-tuning configurations.
Section 4.2; the proxy model's predictive power depends on this representation assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of COSMOS: Predictable and Cost-Effective Adaptation of LLMs." pith.science (2026). https://pith.science/paper/ZGKGLTFH

@misc{pith2026250501449,
  author       = {Pith},
  title        = {Pith review of: COSMOS: Predictable and Cost-Effective Adaptation of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZGKGLTFH}},
  note         = {Machine review of arXiv:2505.01449}
}

read the original abstract

Large language models (LLMs) achieve remarkable performance across numerous tasks by using a diverse array of adaptation strategies. However, optimally selecting a model and adaptation strategy under resource constraints is challenging and often requires extensive experimentation. We investigate whether it is possible to accurately predict both performance and cost without expensive trials. We formalize the strategy selection problem for LLMs and introduce COSMOS, a unified prediction framework that efficiently estimates adaptation outcomes at minimal cost. We instantiate and study the capability of our framework via a pair of powerful predictors: embedding-augmented lightweight proxy models to predict fine-tuning performance, and low-sample scaling laws to forecast retrieval-augmented in-context learning. Extensive evaluation across eight representative benchmarks demonstrates that COSMOS achieves high prediction accuracy while reducing computational costs by 92.72% on average, and up to 98.71% in resource-intensive scenarios. Our results show that efficient prediction of adaptation outcomes is not only feasible but can substantially reduce the computational overhead of LLM deployment while maintaining performance standards.

Figures

Figures reproduced from arXiv: 2505.01449 by the authors.

**Figure 2.** Predicted vs. actual performance-cost analysis for QLoRA fine-tuning. Each plot compares actual ( [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Predicted vs. actual performance-cost analysis for retrieval-based ICL. Each plot compares actual ( [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Actual QLoRA vs. ICL performance-cost trajectories across diverse tasks. Each plot presents the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Benefit of adaptation-augmented routing. The [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Predicted vs. actual performance-cost analysis for QLoRA fine-tuning across eight diverse tasks. Each [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: Predicted vs. actual performance-cost analysis for retrieval-based ICL across eight diverse tasks. Each [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: Scatter plots comparing predicted vs. actual performance (accuracy) for QLoRA fine-tuning across eight [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

**Figure 9.** Figure 9: Scatter plots of predicted vs. actual cost for QLoRA fine-tuning across eight diverse tasks. The near-perfect [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

**Figure 10.** Figure 10: Scatter plots comparing predicted vs. actual accuracy for ICL across eight diverse tasks. Axes are [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 11.** Figure 11: Scatter plots of predicted vs. actual cost for ICL across eight diverse tasks. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

**Figure 12.** Figure 12: Actual QLoRA vs. ICL performance-cost trajectories across eight diverse tasks. Each plot presents the [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 11 canonical work pages

[1]

Llm2vec: Large language models are secretly powerful text encoders

BehnamGhader, P., Adlakha, V ., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961, 2024

arXiv 2024
[2]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020

2020
[3]

Data shunt: Collaboration of small and large models for lower costs and better performance

Chen, D., Zhuang, Y ., Zhang, S., Liu, J., Dong, S., and Tang, S. Data shunt: Collaboration of small and large models for lower costs and better performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11249–11257, 2024

work page 2024
[4]

Frugalgpt: How to use large language models while reducing cost and improving performance

Chen, L., Zaharia, M., and Zou, J. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023

arXiv 2023
[5]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

arXiv 2018
[6]

Training verifiers to solve math word problems, 2021.URL https://arxiv

Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems, 2021.URL https://arxiv. org/abs/2110.14168, 2021

arXiv 2021
[7]

Qlora: Efficient finetuning of quantized llms

Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

2024
[8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024

Show all 42 references

[9]

M., Wen, Y ., Zhang, W., and Wang, J

Feng, X., Wan, Z., Wen, M., McAleer, S. M., Wen, Y ., Zhang, W., and Wang, J. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.17179, 2023

2023 arXiv
[10]

S., Love, J., et al

Gemma, T., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

2024 arXiv
[11]

G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al

Gemma, T., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. 10

2024 arXiv
[12]

Measuring massive multitask language understanding

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

2009 arXiv
[13]

A., Welbl, J., Clark, A., et al

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35:3001...

2022
[14]

J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[15]

J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., and Upadhyay, S

Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., and Upadhyay, S. K. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403.12031, 2024

2024 arXiv
[16]

Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL http...

2023 arXiv
[17]

B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

2001 arXiv
[18]

The power of scale for parameter-efficient prompt tuning

Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, Online a...

2021 doi
[19]

Let’s verify step by step

Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023

2023 arXiv
[20]

Grammar-aligned decoding, 2024

Park, K., Wang, J., Berg-Kirkpatrick, T., Polikarpova, N., and D’Antoni, L. Grammar-aligned decoding, 2024. URL https://arxiv.org/abs/2405.21047

2024
[21]

Qwen2 technical report

Qwen, T. Qwen2 technical report. 2024

2024
[22]

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1–67, 2020

2020
[23]

Rethinking neural operations for diverse tasks

Roberts, N., Khodak, M., Dao, T., Li, L., Ré, C., and Talwalkar, A. Rethinking neural operations for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2021

2021
[24]

Roberts, N., Guo, S., Gao, Z., GNVV , S. S. S. N., Cromp, S., Wu, C., Duan, C., and Sala, F. Pretrained hybrids with mad skills, 2024

2024
[25]

The probabilistic relevance framework: Bm25 and beyond

Robertson, S., Zaragoza, H., et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009

2009
[26]

J., and Hashimoto, T

Ruan, Y ., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938, 2024

2024 arXiv
[27]

G., Natarajan, S., Maru, N., Todorov, H., Guha, E., Buchanan, E

Saad-Falcon, J., Lafuente, A. G., Natarajan, S., Maru, N., Todorov, H., Guha, E., Buchanan, E. K., Chen, M., Guha, N., Ré, C., et al. Archon: An architecture search framework for inference-time techniques. arXiv preprint arXiv:2409.15254, 2024

2024 arXiv
[28]

L., Bhagavatula, C., and Choi, Y

Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

2021
[29]

Fly-swat or cannon? cost-effective language model choice via meta- modeling

Šakota, M., Peyrard, M., and West, R. Fly-swat or cannon? cost-effective language model choice via meta- modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 606–615, 2024

2024
[30]

M., Staten, C., Khodak, M., Neubig, G., and Talwalkar, A

Shen, J., Li, L., Dery, L. M., Staten, C., Khodak, M., Neubig, G., and Talwalkar, A. Cross-modal fine-tuning: Align then refine. In International Conference on Machine Learning (ICML), 2023

2023
[31]

Large language model routing with benchmark datasets

Shnitzer, T., Ou, A., Silva, M., Soule, K., Sun, Y ., Solomon, J., Thompson, N., and Yurochkin, M. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789, 2023. 11

2023 arXiv
[32]

Scaling llm test-time compute optimally can be more effective than scaling model parameters

Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

2024 arXiv
[33]

Fine-tuning and prompt optimization: Two great steps that work better together

Soylu, D., Potts, C., and Khattab, O. Fine-tuning and prompt optimization: Two great steps that work better together. arXiv preprint arXiv:2407.10930, 2024

2024 arXiv
[34]

and Zhou, D

Wang, X. and Zhou, D. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200, 2024

2024 arXiv
[35]

Y ., Guu, K., Yu, A

Wei, J., Bosma, M., Zhao, V . Y ., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V . Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

2021 arXiv
[36]

V ., Zhou, D., et al

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V ., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824–24837, 2022

2022
[37]

From decoding to meta-generation: Inference-time algorithms for large language models

Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, Z. From decoding to meta-generation: Inference-time algorithms for large language models. arXiv preprint arXiv:2406.16838, 2024

2024 arXiv
[38]

Pixiu: A large language model, instruction data and evaluation benchmark for finance

Xie, Q., Han, W., Zhang, X., Lai, Y ., Peng, M., Lopez-Lira, A., and Huang, J. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443, 2023

2023 arXiv
[39]

L., Cao, Y ., and Narasimhan, K

Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y ., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxiv. org/pdf/2305.10601. pdf, 2023

2023 arXiv
[40]

and Zhu, H

Yu, T. and Zhu, H. Hyper-parameter optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05689, 2020

2003 arXiv
[41]

Large language model cascades with mixture of thoughts representations for cost-efficient reasoning

Yue, M., Zhao, J., Zhang, M., Du, L., and Yao, Z. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. arXiv preprint arXiv:2310.03094, 2023

2023 arXiv
[42]

Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. 12 Appendix The appendix is organized as follows: We first detail the datasets (Appendix A) and experimental setup (Ap-...

1905 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Llm2vec: Large language models are secretly powerful text encoders

BehnamGhader, P., Adlakha, V ., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961, 2024

arXiv 2024

[2] [2]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020

2020

[3] [3]

Data shunt: Collaboration of small and large models for lower costs and better performance

Chen, D., Zhuang, Y ., Zhang, S., Liu, J., Dong, S., and Tang, S. Data shunt: Collaboration of small and large models for lower costs and better performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11249–11257, 2024

work page 2024

[4] [4]

Frugalgpt: How to use large language models while reducing cost and improving performance

Chen, L., Zaharia, M., and Zou, J. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023

arXiv 2023

[5] [5]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

arXiv 2018

[6] [6]

Training verifiers to solve math word problems, 2021.URL https://arxiv

Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems, 2021.URL https://arxiv. org/abs/2110.14168, 2021

arXiv 2021

[7] [7]

Qlora: Efficient finetuning of quantized llms

Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

2024

[8] [8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024