Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Routing queries to cheap models with best-of-n sampling can cut LLM serving cost by up to 60% while keeping response quality within 0.8% of GPT-4o.

desk verdict Genuinely new combination of routing and best-of-n, but the headline cost-quality claim is only measured under the same reward model used for training; needs independent validation. read the letter →

arxiv 2506.22716 v1 pith:SQI556FQ submitted 2025-06-28 cs.LG cs.AIcs.CLcs.DB

Dujian Ding , Ankur Mallick , Shaokun Zhang , Chi Wang , Daniel Madrigal , Mirian Del Carmen Hipolito Garcia , Menglin Xia , Laks V.S. Lakshmanan
show 2 more authors
Qingyun Wu Victor Rühle
This is my paper
classification cs.LGcs.AIcs.CLcs.DB
keywords LLMroutingbest-of-nsamplingtest-timecomputeproxyrewardmodelcost-qualitytrade-offmulti-headroutermatchprobability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BEST-Route tries to settle a practical question: when must a serving system pay for a powerful model like GPT-4o on every query, and when is a cheaper model enough? Its answer is that a cheap model can often be made as good as the expensive one by sampling several responses and keeping the best, and that a router can predict ahead of time which (model, sample count) combination will match the reference quality. The paper trains a multi-head router to output, for each cheaper model and each value of n, the probability that best-of-n beats a single reference response; at inference time it picks the cheapest combination whose probability clears a user-set threshold, and falls back to the reference model when none qualifies. On a 10K-example mix of question answering, coding, and safety tasks, the paper reports up to 60% cost reduction with less than 1% response-quality drop, with the trade-off tunable by threshold. If this is right, LLM services could cut GPU spend without swapping models.

What carries the argument

The load-bearing machinery is the match-probability multi-head router plus the proxy reward model. The router uses a shared DeBERTa-v3-small backbone that encodes the query once, and $K$ by $N$ lightweight heads that each output $p_{k,n}(q)$, the probability that best-of-n responses from model $k$ match or beat a single GPT-4o response; it is trained on labels $y_n(q)$ that are ground-truth indicators of that event. The proxy reward model is a DeBERTa-v3-large fine-tuned on worst/median/best pairs from 20 responses per training query with the logistic pairwise ranking loss, and it selects the best of the sampled responses at inference. Algorithm 1 combines these pieces: predict match probabilities, filter by threshold t, estimate costs from input and output token prices and average output lengths, and return the highest-proxy-score response from the cheapest valid combination.

What would settle it

Take held-out queries, generate n=1,5,20 responses from a cheap model, score them with the proxy model and with a more authoritative judge such as human raters or LLM-as-judge, and compare the proxy-best response's judge score against a random sample's judge score; if the gap does not rise with n or turns negative on a nontrivial fraction of queries, the proxy-ranking premise fails. Separately, count how often combinations predicted above threshold t actually beat a single GPT-4o response on a set with known outcomes; systematic overprediction would break the reported cost-quality trade-off on other query distributions.

Watch

Extended reading notes

Core claim

The central claim is that the cost-quality ordering of LLMs is not fixed: a small model combined with best-of-n sampling sits on a new frontier that a router can exploit. Concretely, BEST-Route presents an algorithm that, for each query, predicts a match probability for every (cheap model, sample count) pair against a powerful reference model, filters to pairs whose predicted probability clears a threshold t, and executes the cheapest of those pairs; if no pair clears t, it uses the reference model once. Cost is estimated from token prices and average output length, and the best response is selected by a lightweight proxy reward model trained with a pairwise ranking loss. The paper reports experiments where this achieves up to 60% cost reduction with only a 0.8% quality drop measured by armoRM on the in-distribution test set, and a 1.59% drop on the out-of-distribution MT-Bench set, with negative quality drops on coding queries when a specialized model is added to the pool.

Load-bearing premise

The load-bearing premise is that the proxy reward model ranks the sampled responses in the same order as the true quality score on the queries the router serves; if that ordering is wrong, best-of-n sampling no longer improves quality and the router's cost savings vanish.

Editorial extensions

If this is right

  • A serving system can keep its model portfolio unchanged and still reduce cost by letting the router decide how many responses to sample from each cheap model.
  • The threshold t gives operators a tunable cost-quality dial: lower thresholds favor cheap combinations, higher thresholds protect quality by falling back to the reference model.
  • Router overhead is small enough for real-time use: at n=20, match-probability prediction takes 0.04s and best-of-n scoring adds 0.58s, roughly 18.7x faster than the fastest local model evaluated.
  • Specialized cheaper models can be mixed into the pool without changing the routing algorithm; adding Codestral-22b on coding queries produced better-than-GPT-4o quality at 20% lower cost.
  • The approach carries over to out-of-distribution data and other quality metrics, with MT-Bench quality drop of 1.59% at 60% cost reduction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the router separates a shared query encoder from per-(model, n) heads, adding a new cheap model may only require training new lightweight heads; the paper does not demonstrate this transfer, but the architecture invites it.
  • The threshold rule assumes predicted match probabilities are calibrated well enough that crossing t means the cheap combination is genuinely as good as the reference; a deployment should track realized match rates to detect silent miscalibration.
  • The same proxy-reward best-of-n mechanism could be applied as a test-time compute policy for a single large model, deciding per query how many samples to draw; the paper studies it only in the routing setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes BEST-Route, a routing framework that combines a multi-head router with best-of-n sampling. For each query, the router predicts, for each small model and sample count n, the probability that best-of-n sampling from that model matches the quality of a single GPT-4o response; it then selects the cheapest (model, n) combination whose predicted match probability exceeds a threshold t, falling back to GPT-4o when no combination qualifies. The selected responses are ranked by a fine-tuned proxy reward model, which is itself trained on pairwise preferences derived from armoRM scores. Experiments on a new 10K-example dataset and on MT-Bench report up to 60% cost reduction with less than 1% armoRM score drop, and additional BLEU/ROUGE results. The paper also analyzes latency overhead, model usage before/after adding a specialized coding model, and cost-estimation error.

Significance. If the reported results hold, BEST-Route would be a practically useful contribution: it demonstrates that adaptive best-of-n sampling can extend the Pareto frontier of cost versus quality in LLM serving, and the multi-head router design is a sensible way to avoid training K×N independent routers. The paper also ships a concrete system design, detailed cost model, and a new evaluation dataset, which are valuable assets. However, the headline claim is currently supported mainly by armoRM, which is also the training reward for every learned component; the independent BLEU/ROUGE evidence shows much larger quality drops. The missing reporting of the threshold tuning procedure and the absence of error bars make the quantitative claims difficult to interpret. The central idea is defensible, but the evidence needs to be strengthened before the stated conclusions are established.

major comments (5)
  1. [Section 3.3, Section 4.1, Section 4.2, Table 1] The evaluation metric armoRM is also the training reward for both the proxy reward model and the router. armoRM is used as R_GT to construct the pairwise training set for R_proxy (Section 4.1), as R_GT in the router label y_n(q) (Section 4.2, Eq. 3), and as the response-quality metric in Tables 1–3. The headline claim of '0.8% quality drop' at 60% cost reduction therefore measures closeness to the training objective, not quality as judged by an independent criterion. The paper's own Table 4 shows that under BLEU and ROUGE the same operating point has 18.07% and 21.97% drops, respectively. To support the claimed '<1% performance drop,' the authors should provide an independent evaluation (for example, human judgments, task accuracy, or an out-of-training reward model) and should present the BLEU/ROUGE results as primary evidence alongside armoRM rather than as a secondary robustness check.
  2. [Section 4.2, Eq. (3)] The label y_n(q) is defined as a probability, but the paper never specifies how this probability is estimated during training. It is not stated whether a single reference response is sampled per query, whether the event R_GT(s*_small) >= R_GT(s_ref) is treated as a binary label, whether multiple responses are averaged, or how ties are handled. Without a concrete estimator, the router training procedure is not reproducible. Please specify the exact construction of the training labels, including the number of reference samples and any smoothing or averaging used.
  3. [Algorithm 1, Section 5.2, Tables 1–4] The match probability threshold t is a free parameter, and its values are never reported. The cost-reduction operating points of 10%, 20%, 40%, and 60% in Tables 1–4 appear to be produced by adjusting t, but the paper does not state how t is chosen, whether it is tuned on the validation set, or what values correspond to each row. This is load-bearing because the claimed cost-quality trade-off is a function of a fitted operating point; without reporting the selection procedure, the comparison to baselines that do not have an equivalent tunable parameter is not apples-to-apples. Please report the threshold values and the validation-based selection protocol.
  4. [Tables 1–4] All reported cost-reduction and quality-drop numbers are single point estimates with no error bars, confidence intervals, or significance tests. The test set has only 1K examples, and differences such as 0.19% vs. 0.63% at 10% cost reduction in Table 1 may be within noise. The authors should report means and variances over multiple seeds or bootstrap resamples, at least for the main tables.
  5. [Section 5.5, Table 3, Table 4] The claim that BEST-Route shows 'robustness under distribution shifts and generalizability to alternative quality metrics' is overstated. Table 3 shows that on MT-Bench the 60% cost-reduction point has a 1.59% armoRM drop, which is already larger than the '<1%' headline; Table 4 shows much larger BLEU/ROUGE drops. The MT-Bench result is still an in-training-metric evaluation, since armoRM is the training reward, and BLEU/ROUGE are lexical overlap metrics with known weak correlation with human judgment, as the paper itself notes in Section 3.3. Please temper the generalization claim and provide an evaluation that does not rely on the training reward.
minor comments (6)
  1. [Appendix A.1] The word 'specialized' is misspelled as 'specilized' in the description of Codestral-22b.
  2. [Section 4.1, Figure 2] The y-axis label of Figure 2 reads 'Avg. armoRM score ( )' with an empty parenthetical; please complete the label or remove the empty parentheses.
  3. [Section 4.2] The symbol n is used both as a specific sample count (e.g., in Eq. 2) and as the maximum sample count in Algorithm 1. Please use distinct notation, such as N_max, to avoid confusion.
  4. [Section 4.2, Eq. (3)] The notation y_n(q) for the training label and p_n(q) for the predicted probability is confusingly similar; consider using a different symbol for the label, such as l_n(q).
  5. [Appendix C] The armoRM case study is a single anecdotal example. It is fine as an illustration, but it should not be presented as validation; a sentence clarifying its illustrative status would help.
  6. [Section 5.1] The text says 'Codes will be released upon acceptance of this work' while the code is also listed with a GitHub URL; please clarify the current availability status.

Circularity Check

1 steps flagged · score 4.0 of 10

BEST-Route's headline '<1% performance drop' is measured with armoRM, the same reward model used to generate training labels for both the proxy reward model and the router; the closed loop makes the claim metric-specific, although the routing algorithm itself is not definitionally circular.

  1. self definitional [Section 3.3; Section 4.1; Section 4.2, Eq. (3); Table 1]
    "We assess response quality using armoRM scores Wang et al. [2024a] ... To construct the training set P, we generate n = 20 sample responses S = {s1(q), s2(q), . . . , s20(q)} for each training query q and compute RGT(s(q)) using the armoRM score ... we generate a label yn(q) = P r[RGT(s∗small) ≥ RGT(sref)] (3)"

    The response-quality metric for the headline claim (armoRM, Sections 3.3 and 5) is identical to the ground-truth reward R_GT used to train the proxy reward model (Section 4.1) and to form the router's match-probability labels (Eq. 3). Thus the router is trained to predict exactly the event (small-model best-of-n armoRM score >= reference armoRM score) whose realized frequency is then reported as 'performance drop' in Table 1. With the proxy also trained on armoRM pairs, every learned component is optimized against the same score used for evaluation, so the under-1% drop at 60% cost reduction is a within-objective result rather than an independent quality measurement.

full rationale

The routing algorithm itself is self-contained: Algorithm 1 combines a trained proxy reward model, a multi-head router, and a cost model, and the mechanics of the derivation do not assume the conclusion. The reported cost savings are computed from API prices and held-out test data, and no load-bearing self-citation chain is present; the cited prior work (e.g., Ding et al. 2024) only motivates query-difficulty variation and is not used to justify the main result. However, the central quality claim is not independently validated: armoRM serves as both the evaluation metric and the supervision signal for every learned component (the proxy reward model's ranking pairs and the router's match-probability labels). This is a closed optimization loop, so the under-1% performance drop at 60% cost reduction is partially forced by construction, because the system is explicitly trained to preserve armoRM and then evaluated on armoRM. The paper's own Table 4 shows BLEU/ROUGE drops an order of magnitude larger (18.07% and 21.97% at 60% cost reduction), indicating the headline is metric-specific rather than a general quality guarantee. This is a partial circularity in the evaluation metric, not a definitional equivalence of the derivation, so the score is 4 rather than higher.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the proxy reward model's fidelity and on armoRM as the quality metric, plus the router's ability to predict match probability from text. No invented entities are introduced. The main free parameters are the threshold t and the maximum sample count N.

free parameters (3)
  • match probability threshold t = not reported
    Tunable threshold on p_{k,n} that controls the operating point; Table 1 reports cost reductions at 10-60%, implying t is set to achieve those targets.
  • maximum sample count N = 20 (proxy training), 5 (router experiments)
    The router considers n up to N; the paper uses 20 responses for training the proxy but the routing experiments appear capped at n=5, so N is a design choice affecting cost and quality.
  • average output length per model = training-split averages
    Used to estimate response cost for routing; if actual lengths deviate, cost ordering can be wrong.
assumptions (4)
  • domain assumption Proxy reward model preserves the ground-truth ranking of responses
    Section 4.1 requires Rproxy ordering to match RGT ordering; if not, best-of-n selection degrades.
  • domain assumption armoRM is a reliable proxy for human-judged response quality
    Used as ground truth RGT for training and as the evaluation metric; Appendix C argues a 0.015 score difference is meaningful.
  • domain assumption Query text alone can predict the match probability y_n(q)
    The multi-head router encodes only the query q to predict p_{k,n}(q); if output length or other unobserved features matter, the prediction is incomplete.
  • domain assumption Average training output length predicts per-query cost
    Appendix B.1 reports low average error, but the cost selection in Algorithm 1 uses the average, not query-specific estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute." pith.science (2026). https://pith.science/paper/SQI556FQ

@misc{pith2026250622716,
  author       = {Pith},
  title        = {Pith review of: BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQI556FQ}},
  note         = {Machine review of arXiv:2506.22716}
}
read the original abstract

Large language models (LLMs) are powerful tools but are often expensive to deploy at scale. LLM query routing mitigates this by dynamically assigning queries to models of varying cost and quality to obtain a desired trade-off. Prior query routing approaches generate only one response from the selected model and a single response from a small (inexpensive) model was often not good enough to beat a response from a large (expensive) model due to which they end up overusing the large model and missing out on potential cost savings. However, it is well known that for small models, generating multiple responses and selecting the best can enhance quality while remaining cheaper than a single large-model response. We leverage this idea to propose BEST-Route, a novel routing framework that chooses a model and the number of responses to sample from it based on query difficulty and the quality thresholds. Experiments on real-world datasets demonstrate that our method reduces costs by up to 60% with less than 1% performance drop.

Figures

Figures reproduced from arXiv: 2506.22716 by the authors.

Figure 1
Figure 1. System overview of BEST-Route: Best-of-n Enhanced Sampling and Test-time Route Optimization. developers and users [Yu et al., 2022]. The growing demand for LLM-backed services has spurred the development of innovative solutions to achieve efficiency without sacrificing quality. The rising costs of LLM inference have spurred efforts to develop smaller, more cost-effective models such as self-consistency Wang et al. [… view at source ↗
Figure 2
Figure 2. armoRM score of response selected through best-of- [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Routing performance results. We also examine the impact of best-of-n sampling ( [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Routing performance between GPT-4o and Phi-3-mini with best-of- [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: BEST-Route v.s. best-of-n for each single LLM. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Model usage before and after adding Codestral-22b on coding queries. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Overhead analysis. Cost Reduction (%) Response Quality Drop (armoRM score) w.r.t. always using GPT-4o (%) N-label BEST-Route 10 0.88 0.25 20 2.29 0.43 40 4.41 1.56 60 5.89 1.59 N-class 0% cost reduction with 0% quality drop. Clustering 0% cost reduction with 0% quality…
Figure 8
Figure 8. Figure 8: Routing performance results compared to model cascades baselines. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Latency and Token-Aware Test-Time Compute

    cs.LG 2025-09 conditional novelty 6.0 of 10

    A learned per-query router selects both the inference-scaling method and its compute budget to balance accuracy, token use, and latency, outperforming static strategies on math reasoning.

  2. SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach

    cs.CL 2026-07 reject novelty 5.0 of 10

    A 0.6B router trained by SFT+RL on retrieval-quality rewards reaches 0.771 NDCG@10 across 11 agents, beating intent-prompted LLMs and cutting latency by 82%.

Reference graph

Works this paper leans on

59 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Abdin, J

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Bafna, D

    P. Bafna, D. Pramod, and A. Vaidya. Document clustering: Tf-idf approach. In 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT), pages 61--66. IEEE, 2016

    work page 2016

  3. [3]

    E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots : can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness , Accountability , and Transparency , pages 610--623, Virtual Event Canada, Mar. 2021. ACM. ISBN 978-1-4503-8309-7. doi:10.1145/3442188.3445922. URL https://dl.acm.org/do...

    arXiv 2021

  4. [4]

    A global analysis of metrics used for measuring performance in natural language processing

    K. Blagec, G. Dorffner, M. Moradi, S. Ott, and M. Samwald. A global analysis of metrics used for measuring performance in natural language processing. arXiv preprint arXiv:2204.11574, 2022

    work page Pith review arXiv 2022

  5. [5]

    Brown, J

    B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. R \'e , and A. Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

  6. [6]

    L. Chen, M. Zaharia, and J. Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023

    arXiv 2023

  7. [7]

    L. Chen, J. Q. Davis, B. Hanin, P. Bailis, I. Stoica, M. Zaharia, and J. Zou. Are more llm calls all you need? towards scaling laws of compound inference systems. arXiv preprint arXiv:2403.02419, 2024 a

    arXiv 2024

  8. [8]

    Z. Chen, A. May, R. Svirschevski, Y. Huang, M. Ryabinin, Z. Jia, and B. Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12374, 2024 b

    arXiv 2024

Show all 59 references
  1. [9]

    Chuang, W

    Y.-S. Chuang, W. Fang, S.-W. Li, W.-t. Yih, and J. Glass. Expand, rerank, and retrieve: Query reranking for open-domain question answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12131--12147, 2023

    2023

  2. [10]

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  3. [11]

    D. Ding, S. Amer-Yahia, and L. Lakshmanan. On efficient approximate queries over machine learning models. Proceedings of the VLDB Endowment, 16 0 (4): 0 918--931, 2022

    2022

  4. [12]

    D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V. R \"u hle, L. V. Lakshmanan, and A. H. Awadallah. Hybrid llm: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024

    2024

  5. [13]

    D. Ding, B. Xu, and L. V. Lakshmanan. Occam: Towards cost-efficient and accuracy-aware classification inference. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  6. [14]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Elsken, J

    T. Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20 0 (1): 0 1997--2017, 2019

    1997

  8. [16]

    L. Gui, C. G \^a rbacea, and V. Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.00832, 2024

    2024 arXiv

  9. [17]

    Gupta, H

    N. Gupta, H. Narasimhan, W. Jitkrittum, A. S. Rawat, A. K. Menon, and S. Kumar. Language model cascades: Token-level uncertainty and beyond. arXiv preprint arXiv:2404.10136, 2024

    2024 arXiv

  10. [18]

    Hassibi, D

    B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293--299. IEEE, 1993

    1993

  11. [19]

    P. He, X. Liu, J. Gao, and W. Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020

    2006 arXiv

  12. [20]

    Hinton, O

    G. Hinton, O. Vinyals, J. Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2 0 (7), 2015

    2015 arXiv

  13. [21]

    Hugging face inference api

    HuggingFace. Hugging face inference api. https://huggingface.co/inference-api

  14. [22]

    Jacob, S

    B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704--2...

    2018

  15. [23]

    Jiang, X

    D. Jiang, X. Ren, and B. Y. Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023

    2023 arXiv

  16. [24]

    Jinnai, T

    Y. Jinnai, T. Morimura, K. Ariu, and K. Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment. arXiv preprint arXiv:2404.01054, 2024

    2024 arXiv

  17. [25]

    A. Kag, I. Fedorov, A. Gangrade, P. Whatmough, and V. Saligrama. Efficient edge inference by selective query. In The Eleventh International Conference on Learning Representations, 2022

    2022

  18. [26]

    S. Kim, K. Mangalam, S. Moon, J. Malik, M. W. Mahoney, A. Gholami, and K. Keutzer. Speculative decoding with big little decoder. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  19. [27]

    Lambert, V

    N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

    2024 arXiv

  20. [28]

    LeCun, J

    Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  21. [29]

    Leviathan, M

    Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR, 2023

    2023

  22. [30]

    C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81, 2004

    2004

  23. [31]

    K. Lu, H. Yuan, R. Lin, J. Lin, Z. Yuan, C. Zhou, and J. Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311.08692, 2023

    2023 arXiv

  24. [32]

    Mavromatis, P

    C. Mavromatis, P. Karypis, and G. Karypis. Pack of llms: Model fusion at test-time via perplexity optimization. arXiv preprint arXiv:2404.11531, 2024

    2024 arXiv

  25. [33]

    Nakano, J

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021

    2021 arXiv

  26. [34]

    Narasimhan, W

    H. Narasimhan, W. Jitkrittum, A. S. Rawat, S. Kim, N. Gupta, A. K. Menon, and S. Kumar. Faster cascades via speculative decoding. arXiv preprint arXiv:2405.19261, 2024

    2024 arXiv

  27. [35]

    I. Ong, A. Almahairi, V. Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665, 2024

    2024 arXiv

  28. [36]

    OpenAI. Chatgpt. https://chat.openai.com/, a

  29. [37]

    Openai platform

    OpenAI. Openai platform. https://platform.openai.com/overview, b

  30. [38]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  31. [39]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318, 2002

    2002

  32. [40]

    S akota, M

    M. S akota, M. Peyrard, and R. West. Fly-swat or cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 606--615, 2024

    2024

  33. [41]

    Shnitzer, A

    T. Shnitzer, A. Ou, M. Silva, K. Soule, Y. Sun, J. Solomon, N. Thompson, and M. Yurochkin. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789, 2023

    2023 arXiv

  34. [42]

    Skalse, N

    J. Skalse, N. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 0 9460--9471, 2022

    2022

  35. [43]

    Snell, J

    C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  36. [44]

    Srivatsa, K

    K. Srivatsa, K. K. Maurya, and E. Kochmar. Harnessing the power of multiple minds: Lessons learned from llm routing. arXiv preprint arXiv:2405.00467, 2024

    2024 arXiv

  37. [45]

    Stiennon, L

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

    2020

  38. [46]

    Treviso, J.-U

    M. Treviso, J.-U. Lee, T. Ji, B. v. Aken, Q. Cao, M. R. Ciosici, M. Hassid, K. Heafield, S. Hooker, C. Raffel, et al. Efficient methods for natural language processing: A survey. Transactions of the Association for Computational Linguistics, 11: 0 826--860, 2023

    2023

  39. [47]

    Urban, K

    G. Urban, K. J. Geras, S. E. Kahou, O. Aslan, S. Wang, R. Caruana, A. Mohamed, M. Philipose, and M. Richardson. Do deep convolutional nets really need to be deep and convolutional? arXiv preprint arXiv:1603.05691, 2016

    2016 arXiv

  40. [48]

    Vanhoucke, A

    V. Vanhoucke, A. Senior, and M. Z. Mao. Improving the speed of neural networks on cpus. 2011

    2011

  41. [49]

    H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024 a

    2024 arXiv

  42. [50]

    J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692, 2024 b

    2024 arXiv

  43. [51]

    X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023

    2023

  44. [52]

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun. Orca: A distributed serving system for \ Transformer-Based \ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521--538, 2022

    2022

  45. [53]

    M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. arXiv preprint arXiv:2310.03094, 2023

    2023 arXiv

  46. [54]

    Zhang, J

    S. Zhang, J. Zhang, D. Ding, M. H. Garcia, A. Mallick, D. Madrigal, M. Xia, V. R \"u hle, Q. Wu, and C. Wang. Ecoact: Economic agent determines when to register what action. arXiv preprint arXiv:2411.01643, 2024 a

    2024 arXiv

  47. [55]

    Zhang, J

    S. Zhang, J. Zhang, J. Liu, L. Song, C. Wang, R. Krishna, and Q. Wu. Offline training of language model agents with functions as learnable weights. In Forty-first International Conference on Machine Learning, 2024 b

    2024

  48. [56]

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  49. [57]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023

    2023 arXiv

  50. [58]

    Zoph and Q

    B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016

    2016 arXiv

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.