LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing

Yang Li

REVIEW 4 major objections 5 minor 5 cited by

LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single preference-conditioned bandit policy, trained once, can route queries across arbitrary sets of LLMs and adapt to any accuracy-cost preference at inference time, adding new models from as few as 20-50…

desk verdict Preference-conditioned routing with a valuable cold-start idea, but the key experiments may leak test labels: split overlap is never stated. read the letter →

arxiv 2502.02743 v1 pith:NWMUA2EN submitted 2025-02-04 cs.LG

Yang Li This is my paper · ORCID

classification cs.LG

keywords LLMroutingmulti-armedbanditpreference-conditionedpolicymodelidentityvectoritemresponsetheorycold-startadaptationmulti-objectivereinforcementlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the choice of which large language model should answer a query can be treated as a bandit problem, with a single routing policy that adapts, at inference time, to any user-specified balance between answer quality and cost. The policy is conditioned on compact model identity vectors, so the same trained router can handle arbitrary sets of candidate models, including models it has never seen. The paper claims that a new model can be added by evaluating it on only 20-50 carefully selected prompts and fitting its identity vector, and that routing with these cheap vectors nearly matches routing with vectors fitted on the full benchmark. If that claim holds, LLM deployments could swap in new models and change quality-cost preferences continuously, without retraining the router. Reported experiments include up to a 27% cost reduction, at comparable held-out accuracy, against a two-model baseline on one benchmark.

What carries the argument

The central object is the model identity vector $I_k \in \mathbb{R}^d$, a compact embedding intended to capture model $M_k$'s strengths and weaknesses across tasks. It is learned with a variational item-response-theory model $f(e_n, I_k)$ that predicts a model's score on a prompt from the prompt's pretrained embedding and the identity vector; pairwise winner data add a second head $g(e_n, I_{k_1}) - g(e_n, I_{k_2})$ and a KL regularizer keeps identity vectors near a Gaussian prior. The routing policy is the softmax $\pi_\theta(k' | x, C_K, \omega) \propto \exp(I_{k'}^\top h(x, C_K, \omega))$, where $h$ is a permutation-invariant network over the context set $C_K = \{(I_k, c_k, \hat p_k)\}_{k=1}^K$ containing each model's identity, cost, and predicted score. Conditioning on both the identity vectors and the preference $\omega$ is what lets a single trained policy act on different model sets and different trade-offs; the dot-product form means a newly computed identity vector can be inserted immediately, and the policy is trained with multi-objective PPO, supervised pretraining on pairwise comparisons, Platt-calibrated scores, on-manifold mixup, and normalisation of scores and costs within each candidate set.

What would settle it

Audit the cold-start protocol in Section 2.6 and Appendix E.9: if any of the 20-50 selected prompts also appear in the OpenLLMv2 split used to measure routing quality, the reported parity between small-budget and full-budget identity vectors could be a leakage artifact; rerunning with a provably disjoint prompt subset and with a new model family outside the training set would settle whether 20-50 prompts truly suffice.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that a single preference-conditioned routing policy, trained once on existing models and benchmark queries, generalizes to arbitrary sets of LLMs and to arbitrary user preferences, and that a new model can be added by fitting its identity vector from 20-50 selected evaluation prompts, achieving routing performance nearly matching that of identity vectors computed from the full evaluation set. The paper formulates routing as a multi-objective bandit problem in which each query is sent to a model according to a stochastic policy, and the trade-off between quality and cost is set by a preference parameter at inference time. The authors report that the approach reduces cost by up to 27% on one benchmark while keeping accuracy, and that the single preference-conditioned policy performs comparably to separately trained per-preference policies across multiple model sets.

Load-bearing premise

The load-bearing premise is that the score-prediction model trained on existing models and benchmark prompts can produce a trustworthy compact description of a brand-new model's capabilities from as few as 20-50 selected prompts, so that routing decisions on queries not used in that fitting stay nearly as good as if the new model had been fully evaluated, and that those selected prompts are not themselves part of the later test set.

Editorial extensions

If this is right

Once a model's identity vector is computed, the same trained policy can route across any set of models that includes it, without retraining the policy.
Users can change the accuracy-cost trade-off at inference time through the preference parameter, so one deployment serves applications with different requirements.
Adding a new model costs only 20-50 evaluations, cutting integration overhead by 90% or more compared with full benchmark evaluation.
The routing decision itself is light, about 5 ms and under 100 MB, so the approach is compatible with online query pipelines.
On a two-model comparison, the paper reports up to 27% cost reduction at matched accuracy, and at least parity with separately trained per-preference policies in multi-model settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence the paper does not state explicitly: if the 20-50 selected prompts overlap with the held-out test set, the cold-start parity result could be inflated, so the prompt-selection procedure should be audited for leakage.
The method presumes new models live on the same capability manifold as the training models; an out-of-manifold model, such as a code-specialised or instruction-tuned variant unlike anything in the training list, is the hard case that would stress the claim.
Because identity vectors are updated by optimizing a fixed score predictor, the same quizzing procedure could be applied incrementally as new evaluations arrive, enabling online refresh of a model's identity without retraining the router.
The preference-conditioned architecture could in principle be combined with query-level cost models that depend on input length, which the paper lists as future work, rather than fixed per-model costs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the thing you need to know about this paper: the core idea is genuinely useful and the synthesis is new—a preference-conditioned routing policy that generalizes to arbitrary model sets, with new models added by fitting an IRT-style identity vector from 20-50 quizzing prompts. That cold-start protocol is the kind of practical contribution the routing field needs. But the main empirical claim has a potential leakage problem. Section 4 says each leaderboard is split into training and test, yet Section E.9 never states whether the 20-50 prompts used to fit a new model's identity vector—or the 'full-set' baseline—come from the training split or the full benchmark. If they come from the full benchmark, the same test labels are used to construct the embedding and to score routing decisions. The prompt-selection rule in Eq. 7 uses existing models' evaluation scores, so even the choice of which prompts to quiz can leak test information. The two new families tested (Qwen2.5, aya-23) are close relatives of training models, so the generalization evidence is weak regardless. This is not a manufactured flaw; the text simply never closes the loop on disjointness.

What the paper does well: the method is plausible and the components are sensible. The related work is serious, the action-space generalization via dot-product with identity vectors is clean, and the writing is clear about what is new. The theoretical appendix is thin—Theorem A.1's continuity proof assumes uniform continuity on non-compact X, which is not established—but that is a minor fix and not load-bearing.

The other soft spots: no code, no error bars, no multiple seeds, and the headline 27% cost reduction is a single operating point on MMLU. The cold-start results in Fig. 3 lack variance measures, so we cannot tell if the gap between 50 prompts and the full set is meaningful. The PPO baseline is trained per model set and preference, which is expected, but the comparison is still favorable.

Who gets value: people working on LLM routing, model selection, or cost-efficient inference. The idea deserves referee time, but the authors need to clarify the split disjointness, release code, and report variance before the quantitative claims can be trusted. I would not cite it in my own work until that is done.

Referee Report

4 major / 5 minor

Summary. The paper proposes a preference-conditioned routing framework for LLM selection, formulated as a multi-objective contextual bandit. Model capabilities are represented by learned identity vectors (via an IRT-style model), and a single stochastic policy, conditioned on the available model set, their costs, predicted scores, and a user preference weight, is trained with PPO to trace the accuracy-cost Pareto front. The central claim is that this policy generalizes across arbitrary sets of LLMs and that new models can be added by fitting a 128-dimensional identity vector from only 20-50 selected evaluation prompts, without retraining the policy. Experiments on five benchmarks compare against RouteLLM, a Predictor baseline, per-set PPO, and an oracle, reporting up to 27% cost reduction on MMLU, and cold-start results on two unseen model families (aya-23, Qwen2.5) in Appendix E.9.

Significance. If the cold-start claim holds, the paper would make a useful practical contribution: a routing policy that adapts to new LLMs with a few dozen evaluations and to user preferences at inference time is a natural and valuable extension of existing routing work such as RouteLLM and MetaLLM. The identity-vector formulation and the Pareto-front framing are reasonable and the empirical comparisons cover several benchmarks and model sets. However, the paper currently ships no code, no seeds or error bars, and the cold-start evaluation has a potentially load-bearing data-split ambiguity, so the significance of the quantitative claims cannot yet be assessed at the level the text asserts.

major comments (4)

[Section 2.6 / Appendix E.9] The cold-start evaluation does not state whether the 10/20/50 prompts used to fit the new model identity vector (Eq. 8) and the "full-set" identity baseline are disjoint from the OpenLLMv2 test split on which routing accuracy is measured. Section 4 only says leaderboard prompts are divided into training and test splits; E.9 says identity vectors are obtained over "a selected subset of prompts from the OpenLLMv2 benchmark" without specifying that the subset is drawn from the training split. If the subset, or the prompt-selection statistics ψ_n in Eq. (7), uses test prompts, then the test labels are used both to construct the identity vector and to compute routing accuracy, in which case Figure 3 would be leakage-driven. The authors must state the split provenance of the selected and full-set prompts and rerun with an explicitly disjoint prompt-selection procedure.
[Appendix A.1 (Theorem A.1)] Theorem A.1 is not proven as stated. The proof fixes ε and asserts a δ such that |πθ(k|x) − πθ′(k|x)| < ε/K for all x and k whenever ‖θ − θ′‖ < δ; this is a uniform-in-x continuity statement, which does not follow from pointwise continuity of πθ(k|x) in θ and need not hold when X is noncompact. The theorem needs additional assumptions (for example, compactness of X with joint continuity in θ and x, or a dominated-convergence argument) before it can justify Corollary A.2. Since the existence of πω is used to motivate the preference-conditioned policy, this gap should be closed or the claim should be weakened to a heuristic statement.
[Section 4 / Figure 3] All routing results are reported as single trade-off curves with no error bars, no multiple seeds, and no significance tests. The headline "27% cost reduction" on MMLU and the claim that 50-prompt identity vectors "nearly match" full-set identity vectors are single operating points; without variance estimates, these quantitative claims are not supported. Please provide at least 3-5 independent runs (or seeds) for Figures 2-4 and report mean ± standard deviation or quartiles, especially for the cold-start results where the evaluation budget is small.
[Section 4 / Appendix E.9] The cold-start generalization claim is tested on only two new model families, aya-23 and Qwen2.5, both of which are close relatives of models already present in the training set (Cohere command-r/command for aya-23; Qwen2 for Qwen2.5). This does not substantiate the stated claim that a single policy generalizes to arbitrary sets of LLMs. At minimum, the authors should evaluate a structurally distant family (for example, a code-specialized model or a non-instruction-tuned model) and report the distribution shift, or explicitly narrow the claim to "near-distribution" new models.

minor comments (5)

[Section 4] The benchmark name is misspelled as "AlpaceEval 2.0" in the first paragraph of Section 4; it should be "AlpacaEval 2.0".
[Appendix E.9, Table E.5] The table caption says "Evaluation setting fro new routing candidates" and should be "Evaluation setting for new routing candidates".
[Figure 3 caption] The caption states that identity vectors are obtained using 10, 20, or 50 selected prompts but does not define what "full set" means in the comparison; please specify whether it is the full training split or the full benchmark.
[Section 2.4 / Appendix B.2] Equation (5) in the main text writes π ∝ exp(I⊤h), while the displayed equation in Appendix B.2 omits the exp in the first line and then restores it in the softmax; please make the two notations consistent.
[General] No code, hyperparameters for the policy network, or random-seed protocol are provided; including a code link or a detailed reproducibility appendix would substantially strengthen the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the routing policy and cold-start identity vectors are trained and evaluated on separate splits, and the paper's claims do not reduce to their inputs by construction.

full rationale

The paper's central derivation is self-contained and does not reduce to its inputs. Model identity vectors are learned by an IRT-style score predictor f(e,I) on pairwise data and leaderboard training splits (Eq. 2-4, Sec. 2.3, E.3); the routing policy (Eq. 5, B.3) is trained by PPO on training splits with actual evaluation scores, and evaluated on test splits (Sec. 4). The cold-start procedure (Eq. 7-8, Sec. 2.6, E.9) fits only the new model's identity vector on 20-50 prompts while keeping f fixed, then routes on OpenLLMv2; this is a standard fit-then-predict setup, not a fitted parameter renamed as a prediction. The 'nearly matching full-set identity' claim compares two fitted representations and is an empirical result, not a construction. The paper does not rely on self-citations: the cited multi-objective RL results (Yang et al. 2019; Basaklar et al. 2022) are external and standard. The one material concern is that Sec. 2.6/E.9 never states that the 20-50 quizzing prompts are disjoint from the OpenLLMv2 test split; if they overlap, the cold-start numbers would reflect data leakage. That is a data-split/internal-validity problem, not a derivation-equivalence circularity, and the text provides no equation or definition linking the fitted identity back to the reported test accuracy. Hence no circular step is exhibited.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The central claim rests on the representation and extrapolation assumptions of the IRT identity vectors, the scalarized-preference Pareto coverage assumption, and the cold-start prompt-selection protocol. The IRT model is a learned model with its own parameters; the numbers listed are the fitted or hand-set constants the claim depends on.

free parameters (5)

Binarization threshold eta* per dataset = Optimized via Eq. (B.2) per dataset
Real-valued evaluation scores (F1, RougeL) are binarized by an optimal threshold so that average binarized score matches the original mean; this threshold is fitted to the training scores and affects the IRT training targets.
Platt calibration parameters alpha, beta = Learned from pairwise comparison data
Used to calibrate IRT score predictions to human preferences (Appendix B.3.1); fitted, not derived.
Preference sampling range [omega_min, omega_max] = [0, 2] for the cost weight
Hand-chosen in Appendix B.3.3 to 'effectively capture the Pareto front'; the policy is only trained for preferences in this range, so any user preference outside it is out-of-distribution.
Mixup interpolation weight lambda = Beta(0.2, 0.2)
Hyperparameter chosen following the original mixup paper; not derived from the problem.
Model identity vector I_k = 128-dimensional learned per model
Core capability representation; learned via variational IRT. The cold-start claim assumes new models' identity vectors can be fit from 20-50 prompts.

assumptions (6)

domain assumption A model's performance on arbitrary prompts can be summarized by a low-dimensional identity vector I_k that interacts with prompt embeddings via f(e_n, I_k).
Section 2.3: 'Our goal is to learn a model identity vector I_k in R^d that predicts these evaluation scores.' This is the core representation assumption.
domain assumption Scalarized preferences omega trace the complete Pareto front of achievable performance-cost trade-offs.
Section 2.1: 'as omega varies across R^2_+, the corresponding optimal policies trace out the complete Pareto front.' The proof is deferred to Yang et al. (2019); the paper assumes this result applies.
domain assumption Pretrained prompt embeddings e_n are sufficient; no learnable prompt representation is needed.
Section 2.3: 'we leverage pretrained prompt embeddings e_n rather than learning explicit prompt representations, enabling generalization to unseen prompts.'
standard math Policy gradient with PPO and GAE converges to a good approximation of the scalarized reward optimum.
Section 2.4 and Appendix B.2: standard RL machinery; assumed valid as in prior work.
ad hoc to paper The stratified sampling by discrimination score psi_n yields a representative prompt subset for new models.
Section 2.6: introduced specifically for the cold-start mechanism; no theoretical guarantee that 20-50 prompts from these strata capture routing-relevant capabilities.
domain assumption Costs c_k are fixed and known per model and do not depend on the query.
Section 2.1: c_k in R_+; the Conclusion explicitly flags 'current approach assumes fixed costs per model' as a limitation.

invented entities (1)

Model identity vector I_k independent evidence
purpose: Compact representation of each LLM's capabilities used by the routing policy and cold-start.
A learned latent vector, not a physical entity. It has a falsifiable handle: routing performance on held-out benchmarks and the cold-start tests are external checks. It is an invented representation, but its predictions can be tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing." pith.science (2026). https://pith.science/paper/NWMUA2EN

@misc{pith2026250202743,
  author       = {Pith},
  title        = {Pith review of: LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NWMUA2EN}},
  note         = {Machine review of arXiv:2502.02743}
}

read the original abstract

The rapid advancement in large language models (LLMs) has brought forth a diverse range of models with varying capabilities that excel in different tasks and domains. However, selecting the optimal LLM for user queries often involves a challenging trade-off between accuracy and cost, a problem exacerbated by the diverse demands of individual queries. In this work, we present a novel framework that formulates the LLM selection process as a multi-armed bandit problem, enabling dynamic and intelligent routing of queries to the most appropriate model. Our approach incorporates a preference-conditioned dynamic routing mechanism, allowing users to specify their preferences at inference time, thereby offering a customizable balance between performance and cost. Additionally, our selection policy is designed to generalize to unseen LLMs, ensuring adaptability to new models as they emerge. Experimental results demonstrate that our method achieves significant improvements in both accuracy and cost-effectiveness across various LLM platforms, showcasing the potential of our framework to adaptively optimize LLM selection in real-world scenarios.

Figures

Figures reproduced from arXiv: 2502.02743 by the authors.

**Figure 1.** Overview of our preference-conditioned dynamic routing framework. Model quizzing (left) generates identity vectors capturing model capabilities, while routing policy (right) determines model selection based on user preferences and query. We aim to develop a routing policy π : X → P(K), where P(K) denotes the probability simplex over K models, that maps each query to a distribution over available models. When execu… view at source ↗

**Figure 2.** Evaluate the routing performance across 5 datasets and various sets of LLM candidates. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Evaluate routing performance on two sets of new models. the identity vectors are obtained using 10, 20 or 50 selected prompts, respectively. system, we select several unseen models from the HuggingFace OpenLLM v2 benchmark. These models are not used for training either the IRT model or the routing policy. For detailed evaluation settings, please refer to Appendix E.9. The identity vectors for these models are ob… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Ablation studies on the routing policy components. Ablation Studies Our routing policy consists of a supervised pretraining stage followed by a RL training stage. During training, we incorporate on-manifold mixup regularization to improve generalization to unseen pr…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation
cs.SE 2026-07 conditional novelty 6.5 of 10

A contextual-bandit supervisor learns to route coding agents among default generation, analysis-augmented generation, and stop, yielding large coverage gains over unsupervised agents and non-agentic LLM test generators.
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
cs.CL 2026-07 conditional novelty 6.0 of 10

PyroDash lets a small model decide mid-generation to hand off once to a frozen LLM, achieving 64.04% accuracy (+6.36 points vs LLM) at lambda=0.05 and 96.4% lower estimated cost at lambda=0.6.
Universal Model Routing for Efficient LLM Inference
cs.CL 2025-02 conditional novelty 5.0 of 10

UniRoute represents each language model by its error rates on a few prompt clusters, letting a router choose among models it has never seen during training.
Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
cs.DC 2025-07 conditional novelty 4.0 of 10

A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.
Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems
cs.AI 2025-02 conditional novelty 4.0 of 10

A survey that classifies LLM routing strategies into pre-generation and post-generation approaches and four implementation families, framed as a performance-cost optimization problem.

Reference graph

Works this paper leans on

41 extracted references · 10 canonical work pages · cited by 5 Pith papers

[4]

Contextualize me–the case for context in re- inforcement learning

Benjamins, C., Eimer, T., Schubert, F., Mohan, A., D¨ohler, S., Biedenkapp, A., Rosenhahn, B., Hutter, F., and Lin- dauer, M. Contextualize me–the case for context in re- inforcement learning. arXiv preprint arXiv:2202.04500,

arXiv
[6]

Rt-1: Robotics transformer for real-world control at scale

Brohan, A., Brown, N., Carbajal, J., Chebotar, Y ., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817,

arXiv
[7]

K., Zhou, J., Lu, P., Molaei, S., and Clifton, D

Chauhan, V . K., Zhou, J., Lu, P., Molaei, S., and Clifton, D. A. A brief review of hypernetworks in deep learning. arXiv preprint arXiv:2306.06955,

arXiv
[8]

Frugalgpt: How to use large language models while reducing cost and improving performance

Chen, L., Zaharia, M., and Zou, J. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176,

arXiv
[9]

V ., and Awadallah, A

Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V ., Lakshmanan, L. V ., and Awadallah, A. H. Hy- brid llm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618,

arXiv
[13]

Zero-shot rein- forcement learning via function encoders

Ingebrand, T., Zhang, A., and Topcu, U. Zero-shot rein- forcement learning via function encoders. arXiv preprint arXiv:2401.17173,

arXiv
[14]

9 LLM Bandit Jain, A., Szot, A., and Lim, J. J. Generalization to new actions in reinforcement learning. arXiv preprint arXiv:2011.01928,

work page Pith review arXiv 2011
[16]

Jiang, D., Ren, X., and Lin, B. Y . Llm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.02561,

arXiv

Show all 41 references

[18]

A survey analyzing generalization in deep reinforcement learning

Korkmaz, E. A survey analyzing generalization in deep reinforcement learning. arXiv preprint arXiv:2401.02349,

arXiv
[19]

Lee, J., Lee, Y ., Kim, J., Kosiorek, A., Choi, S., and Teh, Y . W. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning, pp. 3744–3753. PMLR, 2019a. Lee, K., Lee, K., Shin, J., and Lee, H. Netw...

1910 arXiv
[20]

Holistic evaluation of language models

Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y ., Narayanan, D., Wu, Y ., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110,

arXiv
[21]

Routing to the expert: Efficient reward- guided ensemble of large language models.arXiv preprint arXiv:2311.08692,

Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., and Zhou, J. Routing to the expert: Efficient reward- guided ensemble of large language models.arXiv preprint arXiv:2311.08692,

arXiv
[22]

Blending is all you need: Cheaper, better alternative to trillion-parameters llm

Lu, X., Liusie, A., Raina, V ., Zhang, Y ., and Beauchamp, W. Blending is all you need: Cheaper, better alternative to trillion-parameters llm. arXiv preprint arXiv:2401.02994,

arXiv
[23]

P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappa- ganthu, K., Yang, Y ., et al

Madaan, A., Aggarwal, P., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappa- ganthu, K., Yang, Y ., et al. Automix: Automatically mix- ing language models. arXiv preprint arXiv:2310.12963,

arXiv
[24]

H., Hoang, D

Nguyen, Q. H., Hoang, D. C., Decugis, J., Manchanda, S., Chawla, N. V ., and Doan, K. D. Metallm: A high- performant and cost-efficient dynamic framework for wrapping llms. arXiv preprint arXiv:2407.10834,

arXiv
[25]

E., Kadous, M

Ong, I., Almahairi, A., Wu, V ., Chiang, W.-L., Wu, T., Gon- zalez, J. E., Kadous, M. W., and Stoica, I. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665,

arXiv
[26]

Policy gradient approaches for multi- objective sequential decision making

Parisi, S., Pirotta, M., Smacchia, N., Bascetta, L., and Restelli, M. Policy gradient approaches for multi- objective sequential decision making. In 2014 Interna- tional Joint Conference on Neural Networks (IJCNN), pp. 2323–2330. IEEE,

2014
[28]

E., Adi, Y ., Liu, J., Sauvestre, R., Remez, T., et al

Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y ., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950,

arXiv
[29]

High-dimensional continuous control using generalized advantage estimation

10 LLM Bandit Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438,

arXiv
[31]

Large language model routing with benchmark datasets

Shnitzer, T., Ou, A., Silva, M., Soule, K., Sun, Y ., Solomon, J., Thompson, N., and Yurochkin, M. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789,

arXiv
[32]

Learning pareto set for multi-objective continuous robot control

Shu, T., Shang, K., Gong, C., Nan, Y ., and Ishibuchi, H. Learning pareto set for multi-objective continuous robot control. arXiv preprint arXiv:2406.18924,

arXiv
[33]

Learning invariances for policy generalization

Tachet, R., Bachman, P., and van Seijen, H. Learning invariances for policy generalization. arXiv preprint arXiv:1809.02591,

arXiv
[35]

Learning invariant representations for rein- forcement learning without reconstruction

Zhang, A., McAllister, R., Calandra, R., Gal, Y ., and Levine, S. Learning invariant representations for rein- forcement learning without reconstruction. arXiv preprint arXiv:2006.10742,

2006 arXiv
[36]

mixup: Beyond empirical risk minimization

Zhang, H. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412,

arXiv
[38]

Theoretical Analysis A.1

11 LLM Bandit A. Theoretical Analysis A.1. Continuity and Existence of Optimal Policies Consider a policy πθ parameterized by θ ∈ Θ, where Θ is a compact subset of Rd. Let s(x, k) ∈ {0, 1} be a binary reward function and p(x) be the query distribution. Theorem A.1. If the poli...

2019
[39]

With the calibrated evaluation scores ¯p on a prompt x and a user preference vector ω, the routing action is determined by ˆa = arg maxk∈{k1,k2} ωT [¯pk, −ck]

algorithm, which refines the evaluation scores using human-labeled winning labels to produce more accurate predictions. With the calibrated evaluation scores ¯p on a prompt x and a user preference vector ω, the routing action is determined by ˆa = arg maxk∈{k1,k2} ωT [¯pk, −ck...

2020
[41]

To account for varying user preferences, we evaluate RouteLLM using a range of different thresholds

develops a model that predicts the winning label between a pair of LLMs and selects the model based on a threshold applied to the predicted probability. To account for varying user preferences, we evaluate RouteLLM using a range of different thresholds. 19 LLM Bandit Table E.4...

2024
[128]

The estimated cost of invoking the models for processing 1M input tokens and generating 1M output tokens

The 1https://www.together.ai/pricing 2https://huggingface.co/datasets/berkeley-nest/Nectar 3https://huggingface.co/datasets/lmsys/lmsys-arena-human-preference-55k 4https://huggingface.co/datasets/lmsys/chatbot_arena_conversations 5https://huggingface.co/datasets/lmsys/mt_bench...

2024
[1967]

C., and Bowling, M

Farebrother, J., Machado, M. C., and Bowling, M. Gen- eralization and regularization in dqn. arXiv preprint arXiv:1810.00123,

arXiv
[1987]

and Doshi-Velez, F

Konidaris, G. and Doshi-Velez, F. Hidden parameter markov decision processes: an emerging paradigm for modeling families of related tasks. In 2014 AAAI Fall Symposium Series,

2014
[1999]

Optimising calls to large language models with uncertainty-based two-tier selection

Ram´ırez, G., Birch, A., and Titov, I. Optimising calls to large language models with uncertainty-based two-tier selection. arXiv preprint arXiv:2405.02134,

arXiv
[2013]

Hari, S. N. and Thomson, M. Tryage: Real-time, intelligent routing of user prompts to large language model. arXiv preprint arXiv:2308.11601,

arXiv
[2014]

M., Sun, Y ., Kundu, S., Xing, E., and Yurochkin, M

Wang, H., Polo, F. M., Sun, Y ., Kundu, S., Xing, E., and Yurochkin, M. Fusing models with complementary ex- pertise. arXiv preprint arXiv:2310.01542,

arXiv
[2015]

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv
[2017]

and Guo, Y

Zhang, H. and Guo, Y . Generalization of reinforcement learning with policy-aware adversarial data augmentation. arXiv preprint arXiv:2106.15587,

arXiv
[2019]

L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

arXiv
[2020]

Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,

arXiv
[2021]

Basaklar, T., Gumussoy, S., and Ogras, U. Y . Pd-morl: Preference-driven multi-objective reinforcement learning algorithm. arXiv preprint arXiv:2208.07914,

arXiv
[2022]

and Rish, I

Bouneffouf, D. and Rish, I. A survey on practical appli- cations of multi-armed and contextual bandits. arXiv preprint arXiv:1904.10040,

1904 arXiv
[2023]

C., Castro, P

Agarwal, R., Machado, M. C., Castro, P. S., and Bellemare, M. G. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265,

arXiv
[2024]

Ro- bust reinforcement learning through efficient adversarial herding

Dong, J., Hsu, H.-L., Gao, Q., Tarokh, V ., and Pajic, M. Ro- bust reinforcement learning through efficient adversarial herding. arXiv preprint arXiv:2306.07408,

arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [4]

Contextualize me–the case for context in re- inforcement learning

Benjamins, C., Eimer, T., Schubert, F., Mohan, A., D¨ohler, S., Biedenkapp, A., Rosenhahn, B., Hutter, F., and Lin- dauer, M. Contextualize me–the case for context in re- inforcement learning. arXiv preprint arXiv:2202.04500,

arXiv

[2] [6]

Rt-1: Robotics transformer for real-world control at scale

Brohan, A., Brown, N., Carbajal, J., Chebotar, Y ., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817,

arXiv

[3] [7]

K., Zhou, J., Lu, P., Molaei, S., and Clifton, D

Chauhan, V . K., Zhou, J., Lu, P., Molaei, S., and Clifton, D. A. A brief review of hypernetworks in deep learning. arXiv preprint arXiv:2306.06955,

arXiv

[4] [8]

Frugalgpt: How to use large language models while reducing cost and improving performance

Chen, L., Zaharia, M., and Zou, J. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176,

arXiv

[5] [9]

V ., and Awadallah, A

Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V ., Lakshmanan, L. V ., and Awadallah, A. H. Hy- brid llm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618,

arXiv

[6] [13]

Zero-shot rein- forcement learning via function encoders

Ingebrand, T., Zhang, A., and Topcu, U. Zero-shot rein- forcement learning via function encoders. arXiv preprint arXiv:2401.17173,

arXiv

[7] [14]

9 LLM Bandit Jain, A., Szot, A., and Lim, J. J. Generalization to new actions in reinforcement learning. arXiv preprint arXiv:2011.01928,

work page Pith review arXiv 2011

[8] [16]

Jiang, D., Ren, X., and Lin, B. Y . Llm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.02561,

arXiv