Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Error-Aware Reverse Auction Mechanism for Large Language Model Routing

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EA-RAM shows that LLM routing can be implemented as an error-aware reverse auction in which providers' honest bids on success probability and cost remain a Bayesian equilibrium even when both self-prediction and the platform's evaluation…

desk verdict The reverse-auction-for-LLM-routing idea is worth a look, but the BIC proof conditions on a winning event that is correlated with the evaluator noise, so the paper's central theorem doesn't hold as stated. read the letter →

arxiv 2608.12719 v1 pith:FNNIVDHB submitted 2026-08-13 cs.GT cs.AI

Haolong Chen , Zhengyuan Xin , Liang Zhang , Lei Xue , Guangxu Zhu This is my paper · ORCID
classification cs.GTcs.AI
keywords LLMroutingreverseauctionmechanismdesignBayesianincentivecompatibilityDualErrorwelfarelossboundcost-performanceParetoprovider-sideprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing centralized LLM routers, which predict every model's performance at the task center, with a reverse auction in which LLM providers bid their self-predicted success probability and execution cost. It introduces EA-RAM, whose allocation ranks sellers by reported surplus $V\hat{p}_i-\hat{c}_i$ and whose payment to the winner is $V\tilde{\mu}_j-H$, where $\tilde{\mu}_j$ is the buyer's noisy acceptance signal and $H$ is the runner-up surplus; the noise is modeled explicitly as Dual Error, the provider's ex-ante prediction error plus the center's ex-post evaluation error. The paper proves that under this Dual Error the mechanism is Bayesian incentive compatible and individually rational, gives sufficient conditions for the buyer's expected utility to be non-negative, and bounds the expected welfare loss relative to the error-free benchmark by $2VL_{\sigma}(M_{\mathrm{post}}+M_{\mathrm{ante}})$. If correct, this lets routing scale to large model pools without per-model profiling and makes truthful bidding the equilibrium even when predictions and evaluations are imperfect.

What carries the argument

The load-bearing object is the pair consisting of a ranking score and a payment rule: the buyer ranks sellers by reported surplus $\hat{s}_i=V\hat{p}_i-\hat{c}_i$, and the winner receives $V\tilde{\mu}_j-H$, where $\tilde{\mu}_j$ is the binary acceptance signal from the noisy ex-post evaluator and $H$ is the maximum runner-up reported surplus (or zero for the null allocation). Around this pair the paper defines the effective surplus $\overline{T}_i=Vg_i-c_i$ with the belief map $g_i=\sigma(\phi_i+\varepsilon_{\mathrm{post}}+\varepsilon_{\mathrm{ante},i})$, and the central identity is $U_i^{\mathrm{seller}\;\prime}(\hat{s}_i)=f_H(\hat{s}_i)(\overline{T}_i-\hat{s}_i)$, which shows that reporting the effective surplus is the unique maximizer of interim expected utility whenever that surplus is positive. The globally Lipschitz link function $\sigma$ then converts the second-moment radii of the two error channels, $M_{\mathrm{post}}=\sqrt{b_{\mathrm{post}}+a_{\mathrm{post}}^2}$ and $M_{\mathrm{ante}}=\max_i\sqrt{b_{\mathrm{ante},i}+a_{\mathrm{ante},i}^2}$, into the explicit welfare-loss bound.

What would settle it

Give sellers a signal that contains only their own prediction noise, withhold the evaluator's $\varepsilon_{\mathrm{post}}$ from them, and run the auction with nonzero evaluation variance; if truthful reporting of $Vg_i-c_i$ is not the expected-utility maximizer, for instance if a seller can profit by over-reporting when the evaluator noise is positive, the BIC theorem as stated is false. Concretely, compute $E[V\tilde{\mu}_i|\text{seller signal}]$ and compare it with $Vg_i$; the proof's key equality fails whenever the seller's signal is not informative about $\varepsilon_{\mathrm{post}}$.

Watch

Extended reading notes

Core claim

The central discovery is that a Vickrey-style allocation rule can survive both noisy provider prediction and noisy platform evaluation if the auction ranks sellers by the reported surplus $\hat{s}_i=V\hat{p}_i-\hat{c}_i$ and pays the winner $V\tilde{\mu}_j-H$, where $\tilde{\mu}_j$ is the evaluator's acceptance signal and $H$ is the runner-up score. In the error-free setting these rules satisfy dominant-strategy incentive compatibility, individual rationality, center rationality, and economic efficiency. Under Dual Error, with seller belief $g_i=\sigma(\phi_i+\varepsilon_{\mathrm{post}}+\varepsilon_{\mathrm{ante},i})$ and evaluator acceptance probability $h_i=\sigma(\phi_i+\varepsilon_{\mathrm{post}})$, the paper proves that truthful reporting of the effective surplus $\overline{T}_i=Vg_i-c_i$ is a Bayesian best response (Theorem 3.2), that all sellers have non-negative expected utility at equilibrium (Theorem 3.3), that the buyer's expected utility is non-negative when either the runner-up margin is large enough or the evaluation is conservative (Theorem 3.4), and that the expected welfare loss relative to the error-free benchmark is at most $2VL_\sigma(M_{\mathrm{post}}+M_{\mathrm{ante}})$ (Theorem 3.7). The argument turns on the derivative identity $U_i^{\mathrm{seller}\;\prime}(\hat{s}_i)=f_H(\hat{s}_i)(\overline{T}_i-\hat{s}_i)$, which makes the seller's interim expected utility peak exactly at its effective surplus.

Load-bearing premise

The entire incentive guarantee presumes that the seller's reported probability matches its expectation of the buyer's noisy evaluation, specifically that the seller's belief $g_i$ incorporates the evaluator's noise $\varepsilon_{\mathrm{post}}$, so if sellers observe only their own prediction error and not the evaluator's noise, truthful bidding may no longer be a Bayesian equilibrium.

Editorial extensions

If this is right

  • If EA-RAM is deployed, the task center needs only a model-agnostic evaluator and the auction logic; it no longer trains or profiles per-model predictors, so adding a new LLM does not require retraining a router.
  • The welfare-loss bound shrinks to zero as the second-moment radii of prediction and evaluation errors shrink, so improving evaluator and predictor calibration directly brings routing efficiency close to the error-free benchmark.
  • Clear-cut queries, where $|\phi_i|$ is large, are nearly immune to noise when the link function saturates (e.g., logistic), so the mechanism is most sensitive in close competitions between models.
  • Provider-side local information can shift the empirical cost-performance Pareto frontier upward, as shown on RouterBench, meaning sellers' private signals can be harnessed rather than treated as a threat to the mechanism.
  • Center-side latency stays nearly constant as the model pool grows, while communication overhead grows linearly with the number of providers and becomes a bottleneck only at low per-channel bandwidths, as the paper's efficiency experiments indicate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The incentive proof assumes the seller's belief $g_i$ already incorporates the evaluator's noise $\varepsilon_{\mathrm{post}}$; in a real deployment a seller may observe only its own prediction noise, in which case the equality $E[V\tilde{\mu}_i|\cdot]=Vg_i$ used for Bayesian incentive compatibility would not hold and truthful bidding could cease to be an equilibrium. This information-structure as
  • The paper defines the mechanism with respect to an announced task value $V$ and states as a future direction that misspecification of $V$ may change participation incentives; a natural extension is to analyze how a strategic or inaccurate announced value shifts the equilibrium operating point.
  • The same reverse-auction template could apply to any setting where providers hold private success probabilities and a noisy verifier exists, such as code execution, math checking, or translation quality, not only LLM quality routing.
  • The tighter welfare-loss bound under opposite-signed errors suggests that deliberately pairing error regimes, for instance a conservative evaluator with optimistic sellers, could be engineered to reduce allocation distortion in practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes EA-RAM, a reverse-auction mechanism for routing LLM queries. Providers bid with self-predicted success probabilities and execution costs; the buyer allocates to the highest reported surplus V p_hat - c_hat and pays the winner V times a noisy evaluator signal minus the runner-up score. The authors model two sources of noise: provider-side prediction error and buyer-side evaluation error, which they call Dual Error. They claim that under this model EA-RAM is Bayesian incentive compatible and individually rational, satisfies sufficient conditions for center rationality, and has an expected welfare loss bounded by 2 V L_sigma (M_post + M_ante) relative to the error-free benchmark. They also report simulation and RouterBench experiments suggesting that EA-RAM is robust to noise and outperforms centralized routing baselines.

Significance. If the theoretical claims were rigorously established, the paper would make a useful contribution: it shifts ex-ante quality prediction from a central router to LLM providers, explicitly models noisy self-prediction and noisy evaluation in a mechanism-design setting, and provides an error-aware welfare bound. The paper is also empirically grounded, with experiments on RouterBench and an LLM-as-a-judge noise study, and it explicitly labels its oracle and realistic information variants. However, the central BIC/IR proof rests on an unexamined conditional-expectation step, and the welfare-loss proof applies a pointwise Lipschitz bound to a random quantity. These issues currently prevent the main theoretical results from being accepted as stated.

major comments (3)
  1. [Section 3.2 and Appendix C.2, Eq. (5)] The BIC proof replaces E[V tilde_mu_i | H <= s_hat] with V g_i, but under the stated model epsilon_post is a common shock entering every h_i and every g_i, so H = max(0, max_{k neq i}(V g_k - c_k)) is correlated with epsilon_post and therefore with the winner's evaluator signal. Conditioning on the win event H <= s_hat is informative, and E[V tilde_mu_i | H <= s_hat] is not equal to V g_i in general. Consequently Eq. (5) is not the seller's interim utility, the first-order condition U_i'(s_hat) = f_H(s_hat)(Tbar_i - s_hat) does not follow, and Theorems 3.2 and 3.3 are unsupported as written. The authors should either make epsilon_post seller-specific and independent across sellers, or specify the seller's information set precisely and derive the correct conditional expectation; the current text conflates two readings of the information structure.
  2. [Appendix C.3, Eq. (39) and Theorem 3.7] The welfare-loss proof bounds |g_i - p_i| <= L_sigma (M_post + M_ante) as if this held pointwise, but g_i = sigma(phi_i + eta_i) is random and the Lipschitz lemma (C.4) bounds the expectation E|sigma(x + epsilon) - sigma(x)|, not the realized absolute deviation |sigma(x + epsilon) - sigma(x)|. For unbounded errors the pointwise inequality is false. The proof needs an explicit expectation step: it should bound E|g_i - E[sigma(phi_i + epsilon_post)]| using the ante channel and |E[sigma(phi_i + epsilon_post)] - p_i| using the post channel, then sum the two bounds. As written, Theorem 3.7 does not follow from the argument given.
  3. [Appendix C.5, proof of Theorem 3.4] The CR proof conditions on the selected winner's phi_(1) and applies Lemma C.4 to the unconditional distribution of epsilon_post. But the identity of the winner is selected using reported scores that depend on epsilon_post, so the conditional distribution of epsilon_post given the event that seller (1) wins is not the unconditional distribution used in the lemma. The bound |E[h_(1)] - E[p_(1)]| <= Delta_gate needs an argument that accounts for this selection effect; as it stands, sufficient condition (A) of Theorem 3.4 is not established.
minor comments (5)
  1. [Section 1, paragraph on EA-RAM] The sentence 'We therefore propose the Error-Aware Reverse Auction Mechanism (EA-RAM) Crucially, EA-RAM explicitly models...' is missing a period or comma after '(EA-RAM)'.
  2. [Section 2.2] The notation uses epsilon_post without a seller subscript for h_i and inside the definition of g_i, while epsilon_ante,i carries a subscript. Please state explicitly whether epsilon_post is a single common shock across all sellers or a seller-specific shock; this distinction is essential for the BIC conditioning argument.
  3. [Table 2] The rows 'with pi=0.1/0.2' and 'with omega=0.1/0.2' should be labeled as 'oracle local information' and 'realistic local information' in the table itself, since the oracle rows use ground-truth labels and are upper bounds rather than deployable configurations.
  4. [Proposition 3.6] The statement that the error-aware allocation attains the error-free welfare if and only if the error-aware winner is welfare-optimal is tautological; consider stating it as a definition or replacing it with a discussion of when the argmax is preserved.
  5. [Section 4.2.1] The two-layer MLP description would benefit from specifying hidden dimensions and the exact train/test split procedure for the predictor and evaluator, since the Pareto-frontier comparison depends on these implementation details.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EA-RAM's theoretical results are derived from explicit error assumptions and external benchmarks, not from fitted inputs or self-citation chains.

full rationale

The paper's central claims (BIC, IR, CR, welfare-loss bound) are derived in Appendix C from explicitly stated primitives: the link function sigma, the error variables epsilon_post and epsilon_ante,i, and the runner-up score distribution F_H. The payment rule r_j = V tilde_mu_j - H is a standard surplus-externality payment, and the BIC proof follows by differentiating U_i(s_hat) = F_H(s_hat)(Tbar_i - E[H | H <= s_hat]) under the stated belief assumption. No parameter is fitted to data in order to produce a theoretical prediction; the experiments fit MLP predictors and compare against external baselines (RouterBench, EmbedLLM, RouteLLM, etc.), which is evaluation, not derivation. The only citation involving an author of this paper is reference [3], a general survey cited for the prevalence of LLM applications; it is not load-bearing. Although the BIC proof depends delicately on the seller's information about the common evaluation noise epsilon_post, that is a correctness or modeling concern (whether E[V tilde_mu_i | H <= s_hat] = V g_i holds under the stated information structure), not a circularity: the claimed result is not equivalent to an input by construction. Appendix D also candidly states limitations such as bounded-penalty variants, communication overhead, privacy, and evaluator-gaming, none of which reveal circularity. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The theoretical mechanism introduces no fitted constants; the error moments and Lσ are inputs from the model. The main load-bearing assumption is the information structure: the seller is assumed to know the aggregated error η_i, which includes the evaluator's noise, and this is essential for the BIC proof. The experiments train MLPs, but those are not part of the theoretical derivation.

assumptions (6)
  • domain assumption Providers are risk-neutral and maximize expected profit
    Assumed in Section 2.1 and used throughout the utility analysis.
  • domain assumption Task value V is common knowledge and announced by the buyer
    Used in the surplus score and payment rule; Section 2.1 and Limitations in Appendix D.
  • domain assumption Link function σ is strictly increasing, continuously differentiable, and globally Lσ-Lipschitz
    Used in Lemma C.4 and all bounds; Section 2.1.
  • domain assumption Errors ε_post and ε_ante,i are independent with finite second moments
    Used to define Dual Error and to compute welfare-loss bound; Section 2.2.
  • ad hoc to paper The seller's belief g_i is known to the seller and equals σ(ϕ_i + η_i) with η_i = ε_post + ε_ante,i
    This information structure is used in the BIC proof where E[V ˜µ_i|·] = V g_i; Section 2.2 and 3.2. If the seller does not observe ε_post, the certainty of this belief is not justified.
  • standard math The runner-up score H has a distribution independent of the seller's own report
    Standard in second-price auction analysis; used in Theorem 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Error-Aware Reverse Auction Mechanism for Large Language Model Routing." pith.science (2026). https://pith.science/paper/FNNIVDHB

@misc{pith2026260812719,
  author       = {Pith},
  title        = {Pith review of: Error-Aware Reverse Auction Mechanism for Large Language Model Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNNIVDHB}},
  note         = {Machine review of arXiv:2608.12719}
}
read the original abstract

Routing each query to a cost-effective large language model (LLM) is critical for balancing quality and cost, yet most routers rely on a centralized task center to predict model performance, creating an information-risk mismatch and a scalability bottleneck as the model pool grows. We propose a market-based routing paradigm that shifts ex-ante prediction to LLM providers via a reverse auction, where providers bid with self-predicted success probabilities and execution costs. To account for inherently noisy provider predictions and center evaluations, we introduce the \textit{\textbf{E}rror-\textbf{A}ware \textbf{R}everse \textbf{A}uction \textbf{M}echanism} (EA-RAM), which explicitly models this inherent Dual Error. We prove that EA-RAM is Bayesian incentive compatible and individually rational under the Dual Error, establish sufficient conditions for center rationality, and derive an explicit welfare-loss bound. We further identify robustness effects: opposite-signed errors can cancel, vanishing-tail link functions (e.g., logistic) stabilize clear-cut cases via saturation, and extra noise smooths belief maps, reducing the gains from marginal manipulation. Experiments on simulations and real-world benchmarks show that EA-RAM is robust to the Dual Error and achieves a better cost--performance Pareto frontier than centralized baselines, with additional gains when providers contribute local information, validating its practical effectiveness.

Figures

Figures reproduced from arXiv: 2608.12719 by the authors.

Figure 1
Figure 1. Paradigm shift from traditional Centralized Routing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. EA-RAM for LLM routing. (1) Providers bid using ex-ante predictions. (2) The buyer [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ability–Difficulty Matching. Following prior work on capability model￾ing [6, 16], we model the fulfillment proba￾bility pi via the alignment between seller i’s model ability mi ∈ R and the task difficulty d ∈ R ( [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Seller’s strategic decision. The seller bids based on the buyer’s error-involved evaluation, rather than the unobserved ground-truth. LLM Routing faces Dual Error due to noisy pre￾diction and imperfect evaluation, undermining classical mechanisms such as FTMD [13] and …
Figure 5
Figure 5. Figure 5: Robustness to Dual Error. EA-RAM remains stable under evaluator and seller noise, avoiding the misallocation caused by naive reporting. Shaded regions show 95% confidence intervals. 4.2 Real-World Experiments In this section, we benchmark EA-RAM against centralized bas…
Figure 6
Figure 6. Figure 6: Cost–performance Pareto frontiers on real-world benchmarks. As [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Noisy LLM-as-a-Judge evaluation. Varying the judge weight [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Center-side latency. Center-side latency [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 17 canonical work pages

  1. [1]

    A survey of large language models.arXiv preprint arXiv:2303.18223, 1(2), 2023

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models.arXiv preprint arXiv:2303.18223, 1(2), 2023

    arXiv 2023

  2. [2]

    Large language model based multi-agents: A survey of progress and challenges

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. InIJCAI, 2024

    2024

  3. [3]

    An overview of domain-specific foundation model: key technologies, applications and challenges.Science China Information Sciences, 69(1):111301, 2026

    Haolong Chen, Hanzhi Chen, Zijian Zhao, Kaifeng Han, Guangxu Zhu, Yichen Zhao, Ying Du, Wei Xu, and Qingjiang Shi. An overview of domain-specific foundation model: key technologies, applications and challenges.Science China Information Sciences, 69(1):111301, 2026

    work page 2026

  4. [4]

    EmbedLLM: Learning compact representations of large language models

    Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. EmbedLLM: Learning compact representations of large language models. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  5. [5]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  6. [6]

    Icl-router: In-context learned model representations for llm routing.arXiv preprint arXiv:2510.09719, 2025

    Chenxu Wang, Hao Li, Yiqun Zhang, Linyao Chen, Jianhao Chen, Ping Jian, Peng Ye, Qiaosheng Zhang, and Shuyue Hu. Icl-router: In-context learned model representations for llm routing.arXiv preprint arXiv:2510.09719, 2025

    arXiv 2025

  7. [7]

    Auto router

    OpenRouter. Auto router. https://openrouter.ai/docs/guides/routing/routers/ auto-router. Accessed: 2026-05-06

    work page 2026

  8. [8]

    Smart llm routing

    Requesty. Smart llm routing. https://www.requesty.ai/solution/llm-routing. Ac- cessed: 2026-05-06

    work page 2026

Show all 34 references
  1. [9]

    Real-time bidding for online advertising: measure- ment and analysis

    Shuai Yuan, Jun Wang, and Xiaoxue Zhao. Real-time bidding for online advertising: measure- ment and analysis. InProceedings of the seventh international workshop on data mining for online advertising, pages 1–8, 2013

    2013

  2. [10]

    Display advertising with real-time bidding (rtb) and behavioural targeting.Foundations and Trends® in Information Retrieval, 11(4-5):297–435, 2017

    Jun Wang, Weinan Zhang, Shuai Yuan, et al. Display advertising with real-time bidding (rtb) and behavioural targeting.Foundations and Trends® in Information Retrieval, 11(4-5):297–435, 2017

    2017

  3. [11]

    The fcc spectrum auctions: An early assessment.Journal of Economics & Management Strategy, 6(3):431–495, 1997

    Peter Cramton. The fcc spectrum auctions: An early assessment.Journal of Economics & Management Strategy, 6(3):431–495, 1997

    1997

  4. [12]

    Cambridge University Press, 2004

    Paul Robert Milgrom.Putting auction theory to work. Cambridge University Press, 2004

    2004

  5. [13]

    Fault tolerant mechanism design.Artificial Intelligence, 172(15):1783–1799, 2008

    Ryan Porter, Amir Ronen, Yoav Shoham, and Moshe Tennenholtz. Fault tolerant mechanism design.Artificial Intelligence, 172(15):1783–1799, 2008

    2008

  6. [14]

    Strategic design under uncertain evaluations: structural analysis of design- build auctions.The RAND Journal of Economics, 49(3):594–618, 2018

    Hidenori Takahashi. Strategic design under uncertain evaluations: structural analysis of design- build auctions.The RAND Journal of Economics, 49(3):594–618, 2018

    2018

  7. [15]

    Coalesce: Eco- nomic and security dynamics of skill-based task outsourcing among team of autonomous llm agents.arXiv preprint arXiv:2506.01900, 2025

    Manish Bhatt, Ronald F Del Rosario, Vineeth Sai Narajala, and Idan Habler. Coalesce: Eco- nomic and security dynamics of skill-based task outsourcing among team of autonomous llm agents.arXiv preprint arXiv:2506.01900, 2025. 10

    2025 arXiv

  8. [16]

    IRT-router: Effective and interpretable multi-LLM routing via item response theory

    Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, GuanHao Zhao, Fei Wang, and Runze Wu. IRT-router: Effective and interpretable multi-LLM routing via item response theory. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedi...

    2025

  9. [17]

    Routerbench: A benchmark for multi-LLM routing system

    Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-LLM routing system. InAgentic Markets Workshop at ICML 2024, 2024

    2024

  10. [18]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, edi- tors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pa...

    2019

  11. [19]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  12. [20]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  13. [21]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  14. [22]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  15. [23]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  16. [24]

    FrugalGPT: How to use large language models while reducing cost and improving performance.Transactions on Machine Learning Research,

    Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance.Transactions on Machine Learning Research,

  17. [25]

    A unified approach to routing and cascading for LLMs

    Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for LLMs. InForty-second International Conference on Machine Learning, 2025

    2025

  18. [26]

    all-minilm-l6-v2

    Sentence-Transformers. all-minilm-l6-v2. https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2. Accessed: 2026-05-06

    2026

  19. [27]

    Router-r1: Teaching llms multi-round routing and aggregation via reinforcement learning

    Haozhen Zhang, Tao Feng, and Jiaxuan You. Router-r1: Teaching llms multi-round routing and aggregation via reinforcement learning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  20. [28]

    Ad auctions for llms via retrieval augmented generation.Advances in Neural Information Processing Systems, 37:18445–18480, 2024

    MohammadTaghi Hajiaghayi, Sébastien Lahaie, Keivan Rezaei, and Suho Shin. Ad auctions for llms via retrieval augmented generation.Advances in Neural Information Processing Systems, 37:18445–18480, 2024

    2024

  21. [29]

    Mechanism design for large language models

    Paul Duetting, Vahab Mirrokni, Renato Paes Leme, Haifeng Xu, and Song Zuo. Mechanism design for large language models. InProceedings of the ACM Web Conference 2024, pages 144–155, 2024

    2024

  22. [30]

    Auctions with llm summaries

    Avinava Dubey, Zhe Feng, Rahul Kidambi, Aranyak Mehta, and Di Wang. Auctions with llm summaries. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 713–722, 2024. 11 Contents 1 Introduction 1 2 Error-Aware Reverse Auction Mechanism 2 ...

    2024

  23. [32]

    For ability (x=m i), the gradient is non-negative (∂E[U seller i ] ∂mi ≥0)

  24. [33]

    This confirms that the seller’s utility is weakly increasing in ability and weakly decreasing in difficulty

    For difficulty (x=d), the gradient is non-positive ( ∂E[U seller i ] ∂d ≤0). This confirms that the seller’s utility is weakly increasing in ability and weakly decreasing in difficulty. C.3 Welfare Analysis Proposition C.7(Economic Efficiency (EE). Restatement of Proposition 3...

  25. [34]

    For any error ϵ with E[ϵ2]<∞ , the deviation ∆(ϕi) =|E[σ(ϕ i +ϵ)]−σ(ϕ i)| satisfies lim|ϕi|→∞ ∆(ϕi) = 0. Proof.Using the integral form of the Mean Value Theorem, for any realization ofϵ, σ(ϕi +ϵ)−σ(ϕ i) =ϵ Z 1 0 σ′(ϕi +tϵ)dt.(49) Taking absolute values and expectations gives ∆...

  26. [2024]

    Featured Certification

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.