Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An Item-Response-Theory router gives each LLM a 25-dimensional ability and each query a difficulty, routing to the best quality-per-cost model; the paper reports higher performance than GPT-4o alone at 1/30 the cost.

desk verdict A useful, honestly-framed application of IRT to LLM routing with a solid cost-efficiency result, but the headline performance advantage over RouterBench is within sampling error and needs seeds and error bars. read the letter →

arxiv 2506.01048 v2 pith:KIQ7B6MI submitted 2025-06-01 cs.AI

Wei Song , Zhenya Huang , Cheng Cheng , Weibo Gao , Bihan Xu , GuanHao Zhao , Fei Wang , Runze Wu This is my paper · ORCID
classification cs.AI
keywords LLMroutingItemResponseTheorymultidimensionalIRTlatentabilityquerydifficultycost-performancetrade-offcoldstartinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a router can treat LLMs as test-takers and queries as test items: each model gets a 25-dimensional latent ability vector, each query gets difficulty and discrimination vectors, and a logistic function predicts how well that model will answer that query. The router then sends each query to the model that maximizes predicted quality minus a fixed cost term. If this works, it matters because routing becomes cheaper and more explainable than always calling the most powerful model, and because the learned ability and difficulty values provide a reason for each routing choice. On the in-distribution test set with performance priority, the paper reports MIRT-Router reaches 80.67% average performance at a total cost of $0.42, where RouterBench reaches 80.01% at $1.15 and GPT-4o alone reaches 77.53% at $12.93. The paper's own limitation section notes that benchmark queries are short and that the router's sensitivity to the performance-versus-cost weight is limited.

What carries the argument

The load-bearing object is the logistic IRT interaction $$\hat{P}(q_i, M_j) = \frac{1}{1+\exp(-a_i^\top \theta_{M_j} + b_i)},$$ where $\theta_{M_j}\in\mathbb{R}^{25}$ is the latent ability vector of model $M_j$, produced by a linear map from a BERT embedding of the model's text profile, and $a_i$ and $b_i$ are discrimination and difficulty parameters produced by linear maps from the query's BERT embedding. Predicted performance is combined with a fixed cost $C(M_j)$ through the score $S=\alpha \hat{P}-\beta C$, and the query is sent to the model with the highest score. MIRT-Router uses the logistic form directly; NIRT-Router replaces it with a neural interaction layer that uses a per-query relevance vector, obtained by clustering training queries and asking an LLM which of 25 predefined abilities each cluster needs. A warm-up step blends each new query embedding with the averaged embeddings of its five nearest training neighbors to reduce cold-start error.

What would settle it

On a held-out set of queries where all 20 candidate LLMs actually respond to every query, compute the rank correlation between IRT-Router's predicted score $S(q_i, M_j)$ and the empirically measured reward of routing $q_i$ to $M_j$; if the correlation is near zero, the logistic IRT interaction is not capturing query-model fit and the routing decisions built on it are miscalibrated.

Watch

Extended reading notes

Core claim

The central claim is that the response quality of an LLM on a query follows an IRT measurement model, and that the parameters of that model can be learned from query embeddings and LLM profile embeddings alone. On the paper's 20-model, 12-dataset evaluation, both MIRT-Router and NIRT-Router outperform the strongest routing baseline, RouterBench, on the performance-cost reward in every tested setting, while spending a fraction of the cost of always using GPT-4o. The learned parameters also carry the interpretability story: ability vectors of larger models dominate smaller ones in most dimensions, learned query difficulty tracks human difficulty labels, and hard queries are routed to stronger models while easy queries go to cheaper, adequate ones.

Load-bearing premise

The router's predictions stand or fall on the assumption that a single 25-number ability profile per model, plus a per-query difficulty level and a 25-number sensitivity profile, combined through one logistic formula, can predict how well any model will answer any query—including out-of-distribution queries the router never trained on.

Editorial extensions

If this is right

  • A service choosing among many LLM APIs can use a lightweight per-query predictor plus fixed-cost scores, avoiding the need to call every candidate model before deciding.
  • Routing decisions can be explained to users with the model's estimated ability and the query's estimated difficulty, rather than an opaque classifier score.
  • The warm-up mechanism makes the router usable when new queries arrive online, and the paper reports its out-of-distribution reward increases with a larger blending weight $\lambda$.
  • The same framework can be applied to a newly released LLM by encoding its text profile, though the paper reports limited accuracy on one held-out model (ACC 0.67) and treats improvement as future work.
  • Because the predictor is a compact factor model, its parameters can serve as a reusable map of which models are strong in which ability dimensions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the learned ability vectors could serve as a 'model zoo' registry, letting a newly released LLM be inserted by embedding its public description alone; the paper evaluates only one held-out LLM and does not test this workflow.
  • Editorial extension: since the score is a linear combination of predicted performance and a fixed cost, the same router can be repriced for different deployments (for example, free self-hosted models) by changing $C(M_j)$; the paper notes this possibility but does not test it.
  • Editorial extension: comparing the router against an oracle that always picks the empirically best model for each test query would reveal how much headroom remains; the reported top-1 routing accuracy is low (2.72% in-distribution), suggesting the gap may be substantial.
  • Editorial extension: the warm-up step is a form of transductive learning, so one could test whether actively selecting a small set of training queries for new domains improves OOD routing more than the fixed k-nearest-neighbor averaging.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes IRT-Router, a multi-LLM routing framework that models each LLM as a latent-ability vector and each query as a difficulty/discrimination item using Item Response Theory. Two variants are presented: MIRT-Router, based on a multidimensional logistic model, and NIRT-Router, based on a neural cognitive diagnosis model with predefined ability dimensions. A semantic-similarity warm-up mechanism is added for cold-start queries. The method is evaluated on 20 LLMs and 12 datasets, with in-distribution and out-of-distribution tests, and compared against RouterBench, HybridLLM, RouteLLM, and single-model baselines. The authors report that IRT-Router achieves higher performance at lower cost than the baselines, and they present interpretability analyses of learned LLM abilities and query difficulties.

Significance. If the empirical claims hold, the paper makes a useful contribution: it brings a principled psychometric model to LLM routing, provides a compact and cheap predictor, and includes an out-of-distribution evaluation, a new-LLM generalization experiment, a warm-up ablation, and publicly released code. These are concrete strengths that go beyond a purely methodological proposal. The main weaknesses are statistical: the headline comparisons rest on a single split without error bars, the margins over RouterBench are small relative to sampling error, and the interpretability evidence is partly anecdotal and partly affected by parameter identifiability. The approach is nevertheless well motivated and the central idea is testable, so the paper merits a major revision rather than rejection.

major comments (5)
  1. [§5, §6, Tables 1–2] The headline comparative claim is not statistically supported. All results come from a single 70/30 split with no error bars, confidence intervals, or significance tests. At α=0.8, MIRT-Router exceeds RouterBench by 0.66 percentage points on ID (80.67% vs. 80.01%) and by 1.62 points on OOD (87.12% vs. 85.50%). With ID test size about 10,470 and OOD test size about 3,160, the approximate standard error of each difference is 0.5–1.0 percentage points, so neither gap reaches 95% confidence. Please report results over multiple seeds or splits, or provide significance tests, before claiming that IRT-Router outperforms baselines.
  2. [§6.2, Fig. 5, Eq. (5)] The query-difficulty evidence is internally inconsistent with the model definition. In Eq. (5), P = σ(−a_i^T θ_j + b_i), so a larger b_i makes a query harder. In Fig. 5, the Level 2 query has difficulty −0.047 and the Level 4 query has difficulty −0.117; under Eq. (5) the Level 4 query is therefore easier, contradicting the caption's claim that higher Difficulty means more challenging. Please clarify the sign convention or re-examine the difficulty extraction, because as written the figure does not demonstrate that learned difficulty matches level labels.
  3. [§4.2.1, Eqs. (4)–(5)] The interpretability claims are weakened by lack of identifiability constraints on MIRT-Router's parameters. The predicted probability depends only on the product a_i^T θ_j + b_i, so the transformations θ_j → θ_j + d, b_i → b_i − a_i^T d, and θ_j → c θ_j, a_i → a_i/c leave all predictions unchanged. Without anchor items, regularization, or explicit constraints, the absolute ability values shown in Fig. 4 and the difficulty values in Fig. 5 are not uniquely identified. The authors should state what identifies the scale and location of θ and b, or present the interpretability results as relative rather than absolute.
  4. [§4.4, Table 9 (Appendix B)] The warm-up hyperparameter λ is analyzed on the OOD test set itself. Table 9 reports ID and OOD rewards for λ ∈ {0, 0.1, 0.2, 0.3, 0.4} and uses these OOD values to conclude that larger λ improves cold-start generalization. Since the OOD test set is used to select/justify λ, the reported OOD gains of the warm-up mechanism are partially in-sample. Please use a validation split or nested tuning to demonstrate that the warm-up benefit holds for truly unseen queries.
  5. [§6.2, Table 3] The Top-k routing accuracy results raise concerns about routing quality. With 20 candidate LLMs, random Top-1 selection has an expected accuracy of 5%, but MIRT-Router reports Top-1 accuracy of only 2.72% on ID and 2.15% on OOD. The explanation that many models have similar scores needs quantitative support: please report the distribution of score gaps to the oracle-best model, the number of near-optimal ties, and Top-k accuracy under a relaxed optimality threshold. As presented, this table does not support the claim that IRT-Router routes effectively.
minor comments (4)
  1. [Table 5, Appendix A.2] There are typos in the table header ('Iutput') and in the appendix heading ('Cadidate LLMs' should be 'Candidate LLMs').
  2. [Eq. (7)–(8)] The dimensional notation in the NIRT interaction is confusing: Eq. (7) defines x_ij as a vector but Eq. (8) writes W1 x_ij^T; please clarify the row/column convention and the shapes of W1 and b1.
  3. [Table 7] The dataset name 'CEV AL' is inconsistent with the reference to C-EVAL; please unify the spelling.
  4. [Figure 3] The example query text appears with a rendering artifact ('About:miscellaneousQuestion'); please format the figure caption or example properly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IRT-Router's performance claim is tested on held-out queries, not on fitted values.

full rationale

IRT-Router's central routing pipeline is self-contained rather than circular. The performance predictor P̂(q_i, M_j) in Eq. (5) is a parameterized function of query and LLM embeddings, with θ, a, and b produced by learnable transformations (Eq. 4); Wθ, Wa, and Wb are fitted by minimizing cross-entropy on Dtrain = {(q_i, M_j, y_ij)} (Eq. 6), where y_ij are empirical response scores. The main evaluation (Tables 1-2) uses held-out ID queries and OOD datasets not seen during training, so the reported performance, cost, and reward comparisons do not reduce to a fitted value. The interpretability exhibits in Sec. 6.2 are post-hoc visualizations of learned latent variables rather than predictions derived from those latents; comparing learned difficulty to MATH level labels is an external check, and the routing analysis describes the router's own decision function rather than validating the central claim. Self-citations to NCDM and IRT literature are used as modeling inspiration, not as load-bearing proof, and no equation in the paper is identical by construction to the target routing claim. The main caveats—single split, no error bars, and selection of N and λ using the ID test set—are statistical-correctness concerns, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central model relies on the IRT logistic form and on embedding sufficiency. The main hand-chosen values are N, lambda, k, and the cluster label count; the latent ability and difficulty vectors are fitted model outputs, not independently verified entities.

free parameters (4)
  • Ability dimension N = 25
    Chosen via sensitivity sweep; performance varies nonmonotonically with N in Fig. 8.
  • Warm-up weight lambda = 0.2 in ID; 0.3 or 0.4 favored on OOD in Table 9
    Ablated on test sets; OOD reward improves with larger lambda, so the reported OOD numbers may reflect test-set selection.
  • Nearest-neighbor count k = 5
    Used for warm-up in Eq. 10 and for NIRT relevance approximation; set without justification.
  • Sample questions per cluster num = 5
    NIRT relevance vector labeling uses 5 questions per HDBSCAN cluster; chosen ad hoc.
assumptions (5)
  • domain assumption Monotonicity: higher LLM ability implies higher probability of correct response on a query.
    Sec. 3.2 states this psychological IRT assumption; in MIRT Eq. 5 it is not enforced by constraints on learned a_i, so fitted abilities are not guaranteed to be monotone. Interpretability conclusions rely on it.
  • domain assumption BERT embeddings of queries and text profiles of LLMs contain sufficient information to predict response quality.
    Sec. 4.1 constructs all inputs from bert-base-uncased embeddings; if semantics of queries and model profiles are insufficient, the learned IRT parameters cannot capture true performance.
  • domain assumption A single latent ability vector per LLM and difficulty and discrimination per query is a valid generative model of response correctness across heterogeneous benchmarks.
    Eqs. 4 and 5 assume the logistic MIRT form; heterogeneity of MMLU, MATH, code, and reading comprehension is collapsed into 25 ability dimensions.
  • domain assumption yij scores from dataset-specific metrics are comparable labels for binary cross-entropy training.
    Sec. 5 constructs yij as EM, F1, or accuracy values; applying BCE with continuous targets mixes scales across datasets.
  • ad hoc to paper HDBSCAN clusters correspond to coherent ability requirements and GPT-4o Mini's labels for 5 sampled questions per cluster are correct.
    Appendix C.2 uses this to build relevance vectors; errors propagate to NIRT routing and interpretation.
invented entities (2)
  • 25-dimensional latent LLM ability vector theta_Mj
    purpose: Predicts performance and provides interpretable LLM strengths.
    Learned from the same routing training labels; only consistency checks compare pairs of models in Fig. 4, with no external benchmark of ability estimates.
  • Latent query difficulty b_i and discrimination a_i
    purpose: Predicts performance and supports interpretations such as MATH question difficulty.
    Difficulty values are fitted; two MATH examples in Fig. 5 are a post hoc illustration, not a predictive test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory." pith.science (2026). https://pith.science/paper/KIQ7B6MI

@misc{pith2026250601048,
  author       = {Pith},
  title        = {Pith review of: IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KIQ7B6MI}},
  note         = {Machine review of arXiv:2506.01048}
}
read the original abstract

Large language models (LLMs) have demonstrated exceptional performance across a wide range of natural language tasks. However, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. While powerful models deliver better results, they come at a high cost, whereas smaller models are more cost-effective but less capable. To address this trade-off, we propose IRT-Router, a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. Inspired by Item Response Theory (IRT), a psychological measurement methodology, IRT-Router explicitly models the relationship between LLM capabilities and user query attributes. This not only enables accurate prediction of response performance but also provides interpretable insights, such as LLM abilities and query difficulty. Additionally, we design an online query warm-up technique based on semantic similarity, further enhancing the online generalization capability of IRT-Router. Extensive experiments on 20 LLMs and 12 datasets demonstrate that IRT-Router outperforms most baseline methods in terms of effectiveness and interpretability. Its superior performance in cold-start scenarios further confirms the reliability and practicality of IRT-Router in real-world applications. Code is available at https://github.com/Mercidaiha/IRT-Router.

Figures

Figures reproduced from arXiv: 2506.01048 by the authors.

Figure 1
Figure 1. Four representative LLMs’ output pricing and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. LLM Router. Queries are assigned to different [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Framework of IRT-Router. The left side represents query and LLM embedding, the middle performs [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Four LLMs’ ability values across 25 dimensions. From top to bottom, the 4 average ability values are [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Reward of IRT-Router (All and w/o Warm-up) [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Reward of IRT-Router (All and w/o Warm-up) [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Results of MIRT-Router on ID test set when [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 10
Figure 10. Figure 10: Four LLMs’ ability values across 25 dimen [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 9
Figure 9. Figure 9: Prompt for getting relevance vector. C.3 LLM Ability Visualization [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. R2-Router: A New Paradigm for LLM Routing with Reasoning

    cs.CL 2026-02 conditional novelty 6.0 of 10

    R2-Router jointly selects the LLM and an output-token budget, modeling each model as a quality-cost curve rather than a fixed point, and reports 4-5x cost savings on its new R2-Bench.

Reference graph

Works this paper leans on

61 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, et al. 2023. Automix: Automatically mixing language models. arXiv preprint arXiv:2310.12963

    arXiv 2023

  4. [4]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  5. [5]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts. arXiv preprint arXiv:2407.06204

    arXiv 2024

  6. [6]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176

    arXiv 2023

  7. [7]

    Shuhao Chen, Weisen Jiang, Baijiong Lin, James T Kwok, and Yu Zhang. 2024 a . Routerdc: Query-based router by dual contrastive learning for assembling large language models. arXiv preprint arXiv:2409.19886

    arXiv 2024

  8. [8]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2024 b . https://arxiv.org/abs/2412.21187 Do not think that much for 2+3=? on the overthinking of o1-like llms . Preprint, arXiv:2412.21187

    arXiv 2024

Show all 61 references
  1. [9]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv. org/abs/...

    2024 arXiv

  2. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  3. [11]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [12]

    Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John Lui. 2024. Cost-effective online multi-llm selection with versatile reward models. arXiv preprint arXiv:2405.16587

    2024 arXiv

  5. [13]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid llm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618

    2024 arXiv

  6. [14]

    Tao Feng, Yanzhen Shen, and Jiaxuan You. 2024. Graphrouter: A graph-based router for llm selections. arXiv preprint arXiv:2410.03834

    2024 arXiv

  7. [15]

    Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. 2021. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th international ACM SIGIR conference on research and developmen...

    2021

  8. [16]

    Weibo Gao, Hao Wang, Qi Liu, Fei Wang, Xin Lin, Linan Yue, Zheng Zhang, Rui Lv, and Shijin Wang. 2023. Leveraging transferable knowledge concept graph embedding for cold-start cognitive diagnosis. In Proceedings of the 46th international ACM SIGIR conference on research and de...

    2023

  9. [17]

    Maharshi Gor, Hal Daum \'e III, Tianyi Zhou, and Jordan Boyd-Graber. 2024. Do great minds think alike? investigating human-ai complementarity in question answering with caimira. arXiv preprint arXiv:2410.06524

    2024 arXiv

  10. [18]

    Gauthier Guinet, Behrooz Omidvar-Tehrani, Anoop Deoras, and Laurent Callot. 2024. Automated evaluation of retrieval-augmented language models with task-specific exam generation. arXiv preprint arXiv:2405.13622

    2024 arXiv

  11. [19]

    Surya Narayanan Hari and Matt Thomson. 2023. Tryage: Real-time, intelligent routing of user prompts to large language model. arXiv preprint arXiv:2308.11601

    2023 arXiv

  12. [20]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . Preprint, arXiv:2111.09543

    2023 arXiv

  13. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  14. [22]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  15. [23]

    Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. 2024. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403.12031

    2024 arXiv

  16. [24]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36

    2024

  17. [25]

    Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv preprint arXiv:2403.14403

    2024 arXiv

  18. [26]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. https://arxiv.org/abs/2306.09212 Cmmlu: Measuring massive multitask language understanding in chinese . Preprint, arXiv:2306.09212

    2024 arXiv

  19. [27]

    Mingjia Li, Hong Qian, Jinglan Lv, Mengliang He, Wei Zhang, and Aimin Zhou. 2025. Foundation model enhanced derivative-free cognitive diagnosis. Frontiers of Computer Science, 19(1):191318

    2025

  20. [28]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  21. [29]

    Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. 2024 b . Socraticlm: Exploring socratic personalized teaching with large language models. Advances in Neural Information Processing Systems, 37:85693--85721

    2024

  22. [30]

    Qi Liu, Zheng Gong, Zhenya Huang, Chuanren Liu, Hengshu Zhu, Zhi Li, Enhong Chen, and Hui Xiong. 2024 c . Multi-dimensional ability diagnosis for machine learning algorithms. Science China Information Sciences, 67(12):1--2

    2024

  23. [31]

    Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, 33(1):100--115

    2019

  24. [32]

    Yang Liu, Alan Medlar, and Dorota Glowacka. 2023. What we evaluate when we evaluate recommender systems: Understanding recommender systems’ performance using item response theory. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 658--670

    2023

  25. [33]

    Yueyue Liu, Hongyu Zhang, Yuantian Miao, Van-Hoang Le, and Zhiqiang Li. 2024 d . Optllm: Optimal assignment of queries to large language models. arXiv preprint arXiv:2405.15130

    2024 arXiv

  26. [34]

    Yunting Liu, Shreya Bhandari, and Zachary A Pardos. 2024 e . Leveraging llm-respondents for item evaluation: a psychometric analysis. arXiv preprint arXiv:2407.10899

    2024 arXiv

  27. [35]

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311.08692

    2023 arXiv

  28. [36]

    Jie Ma, Zhitao Gao, Qi Chai, Wangchun Sun, Pinghui Wang, Hongbin Pei, Jing Tao, Lingyun Song, Jun Liu, Chen Zhang, et al. 2025. Debate on graph: a flexible and reliable reasoning framework for large language models. In Proceedings of the AAAI Conference on Artificial Intellige...

    2025

  29. [37]

    Fernando Mart \' nez-Plumed, David Castellano, Carlos Monserrat-Aranda, and Jos \'e Hern \'a ndez-Orallo. 2022. When ai difficulty is easy: The explanatory power of predicting irt difficulty. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7719--7727

    2022

  30. [38]

    Fernando Mart \' nez-Plumed, Ricardo BC Prud \^e ncio, Adolfo Mart \' nez-Us \'o , and Jos \'e Hern \'a ndez-Orallo. 2019. Item response theory in ai: Analysing machine learning classifiers at the instance level. Artificial intelligence, 271:18--42

    2019

  31. [39]

    Leland McInnes, John Healy, Steve Astels, et al. 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11):205

    2017

  32. [40]

    Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426

    2018 arXiv

  33. [41]

    Alireza Mohammadshahi, Arshad Rafiq Shaikh, and Majid Yazdani. 2024. https://arxiv.org/abs/2401.13979 Routoo: Learning to route to large language models effectively . Preprint, arXiv:2401.13979

    2024 arXiv

  34. [42]

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665

    2024 arXiv

  35. [43]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822

    2018 arXiv

  36. [44]

    Guillem Ram \' rez, Alexandra Birch, and Ivan Titov. 2024. Optimising calls to large language models with uncertainty-based two-tier selection. arXiv preprint arXiv:2405.02134

    2024 arXiv

  37. [45]

    Mark D Reckase. 2009. Multidimensional item response theory models. In Multidimensional item response theory, pages 79--112. Springer

    2009

  38. [46]

    Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P Lalor, Robin Jia, and Jordan Boyd-Graber. 2021. Evaluation examples are not equally informative: How should that change nlp leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computationa...

    2021

  39. [47]

    Marija S akota, Maxime Peyrard, and Robert West. 2024. Fly-swat or cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 606--615

    2024

  40. [48]

    Tal Shnitzer, Anthony Ou, M \' rian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789

    2023 arXiv

  41. [49]

    Jovan Stojkovic, Chaojie Zhang, \'I \ n igo Goiri, Josep Torrellas, and Esha Choukse. 2024. Dynamollm: Designing llm inference clusters for performance and energy efficiency. arXiv preprint arXiv:2408.00741

    2024

  42. [50]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    2018 arXiv

  43. [51]

    Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. 2020. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6153--6161

    2020

  44. [52]

    David J Woodruff and Bradley A Hanson. 1996. Estimation of item response models using the em algorithm for finite mixtures

    1996

  45. [53]

    Mayi Xu, Yongqi Li, Ke Sun, and Tieyun Qian. 2024. Adaption-of-thought: Learning question difficulty improves large language models for reasoning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5468--5495

    2024

  46. [54]

    Shangzi Xue, Zhenya Huang, Jiayu Liu, Xin Lin, Yuting Ning, Binbin Jin, Xin Li, and Qi Liu. 2024. Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle. Advances in Neural Information Processing Systems, 37:357--385

    2024

  47. [55]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  48. [56]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600

    2018 arXiv

  49. [57]

    Yixuan Zhang and Haonan Li. 2023. https://arxiv.org/abs/2310.09550 Can large language model comprehend ancient chinese? a preliminary test on aclue . Preprint, arXiv:2310.09550

    2023 arXiv

  50. [58]

    Zheng Zhang, Qi Liu, Hao Jiang, Fei Wang, Yan Zhuang, Le Wu, Weibo Gao, and Enhong Chen. 2023. Fairlisa: fair user modeling with limited sensitive attributes information. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 41432--41450

    2023

  51. [59]

    Zheng Zhang, Wei Song, Qi Liu, Qingyang Mao, Yiyan Wang, Weibo Gao, Zhenya Huang, Shijin Wang, and Enhong Chen. 2024. Towards accurate and fair cognitive diagnosis via monotonic data augmentation. Advances in Neural Information Processing Systems, 37:47767--47789

    2024

  52. [60]

    Hongke Zhao, Likang Wu, Yuqing Shan, Zonghan Jin, Yuanpei Sui, Zipeng Liu, Nan Feng, Minqiang Li, and Wei Zhang. 2024. A comprehensive survey of large language models in management: Applications, challenges, and opportunities. Challenges, and Opportunities (August 14, 2024)

    2024

  53. [61]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.