CP-Router: An Uncertainty-Aware Router Between LLM and LRM

Fulin Lin; Han Zheng; Hongwei Wang; Jiayuan Su; Lu Cheng; Teng Wang; XinLong Zhao; Zhaopeng Feng; Zhenyu Xiao; Zuozhu Liu

REVIEW 4 major objections 3 minor 12 references

CP-Router: An Uncertainty-Aware Router Between LLM and LRM

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conformal prediction set size routes prompts and saves tokens

desk verdict Solid idea, unreliable numbers: test-set alpha selection undermines the reported gains; worth a revision, not a desk reject. read the letter →

arxiv 2505.19970 v1 pith:J4E37L5F submitted 2025-05-26 cs.CL

Jiayuan Su , Fulin Lin , Zhaopeng Feng , Han Zheng , Teng Wang , Zhenyu Xiao , Xinlong Zhao , Zuozhu Liu

show 2 more authors

Lu Cheng Hongwei Wang

This is my paper · ORCID

classification cs.CL

keywords LLMroutinglargereasoningmodelsconformalpredictionuncertaintyquantificationmultiple-choiceQAtokenefficiencyadaptivecalibrationoverthinking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CP-Router claims that the expensive long-chain reasoning of large reasoning models can be reserved for genuinely hard questions by using conformal prediction on the cheap LLM's multiple-choice probabilities. For each prompt the router builds a prediction set of plausible answer options; a singleton set means the LLM answers alone, while a larger set sends the prompt to the reasoning model. Because the conformal error rate alpha is hard to choose, the paper adds a Full and Binary Entropy (FBE) criterion that picks the alpha whose prediction-set sizes separate most cleanly into easy and hard prompts. On mathematics, logic, and Chinese chemistry benchmarks, this training-free rule reduces token consumption while keeping accuracy at or above the reasoning-model-alone baseline. The payoff, if the central claim holds, is a principled uncertainty signal that tells an application when expensive reasoning is actually needed.

What carries the argument

The central object is the conformal prediction set and its size distribution. Conformal prediction takes the LLM's softmax scores over answer options, computes the quantile $\hat{q}$ of calibration scores $1 - f(y)$, and outputs all options with score below $\hat{q}$; the size of this set is the uncertainty proxy that decides routing. The complementary mechanism is FBE, which combines full entropy over all set sizes with binary entropy over the singleton/non-singleton split, so that the chosen error rate $\alpha^*$ both spreads uncertainty estimates and avoids routing imbalance. The quantile step is what converts a raw softmax probability into a set with a finite-sample coverage guarantee.

What would settle it

On any benchmark used in the paper, take CP-Router's routing decisions and compare each decision with an oracle that sends the prompt to whichever model answers it correctly. If a substantial share of singleton-set prompts are answered correctly by the reasoning model but not by the LLM, or multi-option prompts are answered correctly by the LLM but not by the reasoning model, then set size is not tracking per-prompt benefit; the central claim would also be falsified if CP-Router's overall accuracy fell below both the LLM-alone and LRM-alone baselines on any dataset.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that prediction-set size from conformal prediction is a usable routing signal between a large language model and a large reasoning model in multiple-choice question answering. The score is $1 - f(y)$, where $f(y)$ is the LLM's softmax probability for option $y$, and the prediction set is $C(x) = \{ y : 1 - f(y) \le \hat{q} \}$ for a calibration quantile $\hat{q}$. The routing rule is set-size aware: small sets go to the LLM, large sets to the LRM. The paper's new component is FBE, $\mathrm{FBE} = \beta H_{\mathrm{full}} + H_{\mathrm{binary}}$, which scores candidate error rates by the entropy of the set-size distribution and the singleton/non-singleton balance, and the rate maximizing FBE is used for routing. Empirically this yields the highest token utility on all six benchmarks for one model pairing and five of six for another, with accuracy sometimes above the LRM alone, and the same rule transfers to new model pairings and to open-ended QA through a generated five-option format.

Load-bearing premise

The load-bearing premise is that the LLM's conformal prediction set size is a trustworthy per-prompt signal of which questions the LLM can answer correctly, so 'singleton means easy, multi-option means hard' holds for individual prompts and not just on average.

Editorial extensions

If this is right

On MCQA workloads, deploying CP-Router removes a large share of reasoning tokens (e.g., 22.8% on Elementary Math with the Qwen pairing) while matching or beating LRM-alone accuracy, so cost savings do not come with an accuracy penalty.
Since the router needs only the LLM's logits and a calibration set, it can be applied to arbitrary LLM-LRM pairs, including cross-family pairs, without retraining or model-specific tuning.
The FBE calibration rule eliminates manual tuning of the conformal error rate, which the paper shows otherwise causes over- or under-routing and poor separation of easy from hard prompts.
The same uncertainty signal extends beyond fixed-choice tasks: by prompting the LLM for its top four answers plus an 'Others' option, CP-Router saves reasoning tokens on about one third of GSM8K questions with only a 1.1% accuracy drop.
On datasets where the LLM is already the stronger model, routing still helps: on CN-Chemistry with the Qwen pairing, CP-Router improves on the LLM's accuracy by 0.8% while cutting token use by 68%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to use the same set-size rule with cost measured in dollars, latency, or energy rather than tokens; the routing signal is model-agnostic and would need only a per-model cost curve.
Because the paper selects alpha by maximizing FBE on the test set, a deployment version would need a separate held-out split; the reported gains may partly reflect favorable test-set calibration rather than the rule itself.
The set-size proxy is a marginal, not conditional, uncertainty statement; an oracle experiment comparing per-prompt routing decisions with both models' correctness would reveal where the proxy misroutes, and could motivate a threshold that depends on dataset difficulty.
The FBE objective could be reused as a cheap data-selection or curriculum signal for deciding when a reasoning model adds value, not just for routing.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

CP-Router has a genuinely nice idea: instead of trying to calibrate a confidence threshold, use conformal prediction set size as the routing signal between a cheap LLM and an expensive LRM, and pick the CP error rate alpha with an entropy heuristic (FBE) that maximizes the spread of set sizes. That is a clean, training-free, model-agnostic way to decide which prompts need the heavy model. The paper is clearly written, covers several model pairings and benchmarks, and includes ablations showing both entropy terms contribute. The CP coverage guarantee is correctly stated, and there is no circular derivation in the method itself.

The problem is the evaluation. Section 3.2 and Method 1 explicitly choose alpha by maximizing FBE on the prediction sets computed on the test set, and then the same test set is used to report accuracy and token reduction. That is test-set adaptation. It is not label leakage, but it gives CP-Router a free pass to pick the alpha that best separates easy and hard prompts on exactly the prompts being evaluated. The baselines, by contrast, are reported at fixed thresholds with no comparable held-out selection rule. So the headline TRR and accuracy gains in Table 1 and Figure 5 are likely inflated relative to what a deployed router would achieve. This is the first thing I would want fixed: pick alpha on a validation split, or at least show sensitivity to the choice of alpha without test-set adaptation.

There are smaller issues. The proxy assumption is validated only at the dataset level (Figure 4), not per prompt; the routing threshold tau appears in the pseudocode but is barely discussed in the text; and the extension to open-ended QA by forcing a five-option MCQ is a bit ad hoc. None of these are fatal. The core method is plausible and the idea is worth pursuing.

Overall, this is a conditional accept in my book. The evaluation protocol is fixable, and the underlying idea is solid enough that a serious referee should engage with it. I would send it to review without hesitation, with the clear message that the test-set alpha selection must be addressed.

Referee Report

4 major / 3 minor

Summary. The paper proposes CP-Router, a training-free router between an LLM and an LRM based on conformal prediction set sizes under the LLM. For MCQA prompts, the router sends prompts with singleton prediction sets to the LLM and all others to the LRM. The error rate α is selected by maximizing a novel entropy criterion, FBE, over a grid, and the paper notes explicitly that this selection is performed on the test set. Experiments on seven MCQA benchmarks and an open-ended QA extension report token reduction with maintained or improved accuracy relative to using the LRM alone, across several model pairings.

Significance. If the reported efficiency gains are real, CP-Router would be a useful lightweight contribution: it requires no training, applies to arbitrary LLM/LRM pairs, and leverages a standard conformal framework with finite-sample coverage guarantees. The FBE criterion is simple, and the ablation isolating the entropy components is informative. The main weakness is the evaluation protocol: α is fitted to the test set, so the headline comparisons in Table 1 and Figure 5 are not trustworthy as they stand. The per-prompt proxy assumption is also under-validated. With a proper held-out selection procedure and per-prompt conditional analysis, the approach deserves further consideration.

major comments (4)

[Section 3.2 / Method 1] Section 3.2, Eq. (5), together with Method 1, Step 2, selects α* by maximizing FBE on prediction sets generated from the test set, and the same test set is then used in Step 3 and Table 1 to report accuracy, TRR, and Utoken. This is test-set adaptation rather than label leakage: the router's threshold is chosen on the exact prompts being evaluated. Baselines such as Random, Top-1, and Entropy are evaluated at fixed thresholds with no comparable held-out tuning, so the reported gains over these baselines may be an artifact of selection. Please re-run with α selected on a calibration or validation split disjoint from the reported test set, report the selected α values, and apply the same selection protocol to the baselines; without this, the central efficiency claim is not established.
[Section 3.1 / Figure 4] The proxy assumption underlying the routing rule, namely that a singleton prediction set implies the LLM is the better model for that prompt and a larger set implies the LRM is needed, is only checked with dataset-level aggregates. Figure 4 shows average prediction set size and aggregate accuracy across two datasets (GPQA and CN-Chemistry). Aggregates do not support per-prompt routing decisions. Please provide conditional calibration curves, such as the accuracy of each model binned by set size or per-prompt accuracy differences against set size, on held-out data, and report how often the router's decision matches the per-prompt better model. Without this, the FBE-selected threshold may simply separate datasets rather than prompts.
[Equations (4)-(5) / Table 3] The FBE objective in Eq. (4) maximizes the entropy of the set-size distribution and does not contain any term for accuracy or token cost of the downstream routing. A more separated set-size distribution need not correspond to a better accuracy/token tradeoff, and Table 3 shows that different entropy weightings lead to very different Acc/TRR operating points. To justify α* selection, please compare FBE-selected α against a small set of fixed α values and against an oracle α chosen on a validation set to optimize Utoken, and report the spread over calibration/test splits. The current table does not establish that FBE is a better calibration rule than selecting α by validation accuracy.
[Method 1 / Section 5.3] Method 1 declares a routing threshold τ and Eq. (4) introduces a weight β, but the main text never specifies how τ is set and states only that a 3:1 weighting is adopted “based on these findings” in Section 5.3. If τ and β are tuned on the test benchmarks, they add further test-set adaptation beyond α; if they are fixed, please say so explicitly and justify the fixed values. Please also report the sensitivity of Table 1 to τ.

minor comments (3)

[Table 3] The column header “Full Binary” and the row labels “1 1”, “2 1”, and “3 1” are confusing; please clarify which entries indicate the included entropy component and which indicate the weight β.
[Equation (6)] Utoken is described as “accuracy gain over token usage ratio,” but the formula can be negative and unbounded; please explain its interpretation and why it is preferable to reporting Acc and TRR separately.
[Table 2] The column header “Standard Reasoning” is ambiguous; please specify which model or mode of inference that column reports.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the routing rule and FBE calibration are not equivalent to the reported accuracy/token metrics by construction.

full rationale

CP-Router's pipeline is: (i) compute softmax scores from the LLM over answer options; (ii) apply CP quantile q_hat to form prediction sets; (iii) route to the LRM iff the set size exceeds tau; (iv) measure accuracy and TRR on the same prompts. The routing decision is a deterministic function of the LLM's own predictive distribution and the calibration quantile; it does not use the true label, the LRM's response, or the final accuracy. Thus the headline result is not true by definition. The FBE criterion (Eq. 4) selects alpha by maximizing a function of the set-size distribution, not by maximizing accuracy or token savings, so the reported Acc/TRR values are not algebraically forced by the selection rule. The empirical premise that low-uncertainty (small-set) prompts are well handled by the LLM is supported by dataset-level comparisons in Figure 4 and by the case studies, rather than being encoded in the prediction-set definition. Self-citations (Su et al. 2024; Pan et al. 2024) occur only in related work and as a baseline, and the conformal coverage theorem is cited to Vovk et al. rather than to the authors' own prior work; none of these citations carries the central claim. The notable protocol issue—alpha is selected by maximizing FBE on the same test set used to report Acc/TRR (Eq. 5; Method 1 steps 7-19)—is an evaluation-fairness/validity concern, not a circular derivation: it does not make the output equivalent to the input, and the FBE objective does not directly optimize the reported metrics. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. The load-bearing assumptions are the proxy relationship between prediction set size and routing benefit, and the FBE heuristic. Key free parameters are beta and tau; the candidate alpha grid is also unspecified. The most significant data-dependent step is the selection of alpha on the test set, which is a methodological concern rather than an entity.

free parameters (3)

beta (FBE weight) = 3
Weight of full entropy in FBE, chosen based on ablation on Elementary Math and High School Math subsets (Table 3), used in all experiments.
tau (routing threshold) = 1
Threshold on prediction set size; if set size is less than or equal to 1, route to LLM. Not systematically varied, despite being a free parameter.
candidate alpha set = not specified
Grid of candidate alpha values used in FBE optimization (Eq. 5) is not reported; a free experimental choice.

assumptions (4)

standard math Calibration and test sets are exchangeable (i.i.d.), as required for CP coverage guarantee.
Assumed in Theorem 2.1 and throughout the CP construction.
domain assumption The softmax probabilities of the LLM over answer choices define a valid nonconformity score S(x,y)=1-f(y).
This score is standard in CP for classification (Sadinle et al., 2019); its validity for LLM logits is assumed.
ad hoc to paper Prediction set size is a reliable per-prompt indicator of whether the LRM will be more accurate than the LLM.
Core hypothesis of the router (Section 3.1), supported only by dataset-level aggregate evidence (Figure 4), not per-prompt validation.
ad hoc to paper Maximizing FBE selects an alpha that yields good accuracy/token trade-offs.
FBE is a heuristic; no formal link to routing performance is given. The ablation (Table 3) is on two subsets only.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CP-Router: An Uncertainty-Aware Router Between LLM and LRM." pith.science (2026). https://pith.science/paper/J4E37L5F

@misc{pith2026250519970,
  author       = {Pith},
  title        = {Pith review of: CP-Router: An Uncertainty-Aware Router Between LLM and LRM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4E37L5F}},
  note         = {Machine review of arXiv:2505.19970}
}

read the original abstract

Recent advances in Large Reasoning Models (LRMs) have significantly improved long-chain reasoning capabilities over Large Language Models (LLMs). However, LRMs often produce unnecessarily lengthy outputs even for simple queries, leading to inefficiencies or even accuracy degradation compared to LLMs. To overcome this, we propose CP-Router, a training-free and model-agnostic routing framework that dynamically selects between an LLM and an LRM, demonstrated with multiple-choice question answering (MCQA) prompts. The routing decision is guided by the prediction uncertainty estimates derived via Conformal Prediction (CP), which provides rigorous coverage guarantees. To further refine the uncertainty differentiation across inputs, we introduce Full and Binary Entropy (FBE), a novel entropy-based criterion that adaptively selects the appropriate CP threshold. Experiments across diverse MCQA benchmarks, including mathematics, logical reasoning, and Chinese chemistry, demonstrate that CP-Router efficiently reduces token usage while maintaining or even improving accuracy compared to using LRM alone. We also extend CP-Router to diverse model pairings and open-ended QA, where it continues to demonstrate strong performance, validating its generality and robustness.

Figures

Figures reproduced from arXiv: 2505.19970 by the authors.

**Figure 2.** Key Components of the CP-Router Framework. (a) CP-Based Routing. For each prompt, it applies CP with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Different prediction set size distributions. A [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: (a) Accuracy comparison between Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Accuracy and prompt allocation of CP-Router [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Accuracy and prompt allocation of CPRouter on the open-ended QA benchmark GSM8K, using Llama-3-8B and DeepSeek-R1-Distill-Llama-8B, demonstrating its generalization to broader QA formats. 5.5 CP-Router Generalizes Across Diverse Model Pairings CP-Router is designed t…

**Figure 7.** Figure 7: CP-Router performance on GPQA under dif [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 6 canonical work pages

[1]

A pre- defined threshold controls the proportion di- rected to the LRM

Random Routing: Prompts are randomly routed to either the LLM or LRM. A pre- defined threshold controls the proportion di- rected to the LRM

work page
[2]

If this probability exceeds a predefined threshold, the LLM is used; otherwise, the LRM is chosen

Top-1 Probability Routing: The model se- lects the option with the highest probability as a proxy for confidence. If this probability exceeds a predefined threshold, the LLM is used; otherwise, the LRM is chosen

work page
[3]

If the entropy exceeds a predefined threshold, the LRM is used; otherwise, the LLM is selected

Response Entropy Routing: The entropy of the option probabilities is computed. If the entropy exceeds a predefined threshold, the LRM is used; otherwise, the LLM is selected

work page
[4]

arXiv preprint arXiv:2412.16720

Openai o1 system card. arXiv preprint arXiv:2412.16720. Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam

arXiv
[5]

bottom line ethics

Explicit Self-Awareness Routing: The LLM generates both the answer and a self- assessment of whether it requires deliberation by the LRM. If so, the LRM is invoked. C Case Study As shown in Tables 4 and 5, CP-Router correctly identifies prompts where the LLM is both confident (i.e., low uncertainty with a prediction set size of 1) and accurate, thereby av...

work page
[6]

In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995, Miami, Florida, USA

API is enough: Conformal prediction for large language models without logit-access. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995, Miami, Florida, USA. Association for Computational Linguistics. Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie. 2025. Between underthinking and overthink- ing: An empiric...

arXiv 2024
[7]

ACM Computing Survey

Conformal prediction: A data perspective. ACM Computing Survey. A Pseudocode We provide the pseudocode of CP-Router in Method 1. Method 1 CP-Router Require: Prompt x(i), Ground truth y(i), LLM fθ, LRM fϕ, Answer options A, Calibration set Dcal, Test set Dtest, Threshold τ, Candidate error rates {αk} 1: Step 1: Calibration (Compute nonconformity scores) 2:...

work page
[11]

Dynathink (Pan et al., 2024): Based on re- peated sampling, if more than half of the sam- pled answers agree on an option, the prompt is routed to the LLM; otherwise, it is routed to the LRM

work page 2024

Show all 12 references

[2020]

arXiv preprint arXiv:2009.03300

Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richard- son, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al

2009 arXiv
[2023]

arXiv preprint arXiv:2305.18404

Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical...

2024 arXiv
[2024]

Advances in Neural Information Processing Systems, 37:114812–114842

Large language model validity via enhanced conformal prediction methods. Advances in Neural Information Processing Systems, 37:114812–114842. Yu-Neng Chuang, Helen Zhou, Prathusha Kameswara Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, and Xia Hu. 2024. Learning to rout...

2024 arXiv
[2025]

arXiv preprint arXiv:2503.05179

Sketch-of-thought: Efficient llm reasoning with adaptive cognitive-inspired sketching. arXiv preprint arXiv:2503.05179. Marthe Ballon, Andres Algaba, and Vincent Ginis. 2025. The relationship between reasoning and performance in large language models–o3 (mini) thinks harder, n...

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

A pre- defined threshold controls the proportion di- rected to the LRM

Random Routing: Prompts are randomly routed to either the LLM or LRM. A pre- defined threshold controls the proportion di- rected to the LRM

work page

[2] [2]

If this probability exceeds a predefined threshold, the LLM is used; otherwise, the LRM is chosen

Top-1 Probability Routing: The model se- lects the option with the highest probability as a proxy for confidence. If this probability exceeds a predefined threshold, the LLM is used; otherwise, the LRM is chosen

work page

[3] [3]

If the entropy exceeds a predefined threshold, the LRM is used; otherwise, the LLM is selected

Response Entropy Routing: The entropy of the option probabilities is computed. If the entropy exceeds a predefined threshold, the LRM is used; otherwise, the LLM is selected

work page

[4] [4]

arXiv preprint arXiv:2412.16720

Openai o1 system card. arXiv preprint arXiv:2412.16720. Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam

arXiv

[5] [5]

bottom line ethics

Explicit Self-Awareness Routing: The LLM generates both the answer and a self- assessment of whether it requires deliberation by the LRM. If so, the LRM is invoked. C Case Study As shown in Tables 4 and 5, CP-Router correctly identifies prompts where the LLM is both confident (i.e., low uncertainty with a prediction set size of 1) and accurate, thereby av...

work page

[6] [6]

In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995, Miami, Florida, USA

API is enough: Conformal prediction for large language models without logit-access. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 979–995, Miami, Florida, USA. Association for Computational Linguistics. Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie. 2025. Between underthinking and overthink- ing: An empiric...

arXiv 2024

[7] [7]

ACM Computing Survey

Conformal prediction: A data perspective. ACM Computing Survey. A Pseudocode We provide the pseudocode of CP-Router in Method 1. Method 1 CP-Router Require: Prompt x(i), Ground truth y(i), LLM fθ, LRM fϕ, Answer options A, Calibration set Dcal, Test set Dtest, Threshold τ, Candidate error rates {αk} 1: Step 1: Calibration (Compute nonconformity scores) 2:...

work page

[8] [11]

Dynathink (Pan et al., 2024): Based on re- peated sampling, if more than half of the sam- pled answers agree on an option, the prompt is routed to the LLM; otherwise, it is routed to the LRM

work page 2024