Pith. sign in

REVIEW 3 major objections 5 minor 61 references

Bi-directional Model Cascading with Proxy Confidence

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that combining internal-state confidence of the small model with a tiny proxy's prediction of the large model's confidence improves cascade deferral, cutting defers to the large model by up to 42.5%.

desk verdict The backward internal-state confidence result is solid and likely reproducible, but the forward proxy adds little on most datasets and the bi-directional claim is oversold. read the letter →

arxiv 2504.19391 v2 pith:5J5B4RBZ submitted 2025-04-27 cs.LG

David Warren , Mark Dras This is my paper · ORCID
classification cs.LG
keywords modelcascadingconfidencecalibrationinternalhiddenstatesproxydeferraldecisionsefficientLLMinferencemultiple-choiceQALLaMA-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to make model cascading cheaper by improving the decision of when to send a question from a small language model to a large one. Its central claim is that deferral should be guided by both models' confidence at once: a richer "backward" confidence drawn from the small model's internal layers, and a "forward" confidence for the large model, predicted by a tiny proxy model before the large model is ever run. The paper reports that this bi-directional method consistently beats deferral curves based on maximum probability or entropy, and on three of its four benchmarks it outperforms even the internal-state-only version. On matched system accuracy, the method reduces the number of items deferred to the 70-billion-parameter model by up to 42.5% compared with the standard maximum-probability baseline, with larger cost savings still when deferred prompt lengths are factored in.

What carries the argument

The object that carries the argument is a deferral feature vector combining backward and forward confidence. Backward confidence is the vector of pseudo-probabilities $q_{M,j}(y|x)=\mathrm{softmax}(\phi(h^{(j)}))$ computed by applying the model's final affine vocabulary layer to hidden states $h^{(j)}$ at selected intermediate layers, giving a picture of how early and stably each candidate answer emerged. Forward confidence is a scalar $M_A(x)$ from a tiny auxiliary model that predicts, from input text only, the binned mean accuracy of the large model's predictions; the bins come from ordering training samples by the large model's output entropy. A random forest meta-model maps these features to the probability that the sample is a "gain" (small model wrong, large model right), and the cascade defers when that probability exceeds the operating threshold.

What would settle it

Compute the proxy model's AUROC for predicting large-model correctness on a held-out fold of a new dataset and compare BI-DIR against BACKINT. The paper's own numbers give the pattern to test: near-chance proxy AUROC on BoolQ (0.4757) coincides with BI-DIR not beating BACKINT there. Finding a dataset with high proxy AUROC where BI-DIR gains nothing, or near-chance proxy AUROC where BI-DIR still gains clearly, would settle whether the proxy is doing the work.

Watch

Extended reading notes

Core claim

Cascades usually defer a sample when the small model's final-layer maximum probability is low. The paper's discovery is that a deferral model fed two complementary confidence signals does better. The first signal comes from applying the model's vocabulary projection at every transformer layer and concatenating the resulting pseudo-probabilities, which gives a vector representation of how the answer evolved through the network; this "backward" representation is better calibrated than final-layer max probability and alone improves cascade deferral. The second signal is a "forward" proxy estimate of the large model's confidence: a small auxiliary model is trained, on a one-off pass of the large model over training data, to predict the large model's binned mean accuracy from the input text alone, so the large model never has to be invoked for the decision. A random forest then learns to flag samples where the small model is wrong and the large model is right. The paper reports that this bi-directional system yields the best deferral AUC on ARC-Easy, ARC-Challenge, and MMLU, matches internal-state-only confidence on the binary BoolQ, and reduces deferrals by up to 42.5% at matched performance.

Load-bearing premise

The load-bearing premise is that a tiny proxy model, reading only the input text, can estimate the large model's confidence or correctness accurately enough to improve deferral decisions over small-model confidence alone.

Editorial extensions

If this is right

  • Cascade operators can replace final-layer maximum probability with internal-state confidence and get better deferral curves without retraining the small or large models.
  • Because the proxy is tiny and runs before the large model, the deferral decision itself costs almost nothing, so the savings are not eaten by the decision overhead.
  • The matching of deferral rate at equal accuracy means the same system accuracy can be had with fewer large-model calls; on BoolQ at the 0.4 deferral rate that is 42.5% fewer defers.
  • The learned meta-model tends to defer shorter prompts on average than the max-probability baseline, so real cost savings exceed the item-count reductions.
  • The method extends in principle to cascades of more than two models, since each adjacent pair can use the same backward/forward confidence pair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • On the paper's own calibration table, the proxy's AUROC for predicting large-model correctness is near chance on BoolQ, ARC-Challenge, and CSQA; this suggests the forward signal's contribution is dataset-dependent, and a practical system should gate the proxy on its validation AUROC and fall back to internal-state confidence when it is near chance.
  • If proxy confidence is learned once per large model, the same auxiliary model could serve many small models or downstream tasks, decoupling the cost of building the proxy from its repeated use in cascades.
  • The bin-accuracy proxy target is coarse; a proxy trained to predict per-sample correctness or calibrated probabilities directly might close the remaining gap to the oracle forward confidence visible on MMLU and CSQA.
  • Because the experiments use only two models from one family, the result does not yet tell whether the proxy signal survives heterogeneous cascades where the small and large models are trained differently; that is a natural test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a bi-directional model-cascading deferral method for large language models. It combines a 'backward' confidence representation derived from internal hidden states of the small model with a 'forward' proxy confidence for the large model, estimated by a small auxiliary model that sees only the input text. A random-forest meta-model then decides whether to defer each sample to the larger model. The method is evaluated on five multiple-choice QA datasets using LLaMA-2 13B as the small model and 70B as the large model, against baselines including max-probability, entropy, a backward-only variant, and a random-deferral baseline, plus an oracle version that uses the large model's actual maximum probability. The paper reports deferral AUC, calibration metrics, bootstrap confidence intervals against MaxProb, ablations, and supplementary analyses of the auxiliary model and deferred prompt lengths.

Significance. If the claims held, the paper would make a useful contribution to LLM cascading: the backward internal-state confidence representation (BACKINT) appears to be a solid, well-evaluated improvement over max-probability deferral, with consistent gains and significant bootstrap intervals on almost all operating points. The experimental methodology is more careful than many cascade papers, including five datasets, an oracle skyline, calibration analyses, ablations, and deferred-length analysis. However, the distinctive forward-proxy contribution is not currently established: the proxy's correctness AUROC is near chance on three of five datasets, and no pairwise significance test against BACKINT is reported. The paper is therefore promising but needs additional evidence or a revised scope.

major comments (3)
  1. [§3.3, Table 7, Table 1] The central claim that the forward proxy contributes to deferral is not supported by the reported evidence. Table 7 shows the auxiliary forward model's correctness AUROC is near chance on BoolQ (0.4757), ARC-Challenge (0.5376), and CSQA (0.5395), and only MMLU shows a clear signal (0.7240). Correspondingly, Table 1 shows BI-DIR vs BACKINT differences of +0.0020 on ARC-Easy, +0.0066 on ARC-Challenge, +0.0096 on MMLU, and negative differences on BoolQ (-0.0045) and CSQA (-0.0018). The bootstrap intervals in Tables 11 and 12 compare each method only against MAXPROB; no confidence interval or test is reported for the BI-DIR vs BACKINT comparison, so the observed proxy gains could be noise. Please report pairwise confidence intervals or significance tests for BI-DIR vs BACKINT, or restrict the proxy-based claim to datasets where the proxy carries signal.
  2. [§1 Contribution list, §5.1 Table 3] The headline 'reductions in data items deferred to the larger model of up to 42.5%' is inconsistent with Table 3. Table 3 reports required deferral rates rd (e.g., BoolQ at the 0.4 operating point: rd=0.1511) and parenthetical values that the text in §5.1 identifies as 'estimated total cost reduction.' The 42.5% value is a cost reduction, not an item-deferral reduction; at the 0.4 operating point on BoolQ, the implied item-deferral reduction is approximately (0.4-0.1511)/0.4 = 62%, not 42.5%. Moreover, no cost model is defined: the assumed relative cost of large versus small model inference is never stated, so the cost-reduction percentages are not reproducible. Please correct the claim, define the cost model, and present item-deferral reductions separately from cost reductions.
  3. [Supplementary C, Table 7; §5.3] The paper states in Supplementary C that 'Auxiliary model confidence is poorly calibrated in general,' yet the method's main innovation relies on this proxy. The current calibration analysis only shows AUROC/Brier/ECE of the proxy against large-model correctness; it does not establish that the proxy provides incremental information for the deferral task beyond BACKINT. To justify the bi-directional design, please report an analysis of the proxy's contribution within the deferral model (e.g., feature importance, or deferral AUC with and without the proxy feature on each dataset), and address why a near-chance proxy on BoolQ, ARC-Challenge, and CSQA still leads to any deferral improvement.
minor comments (5)
  1. [§1] The word 'queryring' in the contribution list is a typo for 'querying'.
  2. [§4.2] The text says 'four commonly used multiple choice question-answering datasets' but then lists five datasets: BoolQ, MMLU, ARC-Easy, ARC-Challenge, and CSQA.
  3. [Table 7] The table caption says 'on the ARC-Easy Dataset' but the table reports results for all five datasets; please correct the caption.
  4. [§5.3] 'App G gives details' should read 'Appendix G gives details.'
  5. [Table 8] The abbreviation 'INDIVACC' in Table 8 is not defined in the caption; please define it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proxy and deferral models are learned estimators evaluated on held-out folds; self-citations are supplementary pointers only.

full rationale

The paper's derivation chain is not circular. The forward proxy MA is trained to predict binned mean accuracy of the large model (Sec 3.3), with targets acc(B_m) computed from large-model outputs and true labels on training folds; ConfF(ML(x)) = MA(x) is then a feature in a separately trained deferral model MD whose target is the gain yG = [MS(x) != y and ML(x) == y] (Sec 3.4). This is standard supervised learning: the proxy is a learned estimator, not a fitted constant, and all reported AUC/deferral results are on unseen test folds (Sec 4.1: 'only unseen test fold results are reported'). No equation in the paper defines the reported prediction as equivalent to its training input by construction. The one self-citation is to the paper's own supplementary material (ref [52], e.g., 'supplementary material I[52]'); it is a pointer, not load-bearing evidence and does not smuggle in an assumption. The manuscript itself flags the proxy's empirical weakness ('Auxiliary model confidence is poorly calibrated in general', Supp C; AUROC near chance on BoolQ 0.4757, ARC-C 0.5376, CSQA 0.5395 in Table 7), and the BI-DIR vs BACKINT differences are small and not bootstrap-tested (Table 11 tests only BI-DIR vs MAXPROB). Those are correctness/robustness concerns, not circularity: a weak or untested feature is not the same as a result that reduces to its own input. The 42.5% figure is an estimated total-cost reduction under an implicit cost model (Table 3), not a definitional identity, and the method is benchmarked against external accuracy labels. Therefore the claim is self-contained and non-circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three empirical premises: internal-layer pseudo-probabilities improve confidence for deferral, a tiny proxy can predict large-model confidence from text alone, and a random forest can learn the gain signal from these features. The number of entropy bins, the internal layer set, and the cost model are user-chosen quantities that affect the reported savings but are not varied in sensitivity analyses. No new physical entities are introduced.

free parameters (4)
  • Number of entropy bins for forward confidence target = 10
    Large-model entropy is divided into 10 equal intervals to compute bin accuracies (Sec 3.3); no sensitivity analysis for the number of bins is reported.
  • Internal layer set J for backward confidence = unspecified (described as all internal blocks)
    BACKINT uses pseudo-probabilities from internal layers of LLaMA-2 13B, but the exact layer indices and aggregation are not specified (Sec 3.2).
  • Deferral model hyperparameters = not reported
    The random forest classifier (Sec 3.4) is used without listing hyperparameters such as tree count or depth; these affect the deferral curves.
  • Cost ratio of large to small model = not stated in main text
    The cost reduction percentages in Table 3 require an assumed cost of LLaMA-2 70B relative to 13B; the main text does not provide the ratio.
assumptions (4)
  • domain assumption Large-model confidence can be predicted from input text alone.
    The forward proxy MA is trained on input text only, relying on the APRICOT ablation (Ulmer et al. [47]) that input-only confidence estimation is feasible; Sec 3.3.
  • domain assumption Internal-layer pseudo-probabilities from the small model provide a confidence representation that is more informative than final-layer maximum probability for deferral.
    This is the premise of BACKINT; it is supported empirically in Sec 5.2 but not derived. Sec 3.2.
  • domain assumption The deferral model can learn the 'gain' event (small model wrong, large model right) from the chosen features using a random forest.
    No theoretical guarantee; relies on the features having signal for the target. Sec 3.4.
  • domain assumption True labels are available for training the gain target and the forward proxy.
    The method requires labeled training examples where both models' outputs are known; in deployment this means building a labeled dataset. Sec 3.3-3.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bi-directional Model Cascading with Proxy Confidence." pith.science (2026). https://pith.science/paper/5J5B4RBZ

@misc{pith2026250419391,
  author       = {Pith},
  title        = {Pith review of: Bi-directional Model Cascading with Proxy Confidence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5J5B4RBZ}},
  note         = {Machine review of arXiv:2504.19391}
}
read the original abstract

Model Cascading, recently applied successfully to LLMs, is a simple but powerful technique that improves the efficiency of inference by selectively applying models of varying sizes. Models are used in sequence from smallest to largest, only deferring samples to large, costly models when smaller models are not sufficiently confident. Existing approaches to deferral use only limited small model confidence estimates because of the inaccessibility of the large model, although large model confidence is known to be important. We therefore propose a bi-directional approach to deferral that considers the confidence of small and large models in the cascade simultaneously through the use of a proxy for the large model. This requires a richer representation of model confidence to enable comparative calibration: we use an analysis of hidden states to improve post-invocation confidence of the small model, which in itself improves cascading results over prior approaches. We then combine this with a tiny proxy model to estimate pre-invocation confidence of the large model. We examine the proposed cascading system over challenging, multiple-choice datasets, finding improvements over standard cascading baselines reflected in reductions in deferrals to more costly models.

Figures

Figures reproduced from arXiv: 2504.19391 by the authors.

Figure 1
Figure 1. Bi-Directional deferral system diagram. Algorithm 1 Cascade algorithm Require: K ≥ 2 models M1, . . . , MK, backward confidence representation ConfB(·), forward confidence representation ConfF (·), deferral model MD(·, ·), thresholds τ1, . . . , τK−1 Require: An input instance x ∈ X 1: for j = 1, . . . , K do 2: D x j = 0 3: end for 4: for j = 1, . . . , K − 1 do 5: if MD(ConfB(Mj (x)), ConfF (Mj+1(x))) > τj then 6:… view at source ↗
Figure 2
Figure 2. Deferral curve for MMLU showing performance of each method over all delegation rates [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Deferral curve for ARC-Easy showing performance of each method over delegation rates. ating points for all datasets using the BI-DIR method. The BACK￾INT method alone provides statistically significant improvement at almost every operating point with full results in supplementary ma￾terial I[52]. To provide an interpretation of these AUC scores, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Confidence Histograms and Reliability Diagrams for baseline max probability confidence and internal representations method on the ARC￾Easy dataset. and is therefore useful in the deferral process, as evidenced in Ta￾ble 1. We note here that the large and diverse MMLU d…
Figure 7
Figure 7. Figure 7: Deferral curve for CSQA showing system performance over del￾egation rates. to predict a confidence around the center of the relevant range. While this is not a problem in terms of the ECE, it leads to less informa￾tive outputs for many samples and under-confidence in s…
Figure 5
Figure 5. Figure 5: Deferral curve for ARC-Challenge showing performance of each method over all delegation rates [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Deferral curve for BoolQ showing system performance over del￾egation rates. C Calibration Backwards internal representations provide strong calibration met￾rics as well as more useful reliability diagrams when compared to MAXPROB. The diagrams demonstrate the tendency …
Figure 8
Figure 8. Figure 8: Confidence Histograms and Reliability Diagrams for baseline max probability confidence and internal representations method on the ARC￾Challenge dataset. is linked to consuming liquids in its definition. Quantitative ques￾tions such as Example 3 are considered more diff…
Figure 9
Figure 9. Figure 9: Reliability diagrams for backwards internal confidence in LLaMA-2 13b on BoolQ [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Reliability diagrams for backwards internal confidence in LLaMA-2 13b on MMLU. there are some limitations as a result. Larger models may better eval￾uate the value of cascades in very difficult tasks, the performance of our small model on difficult tasks shows a signi…
Figure 12
Figure 12. Figure 12: Deferral curves of examined methods on ARC-Challenge con￾textualized with perfect deferral. introduces additional degrees of variability and is not the focus of this work [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 14
Figure 14. Figure 14: Deferral curves for cascading with LLaMA-2 7b as MS on ARC-Challenge [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 51 canonical work pages

  1. [1]

    Arase, P

    Y . Arase, P. Blunsom, M. Diab, J. Dodge, I. Gurevych, P. Liang, C. Raf- fel, A. Rücklé, R. Schwartz, N. A. Smith, E. Strubell, and Y . Zhang. Efficient NLP policy document. Technical report, ACL, 2021

    work page 2021

  2. [2]

    Azaria and T

    A. Azaria and T. Mitchell. The internal state of an LLM knows when it‘s lying. In Findings of ACL: EMNLP 2023 , pages 967–976. ACL, Dec. 2023

    work page 2023

  3. [3]

    Beigi, Y

    M. Beigi, Y . Shen, R. Yang, Z. Lin, Q. Wang, A. Mohan, J. He, M. Jin, C.-T. Lu, and L. Huang. InternalInspector i2: Robust confidence esti- mation in LLMs through internal states. In Findings of ACL: EMNLP 2024, pages 12847–12865. ACL, Nov. 2024

    work page 2024

  4. [4]

    Bhatt, J

    U. Bhatt, J. Antorán, Y . Zhang, Q. V . Liao, P. Sattigeri, R. Fogliato, G. Melançon, R. Krishnan, J. Stanley, O. Tickoo, L. Nachman, R. Chu- nara, M. Srikumar, A. Weller, and A. Xiang. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. In Proc. of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , AIES ’21, pag...

    work page 2021

  5. [5]

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper. Accelerating large language model decoding with speculative sampling. arXiv:2302.01318, 2023

    arXiv 2023

  6. [6]

    L. Chen, M. Zaharia, and J. Zou. Frugalgpt: How to use large language models while reducing cost and improving performance, 2023

    2023

  7. [7]

    T. Chen, J. Navratil, V . Iyengar, and K. Shanmugam. Confidence scor- ing using whitebox meta-models with linear classifier probes. In AIS- TATS 2019, volume 89 ofProc. of Mach. Learn. Res., pages 1467–1475. PMLR, 16–18 Apr 2019

    work page 2019

  8. [8]

    Chuang, Y

    Y .-S. Chuang, Y . Xie, H. Luo, Y . Kim, J. R. Glass, and P. He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth ICLR, 2024

    work page 2024

Show all 61 references
  1. [9]

    Clark, K

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proc. of the 2019 Conference of NAACL: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936. ACL,...

    2019

  2. [10]

    Clark, I

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018

    2018 arXiv

  3. [11]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schul- man. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021

    2021 arXiv

  4. [12]

    J. Geng, F. Cai, Y . Wang, H. Koeppl, P. Nakov, and I. Gurevych. A sur- vey of confidence estimation and calibration in large language models. In Proc. the 2024 Conference of NAACL: Human Language Technolo- gies (Volume 1: Long Papers), pages 6577–6595. ACL, June 2024

    2024

  5. [13]

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th ICML, volume 70 of Proc. Mach. Learn. Res., pages 1321–1330. PMLR, 06–11 Aug 2017

    2017

  6. [14]

    Gupta, H

    N. Gupta, H. Narasimhan, W. Jitkrittum, A. S. Rawat, A. K. Menon, and S. Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth ICLR, 2024

    2024

  7. [15]

    L. He, P. Chen, E. Nie, Y . Li, and J. R. Brennan. Decoding probing: Revealing internal linguistic structures in neural language models using minimal pairs. In LREC/COLING, pages 4488–4497, 2024

    2024

  8. [16]

    P. He, J. Gao, and W. Chen. Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing, 2023

    2023

  9. [17]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021

    2021

  10. [18]

    Huang, J

    Y . Huang, J. Song, Z. Wang, S. Zhao, H. Chen, F. Juefei-Xu, and L. Ma. Look before you leap : An exploratory study of uncertainty analysis for large language models. IEEE Trans. on Software Engineering, PP:1–18, 01 2025

    2025

  11. [19]

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . Bang, D. Chen, W. Dai, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55:1 – 38, 2022

    2022

  12. [20]

    M. Jin, Q. Yu, J. Huang, Q. Zeng, Z. Wang, W. Hua, H. Zhao, K. Mei, Y . Meng, K. Ding, et al. Exploring concept depth: How large language models acquire knowledge at different layers?arXiv:2404.07066, 2024

    2024 arXiv

  13. [21]

    Jitkrittum, N

    W. Jitkrittum, N. Gupta, A. K. Menon, H. Narasimhan, A. S. Rawat, and S. Kumar. When does confidence-based cascade deferral suffice? In NeurIPS 2023, 2023

    2023

  14. [22]

    Kadavath, T

    S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Dodds, N. DasSarma, et al. Language models (mostly) know what they know, 07 2022

    2022

  15. [23]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020

    2020

  16. [24]

    Lebovitz, L

    L. Lebovitz, L. Cavigelli, M. Magno, and L. K. Muller. Efficient infer- ence with model cascades. Trans. Mach. Learn. Res., 2023, 2023

    2023

  17. [25]

    Leviathan, M

    Y . Leviathan, M. Kalman, and Y . Matias. Fast inference from transform- ers via speculative decoding. In ICML, pages 19274–19286. PMLR, 2023

    2023

  18. [26]

    Madras, T

    D. Madras, T. Pitassi, and R. Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. InAdv. NeurIPS, volume 31. Curran Associates, Inc., 2018

    2018

  19. [27]

    A. Mao, C. Mohri, M. Mohri, and Y . Zhong. Two-stage learning to defer with multiple experts. In Thirty-seventh NeurIPS, 2023

    2023

  20. [28]

    R. Miao, Y . Yan, X. Yao, and T. Yang. An efficient inference framework for early-exit large language models, 2024

    2024

  21. [29]

    S. J. Mielke, A. Szlam, E. Dinan, and Y .-L. Boureau. Reducing conver- sational agents’ overconfidence through linguistic calibration. Trans. ACL, 10:857–872, 2022

    2022

  22. [30]

    Narasimhan, W

    H. Narasimhan, W. Jitkrittum, A. K. Menon, A. Rawat, and S. Kumar. Post-hoc estimators for learning to defer to an expert. In Adv. NeurIPS, volume 35, pages 29292–29304. Curran Associates, Inc., 2022

    2022

  23. [31]

    L. Nie, Z. Ding, E. Hu, C. Jermaine, and S. Chaudhuri. Online cascade learning for efficient inference over streams. ArXiv, abs/2402.04513, 2024

    2024 arXiv

  24. [32]

    M. Omar, R. Agbareia, B. S. Glicksberg, G. N. Nadkarni, and E. Klang. Benchmarking the confidence of large language models in clinical ques- tions. medRxiv, 2024

    2024

  25. [33]

    Papernot and P

    N. Papernot and P. D. McDaniel. Deep k-nearest neighbors: To- wards confident, interpretable and robust deep learning. CoRR, abs/1803.04765, 2018

    2018 arXiv

  26. [34]

    E. Park, D. Kim, S. Kim, Y .-D. Kim, G. Kim, S. Yoon, and S. Yoo. Big/little deep neural network for ultra low power inference.2015 Inter- national Conference on Hardware/Software Codesign and System Syn- thesis (CODES+ISSS), pages 124–132, 2015

    2015

  27. [35]

    Pawitan and C

    Y . Pawitan and C. Holmes. Confidence in the reasoning of large lan- guage models, 2024

    2024

  28. [36]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, et al. Scikit-learn: Machine learning in Python. JMLR, 12:2825–2830, 2011

    2011

  29. [37]

    Rahmath P, V

    H. Rahmath P, V . Srivastava, K. Chaurasia, R. G. Pacheco, and R. S. Couto. Early-exit deep neural network - a comprehensive survey. ACM Comput. Surv., 57(3), Nov. 2024. ISSN 0360-0300

    2024

  30. [38]

    S. Ren, B. Tomlinson, R. W. Black, and A. W. Torrance. Reconciling the contrasting narratives on the environmental impact of large language models. Scientific Reports, 14(26310), 2024

    2024

  31. [39]

    M. Shen, Y . Bu, P. Sattigeri, S. Ghosh, S. Das, and G. Wornell. Post-hoc uncertainty learning using a dirichlet meta-model. In Proc. AAAI-23 / IAAI-23 / EAAI-23. AAAI Press, 2023. ISBN 978-1-57735-880-0

    2023

  32. [40]

    Singh, N

    A. Singh, N. P. Patel, A. Ehtesham, S. Kumar, and T. T. Khoei. A survey of sustainability in large language models: Applications, eco- nomics, and challenges, 2024

    2024

  33. [41]

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models. In The Twelfth ICLR, 2024

    2024

  34. [42]

    Talmor, J

    A. Talmor, J. Herzig, N. Lourie, and J. Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proc. of the 2019 Conference of NAACL: Human Language Technolo- gies, Volume 1, pages 4149–4158. ACL, June 2019

    2019

  35. [43]

    Teerapittayanon, B

    S. Teerapittayanon, B. McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks, 2017

    2017

  36. [44]

    Tenney, D

    I. Tenney, D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In Proc. of the 57th Annual Meeting of ACL, pages 4593–4601. ACL, July 2019

    2019

  37. [45]

    Tighidet, J

    Z. Tighidet, J. Mei, B. Piwowarski, and P. Gallinari. Probing language models on their knowledge source. In Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 604–614. ACL, Nov. 2024

    2024

  38. [46]

    Touvron, L

    H. Touvron, L. Martin, K. R. Stone, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288, 2023

    2023 arXiv

  39. [47]

    Ulmer, M

    D. Ulmer, M. Gubri, H. Lee, S. Yun, and S. Oh. Calibrating large lan- guage models using their generations only. In Proc. of the 62nd Annual Meeting of ACL (Volume 1: Long Papers) , pages 15440–15459. ACL, Aug. 2024

    2024

  40. [48]

    Varshney and C

    N. Varshney and C. Baral. Model cascading: Towards jointly improving efficiency and accuracy of NLP systems. In Proc. of the 2022 Confer- ence EMNLP, pages 11007–11021. ACL, Dec. 2022

    2022

  41. [49]

    Viola and M

    P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proc. of the 2001 IEEE CVPR , volume 1, pages I–I, 2001

    2001

  42. [50]

    C. Wang, S. Augenstein, K. Rush, W. Jitkrittum, H. Narasimhan, A. S. Rawat, A. K. Menon, and A. Go. Cascade-aware training of language models, 2024

    2024

  43. [51]

    X. Wang, Y . Luo, D. Crankshaw, A. Tumanov, and J. E. Gonzalez. Idk cascades: Fast deep learning by learning not to overthink. InConference on Uncertainty in Artificial Intelligence, 2017

    2017

  44. [52]

    Warren and M

    D. Warren and M. Dras. Bi-directional model cascading with proxy confidence - this paper incl. supplementary material, 2025. URL https: //arxiv.org/abs/2504.19391

    2025 arXiv

  45. [53]

    Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He. Zeroquant: Efficient and affordable post-training quantization for large- scale transformers. In NeurIPS, volume 35, pages 27168–27183. Curran Associates, Inc., 2022

    2022

  46. [54]

    M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao. Large language model cascades with mixture of thought representations for cost-efficient rea- soning. In The Twelfth ICLR, 2024

    2024

  47. [55]

    Zadrozny and C

    B. Zadrozny and C. P. Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, 2001

    2001

  48. [56]

    H. Zhao, H. Chen, F. Yang, N. Liu, H. Deng, H. Cai, S. Wang, D. Yin, and M. Du. Explainability for large language models: A survey. ACM Trans. Intell. Syst. Technol., 15(2), Feb. 2024. ISSN 2157-6904

    2024

  49. [57]

    Oracle Deferral Curve

    C. Zhu, B. Xu, Q. Wang, Y . Zhang, and Z. Mao. On the calibration of large language models and alignment. In Findings of ACL: EMNLP 2023, pages 9778–9795. ACL, Dec. 2023. 7 Supplementary Materials A Deferral AUC Scores Table 5 presents the AUC scores for deferral rates between...

    2023

  50. [58]

    Question: What is the main source of heat for Earth’s surface? Choices: (A): fire (B): lightning (C): the Sun (D): the ocean C High Yes

  51. [59]

    Question: When people exercise, they often feel thirsty and begin to sweat. It is important for people to feel thirsty when exercising because it makes them realize that they should Choices: (A): take a break (B): consume liquids (C): slow their breathing (D): stop to eat some...

  52. [60]

    What is the atomic mass of potassium? Choices: (A): 19 (B): 20 (C): 38 (D): 39 D Low No

    Question: A potassium (K) atom has 20 neutrons, 19 protons and 19 electrons. What is the atomic mass of potassium? Choices: (A): 19 (B): 20 (C): 38 (D): 39 D Low No

  53. [61]

    Reliability diagrams for backwards internal confidence in LLaMA-2 13b on CSQA

    Question: Which situation is an example of an inherited trait? Choices: (A): lions preying on zebras (B): monkeys using twigs to get food (C): birds following migratory patterns (D): bears opening coolers at campsites A Low No Figure 11. Reliability diagrams for backwards inte...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.