Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees

T0 review · 3 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Any LLM uncertainty score can be turned into a selective answering gate that keeps the error rate among accepted answers below a user-chosen risk level, with a finite-sample high-probability guarantee.

desk verdict Clean UCB calibration for LLM abstention with a real theory–code gap on multiplicity; still worth a referee. read the letter →

arxiv 2607.04430 v1 pith:OBRRMJSA submitted 2026-07-05 cs.CL

Sijin Dong , Hiroyuki Shinnou This is my paper
classification cs.CL
keywords largelanguagemodelsquestionansweringselectiveuncertaintyquantificationupperconfidenceboundriskcontrolabstention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models often produce fluent answers without a trustworthy signal of when those answers are wrong. This paper shows how to take an arbitrary uncertainty score from such a model and, using a held-out calibration set, convert it into a simple answer-or-abstain rule. For every candidate uncertainty threshold the method estimates the error rate among the answers that would be accepted, builds a high-probability upper confidence bound on that rate, and keeps the largest threshold whose bound stays under a risk level the user sets. Under exchangeability of calibration and deployment data, the chosen threshold—if one exists—controls the acceptance-conditioned error rate with probability at least 1−δ. Experiments on closed-ended and open-ended question answering across several models show that the procedure meets the risk targets while still answering a large fraction of queries. A reader who cares about reliability-sensitive deployment would value a statistical promise attached to the answers that are actually returned, rather than average accuracy alone.

What carries the argument

CIC (confidence-interval-based calibration): for each candidate threshold it forms the accepted calibration subset, estimates its error rate, attaches a one-sided upper confidence bound (Hoeffding-style or Clopper–Pearson), and returns the largest threshold whose bound is at most α.

What would settle it

On repeated exchangeable calibration–test splits, when CIC returns a non-null threshold for a stated α and δ, check whether the test-set fraction of wrong accepted answers exceeds α in more than roughly a δ fraction of trials; systematic excess would falsify the central guarantee.

Watch

Extended reading notes

Core claim

Under exchangeability, CIC selects the largest uncertainty threshold whose upper confidence bound on the acceptance-conditioned error rate does not exceed a user-specified risk level α. With probability at least 1−δ the selected non-null threshold controls the error rate among accepted answers at level α. Uncertainty scores are treated only as ranking signals; they need not be calibrated themselves.

Load-bearing premise

Calibration examples and future live queries must be exchangeable—same joint distribution of uncertainty scores and correctness—so the bound computed on the calibration set still applies at deployment.

Editorial extensions

If this is right

  • A finite-sample risk certificate can be attached to selective answering without retraining the LLM or redesigning the uncertainty estimator.
  • When the requested risk level is unattainable for the current model and score, the method reports that fact instead of returning an unsafe threshold.
  • Better base models or more discriminative uncertainty scores yield higher answering rates at the same risk target.
  • The same calibration layer applies to both closed-ended and open-ended QA once an application-specific alignment criterion is fixed.
  • In the one-answer-per-query setting the controlled quantity is the marginal false discovery rate of the selective policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same UCB-threshold construction could wrap non-LLM predictors whenever only a ranking score and binary outcome labels are available.
  • Any distribution shift between calibration and live traffic voids the finite-sample guarantee, so production use would need recalibration or shift detection.
  • Open-ended certificates inherit whatever errors the alignment rule (e.g., a fixed similarity threshold) makes relative to human judgment.
  • Multi-answer, tool-using, or retrieval-augmented pipelines would need a redefined acceptance-conditioned risk before the same certificate applies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CIC, a confidence-interval calibration layer that turns arbitrary LLM uncertainty scores into selective-answering rules with finite-sample control of the acceptance-conditioned error rate R(t)=P(E=1|U≤t). On a held-out calibration set, each response is labeled by an application-specific alignment criterion A and paired with an uncertainty score; for candidate thresholds the method forms Hoeffding or Clopper–Pearson upper confidence bounds on the empirical accepted-error rate and selects the largest threshold whose UCB is ≤α. Under exchangeability, Theorem 3.3 claims that with probability ≥1−δ the selected non-null threshold satisfies R(ˆt)≤α. Experiments on CommonsenseQA and TriviaQA with seven instruction-tuned LLMs and semantic entropy report empirical FDR at or below target α when a feasible threshold exists, together with answering power that increases with α.

Significance. If the finite-sample guarantee attaches to the procedure that is actually deployed, CIC would be a useful, model-agnostic statistical layer for reliability-sensitive LLM QA: it does not require the uncertainty score itself to be calibrated, only that it rank responses, and it yields an explicit risk–coverage trade-off under a user-chosen α. Strengths include a clean problem formulation (acceptance-conditioned error / mFDR rather than set coverage), two standard binomial UCBs with correct fixed-threshold validity (Theorem 3.2), and a reasonably broad empirical sweep (seven models, closed- and open-ended QA, 100 random splits, sensitivity to calibration size). The contribution is incremental relative to concurrent selective-prediction / risk-control work (e.g., COIN, LEC) but would still be of practical interest if the theory–implementation gap is closed and the experimental claims are tightened.

major comments (3)
  1. [§3.4–3.5, Theorem 3.3, Algorithm 1] Theorem 3.3 (Eqs. 13–20) proves a simultaneous guarantee only for a pre-specified finite grid T with Bonferroni levels δ/K. Section 3.5 and Algorithm 1 instead scan the data-dependent ordered scores u(1)…u(N) and use fixed-sequence testing with level δ (not δ/K), justified only by the informal claim that selection-conditioned risk is “near-monotonic.” The manuscript never proves that R(t) is non-decreasing in t, nor that the sequence of UCBs is monotone, so the finite-sample statement claimed for the deployed threshold is not the one proved. Either restore Bonferroni on a fixed grid, or state and prove the monotonicity conditions under which fixed-sequence testing is valid for this selection rule, and align Algorithm 1 with that theorem.
  2. [Abstract; §4.1; Tables 1–4] The abstract and contribution list claim evaluation “across … multiple uncertainty estimators,” but §4.1 and Tables 1–4 use only semantic entropy (10 samples). Either add at least one additional estimator (e.g., SAR or WSE, both cited) under the same CIC pipeline, or revise the abstract/claims to match the actual experimental design. As written, the multi-estimator claim is not supported by the reported results.
  3. [§4.2; Related Work §2] Empirical risk control is shown only for CIC’s two UCB variants against the base error rate. There is no head-to-head comparison with the closest concurrent selective-answering methods with risk guarantees (COIN [24], LEC [25]) or with a simple empirical-threshold baseline that ignores the UCB. Without those baselines it is hard to judge whether the UCB layer improves power or feasibility relative to existing calibrated abstention rules on the same splits and alignment labels.
minor comments (5)
  1. [Table 4] Table 4 (CommonsenseQA power) omits Vicuna-7B-v1.5, which appears in Table 1; either add the missing row or note the omission explicitly.
  2. [Algorithm 1] Algorithm 1 line 3 writes u_i ← U(x_i; G) while the text uses M for the uncertainty measure; unify notation (M vs G).
  3. [Algorithm 1] Typo in Algorithm 1 line 4: “aliged” → “aligned.”
  4. [§4.2, Fig. 2] Figure 2 caption refers to a “target alignment proportion of 0.90” (i.e., α=0.10) without stating α in the main text of that subsection; make the risk level explicit.
  5. [§4.1 Alignment Evaluation] The TriviaQA alignment rule (sentence similarity ≥0.6) is a free modeling choice that defines E; a short sensitivity check over the threshold (or an alternative exact-match / LLM-judge label) would strengthen the open-ended results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: risk control is standard concentration on held-out binary labels; uncertainty scores are black-box ranking signals, not fitted inputs renamed as predictions.

full rationale

The paper's central claim (Theorem 3.3) is that a threshold chosen by upper-bounding the empirical acceptance-conditioned error rate R̂(t) with Hoeffding or Clopper–Pearson UCBs, then taking the largest t with B(t) ≤ α, controls R(t̂) ≤ α with probability ≥ 1−δ under exchangeability. That guarantee is obtained by applying classical binomial concentration (Theorem 3.2) and a union bound over a pre-specified finite grid; R(t) is defined as P(E=1 | U≤t) from application-specific alignment labels, not from any parameter fitted to the same quantity being predicted. Uncertainty estimators (semantic entropy, etc.) are treated explicitly as black-box ranking signals and are never required to be calibrated. There is no self-definitional loop, no fitted parameter re-sold as a prediction, no load-bearing uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via self-citation. Related citations (COIN, LEC, conformal methods) are external prior art used for positioning, not as unverified premises that force the result. The known theory–implementation gap (Algorithm 1 uses fixed-sequence δ and empirical score order rather than Bonferroni δ/K on a fixed grid) is a correctness/coverage issue, not circularity: it does not make the claimed guarantee equivalent to its inputs by construction. Score 0 is therefore appropriate.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central guarantee rests on classical concentration plus one domain assumption (exchangeability) and an application-defined binary alignment oracle. User knobs α and δ, the SE sample count, the open-ended similarity threshold, and the calibration split are free choices that affect feasibility and power but are not fitted to invent the risk claim itself. No new physical or latent entities are postulated.

free parameters (6)
  • risk level α
    User-specified target for acceptance-conditioned error; directly determines which thresholds are certified.
  • significance level δ
    Controls the failure probability of the simultaneous UCB guarantee; chosen by the user.
  • semantic-entropy sample count
    Fixed at 10 samples per question; affects the quality of the ranking signal U.
  • sentence-similarity threshold for TriviaQA
    Fixed at 0.6 to produce binary alignment labels; changes the definition of E and therefore the calibrated threshold.
  • calibration–test split ratio
    Default 0.5; sensitivity study varies it; affects width of UCBs and power.
  • candidate threshold grid T
    Pre-specified finite grid required by the formal theorem; in practice the algorithm scans empirical scores.
assumptions (3)
  • domain assumption Calibration tuples {(Ui, Ei)} and a future deployment pair are exchangeable (Assumption 3.1).
    Load-bearing for transferring the UCB guarantee from calibration to deployment; stated explicitly in §3.4.
  • standard math Conditional on the number of accepted calibration points n(t)=n>0, the number of false discoveries is Binomial(n, R(t)).
    Standard multinomial conditioning under i.i.d./exchangeable sampling; used to justify both Clopper–Pearson and Hoeffding UCBs (Theorem 3.2).
  • domain assumption The application-specific alignment criterion A produces the binary error labels that define the risk of interest.
    For open-ended QA this is operationalized by sentence similarity ≥0.6; if A mismatches user-perceived correctness the controlled quantity is not the operational risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees." pith.science (2026). https://pith.science/paper/OBRRMJSA

@misc{pith2026260704430,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBRRMJSA}},
  note         = {Machine review of arXiv:2607.04430}
}
abstract

Large language models (LLMs) are increasingly deployed in question answering (QA) systems, yet they may generate hallucinated or misaligned responses without reliable confidence estimates. Uncertainty quantification (UQ) offers a natural basis for selective answering, where a system answers only when its prediction is deemed reliable and abstains otherwise. However, existing uncertainty scores for LLMs are often heuristic: a threshold chosen on such scores does not, by itself, provide statistical guarantees on the error rate among accepted answers. We propose CIC, a confidence-interval-based calibration framework that converts arbitrary uncertainty scores into risk-controlled selective answering rules. Given a held-out calibration set, CIC evaluates each generated response using an application-specific alignment criterion and associates it with an uncertainty score and a binary error label. For each candidate uncertainty threshold, CIC estimates the acceptance-conditioned error rate and constructs a high-probability upper confidence bound using either Hoeffding-style or Clopper-Pearson confidence intervals. It then selects the largest threshold whose upper bound is below a user-specified risk level $\alpha$, thereby maximizing the answering rate subject to a finite-sample reliability constraint. Under exchangeability, CIC guarantees with probability at least $1-\delta$ that the selected threshold, if non-null, controls the error rate among accepted answers at level $\alpha$. We evaluate CIC on both closed-ended and open-ended QA benchmarks across seven LLMs and multiple uncertainty estimators. Experimental results show that CIC consistently achieves valid risk control while retaining strong answering efficiency, providing a practical and statistically grounded mechanism for deploying LLMs in reliability-sensitive QA workflows.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 7 linked inside Pith

  1. [24]

    Proceedings of the AAAI Conference on Artificial Intelligence40(40), 33764–33772 (2026)

    Wang, Z., Duan, J., Wang, Q., Zhu, X., Chen, T., Shi, X., Xu, K.: Coin: Uncertainty-guarding selective question answering for foundation models with provable risk guarantees. Proceedings of the AAAI Conference on Artificial Intelligence40(40), 33764–33772 (2026)

    2026

  2. [25]

    arXiv preprint arXiv:2512.01556 (2025)

    Wang, Z., Chen, T., Zhang, Y., Shen, H.T., Shi, X., Xu, K., et al.: Lec: Lin- ear expectation constraints for false-discovery control in selective prediction and routing systems. arXiv preprint arXiv:2512.01556 (2025)

    arXiv 2025

  3. [1]

    IEEE Signal Processing Magazine42(6), 51–63 (2026)

    Wang, S., Xu, T., Li, H., Zhang, C., Liang, J., Tang, J., Yu, P.S., Wen, Q.: Large language models for education: A survey and outlook. IEEE Signal Processing Magazine42(6), 51–63 (2026)

    2026

  4. [2]

    ACM Transactions on Software Engineering and Methodology35(2), 1–72 (2026)

    Jiang, J., Wang, F., Shen, J., Kim, S., Kim, S.: A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology35(2), 1–72 (2026)

    2026

  5. [3]

    Scientific Reports16(1), 5357 (2026)

    Eldem, A., Eldem, H.: The development and evaluation of agricultural question- answering systems based on large language models. Scientific Reports16(1), 5357 (2026)

    2026

  6. [4]

    Automotive Innovation, 1–21 (2026)

    Song, Z., Xu, N., Ding, H.: Decision-making framework for autonomous vehicles in complex scenarios using large language models. Automotive Innovation, 1–21 (2026)

    2026

  7. [5]

    Bi, J., Yan, D., Wang, Y., Huang, W., Chen, H., Wan, G., Ye, M., Xiao, X., Schuetze, H., Tresp, V., Ma, Y.: CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process (2025)

    2025

  8. [6]

    Huang, X., Rishabh, Franke, G., Yang, Z., Bai, J., Bai, W., Bi, J., Ding, Z., Duan, Y., Fan, C., Fan, W., Gao, X., Guo, R., He, Y., He, Z., Hu, X., Johnson, N., Li, B., Lin, F., Lin, S., Liu, T., Ma, Y., Shen, H., Sun, H., Wang, B., Wang, F., Wang, H., Wang, H., Wang, Y., Wang, Y., Wang, Z., Wang, Z., Wu, Y., Xiao, Z., Xie, C., Yang, F., Yang, J., Ye, Q.,...

    2025

Show all 44 references
  1. [7]

    Wan, G., Fu, L., Liu, H., Jin, Y., Leong, H.Y., Jiang, E.H., Geng, H., Bi, J., Ma, Y., Tang, X., Prakash, B.A., Sun, Y., Wang, W.: Beyond Magic Words: Sharpness- Aware Prompt Evolving for Robust Large Language Models with TARE (2025)

    2025

  2. [8]

    Tian, Y., Chen, S., Xu, Z., Wang, Y., Bi, J., Han, P., Wang, W.: Reinforcement Mid-Training (2025) 18

    2025

  3. [9]

    arXiv preprint arXiv:2505.16916 (2025)

    Rong, X., Huang, W., Liang, J., Bi, J., Xiao, X., Li, Y., Du, B., Ye, M.: Back- door cleaning without external guidance in mllm fine-tuning. arXiv preprint arXiv:2505.16916 (2025)

    2025 arXiv

  4. [10]

    arXiv preprint arXiv:2506.14766 (2025)

    Wang, Y., Bi, J., Ma, Y., Pirk, S.: Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm. arXiv preprint arXiv:2506.14766 (2025)

    2025

  5. [11]

    In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T

    Bi, J., Wang, Y., Chen, H., Xiao, X., Hecker, A., Tresp, V., Ma, Y.: LLaVA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T. (eds.) Proceedings of the 63rd Annua...

    2025

  6. [12]

    Bi, J., Wang, Y., Yan, D., Aniri, Huang, W., Jin, Z., Ma, X., Hecker, A., Ye, M., Xiao, X., Schuetze, H., Tresp, V., Ma, Y.: PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection (2025)

    2025

  7. [13]

    Jiang, K., Jiang, N., Du, Y., Ren, Y., Li, Y., Gao, Y., Bi, J., Ma, Y., Liu, Q., Wang, X., Jia, Y., Jiang, H., Hu, Y., Li, B., Liu, L.: MINED: Probing and Updat- ing with Multimodal Time-Sensitive Knowledge for Large Multimodal Models (2025)

    2025

  8. [14]

    Jiang, K., Jiang, H., Jiang, N., Gao, Z., Bi, J., Ren, Y., Li, B., Du, Y., Liu, L., Li, Q.: KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Augmentations and Constraints (2025)

    2025

  9. [15]

    arXiv preprint arXiv:2311.12919 (2023)

    Zhang, G., Bi, J., Gu, J., Chen, Y., Tresp, V.: Spot! revisiting video-language models for event understanding. arXiv preprint arXiv:2311.12919 (2023)

    2023 arXiv

  10. [16]

    Peng, T., Du, Y., Ji, P., Dong, S., Jiang, K., Ma, M., Tian, Y., Bi, J., Li, Q., Du, W., Xiao, F., Cui, L.: Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models (2025)

    2025

  11. [17]

    Li, Z., Wu, X., Wang, Z., Li, J., Tian, Y., Bi, J., Ma, Y., Ye, Y., Zhang, C.: Graph is a Substrate Across Data Modalities (2026)

    2026

  12. [18]

    Artificial intelligence review56, 1513–1589 (2023)

    Gawlikowski, J., Tassi, C.R.N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R.,et al.: A survey of uncertainty in deep neural networks. Artificial intelligence review56, 1513–1589 (2023)

    2023

  13. [19]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., Xu, K.: Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024

  14. [20]

    In: The Eleventh International Conference on Learning Representations (2023)

    Kuhn, L., Gal, Y., Farquhar, S.: Semantic uncertainty: Linguistic invariances 19 for uncertainty estimation in natural language generation. In: The Eleventh International Conference on Learning Representations (2023)

    2023

  15. [21]

    Engineering Applications of Artificial Intelligence139, 109553 (2025)

    Wang, Z., Duan, J., Yuan, C., Chen, Q., Chen, T., Zhang, Y., Wang, R., Shi, X., Xu, K.: Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering applications and beyond. Engineering Applications of Artificial Intelligence139, 109553 (2025)

    2025

  16. [22]

    In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp

    Wang, Z., Duan, J., Cheng, L., Zhang, Y., Wang, Q., Shi, X., Xu, K., Shen, H.T., Zhu, X.: Conu: Conformal uncertainty in large language models with cor- rectness coverage guarantees. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6886–6898 (2024)

    2024

  17. [23]

    Advances in Neural Information Processing Systems37, 73884–73919 (2024)

    Gui, Y., Jin, Y., Ren, Z.: Conformal alignment: Knowing when to trust foundation models with guarantees. Advances in Neural Information Processing Systems37, 73884–73919 (2024)

    2024

  18. [26]

    In: International Conference on Learning Representations (2021)

    Angelopoulos, A.N., Bates, S., Jordan, M., Malik, J.: Uncertainty sets for image classifiers using conformal prediction. In: International Conference on Learning Representations (2021)

    2021

  19. [27]

    Foundations and Trends in Machine Learning16(4), 494–591 (2023)

    Angelopoulos, A.N., Bates, S.: Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning16(4), 494–591 (2023)

    2023

  20. [28]

    arXiv preprint arXiv:2411.11824 (2024)

    Angelopoulos, A.N., Barber, R.F., Bates, S.: Theoretical foundations of conformal prediction. arXiv preprint arXiv:2411.11824 (2024)

    2024 arXiv

  21. [29]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Wang, Z., Wang, Q., Zhang, Y., Chen, T., Zhu, X., Shi, X., Xu, K.: Sconu: Selective conformal uncertainty in large language models. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19052–19075 (2025)

    2025

  22. [30]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Wang, Q., Geng, T., Wang, Z., Wang, T., Fu, B., Zheng, F.: Sample then identify: A general framework for risk control and assessment in multimodal large language models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  23. [31]

    arXiv preprint arXiv:2603.22966 (2026)

    Li, Y., Hu, A., Ye, Y., Tong, S., Wang, Z., Fu, B.: Set-valued prediction for 20 large language models with feasibility-aware coverage guarantees. arXiv preprint arXiv:2603.22966 (2026)

    2026

  24. [32]

    Biometrika26(4), 404–413 (1934)

    Clopper, C.J., Pearson, E.S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika26(4), 404–413 (1934)

    1934

  25. [33]

    arXiv preprint arXiv:2510.17897 (2025)

    Tan, B., Wang, Z., Duan, J., Xu, K., Shen, H.T., Shi, X., Shen, F.: Confor- mal lesion segmentation for 3d medical images. arXiv preprint arXiv:2510.17897 (2025)

    2025

  26. [34]

    Transactions of the Association for Computational Linguistics12, 1497–1516 (2024)

    Campos, M., Farinhas, A., Zerva, C., Figueiredo, M.A., Martins, A.F.: Confor- mal prediction for natural language processing: A survey. Transactions of the Association for Computational Linguistics12, 1497–1516 (2024)

    2024

  27. [35]

    In: The Twelfth International Conference on Learning Representations (2024)

    Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J.H., Jaakkola, T.S., Barzilay, R.: Conformal language modeling. In: The Twelfth International Conference on Learning Representations (2024)

    2024

  28. [36]

    In: Proceedings of the 41st International Conference on Machine Learning, pp

    Mohri, C., Hashimoto, T.: Language models with conformal factuality guarantees. In: Proceedings of the 41st International Conference on Machine Learning, pp. 36029–36047 (2024)

    2024

  29. [37]

    arXiv preprint arXiv:2605.20084 (2026)

    Jia, Z., Ye, Y., Jia, S., Qian, Y., Wang, H., Chen, B., Tang, D., Yu, J., Wang, Z.: Balancerag: Joint risk calibration for cascaded retrieval-augmented generation. arXiv preprint arXiv:2605.20084 (2026)

    2026 arXiv

  30. [38]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Jung, J., Brahman, F., Choi, Y.: Trust or escalate: LLM judges with provable guarantees for human agreement. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  31. [39]

    arXiv preprint arXiv:2309.16609 (2023)

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    2023 arXiv

  32. [40]

    In: International Conference on Learning Representations, vol

    Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., Liu, Y.: Openchat: Advanc- ing open-source language models with mixed-quality data. In: International Conference on Learning Representations, vol. 2024, pp. 57021–57040 (2024)

    2024

  33. [41]

    Advances in neural information processing systems36, 46595–46623 (2023)

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.,et al.: Judging llm-as-a-judge with mt-bench and chat- bot arena. Advances in neural information processing systems36, 46595–46623 (2023)

    2023

  34. [42]

    arXiv preprint arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi` ere, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  35. [43]

    In: Burstein, J., Doran, C., Solorio, T

    Talmor, A., Herzig, J., Lourie, N., Berant, J.: CommonsenseQA: A question 21 answering challenge targeting commonsense knowledge. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computati...

    2019

  36. [44]

    In: Barzilay, R., Kan, M.-Y

    Joshi, M., Choi, E., Weld, D., Zettlemoyer, L.: TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In: Barzilay, R., Kan, M.-Y. (eds.) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2017

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.