Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control

Yuanchang Ye

REVIEW 4 major objections 4 minor 18 references

Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A p-value reformulation of conformal prediction gives black-box LLMs answer sets whose chance of missing the true option is bounded by a user-set α, under exchangeability.

desk verdict Routine split-conformal application whose central equivalence is algebraically wrong; experiments are consistent but do not rescue the theory. read the letter →

arxiv 2508.10022 v1 pith:HV4HYXRB submitted 2025-08-07 cs.CL cs.AI

Yuanchang Ye This is my paper

classification cs.CLcs.AI

keywords conformalpredictionsignificancetestingp-valuesmultiple-choicequestionansweringlargelanguagemodelshallucinationdetectionuncertaintyquantificationaveragesetsize

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a significance-testing version of conformal prediction can control, at any user-specified level α, the probability that the correct answer is left out of a prediction set in multiple-choice question answering. The method treats repeated LLM outputs as votes, computes the empirical frequency of each option, and converts the standard conformal score into a p-value; including or excluding each option is then a hypothesis-testing decision. If correct, this gives LLM users a model-agnostic, distribution-free way to request bounded error rates, which matters for hallucination-prone high-stakes question answering. Experiments on MMLU and MMLU-Pro across several off-the-shelf LLMs show empirical miscoverage below α and monotone shrinkage of prediction sets as α rises.

What carries the argument

The load-bearing object is the conformal p-value. With calibration nonconformity scores $s_1,\dots,s_n$ and a test score $S(x_{\text{test}}, y)$ for a candidate option, $p(y) = \frac{\sum_{i=1}^n \mathbf{1}\{s_i > S(x_{\text{test}}, y)\} + 1}{n+1}$; the rule “include $y$ when $p(y) > \alpha$ and exclude $y$ when $p(y) \le \alpha$” is algebraically equivalent to the conformal quantile construction. The nonconformity score is $S(x,y) = 1 - \hat{f}(y|x)$, where $\hat{f}$ is the observed frequency of option $y$ across repeated single-token generations, so only black-box LLM outputs are needed.

What would settle it

Run a calibration-test split where calibration is confined to one set of subjects (e.g., STEM) and evaluation to another (e.g., humanities) with the same p-value rule; if the empirical miscoverage rate exceeds α beyond bootstrap sampling error, the exchangeability premise is violated and the claimed guarantee does not hold. A second check: fix calibration data and change only the decoding temperature or prompt template at test time, then observe whether coverage stays above $1-\alpha$.

Watch

Extended reading notes

Core claim

The central claim is that the prediction set built by comparing a p-value to α is the same prediction set built by the conformal quantile rule, so it inherits the conformal guarantee $P(y^*_{\text{test}} \in C_\alpha(x_{\text{test}})) \ge 1-\alpha$ under exchangeability. For each candidate option $y$, the paper defines $p(y) = \frac{\sum_{i=1}^n \mathbf{1}\{s_i > S(x_{\text{test}}, y)\} + 1}{n+1}$, using nonconformity scores $s_i$ from calibration; $y$ is excluded when $p(y) \le \alpha$. On the true label, this p-value is the rank of the test score among calibration scores, which is the usual conformal p-value. The paper therefore claims that significance-testing-based conformal prediction i

Load-bearing premise

The whole coverage guarantee rests on the nonconformity scores of calibration and test examples being exchangeable; the paper assumes this by splitting within each subject and treating its 20 sampled responses as i.i.d., but does not test it, and if test questions come from a different subject or distribution the claimed bound $P(y^*_{\text{test}} \in C_\alpha(x_{\text{test}})) \ge 1-\alpha$ can fail.

Editorial extensions

If this is right

A practitioner can set α in advance and receive a prediction set whose expected chance of missing the correct option is at most α, with no access to model internals.
Risk and set size are coupled: increasing α monotonically shrinks average prediction set size, so APSS doubles as an interpretable uncertainty diagnostic.
The same split-conformal calibration procedure works across model families and dataset difficulties; the paper reports below-target miscoverage on both MMLU and MMLU-Pro at a 1:1 calibration-test split.
Because the p-value rule is exactly the conformal rule, significance-testing language (null hypothesis, Type I error) and conformal language (coverage, miscoverage) describe the same single knob.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The guarantee is marginal over the calibration-test draw; it does not promise per-subject or per-question coverage, so subject-level error boxes in the paper should be read as descriptive diagnostics, not guaranteed bounds.
The method is likely sensitive to the exchangeability assumption: calibrating on one subject set and testing on a different subject set, or changing decoding temperature or prompt template between calibration and deployment, could push empirical miscoverage above α. This is a direct test the paper does not run.
Because the derivation never uses the specific form of the nonconformity score beyond exchangeability, the same p-value recipe could be applied to other scores, such as semantic similarity or token-level likelihoods, to extend calibrated abstention from MCQA to open-ended generation.
The 20-sample temperature-1.0 decoding is treated as i.i.d.; if the model's sampling is not actually exchangeable across questions (context dependence, batch artifacts), the nominal α may understate true risk.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is split conformal prediction re-described as a significance-testing framework, and the central claim that the two are equivalent is mathematically wrong. The paper is not ready for publication as is.

What's good: the experimental setup is honest and straightforward. The authors sample 20 tokens per question, compute empirical option frequencies, build calibration scores, and report error rates that track alpha across MMLU and MMLU-Pro for several off-the-shelf models. The APSS curves decreasing with alpha are exactly what conformal theory predicts. As an empirical exercise, it confirms known behavior of split CP, but nothing more.

The soft spots are serious. Equations (3) and (4) are not equivalent. The inclusion rule in Eq (3) counts calibration scores <= the test score, while the p-value in Eq (5) uses a strict inequality, dropping tied calibration scores from the count. For n=10, alpha=0.5, a test score tied with the 6th smallest calibration score satisfies Eq (3) but gives p=5/11<0.5, excluding it. Because the nonconformity scores are discrete (empirical frequencies over 20 draws), ties are generic. The p-value is a strict-smaller version of the standard conformal p-value, making rejection more likely and breaking coverage. Randomized tie-breaking or a >= convention would fix it, but neither is present. Equation (7) uses an undefined N and is not a proof; it simply states the target bound.

The novelty claim is also overstated. The conformal p-value in Eq (5) is standard construction, and the 'integration' with significance testing is a restatement. Existing work applying CP to LLM MCQA is not cited. Exchangeability is assumed without testing, which is normal for CP, but the additional assumption of iid single-token generations across 20 samples is also unexamined.

Overall, the empirical results are consistent with theory only if the theory is corrected. As written, the load-bearing derivation fails, so the paper does not establish provable risk control. A desk rejection is appropriate. The core idea (frequency-based conformity scores for MCQA) is reasonable and could be salvaged with correct proofs and proper tie handling; the author should also release code.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes a conformal prediction (CP) framework for multiple-choice QA in which an LLM is treated as a black-box scorer via 20 single-token generations per question. Nonconformity scores S(x,y)=1-f_hat(y|x) are computed from empirical choice frequencies, calibration is performed per subject, and prediction sets are constructed by comparing a p-value (Eq. 5) with alpha. The paper claims that this p-value rule is equivalent to the split-CP quantile rule and therefore inherits the marginal coverage guarantee P(y*_test in C_alpha(x_test)) >= 1-alpha (Eq. 2). Experiments on MMLU and MMLU-Pro with several open models report empirical error rates close to or below alpha and a monotone decrease of average prediction set size with alpha.

Significance. If the coverage argument were correct, the p-value formulation would provide a useful bridge between hypothesis testing and CP for LLM uncertainty. The empirical study is broad and provides clear evidence that a standard split-CP baseline controls miscoverage on these benchmarks, and the APSS analysis is a useful descriptive tool. The manuscript does not ship code or machine-checked proofs, and the central derivation is not currently sound: the proposed p-value is anti-conservative under ties and the proof of Eq. (7) is missing. The paper's contribution as a 'significance testing-enhanced' framework is also weakened by the fact that the p-value rule is essentially a restatement of the split-CP inclusion rule.

major comments (4)

[Sec. 3.3, Eqs. (3)-(4)] Eqs. (3) and (4) are not algebraically equivalent. For n=10, alpha=0.5, the conformal threshold in Eq. (3) is the 6th smallest calibration score (k=ceil(11*0.5)=6), so a test score equal to that value satisfies Eq. (3) because #{s_i <= S_test}=6 <= 6. However, with the strict '>' in Eq. (5), p=(10-6+1)/11=5/11 <= 0.5, so Eq. (4) excludes the same label. The off-by-one arises because #{s_i > S_test} = n - #{s_i <= S_test}; the '+1' in Eq. (5) does not compensate for the strict inequality. Because the scores are discrete (multiples of 0.05 with P=20), ties are unavoidable; if all n calibration scores and s_test are equal, Eq. (5) gives p=1/(n+1), so for alpha >= 1/(n+1) the true label is excluded with probability 1, violating Eq. (2). The paper needs a conservative rule such as p=(#{s_i >= S_test}+1)/(n+1) or randomized tie-breaking, and a coverage proof for that rule.
[Sec. 3.3, Eq. (7)] Eq. (7) is not a proof of coverage. It uses an undefined N and merely restates the inequality that needs to be established. A rigorous argument must show that, under exchangeability of (s_1,...,s_n,s_{n+1}), the proposed p-value is super-uniform, i.e., P(p <= alpha) <= alpha for all alpha. In particular, the paper does not show that the strict '>' p-value of Eq. (5) is super-uniform; the equal-score example above shows it is not. This is a load-bearing gap because Eq. (7) is the link to the marginal guarantee Eq. (2).
[Algorithm 1 and Eq. (3)] Algorithm 1 contains multiple undefined symbols and a malformed condition. Line 3 iterates over 'D' without defining it; line 4 uses 'y*' instead of y*_i; line 6 sorts '{s_i}_{i=1}^N' although N is never defined (the calibration size is n); line 9 has 'if P(...) <= alpha then continue', where the outer 'P' appears to be a typo and the action of 'continue' conflicts with the surrounding else branch. Eq. (3) also writes S(x_i,y) where the prose (and Eq. (4)) use S(x_test,y). These errors prevent the reader from determining the exact implemented procedure.
[Sec. 4.2 vs. Sec. 3.3] Section 4.2 states that the empirical validation constructs prediction sets via Eq. (3), i.e., the standard split-CP quantile rule, rather than via the p-value decision rule of Eqs. (4)-(5). Consequently the experiments do not validate the paper's claimed significance-testing formulation. If the implementation actually used the p-value rule, the manuscript must say so explicitly and address the tie issue; as written, the empirical findings are consistent with a standard split-CP baseline and do not support the specific ST-CP claims.

minor comments (4)

[Sec. 3.1] 'Typel error rate' should be 'Type I error rate'.
[Sec. 4.2] The sentence 'all results are not significantly lower than our set significance level, but are lower than the set value within a small range' is unclear; no statistical significance test is reported to support 'not significantly lower'.
[Figures] Figure numbering is inconsistent: the text refers to Figures 2-4, but Figure 1 (box plots) appears before Figure 2; the captions and callouts should be normalized.
[Sec. 4.1.3] The choice P=20, temperature=1.0, top-p=0.9 is not justified and no ablation is provided. Since the scores are discrete, this choice directly affects the tie problem discussed in the major comments.

Circularity Check

3 steps flagged · score 6.0 of 10

The ST-CP derivation is partly a restatement of split conformal prediction: the p-value is defined as the conformal inclusion rule, Algorithm 1 uses the very coverage bound it claims to prove, and the reported APSS monotonicity is built into the quantile formula.

self definitional [Algorithm 1, line 9]
"if P(∑_{i=1}^n 1{s_i > s_test} + 1)/(n+1) ≤ α ≤ α then continue else C_α(x_test) ← S(x_test, y_test^k)"

The decision criterion in Algorithm 1 is exactly the probabilistic bound in Eq. (7): it asks whether P(p ≤ α) ≤ α, which is the very coverage theorem the paper claims to prove. Thus the algorithm's inclusion/exclusion decision presupposes the result rather than computing it from data. Even though the surrounding text says to reject H0 when p ≤ α, the pseudocode as written makes the prediction rule dependent on the guarantee it is supposed to establish, so the 'prediction set' cannot fail by construction.
self definitional [Section 3.3, Eqs. (3)-(5)]
"Through algebraic manipulation of Equation (3), we obtain the equivalent condition: [∑_{i=1}^n 1{s_i > S(x_test, y)} + 1]/(n+1) > α (4). This equivalence establishes our hypothesis testing framework with: Null hypothesis H0: y is the true label of x_test ... We define the p-value statistic as: p(y) = [∑_{i=1}^n 1{s_i > S(x_test, y)} + 1]/(n+1) (5)"

The p-value is not derived from an independent significance test; it is defined as exactly the left-hand side of Eq. (4), the conformal inclusion condition. Therefore 'reject H0 iff p(y) ≤ α' is, by construction, the complement of the conformal prediction-set rule. The claimed integration of significance testing with conformal prediction is a restatement/renaming of split conformal prediction (the conformal p-value), not an independent statistical derivation. All subsequent coverage statements inherit the standard conformal proof, so the 'significance testing' component adds no new mathematical content.

1 more flagged steps

self definitional [Section 4.3, Table 2 and following paragraph]
"Tabular data confirm that increasing confidence threshold α monotonically reduces the Average Prediction Set Size (APSS) across all models, consistent with selective prediction theory."

The inclusion rule in Eq. (3) has threshold ⌈(n+1)(1−α)⌉/n, which is non-increasing in α; hence the prediction sets are nested and the average prediction set size must be monotone non-increasing by construction. Presenting this as an empirical confirmation/validation of APSS as an uncertainty metric is a definitional consequence of the quantile formula, not an independent empirical finding. The table cannot fail to show the monotonic trend unless implementation errors occur, so the reported 'validation' does not provide new evidence for the method.

full rationale

The paper's core coverage claim, Eq. (2), is standard split conformal prediction under exchangeability; it is not fitted to data and does not rely on self-citation. However, three definitional equivalences make the presentation circular: (1) Algorithm 1's inclusion condition literally checks the coverage bound Eq. (7) it claims to prove; (2) the p-value in Eq. (5) is defined as the complement of the conformal inclusion rule in Eq. (4), so the 'significance testing' framework is a restatement of split conformal p-values rather than an independent derivation; and (3) the APSS monotonicity reported as experimental validation follows directly from the decreasing quantile threshold in Eq. (3). These are not cases of a fitted parameter being renamed as a prediction, but they do mean that several claimed 'results' are true by construction. Separately, and not counted as circularity, the strict-inequality p-value in Eqs. (5)-(7) lacks handling of discrete score ties, so the coverage proof as written is incomplete; this is a correctness gap rather than a circular step. Overall, the central guarantee retains independent conformal-theory content, but the paper's derivation chain conflates the method with its own guarantee, warranting a score of 6 rather than a higher score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central guarantee rests on exchangeability and the standard conformal quantile lemma; no fitted constants appear. P=20, temperature, top-p, and split ratio are hand-set experimental choices rather than fitted parameters, and alpha is a user-specified risk level.

free parameters (4)

P (number of independent samplings per question) = 20
Chosen by hand in Section 4.1.3; affects empirical frequency estimates and thus the nonconformity scores, but not the coverage proof.
temperature = 1.0
Decoding parameter set in Section 4.1.3; affects response distribution and score estimates.
top-p = 0.9
Decoding parameter set in Section 4.1.3; chosen by hand.
calibration-test split ratio = 0.5
Main experiments use split-ratio=0.5; the paper claims robustness to other ratios but does not report those results.

assumptions (4)

domain assumption Calibration and test nonconformity scores are exchangeable
Required for split conformal coverage; the paper asserts per-subject splitting satisfies this in the introduction and Section 4.1.3.
domain assumption The LLM's per-question sampled responses are i.i.d., so observed option frequencies estimate a fixed score distribution
Used in Section 3.2 to compute fhat(y|x) from 20 generations; needed for the nonconformity scores to be well-defined and exchangeable.
domain assumption A single-token constrained output maps cleanly to one of the K answer choices
Section 4.1.3 constrains output length to single tokens; the frequency counts only make sense if every sampled token corresponds to a valid option.
standard math Standard split conformal quantile construction gives finite-sample marginal coverage
Underlies Eq (2) and Eq (7); it is a known theorem, but the paper does not provide a complete proof.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control." pith.science (2026). https://pith.science/paper/HV4HYXRB

@misc{pith2026250810022,
  author       = {Pith},
  title        = {Pith review of: Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HV4HYXRB}},
  note         = {Machine review of arXiv:2508.10022}
}

abstract

This study introduces a significance testing-enhanced conformal prediction (CP) framework to improve trustworthiness of large language models (LLMs) in multiple-choice question answering (MCQA). While LLMs have been increasingly deployed in disciplinary QA scenarios, hallucination and nonfactual generation substantially compromise response reliability. Although CP provides statistically rigorous marginal coverage guarantees for prediction sets, and significance testing offers established statistical rigor, their synergistic integration remains unexplored. To mitigate hallucination and factual inaccuracies, our framework integrates $p$-value computation with conformity scoring through self-consistency resampling of MCQA responses. This approach calculates option frequencies to address LLMs' black-box nature, subsequently constructing prediction sets via null hypothesis testing ($\mathcal{H}_0$) with empirically derived $p$-values. Evaluations on MMLU and MMLU-Pro benchmarks using off-the-shelf LLMs demonstrate: (1) The enhanced CP achieves user-specified empirical miscoverage rates; (2) Test-set average prediction set size (APSS) decreases monotonically with increasing risk levels ($\alpha$), validating APSS as an effective uncertainty metric. This work establishes a principled statistical framework for trustworthy LLM deployment in high-stakes QA applications.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 9 canonical work pages

[1]

A gentle introduction to conformal prediction and distribution-free uncertainty quantification

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021

arXiv 2021
[2]

Prism: Self-pruning intrinsic selection method for training-free multimodal data selection

Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. arXiv preprint arXiv:2502.12119 , 2025

arXiv 2025
[3]

LL a VA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. LL a VA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2025

work page 2025
[4]

Cot-kinetics: A theoretical modeling assessing lrm reasoning process

Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. Cot-kinetics: A theoretical modeling assessing lrm reasoning process. arXiv preprint arXiv:2505.13408 , 2025

arXiv 2025
[5]

Fedbip: Heterogeneous one-shot federated learning with personalized latent diffusion models

Haokun Chen, Hang Li, Yao Zhang, Jinhe Bi, Gengyuan Zhang, Yueqi Zhang, Philip Torr, Jindong Gu, Denis Krompass, and Volker Tresp. Fedbip: Heterogeneous one-shot federated learning with personalized latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference , 2025

work page 2025
[6]

Does machine unlearning truly remove model knowledge? a framework for auditing unlearning in llms

Haokun Chen, Yueqi Zhang, Yuan Bi, Yao Zhang, Tong Liu, Jinhe Bi, Jian Lan, Jindong Gu, Claudia Grosser, Denis Krompass, et al. Does machine unlearning truly remove model knowledge? a framework for auditing unlearning in llms. arXiv preprint arXiv:2505.23270 , 2025

arXiv 2025
[7]

Conformal alignment: Knowing when to trust foundation models with guarantees

Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees. Advances in Neural Information Processing Systems , 37:73884--73919, 2024

work page 2024
[8]

Do llms know when to not answer? investigating abstention abilities of large language models

Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. Do llms know when to not answer? investigating abstention abilities of large language models. arXiv preprint arXiv:2407.16221 , 2024

arXiv 2024

Show all 18 references

[9]

Backdoor cleaning without external guidance in mllm fine-tuning

Xuankun Rong, Wenke Huang, Jian Liang, Jinhe Bi, Xun Xiao, Yiming Li, Bo Du, and Mang Ye. Backdoor cleaning without external guidance in mllm fine-tuning. arXiv preprint arXiv:2505.16916 , 2025

2025 arXiv
[10]

Sample then identify: A general framework for risk control and assessment in multimodal large language models

Qingni Wang, Tiantian Geng, Zhiyuan Wang, Teng Wang, Bo Fu, and Feng Zheng. Sample then identify: A general framework for risk control and assessment in multimodal large language models. In The Thirteenth International Conference on Learning Representations , 2025

2025
[11]

Conformalized multiple testing after data-dependent selection

Xiaoning Wang, Yuyang Huo, Liuhua Peng, and Changliang Zou. Conformalized multiple testing after data-dependent selection. Advances in Neural Information Processing Systems , 2024

2024
[12]

Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm

Yujun Wang, Jinhe Bi, Yunpu Ma, and Soeren Pirk. Ascd: Attention-steerable contrastive decoding for reducing hallucination in mllm. arXiv preprint arXiv:2506.14766 , 2025

2025 arXiv
[13]

Conu: Conformal uncertainty in large language models with correctness coverage guarantees

Zhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang, Qingni Wang, Xiaoshuang Shi, Kaidi Xu, Heng Tao Shen, and Xiaofeng Zhu. Conu: Conformal uncertainty in large language models with correctness coverage guarantees. In Findings of the Association for Computational Linguistics: EMNL...

2024
[14]

Coin: Uncertainty-guarding selective question answering for foundation models with provable risk guarantees

Zhiyuan Wang, Jinhao Duan, Qingni Wang, Xiaofeng Zhu, Tianlong Chen, Xiaoshuang Shi, and Kaidi Xu. Coin: Uncertainty-guarding selective question answering for foundation models with provable risk guarantees. arXiv preprint arXiv:2506.20178 , 2025

2025 arXiv
[15]

Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering applications and beyond

Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Yue Zhang, Ren Wang, Xiaoshuang Shi, and Kaidi Xu. Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering applications and beyond. Engineering Applications of Artificial ...

2025
[16]

SC on U : Selective conformal uncertainty in large language models

Zhiyuan Wang, Qingni Wang, Yue Zhang, Tianlong Chen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. SC on U : Selective conformal uncertainty in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

2025
[17]

Benchmarking llms via uncertainty quantification

Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification. Advances in Neural Information Processing Systems , 2024

2024
[18]

Spot! revisiting video-language models for event understanding

Gengyuan Zhang, Jinhe Bi, Jindong Gu, Yanyu Chen, and Volker Tresp. Spot! revisiting video-language models for event understanding. arXiv preprint arXiv:2311.12919 , 2023

2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

A gentle introduction to conformal prediction and distribution-free uncertainty quantification

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021

arXiv 2021

[2] [2]

Prism: Self-pruning intrinsic selection method for training-free multimodal data selection

Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. arXiv preprint arXiv:2502.12119 , 2025

arXiv 2025

[3] [3]

LL a VA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. LL a VA steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2025

work page 2025

[4] [4]

Cot-kinetics: A theoretical modeling assessing lrm reasoning process

Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. Cot-kinetics: A theoretical modeling assessing lrm reasoning process. arXiv preprint arXiv:2505.13408 , 2025

arXiv 2025

[5] [5]

Fedbip: Heterogeneous one-shot federated learning with personalized latent diffusion models

Haokun Chen, Hang Li, Yao Zhang, Jinhe Bi, Gengyuan Zhang, Yueqi Zhang, Philip Torr, Jindong Gu, Denis Krompass, and Volker Tresp. Fedbip: Heterogeneous one-shot federated learning with personalized latent diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference , 2025

work page 2025

[6] [6]

Does machine unlearning truly remove model knowledge? a framework for auditing unlearning in llms

Haokun Chen, Yueqi Zhang, Yuan Bi, Yao Zhang, Tong Liu, Jinhe Bi, Jian Lan, Jindong Gu, Claudia Grosser, Denis Krompass, et al. Does machine unlearning truly remove model knowledge? a framework for auditing unlearning in llms. arXiv preprint arXiv:2505.23270 , 2025

arXiv 2025

[7] [7]

Conformal alignment: Knowing when to trust foundation models with guarantees

Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees. Advances in Neural Information Processing Systems , 37:73884--73919, 2024

work page 2024

[8] [8]

Do llms know when to not answer? investigating abstention abilities of large language models

Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. Do llms know when to not answer? investigating abstention abilities of large language models. arXiv preprint arXiv:2407.16221 , 2024

arXiv 2024