{"id":"c568505d-ace7-4331-8d38-98c3bb9e7540","arxiv_id":"2606.16667","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Folding visual evidence acquisition into the conformal score and re-calibrating on post-acquisition scores preserves the hallucination-rate guarantee while recovering coverage in LVLM selective prediction.","lead":"A new method, BCEA, lets vision-language models look again at an image—zooming crops or flipping it—before choosing to answer or abstain, while preserving a certificate on the hallucination rate. It shows that naively adding this re-examination breaks the guarantee, and fixing it by re-calibrating on the final scores restores it and recovers coverage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's 'smallest t with Rδ ≤ α' conflicts with Theorem 1's fixed-sequence scan; the finite-sample guarantee may not apply to the implemented method.","rationale":"The reader's weakest assumption focuses on exchangeability and the existence of claim-type-specific interventions, which are scope conditions rather than internal threats. My stress test identifies a more immediate, potentially fatal inconsistency between the theoretical guarantee and the practical algorithm. Theorem 1's validity rests on a specific fixed-sequence scan that controls for multiple testing by stopping at the first safe threshold in a descending scan. Algorithm 1, as written, selects the smallest safe threshold, which is a different selection rule. Because the paper itself proves the selective risk is not monotone (Proposition 2), the 'smallest safe' rule cannot be justified by the fixed-sequence argument; it requires an additional multiple-testing correction (e.g., closed testing or Learn-then-Test). If the implementation truly uses the smallest t, the finite-sample guarantee is not established, making the central claim unproven. If it uses the descending scan, the reported coverage numbers may be misleadingly high. This is a concrete, testable concern: a synthetic experiment with known ground truth can determine whether the algorithm as specified meets the claimed 1−δ coverage. The verdict should remain CONDITIONAL because the issue is resolvable by aligning the algorithm with the theorem or by adding a valid correction, but it is a more load-bearing concern than the ones the reader raised.","tokens_in":16686,"tokens_out":16349,"duration_ms":177274,"concrete_test":"Run a synthetic simulation reproducing Algorithm 1 exactly. Generate exchangeable calibration and test scores with known labels and a non-monotone risk curve R(τ) (e.g., construct score distributions so risk dips then rises as τ decreases). For each of 10,000 replications, select τ by the 'smallest t with Rδ ≤ α' rule (as written) and by the fixed-sequence scan from largest to smallest (as in the proof). Compute the empirical probability that the true risk at the selected τ is ≤ α. If the 'smallest t' rule yields coverage below 1−δ = 0.9 while the fixed-sequence rule attains at least 0.9, the implemented algorithm violates the theorem. This directly settles whether the central guarantee holds for the method as actually specified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Theorem 1's proof specifies a fixed-sequence procedure: scan candidate thresholds from largest to smallest τ and stop at the first τ for which the Clopper–Pearson upper bound Rδ(τ) ≤ α. This controls the family-wise error rate because smaller (more permissive) thresholds are never tested after a larger threshold fails. However, Algorithm 1 (line 7) selects the 'smallest t' with Rδ ≤ α, i.e., the most permissive threshold satisfying the bound. If implemented literally, this is not the fixed-sequence scan: it tests all thresholds and picks the minimum with a valid p-value, which inflates the probability of selecting an unsafe threshold beyond δ unless a multiple-testing correction is applied. The paper explicitly notes (Proposition 2) that the selective risk R(τ) is not monotone in τ, so the safe set need not be a suffix; thus the gap is real. If the deployed algorithm is the 'smallest t' rule, the guarantee of Theorem 1 does not follow, and the central claim—that BCEA preserves the finite-sample guarantee—is unsupported. If instead the implementation uses the fixed-sequence scan from largest to smallest, then the selected threshold would be the largest safe threshold, likely yielding lower coverage than reported (Table 1), undercutting the coverage-recovery claim. Either way, there is a mismatch between the theorem and the algorithm that must be resolved.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BCEA, a method for adding budgeted visual evidence acquisition to conformal selective prediction for large vision-language models (LVLMs). The central idea is to turn the binary answer/abstain decision into a three-way decision in which borderline claims trigger acquisition of additional visual evidence (CLIP-guided crops, or claim-type-specific interventions such as horizontal flip), and then to fold the acquisition policy into the score function and recalibrate the conformal threshold on post-acquisition scores. The paper proves (Theorem 1) that this restores a finite-sample selective-risk guarantee in the exchangeable setting, characterizes when acquisition improves coverage via an ROC-dominance criterion (Theorem 2), and reports experiments on object-existence and spatial-relation claims across four open VLMs showing that BCEA controls risk near target while improving coverage over guaranteed abstention. The paper explicitly states its scope limits: it evaluates probed atomic claims rather than free-form generation, and it documents claim families for which no truth-flipping intervention recovers signal.","tokens_in":17002,"tokens_out":10674,"duration_ms":116912,"significance":"The problem is timely and important: conformal filters for LVLM hallucination provide distribution-free guarantees but are extremely conservative, and merely adding evidence acquisition destroys the guarantee. The paper makes a clear conceptual contribution by identifying the failure mode of naive acquisition and fixing it through post-acquisition recalibration. The theoretical machinery is standard but cleanly applied: Lemma 1 is the correct exchangeability-transport argument, Theorem 1 is a proper instantiation of fixed-sequence Clopper-Pearson risk control, and Theorem 2's ROC characterization is an elegant, falsifiable criterion for when acquisition helps. The empirical study is broad: four VLMs, the POPE benchmark, multiple claim families, 300 calibration/test splits, and honest reporting of 90th-percentile risk as the matched quantity. The paper also includes a valuable negative result on per-claim budget allocation and an explicit limitations section. If the theorem-algorithm gap described below is resolved, this is a solid contribution to reliable VLM inference.","major_comments":[{"comment":"There is a direct mismatch between the validity theorem and the implemented algorithm. Theorem 1 (and its proof in Appendix A) establishes validity for a fixed-sequence procedure: scan candidate thresholds from largest to smallest and stop at the first threshold whose Clopper-Pearson upper bound satisfies Rδ ≤ α; this controls the family-wise error rate. Algorithm 1 (line 7), however, selects the 'smallest t' with Rδ ≤ α, i.e., the most permissive threshold satisfying the bound. Because Proposition 2 explicitly states that the selective risk R(τ) is not monotone in τ, the set of safe thresholds is not necessarily a suffix, so the fixed-sequence argument does not cover global-minimum selection. Appendix B repeats the 'smallest value' wording while calling the procedure a 'monotone fixed-sequence test,' which is contradictory. The authors must either (i) implement the fixed-sequence scan a","section":"§4.3 / Algorithm 1 / Appendix A"},{"comment":"The paper asserts that 'each extra view weakly improves the ROC' and presents this as an empirical counterpart of Proposition 2 and Theorem 2. This assertion is not proven and does not follow from Proposition 2. Proposition 2 shows that at a fixed threshold τ, both TPR and FPR are non-decreasing with budget and the acceptance sets are nested. Nested acceptance does not imply pointwise ROC dominance: for a fixed FPR of the base score, the augmented score may require a higher threshold, and its TPR at that threshold can be lower. Theorem 2 makes ROC dominance the exact criterion for coverage improvement, so the budget-sweep discussion in §5.4 relies on an unproven theoretical premise. The empirical monotonicity in Figure 6 can stand on its own as an experimental finding, but the paper should either prove ROC dominance under the specific max-of-likelihoods structure or explicitly label the","section":"§4.4 / §5.4"}],"minor_comments":[{"comment":"The quantitative claim about naive acquisition's risk overshoot is inconsistent. The abstract says 'up to 17 points,' §1 says 'up to 13 points,' but Table 1 shows overshoots of 14, 20, and 21 points at α = 0.05, 0.10, and 0.20, respectively. Please harmonize these numbers.","section":"Abstract / §1 / Table 1"},{"comment":"The statement that 'selecting the smallest τ whose confidence-sequence upper bound is ≤α controls the selective risk at level α uniformly over all t' needs a simultaneous guarantee over the threshold grid. The confidence sequence C_t as defined is for a fixed τ; selecting τ after looking at the data requires a union bound or an e-process indexed by τ. This proposition is not used in the experiments, but the claim should be made precise or flagged as a conjecture.","section":"§4.3, Proposition 4"},{"comment":"The sentence 'coverage collapses to zero on those families' for counting, color, and binding is not quantified. A table or a specific number would make the limitation concrete.","section":"§6"},{"comment":"Figure 9's y-axis label 'acquisition gain  sacq sglob' appears to be missing a minus sign; presumably it should be 'sacq − sglob'.","section":"Notation / Figure 9"},{"comment":"No code availability statement is included. If the source code is released, adding a link would aid reproducibility of the CLIP-guided crop selection and the conformal calibration procedure.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea is good and the experimental evidence is substantial, but the Algorithm 1 / Theorem 1 mismatch is a load-bearing correctness issue that must be fixed before the paper can be accepted. The other theoretical concern about ROC monotonicity is also important but more local. I see no reason to reject; the concerns are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is right: when you acquire extra visual evidence, fold the acquisition policy into the score and recalibrate the conformal threshold on the post-acquisition scores. That principle is standard, but the paper earns its keep by showing how badly naive acquisition breaks the guarantee (coverage jumps, risk overshoots by 20+ points), and by demonstrating that a cheap, claim-type-specific intervention—horizontal flip for left/right relations—recovers signal where the global ungrounding score is at chance. The negative result on per-claim budget allocation is also clarifying: it doesn't help because the aggregate score doesn't become more informative. The limitations section is admirably honest: several claim types (counting, binding, above/below) simply have no usable intervention, so coverage collapses there. That is the faithful behavior, not a cover-up.\n\nThe main problem is a mismatch between Algorithm 1 and Theorem 1. Theorem 1 and its proof describe a fixed-sequence scan from the most to least conservative threshold, stopping at the first threshold whose Clopper–Pearson bound is ≤ α. That procedure controls FWER and gives a finite-sample guarantee. Algorithm 1, however, selects the *smallest* threshold (most permissive) where the bound ≤ α. With non-monotone selective risk, which the paper itself notes in Proposition 2, the smallest-safe-threshold rule is not covered by the fixed-sequence guarantee. Either the algorithm is wrong and the guarantee as stated does not apply to the deployed method, or the implementation actually uses the fixed-sequence scan and the reported coverage numbers are optimistic (the first accepted threshold would be the most conservative one, not the most permissive). This needs to be resolved before the central claim—\"BCEA preserves the finite-sample guarantee\"—can be taken at face value.\n\nMinor issues: the abstract's \"up to 17 points\" overshoot doesn't match Table 1's largest overshoot of 21 points; the POPE table reports coverage but not the claimed risk control; and the paper lacks error bars (though it does report 90th-percentile risk over 300 splits, which partly addresses this).\n\nOverall, the paper deserves a serious referee. The mechanism is sound, the interventions are clever, and the empirical work is honest about its boundaries. But the theory-algorithm gap is load-bearing for the guarantee claim, and the authors need to either patch Algorithm 1 to match the fixed-sequence proof or justify the smallest-t rule with a proper multiple-testing correction. I'd want to see that fixed before recommending acceptance.","headline":"A genuinely useful take on conformal evidence acquisition, but the algorithm as written doesn't match the theorem that's supposed to guarantee it.","tokens_in":17494,"tokens_out":4953,"would_cite":true,"duration_ms":53834,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BCEA shows that a vision-language model can be made to 'look again'—acquiring zoomed, cropped, or flipped evidence—without losing the finite-sample guarantee that the hallucination rate among asserted claims stays below a target level.","keywords":["vision-language models","hallucination","conformal prediction","selective prediction","evidence acquisition","recalibration","risk control","claim-type interventions"],"falsifier":"Tune the acquisition budget B (or a stopping threshold) on the same calibration labels used to select the conformal threshold, without a held-out split, and measure the 90th-percentile realized selective risk on exchangeable test claims. If it exceeds α by more than Monte Carlo error, the fixed-sequence bound no longer controls risk. Alternatively, run BCEA on a test set drawn from a different distribution than the calibration set and check whether the realized risk exceeds α, which would show the exchangeability assumption is load-bearing.","tokens_in":16580,"feed_emoji":"🔍","tokens_out":5699,"duration_ms":58889,"temperature":0.7,"pith_summary":"The paper shows that guaranteed selective prediction for vision-language models is unnecessarily conservative: to certify a low hallucination rate, conformal filters abstain on most claims, sometimes over 80%. It introduces Budgeted Conformal Evidence Acquisition (BCEA), which replaces the binary answer/abstain decision with a three-way choice—answer, abstain, or acquire additional visual evidence by re-examining the image—under a bounded compute budget. The central fix is to fold the entire acquisition policy into the score function and recalibrate the conformal threshold on post-acquisition scores, which provably restores the distribution-free hallucination guarantee while improving coverage. The paper also shows that claim-type-specific interventions, such as a horizontal flip for left/right relations, recover signal that global ungrounding scores miss, and gives a precise condition: acquisition helps exactly when it improves the score's ROC curve.","feed_headline":"Look again: recalibrated conformal evidence acquisition controls risk","feed_subtitle":"A new method lets vision-language models zoom or flip instead of abstaining, preserving the finite-sample hallucination guarantee while reco","key_machinery":"The evidence-sufficiency score s(x,c) = ℓ(x) − ℓ(T(x)) measures how much a claim's likelihood drops when the image is removed (T = blank) or a claim-specific intervention is applied; acquisition forms a set of zoomed crops and takes the max likelihood, s_acq(x,c) = max(ℓ(x), max_b ℓ(x^(b))) − ℓ(∅). The load-bearing identity is Theorem 1: applying a fixed, label-independent map A to every claim preserves exchangeability, so the Clopper-Pearson fixed-sequence procedure on post-acquisition scores controls selective risk at level α with probability 1−δ. Theorem 2 converts 'does acquisition help' into a falsifiable ROC-dominance criterion: coverage at fixed risk is a point on the ROC curve, so ac","core_discovery":"The central discovery is that naively plugging evidence acquisition into a calibrated conformal filter breaks the statistical guarantee—realized hallucination rate can overshoot the target by up to 17 points—because acquisition changes the score distribution and destroys the exchangeability between calibration and test that conformal validity relies on. The fix is to treat the acquisition policy as part of the score function and calibrate the threshold on the post-acquisition scores of the calibration claims (Theorem 1), so that calibration and test pass through an identical pipeline. This restores the finite-sample guarantee while still recovering coverage. A second discovery is that the gl","pith_inferences":["The paper evaluates probed atomic claims; composing BCEA with a claim parser for free-form generation is an untested extension. If the parser introduces noise or correlated claims, the exchangeability assumption in Theorem 1 may be violated, and the guarantee would need re-examination—the paper flags this gap explicitly.","The failure of vertical flip and the collapse on counting/color suggest a general principle for intervention design: an intervention is useful only if it is a symmetry of the image distribution that flips the truth value of the claim. This suggests intervention libraries could be built automatically by searching for transformations that are plausible and truth-altering on a small labeled probe set","Theorem 2's ROC criterion suggests a practical model-selection rule: when choosing between acquisition policies (e.g., uniform grid vs. model-guided crops), compare AUROC or the full ROC on a held-out split, not just coverage at one α, because the equivalence is pointwise in α.","The anytime-valid extension implies that one can monitor risk continuously on live traffic and keep calibrating until the desired coverage is reached, without a pre-fixed sample size—useful for deployment where data arrive sequentially."],"forward_implications":["Guaranteed selective prediction for vision-language models can become substantially less conservative: a model may re-examine the image instead of abstaining, while the hallucination rate among asserted claims remains provably bounded.","The invariance principle—fold any fixed, label-independent acquisition policy into the score and recalibrate—extends beyond vision: any evidence-gathering step (querying a tool, running a verifier, asking a follow-up) can be added to a conformal predictor without losing validity, as long as calibration and test traverse the identical pipeline.","Claim-type-specific interventions become a design knob: for any claim family with a truth-flipping, in-distribution intervention, coverage can be recovered without sacrificing the guarantee; for families where no such intervention exists (counting, color binding in current backbones), the honest behavior is to abstain.","Theorem 2 gives a measurable acceptance test for acquisition policies: compute whether the post-acquisition ROC dominates the pre-acquisition ROC; if not, acquisition may not improve coverage at any fixed risk level.","Anytime-valid calibration (Proposition 4) means a deployed reliability monitor can keep collecting labeled claims and stop based on observed coverage without invalidating the guarantee.","The paper's explicit negative result—per-claim budget allocation is coverage-neutral without predictable heterogeneous gains—delimits where adaptive compute spending can help."],"fun_headline_variants":["Budgeted evidence acquisition slashes abstention, keeps guarantee","Look again, not abstain: calibrated evidence acquisition wins","Zoom, crop, verify: conformal guarantee without the abstention toll","Acquire evidence, not abstention: a budgeted conformal fix","Recalibrate after evidence: less abstaining, same safety"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The guarantee rests on exchangeability of test claims with the calibration set after the acquisition policy is applied; if the deployment distribution drifts or the policy is tuned using calibration labels, the finite-sample risk bound no longer holds. Equally load-bearing is the existence of a claim-type-specific, in-distribution, truth-flipping intervention—without one (counting, color binding, above/below), the method's coverage collapses to zero.","fun_headline_variants_meta":{"raw":{"variants":["Budgeted evidence acquisition slashes abstention, keeps guarantee","Look again, not abstain: calibrated evidence acquisition wins","Zoom, crop, verify: conformal guarantee without the abstention toll","Acquire evidence, not abstention: a budgeted conformal fix","Recalibrate after evidence: less abstaining, same safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000415,"raw_usage":{"total_tokens":1952,"prompt_tokens":689,"completion_tokens":1263,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":1174}},"tokens_in":433,"tokens_out":1263,"duration_ms":10539,"temperature":1.0,"reasoning_tokens":1174,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:08:33.418304+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Tune the acquisition budget B (or a stopping threshold) on the same calibration labels used to select the conformal threshold, without a held-out split, and measure the 90th-percentile realized selective risk on exchangeable test claims. If it exceeds α by more than Monte Carlo error, the fixed-sequence bound no longer controls risk. Alternatively, run BCEA on a test set drawn from a different distribution than the calibration set and check whether the realized risk exceeds α, which would show the exchangeability assumption is load-bearing.","supporting_citations":[],"review_version":1}