{"id":"a676fcd2-f99b-474a-ac00-ba6343bf1230","arxiv_id":"2501.00555","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Conformal pruning of answer choices plus a second LLM pass improves MCQ accuracy in most tested settings, and learned scores yield smaller prediction sets.","lead":"This paper proposes CROQ, which uses conformal prediction to prune answer choices before re-asking a large language model, and CP-OPT, a learned scoring scheme that keeps the pruned sets small. Experiments across three datasets show accuracy gains in most setups, but the effect is uneven and absent on standard four-option MMLU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CROQ's accuracy gains may come from re-labeling options to start at A and from the second forward pass, not from pruning; no control isolates the pruning effect.","rationale":"The reader identified Assumption 3.1 as the weakest assumption. I agree it is unsupported: conditioning Table 6's Acc. After on the reported Coverage shows fpost is not monotone for logit scores (e.g., Gemma-2 MMLU-15 logits: k=7 fpost≈69.6%, k=8≈65.2%, k=9≈62.6%, k=10≈65.1%). However, that assumption is used mainly to justify why smaller prediction sets should be preferred (Prop. 3.2, claim 3); the empirical headline could in principle still hold even if the assumption fails. The more load-bearing gap is attribution: CROQ changes the prompt in two ways simultaneously. Step 2 of §3.1 re-labels surviving options to contiguous letters starting at A. Since the model's answer is a greedy argmax over keys, and the paper itself cites option-order bias (Zheng et al. 2024), the re-labeling alone can alter predictions. No experiment separates pruning from re-labeling or from the extra forward pass. The abstract's 'improves accuracy over standard inference' therefore overstates what the current experiments establish. I would not reject the paper: the method is clearly described, the CP coverage framework is standard, and several settings show sizable, significant gains. But those gains need to survive a no-prune control before the central claim can be accepted. Hence the reader's CONDITIONAL verdict is appropriate; I would keep it conditional pending the control.","tokens_in":35218,"tokens_out":11879,"duration_ms":122940,"concrete_test":"On the same test splits and models, add a same-compute control: use the first forward pass exactly as in CROQ, but instead of pruning to the CP set, re-prompt with all m original options after applying a random permutation of the option keys (or, to match set sizes, remove m-k randomly chosen distractors while keeping the correct option and re-key survivors to A...k). Measure the control's accuracy gain over the first-pass answer and compare with the CROQ gains in Tables 2 and 5. If the control reproduces a large share of CROQ's gain, the headline claim that pruning causes the improvement is not established; if the control gain is near zero, the re-labeling/compute confound is ruled out.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that pruning choices improves accuracy. The CROQ procedure in §3.1 Step 2 combines pruning with a prompt modification: the surviving options are re-keyed to contiguous letters starting at A. Because inference is a greedy argmax over option keys (§2.1, Eq. 12), and LLMs are known to be sensitive to option order and key-labeling (the paper itself cites Zheng et al. 2024), this re-labeling can change the answer even if the set of option texts is unchanged. Thus the 'after' accuracies in Tables 2 and 5 (and the conditional tables in Appendix C) measure pruning plus re-labeling, not pruning alone. The comparison to 'standard inference' also mixes in a second forward pass. The monotonicity assumption (Assumption 3.1) is a separate, weaker concern: in Table 6 the logit-score fpost(k), obtained as Acc. After / Coverage, is not monotone (e.g., Gemma-2 MMLU-15 logits: k=7: 60.08/86.31≈69.6, k=8: 57.72/88.60≈65.2, k=9: 56.85/90.75≈62.6, k=10: 58.89/90.45≈65.1), so the formal argument that smaller sets maximize gain is not supported by the paper's own data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CROQ (conformal revision of questions), a two-stage procedure for multiple-choice LLM tasks: first, split conformal prediction is used to build a prediction set of answer options; second, the question is revised to contain only the options in that set, the surviving options are re-keyed to consecutive letters starting at A, and the LLM is prompted again. The authors argue that fewer options improve accuracy, and they introduce CP-OPT, a post-hoc score-learning framework that minimizes expected conformal set size while preserving marginal coverage. Experiments on MMLU, ToolAlpaca, and TruthfulQA with three instruction-tuned LLMs and 4/10/15 options compare accuracy before/after CROQ using logit scores and CP-OPT scores, and report set-size and coverage comparisons. The central claims are that CROQ improves accuracy over standard inference, that CP-OPT reduces set sizes at comparable coverage, and that CROQ with CP-OPT outperforms CROQ with logits.","tokens_in":35454,"tokens_out":6324,"duration_ms":61247,"significance":"If the causal effect were cleanly established, CROQ would be a practically valuable, test-time accuracy booster with a formal coverage guarantee, and CP-OPT would be a useful contribution to conformal score design for LLMs. The paper has real strengths: Proposition 2.1 gives a standard split-conformal coverage proof; the experiments use paired tests and report per-setting conditional tables; the NL2SQL application in Appendix C.3 demonstrates a cost reduction with preserved accuracy; and the set-size histograms give a clear picture of where CP-OPT changes the distribution of conformal sets. However, the central causal claim is currently confounded: the 'after' condition differs from the baseline in three ways at once (pruning, re-keying of options, and a second forward pass), so the reported gains cannot be attributed to pruning alone. The manuscript's own conditional data also undermine the monotonicity assumption that motivates the theoretical analysis. The contribution is promising and the issues are addressable, but the paper as written overstates its central claim.","major_comments":[{"comment":"The CROQ 'after' condition changes three things relative to the baseline: the option set is pruned, the surviving options are re-keyed to contiguous letters starting at A, and a second forward pass is used. Since the LLM answer is the greedy argmax over option keys (Eq. 12), and the paper itself cites Zheng et al. (2024) on LLMs' sensitivity to option ordering and key labeling, the accuracy gains in Tables 2 and 5 cannot be attributed to pruning alone. Please add control conditions: (i) re-ask the original unpruned question as a second-pass control; (ii) prune the option set but preserve the original keys (e.g., keep {C,D} rather than re-keying to {A,B}); and (iii) ideally, also vary the order of the surviving options. At minimum, report how much of the gain survives when re-keying is removed. Without such controls, the central claim that pruning choices improves accuracy is not established.","section":"§3.1, Step 2; Tables 2 and 5"},{"comment":"Assumption 3.1 (monotone conditional accuracy) is load-bearing for Proposition 3.2, but the manuscript's own conditional tables do not support strict monotonicity. In Table 6 (Gemma-2, MMLU-15, Ours), the conditional accuracy on questions where the correct answer is retained, computed as Acc. After / Coverage, is about 62.2% at k=8 (56.14/90.23) and about 62.2% at k=9 (57.49/92.40), and adjacent sizes show non-monotone dips (e.g., k=7 at 60.26/89.74, about 67.1%, is lower than k=6 at 70.81/90.54, about 78.2%). Moreover, the tables report raw 'Acc. After', which includes questions where the correct answer was pruned, rather than the conditional quantity fpost(k) defined in the text; fpost(k) is never estimated or reported. Please either test Assumption 3.1 empirically or explicitly relegate Proposition 3.2 to a conditional statement under an idealization, and report conditional accuracy separately from coverage.","section":"§3.1, Assumption 3.1; Table 6"},{"comment":"The abstract and Contribution 1 state that CROQ 'improves accuracy over the standard inference' and 'consistently improves accuracy', but Tables 2 and 5 show zero or negative gains in 8/27 and 3/27 settings respectively, including all three standard 4-option MMLU settings with logits (gains of -0.19, -1.19, and +0.07). The paper's own discussion in §4.2 correctly says 'in the vast majority of settings' and 'rare small drops'; the front matter should be aligned with those hedged claims. In addition, please report an aggregate summary across the 27 settings (e.g., mean gain with a confidence interval or a sign-test result) so that the reader can assess the overall effect rather than relying on the count of individually significant cells.","section":"Abstract; Contribution 1; §4.2, H2"},{"comment":"Contribution 2 claims CP-OPT yields smaller average set sizes 'at the same level (95%) of coverage', but Table 1 shows that in 6/27 settings the CP-OPT set-size reduction is accompanied by a statistically significant decrease in coverage relative to logits (e.g., MMLU-4 Gemma-2: 94.23 vs 95.16; MMLU-10 Gemma-2: 94.04 vs 95.00; TruthfulQA-4 Llama-3: 92.41 vs 95.95). The split-conformal guarantee is not invalidated because the threshold is recalibrated on a separate set, but the empirical comparison is not 'same coverage' in the settings where coverage drops significantly. Please either compare at a common empirical coverage level, or explicitly state the trade-off and reword the contribution to say 'comparable or slightly lower marginal coverage, with the differences listed in Table 1.'","section":"H1, Table 1; Contribution 2"}],"minor_comments":[{"comment":"The text says 'by the strong law of larger numbers'; this should be 'strong law of large numbers'.","section":"After Eq. (9)"},{"comment":"The architecture description uses d3 = (d + m)/4 in the text but the layer dimensions are W1 ∈ R^{d0×d1}, W2 ∈ R^{d1×d2}, W3 ∈ R^{d2×m}; the intended hidden dimension is d2 = (d + m)/4, so the notation is inconsistent and should be fixed.","section":"§3.2, definition of G"},{"comment":"The regularization term λ1||g||2^2 is introduced without a definition of λ1, and Table 21 lists 'weight decay' rather than λ1; please clarify whether weight decay is λ1 and report its values in the hyperparameter table.","section":"§3.2, Eq. (P2)"},{"comment":"The sufficient condition is written as 'rkρk > a/(mfpost(k))', which is ambiguous; it should be typeset as rkρk > a / (m · fpost(k)) to match the proof in Appendix B.2.","section":"Proposition 3.2, item 2"},{"comment":"The MMLU-Pro result is reported only as a '3% relative improvement' over a 36.4% baseline; please report the actual after-CROQ accuracy and the number of test questions so the reader can judge the magnitude.","section":"Appendix C.2"},{"comment":"The significance analysis uses per-setting paired t-tests at level 0.05 but does not adjust for multiple comparisons across the 27 settings; please note this or apply a simple correction (e.g., Benjamini-Hochberg) and state whether the conclusions change.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv v2 of an ICML 2025 paper, so the report evaluates the current text on its merits. The central issue is not the conformal machinery, which is standard and correctly handled, but the causal attribution of the accuracy gains: the current experimental design cannot separate pruning from re-keying and the second forward pass. This is fixable with control conditions and a more hedged abstract, so I recommend major revision rather than rejection. I would also encourage the editor to ask the authors to report fpost(k) explicitly, since their own tables appear to contradict Assumption 3.1 as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and worth taking seriously: use conformal prediction to prune answer options, then re-ask the LLM. The paper is honest about where the method works and where it doesn't, and the empirical sweep is broader than most in this area—three datasets, three models, 4/10/15 options, with non-significant drops reported alongside gains. The split-conformal machinery is standard, and the theory (Proposition 3.2) is a clean probability decomposition plus a knapsack observation. I also appreciate that they cite prior CP-for-LLM work transparently.\n\nThe main soft spot is the re-labeling confound. In Step 2, surviving options are re-keyed to start at A. The 'after' accuracy therefore measures pruning plus key-label changes, not pruning alone. Since the paper itself cites Zheng et al. on LLM sensitivity to option ordering, this is not a pedantic point. Without a control that re-keys the original option set, or that otherwise isolates the pruning effect, the central claim that 'pruning improves accuracy' is not established. The abstract also overstates: on standard 4-option MMLU, gains are flat or negative for two of three models, and several settings show small drops. The paper is transparent about these numbers, but the abstract says 'improves accuracy' without qualification.\n\nThe monotonicity assumption (Assumption 3.1) is shakier than the prose suggests. Their own Table 6 shows non-monotone conditional accuracy for Gemma-2 on MMLU-15, which weakens the theoretical motivation, though the empirical claim does not stand or fall on that assumption. The lack of released code and data is also a drawback for reproducibility, though the appendix is unusually detailed about hyperparameters and prompt formats.\n\nOverall, this is a genuine new idea—CROQ is not in the cited literature—and the paper deserves a serious referee. But the authors should be asked to add a re-keying control, report per-set-size results more carefully, and soften the abstract. I would engage with it, and I'd probably cite it as related work, but only after the mechanism is clarified.","headline":"A useful, clearly-written test-time pruning idea whose main empirical claim is undercut by a re-labeling confound and an overbroad abstract.","tokens_in":36075,"tokens_out":4971,"would_cite":true,"duration_ms":48785,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pruning an MCQ's answer choices to a conformal prediction set and re-asking the LLM improves accuracy, with learned scores (CP-OPT) making the gains larger.","keywords":["conformal prediction","large language models","multiple-choice question answering","prediction sets","score optimization","accuracy improvement","uncertainty quantification","question revision"],"falsifier":"Measure the conditional accuracy f_post(k) on a held-out MCQ benchmark with enough items per set size and compare overall CROQ accuracy with single-round inference on the same items. If f_post(k) does not decrease with k (for example, if revised 2-option questions are less accurate than revised 3-option questions) or if the paired difference is not positive, then Assumption 3.1 is violated for that setting and the paper's theoretical reason for CROQ helping disappears.","tokens_in":34971,"feed_emoji":"🎯","tokens_out":8532,"duration_ms":75968,"temperature":0.7,"pith_summary":"The paper tries to establish a test-time method for improving LLM accuracy on multiple-choice tasks without retraining: use conformal prediction to build a small set of answer choices that contains the correct answer with high probability, then re-ask the LLM with only those choices. It calls this conformal revision of questions (CROQ) and claims it improves accuracy over standard inference in most settings. Because raw logit scores often produce large sets, the paper also introduces CP-OPT, a post-hoc optimization that learns a score function to shrink prediction sets while keeping the conformal coverage guarantee. The empirical claim is that CROQ works, and works better with CP-OPT than with logit scores, on MMLU, ToolAlpaca, and TruthfulQA across three small instruction-tuned LLMs.","feed_headline":"Conformal pruning lifts LLM accuracy on multiple-choice tests","feed_subtitle":"Re-ask the question with only the choices conformal prediction keeps; learned scores shrink the sets.","key_machinery":"The central object is the conformal prediction set C(x; g, tau) = {y : g(x, y) >= tau}, used to prune the MCQ. The argument runs through the identity $\\Delta$ = sum_{k=1}^{m} r_k rho_k f_post(k) - a, where a is baseline accuracy, r_k is the share of questions whose prediction set has size k, rho_k is the coverage among those questions, and f_post(k) is the LLM's accuracy on a k-option revised question. Proposition 3.2 shows that under Assumption 3.1, which posits that f_post(k) is monotonically decreasing in k, the accuracy gain is maximized by concentrating r_k rho_k on small k, and the proof reduces this to a fractional knapsack problem. CP-OPT is the engine that pursues that concentration: it replaces hard set-size indicators with sigmoid surrogates and solves a differentiable penalty problem over a small neural network that takes the LLM's last-token logits and penultimate-layer representation as input.","core_discovery":"On the paper's own terms, the central discovery is that an LLM answering a multiple-choice question can be made more accurate by a two-step procedure: build a conformal prediction set over answer choices that covers the true answer with probability at least 1 - alpha, then rewrite the question using only the choices in that set and prompt the LLM again. The paper shows that this procedure, CROQ, improves accuracy over single-round inference in 19 of 27 settings with logit scores and in 24 of 27 settings with CP-OPT scores, with statistically significant gains concentrated on questions with 10 or 15 options. The paper further claims that CP-OPT, a neural-network score function trained to minimize expected set size subject to coverage, produces smaller conformal sets than raw logits in most settings while keeping coverage near the 95% target, and that CROQ with CP-OPT outperforms CROQ with logits in 22 of 27 settings. The argument is carried by the decomposition of post-CROQ accuracy into a sum over prediction-set sizes, showing that gains come from shifting probability mass toward smaller sets.","pith_inferences":["The paper's own conditional-accuracy tables show that f_post(k) is not strictly monotone for some models and datasets, so a more general version of the argument might need only a weighted-average or stochastic monotonicity; that weakening is not proved in the paper.","Because CP-OPT consumes only last-token logits and hidden states, it could be inserted into serving stacks as a lightweight pre-answering filter; the paper reports accuracy and set sizes, not serving latency, so that cost remains untested.","The same prune-and-reask loop could be applied to tool or API selection with very large candidate sets, and the NL2SQL experiment is an early indication; whether gains persist when there are hundreds of candidate actions is not tested.","A natural stress test would be to run CROQ with a score function that is deliberately uninformative, such as uniform random scores, to confirm that the accuracy gain is driven by set-size reduction rather than by hidden information in the conformal score."],"forward_implications":["CROQ can be applied to any pretrained LLM with no fine-tuning, so deployment can improve accuracy by adding a conformal scoring step and a second prompt.","Because the conformal coverage guarantee is distribution-free, users can choose alpha to trade a bounded chance of deleting the correct answer against the accuracy gain from smaller sets.","Using CP-OPT scores instead of raw logits reduces average conformal set sizes in most of the 27 settings, which makes the downstream revision more effective.","In the NL2SQL agentic workflow, the same conformal pruning reduced LLM cost by about 45 percent at comparable or better accuracy, so cost and accuracy gains can come together.","Accuracy gains are uneven: they are larger on 10- and 15-option questions and smaller or slightly negative on 4-option questions, so practitioners should tune alpha and verify gains per task."],"supporting_citations":[{"why":"Establishes conformal prediction and its coverage guarantees, which CROQ relies on.","marker":"Vovk et al. (2005)"},{"why":"Supplies the split-conformal score-threshold construction used to build prediction sets.","marker":"Angelopoulos and Bates (2022)"},{"why":"Provides distribution-free split-conformal coverage results that justify the threshold in Proposition 2.1.","marker":"Lei et al. (2018)"},{"why":"Prior use of LLM logits as conformal scores for MCQ answering, the baseline CROQ is compared against.","marker":"Kumar et al. (2023)"},{"why":"Heuristic self-consistency scores without logit access, motivating the need for learned CP-OPT scores.","marker":"Su et al. (2024)"},{"why":"Learning optimal conformal classifiers, the objective that CP-OPT adapts to LLM score learning.","marker":"Stutz et al. (2022)"},{"why":"MMLU dataset, the primary multiple-choice evaluation benchmark.","marker":"Hendrycks et al. (2021)"},{"why":"TruthfulQA dataset, a primary evaluation benchmark for truthful answering.","marker":"Lin et al. (2022)"},{"why":"ToolAlpaca dataset, the tool-selection evaluation benchmark.","marker":"Tang et al. (2023)"},{"why":"Fractional knapsack solution used to prove the greedy gain-maximization claim in Proposition 3.2.","marker":"Cormen et al. (2009)"}],"fun_headline_variants":["CROQ: prune choices by conformal prediction, then re-ask the LLM","Conformal sets shrink, accuracy rises: improved LLM MCQ answering","Ask twice, guess less: conformal pruning lifts LLM test scores","Learn to prune: CP-OPT shrinks sets, CROQ boosts accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that an LLM's accuracy on a revised question declines as more answer choices remain, so that pruning choices reliably makes the task easier; if re-asking with fewer options does not actually improve accuracy, the central mechanism for CROQ's gains collapses.","fun_headline_variants_meta":{"raw":{"variants":["CROQ: prune choices by conformal prediction, then re-ask the LLM","Conformal sets shrink, accuracy rises: improved LLM MCQ answering","Ask twice, guess less: conformal pruning lifts LLM test scores","Learn to prune: CP-OPT shrinks sets, CROQ boosts accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1783,"prompt_tokens":1011,"completion_tokens":772,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":688}},"tokens_in":627,"tokens_out":772,"duration_ms":7639,"temperature":1.0,"reasoning_tokens":688,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:48:15.768291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the conditional accuracy f_post(k) on a held-out MCQ benchmark with enough items per set size and compare overall CROQ accuracy with single-round inference on the same items. If f_post(k) does not decrease with k (for example, if revised 2-option questions are less accurate than revised 3-option questions) or if the paired difference is not positive, then Assumption 3.1 is violated for that setting and the paper's theoretical reason for CROQ helping disappears.","supporting_citations":[],"review_version":1}