{"id":"b53f03a4-e3ec-4672-91f3-0d42a38547e1","arxiv_id":"2506.08295","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AR-Bench shows that LLMs struggle with active reasoning: even GPT-4o solves only 35% of the guessing-number task and 54% of detective cases, while humans reach 80-100%.","lead":"This paper introduces AR-Bench, a benchmark of 6,040 puzzles that tests whether large language models can ask useful questions when they start with incomplete information. Across detective cases, situation puzzles, and number guessing, current models like GPT-4o score far below humans, suggesting that active information-seeking is a weak point of today's AI systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central quantitative claim rests on LLM-judge answers whose accuracy is validated only for GPT-4o SP questions; DC and cross-model reliability, plus the self-referential process scorer, are unmeasured.","rationale":"The reader's weakest_assumption captures the main risk: the benchmark's interactive scores depend on a judge whose reliability has only been spot-checked on one task and one source model. I agree with that, and I add that the process-score oracle is the same model, creating a self-referential loop that the 200-question audit does not address. I do not escalate to REJECT because GN is rule-based, the SP audit is genuine positive evidence, and the benchmark and code are public, making the proposed audit straightforward. A full passive-control comparison (giving models the complete ground truth and asking for the solution) would further separate 'question-asking failures' from 'deduction failures,' but the process-score evidence is the paper's own route to that conclusion, so judge reliability is the more direct load-bearing concern. Reader's verdict of CONDITIONAL remains appropriate pending such an audit.","tokens_in":47951,"tokens_out":5483,"duration_ms":72792,"concrete_test":"Perform a stratified human audit: sample 50 questions from each evaluated model in DC and 50 in SP, have two annotators derive the correct answer from the ground-truth story, compare with Llama-3.1-405B's NPC answer, and independently label f(st, qi) for the same trajectories. If DC or cross-model judge accuracy falls below ~90%, or if human process labels differ from the LLM labels by more than ~5 points, recompute the outcome and process tables using the human-verified answers; the plateau claim survives only if the corrected curves still show the reported gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's conclusion that LLMs fail to actively obtain information (Observations 4.1, 4.5, 4.8) rests on two LLM-judge components: Llama-3.1-405B as the NPC answerer in DC and SP, and the same model as the process-score oracle f(st, qi) in Sec. 3. The only reliability check (Sec. 4.12, Fig. 12) audits 200 GPT-4o questions from SP; it does not cover DC, does not stratify by asking model, and does not validate the process-score labels. This matters because DC NPCs are role-played with separate 'tasks' (Appendix C.5, Listing 6, instructs one suspect to 'sow confusion'), a setting where the 96% SP accuracy may not transfer. If NPC answers are noisy, or if errors correlate with the asking model, the outcome scores and process-score plateau could misattribute oracle failures to question-asking failures, which is the paper's central claim. The GN results are rule-based and unaffected, and SP has partial support, so this is not fatal; but the across-task and cross-model generalization of the judge is load-bearing and unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AR-Bench, a benchmark for evaluating active reasoning in LLMs under incomplete information. It contains three task families: detective cases (DC), situation puzzles (SP), and guessing numbers (GN), spanning commonsense, logical, and symbolic reasoning. The evaluation protocol has the LLM under test play a multi-round questioning game against an NPC implemented by Llama-3.1-405B (for DC and SP) or a rule-based oracle (for GN), after which the model gives a final answer. Outcome metrics (accuracy, F1, exact match) and a process metric based on key questions are reported for eight LLMs and several prompting/training methods. The central finding is that current LLMs, including GPT-4o, achieve low outcome scores (e.g., 35% exact match in GN) and process scores that plateau around half of the key information, suggesting that the bottleneck in active reasoning is question asking rather than final deduction. The authors also report ablations showing limited gains from tree-of-thought, SFT, DPO, and other methods, and provide a public benchmark repository.","tokens_in":48212,"tokens_out":4760,"duration_ms":60303,"significance":"If the central claim holds, AR-Bench fills a real gap in LLM evaluation: existing benchmarks mostly measure passive reasoning with complete information, while many real-world applications require iterative questioning. The paper's strengths include a large-scale generated dataset with human verification of ground truths, a judge-reliability audit on 200 SP questions, a publicly released benchmark, and a systematic comparison of models and methods. The finding that modern LLMs fail to actively acquire the information needed to solve problems is important and falsifiable. However, the headline conclusion rests on an LLM judge whose reliability is only partially measured, and the evaluation lacks error bars or multiple seeds. These issues are fixable and do not invalidate the benchmark's usefulness, but they need to be addressed before the quantitative claims can be fully accepted.","major_comments":[{"comment":"The judge reliability audit is load-bearing but incomplete. The only reliability check reported is based on 200 questions generated by GPT-4o for the SP task, where Llama-3.1-405B achieves 96% accuracy. This audit does not cover DC, where NPCs are instructed to role-play with potentially deceptive behavior (Listing 6 explicitly tells one suspect to 'sow confusion'), nor does it stratify by the model asking the questions. Because Observations 4.1, 4.5, and 4.8 are computed against answers from this same judge, the cross-task and cross-model generalization of the judge is unverified. The authors should extend the audit to DC questions, include questions from all evaluated models, and report agreement per task and per asking model.","section":"Sec. 4.12, Appendix A.2, Listing 6"},{"comment":"The process score f(st, qi) is implemented by prompting Llama-3.1-405B to decide whether the current state resolves a key question, but no human agreement or inter-annotator analysis is reported for these labels. The process-score plateau that underlies Observations 4.5 and 4.8, as well as the positive/negative labels used for SFT and DPO in Sec. 4.1, all depend on this unvalidated oracle. The authors should validate f on a held-out sample with human labels and test sensitivity by comparing with an alternative judge, otherwise the claim that question quality is the primary bottleneck is underdetermined.","section":"Sec. 3, Eq. (1)"},{"comment":"No error bars, seeds, or significance tests are reported. Many comparisons are small in magnitude (e.g., Llama-3.1-8B DC zero-shot 31 vs. few-shot 32; Qwen-2.5-3B GN 0 vs. Qwen-2.5-7B GN 4), and GN scores of 0 vs. 1 are within a single sample. Since AR-Bench is intended as a benchmark for comparing methods and models, the authors should report means and variances over multiple runs, ideally with statistical tests or a stated threshold for meaningful differences.","section":"Sec. 4.1, Tabs. 5-6"}],"minor_comments":[{"comment":"The opening sentence says few-shot and zero-shot instruction methods outperform standard zero-shot, but Fig. 4(a) shows few-shot instruction (27) below zero-shot (31) for Llama-3.1-8B in DC; the wording should be adjusted to describe the actual cross-task pattern.","section":"Observation 4.2"},{"comment":"The GN process score formula states a digit is counted as misplaced if it appears in g but not at position i, with the caveat 'to avoid double-counting, each digit in g is considered only once.' As written, the formula can double-count if the guess contains duplicate digits; please specify that guesses are restricted to unique digits or provide the precise matching algorithm.","section":"Sec. 3, GN score formula"},{"comment":"Within a task, error-pattern proportions can sum to more than 100% (e.g., SP Llama-3.1-8B: 36% + 90% = 126%). If each error case can exhibit multiple error types, state this explicitly so the table is not misread as exclusive categories.","section":"Table 3 / Table 18"},{"comment":"There is a typo: '25 quesions' should be '25 questions', and the output-format instruction is repeated; please clean up the prompt text.","section":"Listing 26"},{"comment":"The explanation that SFT 'tends to memorize training data rather than foster active reasoning skills' is presented as a finding but is not directly supported by the reported experiments; please phrase it as a hypothesis or add supporting evidence such as a training-dynamics analysis.","section":"Appendix D.2, SFT analysis"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a useful contribution and the main direction is publishable if the evaluation reliability is tightened. The most important missing pieces are the DC and cross-model judge audit, validation of the process-score oracle, and error bars or multiple seeds. These are local empirical additions rather than changes to the benchmark design, so I do not see a need for rejection, but the current version overstates the certainty of the quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AR-Bench does something genuinely useful: it takes three known information-seeking games and bundles them into one benchmark with outcome and process metrics, and it makes a plausible case that existing active-reasoning benchmarks lack symbolic feedback and complex reasoning. The construction pipeline is careful for a synthetic benchmark—human verification of ground truths, a judge-reliability check on 200 SP questions, and a fixed open judge model for reproducibility. The headline finding—GPT-4o at 54% on DC, 63% on SP, 35% on GN, with process scores plateauing near half of the key questions resolved—is internally consistent, and the GN numbers are rule-based, so that part is solid.\n\nThe soft spots are real but not fatal. The judge is validated only on SP questions from GPT-4o, yet it also serves as the NPC answerer in DC and as the process-score oracle in both DC and SP. DC includes a suspect explicitly instructed to sow confusion, exactly the setting where judge accuracy might degrade. If NPC answers are noisy or biased by the asking model, both outcome and process scores could overstate the active-reasoning gap. The authors should stratify the judge audit by task and asking model, and ideally by question type. The process metric is self-referential: GPT-4o generates the puzzles, the key questions are extracted from the same generation tree, and the same LLM scores whether the conversation resolves them. That does not taint the outcome metric, but it makes the process numbers suggestive rather than definitive. Missing error bars and multiple seeds also weaken the precision of claims like 'SFT gets 0 on GN.' The human evaluation is a small demo, so the 'humans far exceed models' claim deserves a caveat.\n\nOn balance, the central qualitative finding is likely robust: current LLMs do not ask effective questions under incomplete information. The paper deserves a serious referee and, with the judge audit extended and error bars added, could become a standard eval tool. The reader's CONDITIONAL verdict is the right call.","headline":"Solid benchmark with a likely-true headline finding; the LLM-judge generalization and missing error bars are the main caveats.","tokens_in":48710,"tokens_out":2514,"would_cite":true,"duration_ms":30908,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AR-Bench tests whether LLMs can gather their own missing information; the paper presents evidence they cannot, with GPT-4o reaching only 35% on the guessing-number task and process scores plateauing near half of key questions.","keywords":["active reasoning","question asking","incomplete information","benchmark","large language models","information seeking","multi-turn interaction","reasoning evaluation"],"falsifier":"Give a failing model the complete answer key up front (every key question answered truthfully) and ask it to produce the final solution: if it then scores near ceiling, the question-asking bottleneck is confirmed, while if it still fails, deduction under full information is the weak spot. Conversely, replace the Llama-3.1-405B judge with human annotators on a sample of detective-case and situation-puzzle runs and compare outcome and process scores: if they move materially, part of the reported gap is a judge artifact rather than a model deficiency.","tokens_in":47786,"feed_emoji":"🧩","tokens_out":7589,"duration_ms":77925,"temperature":0.7,"pith_summary":"This paper introduces AR-Bench, a benchmark that asks whether large language models can solve problems when the information needed is not handed to them up front. Its central claim is that state-of-the-art models are far weaker at this 'active reasoning' than at the passive reasoning that existing benchmarks measure: across 25 rounds of interaction, GPT-4o identifies the correct answer in only 54% of detective cases, 63% of situation puzzles, and 35% of guessing-number games, while human participants reach 80%, 67%, and 100%. The paper argues that the bottleneck is asking useful questions rather than drawing conclusions, since process scores plateau at roughly half the key questions resolved, and additional rounds, tree search, and fine-tuning all yield little gain. A sympathetic reading takes the paper to establish that asking the right question under incomplete information is a distinct capability that current LLMs have not acquired.","feed_headline":"LLMs stall at 35% when they must ask for clues","feed_subtitle":"On AR-Bench, even GPT-4o answers only half the key questions; extra rounds and fine-tuning barely help.","key_machinery":"The load-bearing object is AR-Bench itself: three task families — detective cases, where the model interrogates five suspects to find a murderer (commonsense reasoning); situation puzzles, where it asks a judge yes-or-no questions to reconstruct the truth of a lateral-thinking riddle (logical reasoning); and guessing numbers, where it cracks a four-digit secret with digit-position feedback from a rule-based oracle (symbolic reasoning). Each puzzle is played as a 25-round conversation between the model under test and judge agents that hold the ground truth. Two metrics separate the failure modes: the outcome score measures the final answer (accuracy, F1, or exact match), while the process score measures, at each conversation state, how many of the puzzle's key questions have been resolved — with a Llama-3.1-405B verifier judging DC and SP states and an exact digit-count formula for GN. The process metric is what carries the claim that information acquisition, not deduction, is the limiting step.","core_discovery":"The paper's central discovery is that under incomplete information, contemporary LLMs fail in the question-asking phase of reasoning rather than the deduction phase. On AR-Bench, process scores — the fraction of puzzle-defining key questions whose answers the model's conversation has uncovered — rise quickly in the first few rounds and then flatten, with GPT-4o reaching roughly 51% in detective cases and 44% in situation puzzles after 25 rounds. Methods that succeed in passive reasoning transfer poorly: tree-of-thought matches or regresses on detective cases and situation puzzles, supervised fine-tuning scores 0% on guessing numbers, direct preference optimization underperforms zero-shot on two of three tasks, and the two dedicated active-reasoning methods, Proactive CoT and Uncertainty-of-Thought, give marginal or negative results. The paper concludes that the divergence between passive and active reasoning is stark and that improving active reasoning requires training with interactive learning, real-time feedback loops, and environment-aware objectives.","pith_inferences":["The paper's dichotomy suggests a reusable principle for benchmark design: a reasoning benchmark should first verify that the model can acquire the information the reasoning needs, because a model that fails to ask will look equally weak at deduction regardless of its actual deductive ability.","In the guessing-number task the answer space is fully enumerable (5,040 numbers), so the key-question metric could be sharpened into an exact expected-information-gain score per guess; the paper's plateau finding predicts that current models' guesses deliver far less than the information-theoretic optimum.","The judge-reliability check covers only GPT-4o's situation-puzzle questions; a natural extension is to re-run detective cases and all other models with independently verified answers, or with a different judge model, to test whether the reported model ordering is stable.","If the bottleneck is genuinely question generation, then collecting human question-asking traces — as the paper itself suggests in its future-directions appendix — should transfer more directly to AR-Bench performance than the outcome-only DPO data used here."],"forward_implications":["Models that dominate passive-reasoning benchmarks will still fail agentic applications that require gathering missing information, so information-seeking ability must be part of any serious agent evaluation.","Allowing more interaction rounds is not a cure: the process-score gain falls from an average of 7.7% between rounds 5 and 10 to 2.5% between rounds 20 and 25, and scaling to 100 rounds leaves DC and SP outcomes roughly unchanged.","Search and post-training remedies as currently practiced do not close the gap, since ToT, SFT, DPO, Proactive CoT, and UoT all deliver little or no benefit, implying new objectives that reward information gain are needed.","Question-asking quality scales with model size — Llama-3.1-405B and GPT-4o lead on process scores — so progress on active reasoning is partly a scaling story, but even the largest model resolves only about half of the key questions in 25 rounds.","The process metric gives a concrete training target: a method that raises key-question resolution toward 100% should pull outcome scores up with it, making AR-Bench a testbed for interactive-learning algorithms."],"supporting_citations":[{"why":"Supplies GPT-4o and GPT-4o-mini, the strongest evaluated models whose low outcome and process scores constitute the main evidence of the active-reasoning gap.","marker":"(Hurst et al., 2024)"},{"why":"Supplies the Llama-3.1 family, including the 405B model that serves as the judge for DC and SP and carries the evaluation pipeline.","marker":"(Dubey et al., 2024)"},{"why":"Provides tree-of-thought, the search-based method whose failure to improve active reasoning is a central negative result.","marker":"(Yao et al., 2023)"},{"why":"Provides direct preference optimization, one of the post-training methods shown to underperform zero-shot on SP and GN.","marker":"(Rafailov et al., 2023)"},{"why":"Provides Uncertainty-of-Thought, the prior active-reasoning method that performs worse than the zero-shot baseline on AR-Bench.","marker":"(Hu et al., 2024)"},{"why":"Provides Proactive CoT, the clarification method that achieves only marginal gains in SP and none in DC or GN.","marker":"(Deng et al., 2023a)"},{"why":"Inspires the tree-based story-expansion generation pipeline used to construct the DC and SP puzzles at scale.","marker":"(Sprague et al., 2024)"},{"why":"Provides the 20 Questions line of prior active-reasoning benchmarks that AR-Bench extends with symbolic feedback and complex reasoning.","marker":"(Abdulhai et al., 2023)"}],"fun_headline_variants":["LLMs can reason but cannot ask for missing info","Active reasoning: LLMs stall when clues are missing","LLMs fail at asking for clues, not at reasoning","AR-Bench reveals LLMs cannot ask the right questions","LLMs progress stalls after a few rounds of questioning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the assumption that the non-player judges — chiefly Llama-3.1-405B for detective cases and situation puzzles — answer every model's questions correctly and consistently, with the 96% reliability measured on 200 GPT-4o situation-puzzle questions assumed to hold for other tasks and other models, and that each puzzle's ground truth is uniquely correct as keyed.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can reason but cannot ask for missing info","Active reasoning: LLMs stall when clues are missing","LLMs fail at asking for clues, not at reasoning","AR-Bench reveals LLMs cannot ask the right questions","LLMs progress stalls after a few rounds of questioning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000874,"raw_usage":{"total_tokens":3805,"prompt_tokens":989,"completion_tokens":2816,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2738}},"tokens_in":605,"tokens_out":2816,"duration_ms":22729,"temperature":1.0,"reasoning_tokens":2738,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:13:56.442393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a failing model the complete answer key up front (every key question answered truthfully) and ask it to produce the final solution: if it then scores near ceiling, the question-asking bottleneck is confirmed, while if it still fails, deduction under full information is the weak spot. Conversely, replace the Llama-3.1-405B judge with human annotators on a sample of detective-case and situation-puzzle runs and compare outcome and process scores: if they move materially, part of the reported gap is a judge artifact rather than a model deficiency.","supporting_citations":[],"review_version":1}