{"id":"63a06779-8488-4ebd-a291-70e168bd7e19","arxiv_id":"2607.29211","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A reinforcement-learning method that rewards refusal over hallucination cuts LLM 'futile reasoning' from ~66-79% to 1-7% on Countdown while roughly preserving accuracy.","lead":"Large language models often keep making confident but wrong attempts on problems they cannot solve. This paper measures that failure mode and trains models to stop and say \"I don't know\" instead, reducing wasted reasoning without losing much accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HRA assumes every failed trace is refusal-worthy; on solvable Countdown instances this trains over-refusal, and the claimed 'preserved performance' is contradicted by the 14B OOD accuracy drop (50.75→44.87) without error bars.","rationale":"The reader's weakest assumption—that every failed reasoning attempt is refusal-worthy—is also the load-bearing soft spot I find. The concern is not just theoretical: Table 1 shows a measurable accuracy drop on the harder OOD task, and the futile-rate metric is constructed so that increasing refusal mechanically lowers it. The paper's response would need to show that the newly refused instances are ones the model could not solve, not merely ones it failed to solve at inference time. My proposed test directly measures refusal on a vanilla-solvable subset, which would settle whether HRA is doing capability alignment or over-refusal. I do not see a reason to change the reader's CONDITIONAL verdict: the results are plausible and the deltas are large, but the central 'without sacrificing utility' claim is not yet secured. The zero-refusal inconsistency in §3.2/§4.2 versus Table 1 is a secondary accuracy issue in the paper's own reporting, but it does not independently decide the central claim.","tokens_in":15091,"tokens_out":4447,"duration_ms":48241,"concrete_test":"Recompute Table 1 on the subset of OOD Sudoku instances that vanilla Qwen3-14B solves in at least one of its eight samples (i.e., solvable within the model's demonstrated capability). If CaRL's refusal rate on this subset exceeds 10% or its accuracy on this subset is lower than vanilla's, HRA has induced over-refusal on solvable tasks and 'preserving performance' fails. To isolate the mechanism, also retrain CaRL with HRA applied only to failures on instances where no sample in the group succeeds; if the futile-rate reduction mostly disappears, the original result depends on relabeling solvable failures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CaRL reduces futile reasoning 'while preserving performance' (Abstract; §5.4). The mechanism driving the headline futile-rate reductions is Hindsight Refusal Augmentation (§4.2), which relabels every on-policy incorrect trajectory as a refusal demonstration. This is capability-aligned only if failures are confined to queries the model cannot solve. But the Countdown training set consists of solvable instances with golden answers; a failure can be an inference-time miss on a query the model is capable of solving. HRA converts those misses into refusals, so the policy is rewarded for refusing solvable questions. The paper's own numbers expose the risk: on OOD Sudoku, 14B accuracy drops from 50.75 to 44.87 while refusal rises to 38.87%, and RFT collapses to 0% accuracy with >90% refusal (Table 1). Because no error bars or seeds are reported, the −5.9 accuracy change cannot be dismissed as noise. The abstract's 'without sacrificing utility' therefore rests on an untested assumption that the refused instances are exactly the unsolvable ones. This is an internal correctness risk in the training objective, not a disagreement with consensus. The paper also states baseline 0% refusal in §3.2 and §4.2 despite Table 1 reporting 13.92% vanilla refusal on Qwen3-8B, which weakens the motivation but is secondary to the over-refusal concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines and characterizes 'futile reasoning' — long, plausible-looking but incorrect chains produced by LLMs on tasks beyond their capability — using a difficulty-graded Countdown testbed. It reports three phenomena: universal capability overreach (models rarely refuse), specious reasoning as the dominant failure mode, and a capability-behavior misalignment biased toward overconfidence. To address this, the authors propose CaRL, a GRPO-based training method combining reward shaping that ranks refusal above incorrect answers with Hindsight Refusal Augmentation (HRA), which converts every failed rollout into a refusal training example. Experiments on Qwen3-8B/14B report large reductions in futile rate (65.5→7.0% and 78.6→1.0%) while retaining or improving in-distribution accuracy, and a smaller but positive reliability gain on OOD Sudoku and general benchmarks. The central claim is that CaRL achieves capability-aligned behavior without sacrificing utility.","tokens_in":15509,"tokens_out":3773,"duration_ms":43722,"significance":"If the results hold, the paper makes a useful contribution: it provides a clean controlled testbed for studying over-confidence in reasoning models, identifies a concrete failure taxonomy, and proposes a training intervention that appears to substantially reduce hallucinated derivations while keeping task accuracy roughly intact. The authors are appropriately careful to separate reasoning capability from knowledge, and they include OOD and general-task evaluations, which is more than most intervention papers do. The strongest assets are the controlled difficulty design, the explicit refusal-vs-futile metric, and the comparison against Standard RL and RFT baselines, including a cautionary note that RFT's low futile rate on OOD is merely an artifact of near-total refusal. However, the significance is conditional: the central 'without sacrificing utility' claim depends on whether HRA's relabeling of every failed trajectory as refusal-worthy is actually capability-aligned, and the current evidence does not establish that the refused instances are the unsolvable ones. The paper's own OOD accuracy drops and the absence of uncertainty estimates make this a substantive correctness risk rat","major_comments":[{"comment":"HRA relabels every r=-1 trajectory as a refusal demonstration (r=0), based on the premise that 'every failed reasoning attempt implicitly reveals a situation where refusal would have been the appropriate action' (§4.2). Since the Countdown training set is generated with golden answers (Appendix A.1), a substantial fraction of failures are likely inference-time misses on queries the model can solve, not evidence of beyond-capability tasks. Training on these relabeled failures teaches the policy to refuse solvable queries. This is exactly the risk visible in Table 1: the 14B OOD Sudoku accuracy drops from 50.75 to 44.87 while refusal rises to 38.87%, and the abstract's 'without sacrificing utility' therefore rests on an untested assumption. The authors should decompose refused/evaluated instances by whether they are truly unsolvable, and/or ablate HRA using only failures on known-unsolvabl","section":"§4.2, Eq. (1), Algorithm 1"},{"comment":"The capability boundary is operationalized circularly. In §3.4 a query is 'solvable' if the model's own sampled accuracy exceeds 0.5; HRA then uses the same model's failures as ground-truth refusal labels. Consequently, the reported 'calibration' improvements may be self-confirming: the model is trained to treat its own failures as refusal-worthy, then evaluated on whether it refuses its own failures. An external criterion (e.g., provable solvability via the golden answer, or an independent solver's success) is needed to show that refusals concentrate on genuinely unsolvable queries rather than on hard-but-solvable ones. Without this, the headline futile-rate reduction does not establish capability-aligned behavior.","section":"§3.4, §4.2"},{"comment":"The text states that 'vanilla models never refuse (0% across all difficulty levels)' and that baseline models exhibit '0% refusal rate' (§3.2, §4.2). Table 1 reports vanilla refusal rates of 13.92% (Qwen3-8B ID), 7.88% (Qwen3-14B ID), 10.62% (8B OOD), and 18.62% (14B OOD). These are not small rounding effects. The contradiction weakens the motivation for HRA (which is justified by the alleged absence of on-policy refusals) and must be reconciled. If 'refusal' is being counted differently in the two places, that definitional difference should be explicit.","section":"§3.2, §4.2, Table 1"},{"comment":"All headline comparisons are reported as point estimates without error bars, multiple seeds, or significance tests. The paper states that 8 responses per query are averaged, but that does not quantify variation across runs or across query samples. The OOD accuracy drop of 5.9 points for Qwen3-14B, the RFT collapse to 0% accuracy, and the claimed 'preserving performance' are load-bearing; without variance estimates these could be noise. The authors should rerun at least the main comparisons with 3-5 seeds and report means and standard deviations, or provide bootstrap confidence intervals.","section":"Table 1, §5.4, Appendix C.1"}],"minor_comments":[{"comment":"CaRL is spelled 'CARL' once in the introduction; use 'CaRL' consistently.","section":"§1"},{"comment":"Typo: 'Supicous' should be 'Specious'.","section":"Figure 11 caption"},{"comment":"Difficulty levels are inconsistent: §3.2 says N=3 to N=8, Figure 2 shows N=3-8, but §5.1 says only N=4,6,8 are generated and Table 2 uses only those. Clarify which difficulty levels were actually evaluated and why N=3 is excluded from training/evaluation.","section":"§3.1, §3.2, §5.1"},{"comment":"The detection of a 'valid refusal' is not formalized. Please provide the exact refusal-snippet matching procedure (e.g., regex patterns, threshold on confidence) used in both reward computation and evaluation.","section":"§4.1, Eq. (1)"},{"comment":"The notation 'RL unk=0' and 'RL unk=0.5' is hard to read; use subscripts or a table column label such as 'RL_unk=0'.","section":"§5.2, Table 1"},{"comment":"Table 2 reports futile rate and length only for Qwen3-8B. If 14B results differ materially, include them or state why they are omitted.","section":"§5.5, Table 2"},{"comment":"The reliability score weights correct=1.0 and refusal=0.5 arbitrarily. Since the paper's central evaluation metric is not parameter-free, a brief sensitivity analysis (e.g., weight 0.25/0.75) would strengthen the conclusions.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-structured and the intervention is potentially interesting, but the HRA over-refusal concern is real and goes to the heart of the main claim. I would require the authors to add an external solvability label, decompose refusals into correct/incorrect refusals, and provide uncertainty estimates before this can be accepted. The inconsistency between the '0% refusal' text and Table 1's vanilla refusal rates also needs to be resolved; it is not merely cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look: it names a real failure mode — models producing plausible-looking but wrong reasoning instead of refusing — and shows, on a controlled Countdown task, that the failure is systematic across model sizes and that a simple RL recipe (CaRL) cuts futile reasoning from ~65-78% to 1-7% while keeping in-distribution accuracy roughly flat. The diagnosis is solid: the capability-behavior misalignment, the dominance of specious reasoning, and the 6x over-confidence vs over-conservatism are clearly demonstrated. The training idea — reward shaping plus hindsight refusal augmentation — is sensible, and the transfer to Sudoku shows the effect is not just memorized refusals. That is a genuine contribution.\n\nThe soft spots are mostly around overclaiming. The abstract and §5.4 say performance is preserved, but Table 1 shows the 14B model losing 5.9 points on OOD Sudoku (50.75→44.87) with no error bars or seeds, so \"preserved\" is not established. More importantly, HRA (§4.2) relabels every failed trajectory as a refusal, including failures on solvable Countdown instances. That is a structural bias toward over-refusal: it teaches the model that any query it happens to fail is one it should refuse. The 14B OOD accuracy drop and RFT's collapse to 0% accuracy/90%+ refusal are consistent with that risk. The paper needs to either test HRA on failures that are known to be solvable by the model (e.g., only relabel failures on instances above a difficulty threshold) or provide evidence that refused instances are actually unsolvable. Also, §3.2 and §4.2 claim baseline models never refuse (0%), but Table 1 reports vanilla refusal rates of 13.92% and 7.88% on the in-distribution task. That is a factual inconsistency that should be fixed.\n\nNone of this sinks the paper — the effect is large and the mechanism is plausible — but they need to revise before acceptance. As it stands, a reader cannot tell whether the accuracy drop is real noise or a signal of over-refusal. Add seeds, error bars, and an analysis of what fraction of refused queries were actually solvable.\n\nRecommendation: send it to peer review. It deserves referee time. The diagnosis will be useful to people working on RLHF, refusal training, and reasoning models, even if CaRL turns out to need guardrails.","headline":"A solid empirical diagnosis of futile reasoning and a promising RL fix, but the 'preserved performance' claim is weakened by over-refusal on OOD and an internal inconsistency about vanilla refusal rates.","tokens_in":15950,"tokens_out":2665,"would_cite":true,"duration_ms":26344,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models can be trained to refuse unsolvable problems, cutting futile reasoning from 79% to 1% on hard tasks while preserving accuracy.","keywords":["futile reasoning","capability alignment","refusal training","reinforcement learning","reward shaping","hindsight augmentation","LLM reasoning","over-confidence"],"falsifier":"A direct test: take a CaRL-trained model and a set of problems where the model's failures are known to be solvable (e.g., problems it answers correctly under a different seed or with a few extra reasoning steps, or problems verified solvable by a brute-force solver). Measure how often the trained model now refuses these solvable-but-previously-missed problems. If the refusal rate on that set is high (near the failure rate), the method is teaching over-refusal rather than true capability alignment—the Sudoku accuracy drop on 14B (50.75→44.87) is already suggestive of this.","tokens_in":15020,"feed_emoji":"🛑","tokens_out":6917,"duration_ms":58312,"temperature":0.7,"pith_summary":"Large language models, when faced with problems beyond their abilities, often produce long, plausible-sounding but incorrect reasoning instead of admitting they cannot solve them. The paper calls this 'futile reasoning' and shows it is pervasive across model sizes and difficulty levels. It introduces CaRL, a reinforcement-learning method that teaches models when to abort and refuse, using a reward hierarchy that favors refusal over hallucination and a hindsight-augmentation step that turns every failed attempt into a refusal training example. On Countdown and Sudoku tasks, CaRL cuts the futile rate from 65.5% to 7.0% (8B) and 78.6% to 1.0% (14B) while maintaining or improving reliability and shortening outputs. If true, this offers a concrete way to make LLMs safer for high-stakes use.","feed_headline":"LLMs can learn when to quit: futile reasoning down to 1%","feed_subtitle":"Capability-calibrated RL makes 8B and 14B models refuse unsolvable problems while keeping accuracy on solvable ones.","key_machinery":"Futile reasoning is the paper's central object: reasoning traces that look coherent but are mathematically wrong, produced when the model exceeds its capability boundary. The key mechanism is CaRL, which works through two complementary parts. Capability-Calibrated Reward Shaping replaces the usual binary reward with a hierarchy (+1 for correct, 0 for a valid refusal, -1 for incorrect), making refusal strictly better than hallucination. Hindsight Refusal Augmentation addresses the scarcity of refusal examples by taking every incorrect on-policy trajectory and retrofitting it into a refusal response—keeping the reasoning prefix, adding 'Sorry, I don't know', and assigning the refusal reward—so","core_discovery":"The paper's central claim is that 'futile reasoning'—the generation of superficially valid but incorrect derivations on tasks a model cannot solve—is a distinct, dominant failure mode that can be diagnosed and suppressed by aligning model behavior with its actual capability boundary. The authors establish three phenomena: models uniformly overreach (near-zero refusal even when error rates exceed 65%), the prevalent failure style is 'specious reasoning' that escalates with difficulty, and models are miscalibrated by a roughly 6:1 bias toward over-confidence rather than over-conservatism. They then show that their CaRL method—combining a reward function that ranks refusal above hallucination w","pith_inferences":["The core training assumption—that every failed attempt is a refusal-worthy failure—may teach over-refusal on solvable problems the model simply missed; the paper's own OOD Sudoku accuracy drop (50.75 to 44.87 on 14B) hints at this risk, and a dedicated evaluation on 'falsely refused but actually solvable' items would quantify it.","CaRL's reward function treats all refusals as equal (0), so a model could learn to refuse early and cheaply rather than after genuine exploration; ablating on refusal timing or conditioning reward on reasoning effort could distinguish 'knowing when to quit' from 'lazy refusal'.","The findings suggest a general principle for aligning LLMs: capability boundaries can be learned from the model's own errors by relabeling them as refusals, which might be applicable beyond arithmetic puzzles to knowledge-boundary refusal if the hindsight signal can be made reliable.","If scaled, the compute savings from aborting futile reasoning could change the economics of test-time scaling, making reasoning models more practical for budget-constrained deployments."],"forward_implications":["CaRL-trained models should refuse more on unsolvable queries and produce shorter outputs, cutting compute by roughly one-third on hard reasoning tasks.","Reliability, defined as correct answers plus half-credit for refusals, rises by 0.13–0.16 points on 8B and 14B models even when accuracy stays flat, meaning the same model is safer to deploy in high-stakes settings.","The method transfers to out-of-distribution tasks (Sudoku) and to general benchmarks (AIME, GPQA) with less than 2% accuracy change, implying capability-boundary awareness is not just task memorization.","Standard reward-based RL without hindsight augmentation fails to reduce futile reasoning (futile rate stays above 95% on 8B), implying that dense refusal supervision, not reward shaping alone, is the load-bearing ingredient.","Larger models benefit more from CaRL (14B futile rate drops to 1.0% vs 7.0% for 8B), suggesting scale amplifies the ability to internalize refusal boundaries."],"fun_headline_variants":["CaRL trains LLMs to refuse beyond-capability tasks, cutting futile reasoning","Capability-aligned RL: LLMs quit futile reasoning, keep accuracy","LLMs learn to abort specious reasoning with CaRL","Futile reasoning cut to 1% with capability-calibrated training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is stated in Section 4.2: 'every failed reasoning attempt implicitly reveals a situation where refusal would have been the appropriate action.' In other words, the method assumes that whenever the current policy produces a wrong answer, the problem is beyond the model's capability and deserves a refusal. If some failures are merely inference-time misses on problems the model can actually solve, this training signal teaches the model to over-refuse.","fun_headline_variants_meta":{"raw":{"variants":["CaRL trains LLMs to refuse beyond-capability tasks, cutting futile reasoning","Capability-aligned RL: LLMs quit futile reasoning, keep accuracy","LLMs learn to abort specious reasoning with CaRL","Futile reasoning cut to 1% with capability-calibrated training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2732,"prompt_tokens":703,"completion_tokens":2029,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":1951}},"tokens_in":447,"tokens_out":2029,"duration_ms":16569,"temperature":1.0,"reasoning_tokens":1951,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:32:01.931240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: take a CaRL-trained model and a set of problems where the model's failures are known to be solvable (e.g., problems it answers correctly under a different seed or with a few extra reasoning steps, or problems verified solvable by a brute-force solver). Measure how often the trained model now refuses these solvable-but-previously-missed problems. If the refusal rate on that set is high (near the failure rate), the method is teaching over-refusal rather than true capability alignment—the Sudoku accuracy drop on 14B (50.75→44.87) is already suggestive of this.","supporting_citations":[],"review_version":1}