{"id":"27c13e68-a685-46a2-a662-0b0dc73d9013","arxiv_id":"2412.15177","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CQoT, a pipeline that uses argumentation-theoretic critical questions to check LLM reasoning plans, improves MT-Bench reasoning and math scores by roughly 5% over baseline and CoT prompting.","lead":"This paper introduces a prompting method called Critical-Questions-of-Thought (CQoT) that asks a large language model to check its own reasoning plan against eight critical questions before giving a final answer. In tests on the MT-Bench reasoning and math tasks across five models, CQoT improved scores by about 5% on average compared to the baseline and Chain-of-Thought prompting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ablation supporting CQoT confounds critical questions with repeated sampling: the full pipeline may iterate up to 10 times while its Step1+4 control iterates once, so the claimed causal role of the critical questions is not established.","rationale":"I read the paper in good faith. The method is simple and reproducible, and the reported improvements are plausible, but the causal claim that critical questions drive the improvement is not yet supported. The reader's weakest assumption was the reliability of self-assessment in Step 2; I agree this is the pivotal mechanism, but the deeper issue is that the experimental design cannot separate the self-assessment filter from the extra iterations. The ablation compares one plan (Step1+4) with up to ten plans (full CQoT), so even a worthless checklist would likely appear beneficial. The paper's own Discussion admits that CQoT is more likely to output a correct reply if it appears among the model's potential responses, which is exactly the repeated-sampling explanation. My proposed test would settle this by matching iteration counts. If the control matches CQoT, the paper's central contribution collapses to repeated sampling; if CQoT remains significantly better, the CQs add genuine filtering value. I do not think this changes the reader's conditional verdict—it sharpens the conditions under which the claim should be accepted—so verdict_should_be is UNCHANGED.","tokens_in":18790,"tokens_out":9185,"duration_ms":85874,"concrete_test":"Using the released code and prompts, run an iteration-matched control on Llama 3.1-70b and Nemotron-51b: for each test item, first run CQoT and log the number of Step 1 iterations; then run a control that repeats Step 1 that same number of times and sends a randomly selected (or the last) plan to Step 4, omitting Steps 2-3. Score both with the same judge (or, better, with two blind judges) and test the difference with a paired permutation test. If the control is statistically indistinguishable from CQoT, the critical questions are not responsible for the improvement; if CQoT is still significantly better, the self-assessment concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1.3 reports the only evidence for the causal role of critical questions: Table 2 compares Step1+4 (a single reasoning plan, no checkpoint) with the full CQoT pipeline. But the two conditions are not matched for the number of attempts: per Section 3.1, CQoT can regenerate the plan up to ten times before Step 4, while the Step1+4 control generates it exactly once. Any resampling strategy—even one that ignores the CQs entirely—would be expected to improve over a single sample by giving the model extra chances to hit a correct plan. Section 5 even concedes this mechanism: 'the underlying LLM is more likely to output the correct reply if this appears at least once among its potential responses.' Consequently, the observed CQoT > Step1+4 gap in Table 2 cannot be attributed to the critical questions; it may be entirely due to repeated sampling. The Step 3 checkpoint's reliability is never measured: there is no report of how often PASS/FAIL matches independent correctness. Because the novelty of CQoT over ordinary test-time compute lies precisely in the argumentative filter, this confound is load-bearing for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Critical-Questions-of-Thought (CQoT), a four-step prompting pipeline that asks an LLM to produce a structured reasoning plan, evaluates that plan against eight argumentation-theoretic critical questions, iterates the plan when the questions are not answered positively, and only then prompts the model to produce its final answer. The authors evaluate CQoT on the MT-Bench Reasoning and Math tasks using five LLMs (Claude Sonnet 3.5, GPT-4o, Gemini 1.5-pro-001, Llama 3.1-70b-Instruct, Nemotron-51b-Instruct) and report that CQoT outperforms both the standard baseline and a Chain-of-Thought variant, with an average improvement of approximately 5%. They also report an ablation study comparing a Step 1+4-only pipeline with the full CQoT pipeline on two open-source models.","tokens_in":19016,"tokens_out":5286,"duration_ms":34119,"significance":"If the central claim holds, CQoT would be a lightweight, model-agnostic test-time-compute method that improves LLM reasoning without fine-tuning and with modest computational overhead. The paper has several strengths: it uses a diverse set of freely available LLMs, shares code and evaluation results publicly, and is transparent about cases where CQoT underperforms CoT (e.g., Nemotron on Reasoning). The connection to Toulmin's argumentation schemes and critical questions is conceptually interesting and could inspire further work on argumentation-based prompting. However, the current evidence does not yet establish that the critical questions themselves are the cause of the observed improvements, because the ablation control is not matched for the number of reasoning attempts, and the evaluation methodology has unresolved reliability concerns.","major_comments":[{"comment":"The ablation does not isolate the effect of the critical questions. The Step 1+4 control generates a single reasoning plan, while the full CQoT pipeline may iterate up to ten times (Figure 2 and §3.1). Any resampling strategy would be expected to improve over a single sample, and Section 5 explicitly states that 'the underlying LLM is more likely to output the correct reply if this appears at least once among its potential responses.' To attribute the improvement to the critical questions, the control must match the number of attempts (e.g., repeated independent Step 1+4 runs, or an equal-iteration loop that skips the Step 2/3 checkpoint).","section":"§4.1.3, Table 2"},{"comment":"The reliability of the Step 3 checkpoint is never measured. The pipeline treats the model's PASS/FAIL answers to the eight critical questions as a signal of plan quality, but the paper reports no comparison of these answers against independent correctness labels, and no statistics on how often the checkpoint accepts a bad plan or rejects a good one. A validation study (e.g., scoring a sample of accepted and rejected plans by human or ground-truth correctness) is needed to show that the argumentative filter, rather than the loop, is doing the work.","section":"§3.1 Steps 2–3, §5"},{"comment":"The evaluation rests on a single LLM judge (GPT-4o) with manual corrections, and no human validation or inter-judge agreement is reported. The procedure of prompting until the same score occurs three times is ad hoc, and the manual re-prompting described in the Limitations section could introduce bias. Furthermore, with only 40 questions per model and no statistical significance tests, the reported mean differences of roughly 0.3–0.9 points on a 1–10 scale (Table 1) are not shown to be distinguishable from noise. The authors should report confidence intervals, paired significance tests, or a human-judged subset to substantiate the claimed improvement.","section":"§4.1.4, §4.2"}],"minor_comments":[{"comment":"The prompt template for Step 2 contains an unexplained block of garbled text ('sdg sdgd sgasdg ...'); this appears to be a formatting artifact and should be removed.","section":"Figure 7"},{"comment":"The number of test questions is ambiguous: the text says 40 questions evenly divided between reasoning and math, while MT-Bench consists of 80 queries with two sub-questions each; please clarify whether the 40 refers to 10 queries per category with two turns each, and report per-category sample sizes.","section":"§4.1.2"},{"comment":"The phrase 'a majority of the CQs are answered positively' contradicts the actual thresholds of 7/8 and later 5/8; 5/8 is a majority, 7/8 is not. Please rephrase to 'a sufficient number'.","section":"Section 3"},{"comment":"The average improvements quoted in the text (+4.61% reasoning, +5.4% math) do not match the averages shown in Table 3 (+4.74% reasoning Standard, +5.71% math Standard, or +4.48% reasoning CoT, +5.09% math CoT). Please make the text and table consistent.","section":"Section 4.3, Table 3"},{"comment":"The statement that CQoT does not produce replies that never occur in the baseline is an important caveat; it deserves to be mentioned in the abstract or conclusions as a qualification of the claimed improvement.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the open-source release is a plus, but the ablation confound is load-bearing for the claim that critical questions themselves improve reasoning. The fix is within scope: add a repeated-sampling control matched for iterations, and strengthen the evaluation with statistical reporting and at least a small human-judged validation. I therefore recommend major revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of CQoT. The idea—use Toulmin-style critical questions to probe an LLM's reasoning plan and iterate until the plan passes—is genuinely new and worth a look. The paper is cleanly written, tests five models, and ships code and data. That's real work, and the average ~5% improvement over CoT/baseline is plausible as a test-time-compute effect.\n\nBut the stress-test note is right, and it's the crux: the ablation (Section 4.1.3, Table 2) compares one-shot Step1+4 against the full CQoT pipeline, which can regenerate the plan up to ten times. The control is not matched for sampling budget. The authors even concede in Section 5 that CQoT mainly helps because the correct reply appears somewhere in the sampled responses. So the observed CQoT > Step1+4 gap cannot be attributed to the critical questions; it could just be best-of-N sampling. The checkpoint's accuracy—how often PASS/FAIL matches independent correctness—is never measured, so there's no evidence the filter is doing anything.\n\nThat's not a small flaw; it undermines the paper's central claim that the argumentative querying is what drives the improvement. The method might still be useful as an iterative self-check prompt, but the specific contribution of the CQs is unestablished.\n\nBeyond that, the evaluation is thin: 40 questions, one LLM judge (GPT-4o) with manual corrections, no significance tests, and a decision rule (7/8 then 5/8, max 10 iterations) that looks hand-tuned. The contamination point they make is fair enough, but it doesn't fix the sampling confound.\n\nWho gets value? Practitioners looking for prompt-engineering tricks might find this worth prototyping, and the paper is a good teaching example of why an ablation must match compute budget. But as a scientific claim about critical questions, it needs a proper control: e.g., compare against resampling the same number of times without filtering, and measure checkpoint reliability against human judgment.\n\nMy recommendation: send it to peer review, but with a request for major revision. The novelty is real, the writing is honest, and the approach is testable. The authors just haven't yet shown that their specific mechanism matters.","headline":"The argumentative querying idea is fresh, but the key ablation is confounded with repeated sampling, so the paper doesn't yet show that critical questions are what improve LLM reasoning.","tokens_in":19573,"tokens_out":2750,"would_cite":false,"duration_ms":26104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Asking an LLM eight critical questions about its own draft reasoning raises math and logic scores by about five percent.","keywords":["critical questions","argumentation theory","large language models","test-time compute","Chain-of-Thought","MT-Bench","LLM self-assessment","logical reasoning"],"falsifier":"Comparing MT-Bench scores under the real Step-2 self-assessment against a condition where the Yes/No answers are replaced by random coin flips would settle it: if scores do not drop, the critical-question checkpoint is not carrying the gain.","tokens_in":18559,"feed_emoji":"🧠","tokens_out":13102,"duration_ms":107295,"temperature":0.7,"pith_summary":"Large language models still make systematic mistakes on logical and mathematical problems. This paper claims that a lightweight, model-agnostic change to how the model is prompted can narrow that gap: before giving a final answer, the model must write out a step-by-step reasoning plan, then answer eight 'critical questions' about whether the plan rests on clear premises, valid connections, and sound conclusions. If enough answers are positive, the model is told to follow the plan strictly; otherwise it starts over. Across five proprietary and open models on the MT-Bench reasoning and math tasks, this Critical-Questions-of-Thought pipeline beat both the raw baseline and a Chain-of-Thought version in 18 of 20 comparisons, with average gains of roughly 4.6 percent on reasoning and 5.4 percent on math. The significance, if the result holds, is that better reasoning can be bought with inference-time self-questioning rather than more training data or larger models.","feed_headline":"Eight critical questions lift AI models' math and logic scores by 5%","feed_subtitle":"A prompt-only self-checking loop beats plain output and Chain-of-Thought across five AI models on MT-Bench reasoning and math.","key_machinery":"The load-bearing component is the fixed battery of eight critical questions, drawn from argumentation theory's account of presumptive, challengeable reasoning and mapped onto the elements of the classical argumentation schema (claim, data, warrant, backing, qualifier, rebuttal). These questions convert the LLM's draft reasoning into an argument that can be challenged, and the Yes/No answers are counted by a checkpoint that decides whether to iterate or accept the plan. The specific thresholds—7/8 positive answers for the first five attempts, 5/8 thereafter, and acceptance after ten total iterations—implement the paper's governing idea that a conclusion stands unless disproved by the critical probe.","core_discovery":"The paper's central discovery is that argumentation-theoretic critical questions can act as a reasoning gate for LLMs. The pipeline (CQoT) first asks the model to produce a reasoning plan in the form of premises leading to conclusions, without giving the final answer. The model then answers eight Yes/No critical questions targeting the elements of the classical argumentation schema—data, warrant, backing, claim, qualifier, and rebuttal—about its own plan. A verdict of at least seven positive answers (or five, after several failed attempts) allows the plan to proceed; otherwise the model regenerates a plan. Finally, the model must produce its answer by strictly following the accepted plan. Evaluated on the MT-Bench reasoning and math subsets with five LLMs, CQoT clearly won 18 of 20 comparisons against baseline and CoT, tied one, and lost one, and an ablation shows that removing the critical-question stage reduces but does not eliminate the gain.","pith_inferences":["A natural extension the paper leaves implicit would specialize the eight questions to a domain and compare against the generic set; if the generic gain mostly comes from forcing self-explanation, the question wording should matter less than the structured re-planning.","Because the checkpoint trusts the model's own Yes/No self-reports, a direct test that replaces that self-assessment with an independent verifier model (or a human judge on a sample) would show whether accurate self-critique is the mechanism or whether the improvement comes from simply regenerating plans.","The paper's binary pass/fail counting could be replaced by a probabilistic or per-question weighted score, which might recover similar gains at lower iteration cost and would clarify how much of the effect depends on the specific 7/8 and 5/8 thresholds.","If the effect replicates on contamination-free, harder benchmarks, it would strengthen the conclusion that the gain is about reasoning rather than retrieval of familiar answers."],"forward_implications":["Because the pipeline is a prompting wrapper, it can be applied to any LLM without retraining, and the paper's results show open-source models such as Llama 3.1-70b-Instruct and Nemotron-51b-Instruct matching or beating proprietary baselines on several MT-Bench tasks.","The ablation result implies that even a simplified pipeline that only forces a premise-conclusion reasoning plan helps over baseline, with the critical questions adding a further measurable improvement.","CQoT sits inside the test-time compute paradigm: it trades additional inference time and API calls for higher answer quality, without requiring multiple sampled answers.","The authors observe that smaller models, especially below roughly 70 billion parameters, may not follow the instructional protocol reliably enough to benefit fully, suggesting a scale threshold for the method.","The pipeline is orthogonal to other enhancement strategies, so it can be combined with prompt engineering, sampling, or other techniques."],"supporting_citations":[{"why":"Primary source of the six-component argument schema that the eight critical questions are designed to interrogate.","marker":"[52]"},{"why":"Introduces argument schemes and critical questions, the notion the pipeline adapts into an LLM self-assessment battery.","marker":"[56, 55]"},{"why":"Defines Chain-of-Thought prompting, the main baseline technique the evaluation compares against.","marker":"[58]"},{"why":"Supplies the zero-shot 'Let's think step by step' prompting variant used as the CoT condition.","marker":"[25]"},{"why":"Contributes the MT-Bench benchmark and the LLM-as-a-judge scoring protocol that the evaluation builds on.","marker":"[63]"},{"why":"Provides the test-time compute scaling-law argument that motivates giving the model extra reasoning iterations.","marker":"[10]"},{"why":"Used in the discussion to frame why CQoT tends to surface answers the baseline could also produce.","marker":"[45]"}],"fun_headline_variants":["Critical questions sharpen LLM math and logic reasoning","Argumentative self-questioning boosts AI reasoning accuracy","Self-critical AI reasoning improves on math and logic tasks","Toulmin's questions guide LLMs to better reasoning","Probing AI's own reasoning with argumentation theory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's own 'Yes' or 'No' answers to the eight critical questions are reliable enough to decide whether its reasoning plan is sound; if the model is a poor judge of its own reasoning, the checkpoint that makes CQoT work fails.","fun_headline_variants_meta":{"raw":{"variants":["Critical questions sharpen LLM math and logic reasoning","Argumentative self-questioning boosts AI reasoning accuracy","Self-critical AI reasoning improves on math and logic tasks","Toulmin's questions guide LLMs to better reasoning","Probing AI's own reasoning with argumentation theory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1501,"prompt_tokens":1019,"completion_tokens":482,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":406}},"tokens_in":635,"tokens_out":482,"duration_ms":4779,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:32:37.711248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Comparing MT-Bench scores under the real Step-2 self-assessment against a condition where the Yes/No answers are replaced by random coin flips would settle it: if scores do not drop, the critical-question checkpoint is not carrying the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Primary source of the six-component argument schema that the eight critical questions are designed to interrogate."},{"cited_title":"Chain-of-thought prompting elic- its reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Defines Chain-of-Thought prompting, the main baseline technique the evaluation compares against."},{"cited_title":"Judging llm-as-a-judge with mt-bench and chatbot arena","cited_arxiv_id":null,"evidence_quote":"Contributes the MT-Bench benchmark and the LLM-as-a-judge scoring protocol that the evaluation builds on."}],"review_version":1}