Pith. sign in

REVIEW 3 major objections 5 minor 72 references

Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Asking an LLM eight critical questions about its own draft reasoning raises math and logic scores by about five percent.

desk verdict The argumentative querying idea is fresh, but the key ablation is confounded with repeated sampling, so the paper doesn't yet show that critical questions are what improve LLM reasoning. read the letter →

arxiv 2412.15177 v1 pith:MNT6BDH7 submitted 2024-12-19 cs.AI cs.CL

Federico Castagna , Isabel Sassoon , Simon Parsons This is my paper · ORCID
classification cs.AIcs.CL
keywords criticalquestionsargumentationtheorylargelanguagemodelstest-timecomputeChain-of-ThoughtMT-BenchLLMself-assessmentlogicalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models still make systematic mistakes on logical and mathematical problems. This paper claims that a lightweight, model-agnostic change to how the model is prompted can narrow that gap: before giving a final answer, the model must write out a step-by-step reasoning plan, then answer eight 'critical questions' about whether the plan rests on clear premises, valid connections, and sound conclusions. If enough answers are positive, the model is told to follow the plan strictly; otherwise it starts over. Across five proprietary and open models on the MT-Bench reasoning and math tasks, this Critical-Questions-of-Thought pipeline beat both the raw baseline and a Chain-of-Thought version in 18 of 20 comparisons, with average gains of roughly 4.6 percent on reasoning and 5.4 percent on math. The significance, if the result holds, is that better reasoning can be bought with inference-time self-questioning rather than more training data or larger models.

What carries the argument

The load-bearing component is the fixed battery of eight critical questions, drawn from argumentation theory's account of presumptive, challengeable reasoning and mapped onto the elements of the classical argumentation schema (claim, data, warrant, backing, qualifier, rebuttal). These questions convert the LLM's draft reasoning into an argument that can be challenged, and the Yes/No answers are counted by a checkpoint that decides whether to iterate or accept the plan. The specific thresholds—7/8 positive answers for the first five attempts, 5/8 thereafter, and acceptance after ten total iterations—implement the paper's governing idea that a conclusion stands unless disproved by the critical probe.

What would settle it

Comparing MT-Bench scores under the real Step-2 self-assessment against a condition where the Yes/No answers are replaced by random coin flips would settle it: if scores do not drop, the critical-question checkpoint is not carrying the gain.

Watch

Extended reading notes

Core claim

The paper's central discovery is that argumentation-theoretic critical questions can act as a reasoning gate for LLMs. The pipeline (CQoT) first asks the model to produce a reasoning plan in the form of premises leading to conclusions, without giving the final answer. The model then answers eight Yes/No critical questions targeting the elements of the classical argumentation schema—data, warrant, backing, claim, qualifier, and rebuttal—about its own plan. A verdict of at least seven positive answers (or five, after several failed attempts) allows the plan to proceed; otherwise the model regenerates a plan. Finally, the model must produce its answer by strictly following the accepted plan. Evaluated on the MT-Bench reasoning and math subsets with five LLMs, CQoT clearly won 18 of 20 comparisons against baseline and CoT, tied one, and lost one, and an ablation shows that removing the critical-question stage reduces but does not eliminate the gain.

Load-bearing premise

The load-bearing premise is that the LLM's own 'Yes' or 'No' answers to the eight critical questions are reliable enough to decide whether its reasoning plan is sound; if the model is a poor judge of its own reasoning, the checkpoint that makes CQoT work fails.

Editorial extensions

If this is right

  • Because the pipeline is a prompting wrapper, it can be applied to any LLM without retraining, and the paper's results show open-source models such as Llama 3.1-70b-Instruct and Nemotron-51b-Instruct matching or beating proprietary baselines on several MT-Bench tasks.
  • The ablation result implies that even a simplified pipeline that only forces a premise-conclusion reasoning plan helps over baseline, with the critical questions adding a further measurable improvement.
  • CQoT sits inside the test-time compute paradigm: it trades additional inference time and API calls for higher answer quality, without requiring multiple sampled answers.
  • The authors observe that smaller models, especially below roughly 70 billion parameters, may not follow the instructional protocol reliably enough to benefit fully, suggesting a scale threshold for the method.
  • The pipeline is orthogonal to other enhancement strategies, so it can be combined with prompt engineering, sampling, or other techniques.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit would specialize the eight questions to a domain and compare against the generic set; if the generic gain mostly comes from forcing self-explanation, the question wording should matter less than the structured re-planning.
  • Because the checkpoint trusts the model's own Yes/No self-reports, a direct test that replaces that self-assessment with an independent verifier model (or a human judge on a sample) would show whether accurate self-critique is the mechanism or whether the improvement comes from simply regenerating plans.
  • The paper's binary pass/fail counting could be replaced by a probabilistic or per-question weighted score, which might recover similar gains at lower iteration cost and would clarify how much of the effect depends on the specific 7/8 and 5/8 thresholds.
  • If the effect replicates on contamination-free, harder benchmarks, it would strengthen the conclusion that the gain is about reasoning rather than retrieval of familiar answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Critical-Questions-of-Thought (CQoT), a four-step prompting pipeline that asks an LLM to produce a structured reasoning plan, evaluates that plan against eight argumentation-theoretic critical questions, iterates the plan when the questions are not answered positively, and only then prompts the model to produce its final answer. The authors evaluate CQoT on the MT-Bench Reasoning and Math tasks using five LLMs (Claude Sonnet 3.5, GPT-4o, Gemini 1.5-pro-001, Llama 3.1-70b-Instruct, Nemotron-51b-Instruct) and report that CQoT outperforms both the standard baseline and a Chain-of-Thought variant, with an average improvement of approximately 5%. They also report an ablation study comparing a Step 1+4-only pipeline with the full CQoT pipeline on two open-source models.

Significance. If the central claim holds, CQoT would be a lightweight, model-agnostic test-time-compute method that improves LLM reasoning without fine-tuning and with modest computational overhead. The paper has several strengths: it uses a diverse set of freely available LLMs, shares code and evaluation results publicly, and is transparent about cases where CQoT underperforms CoT (e.g., Nemotron on Reasoning). The connection to Toulmin's argumentation schemes and critical questions is conceptually interesting and could inspire further work on argumentation-based prompting. However, the current evidence does not yet establish that the critical questions themselves are the cause of the observed improvements, because the ablation control is not matched for the number of reasoning attempts, and the evaluation methodology has unresolved reliability concerns.

major comments (3)
  1. [§4.1.3, Table 2] The ablation does not isolate the effect of the critical questions. The Step 1+4 control generates a single reasoning plan, while the full CQoT pipeline may iterate up to ten times (Figure 2 and §3.1). Any resampling strategy would be expected to improve over a single sample, and Section 5 explicitly states that 'the underlying LLM is more likely to output the correct reply if this appears at least once among its potential responses.' To attribute the improvement to the critical questions, the control must match the number of attempts (e.g., repeated independent Step 1+4 runs, or an equal-iteration loop that skips the Step 2/3 checkpoint).
  2. [§3.1 Steps 2–3, §5] The reliability of the Step 3 checkpoint is never measured. The pipeline treats the model's PASS/FAIL answers to the eight critical questions as a signal of plan quality, but the paper reports no comparison of these answers against independent correctness labels, and no statistics on how often the checkpoint accepts a bad plan or rejects a good one. A validation study (e.g., scoring a sample of accepted and rejected plans by human or ground-truth correctness) is needed to show that the argumentative filter, rather than the loop, is doing the work.
  3. [§4.1.4, §4.2] The evaluation rests on a single LLM judge (GPT-4o) with manual corrections, and no human validation or inter-judge agreement is reported. The procedure of prompting until the same score occurs three times is ad hoc, and the manual re-prompting described in the Limitations section could introduce bias. Furthermore, with only 40 questions per model and no statistical significance tests, the reported mean differences of roughly 0.3–0.9 points on a 1–10 scale (Table 1) are not shown to be distinguishable from noise. The authors should report confidence intervals, paired significance tests, or a human-judged subset to substantiate the claimed improvement.
minor comments (5)
  1. [Figure 7] The prompt template for Step 2 contains an unexplained block of garbled text ('sdg sdgd sgasdg ...'); this appears to be a formatting artifact and should be removed.
  2. [§4.1.2] The number of test questions is ambiguous: the text says 40 questions evenly divided between reasoning and math, while MT-Bench consists of 80 queries with two sub-questions each; please clarify whether the 40 refers to 10 queries per category with two turns each, and report per-category sample sizes.
  3. [Section 3] The phrase 'a majority of the CQs are answered positively' contradicts the actual thresholds of 7/8 and later 5/8; 5/8 is a majority, 7/8 is not. Please rephrase to 'a sufficient number'.
  4. [Section 4.3, Table 3] The average improvements quoted in the text (+4.61% reasoning, +5.4% math) do not match the averages shown in Table 3 (+4.74% reasoning Standard, +5.71% math Standard, or +4.48% reasoning CoT, +5.09% math CoT). Please make the text and table consistent.
  5. [Section 5] The statement that CQoT does not produce replies that never occur in the baseline is an important caveat; it deserves to be mentioned in the abstract or conclusions as a qualification of the claimed improvement.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: CQoT's gains are empirical and not entailed by its prompt definitions; the self-citations are motivational, and the repeated-sampling ablation confound is a validity issue rather than a circular reduction.

full rationale

CQoT is an empirical prompting pipeline evaluated on MT-Bench. The central claim, that adding eight critical questions improves LLM reasoning, is not derived from the definitions: the thresholds (7/8, 5/8, ten iterations, five strict iterations) are hand-chosen in Section 3.1 rather than fitted to the test data, and the judge scores come from an external protocol based on Zheng et al. [63]. The Toulmin/Walton notions are used as an inspiration for prompts, not as axioms that entail the measured scores. The authors' self-citations, notably [8] ('Can formal argumentative reasoning enhance LLMs performances?') and the EQRbot papers [5,7], are motivational or related-work only; none is load-bearing, since removing them would not change the evaluation or the reported numbers. Thus no step reduces, by construction, to its own input. A separate, non-circularity validity concern: Section 4.1.3's ablation compares a single-plan condition (Step1+4) with the full CQoT pipeline, which may iterate up to ten times, and Section 5 concedes 'the underlying LLM is more likely to output the correct reply if this appears at least once among its potential responses'. This confounds the causal role of the critical questions with repeated sampling, but it is an experimental-methodology issue, not a circular derivation, and it does not make the reported scores equivalent to the pipeline's own inputs by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about the appropriateness of Toulmin's framework, the reliability of LLM self-assessment, and the trustworthiness of an LLM judge. No free parameters are fitted to the data, but the pipeline thresholds and iteration counts are arbitrary design choices. No new entities are postulated.

free parameters (3)
  • Pass threshold for critical questions (7/8) = 7/8
    The pipeline requires at least 7/8 positive answers to CQs to proceed to the final answer. This threshold is chosen by the authors without empirical justification (Section 3.1 Step 3).
  • Relaxed threshold (5/8) after five iterations = 5/8
    After five iterations, the threshold drops to 5/8. This is an arbitrary relaxation designed to avoid infinite loops.
  • Maximum iterations (5, then 10) = 5 and 10
    The pipeline restarts up to 5 times at the strict threshold, then up to 10 total iterations at the relaxed threshold. These limits are arbitrary.
assumptions (4)
  • domain assumption Toulmin's model of argumentation is an appropriate framework for probing LLM reasoning.
    The paper assumes that the six Toulmin components (data, warrant, backing, qualifier, rebuttal, claim) can serve as a battery of queries for evaluating any presumptive reasoning, including that of LLMs. Stated in Section 3: 'we are not trying to claim that an LLM follows the Toulmin model... but rather that LLMs engage in presumptive reasoning, and so a set of CQs that cover all the elements of Toulmin's schema provide a suitable battery of queries.'
  • domain assumption The LLM's self-assessment in Step 2 is a reliable indicator of the quality of its own reasoning plan.
    The pipeline trusts the model's answers to the eight critical questions to decide whether to iterate (Step 3). If the model cannot accurately judge its own reasoning, the pipeline's checkpoint is meaningless. Invoked in Section 3.1 Step 2 and Step 3.
  • domain assumption GPT-4o is a reliable judge of response quality for reasoning and math tasks.
    The evaluation relies on GPT-4o as the sole judge (Section 4.1.4). The paper cites Zheng et al. for LLM-judge agreement with human experts, but does not validate on this specific task set.
  • domain assumption MT-Bench contamination is either absent or equally affects all compared methods, preserving relative conclusions.
    The paper acknowledges possible contamination (Section 4.1.2) but argues it is a fair comparison. This assumes that any contamination raises baseline, CoT, and CQoT equally, which may not hold if CQoT produces longer or structured outputs that align with reference answers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying." pith.science (2026). https://pith.science/paper/MNT6BDH7

@misc{pith2026241215177,
  author       = {Pith},
  title        = {Pith review of: Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNT6BDH7}},
  note         = {Machine review of arXiv:2412.15177}
}
read the original abstract

Studies have underscored how, regardless of the recent breakthrough and swift advances in AI research, even state-of-the-art Large Language models (LLMs) continue to struggle when performing logical and mathematical reasoning. The results seem to suggest that LLMs still work as (highly advanced) data pattern identifiers, scoring poorly when attempting to generalise and solve reasoning problems the models have never previously seen or that are not close to samples presented in their training data. To address this compelling concern, this paper makes use of the notion of critical questions from the literature on argumentation theory, focusing in particular on Toulmin's model of argumentation. We show that employing these critical questions can improve the reasoning capabilities of LLMs. By probing the rationale behind the models' reasoning process, the LLM can assess whether some logical mistake is occurring and correct it before providing the final reply to the user prompt. The underlying idea is drawn from the gold standard of any valid argumentative procedure: the conclusion is valid if it is entailed by accepted premises. Or, to paraphrase such Aristotelian principle in a real-world approximation, characterised by incomplete information and presumptive logic, the conclusion is valid if not proved otherwise. This approach successfully steers the models' output through a reasoning pipeline, resulting in better performance against the baseline and its Chain-of-Thought (CoT) implementation. To this end, an extensive evaluation of the proposed approach on the MT-Bench Reasoning and Math tasks across a range of LLMs is provided.

Figures

Figures reproduced from arXiv: 2412.15177 by the authors.

Figure 1
Figure 1. Toulmin’s schema: the case of Harry’s nationality. before doing that, we should acknowledge other work in what has a significant sub-field within research on artificial intelligence. One way to consider the kind of reasoning captured by argumentation is as a formal representation that extends classical logic to permit defeasible reasoning. This is important because it seems that one way of tractably handling the com… view at source ↗
Figure 2
Figure 2. The four-step process of the CQoT pipeline. tical than relying on expensive and time-consuming human testing. With this in mind, multiple benchmarks were created, each focusing on evaluating specific models’ skill sets [9]. For example, MT-Bench is a comprehensive multi-turn benchmark that presents 80 challenging queries, divided into two sub-questions each, covering 8 different domains (writing, roleplay, reasoning… view at source ↗
Figure 3
Figure 3. Comparison between the responses given by the baseline Llama 3.1 70b-Instruct (wrong, red coloured) and its CQoT counterpart (correct, green coloured). Notice that the multiple ‘Step’ mentioned in the latter reply do not refer to the CQoT pipeline: it is just how the output has been phrased by the model. sions. We also tested the same baseline models by means of an ablation study. Through a slightly modified version… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Two-step pipeline for the ablation study. 4.1.4 LLM as a Judge Drawing from the research of Zheng et al. [63], where it has been established how state-of-the-art LLMs can achieve an agreement rate on par with human ex￾perts when assessing model responses, we followed a…
Figure 5
Figure 5. Figure 5: Comparison between performance achieved by the baseline model with (CQoT) and without (Standard) the Critical-Questions-of-Thought approach. for GPT-4o, Gemini 1.5-pro-001 and Llama 3.1-70b-Instruct. They are some￾what less impressive for Claude Sonnet 3.5, where CoT a…
Figure 6
Figure 6. Figure 6: Comparison between performance achieved by the baseline model (Standard), its CoT augmented version (CoT) and the Critical-Questions-of-Thought approach (CQoT). The point represents the mean score, and the bars are the standard error. on mathematical questions. As a la…
Figure 7
Figure 7. Figure 7: CQoT pipeline prompts for Steps 1, 2 and 4. Recall that the function of Step 3 is only to check and, if required, reiterate the previous two stages. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Pipeline prompts for ablation study. Steps 1 and 4 present the same input as the respective stages of [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: Input used to augment baseline models with Chain-of-Thought reasoning. consist of 40 questions, each of which is split into two sub-questions: an initial query and a sequent one strictly interconnected with the first. For every sub￾question, the judge needs to use a di…
Figure 10
Figure 10. Figure 10: Prompts leveraged by the LLM judge to score the other models’ responses. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Input adjusting judge evaluation in case it unnecessarily penalises the length of the response. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 41 canonical work pages

  1. [1]

    The surprising effectiveness of test-time training for abstract reasoning, 2024

    Ekin Aky¨ urek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for abstract reasoning, 2024

    work page 2024

  2. [2]

    Claude 3.5 Sonnet

    Anthropic. Claude 3.5 Sonnet. Anthropic blog , 2024. https://www. anthropic.com/news/claude-3-5-sonnet (last accessed 26/11/2024)

    work page 2024

  3. [3]

    Handbook of formal argumentation

    Pietro Baroni, Dov Gabbay, Massimilino Giacomin, and Leendert Van der Torre, editors. Handbook of formal argumentation . College Publications, 2018. 20

    work page 2018

  4. [4]

    Large language mon- keys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R´ e, and Azalia Mirhoseini. Large language mon- keys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

  5. [5]

    Federico Castagna, Alexandra Garton, Peter McBurney, Simon Parsons, Isabel Sassoon, and Elizabeth I. Sklar. EQRbot: A chatbot delivering EQR argument-based explanations. Frontiers in Artificial Intelligence , 6, 2023

    2023

  6. [6]

    Computational argumentation-based chatbots: a survey

    Federico Castagna, Nadin K¨ okciyan, Isabel Sassoon, Simon Parsons, and Elizabeth Sklar. Computational argumentation-based chatbots: a survey. Journal of Artificial Intelligence Research , 80:1271–1310, 2024

    work page 2024

  7. [7]

    Federico Castagna, Simon Parsons, Isabel Sassoon, and Elizabeth I. Sklar. Providing explanations via the EQR argument scheme. In Computational Models of Argument: Proceedings of COMMA 2022 , 2022

    work page 2022

  8. [8]

    Can formal argumentative reasoning enhance LLMs performances?

    Federico Castagna, Isabel Sassoon, and Simon Parsons. Can formal ar- gumentative reasoning enhance LLMs performances? arXiv preprint arXiv:2405.13036, 2024

    work page Pith review arXiv 2024

Show all 72 references
  1. [9]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109 , 2023

    2023 arXiv

  2. [10]

    A simple and provable scaling law for the test-time compute of large language models

    Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. A simple and provable scaling law for the test-time compute of large language models. arXiv preprint arXiv:2411.19477 , 2024

    2024

  3. [11]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas An- gelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 , 2024

    2024 arXiv

  4. [12]

    Arc prize 2024

    Francois Chollet, Mike Knoop, Bryan Landers, Greg Kamradt, Hansueli Jud, Walter Reade, and Addison Howard. Arc prize 2024. https: //kaggle.com/competitions/arc-prize-2024, 2024. Kaggle

    2024

  5. [13]

    On the measure of intelligence, 2019

    Fran¸ cois Chollet. On the measure of intelligence, 2019

    2019

  6. [14]

    Lm vs lm: Detect- ing factual errors via cross examination

    Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm vs lm: Detect- ing factual errors via cross examination. arXiv preprint arXiv:2305.13281 , 2023. 21

    2023 arXiv

  7. [15]

    On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games

    Phan Minh Dung. On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. Artificial intelligence, 77(2):321–357, 1995

    1995

  8. [16]

    Argumentative large language models for explainable and contestable decision-making

    Gabriel Freedman, Adam Dejl, Deniz Gorur, Xiang Yin, Antonio Rago, and Francesca Toni. Argumentative large language models for explainable and contestable decision-making. arXiv preprint arXiv:2405.02079 , 2024

    2024 arXiv

  9. [17]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023

    2023 arXiv

  10. [18]

    Harnessing toul- min’s theory for zero-shot argument explication

    Ankita Gupta, Ethan Zuckerman, and Brendan O’Connor. Harnessing toul- min’s theory for zero-shot argument explication. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 10259–10276, 2024

    2024

  11. [19]

    Folio: Natural language reasoning with first-order logic

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, et al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.00840, 2022

    2022 arXiv

  12. [20]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  13. [21]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  14. [22]

    OpenAI and others seek new path to smarter AI as current methods hit limitations

    Krystal Hu and Anna Tong. OpenAI and others seek new path to smarter AI as current methods hit limitations. Reuters, 2024. https://www.reuters.com/technology/artificial-intelligence/ openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11/ (last acces...

    2024

  15. [23]

    Thinking, fast and slow

    Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux , 2011

    2011

  16. [24]

    How far is video generation from world model: A physical law perspective, 2024

    Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective, 2024

    2024

  17. [25]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199–22213, 2022

    2022

  18. [26]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev- Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887 , 2024. 22

    2024 arXiv

  19. [27]

    F. Lin. An argument-based approach to non-monotonic reasoning. Com- putational Intelligence , 9:254–267, 1993

    1993

  20. [28]

    Lin and Y

    F. Lin and Y. Shoham. Argument systems: a uniform basis for nonmono- tonic reasoning. In Proceedings of the 1st International Conference on Knowledge Representation and Reasoning, pages 245–255, San Mateo, CA,

  21. [29]

    R. P. Loui. Defeat among arguments: a system of defeasible inference. Computational Intelligence , 3(3):100–106, 1987

    1987

  22. [30]

    Why do humans reason? Arguments for an argumentative theory

    Hugo Mercier and Dan Sperber. Why do humans reason? Arguments for an argumentative theory. 34(2):57–74, 2011

    2011

  23. [31]

    OpenAI, Google and Anthropic are struggling to build more advanced AI

    Rachel Metz, Shirin Ghaffary, Dina Bass, and Julia Love. OpenAI, Google and Anthropic are struggling to build more advanced AI. Bloomberg,

  24. [32]

    GSM-Symbolic: Understanding the limi- tations of mathematical reasoning in large language models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-Symbolic: Understanding the limi- tations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024

    arXiv

  25. [33]

    R. C. Moore. Semantical considerations on nonmonotonic logic. Artificial Intelligence, 25:75–94, 1985

    1985

  26. [34]

    Arithmetic without algorithms: Language models solve math with a bag of heuristics

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272 , 2024

    2024 arXiv

  27. [35]

    Introducing ChatGPT

    OpenAI. Introducing ChatGPT. OpenAI blog, 2022. https://openai. com/index/chatgpt/ (last accessed 26/11/2024)

    2022

  28. [36]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  29. [37]

    Learning to reason with LLMs

    OpenAI. Learning to reason with LLMs. OpenAI blog , 2024. https: //openai.com/index/learning-to-reason-with-llms/ (last accessed 6/12/2024)

    2024

  30. [38]

    OpenAI Shifts Strategy as Rate of ‘GPT’ AI Improvements Slows

    Stephanie Palazzolo, Erin Woo, and Amir Efrati. OpenAI Shifts Strategy as Rate of ‘GPT’ AI Improvements Slows. The In- formation, 2024. https://www.theinformation.com/articles/ openai-shifts-strategy-as-rate-of-gpt-ai-improvements-slows (last accessed 12/11/2024)

    2024

  31. [39]

    J. L. Pollock. Defeasible reasoning. Cognitive Science, 11:481–518, 1987

    1987

  32. [40]

    J. L. Pollock. How to reason defeasibly. Artificial Intelligence , 57:1–42, 1992. 23

    1992

  33. [41]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bow- man. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  34. [42]

    R. Reiter. A logic for default reasoning. Artificial Intelligence, 13:81–132, 1980

    1980

  35. [43]

    A systematic survey of prompt engineering in large language models: Techniques and applications

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024

    2024 arXiv

  36. [44]

    Aristotle’s Logic

    Robin Smith. Aristotle’s Logic. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, Winter 2022 edition, 2022

    2022

  37. [45]

    Scaling llm test-time compute optimally can be more effective than scaling model pa- rameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model pa- rameters. arXiv preprint arXiv:2408.03314 , 2024

    2024 arXiv

  38. [46]

    Functional benchmarks for ro- bust evaluation of reasoning performance, and the reasoning gap

    Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Phili- pose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for ro- bust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402.19450, 2024

    2024 arXiv

  39. [47]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨ arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022

    arXiv

  40. [48]

    Judgebench: A benchmark for evaluating llm-based judges

    Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Ale- jandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784, 2024

    2024 arXiv

  41. [49]

    DeepSeek-R1-Lite-Preview

    DeepSeek Team. DeepSeek-R1-Lite-Preview. DeepSeek website , 2024. https://chat.deepseek.com/ (last accessed 6/12/2024)

    2024

  42. [50]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024

    2024 arXiv

  43. [51]

    QwQ: Reflect deeply on the boundaries of the unknown, November 2024

    Qwen Team. QwQ: Reflect deeply on the boundaries of the unknown, November 2024

    2024

  44. [52]

    S. Toulmin. The Uses of Argument . Cambridge University Press, Cam- bridge, England, 1958

    1958

  45. [53]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 24

    2017

  46. [54]

    Evaluating arguments based on toulmin’s scheme

    Bart Verheij. Evaluating arguments based on toulmin’s scheme. Argumen- tation, 19:347–371, 2005

    2005

  47. [55]

    Reed, and F

    D Walton, C. Reed, and F. Macagno. Argumentation Schemes. Cambridge University Press, Cambridge, UK, 2008

    2008

  48. [56]

    Argumentation schemes for presumptive reasoning

    Douglas N Walton. Argumentation schemes for presumptive reasoning . Psychology Press, 1996

    1996

  49. [57]

    Mmlu-pro: A more robust and challenging multi-task language understand- ing benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understand- ing benchmark. arXiv preprint arXiv:2406.01574 , 2024

    2024 arXiv

  50. [58]

    Chain-of-thought prompting elic- its reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elic- its reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    2022

  51. [59]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Sid- dhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314 , 2024

    2024 arXiv

  52. [60]

    Enhancing llm reasoning via critique models with test-time and training-time supervision

    Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. Enhancing llm reasoning via critique models with test-time and training-time supervision. arXiv preprint arXiv:2411.16579 , 2024

    2024 arXiv

  53. [61]

    Schemes, critical questions, and complete argument evaluation

    Shiyang Yu and Frank Zenker. Schemes, critical questions, and complete argument evaluation. Argumentation, 34(4):469–498, 2020

    2020

  54. [62]

    Cumulative reasoning with large language models

    Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models. arXiv preprint arXiv:2308.04371, 2023

    2023 arXiv

  55. [63]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595–46623, 2023. 25 Appendi...

    2023

  56. [65]

    Does the reasoning process start with clearly defined premises?

  57. [66]

    Are the premises supported by evidence or accepted facts?

  58. [67]

    Does the reasoning process use logical connections between premises and conclusions?

  59. [68]

    Are the logical connections used in the reasoning process valid?

  60. [69]

    Does the reasoning process avoid fallacies or logical errors?

  61. [70]

    Is the conclusion logically derived from the premises?

  62. [71]

    Is the reasoning process consistent with established knowledge or principles?

  63. [72]

    Let’s think step by step

    Does the reasoning process lead to a conclusion that is plausible and reasonable? </End System Instruction> <User Prompt>{same input as STEP 1}</End User Prompt> <Reasoning Steps>{generated from STEP 1}</End Reasoning Steps> Remember: you MUST reply to EACH question. The repli...

    2024

  64. [2024]

    https://www.bloomberg.com/news/articles/2024-11-13/ openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai (last accessed 20/11/2024)

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.