Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

Logical Reasoning in Large Language Models: A Survey

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey argues that logical reasoning in large language models is a distinct field from chain-of-thought heuristics and maps it through four paradigms: deductive, inductive, abductive, and analogical.

desk verdict Useful survey with a real formal-logic angle, but the inclusion boundary and Table 1 need work before it can claim comprehensiveness. read the letter →

arxiv 2502.09100 v1 pith:EU7IKZJM submitted 2025-02-13 cs.AI cs.CL

Hanmeng Liu , Zhizhang Fu , Mengru Ding , Ruoxi Ning , Chaoli Zhang , Xiaozhang Liu , Yue Zhang This is my paper
classification cs.AIcs.CL
keywords logicalreasoninglargelanguagemodelsdeductiveinductiveabductiveanalogicalneuro-symbolicbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey works to establish that logical reasoning in large language models deserves treatment as its own research area, defined by formal and symbolic logic rather than by heuristic prompting techniques such as chain-of-thought. It organizes the field into four reasoning paradigms—deductive, inductive, abductive, and analogical—and sorts benchmarks and improvement methods into categories. The result is a map of what has been tested, where models succeed and fail, and which strategies (data tuning, reinforcement learning, decoding methods, and neuro-symbolic integration) are supposed to help. The survey concludes that heuristic performance is often strong but rigorous logical inference remains unreliable, especially under rephrasing and out-of-distribution inputs.

What carries the argument

The organizing device is a two-dimensional map: the four reasoning paradigms (deductive, inductive, abductive, analogical) crossed with a benchmark typology (rule-based, expert-designed, exam-based) and an enhancement-method typology (data-centric, model-centric, external-knowledge, neuro-symbolic). The paper also formalizes each enhancement family as an optimization objective: data-centric tuning as $D^* = \arg\max_D R(M_D)$, model-centric optimization as $(\theta^*, S^*) = \arg\max_{\theta,S} R(M_\theta, S)$, and neuro-symbolic coupling as $(M^*, P^*)$ in which the LLM maps natural language to a formal language $z = M(x)$ and the solver produces $y = P(z)$. These equations make explicit what each strategy is supposed to optimize, and the taxonomy is what organizes the survey's evidence.

What would settle it

A systematic literature search with clear inclusion criteria that locates a substantial cluster of LLM logical-reasoning work outside the four paradigms—say modal, deontic, or temporal logic—or that shows researchers routinely treat chain-of-thought as logical reasoning would undercut the survey's central claim that a logic-focused review fills a real gap.

Watch

Extended reading notes

Core claim

The paper's central claim is that logical reasoning in LLMs is a distinct object of study, separate from general reasoning surveys, because the field's core is formal and symbolic inference rather than plausible text generation. On its reading, the state of the art is mixed: models score well on many benchmarks across all four paradigms, but fail on rephrased, out-of-distribution, or extended reasoning tasks, indicating reliance on surface-level statistical patterns. The survey argues that progress comes from three complementary levers: better data (expert-curated, synthetic, and LLM-distilled), better models (instruction tuning and reinforcement learning), and better inference-time machinery (constrained decoding, external solvers, and neuro-symbolic pipelines). It identifies evaluation as the open frontier, since current accuracy-based metrics conflate reasoning with pattern recognition and do not test consistency, soundness, or generalization.

Load-bearing premise

The map is only as trustworthy as the set of papers the authors chose to include, and the survey reports no systematic search or inclusion criteria that would prove the four-paradigm taxonomy covers the field.

Editorial extensions

If this is right

  • Benchmark design should separate logical competence from pattern recognition by adding perturbed rephrasings, negated premises, and swapped quantifiers to evaluation suites.
  • Improvement methods should be compared across tasks rather than on single benchmarks, since the survey's evidence shows that gains from instruction tuning and reinforcement learning are often task-bound.
  • Neuro-symbolic pipelines that translate natural language into formal logic and hand off to solvers or rule-guided LLM chains offer a route to verifiable reasoning, not just higher multiple-choice scores.
  • Evaluation metrics should move from accuracy alone toward consistency and soundness, because current multiple-choice formats conflate reasoning ability with memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the survey's evidence repeatedly ties reasoning failures to paraphrase sensitivity, one testable prediction follows—models trained on logically annotated data should survive rephrasing better than models trained on chain-of-thought traces alone.
  • Editorial extension: the four-paradigm taxonomy omits modal, deontic, temporal, and defeasible logics; checking whether recent LLM work in those areas fits the map would test the taxonomy's completeness.
  • Editorial extension: the survey draws often on benchmarks and systems developed by its own authors, so an independent meta-analysis with transparent inclusion criteria would be the natural next verification of the field map.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper is a survey of logical reasoning in large language models (LLMs), explicitly targeting formal and symbolic logic-based reasoning rather than general heuristic strategies such as chain-of-thought. The survey organizes the field into four reasoning paradigms (deductive, inductive, abductive, analogical), reviews existing benchmarks and datasets, summarizes evaluations of LLM capabilities across these paradigms, and surveys enhancement methods including data-centric tuning, reinforcement learning, inference-time decoding, external knowledge utilization, and neuro-symbolic approaches. It concludes with a discussion of open challenges and future directions. The central claim is that it fills a gap left by prior surveys that conflate logical reasoning with general-purpose heuristics.

Significance. If the survey's scope were carefully enforced, it would be a useful reference for researchers working at the intersection of formal logic and LLMs. It consolidates a dispersed literature, offers a structured taxonomy, and points to important benchmarks (FOLIO, ProofWriter, LogiQA, RuleTaker) and neuro-symbolic methods (LINC, Logic-LM, SymbCoT). The paper does not present new experimental results, but it attempts to synthesize and categorize a fast-moving area. The practical value, however, depends on whether the inclusion boundary between 'logical reasoning' and 'general heuristics' is defensible and whether the factual inventory in Table 1 is reliable. Those conditions are not currently met.

major comments (4)
  1. [§1 and §5.2] The survey's stated scope is formal and symbolic logic-based reasoning rather than general heuristic approaches, but §5.2 (Inference-Time Decoding) lists methods that do not use formal languages, theorem provers, or explicit logical calculi. In particular, Maieutic Prompting, Graph of Thought, Selection-Inference, and DetermLR are general inference-time scaffolding or heuristic methods, not symbolic-logic methods in the sense of LINC or Logic-LM. If these methods are considered logical-reasoning enhancements, the survey must explicitly justify why they fall under the formal/symbolic boundary; if they are not, their inclusion weakens the claim of a distinct focus. This is load-bearing because the survey's stated contribution is precisely to separate logical reasoning from general heuristics.
  2. [Table 1 and §3.1] Table 1 contains factual inconsistencies that undermine the reliability of the benchmark inventory. ProofWriter is labeled 'Exam-based' with size '—', but §3.1 describes it as extending RuleTaker with closed-world and open-world assumptions, i.e., a rule-generated dataset. LogicNLI is labeled 'Exam-based', yet §3.1 states it 'contains 30K entries generated using logical rules.' LogicBench is labeled 'Rule-based', but §3.1 says it is 'GPT-3-generated.' Additionally, GSM is listed as 'Exam-based' with 19K entries, but GSM8K is a collection of grade-school math word problems and GSM-PLUS is a robustness perturbation set; neither is a logical reasoning exam. These errors require correction and verification of all entries in the table.
  3. [§1 (methodology)] The paper claims to be 'a comprehensive review' but provides no systematic literature search strategy, inclusion criteria, or audit trail. Without such a methodology, the representativeness of the surveyed corpus cannot be assessed. This concern is exacerbated by the prominence of the authors' own prior work (ConTRoL, LogiQA, GLoRE, LogiCoT, Logic Agent) in the benchmark and enhancement sections, which raises the risk that the selection is shaped by the authors' research interests rather than by an objective coverage criterion. A short methodology subsection or appendix stating the search dates, databases, and inclusion/exclusion rules is needed to substantiate the comprehensiveness claim.
  4. [§5.2 (Reinforcement Learning)] The statement that DeepSeek-R1 'represents a potential paradigm shift in logical reasoning optimization' is an evaluative claim that is not supported by evidence presented in the survey. DeepSeek-R1 shows strong performance on general reasoning benchmarks, but the paper provides no demonstration that it specifically advances formal or symbolic logical reasoning (e.g., on FOLIO, ProofWriter, or similar logic-focused benchmarks). The claim should be tempered to describe observed performance gains and to note that questions about logical validity and consistency remain open.
minor comments (4)
  1. [Figure 1] Figure 1 contains a typo: 'RulteTaker' should be 'RuleTaker'; also, capitalization is inconsistent for 'Deepseek-R1-Zero' versus 'DeepSeek-R1'. The figure would benefit from a proofread pass.
  2. [References] Reference [liu et al., 2023d] is missing capitalization ('Liu et al.'); check that all citation keys and reference entries are spelled consistently, including 'Logic-of-Thoughts' versus 'Logic-of-thoughts' in the text.
  3. [§3.2] The paragraph on 'GSM' is vague; it should clearly state that it covers GSM8K and GSM-PLUS and explain why these math word problems are categorized under logical reasoning, since they are not typically framed as logic benchmarks.
  4. [§6] In the Discussion, the sentence 'Data-driven methods (e.g., instruction tuning on LogicBench [Parmar et al., 2024])' cites the 2024 version of LogicBench, while the dataset is primarily introduced in the 2023 reference [Parmar et al., 2023]; please clarify the relationship between the two references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey makes no derived predictions, and its self-citations are literature pointers rather than load-bearing premises.

full rationale

This paper is a literature survey, not a derivation. It reports no fitted parameters, no equations that are solved and then renamed as predictions, and no first-principles result whose conclusion is equivalent to an input by construction. The central claim—that prior surveys conflate logical reasoning with general heuristics and that this survey focuses on formal and symbolic logic—is a framing assertion supported by citation of external surveys (Section 1); it is not derived from the authors' own results. The authors do cite their own benchmarks and methods (ConTRoL, LogiQA, GLoRE, LogiCoT, Logic Agent) in the inventory, but those citations are descriptive pointers to existing work, and the survey's organization does not reduce to them. No uniqueness theorem, ansatz, or formal definition is imported from the authors' prior work to force a conclusion. The inclusion boundary is debatable—for example, the placement of heuristic decoding methods like GoT, Maieutic Prompting, Logic-of-Thoughts, and DetermLR under logical-reasoning enhancements in Section 5.2 is not fully reconciled with the stated focus on formal/symbolic logic—and the absence of a reported systematic search methodology weakens the 'comprehensive' label. There are also internal inconsistencies such as Table 1 labeling ProofWriter and LogicNLI as Exam-based even though Section 3.1 describes them as rule-generated. These are correctness and scope concerns, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey rests on framing and coverage assumptions rather than mathematical axioms. Its claims about model behavior are borrowed from the cited papers, and its taxonomy and scope decisions are asserted, not derived. No free parameters or invented entities are introduced.

assumptions (3)
  • domain assumption The four-way taxonomy (deductive, inductive, abductive, analogical) is the correct organizing framework for logical reasoning in LLMs
    Assumed in Section 2.2 without justification; the entire survey structure depends on it.
  • domain assumption The cited papers accurately report their results and are representative of the field
    The survey's claims about model capabilities and method effectiveness are inherited from the cited works; no independent verification is provided.
  • domain assumption Formal and symbolic logic is meaningfully separable from general heuristic reasoning such as chain-of-thought
    Used to justify the survey's scope in the introduction and abstract; it is definitional rather than proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Logical Reasoning in Large Language Models: A Survey." pith.science (2026). https://pith.science/paper/EU7IKZJM

@misc{pith2026250209100,
  author       = {Pith},
  title        = {Pith review of: Logical Reasoning in Large Language Models: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EU7IKZJM}},
  note         = {Machine review of arXiv:2502.09100}
}
read the original abstract

With the emergence of advanced reasoning models like OpenAI o3 and DeepSeek-R1, large language models (LLMs) have demonstrated remarkable reasoning capabilities. However, their ability to perform rigorous logical reasoning remains an open question. This survey synthesizes recent advancements in logical reasoning within LLMs, a critical area of AI research. It outlines the scope of logical reasoning in LLMs, its theoretical foundations, and the benchmarks used to evaluate reasoning proficiency. We analyze existing capabilities across different reasoning paradigms - deductive, inductive, abductive, and analogical - and assess strategies to enhance reasoning performance, including data-centric tuning, reinforcement learning, decoding strategies, and neuro-symbolic approaches. The review concludes with future directions, emphasizing the need for further exploration to strengthen logical reasoning in AI systems.

Figures

Figures reproduced from arXiv: 2502.09100 by the authors.

Figure 1
Figure 1. The structure of this survey theories, and in the 17th century, Leibniz’s universal lan￾guage and calculus ratiocinator bridged logic with mathemat￾ics, foreshadowing modern computational logic. The 19th century saw George Boole’s Boolean algebra, which trans￾formed logic into a mathematical framework, laying the foun￾dation for digital computing. The 20th century ushered in modern logic, with Russell and Whitehead’… view at source ↗
Figure 2
Figure 2. Example tests of Logical reasoning in NLP tasks. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ARGUS: Hallucination and Omission Evaluation in Video-LLMs

    cs.CV 2025-06 conditional novelty 7.0 of 10

    ARGUS measures hallucination and omission in free-form video captions using LLM-based entailment and temporal alignment, finding that even the best video-LLM still produces roughly 40% hallucinated content.

  2. Deductive Logic in Language Models: Horizontal vs Vertical Reasoning

    cs.AI 2025-10 conditional novelty 6.0 of 10

    A 2-layer, single-head attention-only transformer learns to perform multi-step logical deduction through induction-head circuits for rule completion, chaining, and final decision.

  3. MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A new 1,188-question multimodal benchmark covering deductive, inductive, and abductive reasoning shows that leading MLLMs score around 60% and are especially weak at abductive reasoning.

  4. DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching

    cs.SD 2025-08 unverdicted novelty 4.0 of 10

    DAFMSVC swaps source SSL features for similar target features and adds dual cross-attention plus flow matching to improve one-shot singing voice conversion.

  5. Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons

    cs.AI 2025-06 conditional novelty 3.0 of 10

    DeepSeek-R1 outperforms GPT-4o and DeepSeek-V3 on family tree and graph reasoning benchmarks at sizes 10 and 20, but all models collapse at size 40.

Reference graph

Works this paper leans on

71 extracted references · 56 canonical work pages · cited by 5 Pith papers

  1. [1]

    Lingoly: A benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages

    [Bean et al., 2024] Andrew M Bean, Simi Hellsten, Harry Mayne, Jabez Magomere, Ethan A Chi, et al. Lingoly: A benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages. arXiv preprint arXiv:2406.06196,

    arXiv 2024

  2. [3]

    Formal semantics: an introduction

    [Cann, 1993] Ronnie Cann. Formal semantics: an introduction. Cambridge University Press, United States,

    work page 1993

  3. [7]

    Selection-inference: Exploiting large language models for inter- pretable logical reasoning

    [Creswell et al., 2023] Antonia Creswell, Murray Shanahan, and Irina Hig- gins. Selection-inference: Exploiting large language models for inter- pretable logical reasoning. In Proc. of ICLR,

    work page 2023

  4. [10]

    Language models can be deductive solvers

    [Feng et al., 2024] Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, et al. Language models can be deductive solvers. In Proc. of ACL Findings, pages 4026–4042,

    work page 2024

  5. [11]

    Inductive learning of logical theories with llms: A complexity- graded analysis

    [Gandarela et al., 2024] Jo˜ao Pedro Gandarela, Danilo S Carvalho, and Andr´e Freitas. Inductive learning of logical theories with llms: A complexity- graded analysis. arXiv preprint arXiv:2408.16779,

    arXiv 2024

  6. [12]

    Log- ical inferences with comparatives and generalized quantifiers

    [Haruta et al., 2020] Izumi Haruta, Koji Mineshima, and Daisuke Bekki. Log- ical inferences with comparatives and generalized quantifiers. In Proc. of ACL, pages 263–270,

    work page 2020

  7. [15]

    Chen, and Shafiq Joty

    [Jiao et al., 2024] Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen, and Shafiq Joty. Learning planning-based reasoning by trajectories collection and process reward synthesizing. In Proc. of EMNLP , pages 334–350,

    work page 2024

  8. [16]

    Maieutic prompting: Logically consistent rea- soning with recursive explanations

    [Jung et al., 2022] Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, et al. Maieutic prompting: Logically consistent rea- soning with recursive explanations. In Proc. of EMNLP, pages 1266–1279,

    work page 2022

Show all 71 references
  1. [17]

    Boosting logical reasoning in large language models through a new framework: The graph of thought

    [Lei et al., 2023] Bin Lei, Chunhua Liao, Caiwen Ding, et al. Boosting logical reasoning in large language models through a new framework: The graph of thought. arXiv preprint arXiv:2308.08614,

    2023 arXiv

  2. [18]

    Formal-llm: Integrating formal language and natural language for controllable llm-based agents

    [Li et al., 2024b] Zelong Li, Wenyue Hua, Hao Wang, He Zhu, and Yongfeng Zhang. Formal-llm: Integrating formal language and natural language for controllable llm-based agents. arXiv preprint arXiv:2402.00798,

    arXiv

  3. [19]

    Aligning with logic: Measuring, evaluating and improving logical consistency in large language models.arXiv preprint arXiv:2410.02205,

    [Liu et al., 2024c] Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vuli´c, and Nigel Collier. Aligning with logic: Measuring, evaluating and improving logical consistency in large language models.arXiv preprint arXiv:2410.02205,

    arXiv

  4. [20]

    NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints

    [Lu et al., 2021] Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In Proc. of NAACL, pages 4288–4299,

    2021

  5. [21]

    Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models,

    [Luo et al., 2024] Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, et al. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models,

    2024

  6. [22]

    Knowra: Knowledge retrieval augmented method for document-level relation extraction with comprehensive reasoning abili- ties,

    [Mai et al., 2025] Chengcheng Mai, Yuxiang Wang, Ziyu Gong, Hanxiang Wang, and Yihua Huang. Knowra: Knowledge retrieval augmented method for document-level relation extraction with comprehensive reasoning abili- ties,

    2025

  7. [23]

    Ex- ploring the role of reasoning structures for constructing proofs in multi- step natural language reasoning with large language models

    [Malon et al., 2024] Christopher Malon, Martin Min, Xiaodan Zhu, et al. Ex- ploring the role of reasoning structures for constructing proofs in multi- step natural language reasoning with large language models. In Proc. of EMNLP, pages 15299–15312,

    2024

  8. [24]

    McCarthy and P.J

    [McCarthy and Hayes, 1981] J. McCarthy and P.J. Hayes. Some philosophi- cal problems from the standpoint of artificial intelligence. In Readings in Artificial Intelligence, pages 431–450

    1981

  9. [28]

    s1: Simple test-time scaling

    [Muennighoff et al., 2025] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xi- ang Lisa Li, Li Fei-Fei, et al. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393,

    2025 arXiv

  10. [29]

    Newell and H

    [Newell and Simon, 1956] A. Newell and H. Simon. The logic theory machine–a complex information processing system. IRE Transactions on Information Theory,

    1956

  11. [31]

    LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers

    [Olausson et al., 2023] Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, et al. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Proc. of EMNLP, pages 5153–5176,

    2023

  12. [32]

    Learning to reason with LLMs

    [OpenAI, 2024] OpenAI. Learning to reason with LLMs. Technical report,

    2024

  13. [33]

    Fact- driven logical reasoning for machine reading comprehension,

    [Ouyang et al., 2023] Siru Ouyang, Zhuosheng Zhang, and Hai Zhao. Fact- driven logical reasoning for machine reading comprehension,

    2023

  14. [34]

    Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning

    [Pan et al., 2023] Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Proc. of EMNLP Findings, pages 3806–3824,

    2023

  15. [35]

    Logicbench: A benchmark for evaluation of logical reasoning,

    [Parmar et al., 2023] Mihir Parmar, Neeraj Varshney, Nisarg Patel, Santosh Mashetty, Man Luo, et al. Logicbench: A benchmark for evaluation of logical reasoning,

    2023

  16. [36]

    Logicbench: Towards systematic evaluation of logical reasoning ability of large language models

    [Parmar et al., 2024] Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, et al. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proc. of ACL, pages 13679–13707,

    2024

  17. [37]

    Logic for natural language analysis

    [Pereira, 1982] Fernando Carlos Neves Pereira. Logic for natural language analysis

    1982

  18. [39]

    Reasoning with large language models, a survey

    [Plaat et al., 2024] Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511,

    2024

  19. [40]

    Relevant or random: Can llms truly perform analogi- cal reasoning?,

    [Qin et al., 2024] Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, et al. Relevant or random: Can llms truly perform analogi- cal reasoning?,

    2024

  20. [41]

    Investigating transformer-guided chaining for interpretable natural logic reasoning

    [Rajaraman et al., 2023] Kanagasabai Rajaraman, Saravanan Rajamanickam, and Wei Shi. Investigating transformer-guided chaining for interpretable natural logic reasoning. In Proc. of ACL Findings, pages 9240–9253,

    2023

  21. [42]

    AnaLog: Testing analytical and deductive logic learn- ability in language models

    [Ryb et al., 2022] Samuel Ryb, Mario Giulianelli, Arabella Sinclair, and Raquel Fern´andez. AnaLog: Testing analytical and deductive logic learn- ability in language models. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 55–68,

    2022

  22. [43]

    Divide and translate: Compositional first-order logic transla- tion and verification for complex logical reasoning

    [Ryu et al., 2024] Hyun Ryu, Gyeongman Kim, Hyemin S Lee, and Eunho Yang. Divide and translate: Compositional first-order logic transla- tion and verification for complex logical reasoning. arXiv preprint arXiv:2410.08047,

    2024 arXiv

  23. [44]

    Testing the general de- ductive reasoning capacity of large language models using ood examples

    [Saparov et al., 2023] Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mehran Kazemi, et al. Testing the general de- ductive reasoning capacity of large language models using ood examples. In Proc. of NeurIPS, pages 3083–3105,

    2023

  24. [45]

    Chain of logic: Rule-based reasoning with large language models

    [Servantez et al., 2024] Sergio Servantez, Joe Barrow, Kristian Hammond, and Rajiv Jain. Chain of logic: Rule-based reasoning with large language models. In Proc. of ACL Findings, pages 2721–2733,

    2024

  25. [46]

    Neural natural logic inference for interpretable question answering

    [Shi et al., 2021] Jihao Shi, Xiao Ding, Li Du, Ting Liu, and Bing Qin. Neural natural logic inference for interpretable question answering. In Proc. of EMNLP, pages 3673–3684,

    2021

  26. [47]

    Hamilton

    [Sinha et al., 2019] Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. Empirical Methods of Natural Language Processing (EMNLP),

    2019

  27. [48]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    [Srivastava et al., 2022] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615,

    2022 arXiv

  28. [49]

    Logical reasoning with span-level predictions for interpretable and robust NLI models

    [Stacey et al., 2022] Joe Stacey, Pasquale Minervini, Haim Dubossarsky, and Marek Rei. Logical reasoning with span-level predictions for interpretable and robust NLI models. In Proc. of EMNLP, pages 3809–3823,

    2022

  29. [50]

    It is not true that transformers are inductive learners: Probing NLI models with external negation

    [Sullivan, 2024] Michael Sullivan. It is not true that transformers are inductive learners: Probing NLI models with external negation. In Proc. of EACL, pages 1924–1945,

    2024

  30. [51]

    A survey of reasoning with foundation models

    [Sun et al., 2023] Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, et al. A survey of reasoning with foundation models. arXiv preprint arXiv:2312.11562,

    2023 arXiv

  31. [52]

    Determlr: Augmenting llm-based logical reasoning from indetermi- nacy to determinacy

    [Sun et al., 2024] Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, et al. Determlr: Augmenting llm-based logical reasoning from indetermi- nacy to determinacy. In Proc. of ACL, pages 9828–9862,

    2024

  32. [53]

    ProofWriter: Generating implications, proofs, and abductive statements over natural language

    [Tafjord et al., 2021] Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Proc. of ACL Findings, pages 3621–3634,

    2021

  33. [54]

    Assessing the alignment of fol closeness metrics with hu- man judgement

    [Thatikonda et al., 2025] Ramya Keerthy Thatikonda, Wray Buntine, and Ehsan Shareghi. Assessing the alignment of fol closeness metrics with hu- man judgement. arXiv preprint arXiv:2501.08613,

    2025 arXiv

  34. [55]

    Diagnosing the first-order logical reasoning ability through LogicNLI

    [Tian et al., 2021] Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. Diagnosing the first-order logical reasoning ability through LogicNLI. In Proc. of EMNLP, pages 3738–3747,

    2021

  35. [56]

    Verifiable, debuggable, and repairable commonsense logi- cal reasoning via llm-based theory resolution

    [Toroghi et al., 2024] Armin Toroghi, Willis Guo, Ali Pesaranghader, and Scott Sanner. Verifiable, debuggable, and repairable commonsense logi- cal reasoning via llm-based theory resolution. In Proc. of EMNLP, pages 6634–6652,

    2024

  36. [57]

    Donti, Bryan Wilder, and Zico Kolter

    [Wang et al., 2019] Po-Wei Wang, Priya L. Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differ- entiable satisfiability solver,

    2019

  37. [58]

    From lsat: The progress and challenges of com- plex reasoning

    [Wang et al., 2022] Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, et al. From lsat: The progress and challenges of com- plex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing,

    2022

  38. [59]

    Thought- path contrastive learning via premise-oriented data augmentation for logical reading comprehension

    [Wang et al., 2024b] Chenxu Wang, Ping Jian, and Zhen Yang. Thought- path contrastive learning via premise-oriented data augmentation for logical reading comprehension. arXiv preprint arXiv:2409.14495,

    arXiv

  39. [60]

    ANA- LOGICAL - a novel benchmark for long text analogy evaluation in large language models

    [Wijesiriwardene et al., 2023] Thilini Wijesiriwardene, Ruwan Wickrama- rachchi, Bimal Gajera, Shreeyash Gowaikar, Chandan Gupta, et al. ANA- LOGICAL - a novel benchmark for long text analogy evaluation in large language models. In Proc. of ACL Findings, pages 3534–3549,

    2023

  40. [61]

    Training large language models for reasoning through reverse curriculum reinforcement learning

    [Xi et al., 2024] Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. In Proc. of ICML,

    2024

  41. [62]

    Beyond chain-of-thought: A survey of chain-of-x paradigms for llms.arXiv preprint arXiv:2404.15676,

    [Xia et al., 2024] Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, et al. Beyond chain-of-thought: A survey of chain-of-x paradigms for llms.arXiv preprint arXiv:2404.15676,

    2024 arXiv

  42. [63]

    Are large language models really good logical reason- ers? a comprehensive evaluation and beyond,

    [Xu et al., 2023] Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. Are large language models really good logical reason- ers? a comprehensive evaluation and beyond,

    2023

  43. [64]

    Aristotle: Mastering logical reasoning with a logic-complete decompose-search-resolve framework

    [Xu et al., 2024b] Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, et al. Aristotle: Mastering logical reasoning with a logic-complete decompose-search-resolve framework. arXiv preprint arXiv:2412.16953 ,

  44. [65]

    Swope, Alex Gu, Rahul Chala- mala, Peiyang Song, et al

    [Yang et al., 2023] Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chala- mala, Peiyang Song, et al. Leandojo: theorem proving with retrieval- augmented language models. In Proc. of ICONIP,

    2023

  45. [66]

    Re- clor: A reading comprehension dataset requiring logical reasoning

    [Yu et al., 2020] Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Re- clor: A reading comprehension dataset requiring logical reasoning. InProc. of ICLR,

    2020

  46. [67]

    Natural language reasoning, a survey

    [Yu et al., 2024] Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey. ACM Computing Surveys, pages 1– 39,

    2024

  47. [68]

    Can pretrained language models (yet) reason deduc- tively? In Proc

    [Yuan et al., 2023] Zhangdie Yuan, Songbo Hu, Ivan Vuli´c, Anna Korhonen, and Zaiqiao Meng. Can pretrained language models (yet) reason deduc- tively? In Proc. of EACL, pages 1447–1462,

    2023

  48. [69]

    Formal language knowledge corpus for retrieval augmented generation,

    [Zayyad and Adi, 2024] Majd Zayyad and Yossi Adi. Formal language knowledge corpus for retrieval augmented generation,

    2024

  49. [70]

    o1-coder: an o1 replication for coding

    [Zhang et al., 2024] Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, et al. o1-coder: an o1 replication for coding. arXiv preprint arXiv:2412.00154,

    2024 arXiv

  50. [71]

    Marco-o1: Towards open reasoning models for open-ended solutions

    [Zhao et al., 2024] Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, et al. Marco-o1: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411.14405, 2024

    2024 arXiv

  51. [1956]

    How well do sota legal reasoning models support abductive reasoning?,

    [Nguyen et al., 2023] Ha-Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, and Ken Satoh. How well do sota legal reasoning models support abductive reasoning?,

    2023

  52. [1959]

    Artificial intelligence, logic and formaliz- ing common sense

    [McCarthy, 1989] John McCarthy. Artificial intelligence, logic and formaliz- ing common sense. Philosophical Logic and Artificial Intelligence , pages 161–190,

    1989

  53. [1981]

    Programs with common sense

    [McCarthy, 1959] John McCarthy. Programs with common sense. In Pro- ceedings of the Teddington Conference on the Mechanization of Thought Processes,

    1959

  54. [1982]

    Can language models learn analogical reasoning? investigating training ob- jectives and comparisons to human performance

    [Petersen and van der Plas, 2023] Molly Petersen and Lonneke van der Plas. Can language models learn analogical reasoning? investigating training ob- jectives and comparisons to human performance. InProc. of EMNLP, pages 16414–16425,

    2023

  55. [1989]

    Enhancing reasoning capabilities of llms via prin- cipled synthetic logic corpus

    [Morishita et al., 2024] Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reasoning capabilities of llms via prin- cipled synthetic logic corpus. In Proc. of NeurIPS , pages 73572–73604,

    2024

  56. [1993]

    Deeplogic: Towards end-to-end differentiable logical reasoning,

    [Cingillioglu and Russo, 2019] Nuri Cingillioglu and Alessandra Russo. Deeplogic: Towards end-to-end differentiable logical reasoning,

    2019

  57. [2019]

    Trans- formers as soft reasoners over language

    [Clark et al., 2021] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Trans- formers as soft reasoners over language. In Proc. of IJCAI,

    2021

  58. [2020]

    Log- itorch: A pytorch-based library for logical reasoning on natural language

    [Helwe et al., 2022] Chadi Helwe, Chlo´e Clavel, and Fabian Suchanek. Log- itorch: A pytorch-based library for logical reasoning on natural language. In Proc. of EMNLP,

    2022

  59. [2021]

    Training verifiers to solve math word prob- lems

    [Cobbe et al., 2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, et al. Training verifiers to solve math word prob- lems. arXiv preprint arXiv:2110.14168,

    2021 arXiv

  60. [2022]

    Logicpro: Improving complex logical reasoning via program- guided learning

    [Jiang et al., 2024] Jin Jiang, Yuchen Yan, Yang Liu, Yonggang Jin, Shuai Peng, et al. Logicpro: Improving complex logical reasoning via program- guided learning. arXiv preprint arXiv:2409.12929,

    2024 arXiv

  61. [2023]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    [DeepSeek-AI, 2025] DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Technical report,

    2025

  62. [2024]

    A com- prehensive evaluation of inductive reasoning capabilities and problem solv- ing in large language models

    [Bowen et al., 2024] Chen Bowen, Rune Sætre, and Yusuke Miyao. A com- prehensive evaluation of inductive reasoning capabilities and problem solv- ing in large language models. In Proc. of ACL Findings, pages 323–339,

    2024

  63. [2025]

    True detective: A deep abductive reasoning benchmark undoable for GPT-3 and challenging for GPT-4

    [Del and Fishel, 2023] Maksym Del and Mark Fishel. True detective: A deep abductive reasoning benchmark undoable for GPT-3 and challenging for GPT-4. In Proceedings of the 12th Joint Conference on Lexical and Com- putational Semantics (*SEM 2023), pages 314–322,

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.