Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that expert-built decision-tree logic, turned into a knowledge graph and executed as a pseudo-program, lifts LLM reasoning accuracy by 4.0% to 44.3% over existing CoT methods on nine datasets.

desk verdict The framework is coherent and the appendix is transparent, but the central accuracy claim is undermined by a circular evaluation: test questions are generated from the same seeds whose reasoning logic the expert decision trees encode. read the letter →

arxiv 2504.13534 v3 pith:DDCCW6TX submitted 2025-04-18 cs.CL cs.AI

Feiyang Li , Peng Fang , Zhan Shi , Arijit Khan , Fang Wang , Weihao Wang , Xin Zhang , Yongjian Cui This is my paper · ORCID
classification cs.CLcs.AI
keywords chain-of-thoughtreasoningretrieval-augmentedgenerationknowledgegraphsdecisiontreespseudo-programpromptingLLMarithmeticcommonsense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fix two known weaknesses of chain-of-thought prompting: LLMs often generate unreliable reasoning steps, and natural-language reasoning trails code-guided reasoning in accuracy. Its proposed remedy, CoT-RAG, makes a domain expert's reasoning logic the backbone of the prompt by turning an expert-built decision tree into a knowledge graph, retrieving relevant examples and descriptions for each substep, and having the model execute the whole chain as a pseudo-program. The empirical claim is that this pipeline improves accuracy by 4.0% to 44.3% over existing chain-of-thought methods on nine public datasets spanning arithmetic, commonsense, and symbolic reasoning, and reaches 88.3%-99.3% accuracy on four law, finance, and logic datasets. If true, the contribution is practical: a way to raise reasoning reliability without fine-tuning, by injecting expert structure into the prompt itself.

What carries the argument

The load-bearing object is the pseudo-program knowledge graph (PKG). A domain expert authors a coarse-grained decision tree once; the LLM decomposes each node into entities carrying a sub-question, a worked sub-case, a sub-description slot, and an answer slot, with 'Answer Provision' edges encoding dependencies. In the retrieval stage, the model fills each sub-description from the user's query and from predecessor answers; in the execution stage, the model runs the graph line-by-line as a pseudo-program, producing one intermediate answer per entity and a final answer at the terminal node. This object does the argument's work because it constrains every reasoning step to a pre-specified logic while still letting the LLM supply content.

What would settle it

Run CoT-RAG and its strongest baseline on the original, unmodified test splits of the nine benchmarks, using the same expert decision trees, and compare accuracy with the reported results on the LLM-generated sets; if the margin shrinks to near zero, the claimed generalization is an artifact of the logic-matched evaluation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that structured, externally supplied reasoning logic—not the LLM's own free-form chain—can serve as the driver of inference. CoT-RAG decomposes each expert-built decision-tree node into knowledge-graph entities with sub-questions, sub-cases, sub-descriptions, and answers, links them by answer-provision edges, and then executes the graph as a pseudo-program that calls an LLM function at each entity. The paper reports that this design outperforms all compared CoT and graph-RAG baselines on every dataset, with the largest gains in vertical domains: 99.3% on LawBench, 98.6% on LegalBench, 94.7% on CFBenchmark, and 88.3% on AGIEval with GPT-4o mini. It also claims that replacing expert-built trees with LLM-generated ones costs 7.8% average accuracy, and that LLM-based retrieval beats vector-based retrieval in both accuracy and runtime.

Load-bearing premise

The whole comparison rests on the assumption that LLM-generated questions that share the same reasoning logic as the expert decision trees are representative of the original benchmarks; if that assumption fails, the accuracy gains do not transfer.

Editorial extensions

If this is right

  • If the accuracy gains transfer, arithmetic, commonsense, and symbolic reasoning can be improved without fine-tuning by giving each domain a one-time expert decision tree.
  • The vertical-domain results imply that legal, financial, and logical tasks can be handled by encoding domain reasoning procedures as pseudo-programs, making the method a candidate for risk-sensitive applications.
  • The pseudo-program representation is reported to work in Python, C++, and Java, so the same scaffold can be ported across code-centric prompting styles.
  • Ablations identify knowledge-graph decomposition as the largest contributor, so future iterations should concentrate on the quality of node decomposition rather than on better retrieval or prompt phrasing.
  • Because runtime stays linear in the number of decision-tree nodes, the framework is claimed to scale to thousands of user questions after a one-time offline tree-construction cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the accuracy gains are real, the expert decision tree becomes the main bottleneck; automating tree construction from statutes, guidelines, or case histories is the natural next step, and the paper itself flags this direction.
  • The logic-matched synthetic test sets could be reused as a diagnostic: a model that solves these but fails the original benchmark would reveal template-following rather than transferable reasoning.
  • The reported reliance on strong proprietary LLMs suggests a testable extension: distill the pseudo-program traces into a smaller model and measure how much of the accuracy gain survives, which would also address the paper's stated limitation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CoT-RAG, a three-stage reasoning framework that (i) converts expert-built coarse decision trees into knowledge graphs to guide chain-of-thought generation, (ii) augments the knowledge graph with retrieval of sub-cases and sub-descriptions, and (iii) executes the resulting reasoning structure as a pseudo-program. The authors report accuracy gains of 4.0% to 44.3% over CoT baselines on nine datasets and high accuracy on four vertical-domain datasets. However, the evaluation is not conducted on the original benchmark splits: per Appendix H, each dataset is replaced by an LLM-generated test set derived from 200 selected seed questions, and the expert decision trees are constructed to encode the reasoning logic of those same seeds. The measured gains therefore do not support the claim that CoT-RAG improves accuracy on the actual public benchmarks.

Significance. The framework itself is inventive: combining expert decision trees, knowledge-graph decomposition, retrieval-augmented generation, and pseudo-program prompting is a plausible way to inject structured reasoning into LLMs without fine-tuning, and the authors provide code, data, and extensive appendix examples. The central claim, however, is empirical and rests entirely on the validity of the evaluation protocol. Because the test questions are generated to share the reasoning logic encoded in the expert decision trees, the reported 4.0-44.3% gains relative to baselines are not evidence of generalization to the original datasets. If the method were validated on official splits with a fair protocol, it could be a useful engineering contribution, but the current evidence does not establish that.

major comments (3)
  1. [Section 4.1 and Appendix H] The evaluation is circular in a structural sense. Appendix H states that the authors select 200 questions per dataset with 'distinct reasoning logics' and then prompt five LLMs to generate four new questions per seed 'with the same reasoning logic but different content.' The expert-built decision trees are designed to encode those same reasoning logics (Section 3.1, Figure 3, Appendix F). Consequently, the test questions are template variants whose reasoning steps are exactly the ones the decision tree supplies, while the baselines receive no per-logic recipe. Accuracy measured on this logic-matched test set does not measure performance on the original benchmark distributions, and no results on the official splits are reported. This undermines the central claim stated in Section 5 and in the abstract.
  2. [Table 1 and Appendix H] The comparison with 'state-of-the-art methods' is not meaningful under the current protocol. The baseline methods are evaluated on the same LLM-generated test sets, but those test sets are constructed to match the reasoning logic of CoT-RAG's expert decision trees. The baselines have no access to that per-logic recipe, so the reported accuracy gaps of up to 44.3% conflate method quality with test-set construction bias. A reader cannot infer from Table 1 that CoT-RAG would outperform Manual-CoT, Auto-CoT, PS, or KD-CoT on the original data.
  3. [Table 2 and Section 4.1] The vertical-domain results inherit the same circularity. Section 4.1 says that 'following GraphRAG and Graph-CoT, we employ an LLM to adapt the datasets to satisfy our testing needs,' and Appendix H applies the same seed-based generation procedure to each domain-specific dataset. The very high accuracies in Table 2 (93-99% on LawBench and LegalBench) are therefore also measured on generated questions that share the reasoning logic of the seeds used to build the decision trees. These numbers do not demonstrate scalable cross-domain performance on the original LawBench, LegalBench, CFBenchmark, or AGIEval evaluations.
minor comments (4)
  1. [Section 4.1] HotpotQA is described as a commonsense reasoning dataset, but it is a multi-hop question answering benchmark requiring retrieval from Wikipedia paragraphs; this categorization should be justified or corrected, as it affects the interpretation of the commonsense results.
  2. [Table 4] In the GPT-4o mini results table, the Manual-CoT row for the Letter and Coin columns appears as '97.710094.9', which seems to be two numbers run together; the table should be reformatted.
  3. [Appendix F, Table 36] In the AGIEval pseudo-program example, the final lines use sub_question6 and sub_case6 where sub_question5 and sub_case5 are intended, and the comment references 'sub_question6' as well; this appears to be a copy-paste error from the CFBenchmark example.
  4. [Section C.2] The robustness analysis varies knowledge cases and expert designers, but the two experts are both co-authors of the paper and share the paper's assumptions about the target reasoning logic; the robustness result should be interpreted as sensitivity within the author team rather than as an independent expert study.

Circularity Check

1 steps flagged · score 7.0 of 10

Evaluation is circular: the test set is generated from seeds sharing the reasoning logic encoded in the expert decision trees, so the reported gains measure template-filling rather than generalization to the original benchmarks.

  1. fitted input called prediction [Appendix H and Section 3.1 (also Section 4.1)]
    "we first select 200 questions with distinct reasoning logics from each open-source dataset. Then, we prompt each LLM to generate four new questions for each selected question. These new questions had the same reasoning logic but different content. ... experts supply a one-time, coarse-grained decision tree (DT) that encapsulates the underlying reasoning logic for the domain"

    The expert-built decision tree is the method's central input: it supplies the reasoning logic that CoT-RAG decomposes into knowledge-graph entities and pseudo-program sub-questions (Sec. 3.1, Fig. 3). The test questions are generated from seed questions selected to have 'distinct reasoning logics,' and the generated variants are required to have 'the same reasoning logic but different content.' Thus, by construction, every test question is an instance of a reasoning logic that the decision tree already encodes, and CoT-RAG is given that logic as its per-node structure, whereas the baselines are not. The reported 4.0%-44.3% accuracy gains therefore measure how well the LLM fills in a hand-built template on paraphrases of the seeds, not performance on the original benchmark distributions.

full rationale

The paper's methodological contribution—an expert-built decision tree that structures reasoning, a RAG over knowledge cases, and pseudo-program execution—is self-contained and does not rely on self-citation. However, the central empirical claim (the accuracy improvement of 4.0%-44.3% over baselines) is evaluated on a test distribution that is generated from the same reasoning logic that the expert decision trees are designed to encode. Appendix H describes selecting 200 seed questions with 'distinct reasoning logics' and generating four new questions per seed 'with the same reasoning logic but different content.' Section 3.1 states that the decision tree 'encapsulates the underlying reasoning logic for the domain.' CoT-RAG then decomposes this logic into sub-questions, sub-cases, and sub-descriptions that are supplied to the LLM at inference time. Consequently, the test questions are tailored to match the structure the method injects; the comparison against CoT baselines, which receive no such reasoning-logic recipe, is therefore not a test of generalization to the original benchmark distributions. This is a structural circularity in the evaluation protocol rather than in the derivation of the framework itself. The paper does not report results on the original official test splits, and Section 4.1 explicitly states the datasets were adapted 'to suit our testing needs.' For these reasons, the accuracy numbers should be interpreted as performance on a logic-matched, LLM-generated benchmark, not as evidence of state-of-the-art performance on the public benchmarks as originally defined.

Assumptions & free parameters 5 free parameters · 6 assumptions · 4 invented entities

The framework's central mechanism is a hand-designed decision tree per domain, converted by an LLM into a knowledge graph with hand-selected exemplars and slot-filling sub-descriptions. The evaluation adds two further design choices: selecting 200 seed questions per dataset and generating synthetic test questions from those seeds. These choices, rather than fitted numeric constants, are the free parameters that shape the reported results. The axioms are domain assumptions about LLM capabilities and the representativeness of the synthetic test distribution, the latter being ad hoc to this paper.

free parameters (5)
  • 200 seed questions per dataset = 200
    The test sets are generated from these selected seeds, which are chosen for distinct reasoning logics, directly shaping the evaluation distribution.
  • 4 generated questions per seed = 4
    Determines the size and diversity of the synthetic test sets, with each seed producing 4 new questions per LLM, for 20 new questions total plus the original.
  • Expert-built decision tree structure per domain = N/A (structural)
    The hand-designed trees encode the reasoning logic; this is the core mechanism and is chosen by experts, not derived from data.
  • Knowledge case exemplar per DT node = 1 exemplar per node
    Each node contains one hand-selected worked example, such as the Tom carrying plates example, and these exemplars are part of the method's design.
  • Entity count after decomposition = up to 9 in experiments
    The number of sub-questions per node is produced by LLM decomposition and affects accuracy and complexity; it is a design-dependent variable rather than a fitted constant.
assumptions (6)
  • domain assumption LLMs can reliably decompose a coarse decision tree into a fine-grained knowledge graph.
    Stage 1 relies on this capability; the paper provides examples but no systematic verification of decomposition quality.
  • domain assumption LLM-based extraction of sub-descriptions from user query descriptions is accurate.
    Stage 2 assumes the LLM correctly fills each sub-description slot; errors would propagate to answers.
  • ad hoc to paper LLM-generated test questions faithfully represent the original datasets' reasoning distribution.
    Appendix H constructs the test sets by generating variants of 200 selected questions; this assumes the synthetic samples are representative of the full benchmarks.
  • domain assumption Expert-built decision trees capture the domain's reasoning logic.
    The method's reliability depends on expert knowledge; robustness results show accuracy varies with the expert, with Expert A and Expert B producing noticeable fluctuations.
  • domain assumption Pseudo-program prompting improves logical rigor over natural language prompts.
    Stage 3 posits that program-like structure reduces reasoning errors; the ablation shows a moderate contribution, so this assumption is only partially supported.
  • domain assumption String concatenation of intermediate answers is a sufficient mechanism for propagating values between sub-questions.
    The pseudo-programs pass answers as text, such as sub_description4=answer2+answer3; this assumes no information loss or formatting errors in the concatenated text.
invented entities (4)
  • Pseudo-Program Knowledge Graph (PKG)
    purpose: Represents the KG reasoning chain as an executable pseudo-program for the LLM.
    Internal representation introduced by the paper; no external falsifiable handle.
  • Sub-case
    purpose: A concise analogical example attached to each KG entity to guide the LLM's answer.
    Internal component; the exemplars are hand-selected and not independently verified.
  • Sub-description
    purpose: Text extracted from the user query or from previous answers, assigned to each entity.
    Internal slot-filling mechanism without independent evidence beyond the paper's examples.
  • Answer Provision relationship
    purpose: Edge type indicating that one entity's answer feeds another entity's sub-description.
    Graph-theoretic framing for reasoning dependencies; no independent existence outside the method.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models." pith.science (2026). https://pith.science/paper/DDCCW6TX

@misc{pith2026250413534,
  author       = {Pith},
  title        = {Pith review of: CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDCCW6TX}},
  note         = {Machine review of arXiv:2504.13534}
}
read the original abstract

Chain-of-thought (CoT) reasoning boosts large language models' (LLMs) performance on complex tasks but faces two key limitations: a lack of reliability when solely relying on LLM-generated reasoning chains and lower reasoning performance from natural language prompts compared with code prompts. To address these issues, we propose CoT-RAG, a novel reasoning framework with three key designs: (i) Knowledge Graph-driven CoT Generation, featuring knowledge graphs to modulate reasoning chain generation of LLMs, thereby enhancing reasoning credibility; (ii) Learnable Knowledge Case-aware RAG, which incorporates retrieval-augmented generation (RAG) into knowledge graphs to retrieve relevant sub-cases and sub-descriptions, providing LLMs with learnable information; (iii) Pseudo Program Prompting Execution, which promotes greater logical rigor by guiding LLMs to execute reasoning tasks as pseudo-programs. Evaluations on nine public datasets spanning three reasoning tasks reveal significant accuracy gains-ranging from 4.0% to 44.3%-over state-of-the-art methods. Furthermore, tests on four domain-specific datasets demonstrate exceptional accuracy and efficient execution, underscoring its practical applicability and scalability. Our code and data are available at https: //github.com/hustlfy123/CoT-RAG.

Figures

Figures reproduced from arXiv: 2504.13534 by the authors.

Figure 1
Figure 1. Example inputs and outputs of GPT-4o mini with (a) Zero-shot-CoT ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of our CoT-RAG framework. Question: How many trips will the person make? Knowledge case: Tom assists the restaurant staff in collecting dishes, but he can only carry 6 plates at a time. He needs to pick up 15 plates from one table and 9 plates from another. The total number of plates from both tables amounts to 24, so he needs to carry them 24/6=4 times. Sub-question 2: How many trays does the person nee… view at source ↗
Figure 3
Figure 3. An example of decomposing a decision tree into a knowledge graph. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation study results: accuracy when we [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Robustness of CoT-RAG [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Accuracy of problems that can be decomposed [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software

    cs.SE 2026-08 conditional novelty 6.0 of 10

    REAG and a confidence-calibrated cascade generate context-aware test oracles for LLM-based software and produce statistically controlled verdict reliability, demonstrated on a production nutrition advisory app.

  2. Mitigating LLM Sycophancy in Code Smell Detection Using Evidence-Guided Reasoning Prompts

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Sycophancy bias makes LLM code-smell detectors highly prompt-sensitive (DFR up to 72%, FAR >90%); evidence-first EGDP prompting reduces flips to ~12–26% and restores F1.

  3. Improving the Reliability of LLMs: Combining CoT, RAG, Self-Consistency, and Self-Verification

    cs.AI 2025-05 reject novelty 2.0 of 10

    A small benchmark study claims self-verification is the best hallucination-reduction method, but the evaluation leaks the ground-truth answer into the verification prompt.

Reference graph

Works this paper leans on

62 extracted references · 57 canonical work pages · cited by 3 Pith papers

  1. [1]

    - Case: The example describes Tom, who can carry 6 plates at a time

    Subquestion 1: - Question: How many lunch trays can the person carry at once? - Description: Roger can only carry 4 trays at a time. - Case: The example describes Tom, who can carry 6 plates at a time. - Answer: Roger can carry 4 trays at a time

    work page

  2. [2]

    - Case: The example describes Tom needing to pick up 15 plates from one table and 9 plates from another

    Subquestion 2: - Question: How many trays does the person need to pick up from the first table? - Description: Roger have to pick up 10 trays from the first table. - Case: The example describes Tom needing to pick up 15 plates from one table and 9 plates from another. - Answer: Roger needs to pick up 10 trays from the first table

    work page

  3. [3]

    InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 8916–8937, Miami, Florida, USA

    Learning to correct for QA reasoning with black-box LLMs. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 8916–8937, Miami, Florida, USA. Association for Computational Linguistics. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-sh...

    work page 2024

  4. [4]

    - Case: The example explains how Tom calculates his total number of plates

    Subquestion 4: - Question: How many trays does the person need to carry in total? - Description: Based on answers from subquestions 2 and 3, Roger’s total tray count is 10 (from the first table) + 2 (from the second table). - Case: The example explains how Tom calculates his total number of plates. - Answer: 10 + 2 = 12. Therefore, Roger needs to carry 12...

    work page

  5. [5]

    Assuming the speed of the first person is v kilometers per hour, what is the speed of the second person in kilometers per hour?

    Subquestion 5: - Question: How many trips will the person need to make to carry all the trays? - Description: Based on answers from subquestions 1 and 4, Roger can carry 4 trays at a time and he needs to carry 12 trays in total. - Case: The example outlines how Tom calculates the number of trips needed. - Answer: 12 (total trays) / 4 (trays per trip) = 3 ...

    work page

  6. [7]

    InProceedings of the 7th International Conference on Natural Language and Speech Processing (IC- NLSP 2024), pages 439–448, Trento

    SG-RAG: Multi-hop question answering with large language models through knowledge graphs. InProceedings of the 7th International Conference on Natural Language and Speech Processing (IC- NLSP 2024), pages 439–448, Trento. Association for Computational Linguistics. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Co...

    arXiv 2024

  7. [9]

    Ruilin Zhao, Feng Zhao, Long Wang, Xianzhi Wang, and Guandong Xu

    Retrieval-Augmented Generation for AI- Generated Content: A Survey.arXiv e-prints, arXiv:2402.19473. Ruilin Zhao, Feng Zhao, Long Wang, Xianzhi Wang, and Guandong Xu. 2024. Kg-cot: Chain-of-thought prompting of large language models over knowl- edge graphs for knowledge-aware question answer- ing. InProceedings of the Thirty-Third International Joint Conf...

    arXiv 2024

  8. [10]

    Answer Provision

    Pseudo-Program Prompting Execution (Lines 20-23). First, experts construct and input a coarse- grained decision tree related to a field (Line 3), and then ask the LLM to decompose the decision tree into a fine-grained, highly structured knowledge graph (Line 4), which is represented by the Pseudo- Program Prompting we proposed (Line 5), referred to as the...

    work page arXiv 2022

Show all 62 references
  1. [14]

    - Case: The example describes Tom needing to pick up 15 plates from one table and 9 plates from another

    Subquestion 3: - Question: How many trays does the person need to pick up from the second table? - Description: Roger have to pick up 2 trays from the second table. - Case: The example describes Tom needing to pick up 15 plates from one table and 9 plates from another. - Answe...

  2. [17]

    - Description: Friend P’s rate is 15% faster than Friend Q’s

    First Call to LLM (sub_question1): - Question: Assuming the speed of the first person is v kilometers per hour, what is the speed of the second person in kilometers per hour? - Case: Cyclist A’s rate is 20% faster than Cyclist B’s. - Description: Friend P’s rate is 15% faster ...

  3. [18]

    43 / 2.15v

    Second Call to LLM (sub_question2): - Question: How to express the time t (expressed in an equation containing v ) when two people start from both ends and walk towards each other at the same time? - Case: Two cyclists plan to ride along a 60-km route, starting at opposite end...

  4. [19]

    23 kilometers

    Third Call to LLM (sub_question3): - Question: How many kilometers will the first person walk when they meet? - Case: The speed of Cyclist A is 1.2v and the encounter time t = 300/11v hours. - Description: The description from ’answer1’ tells us the speed of Friend P is 1.15v ...

  5. [20]

    - Description: James writes a letter to 2 different friends twice a week

    First Call to LLM (sub_question1): - Question: How many times does the person write letters to each friend per week? - Case: The case provides an example where Tom writes a letter to 4 different friends once a week. - Description: James writes a letter to 2 different friends t...

  6. [21]

    4 letters

    Second Call to LLM (sub_question2): - Question: How many letters does the person write per week? - Case: The case tells us that Tom writes a letter to 4 different friends once a week. - Description: The description tells us James writes a letter to 2 different friends, and **t...

  7. [22]

    12 pages

    Third Call to LLM (sub_question3): - Question: What is the total number of pages written by the person per week? - Case: This case mentions that Tom writes 4 letters per week, and each letter is 5 pages long. - Description: The description tells us that each letter is 3 pages ...

  8. [23]

    624 pages

    Fourth Call to LLM (sub_question4): - Question: What is the total number of pages written by the person in a year? - Case: The case tells us that one year is approximately 52 weeks, and the total number of pages written per week is 12 pages (from ’answer3’). - Description: We ...

  9. [24]

    - Description: Joan found 70 seashells on the beach

    First Call to LLM (sub_question1): - Question: What was the total number of shells initially found by the person? - Case: The case provides an example where Lily initially found 80 seashells. - Description: Joan found 70 seashells on the beach. Answer: The total number of shel...

  10. [25]

    - Description: Joan has 27 seashells left

    Second Call to LLM (sub_question2): - Question: How many shells does this person have left? - Case: The case provides an example where Lily has 32 seashells left. - Description: Joan has 27 seashells left. Answer: According to the description, Joan has 27 seashells left. answe...

  11. [26]

    Third Call to LLM (sub_question3): - Question: How many shells did this person give to others? - Case: This case involves calculating how many shells were given by Lily based on the initial number of shells and how many are left. The formula provided is: Shells given = Initial...

  12. [27]

    Who was the member of the ’Mother Love Bone’ band who passed away before the release of the album ’Apple’?

    Final Answer: The final answer is "43," indicating how many shells Joan gave to others. Final Output: 43 Table 19: The output of executing the pseudo-program knowledge graph in Table 18. #Given a program text, your role is the LLM function,which has three parameters: case, que...

    1990

  13. [28]

    Mother Love Bone

    First Call to LLM (sub_question1): - Question: Who was the member of the "Mother Love Bone" band who passed away before the release of the album "Apple"? - Case: The case asks us to focus on the correlation between the death of a band member and the upcoming release of the alb...

  14. [29]

    Mother Love Bone

    Second Call to LLM (sub_question2): - Question: What band has Andrew Wood joined (except for "Mother Love Bone")? - Case: The case tells us to pay attention to the description of the person’s band experience. - Description: The description talks about Andrew Wood being the lea...

  15. [30]

    Mother Love Bone

    Third Call to LLM (sub_question3): - Question: What band was Andrew Wood in before joining Mother Love Bone? - Case: This call asks to pay attention to the order in which Andrew Wood joined bands. - Description: The description includes the answers from previous questions. Fro...

  16. [31]

    Malfunkshun

    Final Answer: The final answer is "Malfunkshun." Final Output: Malfunkshun Table 21: The output of executing the pseudo-program knowledge graph in Table 20. #Given a program text, your role is the LLM function,which has three parameters: case, question, and description. The de...

  17. [32]

    subdivision

    First Call to LLM (sub_question1): - Question: What are the general characteristics of the area represented by each option in terms of housing distribution and isolation? - Case: The case provides descriptions of different areas: neighborhood, subdivision, city, suburbs, and s...

  18. [33]

    - Description: The description relies on the answer from sub_question1, which identified the subdivision as the area most likely to have access control facilities and isolation

    Second Call to LLM (sub_question2): - Question: Which area is most likely to provide houses with access control facilities and isolation from other areas? - Case: The case asks to compare the characteristics of each area to determine which is most likely to have access control...

  19. [34]

    Is this behavior directly related to understanding and answering the teacher’s questions?

    Final Answer: The final answer is "B", corresponding to **subdivision**. Final Output: B Table 23: The output of executing the pseudo-program knowledge graph in Table 22. #Given a program text, your role is the LLM function,which has three parameters: case, question, and descr...

  20. [35]

    ask for a gold star

    First Call to LLM (sub_question1): - Question: Is this behavior directly related to understanding and answering the teacher’s questions? - Case: Asking for a gold star is a behavior that does not directly contribute to understanding the question. It’s more about receiving a re...

  21. [36]

    skip her class

    Second Call to LLM (sub_question2): - Question: Is this behavior helpful for understanding and answering the teacher’s questions? - Case: Skipping class means not being present to hear the teacher’s questions, which would make it impossible to understand or answer the question...

  22. [37]

    know the information

    Third Call to LLM (sub_question3): - Question: Does this behavior help to understand and answer the teacher’s questions? - Case: Knowing the information related to the problem is essential to understanding the teacher’s question and answering it accurately. - Description: The ...

  23. [38]

    No," "No,

    Fourth Call to LLM (sub_question4): - Question: Which behavior is most likely to occur? - Case: This question asks to select the behavior that best fits the context of understanding and answering the teacher’s question. - Description: The description includes the answers from ...

  24. [39]

    Pretty" is

    First Call to LLM (sub_question1): - Question: What is the last letter of this word? - Case: The description indicates that the last letter of "Pretty" is "y". - Description: The word is "Gavin". Answer: The last letter of "Gavin" is "n". answer1 = "n"

  25. [40]

    Jada" is

    Second Call to LLM (sub_question2): - Question: What is the last letter of this word? - Case: The description indicates that the last letter of "Jada" is "a". - Description: The word is "Neha". Answer: The last letter of "Neha" is "a". answer2 = "a"

  26. [41]

    Sarita" is

    Third Call to LLM (sub_question3): - Question: What is the last letter of this word? - Case: The description indicates that the last letter of "Sarita" is "a". - Description: The word is "Asha". Answer: The last letter of "Asha" is "a". answer3 = "a"

  27. [42]

    Allen" is

    Fourth Call to LLM (sub_question4): - Question: What is the last letter of this word? - Case: The description indicates that the last letter of "Allen" is "n". - Description: The word is "Baltazar". Answer: The last letter of "Baltazar" is "r". answer4 = "r"

  28. [43]

    n" (from Gavin),

    Fifth Call to LLM (sub_question5): - Question: What is the result of concatenating the last letter obtained from each word in order? - Case: The case gives the last letters of the words: "n" (from Gavin), "a" (from Neha), "a" (from Asha), and "r" (from Baltazar). - Description...

  29. [44]

    heads up

    First Call to LLM (sub_question1): - Question: What is the initial state of the coin? - Case: Pay attention to the initial state of the coin at this moment. - Description: The coin is heads up. Answer: Based on the description, the initial state of the coin is "heads up." answ...

  30. [45]

    heads up

    Second Call to LLM (sub_question2): - Question: What is the current state of the coin? - Case: Pay attention to whether this operation changes the original state of the coin and output the state after the operation. - Description: The description includes the answer from ’sub_...

  31. [46]

    heads up

    Third Call to LLM (sub_question3): - Question: What is the current state of the coin? - Case: Pay attention to whether this operation changes the original state of the coin and output the state after the operation. - Description: The description includes the answer from ’sub_q...

  32. [47]

    tails up

    Fourth Call to LLM (sub_question4): - Question: Is the coin still heads up? - Case: Pay attention to the coin’s transition at this moment. - Description: The description includes the answer from ’sub_question3’ ("tails up"), so we check if the coin is still heads up. Answer: S...

  33. [48]

    What is the amount of cash stolen in this theft?

    Final Answer: The final answer is "No" because after the flip, the coin is not heads up. Table 29: The output of executing the pseudo-program knowledge graph in Table 28. #Given a program text, your role is the LLM function,which has three parameters: case, question, and descr...

    2016

  34. [49]

    - Description: The theft took place on January 2, 2016, where Song XX cut open the victim’s coat pocket and stole a small yellow envelope containing 1,500 yuan

    First Call to LLM (sub_question1): - Question: What is the amount of cash stolen in this theft? - Case: Focus on accurately finding the amount of stolen cash in the description, ensuring no confusion with other theft events. - Description: The theft took place on January 2, 20...

    2016

  35. [50]

    - Description: On January 20, 2016, Song XX stole 7,000 yuan from the victim Zhang Mou 1’s coat pocket

    Second Call to LLM (sub_question2): - Question: What is the amount of cash stolen in this theft? - Case: Again, accurately find the amount of stolen cash from the description, ensuring no confusion with other theft events. - Description: On January 20, 2016, Song XX stole 7,00...

    2016

  36. [51]

    - Description: The description contains both ’answer1’ (1,500 yuan) and ’answer2’ (7,000 yuan)

    Third Call to LLM (sub_question3): - Question: What is the total amount involved in all theft crimes? - Case: Perform the addition operation correctly to avoid errors and omissions, using the amounts obtained from the previous answers. - Description: The description contains b...

  37. [52]

    Does the contract mention any content regarding usage permissions?

    Final Answer: The total amount involved in all theft crimes is 8,500 yuan. Table 31: The output of executing the pseudo-program knowledge graph in Table 30. #Given a program text, your role is the LLM function,which has three parameters: case, question, and description. The de...

  38. [53]

    - Description: Roger can carry 4 trays at a time

    First Call to LLM (sub_question1): - Question: How many lunch trays can the person carry at once? - Case: Tom can carry 6 plates at a time. - Description: Roger can carry 4 trays at a time. - Answer: Since Roger can carry 4 trays at a time, the answer is simply 4. answer1 = "4"

  39. [54]

    - Description: Roger needs to pick up 10 trays from one table

    Second Call to LLM (sub_question2): - Question: How many trays does the person need to pick up from the first table? - Case: Tom needs to pick up 15 plates from one table. - Description: Roger needs to pick up 10 trays from one table. - Answer: Roger needs to pick up 10 trays ...

  40. [55]

    - Description: Roger needs to pick up 2 trays from another table

    Third Call to LLM (sub_question3): - Question: How many trays does the person need to pick up from the second table? - Case: Tom needs to pick up 9 plates from another table. - Description: Roger needs to pick up 2 trays from another table. - Answer: Roger needs to pick up 2 t...

  41. [56]

    - Description: The total number of trays that Roger needs to carry is the sum of the trays picked up from both tables

    Fourth Call to LLM (sub_question4): - Question: How many trays does the person need to carry in total? - Case: Tom needs to pick up 15 plates from one table and 9 plates from the second table. - Description: The total number of trays that Roger needs to carry is the sum of the...

  42. [57]

    How many lunch trays can the person carry at once?

    Fifth Call to LLM (sub_question5): - Question: How many trips will the person need to make to carry all the trays? - Case: Tom can carry 6 plates at a time and needs to carry 24 plates in total. - Description: Roger can carry 4 trays at a time and needs to carry 12 trays in to...

  43. [58]

    - Description: Roger can carry 4 trays at a time

    First Call to LLM (sub_question1): - Question: How many lunch trays can the person carry at once? - Case: Tom can carry 6 plates at a time. - Description: Roger can carry 4 trays at a time. - Answer: Based on the description, Roger can carry 4 trays at a time. So, answer1 = "4 trays"

  44. [59]

    10 trays

    Second Call to LLM (sub_question2): - Question: How many trays does the person need to pick up from the first table? - Case: Tom needs to pick up 15 plates from one table. - Description: Roger needs to pick up 10 trays from one table and 2 trays from another. - Answer: Roger n...

  45. [60]

    - Description: Roger needs to pick up 10 trays from one table and 2 trays from another

    Third Call to LLM (sub_question3): - Question: How many trays does the person need to pick up from the second table? - Case: Tom needs to pick up 15 plates from one table and 9 plates from another. - Description: Roger needs to pick up 10 trays from one table and 2 trays from ...

  46. [61]

    12 trays

    Fourth Call to LLM (sub_question4): - Question: How many trays does the person need to carry in total? - Case: Tom needs to pick up 15 plates from the first table and 9 plates from the second table. - Description: By adding up the trays from the first and second tables, we get...

  47. [62]

    If a train travels 300 miles in 5 hours, what is its average speed?

    Fifth Call to LLM (sub_question5): - Question: How many trips will the person need to make to carry all the trays? - Case: Tom can carry 6 plates at a time and needs to carry 24 plates in total. - Description: Roger can carry 4 trays at a time and needs to carry 12 trays in to...

  48. [2015]

    Shanu Kumar, Saish Mendke, Karody Lubna Abdul Rahman, Santosh Kurasa, Parag Agrawal, and Sandi- pan Dandapat

    Parsing algebraic word problems into equa- tions.Transactions of the Association for Computa- tional Linguistics, 3:585–597. Shanu Kumar, Saish Mendke, Karody Lubna Abdul Rahman, Santosh Kurasa, Parag Agrawal, and Sandi- pan Dandapat. 2025. Enhancing zero-shot chain of thought...

    2025 arXiv

  49. [2021]

    Avinava Dubey, Zhe Feng, Rahul Kidambi, Aranyak Mehta, and Di Wang

    Training Verifiers to Solve Math Word Prob- lems.arXiv e-prints, arXiv:2110.14168. Avinava Dubey, Zhe Feng, Rahul Kidambi, Aranyak Mehta, and Di Wang. 2024. Auctions with llm sum- maries. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing,...

    2024 arXiv

  50. [2022]

    How many lunch trays can the person carry at once?

    dataset of questions on whether a coin is still heads up after it is flipped or not flipped based on steps given in the questions. Vertical domains: LawBench (LaB)(Fei et al., 2024): A Chinese legal benchmark including tasks such as entity recognition, reading comprehension, a...

    2024

  51. [2023]

    InFindings of the Associa- tion for Computational Linguistics: EMNLP 2023, pages 2550–2575, Singapore

    Large language models are better reasoners with self-verification. InFindings of the Associa- tion for Computational Linguistics: EMNLP 2023, pages 2550–2575, Singapore. Association for Com- putational Linguistics. Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Gol...

    2023

  52. [2024]

    Association for Computational Linguistics

    Is compound aspect-based sentiment analysis addressed by LLMs? InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 7836–7861, Miami, Florida, USA. Association for Computational Linguistics. Baidu. 2025. Ernie-speed-128k. https://cloud.baidu.com/produ...

    2024

  53. [2025]

    InThe Thirteenth In- ternational Conference on Learning Representations

    Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. InThe Thirteenth In- ternational Conference on Learning Representations. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegre...

    2023 arXiv

  54. [9474]

    Dongyuan Li, Ying Zhang, Zhen Wang, Shiyin Tan, Satoshi Kosugi, and Manabu Okumura

    Curran Associates, Inc. Dongyuan Li, Ying Zhang, Zhen Wang, Shiyin Tan, Satoshi Kosugi, and Manabu Okumura. 2024. Active learning for abstractive text summarization via LLM- determined curriculum and certainty gain maximiza- tion. InFindings of the Association for Computa- tio...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.