{"id":"90792d9d-19cd-4c2e-b57e-71cfc0dce11a","arxiv_id":"2504.13534","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"CoT-RAG steers LLM reasoning by converting expert decision trees into knowledge graphs executed as pseudo-programs, but its claimed gains rest on test questions generated from the same reasoning logic as those templates.","lead":"CoT-RAG is a reasoning framework that turns expert-built decision trees into knowledge graphs, then has an LLM fill in and execute those graphs as pseudo-programs to answer questions. The paper reports large accuracy gains over baselines, but its test sets are LLM-generated variants of seed questions sharing the reasoning logic the decision trees encode, so the evaluation does not establish general-purpose superiority.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is circular: test questions are LLM-generated variants sharing the reasoning logic encoded in the expert decision trees, so the reported gains may not transfer to the original benchmarks.","rationale":"The reader's weakest assumption correctly identifies the load-bearing flaw: the evaluation protocol makes the test distribution non-independent of the method's expert-provided reasoning structure. The paper is otherwise a plausible engineering contribution, with clear methodology and extensive appendix examples, but its headline empirical claim rests entirely on the validity of the 4.0-44.3% accuracy comparisons. Appendix H confirms that the test sets are generated from 200 seed questions per dataset with the same reasoning logic, and the decision trees are built to encode exactly those logics. This makes the evaluation a test of how well CoT-RAG fills templates whose reasoning patterns it was given, rather than how well it generalizes to the original benchmark distributions. The absence of results on the original official splits means the comparison to published state-of-the-art is indirect and potentially misleading. Since the concern is about the soundness of the empirical evidence rather than an external-consensus disagreement, the reader's REJECT verdict stands unchanged.","tokens_in":50784,"tokens_out":6547,"duration_ms":61403,"concrete_test":"Run CoT-RAG and the top three baselines (e.g., Manual-CoT, Auto-CoT, PS) on the original official test splits of GSM8K, AQuA, MultiArith, SingleEq, CSQA, SIQA, and the two symbolic tasks, using decision trees built from training splits only. Report per-dataset accuracy and the average gain; if the 4.0-44.3% range shrinks to near zero or reverses, the state-of-the-art claim is unsupported. Also check whether any of the 200 selected seed questions belong to the official test split; if so, the current evaluation includes direct test-set contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the measured 4.0-44.3% accuracy gain over existing CoT methods on nine public datasets. That claim depends on the test distribution being representative enough to support generalization. The protocol in Appendix H breaks this condition: for each dataset, 200 seed questions with 'distinct reasoning logics' are selected, and five LLMs generate four new questions per seed 'with the same reasoning logic but different content.' The expert-built decision trees are then constructed to encode those same reasoning logics (Sec. 3.1, Fig. 3, Appendix F). The test questions are therefore template variants of the seeds whose reasoning structure is exactly what CoT-RAG's decision tree supplies, while the baselines receive no such per-logic recipe. Accuracy on this logic-matched test set does not measure performance on the original benchmark distribution; no results on the original official splits are reported. Appendix H explicitly says datasets are generated 'to suit our testing needs,' which confirms the non-independence. The ablation and robustness analyses inherit the same circularity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoT-RAG, a three-stage reasoning framework that (i) converts expert-built coarse decision trees into knowledge graphs to guide chain-of-thought generation, (ii) augments the knowledge graph with retrieval of sub-cases and sub-descriptions, and (iii) executes the resulting reasoning structure as a pseudo-program. The authors report accuracy gains of 4.0% to 44.3% over CoT baselines on nine datasets and high accuracy on four vertical-domain datasets. However, the evaluation is not conducted on the original benchmark splits: per Appendix H, each dataset is replaced by an LLM-generated test set derived from 200 selected seed questions, and the expert decision trees are constructed to encode the reasoning logic of those same seeds. The measured gains therefore do not support the claim that CoT-RAG improves accuracy on the actual public benchmarks.","tokens_in":50905,"tokens_out":3078,"duration_ms":32378,"significance":"The framework itself is inventive: combining expert decision trees, knowledge-graph decomposition, retrieval-augmented generation, and pseudo-program prompting is a plausible way to inject structured reasoning into LLMs without fine-tuning, and the authors provide code, data, and extensive appendix examples. The central claim, however, is empirical and rests entirely on the validity of the evaluation protocol. Because the test questions are generated to share the reasoning logic encoded in the expert decision trees, the reported 4.0-44.3% gains relative to baselines are not evidence of generalization to the original datasets. If the method were validated on official splits with a fair protocol, it could be a useful engineering contribution, but the current evidence does not establish that.","major_comments":[{"comment":"The evaluation is circular in a structural sense. Appendix H states that the authors select 200 questions per dataset with 'distinct reasoning logics' and then prompt five LLMs to generate four new questions per seed 'with the same reasoning logic but different content.' The expert-built decision trees are designed to encode those same reasoning logics (Section 3.1, Figure 3, Appendix F). Consequently, the test questions are template variants whose reasoning steps are exactly the ones the decision tree supplies, while the baselines receive no per-logic recipe. Accuracy measured on this logic-matched test set does not measure performance on the original benchmark distributions, and no results on the official splits are reported. This undermines the central claim stated in Section 5 and in the abstract.","section":"Section 4.1 and Appendix H"},{"comment":"The comparison with 'state-of-the-art methods' is not meaningful under the current protocol. The baseline methods are evaluated on the same LLM-generated test sets, but those test sets are constructed to match the reasoning logic of CoT-RAG's expert decision trees. The baselines have no access to that per-logic recipe, so the reported accuracy gaps of up to 44.3% conflate method quality with test-set construction bias. A reader cannot infer from Table 1 that CoT-RAG would outperform Manual-CoT, Auto-CoT, PS, or KD-CoT on the original data.","section":"Table 1 and Appendix H"},{"comment":"The vertical-domain results inherit the same circularity. Section 4.1 says that 'following GraphRAG and Graph-CoT, we employ an LLM to adapt the datasets to satisfy our testing needs,' and Appendix H applies the same seed-based generation procedure to each domain-specific dataset. The very high accuracies in Table 2 (93-99% on LawBench and LegalBench) are therefore also measured on generated questions that share the reasoning logic of the seeds used to build the decision trees. These numbers do not demonstrate scalable cross-domain performance on the original LawBench, LegalBench, CFBenchmark, or AGIEval evaluations.","section":"Table 2 and Section 4.1"}],"minor_comments":[{"comment":"HotpotQA is described as a commonsense reasoning dataset, but it is a multi-hop question answering benchmark requiring retrieval from Wikipedia paragraphs; this categorization should be justified or corrected, as it affects the interpretation of the commonsense results.","section":"Section 4.1"},{"comment":"In the GPT-4o mini results table, the Manual-CoT row for the Letter and Coin columns appears as '97.710094.9', which seems to be two numbers run together; the table should be reformatted.","section":"Table 4"},{"comment":"In the AGIEval pseudo-program example, the final lines use sub_question6 and sub_case6 where sub_question5 and sub_case5 are intended, and the comment references 'sub_question6' as well; this appears to be a copy-paste error from the CFBenchmark example.","section":"Appendix F, Table 36"},{"comment":"The robustness analysis varies knowledge cases and expert designers, but the two experts are both co-authors of the paper and share the paper's assumptions about the target reasoning logic; the robustness result should be interpreted as sensitivity within the author team rather than as an independent expert study.","section":"Section C.2"}],"recommendation":"reject","confidential_remarks":"The central problem is the evaluation protocol, not the idea itself. The generated test sets and the expert decision trees are matched by construction, so the paper's main quantitative claims are unsupported. In principle, the authors could rerun the experiments on original benchmark splits with decision trees built without seed leakage, but that would be a substantial new evaluation and would likely change the headline numbers. Given that the current manuscript's empirical contribution is the abstract's central promise, I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper describes a plausible three-stage framework: expert-built decision trees that LLMs expand into knowledge graphs, case-aware retrieval from those graphs, and pseudo-program execution. Each component is known, but the combination is new and clearly presented. The appendix is genuinely thorough: worked examples for every dataset, pseudo-programs in Python/C++/Java, and the released code and data are real assets. I also give the authors credit for documenting the dataset generation protocol in Appendix H instead of burying it.\n\nThat said, the reader's stress-test note is correct, and it is load-bearing. The central claim is the measured 4–44% accuracy gain. But the test questions are LLM-generated variants of 200 seed questions per dataset, explicitly sharing the same reasoning logic as those seeds. The expert decision trees are then built to encode exactly that reasoning logic. So CoT-RAG is tested on questions whose structure it was handed in advance, while the baselines receive no such per-logic recipe. This is not a minor weakness; it invalidates the reported gains as evidence for generalization. No results on the original benchmark splits appear anywhere in the paper, so the state-of-the-art comparison is indirect and misleading. The ablation and robustness studies inherit the same circularity, since they also run on the logic-matched synthetic test sets.\n\nThe limitation section acknowledges some constraints, like dependence on proprietary LLMs and expert effort, but it does not address the fundamental evaluation issue. That is disappointing, because the method could still be useful if re-tested honestly.\n\nWho is this for? A practitioner who wants to inject expert reasoning structure into an LLM might find the framework a useful recipe, and the appendix is a good template. As a reading-group case study in evaluation design, it is also valuable as a warning. But the paper in its current form does not establish its advertised advantage.\n\nMy recommendation: send it to serious peer review, but with a required major revision. Ask the authors to re-run the experiments on the original official benchmark splits, and to compare against baselines under the same conditions. If the gains survive independent evaluation, this could be a useful contribution. As reported, the central claim does not hold.","headline":"The framework is coherent and the appendix is transparent, but the central accuracy claim is undermined by a circular evaluation: test questions are generated from the same seeds whose reasoning logic the expert decision trees encode.","tokens_in":51566,"tokens_out":1622,"would_cite":false,"duration_ms":18002,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that expert-built decision-tree logic, turned into a knowledge graph and executed as a pseudo-program, lifts LLM reasoning accuracy by 4.0% to 44.3% over existing CoT methods on nine datasets.","keywords":["chain-of-thought reasoning","retrieval-augmented generation","knowledge graphs","decision trees","pseudo-program prompting","LLM reasoning","arithmetic reasoning","commonsense reasoning"],"falsifier":"Run CoT-RAG and its strongest baseline on the original, unmodified test splits of the nine benchmarks, using the same expert decision trees, and compare accuracy with the reported results on the LLM-generated sets; if the margin shrinks to near zero, the claimed generalization is an artifact of the logic-matched evaluation.","tokens_in":50478,"feed_emoji":"🧩","tokens_out":9148,"duration_ms":74556,"temperature":0.7,"pith_summary":"The paper sets out to fix two known weaknesses of chain-of-thought prompting: LLMs often generate unreliable reasoning steps, and natural-language reasoning trails code-guided reasoning in accuracy. Its proposed remedy, CoT-RAG, makes a domain expert's reasoning logic the backbone of the prompt by turning an expert-built decision tree into a knowledge graph, retrieving relevant examples and descriptions for each substep, and having the model execute the whole chain as a pseudo-program. The empirical claim is that this pipeline improves accuracy by 4.0% to 44.3% over existing chain-of-thought methods on nine public datasets spanning arithmetic, commonsense, and symbolic reasoning, and reaches 88.3%-99.3% accuracy on four law, finance, and logic datasets. If true, the contribution is practical: a way to raise reasoning reliability without fine-tuning, by injecting expert structure into the prompt itself.","feed_headline":"Expert decision trees lift LLM reasoning accuracy by up to 44 points","feed_subtitle":"CoT-RAG routes each question through a knowledge graph and pseudo-program, beating chain-of-thought baselines.","key_machinery":"The load-bearing object is the pseudo-program knowledge graph (PKG). A domain expert authors a coarse-grained decision tree once; the LLM decomposes each node into entities carrying a sub-question, a worked sub-case, a sub-description slot, and an answer slot, with 'Answer Provision' edges encoding dependencies. In the retrieval stage, the model fills each sub-description from the user's query and from predecessor answers; in the execution stage, the model runs the graph line-by-line as a pseudo-program, producing one intermediate answer per entity and a final answer at the terminal node. This object does the argument's work because it constrains every reasoning step to a pre-specified logic while still letting the LLM supply content.","core_discovery":"On the paper's own terms, the central discovery is that structured, externally supplied reasoning logic—not the LLM's own free-form chain—can serve as the driver of inference. CoT-RAG decomposes each expert-built decision-tree node into knowledge-graph entities with sub-questions, sub-cases, sub-descriptions, and answers, links them by answer-provision edges, and then executes the graph as a pseudo-program that calls an LLM function at each entity. The paper reports that this design outperforms all compared CoT and graph-RAG baselines on every dataset, with the largest gains in vertical domains: 99.3% on LawBench, 98.6% on LegalBench, 94.7% on CFBenchmark, and 88.3% on AGIEval with GPT-4o mini. It also claims that replacing expert-built trees with LLM-generated ones costs 7.8% average accuracy, and that LLM-based retrieval beats vector-based retrieval in both accuracy and runtime.","pith_inferences":["If the accuracy gains are real, the expert decision tree becomes the main bottleneck; automating tree construction from statutes, guidelines, or case histories is the natural next step, and the paper itself flags this direction.","The logic-matched synthetic test sets could be reused as a diagnostic: a model that solves these but fails the original benchmark would reveal template-following rather than transferable reasoning.","The reported reliance on strong proprietary LLMs suggests a testable extension: distill the pseudo-program traces into a smaller model and measure how much of the accuracy gain survives, which would also address the paper's stated limitation."],"forward_implications":["If the accuracy gains transfer, arithmetic, commonsense, and symbolic reasoning can be improved without fine-tuning by giving each domain a one-time expert decision tree.","The vertical-domain results imply that legal, financial, and logical tasks can be handled by encoding domain reasoning procedures as pseudo-programs, making the method a candidate for risk-sensitive applications.","The pseudo-program representation is reported to work in Python, C++, and Java, so the same scaffold can be ported across code-centric prompting styles.","Ablations identify knowledge-graph decomposition as the largest contributor, so future iterations should concentrate on the quality of node decomposition rather than on better retrieval or prompt phrasing.","Because runtime stays linear in the number of decision-tree nodes, the framework is claimed to scale to thousands of user questions after a one-time offline tree-construction cost."],"supporting_citations":[{"why":"Provides the Manual-CoT baseline and the original chain-of-thought method whose reasoning chains CoT-RAG aims to constrain.","marker":"Wei et al., 2022"},{"why":"Provides the Zero-shot-CoT baseline that CoT-RAG must beat on datasets without exemplars.","marker":"Kojima et al., 2022"},{"why":"Provides the Plan-and-Solve baseline and the reported AQuA and commonsense accuracy figures used to motivate the reliability problem.","marker":"Wang et al., 2023a"},{"why":"Supplies QDMRPS, the decomposition-based baseline whose planned reasoning CoT-RAG extends via knowledge graphs.","marker":"Huang et al., 2024a"},{"why":"Provides the KG-CoT graph-reasoning baseline used in both general and vertical-domain comparisons.","marker":"Zhao et al., 2024"},{"why":"Provides the GraphRAG baseline and the precedent for adapting datasets with LLMs to match testing needs.","marker":"Edge et al., 2024"},{"why":"Provides the Graph-CoT baseline and the second precedent for LLM-based dataset adaptation.","marker":"Jin et al., 2024"},{"why":"Provides IRCoT, the interleaved retrieval-reasoning baseline that motivates the case-aware RAG design.","marker":"Trivedi et al., 2023"}],"fun_headline_variants":["Graph-guided reasoning lifts LLM accuracy by up to 44 points","CoT-RAG: Knowledge graphs boost LLM reasoning up to 44%","Retrieval-augmented knowledge graphs lift LLM reasoning by up to 44%","CoT-RAG turns reasoning into pseudo-program execution, up to 44% better","Expert-built knowledge graphs outdo LLM chain-of-thought by 44%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that LLM-generated questions that share the same reasoning logic as the expert decision trees are representative of the original benchmarks; if that assumption fails, the accuracy gains do not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Graph-guided reasoning lifts LLM accuracy by up to 44 points","CoT-RAG: Knowledge graphs boost LLM reasoning up to 44%","Retrieval-augmented knowledge graphs lift LLM reasoning by up to 44%","CoT-RAG turns reasoning into pseudo-program execution, up to 44% better","Expert-built knowledge graphs outdo LLM chain-of-thought by 44%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001438,"raw_usage":{"total_tokens":5819,"prompt_tokens":987,"completion_tokens":4832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":4726}},"tokens_in":603,"tokens_out":4832,"duration_ms":31170,"temperature":1.0,"reasoning_tokens":4726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:06:22.962863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CoT-RAG and its strongest baseline on the original, unmodified test splits of the nine benchmarks, using the same expert decision trees, and compare accuracy with the reported results on the LLM-generated sets; if the margin shrinks to near zero, the claimed generalization is an artifact of the logic-matched evaluation.","supporting_citations":[],"review_version":1}