Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KnowTrace claims that structuring retrieved knowledge into a growing knowledge graph avoids context overload and enables backtracing to filter self-training data, yielding stronger multi-hop question answering.

desk verdict KnowTrace's structured KG tracing is a real improvement over text-accumulation iterative RAG, and the empirical claim is broadly supported, but the paper needs error bars and a direct test of its backtracing credit-assignment. read the letter →

arxiv 2505.20245 v1 pith:4GRD37UW submitted 2025-05-26 cs.CL cs.AI

Rui Li , Quanyu Dai , Zeyu Zhang , Xu Chen , Zhenhua Dong , Ji-Rong Wen This is my paper
classification cs.CLcs.AI
keywords retrieval-augmentedgenerationmulti-hopquestionansweringknowledgegraphself-trainingprocesssupervisionbacktracingcontextoverloadLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that iterative retrieval-augmented generation fails on multi-hop questions because piling retrieved passages into an ever-growing context overloads the language model, and because futile reasoning steps trigger irrelevant retrievals. To fix this, KnowTrace has the model trace question-relevant knowledge triplets into a small knowledge graph, expanding it round by round until it can answer. The graph gives the model an intelligible structured context, and it also enables a backtracing step that identifies which earlier generations actually contributed to a correct answer. The authors claim this yields better accuracy than existing iterative and restructuring-based RAG methods on three benchmarks, and that fine-tuning on backtraced process supervision amplifies the gain.

What carries the argument

The central objects are the question-specific knowledge graph $G_q$, a set of knowledge triplets (subject, relation, object) the model builds during iterative retrieval, and the supporting subgraph $S_q$ obtained by backtracing. The explore-then-complete loop alternates knowledge exploration, where the model outputs entity-relation pairs or a final answer, with knowledge completion, where the model converts retrieved passages into triplets conditioned on each pair. Backtracing identifies $S_q$ by taking entities appearing in the final thought and answer as targets and walking backward along graph edges to the initial entities; everything on those paths is deemed supportive. This subgraph serves two roles: it is the structured context that keeps the prompt free of accumulated text, and it is the filter that separates contributive from useless generations when synthesizing process supervision data for self-training.

What would settle it

Take a correct KnowTrace trajectory and delete the supporting subgraph $S_q$ from the context, then check whether the model still answers correctly; if it often does, the subgraph is not actually load-bearing and backtracing is mislabeling supervision. A direct statistical check is to remove the backtracing filter entirely and verify that the reported performance decline occurs; if the non-backtraced version matches or beats KnowTrace*, the filtering step is not the source of the bootstrapping gain.

Watch

Extended reading notes

Core claim

KnowTrace claims that multi-hop QA in a RAG setting is better served by structured knowledge tracing than by accumulating text. At each iteration the language model decides whether the current knowledge graph suffices; if not, it names entity-relation pairs to retrieve passages for, and then completes the retrieved text into new triplets. The resulting graph is both the context for further reasoning and the basis for a post-hoc backtracing step: after a correct trajectory, the entities in the final thought and answer are traced backward along graph paths to the initial entities, and only triplets and generations on those paths are kept as process supervision. Training on that filtered supervision is claimed to be what makes bootstrapping work, whereas the same self-training without backtracing degrades performance.

Load-bearing premise

The whole method hinges on the assumption that tracing backward along the graph from the entities in the final answer reliably identifies which earlier steps really helped, and that the graph itself has no wrong or missing links.

Editorial extensions

If this is right

  • KnowTrace consistently outperforms iterative and restructuring-based baselines on EM and F1 across HotpotQA, 2Wiki, and MuSiQue, using both LLaMA3-8B-Instruct and GPT-3.5-Turbo-Instruct as backbones.
  • The bootstrapped version KnowTrace*, trained on backtraced process supervision, further raises EM across bootstrapping iterations, while the same self-training without backtracing causes performance to decline.
  • The gains hold across three retrievers (BM25, DPR, Contriever) and across different numbers of retrieved passages $N$, with KnowTrace staying strong even when $N$ is large.
  • The KG-to-Triplets prompting strategy is both simplest and most effective among the three knowledge prompting strategies studied.
  • KnowTrace is roughly as cheap per query as the simplest iterative baseline IRCoT and much cheaper than the restructuring-based ERA-CoT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If structured tracing works because it keeps context compact and precise, the same explore-then-complete loop may transfer to other long-horizon tasks such as decision-making or mathematics, which the paper itself lists as an open direction.
  • Backtracing could be a general credit-assignment heuristic for self-training whenever intermediate LLM outputs can be represented as a graph; testing it on other self-training setups would clarify whether the benefit is specific to RAG.
  • The rise in the filtered-to-all ratio across bootstrapping iterations suggests that vanilla self-training on correct trajectories compounds noise; a simpler alternative, such as filtering generations by retrieval relevance instead of graph paths, could isolate whether graph structure or filtering is the true driver.
  • Because the method depends on the LLM's own graph construction, a natural stress test is to vary the LLM's instruction-following strength or the retriever quality and measure whether the backtracing gains remain positive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. KnowTrace proposes an iterative RAG framework for multi-hop question answering in which the LLM alternately proposes entity-relation pairs for retrieval and completes knowledge triplets, maintaining a growing knowledge-graph context. A post-hoc 'knowledge backtracing' mechanism selects triplets and exploration steps on backward paths from entities in the final thought/answer to initial entities, filters all other generations from trajectories that produced correct answers, and uses the retained generations as process supervision for LoRA finetuning over several bootstrapping iterations. Experiments on HotpotQA, 2WikiMultihopQA, and MuSiQue with LLaMA3-8B-Instruct and GPT-3.5-Turbo-Instruct, and with BM25, DPR, and Contriever retrievers, report EM/F1 gains over RA-CoT, ReAct, IRCoT, Self-Ask, Iter-RetGen, SG-Prompt, GE-Reasoning, and ERA-CoT, together with an efficiency analysis and an FA ratio intended to quantify the noise removed by backtracing.

Significance. If the results hold, KnowTrace offers a simple and comparatively efficient way to keep iterative RAG contexts structured, and a natural post-hoc credit-assignment scheme for self-training from correct trajectories. The paper's empirical coverage is substantial: three datasets, two LLMs, three retrievers, and a cost analysis, and the code is released. The main empirical pattern is consistent across those settings, and the structured-knowledge-tracing perspective is a plausible and useful alternative to unstructured context accumulation. However, the load-bearing backtracing mechanism is not directly validated, and the headline comparisons are reported as point estimates without uncertainty quantification, so the strength of the 'consistently outperforms' claim and the bootstrapping claim is not fully established.

major comments (3)
  1. [Section 4.2-4.4; Tables 1, 3, 4; Figures 4, 5] The main empirical claims rest on point estimates only. Tables 1, 3, and 4 and Figures 4 and 5 report single EM/F1 numbers with no confidence intervals, number of seeds, or significance tests, so the Section 4.3 statement that KnowTrace 'consistently outperforms all the baselines' is not statistically supported for the smaller gaps (e.g., Table 3, 2Wiki with DPR: KnowTrace .246 vs. ERA-CoT .220; Table 4, MuSiQue: KG-to-Paths .286 vs. KG-to-Triplets .280). In addition, Section 4.2 says LoRA hyperparameters were tuned and bootstrapping was repeated 'until the performance plateaus,' but no held-out validation set or model-selection rule is described; if the reported numbers are selected over iterations or hyperparameters on the test set, the gains would be optimistic. Please report means and variances over at least three seeds and specify the validation protocol used to choose hyperparameters and to stop bootstrapping.
  2. [Section 3.3; Algorithm 2; Figure 4] Backtracing is the only credit-assignment mechanism for the self-bootstrapping claim, yet its correctness is not tested. S_q is defined as all triplets on backward paths from entities appearing in [t,a] to initial entities; this assumes the final thought contains exactly the entities needed, the LLM-built KG has no spurious or missing edges, and path membership is a proxy for causal contribution. None of these assumptions is verified. The FA ratio in Figure 4(d-f) measures only how many tokens were filtered, and a random or length-based filter could produce the same FA while yielding different training data. Because Algorithm 2's training labels are exactly the generations retained by S_q, a mislabeled S_q would make the bootstrapping gains in Figure 4(a-c) ambiguous. A concrete remedy is to compare S_q against HotpotQA's gold supporting facts to measure precision/recall of the filtered triplets, and to add a control baseline that filters the same FA ratio randomly.
  3. [Section 4.4] The comparison between KnowTrace* and Non-Backtracing does not fully isolate the backtracing effect. The paper says the non-backtracing version follows 'the vanilla self-taught finetuning workflow [54]', but it does not state whether it uses the same LoRA architecture, training epochs, learning rate, and number of gradient steps, nor whether it is trained on the same set of correct trajectories with all generations retained. If the two versions differ in data size or optimization budget, the observed gap in Figure 4(a-c) could be due to those differences rather than to credit assignment. Please describe the non-backtracing setup precisely and, ideally, include an additional control trained on unfiltered positive trajectories with the same number of gradient steps.
minor comments (5)
  1. [Section 4.4] 'Fist bootstrapping iteration' should be 'first bootstrapping iteration'.
  2. [Table 3] The heading spells 'MusiQue' and 'Contriver'; use 'MuSiQue' and 'Contriever' for consistency.
  3. [Section 4.5] 'One the one hand' should be 'On the one hand'.
  4. [Algorithm 2 vs. Section 3.3] Algorithm 2 line 10 passes [t_d, \hat{a}_d] to Backtracing, while Section 3.3 defines backtracing on [t,a]; unify the notation.
  5. [Figure 4 caption] The caption should state explicitly what Non-Backtracing is (e.g., STaR applied to the same KnowTrace inference procedure) and whether the same training split and optimization settings are used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KnowTrace's main claims are anchored by held-out benchmark answers, and the backtracing self-reference is a design assumption rather than a forced derivation.

full rationale

The core inference claim (Table 1) is evaluated on held-out EM/F1 against HotpotQA, 2WikiMultihopQA, and MuSiQue, so the reported superiority over IRCoT, ERA-CoT, etc. is not forced by a fitted parameter or by the framework's own definitions. The bootstrapping claim (Section 4.4) is tested by comparing KnowTrace* with a non-backtracing STaR-style baseline under the same training budget; the ground-truth answer is used only to select positive trajectories, and the final evaluation is independent of the filtering rule. The backtracing mechanism in Section 3.3 does use the LLM's final thought/answer to define S_q, making 'supporting triplets' self-referential as a credit-assignment proxy, but the paper does not derive a performance result from that definition alone, and the FA ratio is presented as a descriptive statistic rather than as the evidence that forces the EM gain. No load-bearing premise is justified solely by a self-citation: the only same-author citation [21] supports a peripheral remark about relation mapping properties. Thus no prediction or first-principles result reduces by construction to its own inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method relies on standard domain assumptions about LLM instruction following, retrieval-corpus coverage, and self-training. The backtracing heuristic in Section 3.3 is the only genuinely paper-specific assumption; it is reasonable but unvalidated against human process labels. No new physical or mathematical entities are introduced.

free parameters (4)
  • N (number of retrieved passages per query) = 5 in main experiments; varied to 10, 20, 30, 50
    Selected by the authors; larger N increases context and can hurt baselines, so it interacts with the central comparison.
  • Maximum iterations L in Algorithm 1 = not reported
    The loop runs until the LLM sets FLAG=true, but no explicit cap or distribution of iteration counts is given.
  • Number of in-context examples per prompt = 4
    Chosen by hand for both I_exp and I_com; affects instruction-following quality.
  • LoRA training hyperparameters = epoch in {1,2,3}, batch size in {32,64,128}, learning rate in {1e-5,3e-4}; LoRA rank not reported
    Tuned per dataset during bootstrapping; exact selected values not reported.
assumptions (4)
  • domain assumption The base LLMs (LLaMA3-8B-Instruct, GPT-3.5-Turbo-Instruct) follow the structured exploration and completion prompts and produce parseable FLAG, entity-relation pairs, and triplets.
    Equations (1) and (2) in Section 3.2 assume the model output P and T are usable as structured queries and completions; failures here would break the KG expansion.
  • domain assumption The retrieval corpus, built by collecting all candidate passages including distractors following IRCoT, contains sufficient evidence to answer every evaluation question.
    Section 4.1 and Appendix B define the open-domain setting; if evidence for a question is absent from the corpus, no iterative RAG method can recover it.
  • domain assumption Self-training premise: positive trajectories that reach a correct answer, after filtering, contain reusable high-quality reasoning steps.
    This is the STaR assumption cited in Section 2 and used in Algorithm 2 and Section 4.4; it is not proven for this setting.
  • ad hoc to paper Backtracing premise: entities occurring in the final thought/answer, traced backward through the graph to initial entities, exactly identify the supporting subgraph and therefore the contributive generations.
    Section 3.3 defines the subgraph S_q this way; there is no validation against human process labels or alternative credit-assignment methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing." pith.science (2026). https://pith.science/paper/4GRD37UW

@misc{pith2026250520245,
  author       = {Pith},
  title        = {Pith review of: KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GRD37UW}},
  note         = {Machine review of arXiv:2505.20245}
}
read the original abstract

Recent advances in retrieval-augmented generation (RAG) furnish large language models (LLMs) with iterative retrievals of relevant information to handle complex multi-hop questions. These methods typically alternate between LLM reasoning and retrieval to accumulate external information into the LLM's context. However, the ever-growing context inherently imposes an increasing burden on the LLM to perceive connections among critical information pieces, with futile reasoning steps further exacerbating this overload issue. In this paper, we present KnowTrace, an elegant RAG framework to (1) mitigate the context overload and (2) bootstrap higher-quality multi-step reasoning. Instead of simply piling the retrieved contents, KnowTrace autonomously traces out desired knowledge triplets to organize a specific knowledge graph relevant to the input question. Such a structured workflow not only empowers the LLM with an intelligible context for inference, but also naturally inspires a reflective mechanism of knowledge backtracing to identify contributive LLM generations as process supervision data for self-bootstrapping. Extensive experiments show that KnowTrace consistently surpasses existing methods across three multi-hop question answering benchmarks, and the bootstrapped version further amplifies the gains.

Figures

Figures reproduced from arXiv: 2505.20245 by the authors.

Figure 1
Figure 1. Two challenges of iterative RAG systems: ever [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of two representative workflows (a-b) and our KnowTrace framework (c). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example of KnowTrace’s inference and backtrac [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: EM results (a-c) and FA ratios (d-f) in each bootstrapping iteration. KnowTrace* is the bootstrapped version based on [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: EM results with varying 𝑁 on HotpotQA. 4.5 More Analysis Cost Analysis. We include a detailed cost analysis for KnowTrace and two representative baselines (i.e., IRCoT and ERA-CoT). The statistics are summarized in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modeling

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A working memory represented as a hypergraph, whose hyperedges are updated, inserted, and progressively merged by the LLM, improves multi-step RAG on long-context sense-making benchmarks.

Reference graph

Works this paper leans on

59 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [54]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. STaR: Boot- strapping Reasoning With Reasoning. InAdvances in Neural Information Process- ing Systems

    work page 2022

  2. [1]

    Martin D Braine. 1978. On the relation between the natural logic of reasoning and standard logic.Psychological review85, 1 (1978), 1

    work page 1978

  3. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 2020

  4. [3]

    Xiaoxia Cheng, Zeqi Tan, and Weiming Lu. 2024. Information Re-Organization Improves Reasoning in Large Language Models.arXiv preprint arXiv:2404.13985 (2024)

    work page Pith review arXiv 2024

  5. [4]

    Zhenyun Deng, Yonghua Zhu, Yang Chen, Michael Witbrock, and Patricia Riddle

    work page

  6. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    arXiv 2024

  7. [6]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization.arXiv preprint arXiv:2404.16130 (2024)

    arXiv 2024

  8. [7]

    2013.Constructivism: Theory, perspectives, and practice

    Catherine Twomey Fosnot. 2013.Constructivism: Theory, perspectives, and practice. Teachers College Press

    2013

Show all 59 references
  1. [8]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997(2023)

    2023 arXiv

  2. [9]

    O’Reilly Media, Inc

    Clinton Gormley and Zachary Tong. 2015.Elasticsearch: the definitive guide: a distributed real-time search and analytics engine. " O’Reilly Media, Inc. "

    2015

  3. [10]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reason- ing Steps. InProceedings of the 28th International Conference on Computational Linguistics. 6609–6625

    2020

  4. [11]

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sor- doni, and Rishabh Agarwal. 2024. V-STaR: Training Verifiers for Self-Taught Reasoners. InFirst Conference on Language Modeling

    2024

  5. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InThe Tenth International Conference on Learning Representa- tions

    2022

  6. [13]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bo- janowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised Dense Infor- mation Retrieval with Contrastive Learning.Transactions on Machine Learning Research(2022). 2https://www.mindspore.cn

    2022

  7. [14]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research24, 251 (...

    2023

  8. [15]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.Comput. Surveys55, 12 (2023), 1–38

    2023

  9. [16]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 6769–6781

    2020

  10. [17]

    Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs. InThe Twelfth International Conference on Learning Representations

    2024

  11. [18]

    Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grig- orev. 2022. Internet-augmented language models through few-shot prompting for open-domain question answering.arXiv preprint arXiv:2203.05115(2022)

    2022 arXiv

  12. [19]

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Gener- ation for Knowledge-Intensive NLP Tasks. InAdv...

    2020

  13. [20]

    Ruosen Li and Xinya Du. 2023. Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning. InFindings of the Association for Computational Linguistics: EMNLP. 6779–6789

    2023

  14. [21]

    Rui Li, Chaozhuo Li, Yanming Shen, Zeyu Zhang, and Xu Chen. 2024. Generalizing Knowledge Graph Embedding with Universal Orthogonal Parameterization. In Proceedings of the 41st International Conference on Machine Learning, Vol. 235. 28040–28059

    2024

  15. [22]

    Yanming Liu, Xinyue Peng, Tianyu Du, Jianwei Yin, Weihao Liu, and Xuhong Zhang. 2024. ERA-CoT: Improving Chain-of-Thought through Entity Relation- ship Analysis. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ...

    2024

  16. [23]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. InProceedings of the 61st Annual Meeting of the Association for Comput...

    2023

  17. [24]

    Kanishka Misra, Cicero Nogueira dos Santos, and Siamak Shakeri. 2023. Trigger- ing Multi-Hop Reasoning for Question Answering in Language Models using Soft Prompts and Random Walks.arXiv preprint arXiv:2306.04009(2023)

    2023 arXiv

  18. [25]

    Kamal Nigam and Rayid Ghani. 2000. Analyzing the Effectiveness and Applicabil- ity of Co-training. InProceedings of the 2000 ACM CIKM International Conference on Information and Knowledge Management. 86–93

    2000

  19. [26]

    OpenAI. 2022. Introducing chatgpt.https:// openai.com/ blog/ chatgpt(2022)

    2022

  20. [27]

    Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, and Prathosh A P. 2024. HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2024

  21. [28]

    Jinyoung Park, Ameen Patel, Omar Zia Khan, Hyunwoo J Kim, and Joo-Kyung Kim. 2023. Graph-guided reasoning for multi-hop question answering in large language models.arXiv e-prints arXiv:2311.09762(2023)

    2023 arXiv

  22. [29]

    Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024. Graph Retrieval-Augmented Generation: A Survey.arXiv preprint arXiv:2408.08921(2024)

    2024 arXiv

  23. [30]

    Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela

  24. [31]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. InFindings of the Association for Computational Linguistics: EMNLP. 5687–5711

    2023

  25. [32]

    Peng Qi, Haejun Lee, Oghenetegiri Sido, Christopher D Manning, et al . 2020. Answering open-domain questions of varying reasoning steps from text.arXiv preprint arXiv:2010.12527(2020)

    2020 arXiv

  26. [33]

    Matthew Renze and Erhan Guven. 2024. The effect of sampling temperature on problem solving in large language models.arXiv preprint arXiv:2402.05201 (2024)

    2024 arXiv

  27. [34]

    Stephen Robertson, Hugo Zaragoza, et al . 2009. The probabilistic relevance framework: BM25 and beyond.Foundations and Trends®in Information Retrieval 3, 4 (2009), 333–389

    2009

  28. [35]

    Bhaskarjit Sarmah, Benika Hall, Rohan Rao, Sunil Patel, Stefano Pasquali, and Dhagash Mehta. 2024. HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction.arXiv preprint arXiv:2408.04948(2024)

    2024 arXiv

  29. [36]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing Retrieval-Augmented Large Language Models with KDD ’25, August 3–7, 2025, Toronto, ON, Canada Rui Li, Quanyu Dai, Zeyu Zhang, Xu Chen, Zhenhua Dong, and Ji-Rong Wen Iterative Retrie...

    2023

  30. [37]

    Chi, Nathanael Schärli, and Denny Zhou

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. Large Language Models Can Be Easily Distracted by Irrelevant Context. InInternational Conference on Machine Learning, Vol. 202. 31210–31227

    2023

  31. [38]

    Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J

    Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Ha...

    2024

  32. [39]

    Xin Su, Tiep Le, Steven Bethard, and Phillip Howard. 2023. Semi-Structured Chain-of-Thought: Integrating Multiple Sources of Knowledge for Improved Language Model Reasoning.arXiv preprint arXiv:2311.08505(2023)

    2023 arXiv

  33. [40]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Trac...

    2021

  34. [42]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  35. [43]

    Siyuan Wang, Zhongyu Wei, Jiarong Xu, Taishan Li, and Zhihao Fan. 2023. Unify- ing structure reasoning and language model pre-training for complex reasoning. arXiv preprint arXiv:2301.08913(2023)

    2023 arXiv

  36. [44]

    Transactions of the Association for Computational Linguistics10 (2022), 539–554

    MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics10 (2022), 539–554

    2022

  37. [45]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in Neural Information Processing Systems

    2022

  38. [46]

    Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025. InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales. InThe Thir- teenth International Conference on Learning Representations

    2025

  39. [47]

    Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural Text Generation With Unlikelihood Training. In8th International Conference on Learning Representations

    2020

  40. [48]

    Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph Embedding by Translating on Hyperplanes. InProceedings of the Twenty- Eighth AAAI Conference on Artificial Intelligence. 1112–1119

    2014

  41. [49]

    Diji Yang, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Jie Yang, and Yi Zhang. 2024. IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Infor...

    2024

  42. [50]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Langua...

    2018

  43. [51]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InThe Eleventh International Conference on Learning Representations

    2023

  44. [52]

    Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Training Large Langu...

    2024

  45. [53]

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning math- ematical reasoning with large language models.arXiv preprint arXiv:2308.01825 (2023)

    2023 arXiv

  46. [55]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al . 2023. Siren’s song in the AI ocean: a survey on hallucination in large language models.arXiv preprint arXiv:2309.01219(2023)

    2023 arXiv

  47. [56]

    Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. 2023. Answering Questions by Meta-Reasoning over Multiple Chains of Thought. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 5942–5966

    2023

  48. [60]

    - Birmingham: Find out where Birmingham is located

    Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open- domain question answering.arXiv preprint arXiv:2101.00774(2021). A Discussion on More Related Works This section provides a de...

    2021 arXiv

  49. [2020]

    Unsupervised question decomposition for question answering.arXiv preprint arXiv:2002.09758(2020)

    2020 arXiv

  50. [2022]

    Interpretable AMR-based question decomposition for multi-hop question answering.arXiv preprint arXiv:2206.08486(2022)

    2022 arXiv

  51. [2023]

    InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge- Intensive Multi-Step Questions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10014–10037

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.