Pith. sign in

REVIEW 4 major objections 5 minor 102 references

Xinyu AI Search: Enhanced Relevance and Comprehensive Results with Rich Answer Presentations

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Xinyu AI Search outperforms eight rival generative search technologies in human evaluations, with the highest scores for comprehensiveness and insightfulness.

desk verdict A detailed engineering blueprint for a generative search engine; the system description is the real contribution, and the superiority claim is not statistically supported. read the letter →

arxiv 2505.21849 v1 pith:446XG5NJ submitted 2025-05-28 cs.IR cs.AI

Bo Tang , Junyi Zhu , Chenyang Xi , Yunhang Ge , Jiahao Wu , Yuchen Feng , Yijun Niu , Wenqiang Wei
show 10 more authors
Yu Yu Chunyu Li Zehao Lin Hao Wu Ning Liao Yebin Yang Jiajia Wang Zhiyu Li Feiyu Xiong Jingrun Chen
This is my paper
classification cs.IRcs.AI
keywords generativeAIsearchengineretrieval-augmentedgenerationquery-decompositiongraphbuilt-incitationtimelinevisualizationtextual-visualchoreographymulti-sourceretrievalhumanevaluationofanswers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Generative AI search engines answer questions with synthesized text, but their answers lose relevance on complex queries, omit key facts, and present information in ways that are hard to trace. This paper presents Xinyu, a generative search engine that decomposes each complex query into a graph of dependent sub-queries, retrieves and reranks passages for every sub-query, and then composes a final answer supplemented with sentence-level citations, a timeline of events, and images matched to individual paragraphs. The paper claims that on 300 real-world Chinese queries rated by journalism-trained experts, Xinyu beats eight rival systems, achieving the highest overall score (9.235) and the largest advantages in comprehensiveness (9.143) and insightfulness (7.333). The paper also positions itself as the first full-stack disclosure of a deployed generative search engine, which matters because the winning design is an orchestration of fine-tuned open-source models rather than a single proprietary model, making the approach reproducible.

What carries the argument

The load-bearing object is the query-decomposition graph (QDG), a directed graph in which nodes are sub-queries and edges are dependencies; a fine-tuned LLM builds it by choosing among chain decomposition (sequential, where parent answers inform children), split decomposition (independent parallel sub-queries), or terminal (no decomposition needed). The QDG fixes the execution order of the whole pipeline: retrieval happens per sub-query, answers are generated parents-first with ancestor Q&A inserted as context, and the final response is assembled from all sub-answers. Around this core, three named presentation mechanisms carry the user-facing claims: the built-in citation scheme (a small model extracts entities sentence by sentence, a second small model attaches the best source document, with an embedding-similarity fallback), timeline visualization (event extraction with timestamps, embedding-based deduplication, grouping, and chronological sorting), and textual-visual choreography (image filtering, a three-way paragraph-image relevance score, and Hungarian-algorithm assignment of images to paragraphs).

What would settle it

Re-run the nine-system comparison on a new query set (for instance, 300 English or non-news queries) with raters who cannot see which system produced each answer, and report per-rater agreement; the claim would be refuted if the average gap in comprehensiveness and insightfulness disappears once answer length, timeline structure, images, and citation density are statistically controlled, or once ratings are blinded.

Watch

Extended reading notes

Core claim

The central claim is that a search engine built around a query-decomposition graph resolves the three weaknesses of current generative AI search: insufficient relevance, incomplete coverage, and poor answer presentation. Xinyu breaks a complex query into interdependent sub-queries, retrieves and filters passages for each, generates sub-answers in dependency order so that child questions build on parent answers, and only then assembles the final response. On top of this pipeline it attaches citations at sentence granularity by extracting entities and matching them to retrieved documents, builds a timestamped event timeline from the retrieved passages, and places images next to the paragraphs they illustrate using a learned similarity score. Evaluated by journalism-trained raters on 300 Chinese queries spanning eight domains, Xinyu scores highest overall (9.235 versus 8.810 for the closest rival) and highest on comprehensiveness (9.143) and insightfulness (7.333). Ablation studies attribute the gains to the query-decomposition graph, passage reranking, and the presentation modules, and the authors report that removing any single module does not always improve every metric, with the integrated system winning overall.

Load-bearing premise

The load-bearing premise is that 300 Chinese queries rated by journalism-trained experts, without reported inter-annotator agreement, variance, significance testing, or evidence of blinding, measure search answer quality well enough to support the headline outperformance claim; if those raters systematically preferred long, structured, presentation-rich answers, the result would not generalize to other query distributions or languages.

Editorial extensions

If this is right

  • A competitive generative search engine can be assembled from openly available models — Qwen2.5 variants of 3B to 72B parameters, bge-large-zh embeddings, and bge-reranker-v2-m3 — fine-tuned on datasets of 5K to 37K examples per task, rather than requiring a proprietary foundation model.
  • The presentation modules are not cosmetic: removing built-in citation, timeline visualization, or textual-visual choreography each measurably lowers comprehensiveness ratings, and removing images also lowers clarity.
  • Because the QDG records dependencies between sub-queries and generation follows parents-before-children order, the same pipeline absorbs more complex questions by adding nodes, without redesigning retrieval or generation.
  • The ablation results show the modules interact: omitting query expansion can actually improve relevance on some queries, yet the full system still wins overall, so the value of each component is in combination rather than in isolation.
  • A deployed English version, produced simply by translating the Chinese prompts, reportedly works 'surprisingly well' with the same multilingual models, suggesting the orchestration transfers across languages before any language-specific fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation that carries the headline comparison — 300 Chinese queries, journalism-trained raters, no reported inter-annotator agreement, no variance or significance testing, and no evidence of blinding — leaves open the possibility that the 0.425-point average gap reflects preference for long, structured, presentation-rich answers rather than substantive superiority; a blinded cross-language r
  • The query-decomposition graph is a transferable device: any generation task that benefits from splitting a question into dependent sub-questions, such as legal analysis, medical triage, or competitive intelligence, could reuse the same graph construction and dependency-ordered generation.
  • The sentence-level citation scheme doubles as an audit mechanism: because every sentence must name a supporting document, the system makes each claim individually checkable, which is a stronger hallucination-containment property than paragraph-level citation and could be reused as a verification tool in other RAG systems.
  • The deduplication and citation thresholds (0.8 passage similarity, 0.9 timeline merge, 0.6 citation fallback) are set without sensitivity analysis, so a test that varies these constants would show how much of the presentation gain depends on their exact values.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents Xinyu AI Search, a generative AI search engine that combines a query-decomposition graph (QDG), multi-source retrieval with query expansion, passage deduplication/selection/re-ranking, and three presentation components: fine-grained built-in citations, timeline visualization, and textual-visual choreography. The authors report a full-stack system description with prompts, fine-tuning data sizes, and ablations. The central empirical claim is that, on 300 recent Chinese queries rated by journalism-trained experts, Xinyu outperforms eight existing generative AI search engines and conversational LLM+RAG systems, with the largest advantages in comprehensiveness and insightfulness. The paper also reports component-level comparisons for citations, timelines, and image placement, and ablation studies for most sub-modules.

Significance. If the performance claims were rigorously established, this would be a valuable full-stack reference for generative AI search engine design. The paper is unusually transparent: it discloses concrete prompts (App. B), data-construction procedures and dataset sizes (App. D), content-filtering rules (App. C), and threshold choices. The ablation studies, while mostly LLM-evaluated, provide a reasonable first-order check of the contribution of individual modules. However, the headline comparative claim rests on a single human evaluation that lacks statistical testing, inter-annotator agreement, and blinding, and the winning dimensions coincide with the modules whose presentation style is most distinctive. The evidence as it stands is suggestive but not sufficient to establish that Xinyu 'outperforms' eight systems in the strong sense claimed.

major comments (4)
  1. [Sec. 4.1.1, Table 2] The paper states that Xinyu 'significantly outperforms other methods' in comprehensiveness (9.143 vs. 8.252) and insightfulness (7.333 vs. 6.796), but no statistical tests, confidence intervals, or variance measures are reported. On two metrics, Conciseness (9.813 vs. 9.851) and Coherence (9.810 vs. 9.853), Xinyu scores below Perplexity AI, so the pattern is not uniform even in the aggregate. Without per-query scores, error bars, or pairwise significance tests, the observed differences cannot be distinguished from rating noise, and the central superiority claim is not established.
  2. [Sec. 4.1.1] The human evaluation reports no measure of inter-annotator agreement. The paper does not state how many expert raters scored each answer, whether ratings were collected independently, or any agreement coefficient (e.g., Cohen's kappa or Krippendorff's alpha). For subjective criteria such as Insightfulness, unreported reliability leaves the numerical scores of unknown quality; this is load-bearing because the entire comparative claim derives from these ratings.
  3. [Sec. 4.1.1 and Fig. 4] No blinding procedure is described. Xinyu's answers are visually distinctive due to timeline panels and embedded images, so expert raters could readily identify which system produced which answer. The two dimensions where Xinyu's advantage is largest, Comprehensiveness and Insightfulness, are precisely the dimensions addressed by the timeline and textual-visual modules, so the comparison risks measuring presentation style rather than answer content. The authors should describe how they controlled for this (e.g., rendering all outputs in plain text or using a blinded crowd protocol).
  4. [Sec. 4.1.1, 4.1.2, and Table 1] The word 'significantly' appears repeatedly (Tables 3, 4, and 5) without any accompanying statistical test, and Table 1 reports Pearson correlations between human and LLM scores that appear to be computed on the aggregate mean values of Tables 2 and 15 rather than on per-item pairs. This overstates the agreement between the two evaluation methods and is not an adequate substitute for per-answer validation. The authors should either provide proper statistical support or soften the 'outperforms' and 'significantly' claims to descriptive observations.
minor comments (5)
  1. [Fig. 3] The figure label 'Gropu2 Title' contains a typo; it should read 'Group2 Title'.
  2. [Sec. 3.3.2 and Sec. 3.5.1] Several thresholds are stated without sensitivity analysis: the chunk size of 350 with 25% overlap, the passage retention ratio of 70% (Sec. 3.4.2), the citation fallback cosine threshold of 0.6, and the image-relevance threshold of 0.3. Since these choices are referenced as design decisions, a brief robustness check would increase confidence in the pipeline's reported performance.
  3. [Sec. 4.1.1] The evaluation is conducted exclusively on Chinese queries, while the paper also claims to have an English version. No English-language evaluation is reported, so the external validity of the 'outperforms eight existing technologies' claim is limited to one language and one test-query distribution.
  4. [Table 14 (App. E)] The evaluation protocol for the fine-tuned reranker is unclear: the metrics Precision, Recall, and F1 are reported without a definition of the underlying task (e.g., relevant-passage ranking at a fixed cutoff, or binary classification). A precise task definition would make this ablation interpretable.
  5. [Abstract and Sec. 1.2] The claim of being 'the first comprehensive framework for generative AI search engines' is strong and should be supported with a more explicit comparison to prior system descriptions and open-source projects, or softened to 'one of the first'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical system comparison against external baselines, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

The central claim is that Xinyu outperforms eight existing technologies in human assessments (Sec. 4.1.1, Table 2). This is an external evaluation, not a derivation: the target quantity (human-rated quality according to nine criteria) is not defined in terms of Xinyu's modules, and no parameter is fitted so that the reported advantage is enforced by construction. The ablation studies (Tables 6, 7, 14-16, Fig. 5) compare variants against the full system, but the evaluation criteria are not the optimization objective of the ablation; they are independent human and LLM judgments. The LLM-judge protocol is validated against human scores (Table 1), so it is anchored rather than circular. The closest self-citation is reference [82] in the fine-tuning related work; it is cited merely as an example of fine-tuning and is not load-bearing, and the paper's claims do not depend on its truth. Possible methodological concerns raised by a skeptical reading, such as no reported inter-annotator agreement, no significance testing, no evidence of blinding, and a query-domain skew toward news and history, are validity threats rather than circularity, because none of them shows a prediction reducing to its own input by construction. There is no self-definitional step, no fitted input called a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The paper is self-contained against external systems and human judgment, so the circularity score is 0.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical and does not rest on derived constants, but it depends on several hand-chosen thresholds (0.8, 0.9, 0.6, 0.3, 70%, 6) and on domain assumptions about representativeness of the queries and validity of human ratings. These are standard practice in system papers, but they are not independently grounded, so they are listed as free parameters and assumptions.

free parameters (9)
  • Passage deduplication threshold = 0.8
    Cosine similarity above which two passages are deemed duplicates (Sec. 3.4.1).
  • Timeline event merge threshold = 0.9
    Cosine similarity above which timeline events are merged (Sec. 3.5.2).
  • Citation fallback threshold = 0.6
    Cosine similarity above which a sentence without extracted entities is assigned a citation (Sec. 3.5.1).
  • Image relevance threshold = 0.3
    Reranker score below which images are filtered out (Sec. 3.5.3).
  • Chunk size = 350
    Passage token size for embedding (Sec. 3.3.2).
  • Chunk overlap = 25%
    Overlap between chunks (Sec. 3.3.2).
  • Passage retention ratio = 70%
    Top 70% of passages kept in selection (Sec. 3.4.2).
  • Max sub-queries = 6
    QDG prompt limits decomposition to at most 6 sub-queries (App. B.1).
  • Text-image matching weights = Not specified
    Three similarity measures are combined with a weighted average (Sec. 3.5.3), but the weights are not given.
assumptions (5)
  • domain assumption Human subjective ratings with the nine criteria are a valid measure of answer quality.
    The main evaluation relies on expert ratings rather than automatic metrics; this assumes the criteria reflect real user value.
  • domain assumption The 300 collected Chinese queries are representative of the user population and task distribution.
    Sec. 4 states queries cover eight domains and many trending news topics, but representativeness is not demonstrated.
  • domain assumption Fine-tuning on synthetic data curated by stronger models improves model quality for the target tasks.
    The system relies on fine-tuned Qwen models; ablation supports this for their setup, but the general principle is assumed.
  • domain assumption Multi-source retrieval and query expansion increase answer diversity and quality.
    This motivates the retrieval design, but its benefit is only tested via removal in ablation, not against alternatives.
  • standard math The maximum independent set problem for deduplication can be solved by the described greedy algorithm without harming quality.
    Sec. 3.4.1 uses a greedy approximation for an NP-hard problem; the approximation factor is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Xinyu AI Search: Enhanced Relevance and Comprehensive Results with Rich Answer Presentations." pith.science (2026). https://pith.science/paper/446XG5NJ

@misc{pith2026250521849,
  author       = {Pith},
  title        = {Pith review of: Xinyu AI Search: Enhanced Relevance and Comprehensive Results with Rich Answer Presentations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/446XG5NJ}},
  note         = {Machine review of arXiv:2505.21849}
}
read the original abstract

Traditional search engines struggle to synthesize fragmented information for complex queries, while generative AI search engines face challenges in relevance, comprehensiveness, and presentation. To address these limitations, we introduce Xinyu AI Search, a novel system that incorporates a query-decomposition graph to dynamically break down complex queries into sub-queries, enabling stepwise retrieval and generation. Our retrieval pipeline enhances diversity through multi-source aggregation and query expansion, while filtering and re-ranking strategies optimize passage relevance. Additionally, Xinyu AI Search introduces a novel approach for fine-grained, precise built-in citation and innovates in result presentation by integrating timeline visualization and textual-visual choreography. Evaluated on recent real-world queries, Xinyu AI Search outperforms eight existing technologies in human assessments, excelling in relevance, comprehensiveness, and insightfulness. Ablation studies validate the necessity of its key sub-modules. Our work presents the first comprehensive framework for generative AI search engines, bridging retrieval, generation, and user-centric presentation.

Figures

Figures reproduced from arXiv: 2505.21849 by the authors.

Figure 1
Figure 1. Online evaluation (on February 5th) of the Xinyu AI search system for the query ‘Trump administration latest actions.’ [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Common issues in generative AI search answers. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Xinyu AI search framework. The upper row illustrates the full response pipeline, while the lower row provides a more detailed depiction of several novel approaches integrated into this framework. LRe (𝜃) = −E(𝑥,𝑥+,𝑥 − 1:𝑁 )∼D " log 𝑒 ℎ𝜃 (𝑥,𝑥+ ) 𝑒 ℎ𝜃 (𝑥,𝑥+ ) + Í𝑁 𝑖=1 𝑒 ℎ𝜃 (𝑥,𝑥 − 𝑖 ) # . (2) 3.1 Query preprocessing When a query is input by the user, initial steps are conducted to ensure that the query is safe and harm… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Domain distribution of 300 test queries. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Ablation study of sub-modules for the rich answer [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Xinyu’s Interface for query disambiguation. Sensitive Information Filtering. Personal identifiers, such as phone numbers, email addresses, and platform-specific markers are de￾tected and removed. Text Normalization. Punctuation is standardized to half-width characters,…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

102 extracted references · 36 canonical work pages

  1. [1]

    Baichuan AI. 2024. Baichuan. https://ying.baichuan-ai.com/ Accessed: 2025-02- 05

    2024

  2. [2]

    Baidu AI. 2024. Yiyan. https://yiyan.baidu.com/ Accessed: 2025-02-05

    2024

  3. [3]

    Lepton AI. 2025. Search with Lepton. https://github.com/leptonai/search_with_ lepton Accessed: 2025-02-02

    2025

  4. [4]

    Moonshot AI. 2024. KIMI. https://kimi.moonshot.cn/ Accessed: 2025-02-05

    2024

  5. [5]

    Perplexity AI. 2024. Perplexity AI. https://www.perplexity.ai/ Accessed: 2025- 02-05

    2024

  6. [6]

    Tiangong AI. 2024. Tiangong AI. http://tiangong.cn/ Accessed: 2025-02-05

    2024

  7. [7]

    Tongyi AI. 2024. Tongyi. https://tongyi.ai/ Accessed: 2025-02-05

    2024

  8. [8]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. 2024. A Survey on Data Selection for Language Models.Transactions on Machine Learn- ing Research (2024). https://openreview.net/forum?id...

    2024

Show all 102 references
  1. [9]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations

    2024

  2. [10]

    Orlando Ayala and Patrice Bechard. 2024. Reducing hallucination in structured outputs via Retrieval-Augmented Generation. In Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume...

    2024 doi

  3. [11]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016). 8 Xinyu AI Searc...

    2016 arXiv

  4. [12]

    Alec Berntson. 2023. Azure AI Search: Outperforming Vector Search with Hybrid Retrieval and Reranking. https://techcommunity.microsoft.com/blog/azure- ai-services-blog/azure-ai-search-outperforming-vector-search-with-hybrid- retrieval-and-reranking/3929167 Accessed: 2025-02-01

    2023

  5. [13]

    Sergey Brin and Lawrence Page. 1998. The Anatomy of a Large-Scale Hypertex- tual Web Search Engine. Computer Networks 30 (1998), 107–117

    1998

  6. [14]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  7. [15]

    Harrison Chase. 2022. LangChain. https://github.com/langchain-ai/langchain

    2022

  8. [16]

    ChatGLM. 2024. ChatGLM. https://chatglm.cn/ Accessed: 2025-02-05

    2024

  9. [17]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

  10. [18]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017)

    2017

  11. [19]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference o...

    2024

  12. [20]

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open- source legal large language model with integrated external knowledge bases. CoRR (2023)

    2023

  13. [21]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  14. [22]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. Chain-of-Verification Reduces Hallucina- tion in Large Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre...

    2024

  15. [23]

    Xinya Du and Heng Ji. 2022. Retrieval-Augmented Generative Question An- swering for Event Argument Extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Co...

    2022 doi

  16. [24]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise Zero-Shot Dense Retrieval without Relevance Labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Anna Rogers, Jordan Boyd-Graber, and Naoak...

    2023

  17. [25]

    Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023. Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:2303.14524 (2023)

    2023 arXiv

  18. [26]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs.CL]

    2024 arXiv

  19. [27]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A Survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594 (2024)

    2024 arXiv

  20. [28]

    Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering. In The Thirty-eighth Annual Conference on Neural Informatio...

    2024

  21. [29]

    IBM. 2024. Metadata Enrichment: Highly Scalable Data Classification and Data Discovery. https://www.ibm.com/think/insights/metadata-enrichment-highly- scalable-data-classification-and-data-discovery Accessed: 2025-01-19

    2024

  22. [30]

    Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bender- sky. 2023. Query expansion by prompting large language models. arXiv preprint arXiv:2305.03653 (2023)

    2023 arXiv

  23. [31]

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2024. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems 36 (2024)

    2024

  24. [32]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024 doi

  25. [33]

    Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active Retrieval Augmented Generation. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing , Houda Bouamor, ...

    2023 doi

  26. [34]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  27. [35]

    Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang

  28. [36]

    H. W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2, 1-2 (1955), 83–97. doi:10.1002/nav.3800020109 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/nav.3800020109

    1955 doi

  29. [37]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  30. [38]

    Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023. Making Large Lan- guage Models A Better Foundation For Dense Retrieval. arXiv:2312.15503 [cs.CL]

    2023

  31. [39]

    Xinze Li, Zhenghao Liu, Chenyan Xiong, Shi Yu, Yu Gu, Zhiyuan Liu, and Ge Yu

  32. [40]

    Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. 2024. Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources. In The Twelfth International Conference on Learning Representations

    2024

  33. [41]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. doi:10.1162/tacl_a_00638

    2024 doi

  34. [43]

    Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Augmented large language models with parametric knowledge guiding. arXiv preprint arXiv:2305.04757 (2023)

    2023 arXiv

  35. [44]

    Xinbei Ma, Yeyun Gong, Pengcheng He, hai zhao, and Nan Duan. 2023. Query Rewriting in Retrieval-Augmented Large Language Models. In The 2023 Confer- ence on Empirical Methods in Natural Language Processing . https://openreview. net/forum?id=gXq1cwkUZc

    2023

  36. [45]

    Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692 364 (2019)

    2019 arXiv

  37. [46]

    Yubo Ma, Yixin Cao, Yong Ching Hong, and Aixin Sun. 2023. Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  38. [47]

    Manning, Prabhakar Raghavan, and Hinrich Schütze

    Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Intro- duction to Information Retrieval . Cambridge University Press

    2008

  39. [48]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query Rewriting in Retrieval-Augmented Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Associ...

    2023

  40. [49]

    Metaso. 2024. Metaso. https://metaso.cn/ Accessed: 2025-02-05

    2024

  41. [50]

    Microsoft Azure Architecture Center. 2024. Developing a RAG Solution - Chunk Enrichment Phase. https://learn.microsoft.com/en-us/azure/architecture/ai- ml/guide/rag/rag-enrichment-phase Accessed: 2025-01-19

    2024

  42. [51]

    Richard E. Mayer. 2014. Cognitive Theory of Multimedia Learning. In The Cambridge Handbook of Multimedia Learning , Richard E. Mayer (Ed.). Cambridge University Press, Cambridge, 43–71

    2014

  43. [52]

    Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2024. When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikuma...

    2024 doi

  44. [53]

    OpenAI. 2024. ChatGPT. https://chatgpt.com/ Accessed: 2025-02-05

    2024

  45. [54]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al

  46. [55]

    Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. 2024. Large language model based long-tail query rewriting in taobao search. In Companion Proceedings of the ACM on Web Conference 2024. 20–28. 9 Preprint, Bo Tang and Ju...

    2024

  47. [56]

    Prieto, Kshitij Sharma, Łukasz Kidzinski, María Jesús Rodríguez-Triana, and Pierre Dillenbourg

    Luis P. Prieto, Kshitij Sharma, Łukasz Kidzinski, María Jesús Rodríguez-Triana, and Pierre Dillenbourg. 2018. Multimodal Teaching Analytics: Automated Extrac- tion of Orchestration Graphs from Wearable Sensor Data. Journal of Computer Assisted Learning 34, 2 (April 2018), 193–...

    2018 doi

  48. [57]

    Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training. https://api.semanticscholar.org/CorpusID:49313245

    2018

  49. [58]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35...

    2022

  50. [59]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.Journal of Machine Learning Research 21, 140 (2020), 1–67. http...

    2020

  51. [60]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing Retrieval-Augmented Large Language Models with It- erative Retrieval-Generation Synergy. In Findings of the Association for Com- putational Linguistics: EMNLP 2023 , Houda Bouamor, ...

    2023 doi

  52. [61]

    Statista. 2023. https://www.statista.com/statistics/1377993/us-adults-ai- powered-search-engines-usage-choice/ Accessed: 2025-01-21

    2023

  53. [62]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019)

    2019

  54. [63]

    Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436 (2023)

    2023 arXiv

  55. [64]

    John Sweller, Paul Ayres, and Slava Kalyuga. 2020. Cognitive load theory and educational technology. Educational Technology Research and Development 68, 1 (2020), 1–16. doi:10.1007/s11423-019-09701-3

    2020 doi

  56. [65]

    Ravi Theja. 2023. Evaluating the Ideal Chunk Size for a RAG System using LlamaIndex. https://www.llamaindex.ai/blog/evaluating-the-ideal-chunk-size- for-a-rag-system-using-llamaindex-6207e5d3fec5 Accessed: 2025-01-19

    2023

  57. [66]

    Winnie Street, John Oliver Siy, Geoff Keeling, Adrien Baranes, Benjamin Barnett, Michael McKibben, Tatenda Kanyere, Alison Lentz, Robin IM Dunbar, et al. 2024. LLMs achieve adult human performance on higher-order theory of mind tasks. arXiv preprint arXiv:2405.18870 (2024)

    2024 arXiv

  58. [67]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  59. [68]

    Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query Expansion with Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguist...

    2023 doi

  60. [69]

    Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, and Wei Wang. 2023. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308.11761 (2023)

    2023 arXiv

  61. [70]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  62. [71]

    In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.)

    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge- Intensive Multi-Step Questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.)...

    doi

  63. [72]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597 [cs.CL]

    2023 arXiv

  64. [73]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  65. [74]

    Haoyan Yang, Zhitao Li, Yong Zhang, Jianzong Wang, Ning Cheng, Ming Li, and Jing Xiao. 2023. PRCA: Fitting Black-Box Large Language Models for Re- trieval Question Answering via Pluggable Reward-Driven Contextual Adapter. In Proceedings of the 2023 Conference on Empirical Meth...

    2023 doi

  66. [75]

    Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin

  67. [76]

    arXiv preprint arXiv:2308.13387

    Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs. arXiv preprint arXiv:2308.13387

    arXiv

  68. [77]

    Weiqi Wu, Shen Huang, Yong Jiang, Pengjun Xie, Fei Huang, and Hai Zhao

  69. [78]

    Liangyu Zha, Junlin Zhou, Liyao Li, Rui Wang, Qingyi Huang, Saisai Yang, Jing Yuan, Changbao Su, Xiang Li, Aofeng Su, et al . 2023. Tablegpt: Towards unifying tables, nature language and commands into one gpt. arXiv preprint arXiv:2307.08674 (2023)

    2023 arXiv

  70. [79]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv prep...

    2023 arXiv

  71. [80]

    Chi, Quoc V Le, and Denny Zhou

    Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024. Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. In The Twelfth International Conference on Learning Representations

    2024

  72. [81]

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The Eleventh International Conference on ...

    2023

  73. [82]

    Zhilin Yang. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv preprint arXiv:1906.08237 (2019)

    2019 arXiv

  74. [83]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language...

    2018 doi

  75. [84]

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making Retrieval- Augmented Language Models Robust to Irrelevant Context. In The Twelfth Inter- national Conference on Learning Representations

    2024

  76. [89]

    Blaschko

    Junyi Zhu, Shuochen Liu, Yu Yu, Bo Tang, Yibo Yan, Zhiyu Li, Feiyu Xiong, Tong Xu, and Matthew B. Blaschko. 2024. FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2024 ,...

    2024 doi

  77. [90]

    several" or

    Shengyao Zhuang, Bing Liu, Bevan Koopman, and Guido Zuccon. 2023. Open- source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalik...

    2023

  78. [91]

    complex query

    **Classify the nature of the query** - The query can be classified into one of two types: (a) A "complex query" that consists of multiple sub-queries. (b) A "simple query" that can be directly answered. - If the query is classified as "complex," set 'is_complex ' to **True**. ...

  79. [92]

    complex,

    **Decomposing a Complex Query** - If the query is classified as "complex," break it down into **sub-queries** and store them in the 'sub_queries ' list. - Decomposition principles:

  80. [93]

    What are the latest social news and weather news in Shanghai?

    If a query contains multiple **target entities**, split it into multiple sub-queries. - Example: *"What are the latest social news and weather news in Shanghai?"* - Target entities: *"social news"*, *"weather news"*. - Split into: *"What are the latest social news in Shanghai?...

  81. [94]

    Each sub-query should be **indivisible** and should not require further decomposition

  82. [95]

    No duplicate sub-queries

  83. [96]

    What is the area and population of New Jersey, USA?

    When referring to **names of people, places, or organizations**, ensure full and precise descriptions. - Example: *"What is the area and population of New Jersey, USA?"* - Correct split: *"What is the area of New Jersey, USA?"* and *"What is the population of New Jersey, USA?"...

  84. [97]

    The total number of sub-queries **should not exceed 6**

  85. [98]

    What natural disasters occurred in Indonesia in April?

    **Analyzing Dependencies Between Sub-Queries** - If the query is complex, analyze the **dependency relationships** between sub-queries and store them in ' parent_child '. - Example: - *"What natural disasters occurred in Indonesia in April?"* - *"How long did this natural disa...

  86. [99]

    If sub-queries are **independent**, 'parent_child ' remains an empty list

  87. [100]

    What is the latest iPhone model

    If the **child question cannot be answered without the parent**, it is a dependent relationship. - Example: "What is the latest iPhone model" is the parent node of "What are the specifications of the latest iPhone?" - The first question must be answered before the second

  88. [101]

    Issues Identified

    Every possible pair of sub-queries should be evaluated for dependency. - A query can be both a *parent* and a *child* in different relationships. ### Example: {Few-Shot Examples} Query: {Query} Response: \n Additional options are prepared to present clarifying options for the ...

  89. [103]

    {Retrieved Document}

  90. [104]

    [2]", "[9]

    When making your determination, ensure that the selected reference document matches as much key information from the excerpted sentence as possible. The higher the degree of key information overlap, the more likely the reference document is the source of the excerpted sentence...

  91. [2021]

    Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332 (2021)

    2021 arXiv

  92. [2023]

    InThe 2023 Conference on Empirical Methods in Natural Language Processing

    Tree of Clarifications: Answering Ambiguous Questions with Retrieval- Augmented Large Language Models. InThe 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  93. [2024]

    arXiv:2402.03216 [cs.CL]

    BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216 [cs.CL]

    arXiv

  94. [2025]

    arXiv preprint arXiv:2501.00888 (2025)

    Unfolding the Headline: Iterative Self-Questioning for News Retrieval and Timeline Summarization. arXiv preprint arXiv:2501.00888 (2025)

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.