Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Federated Retrieval Augmented Generation for Multi-Product Question Answering

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Probabilistic federated search lifts multi-product RAG-QA

desk verdict Plausible multi-product RAG framework with a clever gating idea, but the evaluation has no described train/test split and uses the same GPT-4 for label creation and judging, so the reported gains are not yet convincing. read the letter →

arxiv 2501.14998 v1 pith:ZJU7DOIY submitted 2025-01-25 cs.CL

Parshin Shojaee , Sai Sree Harsha , Dan Luo , Akash Maharaj , Tong Yu , Yunyao Li This is my paper · ORCID
classification cs.CL
keywords retrievalaugmentedgenerationfederatedsearchmulti-domainquestionansweringstochasticgatingdomainrouterenterpriseproductQAaccuracyRAGevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that multi-product retrieval-augmented question answering over enterprise documentation can be improved by replacing indiscriminate all-domain search and rigid single-domain routing with a probabilistic federated search. The proposed MKP-QA framework multiplies each domain's router probability by the retriever's document relevance score, selects domains through a stochastic gate with an entropy-adaptive threshold, and aggregates the resulting scores into a unified ranking. On new datasets built from documentation for three Adobe products (Experience Platform, Target, and Customer Journey Analytics), the paper reports higher top-1 retrieval accuracy and better LLM response faithfulness and relevancy than three baselines. The paper also contributes these datasets as multi-domain RAG-QA benchmarks for the three products. A sympathetic reader would care because the claimed approach improves cross-product answers without domain-specific LLM fine-tuning.

What carries the argument

The central object is the unified domain-aware retrieval score $U(j,q,d_i^j) = p_j \cdot s_i^j$, the product of the domain router's sigmoid probability for domain $j$ and the bi-encoder retriever's dot-product similarity for document $d_i^j$. This product is the mechanism that fuses routing softness with passage relevance. Around it sits the stochastic gate $G(q,j) = \mathrm{Bernoulli}(\min(1, p_j/\tau(q)))$, with adaptive threshold $\tau(q) = \tau_0(1 - H(q)/\log m)$, where $H(q)$ is the entropy of the router's probability vector; the gate decides which domains are searched, providing exploration of low-confidence domains while still exploiting high-confidence ones. The federated search then takes the top-$k$ documents by $U$ across the active domains. This machinery is what the paper credits for both retrieval accuracy and response quality gains.

What would settle it

Re-run the evaluation on a held-out set of query-document pairs from the same three Adobe products that were excluded from fine-tuning the router and retriever. If MKP-QA's Acc@Top1 falls to the level of the unified-index baseline on that held-out set, the reported advantage is an artifact of training on the test data; a further check would test whether any evaluation query-document pair appears verbatim in the fine-tuning set.

Watch

Extended reading notes

Core claim

The paper's central claim is that MKP-QA "significantly boosts multi-product RAG-QA performance in terms of both retrieval accuracy and response quality" relative to unified index search (UIS), hard router filtering (RFS), and zero-shot LLM resource selection (LFS). In MKP-QA, a fine-tuned BERT-based domain router outputs query-domain probabilities $p_1,\dots,p_m$, and a Sentence-BERT bi-encoder trained with a symmetric InfoNCE loss produces query-document scores $s_i^j$. These are combined into a unified score $U(j,q,d_i^j)=p_j \cdot s_i^j$, and an active set of domains is sampled as $G(q,j) \sim \mathrm{Bernoulli}(\min(1, p_j/\tau(q)))$, where the threshold $\tau(q)$ shrinks as the router's entropy rises. The final top-$k$ documents are those with the largest $U$ across active domains. The paper reports that this construction outperforms all three baselines on both uni-domain and cross-domain subsets of the Adobe datasets, with the largest gaps on cross-domain queries.

Load-bearing premise

The load-bearing premise is that the queries and document pairs used for evaluation were not also used to train the domain router and retriever; the paper does not report a train/test split, so the reported Acc@Top1 gains could partly reflect memorization rather than generalization.

Editorial extensions

If this is right

  • On the Adobe datasets, MKP-QA's top-1 retrieval accuracy exceeds unified index search, hard router filtering, and zero-shot LLM resource selection, and the gap grows on cross-domain queries.
  • LLM response quality, scored by GPT-4 relevancy judgment and RAGAS faithfulness, is also higher under MKP-QA than under the three baselines.
  • The released datasets provide the first multi-product RAG-QA benchmarks for Adobe Experience Platform, Target, and Customer Journey Analytics, with both uni-domain and cross-domain query-document pairs.
  • The framework achieves its gains without domain-specific LLM fine-tuning or adaptive modules: only the domain router and the bi-encoder retriever are trained.
  • Zero-shot LLM resource selection (LFS) is the weakest baseline, suggesting that learned query-domain relevance is more reliable than a general-purpose LLM's domain judgment for these product domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the stochastic gate, so the marginal contribution of Bernoulli sampling versus deterministic thresholding is unmeasured; a follow-up ablation would tell whether exploration is the source of the cross-domain gain or whether the $p_j$ multiplication alone carries it.
  • Because negative pairs are sampled from the same or closely related documentation pages, the benchmark is easier than a real federated search with unrelated corpora; adding unrelated distractor domains would test the unified score's ability to reject irrelevant resources.
  • The entropy-adaptive threshold implies a precision-recall trade-off across domains that the paper does not plot; varying $\tau_0$ and measuring Acc@Top1 against the number of active domains would map that frontier.
  • All response-quality judgments use GPT-4 or GPT-3.5; whether the gains persist under smaller open-weight LLMs or other enterprise product suites is an untested extrapolation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MKP-QA, a framework for multi-product retrieval-augmented question answering over enterprise product documentation. It combines a fine-tuned BERT domain router, an entropy-based adaptive threshold with stochastic gating, a Sentence-BERT bi-encoder retriever, and a federated aggregation score U = p_j·s_i^j that multiplies the domain probability by the retriever score. The authors also introduce and describe new query-document datasets for three Adobe products, with both uni-domain and cross-domain splits. Experiments compare retrieval accuracy (Acc@Top1) and response quality (relevancy and faithfulness) against unified search (UIS), router filter and search (RFS), and zero-shot LLM resource selection (LFS).

Significance. If the empirical claims were validated, the contribution would be a simple, practical mechanism for domain-aware federated retrieval in RAG pipelines, plus new benchmark datasets for an underexplored setting. The proposed architecture is straightforward and does not require domain-specific LLM fine-tuning. However, because the evaluation has fundamental validity gaps (no described train/test separation, LLM-based data creation and judgment, and an unfavorable zero-shot baseline), the significance cannot currently be assessed. The paper's strengths include a clear problem definition, a fairly detailed description of dataset construction, and the release of new datasets (pending approval).

major comments (5)
  1. [5.3] The paper never states that the evaluation in Section 5 is performed on a held-out subset of the data. Section 3 fine-tunes the router and retriever on 'our multi-domain dataset,' Section 4 describes building that dataset from ExL pages, and Section 5.3 reports results on 'our datasets' without mentioning a split. If the same query-document pairs are used for training and testing, the reported Acc@Top1 and response-quality gains may reflect memorization rather than generalization, undermining the central claim in the abstract. The authors must explicitly describe the train/test split, or if none exists, redo the evaluation with a properly held-out set.
  2. [4.3/5.2] GPT-4 is used to generate queries, annotate query-document relevance, and judge the quality of generated responses, as shown in Figures 5-8. This creates a systematic evaluation circularity: the retriever is trained on GPT-4's relevance labels, and the same model family then measures whether retrieval and responses are correct. The reported gains could be inflated by the judge preferring content that matches GPT-4's own generation style. The authors should include a human evaluation on a sample, or at least use a different model for judging than for data creation.
  3. [5.1] The LFS baseline uses GPT-4 in a zero-shot fashion for domain selection, while MKP-QA's router is fine-tuned on the target domains with labeled queries. This is not an apples-to-apples comparison: the comparison conflates the contribution of the probabilistic federated search mechanism with the benefit of supervised training on the target domain. The paper should include a supervised version of LFS, or a zero-shot variant of MKP-QA, to isolate the effect of the proposed method.
  4. [3.2/3.4] No ablation is reported for the core stochastic gating mechanism or the adaptive threshold. The unified score U = p_j·s_i^j is a simple product, and it is unclear how much of the improvement comes from the gating stochasticity and entropy-based threshold versus the simple multiplicative re-ranking. The experiments should include a deterministic top-domain baseline, a fixed-threshold variant, and possibly a rank-fusion baseline to attribute the gains.
  5. [5.2] No error bars, confidence intervals, or significance tests are reported; all results appear to come from single runs. Given that the claimed improvements are described as 'significant,' the paper needs at least multiple seeds or bootstrap intervals to support this language.
minor comments (5)
  1. [3.2] There are typos in this section: 'Bernouli' should be 'Bernoulli' and 'utilizie' should be 'utilize'.
  2. [3.1] The sentence beginning 'To effectively estimate the query-domain relevance scores.' ends with a period mid-sentence; the period should be a comma, and the sentence should continue without a break.
  3. [Figures 1 and 2] The captions for Figures 1 and 2 are minimal; they should indicate more explicitly what each panel shows, particularly which methods are compared and what the axes represent.
  4. [Section 6] The 'Path to Deployment' section reads like an extended implications or future-work discussion; consider moving it to a shorter subsection or integrating it into the conclusion to keep the paper focused.
  5. [4.3] The sentence 'This approach ensured that positive document pairs per query were designed to span different domains' is grammatically awkward; it should be clarified, for example, 'Each positive pair was designed to span different domains.'

Circularity Check

1 steps flagged · score 6.0 of 10

No train/test split is described: the retriever and router are fine-tuned on the same query-document pairs used to compute Acc@Top1, so the reported gains are in-sample fits rather than predictions.

  1. fitted input called prediction [Sections 3.3, 4.3, 5.3]
    "We fine-tune the retriever model on our multi-domain dataset using a contrastive learning approach with a symmetric supervised variant of the InfoNCE loss ... At inference, we compute the embeddings of all documents in the corpus offline and save in a vector database. For a given query, we compute its embedding and retrieve the top-k documents using similarity score search. Fig. 2 illustrates the retrieval accuracy (Acc@Top1) of our method and baselines across uni-domain and cross-domain datasets."

    The Acc@Top1 golden documents are drawn from the same query-document pairs used to build the contrastive training batches (the D+ pairs in the InfoNCE loss). Section 4.3 explains that relevance labels were annotated for query-document pairs on the ExL pages, and Section 3.3 says the retriever is fine-tuned on 'our multi-domain dataset' using those labels. Section 5.3 then reports Acc@Top1 on 'our datasets' without describing any held-out split. Hence the top-1 'prediction' for a query is evaluated against the very positive document the retriever was trained to rank above in-batch negatives for that same query. The domain router p_j is also trained on the same dataset, and the unified score U = p_j * s_i^j multiplies two in-sample fitted scores.

full rationale

The paper's methodology is not circular in a formal-derivation sense: the unified score U = p_j * s_i^j is a heuristic fusion rule rather than a theorem, the stochastic gating is an explicit modeling choice, and no load-bearing self-citation or uniqueness theorem is invoked. The circularity is empirical and concerns the evaluation protocol. The dataset creation (Section 4.3) uses GPT-4 to generate queries and annotate relevance, with SME review; the same dataset is used to fine-tune the domain router (Section 3.1) and retriever (Section 3.3); Section 5.3 reports Acc@Top1 and response quality on 'our datasets' with no train/test split described anywhere in the paper. Consequently, the central claim—that MKP-QA significantly boosts retrieval accuracy and response quality—rests on in-sample performance as written. The authors may have used a split in practice, but the paper does not state one, so the reader cannot verify that the numbers reflect generalization. This warrants a moderate-to-high circularity score (6), not higher, because the framework itself is not definitionally equivalent to its inputs and the missing split is an evaluation-protocol gap rather than an equation-level reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two trained models: a BERT-based domain router and a Sentence-BERT retriever, both fine-tuned on the new Adobe dataset. The free parameters include the base threshold tau0, the InfoNCE temperature, and top-k. The key axioms are that GPT-4-generated queries and relevance labels are accurate, that the entropy-based gating threshold helps exploration, and that dot-product similarity is a valid relevance signal. No invented entities are introduced.

free parameters (4)
  • tau0 (base threshold)
    Used in the adaptive threshold tau(q) = tau0 * (1 - entropy / log m); the value is not reported.
  • InfoNCE temperature tau
    Temperature in the contrastive retriever loss; value not reported.
  • top-k = 5
    Number of documents retrieved per active domain and in final ranking; the paper uses top-5 in baselines and 'top-k' for MKP-QA, but the exact k is not consistently specified.
  • BERT/SBERT fine-tuning hyperparameters
    Learning rate, epochs, batch size, and negative sampling strategy are not specified.
assumptions (4)
  • domain assumption GPT-4-generated queries and relevance labels are accurate enough to serve as ground truth.
    Queries and binary relevance labels are generated by GPT-4 and only reviewed by experts; Section 4.3.
  • ad hoc to paper The entropy-based adaptive threshold improves exploration-exploitation.
    Section 3.2 introduces this heuristic without theoretical or empirical validation.
  • domain assumption Dot-product similarity between query and document embeddings is a valid relevance signal.
    Section 3.3 uses dot-product in InfoNCE and at inference; no normalization or calibration is discussed.
  • domain assumption The router and retriever generalize to the test queries.
    Section 5 implicitly assumes the trained models transfer to the evaluation queries, but no split is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated Retrieval Augmented Generation for Multi-Product Question Answering." pith.science (2026). https://pith.science/paper/ZJU7DOIY

@misc{pith2026250114998,
  author       = {Pith},
  title        = {Pith review of: Federated Retrieval Augmented Generation for Multi-Product Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZJU7DOIY}},
  note         = {Machine review of arXiv:2501.14998}
}
read the original abstract

Recent advancements in Large Language Models and Retrieval-Augmented Generation have boosted interest in domain-specific question-answering for enterprise products. However, AI Assistants often face challenges in multi-product QA settings, requiring accurate responses across diverse domains. Existing multi-domain RAG-QA approaches either query all domains indiscriminately, increasing computational costs and LLM hallucinations, or rely on rigid resource selection, which can limit search results. We introduce MKP-QA, a novel multi-product knowledge-augmented QA framework with probabilistic federated search across domains and relevant knowledge. This method enhances multi-domain search quality by aggregating query-domain and query-passage probabilistic relevance. To address the lack of suitable benchmarks for multi-product QAs, we also present new datasets focused on three Adobe products: Adobe Experience Platform, Target, and Customer Journey Analytics. Our experiments show that MKP-QA significantly boosts multi-product RAG-QA performance in terms of both retrieval accuracy and response quality.

Figures

Figures reproduced from arXiv: 2501.14998 by the authors.

Figure 1
Figure 1. Overview of the MKP-QA framework. ⃝a The main RAG-QA pipeline: retrieval of multi-domain documents, prompt augmentation, and response generation. ⃝b Detailed view of the multi-domain knowledge augmentation: A domain router estimates query-domain relevance while a retriever finds relevant documents across product domains. A stochastic gating mechanism determines active domains, for which query-domain and query-docume… view at source ↗
Figure 2
Figure 2. Performance comparison of retrieval accu [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of response quality [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: LLM Prompt for Query Generation: Simulat [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 4
Figure 4. Figure 4: Examples of user query and relevant product [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 7
Figure 7. Figure 7: LLM Prompt for Query-Response Relevance Judge: Rating from 1 to 10 for the response to a given query, following the prompt in (Zheng et al., 2024). Your task is to judge the faithfulness of a series of statements based on a given context. For each statement you must re…
Figure 9
Figure 9. Figure 9: LLM Prompt for Resource Selection step in [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 8
Figure 8. Figure 8: LLM Prompt for Response Faithfulness Judge: [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Federated Retrieval-Augmented Generation: A Systematic Mapping Study

    cs.CL 2025-05 reject novelty 4.0 of 10

    A systematic mapping study that classifies 18 federated RAG papers into a taxonomy and highlights evaluation gaps, though its search protocol is not reproducible.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [4]

    arXiv preprint arXiv:2309.15217

    Ragas: Automated eval- uation of retrieval augmented generation. arXiv preprint arXiv:2309.15217. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon

    arXiv

  2. [7]

    arXiv preprint arXiv:2212.14024

    Demonstrate-search- predict: Composing retrieval and language mod- els for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al

    arXiv

  3. [8]

    In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 2511–2522, Singapore

    G-eval: NLG evaluation using gpt-4 with better human align- ment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 2511–2522, Singapore. Association for Com- putational Linguistics. Aaron van den Oord, Yazhe Li, and Oriol Vinyals

    work page 2023

  4. [11]

    arXiv preprint arXiv:2404.14760

    Retrieval augmented generation for domain-specific question answering. arXiv preprint arXiv:2404.14760. Peng Shi, He Bai, and Jimmy Lin

    arXiv

  5. [12]

    In Findings of the Association for Computational Lin- guistics: EMNLP 2020 , pages 2768–2773, Online

    Cross-lingual training of neural models for document ranking. In Findings of the Association for Computational Lin- guistics: EMNLP 2020 , pages 2768–2773, Online. Association for Computational Linguistics. Milad Shokouhi and Luo Si

    work page 2020

  6. [14]

    A Multi-Source Retrieval Question Answering Framework Based on RAG

    A multi- source retrieval question answering framework based on rag. arXiv preprint arXiv:2405.19207. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann

    work page Pith review arXiv

  7. [15]

    arXiv preprint arXiv:2303.17564

    Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564. Xin Xu, Yue Liu, Panupong Pasupat, Mehran Kazemi, et al

    arXiv

  8. [16]

    arXiv preprint arXiv:2401.11624

    In-context learning with retrieved demon- strations for language models: A survey. arXiv preprint arXiv:2401.11624. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al

    arXiv

Show all 17 references
  1. [36]

    question

    Appendix A Query-Document Pair Examples The following AEP query-document examples highlight the necessity for product domain-specific knowledge in providing accurate and detailed re- sponses to the user’s questions: How to create a segment? Question In the Experience Platform ...

    2024

  2. [2017]

    Advances in Neural Information Processing Systems

    Attention is all you need. Advances in Neural Information Processing Systems . Shuai Wang, Ekaterina Khramtsova, Shengyao Zhuang, and Guido Zuccon. 2024a. Feb4rag: Evaluating fed- erated search in the context of retrieval augmented generation. In Proceedings of the 47th Intern...

    arXiv

  3. [2018]

    arXiv preprint arXiv:1807.03748

    Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. N Reimers

    arXiv

  4. [2019]

    arXiv preprint arXiv:1908.10084

    Sentence-bert: Sentence embed- dings using siamese bert-networks. arXiv preprint arXiv:1908.10084. Sanat Sharma, David Seunghyun Yoon, Franck Dernon- court, Dewang Sultania, Karishma Bagga, Mengjiao Zhang, Trung Bui, and Varun Kotte

    1908 arXiv

  5. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online

    Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy L...

    2020

  6. [2021]

    In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1–

    Augmenting scientific papers with just- in-time, position-sensitive definitions of terms and symbols. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1–

    2021

  7. [2022]

    arXiv preprint arXiv:2202.12275

    Partitioned variational inference: A framework for probabilistic federated learning. arXiv preprint arXiv:2202.12275. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia

    arXiv

  8. [2023]

    In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 7889–7901, Singapore

    TheoremQA: A theorem-driven question answering dataset. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 7889–7901, Singapore. Associa- tion for Computational Linguistics. Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Ya...

    2023

  9. [2024]

    arXiv preprint arXiv:2306.16092

    ‘chatlaw: A multi-agent collaborative legal assistant with knowledge graph enhanced mixture- of-experts large language model. arXiv preprint arXiv:2306.16092. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.