REVIEW 4 major objections 4 minor 19 references
Disrupt Your Research Using Generative AI Powered ScienceSage
T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read ScienceSage shows that a RAG querying a custom index that combines vector embeddings and a knowledge graph outperforms either index alone on correctness, relevance, and faithfulness across easy, medium, and hard queries.
desk verdict A transparent system report on an internal GenAI research assistant; the engineering is real, but the faithfulness metric gives the hybrid index an unfair mechanical advantage. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the custom index: a hybrid retrieval structure in which one knowledge base is indexed twice, once as vector embeddings (numerical 'digital fingerprints' capturing semantic context) and once as a knowledge graph of subject-predicate-object triples, and the RAG queries both indexes, merges their retrieved contexts, and then synthesizes a response. The vector index does the work of broad semantic matching; the knowledge graph does the work of entity-anchored factual grounding; the merging step is what lets the two strengths combine. The evaluation machinery is a set of LLM-based evaluators from LlamaIndex, with GPT-4 as a 'gold' judge: correctness is scored against a reference answer on a 1-5 scale, and relevance and faithfulness are computed as ratios of relevant or context-supported statements in the response.
What would settle it
Take a random sample of the 2,295 queries, have three human raters score the vector, knowledge-graph, and custom-index answers on the paper's three metrics, and compare the rank order; if the custom index is not first on a majority of metrics in the human-scored sample, the reported superiority is an artifact of the automated judge.
Extended reading notes
Core claim
The paper's central claim is that a retrieval-augmented generation (RAG) system that queries a custom index retrieves separately from a vector index and a knowledge graph index, concatenates the retrieved contexts, and only then synthesizes an answer, and that this hybrid scheme is consistently more correct, more relevant, and more faithful than either component alone. The authors attribute the effect to complementarity: the vector index supplies broad semantic context, while the knowledge graph anchors responses in explicit entities and relations, so the hybrid avoids the factual drift of pure semantic search and the narrowness of pure graph lookup. In their numerical experiment the pattern holds for easy, medium, and hard queries and after aggregation across keyword-occurrence levels, and the custom index also shows the lowest hallucination rates in the harder query regimes. They describe the cost as a small compromise in speed, which is why the conclusion is framed as a practical engineering choice rather than a new algorithm.
Load-bearing premise
The load-bearing premise is that the GPT-4 'gold' judge gives trustworthy correctness, relevance, and faithfulness scores and that the manually labeled difficulty and keyword-occurrence categories on the 2008 question dataset track genuine retrieval difficulty; neither premise is calibrated against human judgments.
Editorial extensions
If this is right
- RAG deployments that need both semantic breadth and factual grounding should prefer a hybrid vector-plus-graph index over a pure vector index, since the paper finds it higher on correctness, relevance, and faithfulness at a small speed cost.
- Storing generated research reports into the same knowledge base that feeds document chat lets later queries draw on freshly retrieved internet information without a separate ingestion step.
- For easy queries with clear keywords, the vector index alone is competitive; the knowledge graph index is the stronger fallback for entity-heavy medium and hard queries, and the custom index combines both advantages.
- A single persistent knowledge base can serve three query interfaces—structured report generation, document chat, and multimodal chat—so a researcher can accumulate knowledge incrementally and reuse it across tasks.
Reading between the lines
- The reported hybrid advantage has not been separated from the LLM judge's possible preference for longer or more fluent answers; a human-rated subset or a downstream task like fact extraction would tell whether the gain is in retrieval or in presentation.
- If the pattern generalizes beyond the Lincoln-document test set, hybrid retrieval could improve domain question answering on scientific and patent corpora without any change to the underlying language model; that is a cheap, model-agnostic extension worth testing.
- The paper does not calibrate its GPT-4 judge against human annotations, so an immediate check is to rerun a random sample of the 2,295 queries with human raters and compare rank order.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ScienceSage, an MVP web application that lets researchers build, store, update, and query knowledge bases encoded as both vector and knowledge-graph indices. The system supports three functions: generating research reports from the internet, chatting with uploaded documents via a RAG system, and chatting with multimodal data. The central evaluation compares three RAG variants (vector-only, KG-only, and a custom vector+KG hybrid) on a dataset of 2,295 queries derived from Smith et al. (2008), using GPT-4-based correctness, relevance, and faithfulness metrics. The paper claims that the custom hybrid index consistently achieves the highest correctness, relevance, and faithfulness across easy, medium, and hard queries, with only a small compromise in speed.
Significance. If the evaluation were sound, the paper would provide a useful practical demonstration that a hybrid vector-plus-knowledge-graph RAG can outperform either index alone on a research-oriented question-answering task. The system is deployed and used in an industrial R&D setting, and the authors share the code and are transparent about the MVP nature of the work. However, the headline empirical claim rests entirely on an evaluation that has a structural confound in the faithfulness metric and lacks the statistical detail needed to support the stated conclusions. The paper is therefore a reasonable system description, but its central performance claim needs substantially stronger evidence before it can be accepted.
major comments (4)
- [RAG evaluation, Eq. (1)] The faithfulness metric F = |V|/|S| is structurally confounded with the size of the retrieved context c(q). The custom RAG is described as first retrieving from the vector index and KG index separately and then combining all retrieved information, so its c(q) is the union of two contexts, whereas the vector-only and KG-only RAGs each use a single context. For a fixed generated answer, adding retrieved nodes can only increase the number of statements verifiable from c(q), giving the custom index a mechanical advantage on faithfulness that has nothing to do with retrieval quality. The paper does not control for this by, for example, retrieving the same total number of nodes from each index or evaluating faithfulness against a fixed reference context. This confound directly undermines the 'lowest hallucination rates' conclusion in Figure 3(c) and the accompanying text.
- [Results, Figures 3 and A1-A7] The evaluation reports only bar charts with error bars and no numeric means, standard deviations, confidence intervals, or statistical tests. With 2,295 queries, a claim of 'consistent' superiority requires at least a table of means and standard deviations and some pairwise significance testing (e.g., paired bootstrap or a mixed-effects model). Without these, the reader cannot determine whether the visible differences are meaningful or within sampling noise. This is load-bearing because the entire central claim of the paper is the superiority of the custom index.
- [Evaluation metrics, 'RAG evaluation'] All three metrics use GPT-4 as the 'gold' LLM judge, and no human calibration or agreement analysis is reported. The judge is from the same model family as the generator, and LLM judges are known to be biased by response length and fluency, which is especially concerning for the custom RAG whose answers are synthesized from a larger retrieved context and may therefore be longer. The qualitative feedback from CPG researchers mentioned later is not connected to the quantitative metrics, so it does not serve as validation. The paper should report a human-judged sample or at least an inter-annotator agreement study to establish that the metric scores track actual correctness, relevance, and faithfulness.
- [Conclusion] The conclusion states that the custom RAG offers 'superior performance at small compromise of speed,' but no runtime or latency measurements appear anywhere in the paper. Since the speed/quality trade-off is part of the summarized contribution, the absence of any timing data leaves this part of the claim unsubstantiated.
minor comments (4)
- [RAG evaluation, 'Evaluation metrics'] There are several typos in this section: 'There metrics are defined' should be 'These metrics are defined', and 'an response' in the custom-index description should be 'a response'.
- [Throughout] The terminology for the KG-based index is inconsistent: the text and figures use 'Knowledge Index', 'KG index', and 'Knowledge Graph index' interchangeably. Please standardize the terminology.
- [Appendix, Figures A4 and A5] The phrase 'norm number of keywords' appears to be a typo for 'normal number of keywords'.
- [References] The citation of Touvron et al. (2023) for the LlamaIndex CorrectnessEvaluator is questionable; the evaluator is a LlamaIndex component, and the paper should cite the LlamaIndex documentation or a more directly relevant source for the evaluation modules.
Circularity Check
No significant circularity: the paper's RAG comparison is an empirical system evaluation without fitted parameters, self-citation chains, or derivational claims; the hybrid-context faithfulness concern is a benchmark-validity risk, not a circular reduction.
full rationale
The paper reports an empirical comparison of three RAG configurations (vector, KG, custom vector+KG) on a fixed test set, with no fitted parameters, no first-principles derivation, and no load-bearing self-citation. The only quantitative definition is the faithfulness metric in Eq. (1), F = |V|/|S|, where V is the set of statements supported by the retrieved context c(q). The custom RAG does retrieve a union of contexts: 'To answer a user query, this custom RAG first retrieves information from vector index and KG index, separately. It then combines all retrieved information together and finally synthesizes an a response (Liu 2022).' One could worry that a larger context mechanically inflates faithfulness. However, the generated answer as(q) is not held fixed across the compared systems, so a larger context does not force a higher F: the custom answer may contain more unsupported statements, lowering the ratio. The reported superiority is therefore an observed outcome, not an identity forced by the metric's definition. Similarly, the use of GPT-4 as both generator and judge is a self-referential evaluation choice, but it is not a circular derivation of the paper's claims. The Discussion itself disclaims algorithmic novelty: 'ScienceSage is not designed to offer substantial algorithmic innovation for generative AI. It aims to assemble the best and the most efficient GenAI packages.' There are no fitted-input-as-prediction steps, no uniqueness arguments imported from the authors' prior work, and no renamed known results. The union-context faithfulness confound and the LLM-judge reliability issue are legitimate correctness/validity risks, but they do not make any claimed result equivalent to its inputs by construction. Hence no significant circularity is found.
Assumptions & free parameters
free parameters (1)
- Retrieval top-k and chunk size for RAG indices =
not reported
assumptions (4)
- domain assumption GPT-4 as a gold LLM judge produces valid correctness, relevance, and faithfulness scores.
- domain assumption The Smith, Heilman, and Hwa 2008 dataset with the authors' easy/medium/hard and keyword-occurrence labels is a valid testbed for RAG retrieval.
- domain assumption Knowledge graph extraction from text using LlamaIndex produces triples of sufficient quality for KG-based RAG.
- domain assumption Scraped internet content and LLM-generated summaries are accurate enough to support research reports.
Cite this review
Pith. "Pith review of Disrupt Your Research Using Generative AI Powered ScienceSage." pith.science (2026). https://pith.science/paper/O47W7DYZ
@misc{pith2026250218479,
author = {Pith},
title = {Pith review of: Disrupt Your Research Using Generative AI Powered ScienceSage},
year = {2026},
howpublished = {\url{https://pith.science/paper/O47W7DYZ}},
note = {Machine review of arXiv:2502.18479}
}
abstract
Large Language Models (LLM) are disrupting science and research in different subjects and industries. Here we report a minimum-viable-product (MVP) web application called $\textbf{ScienceSage}$. It leverages generative artificial intelligence (GenAI) to help researchers disrupt the speed, magnitude and scope of product innovation. $\textbf{ScienceSage}$ enables researchers to build, store, update and query a knowledge base (KB). A KB codifies user's knowledge/information of a given domain in both vector index and knowledge graph (KG) index for efficient information retrieval and query. The knowledge/information can be extracted from user's textual documents, images, videos, audios and/or the research reports generated based on a research question and the latest relevant information on internet. The same set of KBs interconnect three functions on $\textbf{ScienceSage}$: 'Generate Research Report', 'Chat With Your Documents' and 'Chat With Anything'. We share our learning to encourage discussion and improvement of GenAI's role in scientific research.
Figures
Reference graph
Works this paper leans on
-
[1]
Chase, H. 2022. LangChain
2022
-
[2]
Chen, J.; Lin, H.; Han, X.; and Sun, L. 2023. Benchmarking Large Language Models in Retrieval-Augmented Generation. arXiv:2309.01431
arXiv 2023
-
[3]
Guu, K.; Lee, K.; Tung, Z.; Pasupat, P.; and Chang, M.-W. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. arXiv:2002.08909
arXiv 2020
-
[4]
He, J.; Feng, W.; Min, Y.; Yi, J.; Tang, K.; Li, S.; Zhang, J.; Chen, K.; Zhou, W.; Xie, X.; et al. 2023. Control risk for potential misuse of artificial intelligence in science. arXiv preprint arXiv:2312.06632
arXiv 2023
-
[5]
Jiang, Y.; Shao, Y.; Ma, D.; Semnani, S. J.; and Lam, M. S. 2024. Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations. arXiv:2408.15232
arXiv 2024
-
[6]
Li, T.; Lu, J.; Chu, C.; Zeng, T.; Zheng, Y.; Li, M.; Huang, H.; Wu, B.; Liu, Z.; Ma, K.; et al. 2024. Scisafeeval: a comprehensive benchmark for safety alignment of large language models in scientific tasks. arXiv preprint arXiv:2410.03769
arXiv 2024
-
[7]
Liu, J. 2022. LlamaIndex
work page 2022
-
[8]
H.; He, J.; Tibo, A.; Janet, J
Loeffler, H. H.; He, J.; Tibo, A.; Janet, J. P.; Voronov, A.; Mervin, L. H.; and Engkvist, O. 2024. Reinvent 4: Modern AI--driven generative molecule design. Journal of Cheminformatics, 16(1): 20
work page 2024
Show all 19 references
-
[9]
Bran, A.; Cox, S.; Schilter, O.; Baldassari, C.; White, A
M. Bran, A.; Cox, S.; Schilter, O.; Baldassari, C.; White, A. D.; and Schwaller, P. 2024. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 1--11
2024
-
[10]
Mittal, S.; Joshi, A.; and Finin, T. 2017. Thinking, Fast and Slow: Combining Vector Spaces and Knowledge Graphs. arXiv:1708.03310
2017 arXiv
-
[11]
A.; Xu, P.; Khattab, O.; and Lam, M
Shao, Y.; Jiang, Y.; Kanell, T. A.; Xu, P.; Khattab, O.; and Lam, M. S. 2024. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis...
2024
-
[12]
A.; Heilman, M.; and Hwa, R
Smith, N. A.; Heilman, M.; and Hwa, R. 2008. Question Generation as a Competitive Undergraduate Course Project. In In Proceedings of the NSF Workshop on the Question Generation Shared Task and Evaluation Challenge
2008
-
[13]
Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N....
2023 arXiv
-
[14]
Wang, H.; Fu, T.; Du, Y.; Gao, W.; Huang, K.; Liu, Z.; Chandak, P.; Liu, S.; Van Katwyk, P.; Deac, A.; et al. 2023. Scientific discovery in the age of artificial intelligence. Nature, 620(7972): 47--60
2023
-
[15]
L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A
Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020. Transforme...
2020
-
[16]
Yang, S.; Han, F.; Wu, Y.; and Yan, X. 2016. Fast top-k search in knowledge graphs. In 2016 IEEE 32nd International Conference on Data Engineering (ICDE), 990--1001. Los Alamitos, CA, USA: IEEE Computer Society
2016
-
[17]
Zamani, H.; and Croft, W. B. 2016. Estimating Embedding Vectors for Queries. Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval
2016
-
[18]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[19]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.