TeroSeek: An AI-Powered Knowledge Base and Retrieval Generation Platform for Terpenoid Research

Jiahao Li; Kangwei Xu; Ruibo Wu; Siqi Jiang; Xu Kang

REVIEW 3 major objections 4 minor 3 references

TeroSeek: An AI-Powered Knowledge Base and Retrieval Generation Platform for Terpenoid Research

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a curated terpenoid literature base to a general LLM raises its domain answer accuracy from 0.50 to 0.78 and beats four commercial frontier models.

desk verdict A useful terpenoid RAG resource whose headline accuracy numbers rest on a post-hoc, KB-derived 41-question test set; the direction of the RAG benefit is plausible but the magnitude is not established. read the letter →

arxiv 2505.20663 v1 pith:UQK7KVYV submitted 2025-05-27 cs.IR cs.AIcs.CL

Xu Kang , Siqi Jiang , Kangwei Xu , Jiahao Li , Ruibo Wu This is my paper · ORCID

classification cs.IRcs.AIcs.CL

keywords terpenoidsknowledgebaseretrieval-augmentedgenerationlargelanguagemodelsquestionansweringnaturalproductscitation-linkedanswers

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a curated, literature-derived knowledge base plus retrieval-augmented generation (RAG) can turn a general-purpose large language model into a reliable expert on terpenoids, a large class of natural products studied across chemistry, pharmacology, and biology. The authors build TeroSeek-KB from nearly 48,000 terpenoid publications and pair it with an open-source reasoning model, creating the TeroSeek-normal assistant. On their 41-question terpenoid benchmark, the approach lifts the backbone model's accuracy from 0.50 to 0.78 and beats four commercial frontier models, which score between 0.44 and 0.61. If the result holds, it offers a reusable recipe for turning fragmented, interdisciplinary literature into an answerable, cited knowledge service without fine-tuning.

What carries the argument

TeroSeek-KB is the central object: a vector-searchable collection of 228,400 cleaned text segments drawn from 47,731 terpenoid publications, each segment annotated with up to four LLM-generated hypothetical questions and stored as a 2048-dimensional embedding. Retrieval happens in two layers: a first pass matches the query against an abstract-level summary index and returns up to 400 candidates, and a second pass narrows these to at most 20 sub-chunks whose cosine similarity to the query exceeds 0.7. Before literature retrieval, molecule-specific queries are checked against a companion terpenoid molecular database, and the final prompt combines the user's question with cited passages for the LLM to answer.

What would settle it

Have independent terpenoid experts, without access to TeroSeek-KB, write 100 fresh multiple-choice questions with verified answers; if TeroSeek-normal's 0.78 accuracy falls to the level of the four commercial models (0.44 to 0.61) or below, the reported advantage is an artifact of the knowledge-base-derived test set.

Watch

Extended reading notes

Core claim

The central claim is that externally storing domain knowledge in a retrievable, citation-linked repository is what converts a generic LLM into a high-accuracy terpenoid assistant. Concretely, adding TeroSeek-KB through a two-stage retrieval pipeline raised one open-source reasoning backbone's multiple-choice accuracy from 0.50 to 0.78, a dense open-source model's from 0.37 to 0.66, and a second open-source reasoning model's from 0.46 to 0.74 on the filtered 41-question test set. The best retrieval-augmented configuration scored 0.78, above four commercial frontier models that scored 0.61, 0.51, 0.46, and 0.44. The paper takes these numbers to show that domain-specific retrieval beats larger parametric knowledge for terpenoid questions.

Load-bearing premise

The 41-question test set, largely generated by an LLM from the same article passages stored in the knowledge base and then filtered to remove simpler questions, fairly represents the terpenoid questions real users will ask.

Editorial extensions

If this is right

Terpenoid researchers gain a free web tool whose answers name specific papers, tables, and figures, so claims can be checked at the source rather than taken on trust.
Because the knowledge base and the model are decoupled, adding a better backbone model or new literature improves the service without retraining.
The reported 0.78 accuracy on the curated terpenoid test set is higher than every commercial frontier model tested, suggesting specialized retrieval can beat much larger parametric knowledge in narrow domains.
The construction pipeline—LLM screening of papers, chunk cleaning, hypothetical-question generation, and two-stage vector retrieval—is a reusable template for other interdisciplinary fields.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because many test questions were generated from the knowledge base's own passages, an independent expert-written set could show a smaller gap; this is an inference, not a paper claim.
The reported gain may overstate everyday benefits because 'simpler' questions on which the base model already answers correctly were removed, so the 0.50 baseline is not the accuracy a user would see on typical queries.
A direct testable extension is to freeze the retrieval component and swap different backbone models, isolating how much of the gain comes from retrieval quality versus the LLM's own reasoning.
The same pipeline could be tried on other natural-product families, but the benchmark would need to be built independently of the source corpus to avoid circularity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: TeroSeek is a genuine, useful engineering artifact—a curated 47k-paper terpenoid knowledge base with a RAG pipeline and a live web service—but the headline accuracy numbers (DeepSeek-R1 0.50→0.78, beating Gemini 2.5 Pro, etc.) are measured on a test set that was generated from the same article passages stored in the KB and then pruned from 126 to 41 questions after the models were run. That evaluation design makes the magnitudes unreliable, even though the direction of the RAG benefit is credible.

What's new and good: The KB itself is the contribution. 47,731 documents, 228,400 chunks, 932,577 hypothetical questions for retrieval; hierarchical abstract-to-chunk retrieval with a first layer that returns up to 400 documents and a second layer that trims to 20; and a review mode that generates sub-questions. The web service is public, the paper shows a worked paclitaxel example with real references, and the writing makes clear the system is meant to be a literature assistant, not a new scientific method. They cite prior domain RAG systems (Bailicai, ShennongAlpha, PanKB, PlantGPT), so they're not claiming RAG is theirs.

Where it's soft: The evaluation. The test set is built by having an LLM generate questions from article passages that are also in the KB; then the authors "excluded cases where models produced identical responses for simpler questions," dropping 85 of 126 items. That pruning systematically removes easy, high-agreement questions and leaves a hard-only subset, which inflates the apparent gain of any retrieval system that is correct on easy items. On top of that, the retained questions are from the same documents the retriever was built from, so you're measuring closed-book retrieval on open-book items. There are no error bars, no significance tests, and no comparison against the cited domain RAG systems. The 41-question subset is too small to support the "superior performance" claim, and the paper does not report accuracy on the full 126 set, which would have been useful.

Minor issues: model names are inconsistent (Qwen3 vs Qwen2; Claude 3 vs Claude 3.7), and the text says both "final test set contains 126" and "refined to 41" without clarifying the flow.

Net: it's a useful resource and a reasonable architecture paper. The evaluation should be redone with a held-out test set written by domain experts from sources not indexed in TeroSeek-KB, and with confidence intervals. As it stands, the evidence supports "RAG with a good domain KB helps on these 41 hard questions," not "TeroSeek beats GPT-4-class models on general terpenoid queries."

Recommendation: worth engaging. A serious referee could get this into shape. I'd send it to review with a request for a stronger benchmark, not desk-reject it.

Referee Report

3 major / 4 minor

Summary. The paper presents TeroSeek, a terpenoid-domain knowledge base built from 47,731 academic documents, and a retrieval-augmented generation (RAG) platform that combines this KB with open-source LLMs (DeepSeek-R1, DeepSeek-V3, Qwen3-235B-A22B) to answer terpenoid-related questions. The system uses a two-layer hierarchical retrieval strategy over 228,400 text chunks and 932,577 hypothetical questions, and offers two services: TeroSeek-normal for direct Q&A with citations and TeroSeek-review for multi-stage synthesis. The central empirical claim is that adding TeroSeek-KB to an LLM substantially improves accuracy on a terpenoid test set, with DeepSeek-R1 rising from 0.50 to 0.78, and that TeroSeek-normal outperforms Gemini 2.5 Pro, Claude 3.7 Sonnet, o3, and Grok 3 Beta.

Significance. If the performance gains are real, TeroSeek would be a useful domain-specific RAG resource for the terpenoid community, and the paper's architecture—decoupling the KB, retrieval pipeline, and backbone LLM—is a sensible design with practical benefits for knowledge updates and model iteration. The authors have built a substantial, publicly accessible web service and a large curated KB, which is a tangible contribution. However, the current evaluation does not establish the claimed superiority over general LLMs because the benchmark is constructed from the same article passages that populate the KB and is further pruned post hoc to a 41-question subset. The central claim therefore rests on circular and non-representative evidence; the paper needs a rigorous, independent evaluation before its conclusions can be accepted.

major comments (3)

[Result (benchmark construction, Figure 4)] The test set is circular with respect to the system being evaluated. The paper states that AI-generated questions were created 'based on article passages' and that these passages are the same material indexed in TeroSeek-KB. A RAG system that retrieves from TeroSeek-KB is therefore tested on questions whose answers are contained in the exact passages it was designed to retrieve, while closed-book LLMs must rely solely on parametric knowledge. This inflates the open-book RAG advantage and undermines the reported gains (e.g., DeepSeek-R1 0.50 to 0.78). The authors should construct an independent test set—ideally expert-written or drawn from recent literature not yet in the KB—or at minimum hold out a subset of KB documents and generate questions only from that held-out portion.
[Result (test set refinement, 126 to 41)] The post-hoc exclusion of 'simpler questions' on which models produced identical responses is a serious validity threat. Dropping easy, consensual questions systematically biases the benchmark toward hard items and inflates measured improvements, so the reported accuracies are not representative of general terpenoid queries. Moreover, the text first states that 'The final test set contains 126 multiple-choice questions' and then reports all results on 41 entries; this contradiction must be resolved. The authors should report accuracy on the full 126-question set, justify any exclusion, and show that the main conclusions hold without the pruning.
[Result (statistical reporting, Figure 4)] Accuracy differences are reported without any measure of uncertainty or significance testing, despite each question being answered five times per model. With only 41 items, the differences between, for example, TeroSeek-normal (0.78) and Gemini 2.5 Pro (0.61) could be meaningful, but the paper provides no confidence intervals, error bars, or pairwise significance tests. The authors should report variance across the repeated answers, ideally with bootstrap confidence intervals or a McNemar test for paired comparisons, and also disclose the number of retrieval failures (e.g., the Ref6 error described in the paclitaxel example) and their impact on accuracy.

minor comments (4)

[Result / Figure 4] The in-text references to Figure 4 panels do not match the figure caption: the text cites Figure 4a as the test set construction pipeline and Figure 4b as the disciplinary distribution, while the caption assigns those content types to panels (c) and (d), respectively; the panel letters should be aligned.
[Materials and Methods / Result] The backbone model is referred to as both 'Qwen-3-235B-A22B' in the Methods section and 'Qwen2-235B-A22B' in the Results section; please use a single consistent name.
[Throughout] The capitalization of the product name is inconsistent ('TeroSeek-normal', 'TeroSeek-Normal', 'TeroSeek-normal'); standardize to one form.
[References] Several reference entries are incomplete or missing journal names (e.g., references 2, 13, 23, 32); the reference list should be checked for consistency and completeness.

Circularity Check

2 steps flagged · score 7.0 of 10

RAG gains are measured on a test set generated from the same KB passages and then pruned to hard questions, making the benchmark circular.

self definitional [Materials and Methods, KB construction (Figure 1d); Result, test set construction (Figure 4a)]
"All the Chunks will generate up to 4 hypothetical questions through LLMs, and then these questions will be converted into 2048 -dimensional vectors by the embedding model and stored in the vector database (Figure 1d). Finally, 228,400 text segments with 932,577 hypothetical questions were generated to construct a knowledge base TeroSeek-KB. ... (1) AI-generated questions, where an LLM created numerous potential questions based on article passages, which were then manually screened and verified by human experts."

The benchmark is not external: the test questions are produced by an LLM reading the same article passages that were segmented into TeroSeek-KB, and the KB's retrieval index is built from LLM-generated hypothetical questions for those same chunks. So for AI-generated questions, the correct answer passage is already the retrieval target; TeroSeek's task is to fetch the passage that was used to write the question. Accuracy on such questions measures source-passage retrieval, not terpenoid competence beyond the KB. The comparison to closed-book LLMs therefore builds in the open-book advantage by construction.
other [Result, test-set refinement and accuracy reporting (Figure 4c,d)]
"After excluding cases where models produced identical responses for simpler questions, we refined our test set to 41 entries, with results presented in Figure 4c and Figure 4d. ... When integrated with TeroSeek-KB, the models demonstrated notable performance gains: DeepSeek-V3 improved from 0.37 to 0.66 accuracy, DeepSeek-R1 increased from 0.50 to 0.78, and Qwen2 -235B-A22B rose from 0.46 to 0.74."

All reported accuracy numbers come from the 41 questions remaining after dropping cases where models gave identical responses to 'simpler' questions. This is a post-hoc selection based on model outputs, not the original 126-question protocol. Removing easy items that models already answer correctly inflates both the absolute accuracy of any system that is right on hard items and the relative gain of RAG over closed-book baselines. The headline improvements (e.g., DeepSeek-R1 0.50→0.78) and the 'superior performance' conclusion are therefore conditional on a non-representative, response-selected subset.

full rationale

The paper never claims to derive a mathematical prediction; its central claim is that TeroSeek with RAG outperforms general-purpose LLMs on terpenoid queries. That claim is load-bearing and it rests entirely on a benchmark that is contaminated by construction. In building TeroSeek-KB, each chunk is used to generate up to four hypothetical questions, and those questions/vectors form the retrieval index. The evaluation set is then created by having an LLM generate questions 'based on article passages'—the same passages now stored in the KB. Human screening does not remove this dependence; it only verifies answer correctness. Furthermore, after preliminary model runs the authors discarded all questions on which models produced identical responses for 'simpler' questions, reducing 126 to 41, and all reported accuracies and comparisons are computed on that post-hoc subset. This is not a claim of outright logical equivalence—TeroSeek still has to retrieve the right passage and could answer incorrectly—so the score is high but not maximal. The TeroKit-related self-citations (Refs 1-2) are not load-bearing for this benchmark. The result is a partially circular evaluation: the test set is derived from the system's own knowledge source and then sharpened by model responses, so the reported superiority over closed-book LLMs is not established for general terpenoid queries.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No theoretical entities are introduced. The system relies on hand-set retrieval thresholds and on the assumption that the test set, generated from the KB's own source articles, is an unbiased benchmark.

free parameters (5)

cosine similarity threshold = 0.7
Second-stage retrieval cutoff; chosen by hand, affects precision and recall of retrieved chunks.
first-layer retrieval count = 400
Maximum documents returned in abstract index layer, set to minimize missing relevant info.
second-layer retrieval count = 20
Maximum sub-chunks returned to the LLM after filtering.
hypothetical questions per chunk = up to 4
LLM generates up to 4 hypothetical questions per text chunk for embedding index.
test set exclusion criterion = 126 to 41 questions
Questions where models produced identical responses for simpler questions were excluded post hoc, shrinking the benchmark and affecting the reported accuracy deltas.

assumptions (4)

domain assumption The literature retrieved from Web of Science, Scopus, and PubMed with 'terpene' covers the terpenoid field adequately.
Introduced in Materials and Methods; the KB and all downstream evaluation depend on this coverage.
domain assumption LLM-based screening and cleaning of abstracts and full texts preserves the information needed to answer domain questions.
Used in the KB construction pipeline; errors in screening or cleaning propagate into retrieval quality.
domain assumption The 41-question refined test set is representative of real terpenoid research queries.
The central accuracy comparisons are computed on this set; the refinement process may introduce selection bias.
domain assumption Open-book RAG versus closed-book baseline is a fair comparison for claiming superiority over general-purpose LLMs.
The baseline LLMs cannot access the KB, while the RAG system sees the relevant passages; the comparison conflates retrieval access with model capability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TeroSeek: An AI-Powered Knowledge Base and Retrieval Generation Platform for Terpenoid Research." pith.science (2026). https://pith.science/paper/UQK7KVYV

@misc{pith2026250520663,
  author       = {Pith},
  title        = {Pith review of: TeroSeek: An AI-Powered Knowledge Base and Retrieval Generation Platform for Terpenoid Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQK7KVYV}},
  note         = {Machine review of arXiv:2505.20663}
}

read the original abstract

Terpenoids are a crucial class of natural products that have been studied for over 150 years, but their interdisciplinary nature (spanning chemistry, pharmacology, and biology) complicates knowledge integration. To address this, the authors developed TeroSeek, a curated knowledge base (KB) built from two decades of terpenoid literature, coupled with an AI-powered question-answering chatbot and web service. Leveraging a retrieval-augmented generation (RAG) framework, TeroSeek provides structured, high-quality information and outperforms general-purpose large language models (LLMs) in terpenoid-related queries. It serves as a domain-specific expert tool for multidisciplinary research and is publicly available at http://teroseek.qmclab.com.

Figures

Figures reproduced from arXiv: 2505.20663 by the authors.

**Figure 1.** Data parsing and knowledge base construction process. (a) Preprocessing approach for journal articles, utilizing an LLM to screen candidate literature. (b) Conversion of PDFs to Markdown format for further parsing. (c) Employing an LLM for information cleansing and consolidation. (d) Using an LLM to generate hypothetical questions from knowledge, followed by vectorization and storage in a vector database. The docume… view at source ↗

**Figure 2.** The model architecture of TeroSeek, (a) the vector retrieval process, (b) the response process of the Q&A model, (c) the response process of the research model. Three top-performing open-source models as of May 2025 were utilized, including Qwen-3-235B-A22B (https://qwenlm.github.io/blog/qwen3), DeepSeek-V334, and DeepSeek-R133 . In addition, four commercial models were employed: Gemini 2.5 Pro Preview, Claude 3 Son… view at source ↗

**Figure 3.** Response Demonstration Case - Knowledge Q&A For the question: "What is the target of paclitaxel?", the upper section displays the output from TeroSeek-normal, while the lower section presents the output from DeepSeek-R1. The illustrated outputs have been reformatted and appropriately simplified for clarity. The TeroSeek-normal demonstrates superior capability in extracting information from tables and figures within … view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Benchmark of TeroSeek-normal with other leading LLMs. (a) Performance of backbone models DeepSeek-V3, DeepSeek-R1 and Qwen-3-235B-A22B on the test set and their enhanced performance with TeroSeek-KB. (b) Comparison between TeroSeek-normal and other LLMs in terms of sco…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

[1]

What is the target of paclitaxel?

TeroSeek: An AI-Powered Knowledge Base and Retrieval Generation Platform for Terpenoid Research Xu Kang, Siqi Jiang, Kangwei Xu, Jiahao Li, Ruibo Wu* School of Pharmaceutical Sciences, Sun Yat-sen University, Guangzhou 510006, P.R. China * E-mail: wurb3@mail.sysu.edu.cn Abstract Terpenoids represent a pivotal class of natural products that have garnered s...

arXiv 2020
[2024]

26 Sun, B

8-23 (World Scientific). 26 Sun, B. et al. PanKB: An interactive microbial pangenome knowledgebase for research, biotechnological innovation, and knowledge mining. Nucleic Acids Research 53, D806 -D818, doi:10.1093/nar/gkae1042 %J Nucleic Acids Research (2024). 27 Yeganova, L. et al. LitSense 2.0: AI-powered biomedical information retrieval with sentence ...

work page doi:10.1093/nar/gkae1042 2024
[2025]

30 Zhang, R

Nucleic Acids Research 53, D30-D44, doi:10.1093/nar/gkae978 %J Nucleic Acids Research (2024). 30 Zhang, R. et al. PlantGPT: An Arabidopsis -Based Intelligent Agent that Answers Questions about Plant Functional Genomics. n/a, e03926, doi:https://doi.org/10.1002/advs.202503926. 31 Shuster, K., Poff, S., Chen, M., Kiela, D. & Weston, J. J. a. p. a. Retrieval...

work page doi:10.1093/nar/gkae978 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

What is the target of paclitaxel?

TeroSeek: An AI-Powered Knowledge Base and Retrieval Generation Platform for Terpenoid Research Xu Kang, Siqi Jiang, Kangwei Xu, Jiahao Li, Ruibo Wu* School of Pharmaceutical Sciences, Sun Yat-sen University, Guangzhou 510006, P.R. China * E-mail: wurb3@mail.sysu.edu.cn Abstract Terpenoids represent a pivotal class of natural products that have garnered s...

arXiv 2020

[2] [2024]

26 Sun, B

8-23 (World Scientific). 26 Sun, B. et al. PanKB: An interactive microbial pangenome knowledgebase for research, biotechnological innovation, and knowledge mining. Nucleic Acids Research 53, D806 -D818, doi:10.1093/nar/gkae1042 %J Nucleic Acids Research (2024). 27 Yeganova, L. et al. LitSense 2.0: AI-powered biomedical information retrieval with sentence ...

work page doi:10.1093/nar/gkae1042 2024

[3] [2025]

30 Zhang, R

Nucleic Acids Research 53, D30-D44, doi:10.1093/nar/gkae978 %J Nucleic Acids Research (2024). 30 Zhang, R. et al. PlantGPT: An Arabidopsis -Based Intelligent Agent that Answers Questions about Plant Functional Genomics. n/a, e03926, doi:https://doi.org/10.1002/advs.202503926. 31 Shuster, K., Poff, S., Chen, M., Kiela, D. & Weston, J. J. a. p. a. Retrieval...

work page doi:10.1093/nar/gkae978 2024