{"id":"4c991e9a-4787-49fd-a5d3-c8e8a9a44c1e","arxiv_id":"2501.00223","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CancerKG.ORG is a knowledge graph augmented LLM retrieval system for colorectal cancer literature, described with claims of unsupervised ingestion and hallucination prevention but without comparative evaluation.","lead":"A cancer knowledge system combines a knowledge graph with large language models to answer questions about colorectal cancer treatment. It is presented as faster and more reliable than using either technology alone, but the paper provides no head-to-head evidence for that advantage.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RAG guardrail claim is unsupported: no end-to-end evaluation shows KG-grounded answers are more accurate than the LLM alone, and the fusion pipeline's correctness is unmeasured.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing gap: the system's trustworthiness depends on automatic extraction and fusion preserving medical correctness at scale, and this is not validated. My stress-test agrees and sharpens the point: the paper's only quantitative evidence is a component-level F-measure, which does not bear on the correctness of KG content or the safety of RAG outputs. The unsupported 'unsupervised' claim and the absence of any end-to-end comparison directly undermine the abstract's comparative and safety claims. Since the reader already reached REJECT and my analysis does not change that assessment, the verdict remains UNCHANGED. I am not raising a separate objection; the central claim is unsupported by the evidence provided, but the paper is a systems description rather than a fraudulent claim, so the appropriate disposition is unchanged reject on evidence grounds, with a concrete path to conditional acceptance if the proposed evaluation is performed.","tokens_in":12524,"tokens_out":2620,"duration_ms":26120,"concrete_test":"Run a blind, end-to-end evaluation on a fixed set of 100 colorectal cancer clinical questions with expert-verified reference answers. Query (a) CancerKG's conversational interface, (b) the same underlying LLM without KG retrieval, and (c) a standard PubMed search; have blinded oncologists rate factual accuracy and verifiability. Separately, sample 200 KG edges or fused subtrees and have two clinicians adjudicate them against the cited PubMed sources. If KG-grounded answers are not significantly more accurate than LLM-only answers, or if more than a small fraction (e.g., 5%) of sampled edges are wrong or unsupported, the 'verified guardrail' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the verified KG acts as a RAG guardrail so the hybrid 'does not hallucinate' and serves users better than an LLM, KG, or search engine in isolation (Abstract; §1). This requires two things: the automatically extracted and fused KG content must be medically correct, and KG-grounded answers must be more reliable than the unaided LLM. Neither is demonstrated. §5.2 describes fusion via 'normalized NLP term matching, amended by embedding-driven matching,' with only low-confidence fusions sent to an expert and mistakes to be 'learned by the fusion model' later; no fusion error rates, expert adjudication counts, or post-correction precision figures are reported. The only quantitative result (§4.3) is approximately 95% F-measure for topical table classification on a self-constructed training set, with no error bars; even that number concerns table clustering, not the correctness of KG nodes, edges, or generated answers. Furthermore, the Conclusion's assertion that content is 'extracted and updated in unsupervised manner from PubMed.com' is contradicted by §2 and §5.1, which require a Data Scientist to seed the KG, and §5.2, which requires expert review of low-confidence fusions; peer-review status of a PubMed entry is not a verification of medical correctness. Without measuring KG precision or comparing RAG-grounded outputs against LLM-only outputs, the guardrail claim is an assertion, not a finding. The stated ongoing evaluation at Moffitt Cancer Center is future work, not evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes CancerKG, a web-scale knowledge graph (KG) and large language model (LLM) hybrid for colorectal cancer, with automatic extraction and fusion from PubMed, multiple user interfaces (interactive KG, structural table search, conversational interface, 3D meta-profiles), and a GRU-based table topic classifier. The central claims are that the verified KG acts as a RAG guardrail preventing LLM hallucination and that the hybrid serves users better than an LLM, a KG, or a search engine in isolation. The only quantitative result is an approximately 95% F-measure for topical table classification (§4.3); no end-to-end evaluation, baseline comparison, user study, or hallucination measurement is provided.","tokens_in":12810,"tokens_out":4744,"duration_ms":47433,"significance":"If the central claims were substantiated, the system would be a valuable contribution: an automatically updated, verifiable medical KG-LLM hybrid that addresses hallucination and knowledge freshness at web scale. The paper also demonstrates engineering effort, including structural search over complex medical tables and interactive meta-profiles, and it builds on prior COVIDKG work. However, the current evidence does not support the load-bearing claims. The evaluation is confined to a single component classifier, trained and validated on the authors' own data, and the 'unsupervised' and 'verifiable' characterizations are contradicted by the design described in the manuscript. The paper is a plausible systems description, but it falls far short of demonstrating the claimed benefits.","major_comments":[{"comment":"The central comparative and safety claims are not evaluated. The abstract and introduction assert that the hybrid 'serves the user needs better than just an LLM, KG or a search-engine in isolation' and that the verified KG prevents hallucination, but no comparison against these baselines, no hallucination measurement, and no user study are reported. The only quantitative result (§4.3) is approximately 95% F-measure for topical table classification, which does not speak to answer correctness or KG-grounded generation. Add an end-to-end evaluation with expert or ground-truth assessment, comparing the hybrid against LLM-only, KG-only, and search-only conditions, and report factual accuracy and hallucination rates.","section":"Abstract; §1; §3.3"},{"comment":"The 'unsupervised' and 'verifiable' characterizations are contradicted by the design. Section 2 requires a Data Scientist to initialize a seed KG with 10–20 nodes, §5.1 repeats this dependency, and §5.2 states that low-confidence fusions are evaluated by an expert. The conclusion in §7 that content is 'extracted and updated in unsupervised manner from PubMed.com' is therefore inaccurate. Moreover, the fusion process's correctness is unmeasured: there are no precision/recall figures for extracted nodes and edges, no counts or outcomes of expert interventions, and no evidence for the claim that mistakes 'will be learned by the fusion model.' Clarify the actual degree of supervision and report fusion accuracy on a validation set.","section":"§2; §5.1; §5.2; §7"},{"comment":"The evaluation of the topical table classifier is circular with respect to the clustering claim. The positive training set is selected as tables within an empirical angle (18 degrees) of data-scientist-chosen centroid vectors, and the same centroids define the final clusters. This guarantees strong agreement between the classifier and the cluster definitions. The reported 'approximately 95% F-measure' also lacks error bars, statistical significance, and external validation. Evaluate the classifier on an independent, human-labeled test set and report variance.","section":"§4.3"},{"comment":"The RAG guardrail claim assumes that the KG is a reliable repository of verified medical facts, but the manuscript does not measure KG correctness. The enrichment pipeline uses normalized NLP term matching and embedding-driven matching, with expert review only for low-confidence fusions; no estimate is given of how often expert review is needed or how accurate the resulting graph is. If the KG contains errors, grounding the LLM in it could reinforce false or stale information while giving an appearance of verifiability. Report KG accuracy on a sample of PubMed-derived facts with expert adjudication, and show that KG-grounded outputs are more accurate than those of the unaided LLM.","section":"§5.2; Abstract"}],"minor_comments":[{"comment":"The phrase 'Large Language Models (LMM)' should be 'Large Language Models (LLM)'.","section":"§2"},{"comment":"References [2] and [3] appear mismatched: [2] is listed as the COVID-19 Infectious Disease Ontology, yet it is cited for LLM hallucinations, and [3] is COVIDScholar, cited for catastrophic forgetting. Verify the intended sources and correct the citations.","section":"References"},{"comment":"Spelling of 'Moffitt' is inconsistent between the author affiliation and the body text; similarly, 'Llama 2' / 'LLama 2' variants should be normalized.","section":"Throughout"},{"comment":"The paper mentions a '100,000 dimensional feature space' and also '100K English terms in our vocabulary'; clarify whether the feature dimension equals the vocabulary size and how the 100K was selected beyond frequency sorting.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript describes an interesting system, but the gap between the strong claims in the abstract and the evidence is very large. The paper would need substantial new evaluation—end-to-end RAG comparisons, KG precision measurement, and a user study—before it could be considered for publication. The internal contradiction between the 'unsupervised' characterization and the Data-Scientist/expert supervision described in Sections 2, 5.1, and 5.2 must also be corrected. I see this as a major revision rather than a reject because the architecture is plausible and the missing evaluation is, in principle, within the authors' capacity to supply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: CancerKG is a real engineering effort—a web-scale KG-LLM hybrid for colorectal cancer with multiple interfaces, structural table search, and 2.5 million PubMed papers processed. That part is genuine and could be useful if evaluated properly. But the central claims—that it serves users better than an LLM, KG, or search engine alone, and that the KG acts as a RAG guardrail so the system 'does not hallucinate'—are assertions, not findings. There is no end-to-end evaluation, no baseline comparison, no user study, and no measurement of KG precision or hallucination rates. The only quantitative result is about 95% F-measure for topical table classification on a self-constructed training set, which tests a component, not the hybrid.\n\nWhat is actually new: applying the authors' prior COVIDKG architecture to colorectal cancer, with different LLMs, datasets, and meta-profiles. The paper admits this similarity, so novelty is an application rather than a technique. The 3D meta-profile interface and structural search over non-normalized medical tables are nice touches.\n\nThe soft spots are proportional to the claims. The abstract and conclusion say content is 'extracted and updated in unsupervised manner,' but Section 5.1 has a Data Scientist seed the KG and Section 5.2 sends low-confidence fusions to an expert. That is a direct internal contradiction. Even with automatic extraction, peer-reviewed status of a PubMed article is not verification of medical correctness. The fusion section reports no error rates, no expert adjudication counts, no post-correction precision. The guardrail claim is unfalsifiable as presented—there is no controlled comparison of grounded versus ungrounded LLM outputs. The Moffitt evaluation is explicitly future work.\n\nThe system may work in practice, and the engineering is substantial, but the paper is not a research contribution as it stands. It is a systems report. That said, it is the kind of paper a good referee could push into shape: ask for a small user study, a KG precision audit, and a direct comparison against the LLM alone. Those are demanding but doable.\n\nMy recommendation: do not desk-reject. Send it to review with a clear request for end-to-end evaluation. A serious referee will identify the same gaps, but the systems community might still value a well-scoped description once the overclaims are trimmed.\n\nBest","headline":"CancerKG is a useful system description whose superiority and safety claims are unsupported; give it a serious referee to force an honest evaluation.","tokens_in":13363,"tokens_out":3292,"would_cite":false,"duration_ms":30011,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that CancerKG, a web-scale knowledge-graph/large-language-model hybrid built largely automatically from 2.5 million peer-reviewed colorectal cancer publications, serves user needs better than any component alone, with the…","keywords":["knowledge graph","large language model","retrieval-augmented generation","colorectal cancer","medical information retrieval","table search","hallucination mitigation","knowledge fusion"],"falsifier":"Take a set of 100 clinical questions with known answers from recently published colorectal cancer trials, then check whether every cited graph node exists, matches the cited publication, and supports the language model's answer; if even a small percentage of answers cite nodes absent from the graph or contradict the cited paper, the verifiability and anti-hallucination claim collapses.","tokens_in":12287,"feed_emoji":"🩺","tokens_out":10515,"duration_ms":91996,"temperature":0.7,"pith_summary":"CancerKG.ORG is a web-scale hybrid that joins a knowledge graph, built largely automatically from the latest peer-reviewed colorectal cancer literature after a small manual seed, with fine-tuned large language models; the paper's central claim is that this combination serves users better than any component alone. The authors argue that large language models hallucinate, forget, and train on outdated data, while manually curated medical knowledge graphs quickly become stale, so they make the verified graph act as a retrieval-augmented generation guardrail that keeps answers traceable to cited publications. If the claim holds, clinicians, researchers, and patients could get current, actionable cancer information without the staleness of curated databases or the fabrication risk of bare language models. Colorectal cancer is presented as a model domain, with the architecture claimed to transfer to other scientific and medical areas.","feed_headline":"CancerKG grounds LLM cancer answers in a verified knowledge graph","feed_subtitle":"It ingests 2.5 million peer-reviewed papers so clinicians can query current, cited knowledge without invented facts.","key_machinery":"The central object is the verified hierarchical knowledge graph: a scalable graph database seeded with a small vetted set of nodes and edges, then continuously enriched by a fusion process. Fusion matches the root of each newly extracted subtree to existing graph nodes through normalized term matching, amended by embedding-driven matching for terms never seen before; subtrees with several layers and low-confidence matches are kept separate or sent to expert review, and the fusion model is expected to learn from expert corrections over time. The other load-bearing mechanism is the retrieval loop around the graph: a conversational query parser splits a natural-language question into a structural table query and a text query, a table search engine ranks results with custom table embeddings, and a fine-tuned large language model generates the final answer using the retrieved evidence. Supporting machinery includes binary topical table classifiers built on fine-tuned biomedical language-model embeddings and the 3D meta-profile, a summary of a table cluster's horizontal and vertical metadata used for browsing and filtering.","core_discovery":"The paper reports construction of one of the first web-scale knowledge-graph/large-language-model hybrids for a cancer domain, storing more than 2.5 million parsed, peer-reviewed publications and organizing them into a hierarchical, browsable graph. The discovery at the center of the paper is that an automated pipeline can keep such a graph current and can use it to discipline the language model: a small vetted graph is seeded manually, topical clusters of tables are extracted by trained classifiers, new knowledge is fused in by matching normalized terms and embedding vectors, and the resulting verified graph is used as a retrieval-augmented generation guardrail so answers point back to the literature. The authors report approximately 95% F-measure for the topical table classifiers under 10-fold cross-validation and describe the system as currently being evaluated for medical research and clinical information retrieval tasks at a major U.S. cancer center. Five interfaces are presented as serving different data modalities: interactive graph browsing, publication and structural table search engines, a conversational interface, and 3D meta-profiles.","pith_inferences":["The decisive experiment the paper does not report is a head-to-head comparison on identical clinical questions between the hybrid, a bare large language model, and a keyword search engine, scored against expert-verified answers; that comparison would directly test the better-than-any-part-in-isolation claim.","Because the paper reports only that low-confidence fusions go to an expert and that mistakes are later learned, the unsupervised label is partial; measuring the fraction of graph updates that require human sign-off would reveal the true maintenance cost.","Embedding-driven fusion could merge or confuse distinct entities whose names are similar, so a targeted evaluation of entity identity under synonym and spelling variation would show whether graph precision holds at scale.","The same guardrail design could plausibly transfer to other clinical domains, but only if each domain can supply the seed nodes and table schemas the pipeline needs."],"forward_implications":["Users can trace every answer to specific peer-reviewed publications, because the graph constrains generation to verified content.","Because ingestion is automatic, the system can stay current in a way fixed-training-cutoff language models and manually curated databases cannot.","Structural table search lets clinicians query concrete clinical attributes such as tumor size and lymph-node status instead of relying on keyword search over whole documents.","If the architecture is truly domain-independent, the same pipeline can be retrained for other fast-moving medical or scientific fields.","The verified-graph guardrail idea applies to any setting where language-model answers must be checked against an evolving corpus."],"supporting_citations":[{"why":"Supplies the high-level cancer characteristics used to seed the knowledge graph and represents the manually curated baseline the hybrid aims to supersede.","marker":"[26]"},{"why":"Provides the predecessor architecture for knowledge-graph construction and table classification that this system extends to a new disease domain and to language-model grounding.","marker":"[76]"},{"why":"Offers pretrained language-model embeddings that the pipeline fine-tunes for table classification and embedding-driven fusion matching.","marker":"[31]"},{"why":"Supplies scalable hierarchical metadata classification for tables with horizontal and vertical metadata, which underlies the structural table search.","marker":"[53]"},{"why":"Provides a large public web-table corpus used, alongside the cancer dataset, to compose training sets for the topical table classifiers.","marker":"[60]"},{"why":"One of the large language models available in the conversational interface, representing the generation component the graph is designed to guard.","marker":"[61]"}],"fun_headline_variants":["CancerKG uses a verified knowledge graph to stop LLM hallucinations","Automated cancer knowledge graph keeps LLM answers current and cited","Web-scale cancer KG-LLM hybrid cites 2.5M papers on demand","CancerKG's self-updating graph grounds AI cancer answers in evidence","Hybrid system merges LLM with 2.5M-paper graph for cancer care"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic extraction-and-fusion pipeline keeps the graph medically correct: if embedding similarity plus occasional expert review of low-confidence merges cannot prevent wrong or outdated nodes from entering the graph, the verified guardrail would lock in and amplify errors rather than prevent them.","fun_headline_variants_meta":{"raw":{"variants":["CancerKG uses a verified knowledge graph to stop LLM hallucinations","Automated cancer knowledge graph keeps LLM answers current and cited","Web-scale cancer KG-LLM hybrid cites 2.5M papers on demand","CancerKG's self-updating graph grounds AI cancer answers in evidence","Hybrid system merges LLM with 2.5M-paper graph for cancer care"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2462,"prompt_tokens":965,"completion_tokens":1497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1398}},"tokens_in":581,"tokens_out":1497,"duration_ms":30683,"temperature":1.0,"reasoning_tokens":1398,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:55:51.251600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of 100 clinical questions with known answers from recently published colorectal cancer trials, then check whether every cited graph node exists, matches the cited publication, and supports the language model's answer; if even a small percentage of answers cite nodes absent from the graph or contradict the cited paper, the verifiability and anti-hallucination claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the predecessor architecture for knowledge-graph construction and table classification that this system extends to a new disease domain and to language-model grounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies scalable hierarchical metadata classification for tables with horizontal and vertical metadata, which underlies the structural table search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a large public web-table corpus used, alongside the cancer dataset, to compose training sets for the topical table classifiers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the large language models available in the conversational interface, representing the generation component the graph is designed to guard."}],"review_version":1}