{"id":"66d75f9c-3174-4768-a2de-cd1bdce1a609","arxiv_id":"2501.13984","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A system that enriches an NCCN NSCLC guideline knowledge graph with LLM node labels and uses LLM-generated Cypher queries for question answering, without a direct evaluation of answer correctness.","lead":"This paper builds a knowledge graph from an NCCN lung cancer guideline, automatically labels its nodes using GPT-3.5, and answers natural language questions by converting them to graph queries. The authors claim high-quality QA with reduced hallucination, but they never evaluate the factual correctness of the final answers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated relation-to-template mapping (Table II) can misrepresent guideline qualifiers and conditions, so the claimed factual accuracy of generated answers is unsupported.","rationale":"The reader's weakest_assumption pointed to the same unvalidated mapping in Table II. I agree that this is the most load-bearing concern because it is a concrete, necessary component of the answer-generation pipeline and a plausible source of clinical errors. The paper has genuine strengths: the manual construction of the enriched knowledge graph, the transparent LLM classification experiments, and the clear pipeline description. However, the central claim of 'factual accuracy' in medical QA is not supported by any evaluation of the final answers. The Cypher query error rates only measure whether the right subgraph is retrieved, not whether the natural-language answer is correct. The template mapping is the bridge between subgraph and answer, and its fidelity is assumed without evidence. A wrong template could produce confident but misleading answers even when retrieval is perfect, so this is the weakest link in the argument. The concrete test I propose would settle the concern by directly assessing answer correctness on a subset of error-free queries. If the test passes, the concern is mitigated; if it fails, the paper's headline claim is overstated. Since the reader's verdict was REJECT and my analysis reinforces that conclusion, the verdict should remain unchanged.","tokens_in":14273,"tokens_out":3928,"duration_ms":36699,"concrete_test":"Take the 59 test questions for which the auto-generated Cypher query was error-free (Table III) and run the full pipeline to produce final natural language answers. Have two independent oncologists compare each answer directly against the relevant NCCN guideline passages, rating each as 'consistent', 'partially consistent' (missing a qualifier/condition), or 'contradictory'. If any answer is rated 'contradictory', or if more than 10% are rated 'partially consistent', the template mapping is not faithful and the factual accuracy claim fails. This test isolates the template mapping from query-generation errors by using only error-free queries, and it directly checks whether the generated text preserves guideline semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that answers generated from subgraph paths and semantic information ensure factual accuracy. This depends on the hand-crafted mapping in Table II, which translates the three relation types ('requires', 'indicates', 'is followed by') into fixed natural-language templates. The paper assumes this mapping faithfully preserves the clinical meaning of every node combination, but the mapping is never validated. For example, 'requires' from a Disease Condition to a Treatment Option becomes 'use the treatment', which flattens critical qualifiers like 'consider', 'preferred', or 'category 2B' that appear in the node text. Similarly, 'is followed by' between Treatment Options becomes 'After the treatment is over, further use the treatment', which ignores conditions such as 'if progression' or 'if not previously done'. Because the three relation types are coarse, the templates cannot capture the conditional logic and nuances of the NCCN guidelines. Even when the retrieved subgraph is correct, the final answer may be clinically misleading or outright wrong. This directly undermines the paper's claim of 'ensuring factual accuracy'. The absence of any evaluation of the final answers against clinician judgment or a reference set means this assumption is not just unproven but entirely unexamined. The two sample answers in the paper are illustrative only and show awkward, possibly incomplete phrasing, but no correctness assessment is provided. Thus, the load-bearing weak point is the unvalidated template mapping, which is a necessary step between the subgraph and the claimed factually accurate answer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a method for converting the graphical flows of the NCCN Non-Small Cell Lung Cancer guideline into a knowledge graph with three node types and three relation types, using manual labeling and an automated LLM-based node classification with zero-shot and few-shot prompting. It then proposes a question-answering pipeline in which an LLM converts natural language questions into Cypher queries over a Neo4j database, retrieves a subgraph, and renders the subgraph as a natural language answer using a fixed template mapping. The authors report node classification accuracies of 80.86% (zero-shot) and 88.47% (few-shot), and report that 66.10% of automatically generated Cypher queries had no errors. They claim that the approach provides factually accurate answers and mitigates LLM hallucination while requiring far less training data than prior deep learning methods.","tokens_in":14532,"tokens_out":6065,"duration_ms":52242,"significance":"If the results were valid, the paper would offer a low-data method for building executable representations of cancer treatment guidelines and answering guideline-restricted questions. The paper has some strengths: it uses real NCCN guideline data, provides a clear error taxonomy for Cypher queries, gives full prompts for node classification, and makes the comparison between zero-shot and few-shot explicit. However, the two central claims are not supported. The few-shot node classification accuracy is obtained with few-shot examples selected from the test set, so the reported gain is not evidence of generalization. More importantly, the claimed 'factual accuracy' of the QA system is never evaluated; only Cypher query errors are measured, and the hand-crafted template mapping that produces the final answers is unvalidated. These issues are load-bearing rather than cosmetic.","major_comments":[{"comment":"The few-shot node-classification evaluation is circular. The paper states that 'Twenty-three distinct nodes from the incorrect node classifications obtained using the previous technique are given as training instances' and then reports few-shot accuracy of 88.47% on the same set of nodes used for the zero-shot evaluation. Because the prompt examples are drawn from the very test instances being classified, the few-shot result is at least partly a fit to the test distribution and does not measure generalization. A valid few-shot evaluation would select training examples from a separate development split, not from the errors on the test set, or would use cross-validation.","section":"Section II.B"},{"comment":"The central claim that the method ensures 'factual accuracy' and mitigates hallucination is not evaluated anywhere. Section IV.B only reports error rates of the auto-generated Cypher queries (Table III), not the correctness of the final natural language answers. No reference answer set, no clinician review, and no automatic metric are applied to the generated answers. The two sample answers in the text are illustrative only and are not scored. Without an evaluation of answer correctness, the paper provides no evidence for its headline claim.","section":"Section IV.B and Abstract"},{"comment":"The hand-crafted graph-to-text mapping in Table II is unvalidated and may not preserve clinically important qualifiers. For instance, the template for 'Disease Condition requires Treatment Option' renders 'use the treatment <destination>' regardless of whether the node text says 'consider', 'preferred', 'category 2B', or includes conditions such as 'if not previously done'. Likewise, 'Treatment Option is followed by Treatment Option' becomes 'After the previous treatment is over, further use the treatment', dropping branching conditions like 'if progression'. Because the final answers are generated by these templates, even a correct subgraph extraction can produce a misleading or overcommittal answer, which directly undermines the claimed factual accuracy. The authors should validate the templates against expert paraphrases or a reference answer set.","section":"Section II.C and Table II"},{"comment":"The comparison with prior work [15] is not well posed. The paper states that the proposed method requires only 13 training instances versus 5810 in [15], but it does not compare end-to-end answer accuracy. The only comparable quantity reported is Cypher query generation accuracy (66.1% no-error), which is not an answer accuracy. Thus the claim that the method achieves comparable or better QA performance with much less training data is unsupported.","section":"Section IV.B"}],"minor_comments":[{"comment":"The manuscript uses 'NSCL' instead of the standard abbreviation 'NSCLC' for non-small cell lung cancer in several places, including the Abstract and Section II.A.","section":"Throughout"},{"comment":"The manual node and relation labels serve as ground truth for the accuracy numbers, but no inter-annotator agreement or external validation is reported, so the reliability of the ground truth is unclear.","section":"Section II.A"},{"comment":"The 'Overall Error (%)' column appears to be computed over the combined test sets (e.g., 4/59 = 6.78% for Type-I), but this is not stated explicitly; the table should clarify the denominators used.","section":"Table III"},{"comment":"The sample answer for the Set B question concatenates 'Stage IIIB (T4, N2) Stage IIIC (T4, N3)' without a conjunction, which is a readability issue and also illustrates how the template mapping flattens node boundaries.","section":"Section IV.B"},{"comment":"Reference [40] points to a ChatGPT product page rather than to the official model documentation or paper for gpt-3.5-turbo-instruct; the citation should be updated.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper has two fundamental evaluation gaps that go beyond presentation issues: the few-shot node classification experiment leaks test information into the prompt examples, and the QA system's central claim of factual accuracy is never measured. These are not local fixes; they require new experimental designs and substantial additional evaluation. The manuscript in its current form is not a reliable basis for the claimed contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds a context-enriched knowledge graph from the NCCN NSCLC guidelines (538 nodes, 707 relations) and then uses GPT-3.5 to classify nodes and translate natural-language questions into Cypher queries. That pipeline is real and described in enough detail that someone could rebuild it. The enriched graph itself, with the 'context' property and the three-node/three-relation taxonomy, is a reasonable contribution for a resource paper. The zero-shot node classification at 80.86% is a legitimate result, even if not surprising for GPT-3.5 on a three-way task.\n\nThe problems are in the evaluation. The few-shot node classification number (88.47%) is indeed inflated: the 23 training examples are exactly the nodes misclassified by zero-shot on the same test set, and then accuracy is measured again on that same set. That is circular. It measures memorization of the test distribution, not generalization. This is not a minor caveat; it breaks the comparison between 80.86% and 88.47% as evidence for few-shot improvement.\n\nThe QA evaluation has a different but equally load-bearing gap. The paper reports only Cypher query error rates (66.10% error-free overall) and never evaluates the medical correctness of the final natural-language answers. The two sample answers read awkwardly and in places literally restate the same stage text twice, but no reference-answer comparison or clinician assessment is provided. On top of that, the template mapping in Table II is coarse: 'requires' from Disease Condition to Treatment Option always becomes \"use the treatment,\" which flattens qualifiers like 'consider', 'preferred', or 'category 2B' that appear in the node text. The stress-test note is right that this can produce confidently worded but clinically misleading answers even when the retrieved subgraph is correct. The paper's abstract claims 'ensuring factual accuracy,' and that claim is simply not supported by the evidence presented.\n\nThere is no code or data release, so the resource value is harder to verify independently. Still, the paper is not incoherent. It identifies a real problem, assembles a plausible pipeline, and honestly describes the Cypher error types. It just overclaims on both node classification and answer accuracy.\n\nMy take: this deserves a serious referee, but with the expectation of major revision. The authors need to re-run the few-shot evaluation with held-out examples, add an evaluation of answer correctness against a clinician-built reference set, and soften or qualify the claims about hallucination and factual accuracy. For me personally, I would not cite the results as they stand, but I would bring it to a reading group to discuss evaluation leakage in LLM pipelines.","headline":"A useful systems description undermined by a circular few-shot evaluation and no direct check of answer correctness; the right call is major revision, not outright rejection without review.","tokens_in":15105,"tokens_out":1372,"would_cite":false,"duration_ms":14830,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A typed guideline graph lets an LLM answer cancer questions from guideline text rather than from memory.","keywords":["clinical practice guidelines","knowledge graph","large language models","question answering","node classification","few-shot learning","NCCN","non-small cell lung cancer"],"falsifier":"Have independent oncologists check the answers generated for the paper's 59 held-out questions against the NCCN text, separating retrieval failures from wording failures: if a large fraction of correctly retrieved subgraphs still produce clinically wrong sentences because of the template, the factual-accuracy claim fails.","tokens_in":14062,"feed_emoji":"🩺","tokens_out":7366,"duration_ms":796303,"temperature":0.7,"pith_summary":"The paper claims that the treatment flows in an NCCN cancer guideline can be converted into a typed, context-enriched graph, and that this graph can ground a question-answering system where a large language model only writes the database query rather than the medical answer. The answer sentence is assembled from the retrieved subgraph's nodes and edges using a hand-built mapping, so the model never has to recall clinical facts from memory. The payoff, if the claim holds, is a guideline QA system that needs only 13 labeled query examples instead of the roughly 5,810 used by the prior deep-learning baseline, while keeping answers traceable to guideline text. The paper reports 80.86% zero-shot and 88.47% few-shot accuracy for automated node labeling, and 66.1% of auto-generated Cypher queries ran without error.","feed_headline":"Graph-grounded LLM answers NCCN lung-cancer queries with 13 examples","feed_subtitle":"The LLM writes only the query; the guideline graph supplies the path, keeping answers inside the text.","key_machinery":"The central object is the enriched guideline knowledge graph: 538 nodes typed as Disease Condition, Treatment Option, or Evaluation, carrying the guideline text plus a context property taken from page-level labels, connected by 707 directed edges typed as requires, indicates, or is followed by. The argument runs on three coupled mechanisms: LLM prompting for node labeling, LLM-based Cypher generation against the Neo4j graph, and a graph-semantics-to-text template (Table II) that converts each typed edge in the retrieved subgraph into a fixed natural-language sentence. The template step is what turns graph structure into clinically worded output.","core_discovery":"The central discovery claimed is that a semantically enriched guideline graph is enough to make LLM-based question answering faithful: the LLM is confined to translating a natural-language question into Cypher, the graph database returns a subgraph, and a template mapping from relationship types to natural-language phrases turns each path into a clinical sentence such as \"If the disease condition is X, use the treatment Y.\" The authors state that this design mitigates the hallucination risk that has blocked LLMs in medicine, because every generated statement corresponds to a node or edge present in the guideline graph. They support the claim with the node-classification accuracies, the error analysis of auto-generated Cypher queries, and the comparison with the prior deep-learning system that required far more training data.","pith_inferences":["A fair test of the approach should validate the Table II template mapping against oncologist-authored answers; the paper does not report such a validation, and a template that misphrases a relationship could produce fluent but misleading answers from a correctly retrieved subgraph.","The node-classification accuracy figures are computed against the authors' own manual labels, which is a reasonable first check but not the same as agreement with an independent clinician-annotated gold standard.","The same retrieve-then-render design could be applied to open-weight LLMs and to other guideline publishers, and the comparison the authors list as future work would show whether the Cypher-generation step depends on a particular model.","A direct ablation would compare the template-based answers with answers that quote the retrieved subgraph's node texts verbatim, isolating how much medical wording the templates contribute and whether they ever distort it."],"forward_implications":["A guideline QA system can be built with 13 labeled query examples rather than thousands of question-answer pairs, which lowers the cost of updating answers when guidelines change.","Because every answer sentence is derived from an explicit graph path, a clinician can trace a recommendation back to the guideline nodes that generated it.","Few-shot node labeling at 88.47% makes it practical to enrich new guideline graphs automatically, with only a small set of corrected examples per guideline.","LLM failure modes appear before answer generation: roughly one third of Cypher queries need manual fixing, and content mismatches are the most common error."],"supporting_citations":[{"why":"provides the automated guideline extraction tool that produces the initial JSON-LD knowledge graph from the NCCN CPG.","marker":"[14]"},{"why":"supplies the prior decision-knowledge-graph QA system with 67.6% accuracy and 5,810 training instances that this work is compared against.","marker":"[15]"},{"why":"is the NCCN Non-Small Cell Lung Cancer guideline version 2.2024 that the graph and all questions are built from.","marker":"[28]"},{"why":"is the GPT-3.5 model used for zero-shot and few-shot node classification and for Cypher query generation.","marker":"[40]"},{"why":"is the Neo4j graph database that stores the guideline knowledge model and executes the subgraph extraction queries.","marker":"[41]"},{"why":"is the Natural Language Queries extension that translates the user's question into Cypher within the NeoDash plugin.","marker":"[42]"},{"why":"motivates the design of constraining the LLM to extract information from a structured knowledge base rather than generating answers from memory.","marker":"[27]"},{"why":"supports the few-shot prompting strategy for knowledge-base question answering that the node-labeling step adopts.","marker":"[44]"}],"fun_headline_variants":["Guideline graph confines LLM to truthful cancer answers","Graph-grounded LLM answers cancer queries with 80-88% accuracy","Cancer CPGs as graphs keep LLM answers on the page","Subgraph paths keep LLM cancer answers factual"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-built mapping from graph relationships to natural-language sentences preserves the medical meaning of every pathway, since the paper never checks that mapping against expert clinician phrasing or a gold answer set.","fun_headline_variants_meta":{"raw":{"variants":["Guideline graph confines LLM to truthful cancer answers","Graph-grounded LLM answers cancer queries with 80-88% accuracy","Cancer CPGs as graphs keep LLM answers on the page","Subgraph paths keep LLM cancer answers factual"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1566,"prompt_tokens":925,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":572}},"tokens_in":541,"tokens_out":641,"duration_ms":6602,"temperature":1.0,"reasoning_tokens":572,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:57:07.851416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent oncologists check the answers generated for the paper's 59 held-out questions against the NCCN text, separating retrieval failures from wording failures: if a large fraction of correctly retrieved subgraphs still produce clinically wrong sentences because of the template, the factual-accuracy claim fails.","supporting_citations":[{"cited_title":"Automated Knowledge Modeling for Cancer Clinical Practice Guidelines,","cited_arxiv_id":null,"evidence_quote":"provides the automated guideline extraction tool that produces the initial JSON-LD knowledge graph from the NCCN CPG."},{"cited_title":"Decision Knowledge Graphs: Construction of and Usage in Question Answering for Clinical Practice Guidelines","cited_arxiv_id":"2308.02984","evidence_quote":"supplies the prior decision-knowledge-graph QA system with 67.6% accuracy and 5,810 training instances that this work is compared against."},{"cited_title":"Uncertainty-Aware Text- to-Program for Question Answering on Structured Electronic Health Records,","cited_arxiv_id":null,"evidence_quote":"is the NCCN Non-Small Cell Lung Cancer guideline version 2.2024 that the graph and all questions are built from."},{"cited_title":"Query a neo4j database using cypher,","cited_arxiv_id":null,"evidence_quote":"is the GPT-3.5 model used for zero-shot and few-shot node classification and for Cypher query generation."},{"cited_title":"ChatGPT - open a I gpt 3.5,","cited_arxiv_id":null,"evidence_quote":"is the Neo4j graph database that stores the guideline knowledge model and executes the subgraph extraction queries."},{"cited_title":"Neo4j graph database & analytics,","cited_arxiv_id":null,"evidence_quote":"is the Natural Language Queries extension that translates the user's question into Cypher within the NeoDash plugin."},{"cited_title":"Assessing the research landscape and clinical utility of large language models: a scoping review,","cited_arxiv_id":null,"evidence_quote":"motivates the design of constraining the LLM to extract information from a structured knowledge base rather than generating answers from memory."},{"cited_title":"NeoDash - Dashboard Builder for Neo4j,","cited_arxiv_id":null,"evidence_quote":"supports the few-shot prompting strategy for knowledge-base question answering that the node-labeling step adopts."}],"review_version":1}