{"id":"945b6b72-3895-49ac-a982-053de6db7a5e","arxiv_id":"2504.12422","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LinkQ's knowledge-graph-guided querying beats plain GPT-4 on a 120-question benchmark, but still struggles with multi-hop and intersection questions.","lead":"This paper evaluates LinkQ, a system that forces a chatbot to answer questions by querying a knowledge graph of verified facts instead of relying on memory, and reports that it outperforms plain GPT-4 on a 120-question benchmark. It matters because it demonstrates a concrete guardrail against AI hallucinations in high-stakes fields such as cybersecurity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The lenient 'correct at least once in three attempts' scoring rule is the load-bearing assumption; without per-attempt counts or stricter-rule results, the claimed advantage on Multi-hop and Intersection questions is not yet established.","rationale":"The central claim is directional and supported by a clear mechanism: LinkQ grounds answers in KG query execution rather than free-form LLM generation, and the evaluation uses a standard KGQA dataset. The main weakness is that the reported metric is an upper-bound 'ever correct' rate, which is especially problematic because the paper targets high-stakes cyber operations where reliability, not occasional success, is the relevant property. The reader already identified this scoring rule as the weakest assumption, and the CONDITIONAL verdict is appropriate: the result is plausible but requires additional per-attempt analysis to confirm that the advantage is not an artifact of the lenient rule. I do not see a more fundamental flaw: the baseline uses the same underlying model, the evaluation is framed as a case study, and the qualitative study is exploratory rather than evidence for the central quantitative claim. The concrete check is straightforward because the per-attempt breakdown is partially visible in Figure 3 and the repository is linked; if the advantage persists under stricter rules, the paper's conclusions stand, and if not, the claims should be softened. Therefore, the reader's verdict does not need to change.","tokens_in":7265,"tokens_out":4652,"duration_ms":50260,"concrete_test":"Using the open-source repository linked in the paper, reconstruct per-question, per-attempt outcomes for all 120 questions and recompute accuracy under three scoring rules: at least 1/3 (as reported), at least 2/3, and 3/3, for both LinkQ and GPT-4, broken down by question type. Report exact counts, Clopper-Pearson 95% confidence intervals, and a paired McNemar test for each type. If, under the 2/3 or 3/3 rule, LinkQ no longer beats GPT-4 on Multi-hop and Intersection, the central claim should be revised to a scoring-rule-dependent result; if the advantage persists, this concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines correctness as 'a correct answer at least once from those three attempts.' This rule measures peak capability, not reliability, while the paper's central claim is about mitigating hallucinations in high-stakes QA: a system that succeeds in only one of three attempts still produces wrong or empty answers most of the time. The same rule is applied to GPT-4, so the comparison is not unfair, but it inflates both systems' accuracies; if LinkQ's advantage is concentrated in 1/3 successes, the conclusion that LinkQ 'strongly outperforms GPT-4' (Section 3.2) may not survive a 2/3 or 3/3 rule. Figure 3(a) shows a 1/3, 2/3, and 3/3 breakdown, so the per-attempt information likely exists, but the paper reports no exact counts, confidence intervals, or paired significance tests. For example, Intersection is reported as 54.2% (13/24) versus 12.5% (3/24) under the 'at least once' rule; if many of LinkQ's 13 successes are 1/3 outcomes, an all-three rule could leave LinkQ much closer to GPT-4, and the headline finding would shrink to 'sometimes better.' With only n=24 per question type, a small number of discordant outcomes can change the conclusion. The text gives no justification for why occasional success is the appropriate reliability metric for a system intended to prevent hallucinations in high-stakes domains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LinkQ, an open-source LLM-assisted visual interface that translates natural-language questions into knowledge-graph (KG) queries to suppress hallucinations. It reports a quantitative evaluation on 120 Mintaka questions spanning five complexity types (Multi-hop, Comparative, Yes/No, Generic, Intersection), comparing LinkQ (which uses GPT-4 under the hood) against plain GPT-4. The authors find that LinkQ outperforms GPT-4 on every question type, with the largest gaps on Multi-hop (75.0% vs 16.7%) and Intersection (54.2% vs 12.5%). A qualitative study with two cybersecurity experts on the BRON KG provides feedback and suggestions for future systems. The paper concludes that LinkQ can improve the objective correctness of LLM-to-KG query translation without fine-tuning, though some complex question types remain challenging.","tokens_in":7592,"tokens_out":2899,"duration_ms":29368,"significance":"If the reported results hold, the paper offers a practical, fine-tuning-free approach to grounding LLM answers in structured knowledge, which is relevant for high-stakes domains such as cyber operations. Its strengths include the use of an external benchmark dataset (Mintaka), a human-in-the-loop design that is clearly described, and an open-source implementation that supports reproducibility. The qualitative study adds real-world context, but the evidence for the central quantitative claim is weakened by the lenient 'correct at least once in three attempts' scoring rule and the absence of statistical inference on small per-type samples.","major_comments":[{"comment":"The scoring rule, which counts a question as correct if at least one of three independently generated queries returns the right answer, measures peak capability rather than reliability. In a high-stakes setting, a system that succeeds only one time out of three still produces wrong or empty answers most of the time, so this rule is not clearly aligned with the paper's stated goal of mitigating hallucinations. Because Figure 3(a) already includes a 1/3, 2/3, 3/3 breakdown, the authors should report the exact per-attempt counts and present accuracy under stricter rules (e.g., at least two or all three attempts correct) and justify the chosen threshold.","section":"Section 3.1, Correctness Criteria"},{"comment":"With only 24 questions per type, the headline differences (e.g., Multi-hop 75.0% vs 16.7% and Intersection 54.2% vs 12.5%) are not accompanied by confidence intervals or significance tests. A Fisher's exact test or a bootstrap confidence interval would show whether the observed gaps are compatible with chance. Without such measures, the claim that LinkQ 'strongly outperforms' GPT-4 is not yet statistically supported, particularly for the categories with the largest reported advantage.","section":"Section 3.2 and Figure 3"},{"comment":"The prompt and parameters used for the plain GPT-4 baseline are described only as 'instructing it to generate an appropriate KG query.' This level of detail makes it difficult to assess whether the comparison is fair (e.g., whether the baseline received the same few-shot examples as LinkQ). The authors should include the baseline prompt, temperature, and other relevant settings in the repository or an appendix so that the comparison is reproducible.","section":"Section 3.1, Baseline"}],"minor_comments":[{"comment":"The flow diagram is information-dense; adding a concise bulleted list of the protocol steps in the text would help readers who do not parse the figure in detail.","section":"Section 2, Figure 2"},{"comment":"The phrase 'increases the objective correctness (and consequently the runtime)' is ambiguous about whether runtime increases or decreases and why correctness would affect runtime. Please clarify the intended relationship.","section":"Section 3.2"},{"comment":"There is a typo: 'BRON was too complex to for an LLM to navigate' should read 'too complex for an LLM to navigate.'","section":"Section 4"},{"comment":"The qualitative study relies on only two subject-matter experts; the paper should state this limitation explicitly when drawing broader conclusions about domain-expert usability.","section":"Section 4"},{"comment":"The procedure for selecting the 120 questions (random selection with criteria) is described, but the specific seed or code for the selection is not provided; including it would strengthen reproducibility.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a workshop-style venue focused on human-centered XAI. The main concern is the quantitative evaluation: the 'at least once' rule and lack of statistical inference make the headline claims more tentative than the text suggests. This is fixable with additional analysis of data that already appears to exist from the three runs. The qualitative study is a nice complement, but it is anecdotal and should be framed as such."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it is a genuine, reproducible evaluation of an existing system (LinkQ, from IEEE VIS 2024) against a fair baseline, not a new model or theory. The authors ran fresh experiments on a 120-question Mintaka subset and report LinkQ outperforming GPT-4 on every question type, with the biggest gaps on Multi-hop (75% vs 16.7%) and Intersection (54.2% vs 12.5%). They also provide a qualitative study with two cybersecurity experts using the BRON graph. On the positive side, the work is refreshingly honest: they openly state that LinkQ sometimes returns empty results, they give concrete expert suggestions for improvement, and they released the evaluation artifacts on GitHub. The baseline is fair—plain GPT-4 asked to generate a KG query—and the evaluation is anchored to an external dataset and an external KG, so there is no circularity issue. The main soft spot is the scoring rule. Counting a question as correct if only one of three independent attempts succeeds measures peak capability, not reliability. For a system designed to mitigate hallucinations in high-stakes domains, that is a meaningful mismatch. The comparison to GPT-4 uses the same rule, so it is not unfair, but it likely inflates both systems' absolute numbers, and the margin on Intersection and Multi-hop could shrink under a stricter 2-of-3 or 3-of-3 rule. To their credit, the authors show a 1/3, 2/3, 3/3 breakdown in Figure 3, but they do not report exact per-category counts or confidence intervals, and with only 24 questions per type, a few discordant trials could change the story. The curated question selection (manually screened to be answerable in Wikidata) is another limitation, though acceptable for a case study. The qualitative study with n=2 is exactly that—illustrative, not conclusive. These concerns are addressable: a sensitivity table with stricter scoring rules, per-attempt counts, and exact prompts would make the claim much stronger. This is a workshop-level contribution, useful for practitioners building grounded KGQA interfaces and for HCI researchers studying interactive systems in security domains. It deserves a serious referee, but a referee should ask the authors to add the stricter-rule analysis before acceptance.","headline":"A modest but honest case study: LinkQ beats plain GPT-4 at KG query generation on a curated 120-question Mintaka subset, though the 'at least one of three' scoring rule makes the size of the advantage uncertain on harder types.","tokens_in":8072,"tokens_out":2422,"would_cite":false,"duration_ms":25784,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Forcing an LLM to query a knowledge graph beats plain GPT-4 on every question type tested.","keywords":["LLM hallucination","knowledge graph question answering","LinkQ","Mintaka","BRON","GPT-4","human-in-the-loop","trustworthy AI"],"falsifier":"Re-run the same 120 Mintaka questions with LinkQ and plain GPT-4 scoring by majority or all-three agreement rather than at-least-one; if the gap on multi-hop and intersection questions disappears or reverses, the claim that LinkQ increases objective correctness without fine-tuning would be refuted.","tokens_in":7082,"feed_emoji":"🧠","tokens_out":5533,"duration_ms":51128,"temperature":0.7,"pith_summary":"This paper reports a case study of LinkQ, an interface that prevents a large language model from answering questions directly and instead requires it to build and run queries against a knowledge graph, so every answer is grounded in up-to-date, verified data. On 120 questions drawn from the Mintaka dataset, LinkQ outperformed a plain GPT-4 baseline on all five tested question types, with the largest gaps on multi-hop (75.0% versus 16.7%) and intersection (54.2% versus 12.5%) questions. The authors' central claim is that LinkQ's message-passing and prompting strategy increases the objective correctness of LLM question-to-query translation without fine-tuning. A qualitative session with two cybersecurity experts using the BRON knowledge graph showed the approach can answer domain-specific questions, though it sometimes returned empty results when data existed.","feed_headline":"Knowledge-graph grounding beats plain GPT-4 on every question type","feed_subtitle":"On 120 Mintaka questions, LinkQ's KG-query protocol scored higher on all five types, including multi-hop and intersection.","key_machinery":"The load-bearing mechanism is LinkQ's human-in-the-loop prompting protocol. When a user asks a question, the system intercepts it and has the LLM clarify the question, then lets the LLM fuzzy-search for entity and relation IDs in the knowledge graph; every data access is routed through the KG API, and the retrieved graph structure and ground-truth IDs are given back to the LLM in system messages. The LLM writes a graph query from a few-shot prompt, the user sees a query preview and can edit it, and the system executes the query and displays the tabular results with an LLM summary. This forces the LLM to derive answers from the KG's actual nodes and edges rather than from parametric memory, which is what suppresses hallucinations.","core_discovery":"The paper's central discovery is that routing an LLM's knowledge through a knowledge-graph query protocol measurably reduces hallucination. The authors show that LinkQ, which uses GPT-4 as its underlying model but adds system-intercepted entity and relation lookups, few-shot query examples, and a user preview step, answered more questions correctly than off-the-shelf GPT-4 on every complexity type tested: comparative 91.7% versus 20.8%, yes/no 87.5% versus 54.2%, generic 79.2% versus 33.3%, multi-hop 75.0% versus 16.7%, and intersection 54.2% versus 12.5%. They interpret the result as evidence that message passing and prompting alone, without fine-tuning, can raise objective correctness of KG query generation. They also find that LinkQ still struggles on multi-hop and intersection questions, and that in the BRON domain study it produced empty results rather than fabricated facts when the KG structure was not fully explored.","pith_inferences":["The reported accuracies use an at-least-one-of-three scoring rule; re-scoring the same outputs under a majority or all-three rule would lower both systems' scores and could shrink the gap on the hardest question types.","The qualitative finding that domain experts could correct empty-result queries suggests that LinkQ's benefit may depend on user familiarity with the KG schema; a non-expert may not know how to repair a query that returns no data.","The SME suggestion to search by node descriptions rather than properties points toward a hybrid that combines LinkQ's grounded queries with retrieval-augmented generation, a direction the paper does not implement.","Because only one baseline model was compared, the magnitude of LinkQ's advantage may vary with the underlying LLM; re-running the same protocol with a different model would test the generality of the claim."],"forward_implications":["LinkQ-style grounding is sufficient for straightforward analytical questions such as comparative, yes/no, and generic single-hop queries, which are common in initial data exploration.","Multi-hop and intersection questions remain the bottleneck, so future systems should decompose complex questions into smaller steps, possibly through chain-of-thought prompting, or ask the user to refine the query step by step.","Because the improvement comes from prompting and message passing rather than fine-tuning, the approach can be transferred to new domain-specific knowledge graphs without retraining the model.","When a translation fails, the system should hand control back to the user rather than improvise an answer, preserving the no-hallucination guarantee.","The accuracy gain comes at a runtime cost, since LinkQ takes longer than plain GPT-4 to generate a query."],"supporting_citations":[{"why":"Describes LinkQ's original system and prompting protocol that this case study evaluates.","marker":"[15]"},{"why":"Provides the Mintaka complex-question dataset from which the 120 evaluation questions were sampled.","marker":"[20]"},{"why":"Supplies the few-shot learning technique used to teach the LLM to write KG queries.","marker":"[4]"},{"why":"The Wikidata knowledge graph used as ground truth in the quantitative evaluation.","marker":"[24]"},{"why":"Introduces the BRON cybersecurity knowledge graph used in the qualitative expert study.","marker":"[8]"},{"why":"Motivates running each question three times because LLM outputs are probabilistic at non-zero temperature.","marker":"[3]"}],"fun_headline_variants":["KG grounding lifts LLM accuracy by up to 71 points","Knowledge-graph queries cut LLM hallucinations across question types","LinkQ outperforms GPT-4 on every Mintaka question type","Grounding LLMs in KGs beats off-the-shelf GPT-4","LLM with KG lookups outperforms plain GPT-4 in QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A question counts as correctly answered if at least one of three independently generated queries returns the right answer, and the paper reports accuracies without confidence intervals; if correctness instead required all three or a majority of attempts, both systems' scores would fall and LinkQ's margin could shrink.","fun_headline_variants_meta":{"raw":{"variants":["KG grounding lifts LLM accuracy by up to 71 points","Knowledge-graph queries cut LLM hallucinations across question types","LinkQ outperforms GPT-4 on every Mintaka question type","Grounding LLMs in KGs beats off-the-shelf GPT-4","LLM with KG lookups outperforms plain GPT-4 in QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1761,"prompt_tokens":910,"completion_tokens":851,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":760}},"tokens_in":526,"tokens_out":851,"duration_ms":8723,"temperature":1.0,"reasoning_tokens":760,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:31:25.069291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 120 Mintaka questions with LinkQ and plain GPT-4 scoring by majority or all-three agreement rather than at-least-one; if the gap on multi-hop and intersection questions disappears or reverses, the claim that LinkQ increases objective correctness without fine-tuning would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Wikidata knowledge graph used as ground truth in the quantitative evaluation."}],"review_version":1}