{"id":"5090a5ce-0e93-4065-9c9a-179dbff5ad6f","arxiv_id":"2412.12643","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"READS reframes KGQA as three constrained selection tasks and reports SOTA results on WebQSP and CWQ with a 7B LLM.","lead":"READS decomposes knowledge graph question answering into three discriminative subtasks, searching, pruning, and answering, and uses a fine-tuned Llama2-7b to select options instead of generating free text. It reports state-of-the-art Hits@1 of 0.840 on WebQSP and 0.802 on CWQ, improving over generative baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA may depend on gold SPARQL queries being used to expand test-time search subgraphs (Appendix B); the paper does not state whether this expansion is train-only, so the central comparison may not be clean.","rationale":"Reading in good faith, READS is a coherent framework: the three subtasks are well-motivated, the discriminative inference rules in Equations (6)-(8) are parameter-light, and the ablations in Table 3 support the claim that each subtask contributes. The reader's verdict of CONDITIONAL is appropriate. However, the single most load-bearing assumption is not merely that entities are pre-linked or that gold SPARQL is used to construct training data; it is whether gold SPARQL structure is injected into the test-time search space during preprocessing. The wording in Appendix B is genuinely ambiguous: it describes expanding subgraphs 'using the SPARQL queries provided in the benchmarks' immediately after describing per-question BFS extraction, without restricting the expansion to training splits. If the expansion reaches test questions, the graph-searching subtask becomes a selection task over a gold-informed candidate graph, which would explain the large retrieval-quality gains in Table 2 and would undermine the headline SOTA comparison. This is not a claim of misconduct; it is a specific, checkable ambiguity in a paper that has not released code or data. Until the authors either confirm the expansion is train-only or provide a leakage-free rerun, the empirical central claim cannot be fully verified. The appropriate verdict therefore remains CONDITIONAL, unchanged from the reader's assessment, with the condition sharpened to require clarification of this preprocessing detail.","tokens_in":14768,"tokens_out":6436,"duration_ms":55901,"concrete_test":"Inspect the released preprocessing code (or ask the authors) to determine whether the SPARQL-query expansion in Appendix B is applied to the test split. Then rerun READS on WebQSP and CWQ with the expansion disabled for test questions (training-only expansion), keeping all other settings fixed. If Hits@1 or F1 falls by more than about 0.03 relative to Table 1, the reported SOTA is substantially attributable to gold-query leakage rather than to the discriminative strategy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that discriminative subtask decomposition beats generative and interactive KGQA baselines, rests on a clean evaluation. The cleanest threat is in Appendix B's 'Freebase preprocessing.' The paper says it 'extract[s] subgraphs from Freebase using breadth-first search for each question' and then 'expand[s] these subgraphs using the SPARQL queries provided in the benchmarks to ensure the presence of constraint branches,' adding these to 'the original dataset.' It never states that this SPARQL-query expansion is limited to training examples. If test-set subgraphs are also expanded from the benchmark's gold SPARQL, then the graph-searching subtask in Equation (6) is not searching an unbiased knowledge graph; it is selecting edges from a candidate graph already engineered to contain the gold reasoning structure. That would inflate the relation-recall and graph-edit-distance numbers in Table 2 and the end-to-end Hits@1/F1 in Table 1, and it would make the comparison to RoG, ToG, and KG-Agent unfair. The Limitations section only acknowledges gold queries as a labeled-data requirement (item 2), not as a possible test-time search-space augmentation. Since code is not released, the ambiguity cannot be resolved from the manuscript alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes READS, a KGQA framework that replaces open-ended LLM generation for subgraph planning and answer reasoning with three discriminative subtasks: (i) iterative subgraph searching in which the LLM selects the next (entity, relation) triple from an option pool built from the current abstract subgraph structure; (ii) subgraph pruning in which constraint entities are mapped to positions in the abstract structure with comparison operators; and (iii) answer inference in which the LLM selects a semantic node whose entity group forms the answer. The framework is trained on samples derived from gold SPARQL queries in the WebQSP and CWQ training sets and evaluated on those benchmarks (plus a GrailQA dev-set experiment), reporting Hits@1/F1 of 0.840/0.845 on WebQSP and 0.802/0.820 on CWQ, above the listed generative and interactive baselines. The paper also reports search-quality metrics (relation recall and minimum graph edit distance), ablations, error-type frequencies, data-efficiency curves, and a base-model universality table.","tokens_in":14927,"tokens_out":8647,"duration_ms":75918,"significance":"Conditional on a clean evaluation, the paper makes a valuable contribution: it demonstrates that reformulating KGQA as constrained discriminative selections can outperform generative and interactive KGQA methods with a 7B-scale model, and it offers a concrete mechanism for the improvement (reduced ungrounded subgraph and reasoning generation). The ablations in Table 3 and the error-type analysis in Table 11 are consistent with that mechanism, and Table 2 provides a useful decomposition of search quality. The data-efficiency and model-universality analyses are also informative. The main caveat is that the headline state-of-the-art numbers depend on the exact construction of the test-time search space, which is not currently described unambiguously in Appendix B; the entity-linking assumption also needs quantification. The paper does not yet release code or data, so its reproducibility strength cannot currently be verified.","major_comments":[{"comment":"The manuscript does not state whether the benchmark-SPARQL subgraph expansion in Appendix B is applied only to training questions or also to test questions. The passage says the authors 'extract subgraphs from Freebase using breadth-first search for each question' and then 'expand these subgraphs using the SPARQL queries provided in the benchmarks to ensure the presence of constraint branches,' adding these to 'the original dataset.' If test-time subgraphs are also expanded from gold SPARQL, the 'searching' subtask in Equation (6) is not searching an unbiased graph; the candidate option pool already contains branches derived from the gold reasoning structure. That would inflate the relation-recall and graph-edit-distance numbers in Table 2 and the end-to-end Hits@1/F1 in Table 1, and it would make the comparison to RoG, ToG, and KG-Agent unfair. Please state explicitly that the SPARQL-query expansion is train-only, or, if it is not, rerun the test evaluation without test-time expansion and report both settings.","section":"Appendix B, Freebase preprocessing"},{"comment":"Section 3.2 assumes the question entities are already linked to Freebase entities through rule-based recognition, and Limitations item 1 acknowledges this, but the paper never reports which linker or rules are used or how often they succeed on WebQSP and CWQ. Because every later subtree is rooted at the linked starting entity, a wrong linker output cannot be repaired by any subsequent subtask. Please name the linker, report its coverage on the two benchmarks, and quantify the fraction of test questions whose starting node is incorrect; without this, the comparison against methods that do not assume such a root is not fully interpretable.","section":"Section 3.2; Limitations item 1"},{"comment":"The WebQSP state-of-the-art claim relies on discounting the original RoG Hits@1 of 0.857 as miscalculated, but the manuscript only points to an external author response. Please provide the corrected calculation in-paper using the Appendix C formula, and confirm that the same evaluation script, answer normalization, and entity linking were used for every row in Table 1. At a minimum, report variance or exact reproducibility of the reproduced RoG run (0.795), since a 4-point gap relative to the original paper is too large to leave undocumented.","section":"Table 1 and Appendix C"}],"minor_comments":[{"comment":"The text calls the F1 score 'Macro-F1,' but the described procedure (averaging per-sample precision/recall and then taking the harmonic mean of the aggregate precision and recall) is not the usual macro-F1; please rename it or clarify the aggregation.","section":"Appendix C"},{"comment":"The text says 'see Figure 12' but the statistic is presented in Table 12; please fix the cross-reference.","section":"Appendix G"},{"comment":"The 'READS' row for GPT-4 is described as not fine-tuned or constrained in the same way as the Llama2-7b READS model; please label it 'READS (zero-shot selection)' to avoid the impression that the same READS procedure is used.","section":"Table 4"},{"comment":"No confidence intervals or repeated-seed statistics are reported for the headline numbers; given the small differences to some baselines, a brief statement of run-to-run variability (or that the pipeline is deterministic) would improve confidence.","section":"Table 1 and Section 5.2"},{"comment":"The caption does not specify what the two curves represent (generative reasoning with ToG-style in-context prompting versus READS discriminative selection) or how the averages are computed over test questions; please expand the caption.","section":"Figure 4"},{"comment":"The column headers 'Total' and 'Seperate' should be corrected, and the interpretation of those columns (per-dataset totals versus per-error-type counts) should be stated explicitly.","section":"Table 11 and Appendix F"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is promising but rests on one evaluation-cleanliness ambiguity (Appendix B) that the authors must resolve. If the SPARQL expansion is train-only and the entity linking is quantified, I would expect a minor-revision-level outcome. The lack of released code and data at review time is a concern but not disqualifying given the promise to release upon acceptance; however, the external-author-response footnote for the RoG comparison should be verified during the revision process."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"READS is a solid empirical paper. The three-way split of KGQA into subgraph search, constraint pruning, and answer-position selection is a real combination—PANGU gives the 'don't generate, discriminate' idea and UniKGQA gives semantic nodes, but nobody has put them together this way. The ablations in Table 3 show each subtask contributes, and the error analysis in Section 5.6 supports the hallucination-reduction story. The cost table (6) is a genuine plus: 7B models get better Hits@1 than the reproduced RoG numbers while using roughly half the input tokens.\n\nThe main soft spot is in Appendix B. The 'Freebase preprocessing' paragraph says the authors extract BFS subgraphs and then 'expand these subgraphs using the SPARQL queries provided in the benchmarks to ensure the presence of constraint branches,' adding them 'to the original dataset.' They never state explicitly that this expansion is limited to training examples. If test-time search also runs on these gold-expanded subgraphs, the graph-searching subtask is selecting edges from a candidate graph already engineered to contain the reasoning structure—and the relation-recall numbers in Table 2 plus the end-to-end comparison to RoG, ToG, and KG-Agent would all be inflated. I don't think this is deliberate; the surrounding text reads like a training-data construction step, but the ambiguity is exactly the kind of thing that sinks a SOTA claim. The reader's report didn't flag this; I think it's the most important thing to resolve.\n\nOther issues are minor: single-run results without variance, code and data promised only on acceptance, and the acknowledged assumptions of pre-linked entities and gold SPARQL queries for training. GrailQA results are below overall SOTA, which the paper admits. The math is clean—the equations are argmax selection rules, no parameter is fitted to test results.\n\nFor a KGQA researcher, especially one comparing generative vs discriminative grounding in 7B-scale models, this is worth reading. The preprocessing question makes it a good reading-group case too. I'd send it to peer review. The referees should ask for a precise statement of when the SPARQL expansion applies, plus variance bars and code/data release. If the expansion is train-only, this is a solid contribution; if not, the main comparison has to be redone. My vote: conditional.","headline":"READS is a genuinely different discriminative decomposition for KGQA, and the ablations hold up; the only thing that keeps me from trusting the SOTA numbers is the ambiguous SPARQL-expansion step in Appendix B.","tokens_in":15536,"tokens_out":3749,"would_cite":false,"duration_ms":32927,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reformulating knowledge graph QA into three discriminative subtasks (search, prune, answer) where an LLM picks from option pools instead of generating free text yields state-of-the-art Hits@1/F1 of 0.840/0.845 on WebQSP and 0.802/0.820 on…","keywords":["knowledge graph question answering","discriminative reasoning","large language models","hallucination mitigation","subgraph retrieval","answer inference","WebQSP","CWQ"],"falsifier":"Take WebQSP and CWQ, and for a random subset of test questions either corrupt or delete the pre-linked starting entity before READS runs; compare Hits@1 and F1 to the reported 0.840/0.845 and 0.802/0.820. If scores do not fall well below those numbers, the entity-linking assumption is not doing the work; if they fall sharply, the headline results depend on the assumed linker rather than on the discriminative subtasks alone.","tokens_in":14506,"feed_emoji":"🧠","tokens_out":6678,"duration_ms":49320,"temperature":0.7,"pith_summary":"This paper tries to establish that the hallucination and ungrounded reasoning of large language models on knowledge graph question answering can be reduced by converting the task into three discriminative subtasks rather than an open-ended generation problem. Given a question, the LLM first chooses which neighboring relation to expand in the question-related subgraph, then selects which constraints in the question apply where, then picks the node position that holds the answer. The claim is that constraining the model to choose from option pools at each step grounds its reasoning in the retrieved graph. If correct, this gives a recipe for making 7-billion-parameter LLMs reliable on knowledge graph QA without relying on their internal knowledge or toolboxes.","feed_headline":"Choosing beats generating in knowledge-graph QA","feed_subtitle":"Splitting QA into search, prune, and answer tasks lifts Hits@1 to 0.840 on WebQSP and 0.802 on CWQ.","key_machinery":"The central object is the abstract subgraph structure that groups entities in the retrieved subgraph into semantic nodes by structural position while keeping relation names, optionally annotated with entity types (entity, topic, num, date). READS performs all three decisions on this skeleton: choose the next (node, relation) pair to expand, choose constraints as (position, operator, target) triples, and choose the answer position. This lets a 7-billion-parameter LLM reason over a small typed structure instead of raw triples, and the discriminative option-pool formulation with logit-constrained beam search is what carries the argument.","core_discovery":"READS decomposes knowledge graph question answering into graph searching, graph pruning, and answer inference, and casts each as a multiple-choice selection over an option pool. The next triple to expand is chosen by constrained beam search over the LLM's output logits; constraints are mapped to positions with one of seven operators (=, <, ≤, >, ≥, min, max); and the answer is located as a semantic node in the abstract structure, with every entity grouped at that node returned as an answer. On WebQSP and CWQ the method reports Hits@1/F1 of 0.840/0.845 and 0.802/0.820, exceeding the reproduced generative baseline and the interactive baselines, with the largest margin on the harder CWQ benchmark. The authors interpret this as evidence that discriminative subtask decomposition alleviates ungrounded subgraph planning and answer hallucination.","pith_inferences":["A testable extension is to run READS with a learned entity linker instead of the assumed rule-based linking; the reported scores probably bound what is achievable when linking is imperfect.","Because the framework does not use the model's internal knowledge and transfers across several backbones, it may generalize to other deterministic graph tasks such as database query selection or document-graph retrieval, though the paper does not test them.","The paper's data-efficiency curves suggest about 25% of the training data is enough to reach near-best performance, so the gold-SPARQL dependency could be relaxed with a smaller set of pseudo-labels from a strong model without losing much accuracy."],"forward_implications":["On WebQSP and CWQ, the method attains state-of-the-art Hits@1 and F1 of 0.840/0.845 and 0.802/0.820, surpassing generative and interactive baselines.","All three subtasks contribute to the performance: replacing any one with a generative counterpart lowers Hits@1, and the search subtask is the largest single source of error.","Entity-type labels on semantic nodes are load-bearing, since removing them drops Hits@1 on both benchmarks.","The decomposition cuts average input tokens per request roughly in half relative to the generative baseline while keeping a similar number of model calls.","On the GrailQA development set, the method improves Hits@1 over the single-task discriminative baseline on i.i.d. and compositional splits, though not on the zero-shot split."],"supporting_citations":[{"why":"Supplies the WebQSP benchmark and its gold SPARQL queries, which READS uses to construct search, pruning, and answer training data.","marker":"(Yih et al., 2016)"},{"why":"Supplies the CWQ benchmark and the harder multi-hop test set where READS shows its largest gains.","marker":"(Talmor and Berant, 2018)"},{"why":"Provides the Freebase knowledge graph from which subgraphs are extracted.","marker":"(Bollacker et al., 2008)"},{"why":"Introduces the semantic-node abstraction and the BFS subgraph extraction that READS builds on.","marker":"(Jiang et al., 2022)"},{"why":"Provides the discriminative-strategy idea and the PANGU baseline whose single-task approach READS extends to three subtasks.","marker":"(Gu et al., 2023)"},{"why":"The generative baseline RoG whose planning method READS replaces and whose reproduced results are the main comparison.","marker":"(LUO et al., 2024)"},{"why":"The interactive reasoning baseline ToG, used in the comparison table and in the in-context reasoning analysis.","marker":"(Sun et al., 2024a)"}],"fun_headline_variants":["READS: Discriminative reasoning lifts KGQA to new state of the art","Splitting KGQA into discriminative subtasks beats generative LLMs","Discriminative KGQA: Higher accuracy without hallucination","READS turns KGQA into multiple-choice to boost Hits@1","From generation to selection: READS tops WebQSP and CWQ"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that question entities have already been correctly linked to Freebase entities by rule-based recognition before READS runs (with gold SPARQL queries available to build the training data), because a wrong starting entity or missing gold query cannot be repaired by any later subtask.","fun_headline_variants_meta":{"raw":{"variants":["READS: Discriminative reasoning lifts KGQA to new state of the art","Splitting KGQA into discriminative subtasks beats generative LLMs","Discriminative KGQA: Higher accuracy without hallucination","READS turns KGQA into multiple-choice to boost Hits@1","From generation to selection: READS tops WebQSP and CWQ"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1749,"prompt_tokens":866,"completion_tokens":883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":789}},"tokens_in":482,"tokens_out":883,"duration_ms":6876,"temperature":1.0,"reasoning_tokens":789,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:51:34.538119+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take WebQSP and CWQ, and for a random subset of test questions either corrupt or delete the pre-linked starting entity before READS runs; compare Hits@1 and F1 to the reported 0.840/0.845 and 0.802/0.820. If scores do not fall well below those numbers, the entity-linking assumption is not doing the work; if they fall sharply, the headline results depend on the assumed linker rather than on the discriminative subtasks alone.","supporting_citations":[],"review_version":1}