{"id":"e3afd049-941e-4362-814a-18afc4d55c43","arxiv_id":"2412.19544","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TARGA creates on-the-fly synthetic query demonstrations from the knowledge graph around each test question, and this is enough to beat non-fine-tuned KBQA baselines without any manual annotation.","lead":"TARGA generates a handful of practice question-answer examples on the spot for each new question by exploring the knowledge graph around the entities and relations in the question, then uses them as examples for a 7B language model. This removes the need for manually labeled training data and improves accuracy on several complex question-answering benchmarks, especially on questions that differ from training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'zero annotation' claim is potentially compromised because TARGA's entity linking comes from Pangu, a fine-tuned model trained on the benchmark; the paper neither discloses this dependency nor ablates with an annotation-free linker.","rationale":"The reader's weakest assumption was coverage: Table 9 shows that the correct query is absent from the synthetic pool for 21--31% of Freebase questions, and the paper does not analyze performance on those cases. That is a real concern, but it is not the most load-bearing condition for the central claim. Even if coverage were 100%, the claim 'without any data annotation' would still fail if TARGA relies on an entity linker fine-tuned on the same benchmark's annotated training data. Conversely, even with incomplete coverage, TARGA could still be a valid annotation-free method that outperforms baselines on average. The Pangu provenance issue is more fundamental because it concerns the truth of the 'no annotation' claim itself, and it is checkable. The paper is transparent about using Pangu linking but does not connect that fact to the annotation-free framing; it also does not ablate the linker. I keep the CONDITIONAL verdict because the empirical results may survive a corrected framing, but the paper should either demonstrate that Pangu's linker is annotation-free or replace it with an annotation-free linker and re-run the headline comparisons. If the proposed test shows a large performance drop, the verdict should move to REJECT; if not, ACCEPT becomes defensible.","tokens_in":20772,"tokens_out":9734,"duration_ms":88125,"concrete_test":"Determine the training provenance of the Pangu entity linker used in Section 3.2, and re-run TARGA with FACC1+BM25 entity linking on GrailQA and GraphQ while keeping all other components fixed. Concretely: (1) Inspect Gu et al. (2023) and the released artifacts to establish whether the linker is trained on the GrailQA/GraphQ training splits or is unsupervised. (2) Replace the Pangu entity linking with FACC1+BM25, the same linking setup used by KB-Binder and KB-Coder, and measure F1 on GrailQA and GraphQ. If the F1 drops below the non-fine-tuned baselines (e.g., KB-Coder-R at 61.3 on GrailQA), the zero-annotation claim fails; if it remains above those baselines, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing condition for the headline claim is that TARGA operates 'without any data annotation.' Section 3.2 obtains candidate entities from 'the linking result provided by Gu et al. (2023)'—i.e., Pangu. Table 1 lists Pangu as a Seq2Seq fine-tuning method trained on the full training set (GrailQA 83.4 F1), and Appendix E.2 confirms that TARGA uses Pangu's entity linking on GrailQA and GraphQ. If that linking output comes from a model fine-tuned on the same benchmark's annotated training split, then the 'zero annotation (ICL)' row in Table 1 is mislabeled: TARGA is not annotation-free, and its comparison with KB-Binder and KB-Coder, which use FACC1+BM25 and not a benchmark-trained linker, is not a no-annotation comparison. The paper does not disclose this dependency, does not report an ablation with an annotation-free linker, and does not state whether the Pangu linker was trained on the test datasets' training splits. This bears directly on the abstract's 'without any data annotation' claim and on the claimed advantage over non-fine-tuned methods. If the Pangu linker is benchmark-trained, part of the reported gain could come from a supervised component, and the practical claim 'no annotation needed' is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TARGA is a framework for KBQA and Text2SQL that, for each test question, dynamically constructs a small set of candidate SPARQL/PyQL queries by expanding from retrieved entities and relations, re-ranks them, and feeds the top ones as (query, pseudo-question) demonstrations to a 7B LLM in a few-shot manner. The paper reports state-of-the-art results among non-fine-tuned methods on GrailQA, GraphQ, and KBQA-Agent, competitive results on MetaQA and WikiSQL, and provides ablations, robustness, efficiency, and model-size analyses. The central claim is that this online synthetic-data generation achieves these results without any manual annotation.","tokens_in":21056,"tokens_out":7273,"duration_ms":62383,"significance":"If the results hold, TARGA offers a practical recipe for reasoning over structured data with a small open-source LLM and no task-specific fine-tuning of the reasoning model. The paper's strengths include the clear pipeline description, the extensive ablations in Table 5, the robustness experiments in Figures 4-5, the efficiency analysis in Table 3, the model-size study in Table 4, and a public code repository. The main empirical contribution—strong KBQA results from a 7B model without fine-tuning—is interesting and potentially useful. However, the 'zero annotation' claim is currently not clean because the entity linker is a fine-tuned model trained on the same benchmark, and the coverage analysis reveals that a substantial fraction of test questions have no correct synthetic query, a fact the paper does not analyze further. The paper is transparent about its components, which is a credit, but the claims need to be realigned with the actual dependencies.","major_comments":[{"comment":"The abstract and introduction state that TARGA works 'without any data annotation,' but for GrailQA and GraphQ the candidate entities come from the entity linking output of Pangu (Gu et al., 2023), which Table 1 lists as a sequence-to-sequence model fine-tuned on the full training set of the same benchmark. Appendix E.2 confirms that TARGA uses this Pangu linking result on GrailQA and GraphQ. The method therefore relies on a supervised component trained with human-annotated data, so the 'zero annotation' framing is misleading and the comparison with KB-Binder and KB-Coder, which use FACC1+BM25/SimCSE linking, is not an annotation-free comparison. In addition, on KBQA-Agent and MetaQA-3Hop, TARGA uses golden entity linking (Table 8). The paper should either add an ablation with an annotation-free linker (e.g., FACC1 or a pure string-match linker) and report the performance difference, or explicitly qualify the 'zero annotation' claim to refer only to the reasoning component.","section":"Section 3.2, Appendix E.2, Table 1"},{"comment":"Table 9 reports coverage, the proportion of questions with at least one F1=1 synthetic query, as only 0.79 on GrailQA, 0.69 on GraphQ, 0.78 on KBQA-Agent, and 0.75 on WikiSQL. For the remaining 21-31% of test questions, the synthetic pool contains no correct query, yet the paper does not report performance separately for covered versus uncovered questions. This is essential for interpreting the headline results: if the F1 gains are concentrated on covered questions, the method is largely copying or lightly editing a correct query from the demonstrations; if the model also solves many uncovered questions, that would support the generalization claim. Please report the covered/uncovered F1 breakdown on the main KBQA datasets and discuss the model's behavior on uncovered questions.","section":"Table 9"},{"comment":"The query construction stops at three hops and five edges based on informal statements that coherent questions 'typically' fall within these bounds. The paper provides no empirical distribution of the number of hops or edges required by the test sets, nor an analysis of how many correct queries are missed because of these limits beyond the aggregate coverage in Table 9. Since the coverage values already show that a substantial fraction of questions have no correct query in the pool, the hop and edge limits are a likely contributor. Please provide the distribution of required query complexity in the datasets and show that the 3-hop/5-edge thresholds do not systematically exclude a meaningful class of correct queries.","section":"Section 3.3, Eqs. (1)-(3)"},{"comment":"All reported F1 and accuracy numbers are single-run values with no confidence intervals, standard deviations, or significance tests. Given that the pipeline involves stochastic elements (embedding, reranking, and LLM decoding), it would be useful to report multiple runs or seed variance for at least the main comparisons, so that the reader can assess whether the reported gaps (e.g., +7.7 on GrailQA and +12.2 on KBQA-Agent) are robust to this variability.","section":"Section 4.2, Tables 1-6"}],"minor_comments":[{"comment":"'close-sourced model' should be 'closed-source model' in both occurrences.","section":"Abstract and Section 4.2"},{"comment":"The phrase 'top tanked candidates' should be 'top-ranked candidates'.","section":"Section 3.4, Eq. (4)"},{"comment":"The category label 'zero annotation (ICL)' is misleading because TARGA uses the fine-tuned Pangu entity linker for GrailQA and GraphQ and golden linking for KBQA-Agent and MetaQA-3Hop; please rename or footnote this category to reflect the actual entity-linking dependencies.","section":"Table 1"},{"comment":"The hierarchical ranking strategy retains 'top n candidates' per parent query, but the value of n is not reported in the main text or appendices; please state the hyperparameter value used in the experiments.","section":"Section 3.4"},{"comment":"The phrase 'make a fair comparison' is used to justify following the Pangu entity linking setup; please clarify whether this is a claim about no-annotation conditions or simply a statement about matching the baselines' preprocessing.","section":"Appendix E.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core and the writing is generally clear, but the 'zero annotation' claim needs to be re-framed or supported by an annotation-free linking ablation. The missing covered/uncovered analysis is the second main gap. The single-run results are a lesser concern but should be acknowledged. I believe a major revision with these additions and clarifications would make the paper suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this paper is worth reading and the method is genuinely new. TARGA builds a pool of candidate queries online for each test question by expanding entity/relation seeds into valid query graphs, executing them, reranking, and using the top ones as ICL demonstrations. That is a real departure from BYOKG's offline synthetic corpus and from KB-Binder/KB-Coder's retrieval from annotated sets. The empirical results are strong: 69.0 F1 on GrailQA with a 7B open model, big gains on KBQA-Agent, good transfer to WikiSQL, and ablations that make sense. The runtime numbers are also convincing.\n\nBut the central headline claim—'without any data annotation'—does not hold as stated. Section 3.2 takes entity linking from Pangu, and Appendix E.2 confirms it on GrailQA and GraphQ. Pangu is a model fine-tuned on those benchmarks' training sets. So TARGA is not annotation-free in the sense the abstract claims; it inherits a supervised component. The paper neither discloses this in the main text nor ablates with an annotation-free linker. That is a load-bearing issue. If the comparison baselines use FACC1+BM25, then TARGA has an unfair advantage.\n\nSecond, coverage is concerning. Table 9 reports that the correct query is absent from the synthetic pool for 21–31% of Freebase questions. The paper doesn't say what happens on those cases, or how the ICL model copes. That gap matters because the whole pitch is high-relevance demonstrations.\n\nThird, the formal query construction in Section 3.3 only defines triplet patterns, yet the logic form includes filters, argmax/argmin, and count. How those are synthesized is not specified. The paper needs to close that gap.\n\nWeaker points: results are single runs with no variance; some baseline numbers are copied from prior work; code/data are not obviously released despite a GitHub link.\n\nNone of these kill the core idea. The method is novel, the ablations are honest, and the main effect is likely real—but the 'zero annotation' framing needs a major revision, and the coverage and formal-construction gaps need explicit treatment.","headline":"Strong empirical method for targeted synthetic demonstrations in KBQA, but the 'zero annotation' claim is overstated because the entity linking comes from a benchmark-trained model.","tokens_in":21587,"tokens_out":2914,"would_cite":true,"duration_ms":26192,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TARGA claims that a 7B model with no annotated data can beat closed-source KBQA systems by generating targeted synthetic demonstrations for each test question.","keywords":["semantic parsing","knowledge base question answering","in-context learning","synthetic data generation","zero-shot generalization","query graph expansion","text-to-SQL","targeted demonstrations"],"falsifier":"Compute TARGA's F1 separately on the questions whose synthetic pool contains no perfect (F1=1) query, using the paper's coverage split; if that F1 is close to zero, the headline results are fully explained by candidate coverage and the LLM contributes no structure beyond the pool, whereas if it is substantial, the ICL model is generalizing beyond the generated candidates and the coverage ceiling is not a binding constraint.","tokens_in":20531,"feed_emoji":"🤖","tokens_out":9824,"duration_ms":82897,"temperature":0.7,"pith_summary":"This paper argues that the bottleneck in knowledge-base question answering is not model size or annotated data but the relevance of demonstrations: instead of retrieving examples from a static labeled corpus, TARGA builds a small set of synthetic query-question pairs tailored to the test question, in real time, and feeds them to an in-context learning model. With only a 7B-parameter open model and no manual annotation, it reports F1 scores of 69.0 on GrailQA, up from 61.3 for the best previous non-fine-tuned method, and 46.5 on KBQA-Agent, up from 34.3. If correct, this makes semantic parsing over structured data substantially cheaper and more adaptable: no annotation budget, no offline retrieval index, and less dependence on expensive closed-source APIs. It also reframes generalization as a data-generation problem rather than a data-collection problem.","feed_headline":"Zero-annotation synthetic data lets a 7B model top KBQA baselines","feed_subtitle":"It builds targeted examples per question, beating retrieval from labeled corpora at one-tenth the tokens.","key_machinery":"The load-bearing mechanism is the two-stage synthetic query constructor: layer-wise expansion grows chain-like multi-hop queries one relation at a time, and cross-layer combination merges two verified queries through a shared variable to produce multi-constraint graphs, with every query executed against the knowledge base to require a non-empty result. Around this sits a re-ranking stage using bge-reranker-v2-m3 on query textification outputs plus a hierarchical top-n per parent query, which keeps the candidate pool small while keeping relevant queries retrievable. The final in-context learning prompt pairs the top-ranked textified queries with their generated natural-language questions, so the LLM sees examples that match the target question's own entities and relations.","core_discovery":"TARGA's central claim is that the most effective demonstrations for in-context semantic parsing are not the most similar labeled examples from a training set but synthetic examples generated from the knowledge base itself, targeted to the entities and relations of the incoming question. Concretely, it links the question's entities via Pangu's linker and takes the top-20 relations by text-embedding similarity, then explores valid query graphs by layer-wise expansion (multi-hop chains grown one edge at a time) and cross-layer combination (multi-constraint graphs merged through a shared variable). Each candidate is executed against the knowledge base and kept only if it returns a non-empty result. Queries are re-ranked with a cross-encoder reranker after 'textification' into natural-language-like strings, and the top 10 are presented as (question, query) demonstrations to a 7B instruct model. The paper reports that this procedure, with no annotations and no static corpus, outperforms all non-fine-tuned baselines on GrailQA, GraphQ, KBQA-Agent, and WikiSQL, while using about a tenth of the tokens of ICL baselines.","pith_inferences":["The paper's own coverage statistics imply an upper bound it does not discuss: on GrailQA, GraphQ, and KBQA-Agent, only 79%, 69%, and 78% of questions have at least one F1=1 synthetic query, so the remaining 21-31% must depend on the LLM generalizing beyond the candidate pool; improving coverage, not just ranking, is the natural next lever.","Because TARGA's relation candidates come from text-embedding similarity, the framework inherits the embedding model's lexical and multilingual blind spots; in domains with heavy paraphrase or low-resource languages, coverage may be lower than the paper's English Freebase results, which the paper did not test.","If the ranking step is doing most of the work, TARGA could be combined with any generator of candidate logical forms, not just graph expansion: for example, an LLM's own drafts could be executed, filtered, and re-ranked with the same textification-plus-reranker loop, a testable extension the paper does not explore.","The 3-hop and 5-edge limits, chosen because most benchmark questions fit within them, mean TARGA is calibrated to current dataset complexity; questions requiring deeper chains or more constraints would need those thresholds raised, with a likely quadratic cost in candidate queries."],"forward_implications":["Zero-annotation semantic parsing becomes practical: TARGA needs no human-labeled training set, no retrieval index over annotations, and no closed-source API, yet improves the best non-fine-tuned F1 on GrailQA from 61.3 to 69.0 and on KBQA-Agent from 34.3 to 46.5.","Small open models can compete with large closed models on structured reasoning: at 7B, TARGA matches or exceeds GPT-3.5-turbo on GrailQA and beats all non-fine-tuned methods on GraphQ and KBQA-Agent, while a 1.5B variant already surpasses the previous best non-fine-tuned method.","Generalization to non-I.I.D. settings stops being a retrieval failure: on GrailQA's zero-shot split, TARGA scores 71.7 versus 41.8-43.2 for KB-Binder and KB-Coder, because every test question receives fresh targeted demonstrations instead of similar-but-mismatched corpus examples.","The same recipe transfers to other logic-form tasks: on WikiSQL, 10 synthetic demonstrations outperform methods using 32 annotated examples and approach the accuracy of a fine-tuned 3B model.","Robustness to bad demonstrations improves markedly: corrupting every synthetic demonstration drops TARGA's F1 by only about 25%, while similarity-retrieval and random ICL baselines fall by roughly 40% and 75%, suggesting the model leans on the per-question target rather than exact demonstration content."],"supporting_citations":[{"why":"Supplies the Pangu entity-linking results TARGA adopts for GrailQA and GraphQ and serves as the grounding baseline TARGA contrasts with its generation approach.","marker":"(Gu et al., 2023)"},{"why":"KB-Binder, the few-shot ICL baseline that retrieves demonstrations from an annotated training set, is the main retrieval paradigm TARGA replaces.","marker":"(Li et al., 2023b)"},{"why":"KB-Coder is the code-style ICL baseline whose 61.3 F1 on GrailQA TARGA improves to 69.0.","marker":"(Nie et al., 2023)"},{"why":"Defines KBQA-Agent, the most difficult benchmark, and FUXI, the prior 34.3-F1 SOTA that TARGA raises to 46.5.","marker":"(Gu et al., 2024)"},{"why":"QueryAgent is the agent-based baseline whose re-implemented results and efficiency metrics frame TARGA's speed comparisons.","marker":"(Huang et al., 2024)"},{"why":"Supplies bge-reranker-v2-m3, the cross-encoder reranker used in the query re-ranking stage.","marker":"(Chen et al., 2024)"},{"why":"Constructs GrailQA and its I.I.D., compositional, and zero-shot generalization splits, which ground TARGA's non-I.I.D. claims.","marker":"(Gu et al., 2021)"},{"why":"BYOKG is the zero-annotation self-supervised baseline that TARGA roughly doubles in F1 on GrailQA and MetaQA-3Hop, showing the gain of targeted online synthesis over offline synthetic corpora.","marker":"(Agarwal et al., 2024)"}],"fun_headline_variants":["Zero-annotation synthetic data boosts KBQA with 7B model","Targeted synthetic data from KB beats retrieval for 7B model","Zero-annotation synthetic demonstrations outperform on KBQA","On-the-fly synthetic data: 7B model tops closed-source baselines","Dynamic synthetic examples beat static retrieval for KBQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"TARGA's success rests on the assumption that its candidate retrieval and expansion steps place the correct query in the synthetic pool: top-20 embedding-similar relations, at most three hops and five edges, and valid execution against the KB; the paper's own Table 9 shows this pool is complete for only 0.79 of GrailQA, 0.69 of GraphQ, and 0.78 of KBQA-Agent questions, so for the remaining 21-31% the LLM must succeed by guessing.","fun_headline_variants_meta":{"raw":{"variants":["Zero-annotation synthetic data boosts KBQA with 7B model","Targeted synthetic data from KB beats retrieval for 7B model","Zero-annotation synthetic demonstrations outperform on KBQA","On-the-fly synthetic data: 7B model tops closed-source baselines","Dynamic synthetic examples beat static retrieval for KBQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000692,"raw_usage":{"total_tokens":3145,"prompt_tokens":973,"completion_tokens":2172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2096}},"tokens_in":589,"tokens_out":2172,"duration_ms":14636,"temperature":1.0,"reasoning_tokens":2096,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:13:06.321619+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute TARGA's F1 separately on the questions whose synthetic pool contains no perfect (F1=1) query, using the paper's coverage split; if that F1 is close to zero, the headline results are fully explained by candidate coverage and the LLM contributes no structure beyond the pool, whereas if it is substantial, the ICL model is generalizing beyond the generated candidates and the coverage ceiling is not a binding constraint.","supporting_citations":[{"cited_title":"Code-Style In-Context Learning for Knowledge-Based Question Answering","cited_arxiv_id":"2309.04695","evidence_quote":"KB-Coder is the code-style ICL baseline whose 61.3 F1 on GrailQA TARGA improves to 69.0."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BYOKG is the zero-annotation self-supervised baseline that TARGA roughly doubles in F1 on GrailQA and MetaQA-3Hop, showing the gain of targeted online synthesis over offline synthetic corpora."}],"review_version":1}