{"id":"bd39367f-c950-4324-ab48-1c179c34928d","arxiv_id":"2411.08165","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"KGR3 combines retrieval of entity contexts, LLM reasoning, and supervised re-ranking to improve knowledge graph completion, reporting state-of-the-art Hits@1 on FB15k237 and WN18RR.","lead":"KGR3 is a three-stage framework that retrieves supporting triples and entity descriptions, asks an LLM to propose answers, then fine-tunes the LLM to re-rank candidate answers for knowledge graph completion. It reports large Hits@1 gains on FB15k237 and WN18RR across six base KGC models, though no code, data, or error bars are released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Base-model candidate ceiling is unquantified: Eqs. (2)/(4)/(5) imply Hits@1 can only improve when ground truth is already in Ac, but oracle recall is never reported.","rationale":"The reader's weakest assumption identifies the same structural constraint: the base model's top-50 candidate filter in Eq. (2) bounds what the LLM can contribute, and the Ac construction in Eq. (4) limits the re-ranker to at most 20 re-ranked entities. My stress test sharpens this into an explicit upper bound: because Eq. (5) preserves AKGC[n:] after Ac, no entity outside Ac can ever be promoted to position 1. The absence of an oracle-recall analysis is therefore load-bearing for the headline claim. A single oracle-selector run would determine whether the reported Hits@1 is close to the candidate-availability ceiling or far below it, and would settle whether the gains should be credited to the framework or to the base models' candidate lists. Since the paper is otherwise a plausible engineering contribution with consistent trends across base models and LLM backbones, but lacks code, variance estimates, and this key upper-bound analysis, the original CONDITIONAL verdict remains appropriate.","tokens_in":17126,"tokens_out":11581,"duration_ms":118404,"concrete_test":"Run an oracle-selector experiment on FB15k237 with NBF-Net/Llama3-8B and on WN18RR with SimKGC/Llama3-8B: construct Ac exactly as in Eq. (4) with p=10, n=20, delta=50, and instead of the fine-tuned re-ranker, always select the ground-truth entity when it appears in Ac. Report oracle Hits@1, Hits@10, and MRR, together with base-model-only Recall@20 and Recall@50. If oracle Hits@1 is close to the reported 0.475/0.656, the re-ranker is operating near the candidate ceiling and the gains are explained by candidate availability plus selection; if oracle Hits@1 is much higher, the re-ranker is the bottleneck and the framework's stated contribution needs re-scoping. Also report the fraction of test triples whose ground truth appears in AKGC[0:20] to expose the cost of the delta=50 filter in Eq. (2).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that KGR3's reasoning and re-ranking modules recover answers the base KGC models miss is not separable from the recall of the base models' candidate lists. Equation (2) filters every LLM answer through AKGC[0:delta] with delta=50, and Eq. (4) builds the re-ranking set Ac = AKGC[0:p] union {ALLM\\AKGC[0:p]}[0:n-p] with p=10, n=20. Equation (5) then places only the selected entity at the front and leaves every entity outside Ac in AKGC[n:|E|]. Therefore, any test triple whose ground-truth entity is absent from Ac has zero probability of being ranked first: Hits@1(KGR3) is bounded above by the fraction of test triples whose ground truth is present in Ac. The paper never reports this oracle recall. The Limitations section concedes only that the base KGC model 'may inevitably constrain' final performance, without quantifying how often the correct entity is even available to the re-ranker. Without that number, the reported absolute Hits@1 gains of 12.3% and 5.6% cannot be attributed to the framework rather than to favorable candidate lists inherited from the base models. This is not a circularity; it is a missing bound on the central empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KGR3, a three-module pipeline for transductive knowledge graph completion: a Retrieval stage that gathers supporting triples, entity contexts, and a candidate ranking from a base KGC model; a Reasoning stage that prompts an LLM to propose answers, filtered to the base model's top-δ list; and a Re-ranking stage that fine-tunes an LLM to select the best answer from a merged candidate set. Experiments on FB15k237 and WN18RR with six base KGC models and several LLMs report consistent gains, with the best variant reaching MRR 0.535/0.543 and Hits@1 0.475/0.487 on FB15k237, and the paper claims absolute Hits@1 improvements of 12.3% and 5.6% on the two datasets. The central design is that the LLM re-ranker reorders a candidate set built from base-model top-p predictions and LLM answers, rather than scoring the full entity vocabulary.","tokens_in":17366,"tokens_out":7027,"duration_ms":69610,"significance":"If the reported gains reproduce, KGR3 is a practical and modular contribution: it wraps any base KGC model without retraining, and the ablations isolate the contributions of reasoning, entity descriptions, neighbor facts, and re-ranking strategy. The paper also provides detailed prompts, hyperparameters, and time costs, which are useful for replication. However, the empirical claim is currently not fully bounded: the final candidate set Ac is constructed entirely from the base model's top-δ list, so the headline gains cannot be separated from the base models' oracle recall. This missing quantity, together with the absence of variance information, prevents a conclusive assessment of the framework's own contribution.","major_comments":[{"comment":"The final Hits@1 ceiling is the oracle recall of Ac, and this is never reported. Eq. (2) filters every LLM answer through AKGC[0:δ] with δ=50, and Eq. (4) builds Ac from AKGC[0:p] plus at most n−p LLM answers that are already in that top-50 slice. Eq. (5) then places all entities outside Ac in AKGC[n:|E|], so any test triple whose ground truth is not in Ac cannot be ranked first. The Limitations section concedes that the base KGC model 'may inevitably constrain' performance, but it does not quantify how often the ground truth is even present in Ac. Please report, for each base model and dataset: the oracle Hits@n of Ac (the fraction of test triples whose ground truth is in the final candidate set), the Hits@δ of AKGC, and KGR3's Hits@1 conditioned on ground-truth-in-Ac. Without these numbers, the claimed 12.3% and 5.6% improvements cannot be attributed to the reasoning and re-ranking modules rather than to favorable candidate lists inherited from the base models.","section":"§4.2–4.3, Eqs. (2), (4), (5)"},{"comment":"There is a train/inference distribution mismatch in the re-ranking module that is not analyzed. The SFT loss in Eq. (3) is trained on randomly sampled negatives (one ground truth plus n−1 random negatives, about half relation-hard), whereas inference re-ranks Ac, which is built from the base model's top-p candidates and LLM answers. These two distributions can be substantially different: base-model candidate lists are typically harder and more relation-confusable than random negatives. The paper does not measure the overlap between training negatives and inference candidates, nor does it ablate an SFT variant that trains on base-model top-n candidates. Without this analysis, the re-ranking module's contribution in Table 3 and Table 5 is not isolated as cleanly as the text claims; the authors should either provide this ablation or at least quantify the train/inference candidate overlap.","section":"§4.3, Eq. (3)"},{"comment":"The main results are reported as a single run per configuration, with no standard deviations, confidence intervals, or significance tests. LLM generation is stochastic (temperature and decoding strategy are not specified for inference), so a single run does not establish that the large gains are stable across seeds. Please state the decoding settings, run at least a few seeds for the best variants, and report the spread; even a small number of repeats would materially strengthen the 'consistently' claim made throughout the paper.","section":"§5.3, Table 2"}],"minor_comments":[{"comment":"The headline numbers are inconsistent across the paper: the abstract claims absolute Hits@1 improvements of 12.3% and 5.6%, while §5.3 reports 11.1% and 4.0% against 'previous state-of-the-art' with Llama3-8B, with a footnote saying Qwen2-7B is better. The 12.3% and 5.6% appear to correspond to comparisons against DIFT+CoLE and DIFT+SimKGC, but this is not stated. Please specify the exact baselines and LLM configuration for each headline number.","section":"Abstract and §5.3"},{"comment":"The caption labels the three modules as '1) Reasoning (left), 2) Reasoning (middle), 3) Re-ranking (right)', but the left module is Retrieval; the caption should be corrected to Retrieval, Reasoning, and Re-ranking.","section":"Figure 2 caption"},{"comment":"Eq. (2) uses set intersection notation between a list (ALLM) and a slice (AKGC[0:δ]), which does not specify whether the order of LLM answers is preserved or how duplicate aliases and entity labels are resolved. Please define the operation more formally, including the handling of the δ threshold.","section":"§4.2.2, Eq. (2)"},{"comment":"Section 4.3 says 'half of them' are hard negatives, while Table 8 specifies 9 easy and 10 hard negatives; the prose should be aligned with the exact split in Table 8.","section":"§4.3 and Table 8"},{"comment":"No code, processed contexts, or candidate lists are released; given that results depend on the exact retrieved contexts and the base-model candidate filtering, releasing these artifacts would substantially aid reproducibility and verification of the candidate-ceiling analysis.","section":"Reproducibility"},{"comment":"There are several typographical and formatting issues, including 'Y AGO' in §3, 'LLama' capitalization in Table 2, 'pompts' in Appendix A.2, and inconsistent use of 'adjoints' in the appendix prompts; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper reports large, consistent Hits@1 gains by adding a retrieval-reasoning-re-ranking pipeline over existing KGC models: up to 12.3% absolute improvement on FB15k237 over NBF-Net, and the pattern holds across six base models and four LLMs. That is a genuinely useful engineering contribution, not a trick that works on one model.\n\nWhat's new is the specific integration. Each piece has been seen before in KICGPT, DIFT, and SimKGC, but the three-stage design with entity context retrieval and the plug-and-play property is not in the cited prior work. The ablations are thoughtful: they isolate the reasoning module, entity descriptions, neighbor facts, LLM choice, ranking strategy, and SFT data volume. The decision to move only the selected candidate to the front, rather than re-sorting the entire list, is a sensible way to limit damage from false positives. This is a solid empirical study.\n\nThe soft spots are real but not fatal. No code or processed contexts are released, and there are no error bars or significance tests. The more specific concern is the candidate ceiling. Equation (2) filters every LLM answer through the base model's top-50, and Eq. (4) builds the re-ranking set from that same window. So any test triple whose true entity is outside the base model's top-50 has zero chance of being answered correctly. The paper acknowledges this in the Limitations section but never quantifies it. We don't know how often the base model's top-50 actually contains the ground truth. This is a missing bound, not a circularity. It doesn't invalidate the empirical gains, but it leaves open how much of the improvement is reordering within a good candidate list versus recovering answers the base model never had. An oracle-recall plot would settle this quickly.\n\nThe central claim holds up as an empirical statement. I'd send this to peer review. A serious referee should ask for the oracle analysis and for code and data, but the work is worth refereeing. For a reading group, it's a reasonable \"how to squeeze LLMs into KGC\" paper. I would not cite it in my own work in the next 12 months, but anyone building KGC pipelines should have it on their radar.","headline":"A useful plug-and-play KGC pipeline with real, consistent gains; the missing oracle-recall analysis leaves the central claim under-supported but not wrong.","tokens_in":17961,"tokens_out":3031,"would_cite":false,"duration_ms":28634,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three-stage LLM pipeline lifts KG-completion Hits@1 by up to 12.3%","keywords":["knowledge graph completion","retrieval-reasoning-re-ranking","large language models","entity context","in-context learning","supervised fine-tuning","link prediction","Wikidata"],"falsifier":"On a random sample of FB15k237 test triples, measure whether the ground-truth tail appears in the base model's top-50 ranking. If most KGR3 failures have ground truths outside that list, the bottleneck is candidate recall of the base model; if failures occur inside the list, the LLM or the prompt is the limiting component.","tokens_in":16892,"feed_emoji":"🧠","tokens_out":11159,"duration_ms":97889,"temperature":0.7,"pith_summary":"The paper sets out to show that knowledge graph completion improves when structural triples are enriched with entity contexts before a large language model is asked to reason and re-rank. It proposes KGR3, a three-stage pipeline that retrieves supporting triples and textual contexts, asks an LLM to propose candidate answers, and fine-tunes the same LLM to select the best answer from a merged candidate list. The claimed payoff is large and consistent: absolute Hits@1 improvements of 12.3% on FB15k237 and 5.6% on WN18RR, with gains across all six base KGC models tested. The framework matters because it is plug-and-play, improving existing completion models without retraining them and concentrating its gains at the top of the ranking, where completions are actually used.","feed_headline":"Three-stage LLM pipeline lifts KG-completion Hits@1 by up to 12.3%","feed_subtitle":"Entity contexts and a fine-tuned LLM re-ranker lift exact-answer Hits@1 on FB15k237 and WN18RR without retraining the base model.","key_machinery":"The load-bearing machinery is the composition of three ranking operations. A base KGC model produces an initial ranking $A_{KGC}$ over all entities. The LLM's free-form answers are intersected with the base model's top-$\\delta$ positions ($\\delta=50$) to form $A_{LLM}$, keeping generative answers inside the KG. At inference the candidate set is $$A_c = A_{KGC}[0:p] \\cup \\{A_{LLM}\\setminus A_{KGC}[0:p]\\}[0:n-p]$$ with $p=10$ and $n=20$, so every final candidate is either a base-model top-10 entity or an LLM-suggested entity the base model ranked no lower than 50. A supervised-fine-tuned LLM is trained with a cross-entropy loss to select the true tail from the $n$ candidates; at test time its chosen answer is moved to the front and the rest of $A_c$ follows in base order, yielding the final list $A_{RR}$. Entity labels, descriptions, and aliases from Wikidata are the substance that lets the LLM connect entity IDs to ordinary language.","core_discovery":"The central claim is that the semantic gap between structured KG triples and natural language can be bridged by retrieved entity contexts, and that once bridged, an LLM can both propose and select answers better than either a pure embedding model or a pure text model. Concretely, KGR3 takes a query triple, retrieves supporting triples with the same relation or similar entities, maps entity IDs to Wikidata labels, descriptions, and aliases, and asks an LLM to generate possible answers only from the base model's top-50 ranked entities. A LoRA-fine-tuned LLM is then trained to choose the ground-truth entity among twenty candidates assembled from the base model's top-10 and the LLM's suggestions. The paper reports that the best variant raises absolute Hits@1 by 12.3 percentage points on FB15k237 and 5.6 percentage points on WN18RR, and that every tested base KGC model is improved, with larger gains in Hits@1 than in Hits@3 or Hits@10.","pith_inferences":["A direct test of the framework's ceiling would be to report oracle recall of the base model's top-50 list; if that recall is near 100%, KGR3 is adding precision only, whereas if it is low, improving the base model's candidate recall is the next bottleneck.","The same retrieve-candidates-then-let-an-LLM-select pattern should transfer to other KG tasks such as relation prediction or entity typing, where the final decision is a choice among a small candidate set.","The design decision to train the LLM to pick one best answer rather than sort a full list suggests that discriminative selection is a better use of LLMs for structured ranking than full list ordering."],"forward_implications":["Any existing KGC model can be wrapped by KGR3 without retraining; the base model only needs to produce a ranked candidate list.","Because the final candidates come from the base model's top-10 plus LLM answers already in its top-50, the framework's ceiling is the base model's top-50 recall, a constraint the paper acknowledges in its Limitations section.","The largest gains are in Hits@1, so the practical effect is to convert near-miss rankings into exact top-ranked answers rather than to discover entirely new entities.","Neighbor facts matter more than entity descriptions in the ablations, indicating that structured triples remain indispensable even when an LLM supplies broad world knowledge.","The framework works with LLMs as small as 1.5B parameters and reports lower fine-tuning time, suggesting the re-ranking benefit is available without the largest models."],"supporting_citations":[{"why":"Supplies NBF-Net, the graph neural network base model whose candidate list anchors the strongest reported KGR3 variant on FB15k237.","marker":"(Zhu et al., 2021)"},{"why":"Provides Wikidata, the knowledge base from which entity labels, descriptions, and aliases are retrieved to bridge triples and text.","marker":"(Vrandečić and Krötzsch, 2014)"},{"why":"Introduces DIFT, the LLM fine-tuning baseline KGR3 compares against and whose discrimination-instruction idea shapes the re-ranking training objective.","marker":"(Liu et al., 2024)"},{"why":"Introduces KICGPT, the in-context learning baseline from which the top-delta filtering of LLM answers is adapted.","marker":"(Wei et al., 2023)"},{"why":"Supplies SimKGC, a text-based base model whose candidate list and WN18RR entity contexts are used and improved by KGR3.","marker":"(Wang et al., 2022)"},{"why":"Provides the entity labels and descriptions for WN18RR that KGR3 adopts as textual context.","marker":"(Yao et al., 2019)"}],"fun_headline_variants":["LLM context retrieval boosts KG Hits@1 by 12.3%","KGR3: context-aware LLM re-ranker lifts KG completion","Retrieval, reasoning, re-rank: KG completion gets LLM boost","Entity contexts help LLM fine-tune KG answer selection","Twelve-point gain in Hits@1 from LLM-reasoned KG completion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework can only return an entity the base KGC model already placed in its top-50 candidate list; if the true answer is outside that list, neither LLM reasoning nor re-ranking can recover it.","fun_headline_variants_meta":{"raw":{"variants":["LLM context retrieval boosts KG Hits@1 by 12.3%","KGR3: context-aware LLM re-ranker lifts KG completion","Retrieval, reasoning, re-rank: KG completion gets LLM boost","Entity contexts help LLM fine-tune KG answer selection","Twelve-point gain in Hits@1 from LLM-reasoned KG completion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1414,"prompt_tokens":1018,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":298}},"tokens_in":634,"tokens_out":396,"duration_ms":4291,"temperature":1.0,"reasoning_tokens":298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:53:43.927123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a random sample of FB15k237 test triples, measure whether the ground-truth tail appears in the base model's top-50 ranking. If most KGR3 failures have ground truths outside that list, the bottleneck is candidate recall of the base model; if failures occur inside the list, the LLM or the prompt is the limiting component.","supporting_citations":[],"review_version":1}