REVIEW 4 major objections 5 minor 30 references
RECIPE-TKG: From Sparse History to Structured Reasoning for LLM-based Temporal Knowledge Graph Completion
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper argues that LLM-based temporal knowledge graph completion is bottlenecked not by model capacity but by shallow history retrieval and evaluation choices, and that a three-part framework — rule-based multi-hop sampling…
desk verdict Useful framework, but the filter's credit is probably a historical fallback in disguise, and the LLM baseline comparison is incomplete. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a three-component pipeline rather than a single identity. The sampler's load-bearing object is the composite weight $w = w_n \cdot w_f \cdot (w_t + w_c + w_{cp})$ over candidate quadruples, which multiplicatively combines reachability, anti-frequency, temporal recency, subject/relation co-occurrence, and connectivity to the initial rule-based context, so structurally distant but relevant facts enter the prompt. The learner uses a margin-based contrastive loss $\mathcal{L}_{\text{contrastive}} = \frac{1}{N_c} \sum_{i=1}^{N_c} \max(0, \|a_i - \text{pos}_i\|_2 - \|a_i - \text{neg}_i\|_2 + m)$ over relation-labeled positive/negative pairs, computed on attention-aggregated entity embeddings. The refiner applies the decision rule $p' = p$ if $p \in H$ or $\phi(p,c) \ge \tau$, otherwise regenerate, otherwise $\arg\max_{h \in H} \psi(h)$ with $\psi(h) = \beta f(h) + (1-\beta) r(h)$, where $\phi(p,c)$ is the cosine similarity from the frozen all-mpnet-base-v2 sentence transformer. The ablation attributes the single largest Hits@10 jump (0.580 to 0.651 on ICEWS14) to this final filtering step.
What would settle it
On each dataset, re-estimate the filtering threshold $\tau$ on the test predictions themselves and on a scrambled (prediction, context) pairing; if the Hits@10 gain from filtering (0.580 to 0.651 on ICEWS14) shrinks to noise once $\tau$ is re-tuned on the test distribution, the filter is exploiting threshold over-fitting rather than semantic alignment. The paper reports the relevant similarity gap only for ICEWS14 ($\Delta\mu = 0.057$ between correct and incorrect predictions), so repeating the same histogram analysis on GDELT and YAGO — where filtering is credited with a 16.8% Hits@10 gain — would settle whether the assumption transfers.
Extended reading notes
Core claim
The central discovery is that the apparent gains of LLM-based TKG forecasting are dominated by how history is retrieved and how outputs are post-processed, not by how well the model is trained: under a unified evaluation harness, switching the evaluation pipeline alone raises Hits@1 from 25.8% to 34.4%, while switching sampling adds roughly one more point and fine-tuning adds about 1.3 points. RECIPE-TKG turns this diagnosis into a method. First, rule-based multi-hop history sampling augments subject-anchored temporal-rule facts with a weighted multi-hop expansion that favors reachable, recent, low-frequency, and structurally connected quadruples. Second, contrastive fine-tuning with LoRA adapters, using relation-guided positive and negative neighbor pairs and a margin loss on attention-aggregated entity embeddings, sharpens the latent space so semantically compatible entities cluster. Third, test-time semantic filtering accepts a generation only if its sentence-embedding cosine similarity to the prompt exceeds a tuned threshold or the entity already appears in retrieved history; otherwise it regenerates and ultimately falls back to a frequency-and-recency-scored historical candidate. The paper reports that this combination yields relative improvements of 8.0% to 30.6% over the best LLM baselines across the four benchmarks, lifts Hits@10 above 60% even in the sparsest history bins where prior methods stay below 30%, and produces semantically coherent near-miss predictions even when exact matches are absent from history.
Load-bearing premise
The largest single accuracy jump comes from the test-time filter, and that filter assumes that the cosine similarity between a generated answer and the prompt text — computed with one fixed sentence transformer — reliably separates good from bad predictions, and that the single threshold tuned on a development set keeps working on test queries without retuning.
Editorial extensions
If this is right
- On ICEWS14 and YAGO the method reports new state-of-the-art scores among all compared methods — up to 11.9% relative improvement over the strongest competitor — and on ICEWS18 it beats the best LLM baseline by 30.6% relative Hits@10 while matching the top embedding-based method.
- Sparse-history queries are not inherently harder: with only 0–2 retrieved facts, Hits@10 rises above 60% where previous LLM methods sit below 30%, so retrieval depth, not model capacity, was the binding constraint.
- Because the controlled re-evaluation shows that most of GenTKG's reported gain over ICL comes from evaluation postprocessing and sampling rather than supervised fine-tuning, future LLM-based TKG comparisons need a standardized decoding-and-evaluation harness before attributing gains to reasoning.
- Contrastive tuning produces semantically plausible near-miss predictions even when the exact gold entity is absent from history, which improves ranked candidate lists in forecasting and decision-support settings.
- The framework needs only a few thousand training shots and LoRA adapters, and it transfers to a newer base model (LLaMA-3-8B) with comparable results, supporting deployment without full model retraining.
Reading between the lines
- The method redirects compute from training to retrieval and verification, suggesting a template for other knowledge-intensive generation tasks: keep the base model frozen, enrich the prompt with graph-structured evidence, and gate outputs by embedding similarity to the context.
- Because cosine similarity to the prompt is treated as a correctness proxy, the filter amounts to a generic hallucination guard; a natural transfer test is to apply the same thresholded filtering to closed-book question answering or fact checking, where context-misaligned generations are also the dominant error type.
- The re-evaluation result (evaluation pipeline moves Hits@1 by +8.6 points, fine-tuning by +1.3) implies that published LLM-based TKG comparisons may be comparing pipelines rather than models; a shared evaluation harness, not a new model, would be the cheapest next contribution to the field.
- The paper's own limitations — clean, fully observed graphs and rule mining that must be rerun whenever the TKG changes — point to the stress test that would bound the approach's usefulness: injecting dropped or delayed events into the history and measuring how the sampling weights and the filter threshold degrade.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RECIPE-TKG, a three-component framework for LLM-based temporal knowledge graph completion: rule-based multi-hop history sampling (RBMH), contrastive fine-tuning of LoRA adapters with a relation-guided contrastive loss, and test-time semantic filtering with a historical fallback. The method is evaluated on ICEWS14, ICEWS18, GDELT, and YAGO using LLaMA2-7B (and LLaMA3-8B in a case study) against embedding-based, rule-based, and two LLM-based baselines (ICL and GenTKG). The authors report consistent improvements on most metrics, with the largest gains in Hits@10 and in low-history settings, and they also re-evaluate prior baselines under a unified setup to show that much of GenTKG's reported gain comes from evaluation and sampling choices.
Significance. If the central claims hold, RECIPE-TKG is a useful modular contribution for LLM-based TKG forecasting in sparse-history conditions, and the re-evaluation of prior baselines under consistent evaluation is valuable. The paper is also commendable for releasing code and for providing hyperparameter-sensitivity analyses. However, the significance is currently limited by three weaknesses: the abstract overstates the GDELT results, the largest single contribution (test-time filtering) is not isolated from the historical fallback, and recent LLM-based baselines are omitted from the comparison. These gaps, if addressed, would substantially strengthen the paper.
major comments (4)
- [Abstract and Section 5.2, Table 2] The abstract and introduction claim that RECIPE-TKG 'outperforms previous LLM-based approaches' with 'relative gains on Hits@1/3/10 ranging from 8% to 30.6%.' Table 2 shows this is not accurate for GDELT: RECIPE-TKG's Hits@1 is 0.095 versus 0.099 for GenTKG (a relative decrease of 4.0%) and Hits@3 is 0.192 versus 0.193 (relative decrease of 0.5%); only Hits@10 improves. The headline claim should be qualified to reflect that the method is better on most metrics but not uniformly so.
- [Section 4.3, Appendix C, Table 4] The test-time filtering contribution is not isolated from the historical fallback. In Eq. 8, a prediction is accepted if it appears in H or if cosine similarity exceeds tau; otherwise the model regenerates once (k=1) and then falls back to a frequency/recency scorer psi(h) in Eq. 9. Appendix C reports that only 1.5% of non-historical predictions are correct and that 9.1% of predictions are non-historical while the gold entity is in H. These facts imply that replacing low-similarity predictions with the historical fallback alone would correct most of these errors. Indeed, Table 4 shows the filter raises Hits@10 from 0.580 to 0.651 while Hits@1 moves only 0.392 to 0.393, a pattern consistent with the fallback filling lower-ranked slots. To substantiate the claim that the semantic similarity threshold is load-bearing, the authors must (i) report per-dataset tau and beta values and the development-set sizes, and (ii) ablate the similarity test against a random acceptance rule matched on acceptance rate, or against a rule that always accepts predictions in H and falls back after one regeneration. Without this, the paper does not establish that the embedding-based filter contributes beyond the hard 'p in H' acceptance and the historical fallback.
- [Section 1 and Section 5.1 (Baselines), Table 2] The paper cites Luo et al. (2024) and Xia et al. (2024) as recent LLM-based TKG methods in the introduction, but Table 2 compares only ICL (Lee et al., 2023) and GenTKG (Liao et al., 2024). The claim of state-of-the-art among LLM-based methods on ICEWS14 and YAGO is therefore only with respect to two baselines. The authors should include these recent methods (or explicitly justify their exclusion), since the reader cannot otherwise evaluate the competitiveness of the proposed approach.
- [Section 5.1 and Table 2] The main results are reported without error bars or multiple runs. Given the near-ties on GDELT (e.g., Hits@1 0.095 vs 0.099) and the stochastic components (LoRA training, sampling, generation), the reader cannot assess whether the negative deltas are significant. Reporting mean and standard deviation over at least three seeds, or providing significance tests, is necessary to support the claimed improvements.
minor comments (5)
- [Section 6.1, Table 4 paragraph] The sentence 'combining test-time filtering with RBMH Sampling and Test-time Filtering (RECIPE-TKG)' appears to have a typo; it should likely read 'combining RBMH Sampling, contrastive fine-tuning, and test-time filtering (RECIPE-TKG).'
- [Table 1 caption] The caption does not specify the dataset on which the re-evaluation was performed; the text implies ICEWS14 but this should be stated explicitly.
- [Section 6.1] The text says 'in-context learning (ICT)' but the abbreviation should be ICL, as used elsewhere in the paper.
- [Appendix C] The description of k is ambiguous: the text says 'we use k=1' but Eq. 8 says 'after k attempts.' Please clarify whether k denotes the number of regeneration attempts after the initial generation or the total number of attempts.
- [Appendix B.4] There is a missing space in 'fine-tuneLLaMA-2-7B'; it should read 'fine-tune LLaMA-2-7B.'
Circularity Check
No significant circularity: the framework's components are tested against external baselines and independent test labels, and no prediction reduces to its inputs by construction.
full rationale
RECIPE-TKG's three components are not defined in terms of the target metrics. Rule-based multi-hop sampling adapts the external TLR algorithm and adds explicitly defined weighting components; contrastive fine-tuning uses relation labels from an external GPT-4o system and a standard contrastive loss; test-time filtering uses a frozen external sentence transformer (all-mpnet-base-v2), with the threshold tau optimized on a development set rather than on test labels, and the fallback is a transparent frequency/recency score over retrieved history. The headline results compare against independent embedding-based, rule-based, and LLM-based baselines using held-out test splits. The only self-citation (Wang et al., 2025, on which one co-author appears) is used in a general statement about LLM reasoning benchmarks and is not load-bearing. The large Hits@10 gain attributed to filtering may partly reflect the historical fallback mechanism rather than semantic similarity, but that is a potential experimental confound, not a circular reduction: the filter's output is not equivalent to its input by definition, and the evaluation remains grounded in external test labels. Therefore no circular step is established, and the derivation chain is self-contained with respect to the concerns enumerated in the circularity analysis.
Assumptions & free parameters
free parameters (7)
- tau (semantic similarity threshold) =
learned on dev set; ~0.6 for ICEWS14 (Figure 11)
- alpha (contrastive loss weight) =
0.2
- margin m =
1.0
- beta (fallback balance) =
0.6
- gamma_1 to gamma_4 (sampling weights) =
0.6, 0.6, 0.01, 0.1
- N (max history length) =
50
- regeneration attempts k =
1
assumptions (4)
- domain assumption Temporal knowledge graphs are clean and fully observed; all historical events are available and correct.
- domain assumption GPT-4o relation classification into positive/negative/neutral is accurate enough for contrastive pair construction.
- domain assumption Sentence-transformer cosine similarity between prediction and prompt is a reliable proxy for correctness.
- ad hoc to paper The 'enemy of my enemy is my friend' heuristic is a valid relational prior for geopolitical TKGs.
Cite this review
Pith. "Pith review of RECIPE-TKG: From Sparse History to Structured Reasoning for LLM-based Temporal Knowledge Graph Completion." pith.science (2026). https://pith.science/paper/JJ4VEJT3
@misc{pith2026250517794,
author = {Pith},
title = {Pith review of: RECIPE-TKG: From Sparse History to Structured Reasoning for LLM-based Temporal Knowledge Graph Completion},
year = {2026},
howpublished = {\url{https://pith.science/paper/JJ4VEJT3}},
note = {Machine review of arXiv:2505.17794}
}
read the original abstract
Temporal Knowledge Graphs (TKGs) represent dynamic facts as timestamped relations between entities. TKG completion involves forecasting missing or future links, requiring models to reason over time-evolving structure. While LLMs show promise for this task, existing approaches often overemphasize supervised fine-tuning and struggle particularly when historical evidence is limited or missing. We introduce RECIPE-TKG, a lightweight and data-efficient framework designed to improve accuracy and generalization in settings with sparse historical context. It combines (1) rule-based multi-hop retrieval for structurally diverse history, (2) contrastive fine-tuning of lightweight adapters to encode relational semantics, and (3) test-time semantic filtering to iteratively refine generations based on embedding similarity. Experiments on four TKG benchmarks show that RECIPE-TKG outperforms previous LLM-based approaches, achieving up to 30.6\% relative improvement in Hits@10. Moreover, our proposed framework produces more semantically coherent predictions, even for the samples with limited historical context.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
Elizabeth Boschee, Jennifer Lautenschlager, Sean O’Brien, Steve Shellman, James Starz, and Michael Ward. 2015. https://doi.org/10.7910/DVN/28075 ICEWS Coded Event Data
-
[2]
Rochana Chaturvedi. 2024. https://dl.acm.org/doi/10.1145/3589335.3651256 Temporal knowledge graph extraction and modeling across multiple documents for health risk prediction . In Companion Proceedings of the ACM Web Conference 2024, pages 1182--1185
arXiv 2024
- [3]
-
[4]
Julia Gastinger, Timo Sztyler, Lokesh Sharma, Anett Schuelke, and Heiner Stuckenschmidt. 2023. https://dl.acm.org/doi/abs/10.1007/978-3-031-43418-1_32 Comparing apples and oranges? on the evaluation of methods for temporal knowledge graph forecasting . In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 533--549. Springer
-
[5]
Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2020. https://openreview.net/forum?id=pGIHq1m7PU Explainable subgraph reasoning for forecasting on temporal knowledge graphs . In International Conference on Learning Representations
work page 2020
-
[6]
Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021. https://aclanthology.org/2021.emnlp-main.658/ Learning neural ordinary equations for forecasting future links on temporal knowledge graphs . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8352--8364
work page 2021
-
[7]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeGNn4cx5c Lora: Low-rank adaptation of large language models . In International Conference on Learning Representations (ICLR)
work page 2022
-
[8]
Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Kai Yao, Jia Xu, Linjian Mo, and Min Zhang. 2025. https://doi.org/10.48550/arXiv.2501.02497 Test-time compute: from system-1 thinking to system-2 thinking . Preprint, arXiv:2501.02497
Show all 30 references
-
[9]
Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020. https://aclanthology.org/2020.emnlp-main.541/ Recurrent event network: Autoregressive structure inference over temporal knowledge graphs . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proc...
2020
-
[10]
Dong-Ho Lee, Kian Ahrabian, Woojeong Jin, Fred Morstatter, and Jay Pujara. 2023. https://arxiv.org/abs/2305.10613 Temporal knowledge graph forecasting without knowledge using in-context learning . Preprint, arXiv:2305.10613
2023 arXiv
-
[11]
Kalev Leetaru and Philip A Schrodt. 2013. Gdelt: Global data on events, location, and tone, 1979--2012. In ISA annual convention, volume 2, pages 1--49. Citeseer
2013
-
[12]
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. https://arxiv.org/abs/2206.14858 Solving quantitative re...
2022 arXiv
-
[13]
Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021. https://dl.acm.org/doi/10.1145/3404835.3462963 Temporal knowledge graph reasoning based on evolutional representation learning . In Proceedings of the 44th Internatio...
2021
-
[14]
Ruotong Liao, Xu Jia, Yangzhe Li, Yunpu Ma, and Volker Tresp. 2024. https://arxiv.org/abs/2310.07793 Gentkg: Generative forecasting on temporal knowledge graph with large language models . Preprint, arXiv:2310.07793
2024 arXiv
-
[15]
Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. 2022. https://cdn.aaai.org/ojs/20330/20330-13-24343-1-2-20220628.pdf Tlogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs . In Proceedings of the Thirty-Sixth A...
2022
-
[16]
Ruilin Luo, Tianle Gu, Haoling Li, Junzhe Li, Zicheng Lin, Jiayi Li, and Yujiu Yang. 2024. https://arxiv.org/abs/2401.06072 Chain of history: Learning and forecasting with llms for temporal knowledge graph completion . Preprint, arXiv:2401.06072
2024 arXiv
-
[17]
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. 2013. Yago3: A knowledge base from multilingual wikipedias. In CIDR
2013
-
[18]
Shreyas Mangrulkar and 1 others. 2022. Peft: Parameter-efficient fine-tuning. https://github.com/huggingface/peft. GitHub repository, accessed May 2025
2022
-
[19]
Meta AI . 2024. Meta llama 3: Open foundation and fine-tuned chat models. https://ai.meta.com/blog/meta-llama-3/. Accessed: 2025-05-16
2024
-
[20]
all-mpnet-base-v2
Sentence-Transformers. all-mpnet-base-v2. https://huggingface.co/sentence-transformers/all-mpnet-base-v2. Accessed: 2025-05-19
2025
-
[21]
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters . Preprint, arXiv:2408.03314
2024 arXiv
-
[22]
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://proceedings.neurips.cc/paper/2020/file/c3a690be93aa602ee2dc0ccab5b7b67e-Paper.pdf Mpnet: Masked and permuted pre-training for language understanding . In Advances in Neural Information Processing Systems...
2020
-
[23]
Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021. https://arxiv.org/abs/2109.04101 Timetraveler: Reinforcement learning for temporal knowledge graph forecasting . arXiv preprint arXiv:2109.04101
2021 arXiv
-
[24]
Hugo Touvron, Louis Martin, Kevin Stone, Abdullah Al-Dujaili, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . arXiv preprint arXiv:2307.09288
2023 arXiv
-
[25]
Volker Tresp, Crist \'o bal Esteban, Yinchong Yang, Stephan Baier, and Denis Krompa . 2015. https://arxiv.org/abs/1511.07972 Learning with memory embeddings . arXiv preprint arXiv:1511.07972
2015 arXiv
-
[26]
Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017. https://proceedings.mlr.press/v70/trivedi17a.html Know-evolve: Deep temporal reasoning for dynamic knowledge graphs . In Proceedings of the 34th International Conference on Machine Learning, pages 3462--3471. PMLR
2017
-
[27]
O mer Faruk Akg \
Shangshang Wang, Julian Asilis, \"O mer Faruk Akg \"u l, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. 2025. https://arxiv.org/abs/2504.15777 Tina: Tiny reasoning models via lora . arXiv preprint arXiv:2504.15777
2025 arXiv
-
[28]
Yuwei Xia, Ding Wang, Qiang Liu, Liang Wang, Shu Wu, and Xiao-Yu Zhang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.955 Chain-of-history reasoning for temporal knowledge graph forecasting . In Findings of the Association for Computational Linguistics: ACL 2024, pages 1...
2024 doi
-
[29]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.