{"id":"12e9511e-bbc6-40fc-88c8-1a26b7793c8f","arxiv_id":"2501.11911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TGL-LLM combines temporal graph embeddings with LLM tokenization and two-stage fine-tuning, achieving higher multiple-choice forecasting accuracy than existing TKGF baselines.","lead":"This paper introduces TGL-LLM, a framework that feeds temporal graph embeddings into an LLM via learned adapters and a two-stage fine-tuning schedule, and reports large accuracy gains on three POLECAT subsets for temporal knowledge graph forecasting. The work is a candidate improvement for event forecasting systems that combine graph structure with language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA may be an artifact of the recent-3-day multiple-choice candidate construction; standard full-entity ranking is needed before the central claim is credible.","rationale":"The reader identified the same weakest assumption: the recent-3-day multiple-choice candidate set and the risk that it favors models with recent temporal embeddings. My stress-test sharpens this into a concrete mechanism: TGL-LLM's hybrid prompt injects temporal graph tokens for every candidate, so the model is effectively solving a supervised matching problem over a small, time-localized candidate pool. Under the standard TKGF evaluation (rank against all entities with time-aware filtering), the reported Acc@4/6/10 are not evidence of state-of-the-art forecasting. The paper's own justification in Appendix A.3 is on different datasets and does not address the negative-sampling distribution. This concern is load-bearing because the entire empirical claim is the performance comparison; if full-entity ranking reverses the gap, the contribution is not established. The reader's CONDITIONAL verdict already captures the need for code/data and standard evaluation, so I do not propose changing the verdict. A single concrete test—full-entity ranking plus uniform-negative ablation—would settle whether the reported margin is real or an artifact of the MCQ protocol.","tokens_in":79,"tokens_out":4147,"duration_ms":54961,"concrete_test":"Re-run TGL-LLM and all baselines on POLECAT-IR/IS/EG under the standard time-aware filtered ranking over the full entity set, reporting MRR and Hits@1/10. Additionally, ablate the candidate distribution: replace the recent-3-day negatives with K=10 negatives sampled uniformly from all entities not appearing at the query timestamp. If TGL-LLM's margin over KoPA/CoH collapses or reverses under either test, the reported SOTA is an artifact of the candidate-construction protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 2, but the evaluation protocol in Sec 5.1.2 constructs candidates by randomly sampling 4/6/10 entities from the most recent three days, always including the positive. This makes the task a recency/biography-matching game rather than open TKGF: the candidate pool is restricted to entities active in a narrow recent window, so the model never has to rule out long-tail or unseen entities. TGL-LLM explicitly feeds temporal graph tokens of the subject, relation, and each candidate into the LLM, so it can directly compare the query against each candidate's recent history. A model that has no real forecasting ability could score well by choosing the candidate whose recent graph embedding best co-occurs with the query. The standard TKGF protocol instead ranks all entities with time-aware filtering; the reported Acc@4/6/10 are therefore not comparable to Hits@K/MRR in prior work. The Appendix A.3 attempt to justify MCQ uses GenTKG on ICEWS/GDELT, not the POLECAT subsets or TGL-LLM, and it only shows generated answers often appear in the prompt; it does not validate the recent-3-day negative distribution. Without full-entity evaluation, the 'large margin' over KoPA/CoH may reflect the MCQ candidate pool, not better forecasting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TGL-LLM, an LLM-based framework for temporal knowledge graph forecasting. It pre-trains a temporal graph model (RGCN with GRU and ConvTransE) to obtain historical entity and relation embeddings, converts the recent T steps of these embeddings into language tokens through entity/relation adapters and a learned feature token, and fine-tunes Llama2-7B with LoRA in two stages: first on a high-quality subset selected by an influence function, then on a smaller diversity subset. Experiments on three POLECAT subsets (Iran, Israel, Egypt) under a multiple-choice evaluation protocol report large gains over Non-LLM and LLM-based baselines, with ablations showing contributions from hybrid graph tokenization and, to a smaller degree, the two-stage training paradigm.","tokens_in":18850,"tokens_out":6782,"duration_ms":70557,"significance":"If the reported results hold, the paper's core idea of feeding temporally ordered graph tokens into an LLM is a plausible and useful direction for LLM-based TKGF, and the choice of POLECAT data after the LLM training cutoff is a genuine strength for avoiding leakage. However, the central SOTA claim currently rests on a non-standard candidate-construction protocol, and the statistical evidence is weak. The paper does not release code or data, does not report error bars, and the ANOVA significance claim is not supported by the presented experimental detail. The contribution is therefore promising but not yet convincingly established; the evaluation protocol and experimental rigor need substantial work before the claims can be accepted.","major_comments":[{"comment":"The central claim that TGL-LLM outperforms all baselines by a large margin rests on a multiple-choice protocol in which the 4/6/10 candidate entities are randomly sampled from the most recent three days, always including the positive entity. This is not the standard TKGF evaluation (full-entity ranking with time-aware filtering), so the Acc@4/6/10 numbers in Table 2 are not comparable to Hits@K/MRR reported in prior work. More importantly, the protocol may systematically favor TGL-LLM because its hybrid prompt feeds recent graph tokens of each candidate to the LLM, turning the task into a recency/co-occurrence matching problem rather than open forecasting. The justification in Appendix A.3 studies GenTKG on ICEWS/GDELT and only shows that generated answers usually appear in the prompt; it does not validate that recent-three-day negatives are representative for POLECAT or that the conclusion transfers to TGL-LLM. Please add a full-entity ranking evaluation (or at least negatives sampled from the full entity set or from a longer history) and report Hits@K/MRR; without this, the 'large margin' conclusion is not established.","section":"§5.1.2, Table 2"},{"comment":"The training-data budget appears inconsistent across methods. Section 5.1.4 says 'each experiment is trained on 100,000 training samples randomly selected from the training dataset,' but the same paragraph sets the high-quality subset |D_h| to 100,000 and the diversity subset |D_p| to 10,000. If TGL-LLM is fine-tuned on 100,000 + 10,000 = 110,000 samples while KoPA, CoH, and GenTKG receive 100,000 samples, the comparison in Table 2 is not matched in training data. Please clarify how the random 100,000-sample set relates to D_h and D_p, and either match the total number of training samples or updates across all methods or add a baseline trained on 110,000 samples.","section":"§5.1.4, §4.3.2"},{"comment":"The paper states that 'ANOVA significance test [over] the performances of all baselines. All of the evaluated p-values are below 0.01,' but Table 2 contains a single performance number per condition and no variance estimate or number of repeated runs. ANOVA requires within-condition variability; none is reported. The same problem affects Tables 3 and 4, where several differences are small (e.g., Acc@4 on POLECAT-IR: Random 0.8467 vs. TGL-LLM 0.8514; w/o-DS 0.8354). Please report means and standard deviations over at least three seeds and use an appropriate significance test, or remove the ANOVA claim.","section":"§5.2, Tables 2–4"},{"comment":"The data-pruning component, one of the two key innovations, is under-specified. Equation (11) defines influence scores, but the text does not describe how the scores are converted into the high-quality subset D_h: what 'stratified sampling' stratifies on, what threshold or budget is used, and how the 100,000-sample size is chosen. Since the surrogate graph model is trained on the same training set, the influence scores measure fit to the graph model's training loss, not necessarily informativeness for LLM cross-modal alignment; the assertion that high influence indicates high quality for LLM fine-tuning needs a more direct justification or an experiment showing that the selected subset is not simply the graph model's easy samples. Please provide the full selection algorithm and implementation details.","section":"§4.3.1, Eq. (11)"}],"minor_comments":[{"comment":"The POLECAT-IS Acc@10 value for ConvTransE is written as '0,6033' instead of '0.6033'.","section":"Table 3"},{"comment":"The legend and axis contain the typo 'Saprse'; it should be 'Sparse'.","section":"Figure 3"},{"comment":"There are several typos: 'issuse' should be 'issues', 'classicial' should be 'classical', and 'CovTransE' should be 'ConvTransE'.","section":"§4.3.1, §3, §5.4.1"},{"comment":"The statistical analysis uses GenTKG on ICEWS/GDELT, not the POLECAT subsets or TGL-LLM. Please clarify how these results support the multiple-choice protocol for the datasets and model used in this paper, or soften the equivalence claim.","section":"Appendix A.3"},{"comment":"The historical-length study reports a single curve per dataset without error bars, so the conclusions about the optimal history length (5 vs. 7) may reflect noise; please add variance estimates.","section":"§5.4.2, Figure 4"},{"comment":"The KoPA citation appears with year 2021; please verify that this is the correct publication year and venue for the cited work.","section":"Reference [47]"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible and interesting idea, and the post-cutoff POLECAT evaluation is a genuine strength for LLM-based TKGF. However, the central SOTA claim is not credible until the authors evaluate under a standard full-entity ranking protocol or at least justify the recent-three-day negative distribution on their own datasets. The ANOVA claim and the inconsistency in the training-data budget between TGL-LLM and the baselines also need to be fixed. I would be willing to reconsider after these issues are addressed, but I would not support acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a coherent, well-built method with a genuinely new combination, but the headline numbers come from a multiple-choice protocol that is not the standard TKGF evaluation, and the authors have not yet shown the margin survives full-entity ranking. The right verdict is conditional, not reject.\n\nWhat's new: feeding recent temporal graph embeddings as ordered graph tokens (subject, relation, candidate) through adapters into a LoRA-tuned LLM, plus a two-stage data-pruning schedule using influence functions for quality and random sampling for diversity. The pieces are known, but putting temporal graph embeddings directly in the LLM's token stream is a sensible next step over KoPA and GraphTranslator. I also credit the choice of POLECAT with a post-cutoff split for Llama2; that is a real attempt to dodge leakage, and the cost table is useful.\n\nWhere I push back: the evaluation. In Sec 5.1.2, candidate sets are built by sampling negatives from the most recent three days. That is a recency/biography-matching task, not open forecasting. TGL-LLM can compare the query against each candidate's recent graph tokens directly, so the large margin over KoPA and CoH may be an artifact of the candidate pool. The stress-test concern lands. The Appendix A.3 defense only shows that GenTKG's generated answers often come from the context on ICEWS/GDELT; it does not show that recent-three-day negatives are a faithful proxy for full ranking. Without standard Hits@K/MRR or full-entity ranking on at least one benchmark, the central SOTA claim is not established.\n\nThe other soft spots are smaller. No error bars or code/data release, and some hyperparameters (sampling procedure, influence-function details) are underspecified. In the ablations, the two-stage pipeline's gains over random sampling are real but uneven: on Acc@4 the IR gain is under a point, and the influence function's contribution is often marginal. That weakens the cross-modal alignment story, though not the overall recipe.\n\nBottom line: the paper is coherent, honestly motivated, and worth referee time, but it needs a standard-protocol evaluation, error bars, and code/data before the SOTA claim is credible. I would send it to review with those demands rather than desk-reject, and I would ask the authors to either add full ranking on ICEWS/GDELT or trim the claims.","headline":"A well-engineered embedding-based LLM method with a genuinely new combination, but the headline margin rests on a non-standard multiple-choice protocol and has not been shown to survive full-entity ranking.","tokens_in":19384,"tokens_out":3472,"would_cite":false,"duration_ms":38128,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TGL-LLM claims that an LLM forecasts events in a temporal knowledge graph far better when it receives the last few timestamps' entity embeddings as a time-ordered token sequence than when it receives retrieved text or a static graph…","keywords":["Temporal Knowledge Graph Forecasting","Large Language Models","Hybrid Graph Tokenization","Temporal Graph Learning","Influence Function Data Pruning","LoRA Fine-tuning","POLECAT Dataset","Multiple-Choice Event Forecasting"],"falsifier":"Two probes would settle it: replace the random-recent-negative MCQ protocol with full-entity ranking or with near-miss negatives, and shuffle the order of the temporal graph tokens; if accuracy collapses in either case, the mechanism claimed is not the one driving the results.","tokens_in":18373,"feed_emoji":"📈","tokens_out":8001,"duration_ms":80442,"temperature":0.7,"pith_summary":"TGL-LLM sets out to show that the bottleneck in LLM-based temporal knowledge graph forecasting is the form in which graph history is presented: retrieved text or a static graph embedding hides the order in which events unfold. The paper's claim is that encoding the most recent timestamps as entity embeddings and injecting them into the LLM as a time-ordered token sequence lets the language model exploit temporal patterns it otherwise cannot see. On three POLECAT subsets the authors report Acc@4 of 0.8514, 0.8779, and 0.8109, against a best baseline of 0.6083, 0.6110, and 0.5315. If this holds, the path to better LLM forecasting is to model time explicitly in the prompt rather than to add more textual history.","feed_headline":"Temporal graph tokens lift LLM event forecasting past 85% accuracy","feed_subtitle":"Injecting time-ordered entity embeddings into a fine-tuned LLM beats text-only and static-graph baselines on POLECAT.","key_machinery":"The load-bearing object is the hybrid prompt: a time-ordered sequence of recent historical entity embeddings, one token per snapshot, projected into the LLM's token space by an entity adapter and separated by a learned feature token. The same construction describes the query subject, the query relation, and each candidate answer, so the LLM reasons over an evolving entity state rather than a frozen snapshot or a textual list of past facts.","core_discovery":"The framework, TGL-LLM, learns an entity's recent history with a temporal graph model: an RGCN aggregates each timestamp's relational neighborhood, and during training a GRU passes over the snapshots while a ConvTransE decoder with cross-entropy loss fits the graph model. For the LLM input, the framework keeps the recent T RGCN embeddings rather than the recurrent summary, and projects each snapshot into token space with small entity and relation adapters. A learned feature token marks the start of each graph segment, and the tokens appear in chronological order as the description of the subject, relation, and candidate answers. A two-stage LoRA fine-tune, first on influence-function-selected high-quality data and then on a smaller diversity subset, aligns this graph modality with language. The paper reports that this combination outperforms every Non-LLM and LLM-based baseline on all three POLECAT subsets by a large margin.","pith_inferences":["Editorial inference: the random-negatives-from-the-last-three-days MCQ protocol may be an easier test than full-entity ranking; near-miss negatives with similar recent activity would be a harder probe, and full ranking would show whether the margin survives.","Editorial inference: the ablation results suggest the ordered recent snapshots themselves, not the GRU summary, may be what the LLM exploits; a probe that shuffles the token order would isolate temporal order as the active ingredient.","Editorial inference: the influence-plus-diversity data recipe is not tied to TKGF and could be transferred to other graph-to-text LLM alignments, such as static knowledge graph completion or recommendation, with the same adapter design.","Editorial inference: on POLECAT-IS the diversity stage adds little, so the optimal data recipe may vary by dataset; a cheaper single-stage variant could match performance when test patterns resemble training patterns."],"forward_implications":["Temporal graph tokens make the prompt shorter than text-based retrieval: TGL-LLM uses about 245 tokens per query versus 466 for CoH, while reaching higher accuracy, so the gain does not come from a larger context window.","Historical length is a real hyperparameter: accuracy rises from one day to about five to seven days of snapshots and then converges, which the paper reads as longer histories adding noise rather than signal.","The two-stage training data matters: removing either the influence-selected high-quality subset or the diversity subset lowers accuracy in most settings, showing that data quality and diversity both contribute to graph-language alignment.","On long-tail entities with few training links, TGL-LLM keeps a much smaller accuracy gap than non-LLM baselines, supporting the paper's argument that LLM knowledge plus aligned graph tokens handles sparse entities better.","Because the test window lies after the Llama2 training cutoff, the reported advantage is less likely to come from the LLM having memorized the test events."],"supporting_citations":[{"why":"Supplies the RGCN-plus-GRU temporal graph encoder that TGL-LLM follows to produce per-timestamp entity embeddings.","marker":"[22]"},{"why":"Used as the decoder whose cross-entropy loss pre-trains the temporal graph model.","marker":"[38]"},{"why":"The strongest embedding-based LLM baseline and the static-graph approach TGL-LLM is designed to beat.","marker":"[47]"},{"why":"Provides the influence-function formula used to score and prune training data for the high-quality subset.","marker":"[14]"},{"why":"Prior work on aligning graph embeddings with LLM token space that the adapter design follows.","marker":"[46]"},{"why":"Prior alignment method with a linear projector that motivates the temporal adapter approach.","marker":"[41]"},{"why":"Text-based LLM baseline that constructs prompts from temporal logical rules; a comparison target for the hybrid prompt.","marker":"[23]"},{"why":"Text-based fine-tuned LLM baseline whose accuracy TGL-LLM must beat and whose prompt style is used for zero-shot baselines.","marker":"[27]"},{"why":"The live dataset from which the three test subsets are cropped; its post-cutoff test window underpins the leakage argument.","marker":"[36]"},{"why":"The LLM backbone; its July 2023 knowledge cutoff defines the train/validation/test split.","marker":"[40]"}],"fun_headline_variants":["TGL-LLM: Temporal graph tokens teach LLMs to forecast events better than SOTA","Integrating temporal graph learning into LLMs: TGL-LLM beats all baselines on POLECAT","LLMs get a temporal graph memory: TGL-LLM outperforms on TKGF benchmarks","Temporal graph embeddings inside LLM: TGL-LLM improves event forecasting","TGL-LLM: How to make LLMs reason over time with graph snapshots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation turns forecasting into a multiple-choice test whose wrong answers are random recent entities, so the reported accuracy may overstate how well the model would rank the true answer against harder near-miss candidates or the full entity set.","fun_headline_variants_meta":{"raw":{"variants":["TGL-LLM: Temporal graph tokens teach LLMs to forecast events better than SOTA","Integrating temporal graph learning into LLMs: TGL-LLM beats all baselines on POLECAT","LLMs get a temporal graph memory: TGL-LLM outperforms on TKGF benchmarks","Temporal graph embeddings inside LLM: TGL-LLM improves event forecasting","TGL-LLM: How to make LLMs reason over time with graph snapshots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3392,"prompt_tokens":958,"completion_tokens":2434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2314}},"tokens_in":574,"tokens_out":2434,"duration_ms":16102,"temperature":1.0,"reasoning_tokens":2314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:43:52.353743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two probes would settle it: replace the random-recent-negative MCQ protocol with full-entity ranking or with near-miss negatives, and shuffle the order of the temporal graph tokens; if accuracy collapses in either case, the mechanism claimed is not the one driving the results.","supporting_citations":[{"cited_title":"MM-Forecast: A Multimodal Approach to Temporal Event Forecasting with Large Language Models","cited_arxiv_id":"2408.04388","evidence_quote":"Text-based LLM baseline that constructs prompts from temporal logical rules; a comparison target for the hybrid prompt."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RGCN-plus-GRU temporal graph encoder that TGL-LLM follows to produce per-timestamp entity embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The strongest embedding-based LLM baseline and the static-graph approach TGL-LLM is designed to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the influence-function formula used to score and prune training data for the high-quality subset."},{"cited_title":"Scarborough, Benjamin E","cited_arxiv_id":null,"evidence_quote":"The live dataset from which the three test subsets are cropped; its post-cutoff test window underpins the leakage argument."}],"review_version":1}