{"id":"69b76cfb-dd1d-4b85-858a-11d3d6384c7f","arxiv_id":"2506.00445","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A general-to-specific training framework with an anonymization stage improves LLM forecasting on temporal knowledge graphs across datasets.","lead":"This paper proposes G2S, a two-stage framework that trains LLMs on anonymized Temporal Knowledge Graph structures and then adapts them to specific datasets via in-context learning or fine-tuning. It reports improved forecasting accuracy on ICEWS and YAGO benchmarks in zero-shot, low-resource, and standard settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot gains are confounded: no control training on the same 130k samples without anonymization, so the improvement attributed to disentanglement may just be exposure to more TKG data.","rationale":"The paper proposes a two-stage framework whose stated contribution is disentangling general temporal patterns from scenario information via anonymization. The zero-shot results in Table 2 are the primary evidence for this contribution. However, the comparison is asymmetric: G2SGL variants receive 130k supervised samples of the same task format, while the ICL baselines receive none. The only ablation that removes the general stage (w/o GL in Table 4) drops from 32.02 to 13.67 H@1, but this conflates the effect of anonymization with the effect of additional training data. A proper control would train on the same 130k samples with scenario information intact (original names/timestamps) and then evaluate zero-shot on the target datasets. The paper's G2SGL(F+Map) condition partially addresses this but still uses frequency-based IDs rather than original names and is trained only on GDELT. That condition's near-tie with G2SGL(F) on ICEWS14, despite the paper's claim that scenario information interferes with pattern learning, is a red flag: it suggests that scenario information from a schema-overlapping dataset is not harmful and may even transfer. The YAGO result is similarly explainable by WIKI-YAGO ontology overlap. I agree with the reader's weakest assumption and believe the missing control is the single most load-bearing gap. It does not, however, require rejecting the paper; the framework may still be effective, but the causal story needs direct evidence. The reader's CONDITIONAL verdict is appropriate, hence 'UNCHANGED'.","tokens_in":14358,"tokens_out":8233,"duration_ms":77875,"concrete_test":"Fine-tune Llama3-8B with the same LoRA hyperparameters (LR 1e-4, batch 8, 1 epoch, L=50) on the same 130k samples from GDELT+WIKI but WITHOUT anonymization: keep original entity/relation names and calendar timestamps, using the G2S prompt with entity/relation mapping blocks (as in Figure 2) and the answer as the entity name. Then evaluate zero-shot on ICEWS14/18/YAGO using the same ICL inference protocol (top-10 generation, time-aware filtering). Compare H@1/H@10 against G2SGL(R)w.WIKI in Table 2. If the non-anonymized control reaches the same or higher scores (within 1-2 H@1 points on all datasets), the anonymization/disentanglement step is not the cause of the zero-shot gains; if G2SGL stays clearly ahead across datasets, the concern is resolved. Run each setting with at least 3 seeds and report mean±std.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that anonymization in the general learning stage drives the zero-shot gains is not supported by the experimental design. G2SGL(R)w.WIKI is fine-tuned on 130k samples from GDELT+WIKI, whereas the zero-shot baselines (Llama-ICL, GPT-NeoX-ICL, Frequency) use no fine-tuning at all. The observed gains could therefore come from exposure to 130k TKG-forecasting samples (task-format adaptation) rather than from the removal of scenario information. The paper's own ablation w/o GL only removes the whole general stage; it does not control for training on the same data without anonymization. The closest condition, G2SGL(F+Map) (Table 2), adds scenario information to general learning but still uses frequency IDs and GDELT only, and it actually matches or slightly beats G2SGL(F) on the schema-overlapping ICEWS14 (H@1 32.13 vs 31.52). That result contradicts the interference hypothesis and suggests dataset-specific transfer (GDELT/ICEWS share the CAMEO relation schema) may explain the improvements better than 'general patterns'. Similarly, the large YAGO gain from adding WIKI (86.07 vs 80.40) is plausibly due to WIKI and YAGO both being Wikipedia-derived, not to an invariant temporal structure. Without a control trained on the same 130k samples with original entity/relation names and timestamps, the disentanglement mechanism remains unidentified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes G2S, a two-stage framework for temporal knowledge graph (TKG) forecasting with large language models. In the general learning stage, the model is fine-tuned on anonymized temporal structures from GDELT and WIKI, where entities, relations, and timestamps are replaced by abstract IDs. In the specific learning stage, scenario information is injected either via in-context learning or supervised fine-tuning. The framework is evaluated on ICEWS14, ICEWS18, and YAGO under standard, zero-shot, and low-resource settings, reporting improvements over in-context learning and prior fine-tuning baselines. The paper also includes ablations and an analysis of anonymization strategies.","tokens_in":14710,"tokens_out":4132,"duration_ms":36493,"significance":"If the disentanglement claim holds, G2S provides a practical recipe for improving LLM generalization to unseen TKG scenarios, which is timely given the growing interest in LLM-based temporal reasoning. The paper has several strengths: it evaluates across three distinct settings, includes multiple ablation variants, reports hyperparameter and infrastructure details, and provides a concrete case study illustrating a shared pattern across datasets. However, the central causal claim that anonymization in the general learning stage is what drives the zero-shot gains is not isolated by the experimental design. The general stage adds 130k training samples, and the ablations do not control for the effect of additional data or for the change in input format induced by anonymization. As a result, the current evidence supports the conclusion that the proposed framework works well, but not the stronger conclusion that disentanglement is the operative mechanism.","major_comments":[{"comment":"The zero-shot comparison is confounded by the amount of training data. G2SGL(R)w.WIKI is fine-tuned on 130k general-stage samples (100k from GDELT and 30k from WIKI), whereas the ICL baselines (Llama2-ICL, Llama3-ICL, GPT-NeoX-ICL) and the Frequency baseline are not fine-tuned at all. The observed gains could therefore be explained by exposure to additional TKG-forecasting samples and task-format adaptation rather than by the anonymization-induced disentanglement. The paper should include a control trained on the same 130k samples with original entity/relation names and timestamps (i.e., without anonymization) in the general stage. Without such a control, the central claim that anonymization drives the improvement is not supported.","section":"§5.1, §6.2, Table 2"},{"comment":"The w/o GL ablation removes the entire general learning stage, but the remaining specific-stage ICL prompt still uses anonymized IDs with entity/relation mapping parts. This is not the same input format as the standard ICL baselines in Table 2 (Llama3-ICL reports H@1 31.9 on ICEWS14, while w/o GL reports 13.67). The large drop could be due to the model being unfamiliar with the ID-based format rather than losing access to general temporal patterns. A control that uses the same G2S prompt structure but with non-anonymized names in the general stage is needed to separate format adaptation from pattern learning.","section":"§6.4, Table 4"},{"comment":"The result for G2SGL(F+Map) weakens the interference hypothesis. This variant deliberately injects scenario information during general learning, yet it matches or slightly exceeds G2SGL(F) on ICEWS14 (H@1 32.13 vs 31.52) and ICEWS18 (H@3 31.87 vs 31.17). The authors attribute this to the CAMEO schema overlap between GDELT and the ICEWS datasets, but that explanation suggests the improvements come from dataset-specific transfer rather than from learning invariant general patterns. The paper should explicitly reconcile this observation with the claim that simultaneously learning scenario information hurts generalization.","section":"§6.2, Table 2"},{"comment":"zrLLM (Ding et al., 2024) is cited in the related work as an LLM-based method for zero-shot relational learning on TKGs, but it is not included among the baselines in Table 2. Since zrLLM is specifically designed for zero-shot TKG forecasting with LLMs, it is directly relevant to the zero-shot generalization claims. The authors should either add this comparison or justify its exclusion in the experimental section.","section":"§2, §5.3"}],"minor_comments":[{"comment":"The description of G2SGL(R)w.WIKI states that it 'jointly utilizes GDELT and Y AGO' and adopts the RID strategy. Given the model name and the subsequent discussion about WIKI, this appears to be a typo: it should read 'GDELT and WIKI'.","section":"§6.2"},{"comment":"The notation 'w. W IKI' contains an extraneous space; it should be 'w.WIKI' (or 'w/ WIKI') for consistency with the model name in Tables 2 and 4.","section":"§6.2"},{"comment":"The statistics on multi-token IDs are useful, but the text could clarify whether the 'multi-token ID' percentages are computed on the validation set only and whether the same issue affects the test set used for the reported results.","section":"Appendix A, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially interesting, but the main mechanistic claim needs a cleaner experimental design. The most important missing experiment is a non-anonymized control trained on the same 130k samples in the general stage; without it, the paper reads as a strong empirical recipe rather than a validated disentanglement framework. I would also encourage the authors to compare with zrLLM under zero-shot settings, as the current baseline set omits the most closely related method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is sensible: separate the learning of general temporal patterns from scenario-specific facts by first fine-tuning on anonymized structures, then adapting to the target dataset. That framing is clean, and the paper does some things well. The relative timestamp encoding (Eq. 2) is a simple improvement over prior absolute-time IDs. The systematic comparison of FID, GID, and RID anonymization strategies is new and practically useful. The writing is clear, the experimental coverage (standard, zero-shot, low-resource) is appropriate, and the authors are upfront about limitations like the multi-token ID problem and the one-hop history selection.\n\nWhere the paper falls short is the evidence for its central claim. The zero-shot comparison pits G2S, which fine-tunes on 100k-130k general-stage samples, against ICL baselines that use no fine-tuning at all. The gain could come from exposure to more TKG-forecasting data, not from anonymization. The w/o GL ablation removes the entire general stage, so it does not control for data volume. The closest control, G2SGL(F+Map), adds scenario information during general learning and actually matches or slightly beats G2SGL(F) on the schema-overlapping ICEWS14/18. That cuts against the interference hypothesis and suggests dataset-specific transfer (GDELT and ICEWS share the CAMEO relation schema) may be doing the work.\n\nAlso notable: the paper cites zrLLM (Ding et al., 2024) but never compares against it, which is a real gap for a zero-shot claim. Minor issues: no code release, and anonymization strategies are selected on validation, which is standard but should be flagged.\n\nAll that said, this is an honest, well-structured empirical paper, not a circular one. The results are genuine test-set forecasts, and the framework is a practical contribution to LLM-based TKG forecasting. The main confound is addressable with a control that trains on the same general-stage data without anonymization, plus a zrLLM comparison. With those, the paper would be much stronger. I would send it to a serious referee, and I would expect a revision rather than a desk reject.","headline":"A useful two-stage recipe for LLM-based TKG forecasting, but the anonymization claim is confounded by extra training data and a missing zrLLM baseline.","tokens_in":15224,"tokens_out":3958,"would_cite":true,"duration_ms":36922,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Anonymized time-graphs lift LLM forecasting in zero-shot tests","keywords":["temporal knowledge graph forecasting","large language models","generalization","anonymization","in-context learning","supervised fine-tuning","zero-shot learning","low-resource learning"],"falsifier":"Train a control model on the same GDELT+WIKI samples but with raw, non-anonymized facts (or with a fixed random ID mapping shared between training and target datasets), and measure zero-shot ICEWS14 performance; if the control matches G2S, the improvement is not caused by anonymized structure learning. Alternatively, a dataset with a radically different schema where anonymous patterns cannot transfer should show no zero-shot gain, which would bound the claim.","tokens_in":14198,"feed_emoji":"🧩","tokens_out":5749,"duration_ms":49986,"temperature":0.7,"pith_summary":"The paper claims that LLM-based temporal knowledge graph forecasting fails to generalize across domains because the model learns two kinds of knowledge at once: reusable temporal patterns and scenario-specific facts (who did what, under which relation, at which time granularity). It proposes a General-to-Specific framework (G2S) that first trains on anonymous temporal structures—entities, relations, and timestamps replaced by abstract IDs—from GDELT and WIKI, and only later injects the concrete entities and relations of the target dataset through in-context learning or fine-tuning. The result is a forecast model that outperforms existing ICL and SFT baselines under zero-shot, low-resource, and standard settings on ICEWS14, ICEWS18, and YAGO. If true, this separates a transferable 'grammar' of temporal events from the vocabulary of a particular scenario, making LLM forecasters cheaper to adapt to new domains.","feed_headline":"Anonymized time-graphs lift LLM forecasting in zero-shot tests","feed_subtitle":"Structure-first, names-second training lifts an 8B LLM past stronger baselines on unseen temporal graphs.","key_machinery":"The carrying object is the anonymous temporal structure (ATS): a query-time-relative rendering in which each history line is \"A(t):[A(s),A(r),A(o)]\" with $A(t) = t_q - t$, and entities/relations mapped to IDs by one of three strategies (frequency rank, global dataset ID, or random ID). The general learning stage fine-tunes an LLM (8B) on these ATS samples from GDELT and WIKI, forcing it to learn structural patterns independent of names; the specific learning stage prepends entity/relation ID mappings and continues with ICL or SFT. The relative timestamp (query time = 0) is what lets patterns from 15-minute-granularity GDELT transfer to daily ICEWS and yearly YAGO.","core_discovery":"The central claim is that temporal knowledge graphs contain two entangled kinds of knowledge—general patterns (invariant structures such as recurring periodic sequences like [1,1,2],[1,1,4],[1,1,2] → [1,1,4]) and scenario information (the concrete entities, relations, and time granularity). G2S disentangles them by anonymizing the scenario information into abstract IDs during a general learning stage, then restoring it in a specific learning stage. Evidence: zero-shot variants trained only on GDELT and WIKI achieve accuracy close to or above a 20B-parameter ICL model, and fine-tuned G2S beats or matches baselines on three benchmarks. The paper interprets these gains as confirming that separate learning stages prevent interference between general patterns and scenario information.","pith_inferences":["Editorial inference: the same anonymize-then-specialize recipe could apply to other LLM sequence tasks with a procedural core and a lexical surface—e.g., event extraction, procedural planning, or code synthesis—where the general pattern (control flow) is masked from domain names.","Editorial inference: the paper's relative-time encoding (query time = 0) suggests that future work could test whether shifting from absolute calendar time to relative intervals is itself the main transfer mechanism, independent of entity anonymization.","Editorial inference: a testable extension would be to train the general stage on purely synthetic anonymous temporal structures with known planted patterns; if G2S improves as planted pattern complexity increases, the causal link between structure-learning and generalization strengthens."],"forward_implications":["If the general stage truly learns transferable patterns, a single anonymous-pattern pretrained model should adapt to new TKG domains with only a handful of example mappings, reducing the labeled data needed for forecasting.","The framework implies that ICL and SFT are complementary rather than competing: the general stage pretrains on structure, and the specific stage can switch modes depending on data availability.","On ICEWS18 the paper notes G2S slightly underperforms GenTKG, attributing it to multi-token ID generation; fixing single-token decoding for global IDs should close that gap.","The anonymization strategies matter: global IDs work best with enough data, while frequency IDs are better in low-resource settings, suggesting practical guidance for choosing a strategy per dataset."],"supporting_citations":[{"why":"Supplies the in-context learning baseline and the frequency-based ID strategy that G2S builds on and must outperform.","marker":"Lee et al. (2023)"},{"why":"GenTKG is the supervised-fine-tuning baseline whose simultaneous learning of patterns and scenario information G2S is designed to beat; also provides low-resource comparison numbers.","marker":"Liao et al. (2024)"},{"why":"Defines the TKG forecasting formulation, the one-hop history selection, and the GDELT/ICEWS18 datasets used in training and evaluation.","marker":"Jin et al. (2020)"},{"why":"Provides the Llama 3 backbone that G2S fine-tunes with LoRA.","marker":"AI@Meta (2024)"},{"why":"The LoRA method used for parameter-efficient fine-tuning in both learning stages.","marker":"Hu et al. (2022)"},{"why":"Source of the WIKI TKG used in the general learning stage.","marker":"Leblay and Chekol (2018)"}],"fun_headline_variants":["Disentangle to generalize: G2S boosts LLM forecasting on temporal graphs","Anonymize first, then specialize: G2S lifts zero-shot TKG forecasting","LLM forecasting gets a G2S boost by separating patterns from specifics","G2S: Teach LLMs temporal patterns before adding scenario details","Two-stage learning helps LLMs foresee on unseen temporal graphs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that anonymizing entities, relations, and timestamps into abstract IDs removes exactly the scenario-specific information while preserving the generalizable temporal structure; if the zero-shot gains instead come from the extra 130,000 training samples or from latent leakage through ID-frequency correlations, the disentanglement story collapses.","fun_headline_variants_meta":{"raw":{"variants":["Disentangle to generalize: G2S boosts LLM forecasting on temporal graphs","Anonymize first, then specialize: G2S lifts zero-shot TKG forecasting","LLM forecasting gets a G2S boost by separating patterns from specifics","G2S: Teach LLMs temporal patterns before adding scenario details","Two-stage learning helps LLMs foresee on unseen temporal graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1737,"prompt_tokens":948,"completion_tokens":789,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":692}},"tokens_in":564,"tokens_out":789,"duration_ms":7241,"temperature":1.0,"reasoning_tokens":692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:04:16.329771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a control model on the same GDELT+WIKI samples but with raw, non-anonymized facts (or with a fixed random ID mapping shared between training and target datasets), and measure zero-shot ICEWS14 performance; if the control matches G2S, the improvement is not caused by anonymized structure learning. Alternatively, a dataset with a radically different schema where anonymous patterns cannot transfer should show no zero-shot gain, which would bound the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the WIKI TKG used in the general learning stage."}],"review_version":1}