{"id":"5ac61e26-c178-47d3-90d9-91cd08879989","arxiv_id":"2412.13582","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EvoWiki categorizes facts as stable, evolved, or uncharted and shows that LLMs perform much worse on evolved and uncharted knowledge, with RAG plus continual learning providing the best adaptation.","lead":"This paper introduces EvoWiki, a benchmark built from Wikipedia and Wikidata that sorts facts into stable, evolved, or uncharted categories depending on when they changed. It tests whether language models can answer questions about recently changed or brand-new facts, and finds they lag on updated knowledge while retrieval and fine-tuning help more together than apart.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The global January 2024 cutoff is the linchpin of EvoWiki's labels; neither Llama-3.1-8B nor Mistral-7B has a verified January 2024 knowledge boundary, so the headline stable/evolved/uncharted gaps may partly reflect mislabeled facts rather than knowledge adaptation.","rationale":"The reader's conditional verdict rests on the same load-bearing concern I identify: a single global cutoff date, January 2024, is used to label all knowledge states for all evaluated models, and the model cards cited in Appendix A do not actually establish that either Llama-3.1-8B-Instruct or Mistral-7B-Instruct-v0.3 has that cutoff. This is the root of the benchmark's validity because every downstream claim—the closed-book performance gap, the 'models struggle to adapt' conclusion, and the contamination-free framing—is computed relative to those labels. Other weaknesses in the paper, such as the limited human evaluation of 180 samples and the statistically unquantified 'synergistic effect' of RAG plus CL, are real but secondary: they can be patched without invalidating the benchmark's core design, whereas a wrong model-specific cutoff would make the labels themselves unreliable. The concrete test I propose is directly feasible because EvoWiki's construction already has access to Wikidata revision history, so each fact's change date can be compared against independently estimated per-model cutoffs. Since this concern is precisely the reader's weakest assumption, my stress-test does not move the verdict; the paper should remain CONDITIONAL pending this check.","tokens_in":17515,"tokens_out":5069,"duration_ms":48351,"concrete_test":"Estimate each model's effective knowledge cutoff independently: build a temporal probe set of several hundred Wikidata facts whose object changed at known dates between September 2021 and May 2024, query each model closed-book, and fit accuracy versus event date to find the date at which recall drops sharply. Then relabel EvoWiki's stable/evolved/uncharted categories using each model's estimated cutoff instead of the fixed January 2024 date, recompute the single-hop closed-book rows of Table 4, and compare the stable-versus-evolved/uncharted gaps. If the gaps persist under per-model cutoffs, the concern is resolved; if they shrink materially or reverse, the benchmark needs model-specific timestamps before its contamination-free and knowledge-adaptation claims can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 fixes init-time to September 2021, cutoff-time to January 2024, and current-time to May 2024, and every stable/evolved/uncharted label is computed from those dates. Appendix A undermines the key premise: Mistral has no official cutoff, and the Llama-3 entry refers to Llama 3 (7B: March 2023; 70B: December 2023), not the evaluated Llama-3.1-8B-Instruct, whose release and training boundary are not the same. If Llama-3.1-8B's real cutoff is later than January 2024, then 'uncharted' facts introduced between January and May 2024 may already be in its training data; if its real cutoff is earlier, some 'evolved' facts may have changed before the model was trained and thus are not actually post-cutoff knowledge. The same uncertainty applies a fortiori to Mistral. This matters because the paper's central evidence—the closed-book gap between 31.61% on stable and 6.96%/10.84% on evolved/uncharted (Table 4)—is interpreted as 'models struggle to adapt to new knowledge,' and the benchmark's 'contamination-free' framing depends entirely on the cutoff being a true boundary for each model. The Limitations section acknowledges Wikidata noise but does not address the model-specific cutoff problem. Until the labels are shown to be correct relative to each model's actual training boundary, the headline performance gaps and the contamination-free claim remain conditional on an unverified assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EvoWiki, a QA benchmark constructed from Wikidata and Wikipedia snapshots at three timestamps (September 2021, January 2024, May 2024) and labels facts as stable, evolved, or uncharted relative to a global knowledge cutoff. The benchmark adds referenced contexts, multi-hop questions, and popularity attributes, and the paper evaluates Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 under closed-book, open-book, BM25/Contriever retrieval, continual pretraining/SFT, and combinations. The main empirical findings are that models score markedly higher on stable than on evolved/uncharted facts in closed-book QA, that RAG strongly improves single-hop but not multi-hop questions, that continual learning gives modest gains, and that combining RAG and CL is claimed to be synergistic.","tokens_in":17891,"tokens_out":9010,"duration_ms":75397,"significance":"EvoWiki addresses a real need: most QA benchmarks are static, and an updatable benchmark with explicit stable/evolved/uncharted labels and contextual evidence is a useful infrastructure contribution. The dataset construction is described concretely, the classification rule is stated from Wikidata snapshots and is independent of the evaluated models, and human evaluation provides some quality assurance. If the temporal labels are validated, the large closed-book gaps in Table 4 would be a valuable quantitative demonstration of LLMs' difficulty with post-training knowledge changes. The main empirical pattern is consistent across two models and several configurations. However, the paper's central interpretation is load-bearing on the temporal classification, and the current operational definitions and cutoff choices have not been shown to support it; the experimental claims also lack uncertainty quantification.","major_comments":[{"comment":"The definition of evolved facts in Section 3.1 ('facts that are established before init-time and exhibit changes between cutoff-time (or init-time) and current-time') and the implementation in Appendix B ('determine whether the triple has changed at cutoff-time or current-time') classify any fact that changed between September 2021 and January 2024 as evolved, even if the change occurred well before the January 2024 cutoff. This contradicts the preceding sentence in Section 3.1 that evolved facts 'were established before the cut-off date but have undergone changes since.' As a result, some 'evolved' facts may already be present in the model's training data, and the closed-book gaps reported in Table 4 conflate pre-cutoff updates with post-cutoff knowledge adaptation. Please restrict the evolved class to changes occurring after the cutoff (or explicitly redefine and justify the init-to-current criterion).","section":"Section 3.1, Appendix B"},{"comment":"Section 3.1 fixes one global cutoff (January 2024) for both models, but Appendix A does not supply a verified training boundary for either evaluated model: the Llama entry refers to Llama 3 7B/70B, not the evaluated Llama-3.1-8B-Instruct, and Mistral is listed as having no official cutoff. Because every stable/evolved/uncharted label and the 'contamination-free' framing is computed relative to this single date, the headline Llama single-hop closed-book gaps (31.61% vs 6.96% and 10.84% in Table 4) are only meaningful if the cutoff is correct for each model. Please provide per-model evidence of the knowledge boundary, use model-specific timestamps, or report a sensitivity analysis over cutoff dates showing that the label distribution and main results are stable.","section":"Section 3.1, Appendix A, Table 4"},{"comment":"All experimental numbers in Table 4 appear to be single runs without error bars, confidence intervals, or significance tests, yet the paper draws conclusions from small differences, e.g., 'modest yet consistent' CL gains of +1.87 pp and +1.57 pp on Llama single-hop evolved facts. The Section 4.5 'synergistic effect' is also asserted from point comparisons; for Llama multi-hop stable facts, SFT+Contriever (24.02) is actually below SFT closed-book (24.41), so the combination is not uniformly better than its best component. Please report multiple seeds or resampling-based intervals, and define and test the synergy claim against both RAG-only and CL-only baselines.","section":"Section 4.1, Table 4, Section 4.5"},{"comment":"The 552K SFT question-answer pairs (Appendix C) are generated from the same 15K Wikipedia chunks that define the golden contexts and the RAG corpus, yet the paper reports no overlap analysis between these training pairs and the EvoWiki evaluation questions. If the SFT data contains exact or near-duplicate questions and answers, the closed-book gains after SFT in Table 4 and the apparent RAG+CL synergy could reflect memorization of the evaluation answers rather than generalizable knowledge updating. Please quantify n-gram/embedding overlap between the SFT data and the test questions, remove overlapping examples, and confirm the conclusions on a disjoint evaluation split.","section":"Appendix C, Section 4.4"}],"minor_comments":[{"comment":"The abstract contains the typo 'Contunual Learning'; it should read 'Continual Learning'.","section":"Abstract"},{"comment":"Appendix C contains several typographical and grammatical errors: 'pre-tranining' should be 'pre-training', 'implementate' should be 'implemented', and 'Each document of Wikipedia are splited' is ungrammatical.","section":"Appendix C"},{"comment":"Appendix E.2 has 'descirption' for 'description', and Appendix E.5 has 'studen' for 'student'.","section":"Appendix E"},{"comment":"The outdated-answer scores are typeset inconsistently, e.g., '48.99(72.70)' without a space next to '36.13 (53.78)' with a space; use a uniform format.","section":"Table 4"},{"comment":"Appendix A should state explicitly how the listed Llama 3 cutoff dates relate to the evaluated Llama-3.1-8B-Instruct, or label the entry as not directly applicable to that model.","section":"Appendix A"},{"comment":"Figures 3 and 5 do not report the number of runs or any measure of variability; please either add error bars or explicitly state that these are single-run point estimates.","section":"Figures 3 and 5"},{"comment":"The Introduction refers to 'NaturalQuestion'; the correct benchmark name is 'Natural Questions'.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: EvoWiki is a solid, useful benchmark contribution, and the main pattern in the results is credible. The soft spot is the universal January 2024 cutoff for all models, which is an unverified assumption that the \"contamination-free\" framing rests on.\n\nWhat's actually new: the combination of three-level temporal classification (stable/evolved/uncharted) with referenced context, multi-hop, and popularity in a single auto-updatable dataset. Table 1 shows no prior benchmark covers all those cells. The construction is transparent: Wikidata snapshots at three timestamps, distant supervision against Wikipedia, human eval on 180 samples. The classification rule is explicit and independent of the evaluated models, so there's no circularity burden.\n\nThe experiments are fairly extensive: two models, RAG (BM25/Contriever, two corpus sizes), CL (CPT/SFT, orders), and combinations. The consistent pattern—closed-book stable >> evolved/uncharted, RAG helps single-hop but not multi-hop, CL gives modest gains—is believable and aligns with prior work on knowledge conflicts. I also appreciate the self-critique experiments showing a negative result; that's honest.\n\nSoft spots, in proportion. The biggest is the cutoff assumption. Section 3.1 fixes init/cutoff/current times once, and Appendix A lists Mistral as having no official cutoff and the Llama entries as Llama 3, not the evaluated Llama-3.1-8B-Instruct. If those models' actual boundaries differ from January 2024, some \"evolved\" and \"uncharted\" facts may already be in training data, which would weaken the contamination-free claim and the interpretation of the closed-book gaps. This isn't fatal to the benchmark—misclassification would likely attenuate the gaps, not create them—but it should be acknowledged and ideally addressed with per-model cutoff sensitivity analysis.\n\nSecond, all results are single runs with no error bars or statistical tests. For the headline gaps that's fine, but the \"synergistic effect of RAG and CL\" is asserted without a formal interaction test; it might just be additive gains. Minor in the context of a dataset paper.\n\nThird, the SFT data for CL is generated from the same 15K Wikipedia corpus that provides the golden contexts, so the CL results have a mild contamination risk of their own. This is a minor concern for the CL conclusions, not for the benchmark itself.\n\nBottom line: this is a worthwhile dataset paper for anyone working on temporal knowledge, RAG, or continual learning. The central claim—that current LLMs struggle more with evolved and uncharted knowledge—holds up. Send it to review; reviewers should push on the cutoff assumption and request variance reporting, but it's a solid paper.","headline":"EvoWiki is a genuinely useful benchmark with a credible central finding; the main weakness is the unverified single global cutoff date behind its contamination-free claim.","tokens_in":18400,"tokens_out":2651,"would_cite":true,"duration_ms":21792,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By labeling facts as stable, evolved, or uncharted relative to a cutoff date, EvoWiki finds that LLMs answer stable facts correctly about 31% of the time versus about 7% for evolved facts.","keywords":["evolving knowledge","knowledge evolution","LLM evaluation","retrieval-augmented generation","continual learning","knowledge contamination","benchmark","Wikidata"],"falsifier":"For a sample of facts labeled evolved, check whether the model already knew the post-change answer before the change was published—for instance, by querying an earlier model checkpoint trained strictly before January 2024 and seeing if it still produces the new answer at high confidence. Alternatively, re-running the full evaluation using each model's stated or estimated per-model cutoff instead of the fixed January 2024 date and showing that the stable/evolved/uncharted accuracy gaps largely disappear would directly disprove the claim that the split measures knowledge evolution rather than cutoff misalignment.","tokens_in":17356,"feed_emoji":"📊","tokens_out":8082,"duration_ms":64409,"temperature":0.7,"pith_summary":"This paper aims to establish that language models' ability to handle knowledge depends sharply on the temporal state of that knowledge relative to the model's training cutoff. To show this, the authors build EvoWiki, a continuously auto-updated benchmark that labels facts as stable, evolved, or uncharted by comparing Wikidata snapshots at three timestamps. Their measurements on two open models show closed-book accuracy of about 31% on stable single-hop facts but only about 7% on evolved facts and 11% on uncharted facts. They also find that retrieval-augmented generation helps single-hop questions but hurts multi-hop ones, while continual learning gives small and consistent gains, and combining the two improves performance further.","feed_headline":"LLMs score 7% on updated facts, 32% on stable ones","feed_subtitle":"An auto-updating benchmark separates stable, evolved, and new facts, exposing how poorly models adapt to change.","key_machinery":"The load-bearing object is the three-timestamp classification scheme (init-time, cutoff-time, current-time) applied to Wikidata/Wikipedia snapshots. Facts are labeled stable if unchanged across all three snapshots, evolved if they existed before the cutoff but changed afterward, and uncharted if they first appear after the cutoff. This scheme converts the vague notion of 'knowledge freshness' into discrete, automatically updatable categories that anchor every subsequent measurement, including retrieval and continual-learning comparisons.","core_discovery":"EvoWiki's central claim is that knowledge evolution can be measured as a three-way split defined by a fixed knowledge cutoff date: stable facts (unchanged from before the cutoff to the present), evolved facts (changed after the cutoff), and uncharted facts (introduced after the cutoff). Built from Wikidata triple histories and verified against Wikipedia text via distant supervision, the benchmark yields a reproducible measurement protocol with referenced context, multi-hop questions, and popularity attributes. Using this protocol, the paper demonstrates that Llama-3.1-8B answers stable single-hop facts correctly in 31.61% of cases but only 6.96% of evolved facts and 10.84% of uncharted facts, that RAG recovers much of the gap on single-hop but degrades multi-hop accuracy, and that continual learning combined with RAG produces a synergistic effect.","pith_inferences":["A testable extension of this result would be to replace the fixed January 2024 cutoff with per-model estimated cutoffs in future runs, which would clarify whether the stable/evolved/uncharted split is truly about knowledge evolution or partly about cutoff misalignment.","The finding that RAG hurts multi-hop accuracy on stable facts suggests that adding a separate reasoning or verification step to retrieval pipelines should recover the lost multi-hop performance.","Because EvoWiki is auto-updatable, it could be used to track whether the gap between stable and evolved knowledge closes across successive model releases, offering a public signal of progress on knowledge updating."],"forward_implications":["Static benchmarks likely overstate what current LLMs know about the world after their training cutoff.","Retrieval augmentation should not be assumed to fix multi-hop questions: noisy retrieval can actively hurt performance on stable knowledge.","Continual learning offers small but consistent gains and, when combined with retrieval, yields better adaptation than either alone.","Because EvoWiki is auto-updatable, it can be re-run as new models and new Wikidata changes appear, producing a moving picture of knowledge adaptation.","Popularity should be a factor in designing continual-learning training data: rare new facts may be learned better than popular ones when using supervised fine-tuning."],"supporting_citations":[{"why":"Supplies the NaturalQuestions static-benchmark baseline that EvoWiki contrasts with static QA evaluation.","marker":"Kwiatkowski et al., 2019"},{"why":"HotpotQA provides the multi-hop question answering basis that EvoWiki extends with temporal states.","marker":"Yang et al., 2018"},{"why":"CKL-LAMA is compared in Table 1 to show EvoWiki's additional evolution levels and attributes.","marker":"Jang et al., 2022b"},{"why":"Realtime QA motivates the auto-update requirement by demonstrating real-time answer evaluation.","marker":"Kasai et al., 2023"},{"why":"Foundational RAG method that EvoWiki uses as the main retrieval-augmented generation baseline.","marker":"Lewis et al., 2020"},{"why":"BM25 is one of the two retrievers whose performance on EvoWiki is reported.","marker":"Robertson and Zaragoza, 2009"},{"why":"Contriever is the dense retriever used in the RAG experiments.","marker":"Izacard et al., 2022"},{"why":"Provides the continual pre-training and supervised fine-tuning combination recipe that the paper follows.","marker":"Jiang et al., 2024"},{"why":"Establishes popularity as a factor in knowledge acquisition, which EvoWiki uses for continual-learning analysis.","marker":"Mallen et al., 2023"}],"fun_headline_variants":["EvoWiki: LLMs score 7% on evolved facts","LLMs answer only 7% of evolved facts","EvoWiki: knowledge evolution stumps LLMs","Benchmark exposes LLMs' failure to adapt","Auto-updating benchmark: LLMs lag on new facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire benchmark depends on a single global cutoff date of January 2024 being an accurate representation of each evaluated model's actual training-knowledge boundary; if a model's real cutoff differs (as the paper itself notes for Mistral and Llama-3.1), facts labeled evolved or uncharted may already be in the model's weights, and the reported gaps would be distorted.","fun_headline_variants_meta":{"raw":{"variants":["EvoWiki: LLMs score 7% on evolved facts","LLMs answer only 7% of evolved facts","EvoWiki: knowledge evolution stumps LLMs","Benchmark exposes LLMs' failure to adapt","Auto-updating benchmark: LLMs lag on new facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3259,"prompt_tokens":894,"completion_tokens":2365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2286}},"tokens_in":510,"tokens_out":2365,"duration_ms":16732,"temperature":1.0,"reasoning_tokens":2286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:59:14.073919+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a sample of facts labeled evolved, check whether the model already knew the post-change answer before the change was published—for instance, by querying an earlier model checkpoint trained strictly before January 2024 and seeing if it still produces the new answer at high confidence. Alternatively, re-running the full evaluation using each model's stated or estimated per-model cutoff instead of the fixed January 2024 date and showing that the stable/evolved/uncharted accuracy gaps largely disappear would directly disprove the claim that the split measures knowledge evolution rather than cutoff misalignment.","supporting_citations":[],"review_version":1}