Pith. sign in

REVIEW 4 major objections 7 minor 2 cited by

EvoWiki: Evaluating LLMs on Evolving Knowledge

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By labeling facts as stable, evolved, or uncharted relative to a cutoff date, EvoWiki finds that LLMs answer stable facts correctly about 31% of the time versus about 7% for evolved facts.

desk verdict EvoWiki is a genuinely useful benchmark with a credible central finding; the main weakness is the unverified single global cutoff date behind its contamination-free claim. read the letter →

arxiv 2412.13582 v1 pith:I2B3SJCR submitted 2024-12-18 cs.CL

Wei Tang , Yixin Cao , Yang Deng , Jiahao Ying , Bo Wang , Yizhe Yang , Yuyue Zhao , Qi Zhang
show 3 more authors
Xuanjing Huang Yugang Jiang Yong Liao
This is my paper · ORCID
classification cs.CL
keywords evolvingknowledgeevolutionLLMevaluationretrieval-augmentedgenerationcontinuallearningcontaminationbenchmarkWikidata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that language models' ability to handle knowledge depends sharply on the temporal state of that knowledge relative to the model's training cutoff. To show this, the authors build EvoWiki, a continuously auto-updated benchmark that labels facts as stable, evolved, or uncharted by comparing Wikidata snapshots at three timestamps. Their measurements on two open models show closed-book accuracy of about 31% on stable single-hop facts but only about 7% on evolved facts and 11% on uncharted facts. They also find that retrieval-augmented generation helps single-hop questions but hurts multi-hop ones, while continual learning gives small and consistent gains, and combining the two improves performance further.

What carries the argument

The load-bearing object is the three-timestamp classification scheme (init-time, cutoff-time, current-time) applied to Wikidata/Wikipedia snapshots. Facts are labeled stable if unchanged across all three snapshots, evolved if they existed before the cutoff but changed afterward, and uncharted if they first appear after the cutoff. This scheme converts the vague notion of 'knowledge freshness' into discrete, automatically updatable categories that anchor every subsequent measurement, including retrieval and continual-learning comparisons.

What would settle it

For a sample of facts labeled evolved, check whether the model already knew the post-change answer before the change was published—for instance, by querying an earlier model checkpoint trained strictly before January 2024 and seeing if it still produces the new answer at high confidence. Alternatively, re-running the full evaluation using each model's stated or estimated per-model cutoff instead of the fixed January 2024 date and showing that the stable/evolved/uncharted accuracy gaps largely disappear would directly disprove the claim that the split measures knowledge evolution rather than cutoff misalignment.

Watch

Extended reading notes

Core claim

EvoWiki's central claim is that knowledge evolution can be measured as a three-way split defined by a fixed knowledge cutoff date: stable facts (unchanged from before the cutoff to the present), evolved facts (changed after the cutoff), and uncharted facts (introduced after the cutoff). Built from Wikidata triple histories and verified against Wikipedia text via distant supervision, the benchmark yields a reproducible measurement protocol with referenced context, multi-hop questions, and popularity attributes. Using this protocol, the paper demonstrates that Llama-3.1-8B answers stable single-hop facts correctly in 31.61% of cases but only 6.96% of evolved facts and 10.84% of uncharted facts, that RAG recovers much of the gap on single-hop but degrades multi-hop accuracy, and that continual learning combined with RAG produces a synergistic effect.

Load-bearing premise

The entire benchmark depends on a single global cutoff date of January 2024 being an accurate representation of each evaluated model's actual training-knowledge boundary; if a model's real cutoff differs (as the paper itself notes for Mistral and Llama-3.1), facts labeled evolved or uncharted may already be in the model's weights, and the reported gaps would be distorted.

Editorial extensions

If this is right

  • Static benchmarks likely overstate what current LLMs know about the world after their training cutoff.
  • Retrieval augmentation should not be assumed to fix multi-hop questions: noisy retrieval can actively hurt performance on stable knowledge.
  • Continual learning offers small but consistent gains and, when combined with retrieval, yields better adaptation than either alone.
  • Because EvoWiki is auto-updatable, it can be re-run as new models and new Wikidata changes appear, producing a moving picture of knowledge adaptation.
  • Popularity should be a factor in designing continual-learning training data: rare new facts may be learned better than popular ones when using supervised fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of this result would be to replace the fixed January 2024 cutoff with per-model estimated cutoffs in future runs, which would clarify whether the stable/evolved/uncharted split is truly about knowledge evolution or partly about cutoff misalignment.
  • The finding that RAG hurts multi-hop accuracy on stable facts suggests that adding a separate reasoning or verification step to retrieval pipelines should recover the lost multi-hop performance.
  • Because EvoWiki is auto-updatable, it could be used to track whether the gap between stable and evolved knowledge closes across successive model releases, offering a public signal of progress on knowledge updating.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces EvoWiki, a QA benchmark constructed from Wikidata and Wikipedia snapshots at three timestamps (September 2021, January 2024, May 2024) and labels facts as stable, evolved, or uncharted relative to a global knowledge cutoff. The benchmark adds referenced contexts, multi-hop questions, and popularity attributes, and the paper evaluates Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 under closed-book, open-book, BM25/Contriever retrieval, continual pretraining/SFT, and combinations. The main empirical findings are that models score markedly higher on stable than on evolved/uncharted facts in closed-book QA, that RAG strongly improves single-hop but not multi-hop questions, that continual learning gives modest gains, and that combining RAG and CL is claimed to be synergistic.

Significance. EvoWiki addresses a real need: most QA benchmarks are static, and an updatable benchmark with explicit stable/evolved/uncharted labels and contextual evidence is a useful infrastructure contribution. The dataset construction is described concretely, the classification rule is stated from Wikidata snapshots and is independent of the evaluated models, and human evaluation provides some quality assurance. If the temporal labels are validated, the large closed-book gaps in Table 4 would be a valuable quantitative demonstration of LLMs' difficulty with post-training knowledge changes. The main empirical pattern is consistent across two models and several configurations. However, the paper's central interpretation is load-bearing on the temporal classification, and the current operational definitions and cutoff choices have not been shown to support it; the experimental claims also lack uncertainty quantification.

major comments (4)
  1. [Section 3.1, Appendix B] The definition of evolved facts in Section 3.1 ('facts that are established before init-time and exhibit changes between cutoff-time (or init-time) and current-time') and the implementation in Appendix B ('determine whether the triple has changed at cutoff-time or current-time') classify any fact that changed between September 2021 and January 2024 as evolved, even if the change occurred well before the January 2024 cutoff. This contradicts the preceding sentence in Section 3.1 that evolved facts 'were established before the cut-off date but have undergone changes since.' As a result, some 'evolved' facts may already be present in the model's training data, and the closed-book gaps reported in Table 4 conflate pre-cutoff updates with post-cutoff knowledge adaptation. Please restrict the evolved class to changes occurring after the cutoff (or explicitly redefine and justify the init-to-current criterion).
  2. [Section 3.1, Appendix A, Table 4] Section 3.1 fixes one global cutoff (January 2024) for both models, but Appendix A does not supply a verified training boundary for either evaluated model: the Llama entry refers to Llama 3 7B/70B, not the evaluated Llama-3.1-8B-Instruct, and Mistral is listed as having no official cutoff. Because every stable/evolved/uncharted label and the 'contamination-free' framing is computed relative to this single date, the headline Llama single-hop closed-book gaps (31.61% vs 6.96% and 10.84% in Table 4) are only meaningful if the cutoff is correct for each model. Please provide per-model evidence of the knowledge boundary, use model-specific timestamps, or report a sensitivity analysis over cutoff dates showing that the label distribution and main results are stable.
  3. [Section 4.1, Table 4, Section 4.5] All experimental numbers in Table 4 appear to be single runs without error bars, confidence intervals, or significance tests, yet the paper draws conclusions from small differences, e.g., 'modest yet consistent' CL gains of +1.87 pp and +1.57 pp on Llama single-hop evolved facts. The Section 4.5 'synergistic effect' is also asserted from point comparisons; for Llama multi-hop stable facts, SFT+Contriever (24.02) is actually below SFT closed-book (24.41), so the combination is not uniformly better than its best component. Please report multiple seeds or resampling-based intervals, and define and test the synergy claim against both RAG-only and CL-only baselines.
  4. [Appendix C, Section 4.4] The 552K SFT question-answer pairs (Appendix C) are generated from the same 15K Wikipedia chunks that define the golden contexts and the RAG corpus, yet the paper reports no overlap analysis between these training pairs and the EvoWiki evaluation questions. If the SFT data contains exact or near-duplicate questions and answers, the closed-book gains after SFT in Table 4 and the apparent RAG+CL synergy could reflect memorization of the evaluation answers rather than generalizable knowledge updating. Please quantify n-gram/embedding overlap between the SFT data and the test questions, remove overlapping examples, and confirm the conclusions on a disjoint evaluation split.
minor comments (7)
  1. [Abstract] The abstract contains the typo 'Contunual Learning'; it should read 'Continual Learning'.
  2. [Appendix C] Appendix C contains several typographical and grammatical errors: 'pre-tranining' should be 'pre-training', 'implementate' should be 'implemented', and 'Each document of Wikipedia are splited' is ungrammatical.
  3. [Appendix E] Appendix E.2 has 'descirption' for 'description', and Appendix E.5 has 'studen' for 'student'.
  4. [Table 4] The outdated-answer scores are typeset inconsistently, e.g., '48.99(72.70)' without a space next to '36.13 (53.78)' with a space; use a uniform format.
  5. [Appendix A] Appendix A should state explicitly how the listed Llama 3 cutoff dates relate to the evaluated Llama-3.1-8B-Instruct, or label the entry as not directly applicable to that model.
  6. [Figures 3 and 5] Figures 3 and 5 do not report the number of runs or any measure of variability; please either add error bars or explicitly state that these are single-run point estimates.
  7. [Introduction] The Introduction refers to 'NaturalQuestion'; the correct benchmark name is 'Natural Questions'.

Circularity Check

1 steps flagged · score 4.0 of 10

Benchmark labels are externally derived from Wikidata snapshots, but CL/RAG findings are partially circular because SFT data is generated from the same 15K Wikipedia corpus used as golden contexts.

  1. fitted input called prediction [Section 4.1 (Experimental Settings); Appendix C (Implementation Details of Continual Learning); Table 4 CL rows]
    ""The corpus is built from a 15K Wikipedia dump of golden documents" ... "For supervised fine-tuning, we first generate the SFT data with Meta-Llama-3.1-8B-Instruct. Each document of Wikipedia are splited into multiple chunks with a maximum 512 tokens. Then we prompt the model to generate 6 questions for each chunk. We finally get 552K question-answer pairs as the SFT data." ... "In open-book QA, models are provided with a golden context, a concise yet informative sentence extracted from Wikipedia (Section 3.2), ensuring minimal noise and optimal contextual support.""

    The SFT data used to fine-tune CL models is generated from the same 15K Wikipedia documents that define EvoWiki's golden contexts and fact triples; the prompt explicitly asks Llama to write questions 'according to the given context' for chunks of those documents. Open-book and RAG evaluation then provides golden context sentences extracted from the same pages, and closed-book evaluation asks questions built from those same triples. Therefore the reported CL gains and the claimed 'synergistic effect' of RAG+CL are, to an unknown degree, a measure of memorizing the evaluation corpus rather than of adapting to genuinely new knowledge. The EvoWiki evolution labels themselves come from Wikidata snapshot comparisons and are not circular.

full rationale

The stable/evolved/uncharted partition is defined by explicit snapshot rules (init-time September 2021, cutoff-time January 2024, current-time May 2024) applied to Wikidata/Wikipedia, independent of model outputs; this is self-contained and not circular. The global-cutoff assumption is an empirical limitation, not circularity. The one concrete circular step is the continual-learning evaluation: SFT question-answer pairs are generated from the same 15K Wikipedia dump that supplies golden contexts, so CL improvements and the RAG+CL synergy claim are partially forced by training on the evaluation corpus. Since the central benchmark construction remains independent and no fitted parameter is renamed as a prediction, the overall score is 4 rather than higher.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All evolution labels and performance comparisons rest on domain assumptions about Wikidata and Wikipedia fidelity, per-model cutoff alignment, distant supervision adequacy, and exact-match scoring. The paper introduces no formal axioms or new physical entities; the only hand-set quantities are the three timestamps, retrieval top-k, and max-hop count.

free parameters (4)
  • Global timestamps (init, cutoff, current) = 2021-09 / 2024-01 / 2024-05
    Chosen by hand in Section 3.1 to align with 'popular LLMs'; every evolution label and all downstream comparisons depend on these dates.
  • Retrieval top-k = 15
    Section 4.4 and Figure 3 use top-15 chunks; the paper notes performance flattens beyond 15, but the value is a design choice, not a fitted constant.
  • Max hops = 3
    Multi-hop questions are extended up to three hops in Section 3.2; the cutoff is arbitrary and the paper admits 3-hop items can degenerate into 2-hop.
  • Questions per Wikipedia chunk for SFT data = 6
    Section C stipulates 6 generated questions per chunk, yielding 552K pairs; this choice affects CL results but is not justified by an analysis.
assumptions (4)
  • domain assumption Wikidata and Wikipedia snapshots accurately represent real-world facts and their temporal changes.
    Section 3.1 classifies stable/evolved/uncharted solely from Wikidata and Wikipedia diffs; the Limitations section admits these sources contain noise.
  • domain assumption Public model-card cutoff dates delimit each model's training knowledge.
    Appendix A lists cutoff dates; Section 3.1 sets a single global cutoff-time of January 2024 for all evaluated models, though Mistral has no official cutoff.
  • domain assumption Distant supervision via explicit mention in Wikipedia confirms that a fact is answerable from the referenced context.
    Section 3.2 retains only triples whose object or subject is explicitly mentioned in the corresponding Wikipedia page; human evaluation covers only 180 samples.
  • domain assumption Exact-match scoring with the latest answer as gold is a valid measure of knowledge utilization.
    Section 4.1 uses exact match; evolved facts are scored correct only with the newest answer, and outdated answers are counted separately in parentheses.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvoWiki: Evaluating LLMs on Evolving Knowledge." pith.science (2026). https://pith.science/paper/I2B3SJCR

@misc{pith2026241213582,
  author       = {Pith},
  title        = {Pith review of: EvoWiki: Evaluating LLMs on Evolving Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I2B3SJCR}},
  note         = {Machine review of arXiv:2412.13582}
}
read the original abstract

Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updatable, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Contunual Learning (CL), we evaluate how effectively LLMs adapt to evolving knowledge. Our results indicate that current models often struggle with evolved knowledge, frequently providing outdated or incorrect responses. Moreover, the dataset highlights a synergistic effect between RAG and CL, demonstrating their potential to better adapt to evolving knowledge. EvoWiki provides a robust benchmark for advancing future research on the knowledge evolution capabilities of large language models.

Figures

Figures reproduced from arXiv: 2412.13582 by the authors.

Figure 1
Figure 1. EvoWiki categorizes knowledge into three [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Evolution level identification process. 3 EvoWiki Dataset In this section, we outline the construction process of the EvoWiki dataset, which integrates several features, such as knowledge evolution levels, ref￾erenced context, multi-hop reasoning capabilities, and popularity attributes. We identify facts at var￾ious stages of evolution by comparing different temporal versions of English Wikidata2 (referred to as Wik… view at source ↗
Figure 3
Figure 3. RAG performance across top-k values of Contriever; the dashed line represents closed-book QA results. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Probability shift (%) of CL methods on Llama [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Popularity effects of SFT on Llama. Due to [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ScienceMeter: Tracking Scientific Knowledge Updates in Language Models

    cs.CL 2025-05 reject novelty 6.0 of 10

    ScienceMeter evaluates language model knowledge updates across three axes, preservation of old scientific claims, acquisition of new claims, and projection to future findings, and finds all current methods fall short.

  2. Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

    cs.CL 2025-04 conditional novelty 4.0 of 10

    A survey that frames the core problem of LLM evaluation as 'evaluation generalization': finite test sets cannot scale with unbounded model capabilities, and proposes two transitions in evaluation design.

Reference graph

Works this paper leans on

46 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. https://openreview.net/forum?id=9-LSfSU74n- A dataset for answering time-sensitive questions . In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    work page 2021

  4. [4]

    Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2023. Lift yourself up: Retrieval-augmented text generation with self memory. arXiv preprint arXiv:2305.02437

    arXiv 2023

  5. [5]

    Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W

    Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2021. https://api.semanticscholar.org/CorpusID:235669861 Time-aware language models as temporal knowledge bases . Transactions of the Association for Computational Linguistics, 10:257--273

    work page 2021

  6. [6]

    White, Aaron Schein, and Ryan Cotterell

    Kevin Du, Vésteinn Snæbjarnarson, Niklas Stoehr, Jennifer C. White, Aaron Schein, and Ryan Cotterell. 2024. https://arxiv.org/abs/2404.04633 Context versus prior knowledge in language models . Preprint, arXiv:2404.04633

    arXiv 2024

  7. [7]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    arXiv 2023

  8. [8]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    arXiv 2021

Show all 46 references
  1. [9]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. https://arxiv.org/abs/2112.09118 Unsupervised dense information retrieval with contrastive learning . Preprint, arXiv:2112.09118

    2022 arXiv

  2. [10]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. https://arxiv.org/abs/2403.07974 Livecodebench: Holistic and contamination free evaluation of large language models for code . Preprint, a...

    2024 arXiv

  3. [11]

    Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022 a . https://api.semanticscholar.org/CorpusID:248476156 Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models . In Conferen...

    2022

  4. [12]

    Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo. 2022 b . https://openreview.net/forum?id=vfsRB5MImo9 Towards continual knowledge learning of language models . In International Conference on Learning Repres...

    2022

  5. [13]

    Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Str\" o tgen, and Gerhard Weikum. 2018 a . https://doi.org/10.1145/3184558.3191536 Tempquestions: A benchmark for temporal question answering . In Companion Proceedings of the The Web Conference 2018, WWW '18, page 1057–...

    2018

  6. [14]

    Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Str\" o tgen, and Gerhard Weikum. 2018 b . https://doi.org/10.1145/3269206.3269247 Tequila: Temporal question answering over knowledge bases . In Proceedings of the 27th ACM International Conference on Information and Kn...

    2018

  7. [15]

    Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021. https://doi.org/10.1145/3459637.3482416 Complex temporal question answering on knowledge graphs . In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, ...

    2021

  8. [16]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736

    2023 arXiv

  9. [17]

    Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, and Srinivasan Iyer. 2024. Instruction-tuned Language Models are Better Knowledge Learners . In Annual Meeting of the Association for Computational Linguistics...

    2024 arXiv

  10. [18]

    Smith, Yejin Choi, and Kentaro Inui

    Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. https://openreview.net/forum?id=HfKOIPCvsv Realtime QA : What's the answer right now? In Thirty-seventh Conferenc...

    2023

  11. [19]

    Diederik P Kingma. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  12. [20]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  13. [21]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  14. [22]

    Xinze Li, Zhenghao Liu, Chenyan Xiong, Shi Yu, Yu Gu, Zhiyuan Liu, and Ge Yu. 2023. Structure-aware language model pretraining improves dense retrieval on structured data. arXiv preprint arXiv:2305.19912

    2023 arXiv

  15. [23]

    Ron Litman, Oron Anschel, Shahar Tsiper, Roee Litman, Shai Mazor, and R Manmatha. 2020. Scatter: selective context attentional scene text recognizer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11962--11972

    2020

  16. [24]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  17. [25]

    Sara Vera Marjanovi'c, Haeun Yu, Pepa Atanasova, Maria Maistro, Christina Lioma, and Isabelle Augenstein. 2024. Dynamicqa: Tracing Internal Knowledge Conflicts in Language Models . In Conference on Empirical Methods in Natural Language Processing , volume abs/2407.17023, pages...

    2024 arXiv

  18. [26]

    Jannat Ara Meem, Muhammad Shihab Rashid, Yue Dong, and Vagelis Hristidis. 2024. https://api.semanticscholar.org/CorpusID:267750097 Pat-questions: A self-updating benchmark for present-anchored temporal question-answering . In Annual Meeting of the Association for Computational...

    2024

  19. [27]

    Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. https://arxiv.org/abs/2404.08700 Dyknow: Dynamically verifying time-sensitive factual knowledge in llms . Preprint, arXiv:2404.08700

    2024 arXiv

  20. [28]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  21. [29]

    Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Found. Trends Inf. Retr., 3(4):333–389

    2009 doi

  22. [30]

    Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. https://doi.org/10.18653/v1/2021.acl-long.520 Question answering over temporal knowledge graphs . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International...

    2021 doi

  23. [31]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://arxiv.org/abs/2303.11366 Reflexion: Language agents with verbal reinforcement learning . Preprint, arXiv:2303.11366

    2023 arXiv

  24. [32]

    Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.acl-long.828 Towards benchmarking and improving the temporal reasoning capability of large language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023 doi

  25. [33]

    Wei Tang, Yixin Cao, Jiahao Ying, Bo Wang, Yuyue Zhao, Yong Liao, and Peng Zhou. 2024. https://doi.org/10.18653/v1/2024.findings-acl.219 A + B : A general generator-reader framework for optimizing LLM s to unleash synergy potential . In Findings of the Association for Computat...

    2024 doi

  26. [34]

    Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023. https://arxiv.org/abs/2310.08118 Can large language models really improve by self-critiquing their own plans? Preprint, arXiv:2310.08118

    2023 arXiv

  27. [35]

    Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023. Self-knowledge guided retrieval augmentation for large language models. arXiv preprint arXiv:2310.05002

    2023 arXiv

  28. [36]

    Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.100 M enat QA : A new dataset for testing the temporal comprehension and reasoning abilities of large language models . In Fi...

    2023 doi

  29. [37]

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2024. https://arxiv.org/abs/2406.19314 Livebench:...

    2024 arXiv

  30. [38]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 arXiv

  31. [39]

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. https://openreview.net/forum?id=auKAUJZMO6 Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts . In The Twelfth International Conference on Learning Representations

    2024

  32. [40]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025

    2023 arXiv

  33. [41]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  34. [42]

    Jiahao Ying, Yixin Cao, Yushi Bai, Qianru Sun, Bo Wang, Wei Tang, Zhaojun Ding, Yizhe Yang, Xuanjing Huang, and Shuicheng Yan. 2024 a . https://arxiv.org/abs/2402.11894 Automating dataset updates towards reliable and timely evaluation of large language models . Preprint, arXiv...

    2024 arXiv

  35. [43]

    Jiahao Ying, Yixin Cao, Kai Xiong, Long Cui, Yidong He, and Yongbin Liu. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.232 Intuitive or dependent? investigating LLM s ' behavior style to conflicting prompts . In Proceedings of the 62nd Annual Meeting of the Association fo...

    2024 doi

  36. [44]

    Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063

    2022 arXiv

  37. [45]

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024 a . When Scaling Meets LLM Finetuning : The Effect of Data , Model and Finetuning Method . In The Twelfth International Conference on Learning Representations

    2024

  38. [46]

    Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Yipeng Zhang, Haitao Mi, and Helen Meng. 2024 b . https://arxiv.org/abs/2406.06326 Self-tuning: Instructing llms to effectively acquire new knowledge through self-teaching . Preprint, arXiv:2406.06326

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.