Pith. sign in

REVIEW 2 major objections 6 minor 50 references

Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations

T0 review · 2 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Even the most robust language model tested, gemma-2-27b-it, assigns the highest probability to the correct temporal context for only about 11 percent of 2,003 popular historical facts, and most models score below 3 percent.

desk verdict A solid, transparent empirical study with a reusable dataset; the 11% headline is real but hostage to one GPT-4o question per fact, so treat it as provisional. read the letter →

arxiv 2502.01220 v6 pith:FK3VP34F submitted 2025-02-03 cs.CL cs.LG

Hichem Ammar Khodja , Frédéric Béchet , Quentin Brabant , Alexis Nasr , Gwénolé Lecorvé This is my paper · ORCID
classification cs.CLcs.LG
keywords temporalrobustnessfactualknowledgelanguagemodelsTimeStresscontextprobingmetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether language models can tell when a fact is true. The authors build TimeStress, a dataset of 2,003 popular historical facts with natural-language statements paired against correct and incorrect temporal contexts at three granularities: year, month, and day. They test 18 models by asking, for each pair of statements, which one the model considers more probable. The central finding is that models have a coarse sense of time but no robust one: the best model, gemma-2-27b-it, prefers the correct context over every incorrect context for only about 11 percent of facts, and most models stay below 3 percent. If this holds, current language models cannot be trusted to answer when a fact held, even for facts they plainly know.

What carries the argument

The central object is the pairwise temporal-context comparison protocol instantiated by TimeStress. For each fact (s, r, o, [a, b]), the dataset constructs statements of the form 'In [tau], [question about s and r]? o' and samples many correct and incorrect contexts at year, year-month, and year-month-day granularity, with equal counts across granularities and a relative-distance measure alpha = (midpoint(tau) - midpoint([a, b])) / duration([a, b]). A model is said to distinguish a correct from an incorrect context when Pr_M(o | s, r, tau+) > Pr_M(o | s, r, tau-). These pairwise wins are aggregated into a per-fact win rate W(M, f), and robustness R(M, f) is the indicator that all comparisons are won; the dataset is large enough, with about 11 correct and 74 incorrect contexts per fact on average, that random guessing essentially never produces a robust fact. This protocol is what lets the paper separate coarse temporal sensitivity from reliable temporal binding.

What would settle it

Recompute the all-granularity robustness of gemma-2-27b-it on TimeStress using ten independently generated, human-verified questions per fact instead of one; if the aggregate robustness rises well above 11 percent, the low score is partly a verbalization artifact. A complementary check is to take 100 facts with manually verified validity intervals and see whether the same win-rate-by-distance curve and the persistent failures at relative distances of four or more still appear.

Watch

Extended reading notes

Core claim

The paper's central claim is that the factual knowledge stored in language models is only weakly bound to its validity period. Models assign higher probability to an answer when the stated date lies inside a fact's validity interval, and that probability decays as the date moves away, evidence of a genuine but coarse temporal representation. Yet robustness, defined as winning every correct-versus-incorrect comparison for a fact, is almost never achieved: gemma-2-27b-it, the most robust of the 18 models, scores 11 percent when all granularities are combined and 17 percent for years alone, while nearly all other models stay under 3 percent. The failures are not confined to dates near the boundaries of the validity period; even facts with a 95 percent win rate are sometimes defeated by dates very far away, with errors persisting at relative distances of four validity periods or more. The paper also shows that knowing a fact at year granularity does not reliably transfer to month or day granularity, with average transfer rates around 72 percent and no model achieving perfect transfer.

Load-bearing premise

The load-bearing premise is that the single automatically generated question assigned to each fact is a faithful and neutral expression of that fact and that the knowledge base's recorded validity window is complete, so a fact with a slightly off wording or an incomplete validity window can score zero robustness even when the model has correct temporal knowledge.

Editorial extensions

If this is right

  • Average win rates of 78 to 87 percent on the top five models show that language models are not temporally blind; the failure is specifically a lack of consistency across all comparisons.
  • Because robustness requires every comparison to be won, an all-or-nothing score near 11 percent means even the best model has only a fragile hold on when facts hold.
  • Instruction-tuned models usually beat their base versions, but reformatting statements as chat instructions does not reliably improve robustness and can degrade it for some models.
  • Vulnerability to easy incorrect contexts persists at relative distances of four validity periods or more even when a fact's win rate is 99 percent, so the errors are not just boundary effects.
  • Knowledge at one granularity transfers to another in roughly 72 percent of cases on average, with coarse-to-fine transfers failing more often, and adding explanatory prompts about date hierarchies improves this only slightly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because each fact is assigned a single randomly chosen question, the 11 percent figure conflates temporal robustness with verbalization robustness; evaluating the same facts with several human-verified phrasings would separate the two and could move the number either way.
  • The pairwise-preference design measures what a model would choose under forced comparison, not what it would volunteer in open generation, so applications that let a model abstain or retrieve a timestamp may behave better than these numbers suggest.
  • A natural next probe is to shift the temporal phrase from the beginning of the statement to the end, or embed it in a relative clause, to test whether the observed failures are tied to the 'In [year],' surface pattern.
  • If temporal binding is this weak for popular facts, temporal knowledge editing should target validity intervals rather than only object replacement, and evaluation should include the granularity-transfer and far-date failure modes introduced here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces TimeStress, a dataset of 2,003 past Wikidata facts with a total of 521,000 natural-language statements, and uses it to evaluate 18 language models on the task of preferring correct temporal contexts over incorrect ones. The evaluation defines a win rate W(M,f) and an all-or-nothing robustness metric R(M,f), and analyzes performance along two dimensions: the distance of an incorrect context from the fact's validity period, and the granularity of the context (year, year-month, year-month-day). The central empirical claim is that the best model, gemma-2-27b-it, achieves a global robustness of only about 11% of facts, meaning it prefers every correct context to every incorrect context for only a small fraction of the dataset. The paper further reports that LMs are vulnerable to 'easy' incorrect contexts far from the validity period even on facts with near-perfect win rates, and that knowledge does not fully transfer between granularities.

Significance. The paper makes a useful empirical contribution: it provides a publicly released, temporally annotated dataset with careful experimental detail, evaluates a diverse set of 18 open models, reports bootstrap confidence intervals, and gives a transparent account of the probability-based evaluation protocol in Appendix D. The headline result—that even the strongest model is robust on only about 11% of facts—is striking and, if stable, would be a meaningful data point for the community's understanding of temporal knowledge in LMs. The anomaly analysis (e.g., failures on distant incorrect contexts despite high win rates) is a falsifiable finding that can inform future benchmarks. However, the central number inherits a nontrivial sensitivity to the single verbalization used per fact, so the strength of the claim currently outruns the evidence.

major comments (2)
  1. [§3.2.2 and §3.1] The headline robustness figure is confounded with a single arbitrarily chosen question per fact. Section 3.2.2 states that 'each fact is randomly assigned one of its four associated questions,' while Section 3.1 defines R(M,f)=1 only when every correct-context statement beats every incorrect-context statement for that fact. With an average of 11 correct and 74 incorrect contexts, one fact contributes to the 11% only if the model wins roughly 800 pairwise comparisons under one fixed verbalization. The Appendix A.3 QA (1 error in 50 questions; Wilson CI [0.85,0.99]) verifies grammaticality and answerability, but not that the selected question is neutral across temporal contexts or that it does not inflate the probability of the object for dates outside the validity period. A single question with mild lexical ambiguity, an implicit default date, or a relation whose object is plausible in many years can zero out an entire fact even if the model knows the validity period perfectly. The authors should report robustness computed per question (e.g., the average of R over the four generated questions per fact, or the distribution across questions) to show that the 11% is not an artifact of this random selection.
  2. [§4.2, Figure 6] The granularity-transfer analysis inherits the same single-question sensitivity. The 'known' definition used in Figure 6 is R(M,f)=1 for a given granularity, and the claim that LMs fail to generalize across granularities in 28% of cases is based on this all-or-nothing condition. Since R=1 requires winning all comparisons under the one assigned question for that fact, a fact that is robust under the selected question need not be robust under another of the four generated questions. The observed transfer rates (e.g., 74±5% for gemma-2-27b-it from Y to YM) could therefore partly reflect question-level noise rather than a genuine inability to propagate temporal knowledge across granularities. The authors should recompute the transfer matrices using an aggregate over the four questions (majority voting or average R) to demonstrate that the reported gap is robust to verbalization choice.
minor comments (6)
  1. [Figure 1 caption] The caption contains the garbled fragment 'Thetrophymeans'; the intended symbol is not rendered. Please clarify the caption and ensure the figure displays the checkmark/cross symbols correctly.
  2. [Table 4 reference] The text refers to 'Table 4' for the vulnerability-to-easy-contexts analysis, but the displayed table is labeled 'Figure 4.' Update the cross-reference and the surrounding citations for consistency.
  3. [Appendix C.3, Figure 10] The labels in Figure 10 use the French word 'connu' (e.g., 'P(X connu | Y connu)') while the rest of the paper uses 'known.' Please localize these labels to English.
  4. [References] The Gemma Team reference lists 'et el' instead of 'et al.'; correct this typo.
  5. [Appendix C.1 and Figure 8] The paper uses 'MLs' (e.g., 'the 5 most robust MLs') in several places; these should be 'LMs' to match the terminology used throughout the manuscript.
  6. [Abstract and §4.2] The phrase 'errors, certainly rare, but critical that humans would not make' is grammatically ambiguous; consider rewriting, e.g., 'errors that are rare but critical, and that humans would not make.'

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the 11% robustness result is an empirical measurement on LM probabilities; the only self-citation is a non-load-bearing dataset-construction source.

full rationale

The paper is an empirical benchmark rather than a derivation. The central claim, that the best LM attains R = 11% over TimeStress facts, is computed from held-out LM conditional probabilities over independently sampled temporal contexts; no parameter is fitted to the target result and no equation reduces the reported robustness to its own input. The only self-citation in the construction chain is Section 3.2.1: 'The quintuplet collection process begins with a preprocessed version of Wikidata provided in Ammar Khodja et al. (2025).' That prior work supplies a Wikidata snapshot and a popularity measure based on Wikipedia page views. These are external inputs used to select facts and do not determine the measured robustness values. The adapted GPT-4o verbalization prompt (Appendix A.2) is also sourced from the authors' prior work, but its output is manually QA-checked and the robustness scores do not depend on any claimed theorem from that work. The manual QA of 50 questions is a dataset-validity limitation, not a circular reduction. The abstract's '11%' is simply the paper's own defined all-or-nothing metric R(M,f)=1[W(M,f)=1], so reporting that number is a definitional recap rather than a derivation. No 'prediction' is equivalent to an input by construction, and no load-bearing argument reduces to an unverified self-citation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or formal entities are postulated. The dataset construction uses hand-chosen thresholds and sampling parameters, and the evaluation rests on three domain assumptions: Wikidata validity periods are accurate, GPT-4o-generated questions are faithful verbalizations, and conditional probabilities of the object given the statement are a valid probe of knowledge. The popularity threshold and sampling window affect which facts and how many comparisons enter the robustness metric, so they are listed as free parameters even though they are not fitted to the target result.

free parameters (3)
  • popularity threshold = 90,000 median monthly Wikipedia visits (geometric mean of subject and object popularity)
    Chosen by iteratively lowering from 150,000 in 10,000 steps until more than 2,000 facts were kept (Appendix A.1). It determines which facts enter TimeStress and, because popularity correlates with robustness, sets an upper bound on measured performance.
  • temporal sampling window = m ± 5d with step 0.05d
    Arbitrary design choice limiting each fact to about 21 correct and 180 incorrect year contexts (Appendix A.4). The size of the window and sampling density directly affect the number of matches and therefore the all-or-nothing robustness score.
  • minimum validity duration = 3 years
    Introduced to guarantee at least one fully correct year-level context per fact (Section 3.2.1). Longer facts are over-represented, and the paper later finds duration correlates with win rate.
assumptions (4)
  • domain assumption Wikidata validity periods [a,b] for each (s,r,o) correctly bound the times when the statement is true.
    All correct/incorrect labels and the robustness metric depend on these intervals (Section 3.1). The paper removes only facts with two recorded validity periods, not unrecorded ones.
  • domain assumption The conditional probability Pr_M(o | s, r, tau) is a valid probe of the LM's factual knowledge of (s,r,o).
    The entire win-rate and robustness protocol compares these probabilities; the paper defends the choice in Section 5 and Limitations but does not establish it empirically beyond citing prior correlations.
  • domain assumption GPT-4o-generated questions faithfully verbalize each Wikidata fact without adding clues or ambiguity.
    Only 53 of 2,098 generation failures and 64 subject-as-answer errors were auto-removed; a random sample of 50 questions found 1 incorrect (Appendix A.3). The remaining 1,953 facts are assumed correct.
  • ad hoc to paper A single randomly chosen question per fact represents the fact well enough for all robustness tests.
    Each fact is assigned one of four GPT-4o questions (Section 3.2.2). Awkward wording can lower all win rates for that fact and is not controlled for.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations." pith.science (2026). https://pith.science/paper/FK3VP34F

@misc{pith2026250201220,
  author       = {Pith},
  title        = {Pith review of: Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FK3VP34F}},
  note         = {Machine review of arXiv:2502.01220}
}
read the original abstract

This paper explores the robustness of language models (LMs) to variations in the temporal context within factual knowledge. It examines whether LMs can correctly associate a temporal context with a past fact valid over a defined period, by asking them to differentiate correct from incorrect contexts. The LMs' ability to distinguish is analyzed along two dimensions: the distance of the incorrect context from the validity period and the granularity of the context. To this end, a dataset called TimeStress is introduced, enabling the evaluation of 18 diverse LMs. Results reveal that the best LM achieves a perfect distinction for only 11% of the studied facts, with errors, certainly rare, but critical that humans would not make. This work highlights the limitations of current LMs in temporal representation.

Figures

Figures reproduced from arXiv: 2502.01220 by the authors.

Figure 1
Figure 1. The robustness of the LM on a fact is evaluated [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Average metrics on the TimeStress dataset for the 5 most robust models (95% CIs were determined using [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Evolution of log Pr(o|f, τ ) with respect to the relative distance α, averaged across all facts in TimeStress and all LMs, for granularity Y (Bootstrap 95% CIs). The number of points used to compute each bar is indicated above it. to contexts with a distance of |α| ≥ 1 in 19% of cases. This proportion decreases to 6% for |α| ≥ 3, which remains significant given the proximity of the win rate to 100% for the facts obs… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Proportion of incorrect dates favored over [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Average R across all granularities for facts in TimeStress based on the format of statements submitted to the models: raw text (blue) or instruction (orange). 95% CIs were determined using bootstrapping. Y YM YMD B Y YM YMD A 1 0.78± 0.02 0.77± 0.02 0.68± 0.02 1 0.83± …
Figure 6
Figure 6. Figure 6: Average success rate of knowledge transfer [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Proportion of incorrect contexts favored over correct contexts that are beyond a relative distance [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: For each α segment, the average local robustness correlation across all granularity pairs is calculated over all facts and the 5 most robust LMs. Y YM YMD B Y YM YMD A 1 0.84± 0.04 0.84± 0.05 0.74± 0.05 1 0.88± 0.04 0.68± 0.05 0.80± 0.05 1 P(A known | B known) 0.0 0.2 …
Figure 9
Figure 9. Figure 9: Generalization matrics between pairs of granularities on the 5 most robust LMs. In the [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Effect of adding explanations on temporal concepts through an explanatory prompt [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Relationship between the number of parameters in an LM and the metric used (across all granularities Y, [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Average metrics across all facts in TimeStress for the 18 studied LMs with 95% confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 14
Figure 14. Figure 14: Distribution of the the years of all the tem [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: The evolution of logP(o|f, τ ) with respect to the relative distance of the context from the validity period α. Each point is an average over many data points. 0.0 0.2 0.4 Average officeholder head coach head of government chairperson head of state presenter secretary…
Figure 17
Figure 17. Figure 17: Examples of vulnerability to easy incorrect contexts for different LMs. The color blue represents the boundaries of the validity period, the color green represents incorrect contexts that are never preferred to correct contexts, and the color red, on the contrary, rep…
Figure 18
Figure 18. Figure 18: The influence of two factors on the robustness and win rate of the 5 most robust LMs. All correlations [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hichem Ammar Khodja, Abderrahmane Ait gueni ssaid, Frederic Bechet, Quentin Brabant, Alexis Nasr, and Gw \'e nol \'e Lecorv \'e . 2025. https://aclanthology.org/2025.coling-main.537/ Factual knowledge assessment of language models using distractors . In Proceedings of the 31st International Conference on Computational Linguistics, pages 8043--8056, Abu Dh...

    work page 2025

  4. [4]

    Hichem Ammar Khodja, Fr \'e d \'e ric B \'e chet, Quentin Brabant, Alexis Nasr, and Gw \'e nol \'e Lecorv \'e . 2024. https://aclanthology.org/2024.lrec-main.1532/ W iki F act D iff: A large, realistic, and temporally adaptable dataset for atomic factual knowledge update in causal language models . In Proceedings of the 2024 Joint International Conference...

    work page 2024

  5. [5]

    Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/1f0e3dad99908345f7439f8ffabdffc4-Abstract-round2.html A dataset for answering time-sensitive questions . In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021...

    2021

  6. [6]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. https://doi.org/10.18653/v1/2024.acl-long.66 T ime B ench: A comprehensive evaluation of temporal reasoning abilities in large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    doi:10.18653/v1/2024.acl-long.66 2024

  7. [8]

    Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/5f0a4cd23e1c6eedd3edebba674ab877-Abstract-Conference.html Statistical knowledge assessment for large language models . In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing System...

    work page 2023

  8. [10]

    Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. https://aclanthology.org/L18-1544 T - RE x: A large scale alignment of natural language with knowledge base triples . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018) , M...

    2018

Show all 50 references
  1. [11]

    Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. 2025. https://openreview.net/forum?id=44CoQe6VCq Test of time: A benchmark for evaluating LLM s on temporal reasoning . In The Thirt...

    2025

  2. [12]

    Wee Chung Gan and Hwee Tou Ng. 2019. https://doi.org/10.18653/V1/P19-1610 Improving the robustness of question answering systems to question paraphrasing . In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 2...

    2019 doi

  3. [13]

    Xiou Ge, Ali Mousavi, Edouard Grave, Armand Joulin, Kun Qian, Benjamin Han, Mostafa Arefiyan, and Yunyao Li. 2024. https://aclanthology.org/2024.acl-short.53 Time sensitive knowledge editing through efficient finetuning . In Proceedings of the 62nd Annual Meeting of the Associ...

    2024

  4. [14]

    Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  5. [15]

    Tim Hagen, Harrisen Scells, and Martin Potthast. 2024. https://aclanthology.org/2024.findings-emnlp.248 Revisiting query variation robustness of transformer models . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, ...

    2024

  6. [16]

    Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/95b6e2ff961580e03c0a662a63a71812-Abstract-Conference.html Aging with GRACE: lifelong model editing with discrete key-value adaptors ....

    2023

  7. [17]

    Prachi Jain, Sushant Rathi, Mausam , and Soumen Chakrabarti. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.305 T emporal K nowledge B ase C ompletion: N ew A lgorithms and E valuation P rotocols . In Proceedings of the 2020 Conference on Empirical Methods in Natural Langua...

    2020 doi

  8. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  9. [20]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  10. [21]

    Jan - Christoph Kalo and Leandra Fichtel. 2022. https://akbc.ws/2022/papers/15\_kamel\_knowledge\_analysis\_with\_ KAMEL: knowledge analysis with multitoken entities in language models . In 4th Conference on Automated Knowledge Base Construction, AKBC 2022, London, UK, Novembe...

    2022

  11. [22]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. https://proceedings.mlr.press/v202/kandpal23a.html Large language models struggle to learn long-tail knowledge . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honol...

    2023

  12. [23]

    Cheongwoong Kang and Jaesik Choi. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.518 Impact of co-occurrence on factual knowledge of large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7721--7735, Singapore. Associatio...

    2023 doi

  13. [24]

    Smith, Yejin Choi, and Kentaro Inui

    Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/9941624ef7f867a502732b5154d30cb7-Abstract-Datasets\_and\_Benchmarks.htm...

    2023

  14. [25]

    Nora Kassner and Hinrich Sch \"u tze. 2020. https://doi.org/10.18653/v1/2020.acl-main.698 Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p...

    2020 doi

  15. [26]

    Yujin Kim, Jaehong Yoon, Seonghyeon Ye, Sangmin Bae, Namgyu Ho, Sung Ju Hwang, and Se - Young Yun. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.302 Carpe diem: On the evaluation of world knowledge in lifelong language models . In Proceedings of the 2024 Conference of the ...

    2024 doi

  16. [27]

    Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gim \'e nez, Cyprien de Masson d'Autume, Tom \'a s Kocisk \'y , Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. https://api.semanticscholar....

    2021

  17. [28]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  18. [29]

    Adam Liska, Tom \' a s Kocisk \' y , Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d'Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan - McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. https://proceedings...

    2022

  19. [30]

    Chenyang Lyu, Minghao Wu, and Alham Aji. 2024. https://doi.org/10.18653/v1/2024.knowllm-1.10 Beyond probabilities: Unveiling the misalignment in evaluating large language models . In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages...

    2024 doi

  20. [31]

    Katerina Margatina, Shuai Wang, Yogarshi Vyas, Neha Anna John, Yassine Benajiba, and Miguel Ballesteros. 2023. https://doi.org/10.18653/V1/2023.EACL-MAIN.211 Dynamic benchmarking of masked language models on temporal concept drift with multiple views . In Proceedings of the 17...

    2023 doi

  21. [32]

    Takumi Matsuno and Masatoshi Tsuchiya. 2023. https://doi.org/10.1109/ICAICTA59291.2023.10390252 Evaluating the robustness of question answering model against context variations . In 2023 10th International Conference on Advanced Informatics: Concept, Theory and Application (IC...

    2023

  22. [33]

    Sachin Mehta, Mohammad Sekhavat, Qingqing Cao, Max Horton, Yanzi Jin, Frank Sun, Iman Mirzadeh, Mahyar Najibikohnehshahri, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. https://arxiv.org/abs/2404.14619 Openelm: An efficient language model family with open trai...

    2024 arXiv

  23. [34]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html Locating and editing factual associations in GPT . In Advances in Neural Information Processing System...

    2022

  24. [35]

    Ishani Mondal and Abhilasha Sancheti. 2024. https://openreview.net/forum?id=cyN5Ck1RFT On the robustness of chatgpt under input perturbations for named entity recognition task . In The Second Tiny Papers Track at ICLR 2024, Tiny Papers @ ICLR 2024, Vienna, Austria, May 11, 202...

    2024

  25. [36]

    Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. https://aclanthology.org/2024.findings-emnlp.471 Dyknow: Dynamically verifying time-sensitive factual knowledge in llms . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida,...

    2024

  26. [37]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. https://doi.org/10.18653/v1/D19-1250 Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P...

    2019 doi

  27. [38]

    Harsh Raj, Domenic Rosati, and Subhabrata Majumdar. 2022. https://openreview.net/forum?id=SgbpddeEV-C Measuring reliability of large language models through semantic consistency . In NeurIPS ML Safety Workshop

    2022

  28. [39]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton - Brown, and Yoav Shoham. 2023. https://doi.org/10.1162/TACL\_A\_00605 In-context retrieval-augmented language models . Trans. Assoc. Comput. Linguistics, 11:1316--1331

    2023 doi

  29. [40]

    Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, and Min Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.703 Living in the moment: Can large language models grasp co-temporal reasoning? In Proceedings of the 62nd Annual Meeting of t...

    2024 doi

  30. [41]

    Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.18 Head-to-tail: How knowledgeable are large language models (llms)? A.K.A. will llms replace knowledge graphs? In Proceedings of the 2024 Conference of the North...

    2024 doi

  31. [42]

    Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. https://api.semanticscholar.org/CorpusID:259165281 Towards benchmarking and improving the temporal reasoning capability of large language models . In Annual Meeting of the Association for Computational Linguistics

    2023

  32. [43]

    Gemma Team, Morgane Riviere, and Shreya Pathak et el. 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . Preprint, arXiv:2408.00118

    2024 arXiv

  33. [44]

    Lukas Thede, Karsten Roth, Matthias Bethge, Zeynep Akata, and Tom Hartvigsen. 2025. https://arxiv.org/abs/2503.05683 Understanding the limits of lifelong knowledge editing in llms . Preprint, arXiv:2503.05683

    2025

  34. [45]

    Paulo Alting von Geusau and Peter Bloem. 2020. https://doi.org/10.1007/978-3-030-76640-5\_1 Evaluating the robustness of question-answering models to paraphrased questions . In Artificial Intelligence and Machine Learning - 32nd Benelux Conference, BNAIC/Benelearn 2020, Leiden...

    2020 doi

  35. [46]

    Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023. https://api.semanticscholar.org/CorpusID:263831019 Menatqa: A new dataset for testing the temporal comprehension and reasoning abilities of large language models . In Confer...

    2023

  36. [47]

    Jacek Wiland, Max Ploner, and Alan Akbik. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.155 BEAR: A unified framework for evaluating relational knowledge in causal and masked language models . In Findings of the Association for Computational Linguistics: NAACL 2024, Me...

    2024 doi

  37. [48]

    Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. https://doi.org/10.18653/v1/2024.acl-long.563 Large language models can learn temporal reasoning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2024 doi

  38. [50]

    Xunjian Yin, Jin Jiang, Liming Yang, and Xiaojun Wan. 2024 b . https://doi.org/10.1609/AAAI.V38I17.29912 History matters: Temporal knowledge editing in large language model . In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Inn...

    2024 doi

  39. [51]

    Lang Yu, Qin Chen, Jie Zhou, and Liang He. 2024. https://doi.org/10.1609/AAAI.V38I17.29916 MELO: enhancing model editing with neuron-indexed dynamic lora . In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Application...

    2024 doi

  40. [52]

    Michael Zhang and Eunsol Choi. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.586 S ituated QA : Incorporating extra-linguistic contexts into QA . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371--7387, Online and Punta C...

    2021 doi

  41. [53]

    Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.516 How do large language models capture the ever-changing world knowledge? a review of recent advances . In Proceedings of the 2023 Conference on Empir...

    2023 doi

  42. [54]

    Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hanna Hajishirzi, and Noah A. Smith. 2024. https://api.semanticscholar.org/CorpusID:268033102 Set the clock: Temporal alignment of pretrained language models . In Annual Meeting of the Association for Computational Linguistics

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.