Pith. sign in

REVIEW 3 major objections 4 minor 63 references

ScienceMeter: Tracking Scientific Knowledge Updates in Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No knowledge-update method can simultaneously preserve past scientific claims, acquire new ones, and project future ones; the best scores only 85.9%, 71.7%, and 37.7% on the three axes.

desk verdict ScienceMeter has a valuable dataset and a sensible preservation/acquisition framework, but its headline projection metric asks models to judge claim truth while scoring them on paper stance, so the 37.7% projection figure should not be taken at face value. read the letter →

arxiv 2505.24302 v3 pith:DQAN7I2D submitted 2025-05-30 cs.CL

Yike Wang , Shangbin Feng , Yulia Tsvetkov , Hannaneh Hajishirzi This is my paper
classification cs.CL
keywords knowledgeupdatescientificclaimspreservationacquisitionprojectionevaluationframeworkdomainvolatilitylargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a useful scientific knowledge update for an LLM must do three things at once: keep previously learned claims intact, absorb claims from newly added papers, and anticipate claims that will appear only in future papers. To test this, the authors build ScienceMeter, a ten-domain dataset of 15,444 papers and 30,888 atomic, verifiable claims organized into prior, new, and future literature by publication date, and define three metrics—knowledge preservation, acquisition, and projection. Evaluating five update methods on mid-size and frontier models, the paper finds that the best method reaches 85.9% preservation, 71.7% acquisition, and 37.7% projection, and that no method achieves all three objectives. The finding matters because it identifies future-knowledge projection as the unmet bottleneck for language models in scientific work, with consequences for their use in hypothesis generation and staying current.

What carries the argument

ScienceMeter is the load-bearing object: an evaluation framework that treats scientific knowledge as atomic, verifiable claims tied to papers and measures an update along three axes—preservation of prior claims, acquisition of new claims, and projection of future claims. Each claim is scored correct, incorrect, or unknown by combining factual accuracy with an estimated confidence, which yields a two-error decomposition into distortion (confident and wrong) and loss (low confidence). The dataset is built as citation-linked triplets of prior, new, and future papers across ten domains, and the headline numbers are the best scores any tested update method achieves on the three axes averaged over tasks and models.

What would settle it

Have expert scientists independently judge the truth of a sample of future claims without knowing which paper cites them, recompute the 37.7% projection figure with those expert labels, and see whether it moves materially; if it does, the projection metric tracks citation stance rather than future knowledge.

Watch

Extended reading notes

Core claim

The central claim is an empirical ceiling: today's knowledge-update methods cannot satisfy preservation, acquisition, and projection simultaneously. Based on claim judgment and generation over the ScienceMeter corpus, the best-performing method averaged across tasks and models preserves 85.9% of prior correct claims, acquires 71.7% of new claims, and projects 37.7% of future claims, and no method meets all three. The paper also reports that inference-time update methods work for a 32B-parameter model but not for an 8B model, which needs training-based updates; that cross-domain scores correlate with domain volatility rather than pretraining exposure; and that a domain-adapted scientific model still falls short. The authors conclude that projection, not acquisition, is the open challenge for keeping LLMs scientifically current.

Load-bearing premise

The projection score assumes that a future paper's support or refutation of a claim equals the claim's truth, even though a claim can be true without appearing in that paper or false while being supported by it.

Editorial extensions

If this is right

  • No robust scientific knowledge update method exists: any current approach sacrifices at least one of preservation, acquisition, or projection.
  • Projection is the hardest objective, so today's LLMs are not yet reliable for anticipating future findings, limiting their use in hypothesis generation and research idea formulation.
  • Model scale determines the right update strategy: inference-time injection works for large models, while small models require training-based updates to absorb new claims.
  • Domain volatility, not pretraining data availability, drives preservation and projection; fast-moving fields such as materials science and environmental science are where updates fail most.
  • Even domain-specialized scientific LLMs do not collectively satisfy the three objectives, so the problem is not solved by better pretraining alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The projection metric likely understates true projection ability: it scores a claim as 'projected' only if it matches the stance of one specific future paper, even though a true claim could be missing from that paper and a false claim could appear in it; the paper itself notes the true magnitude may be higher.
  • The temporal-triplet design can serve as a rolling benchmark: as today's 'future' papers become published, the same claims become verifiable, allowing head-to-head comparison of update methods over time without re-annotation.
  • The distortion-loss decomposition points to a concrete design goal for the next generation of update methods: add an abstention mechanism that refuses to modify the model on low-confidence content, which could raise preservation without hurting acquisition.
  • The scale-dependent result suggests a practical deployment rule: retrieval-based updating is a premium feature of large models, not a universal replacement for training in smaller deployed systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. ScienceMeter proposes an evaluation framework for scientific knowledge updates in LLMs, defining three metrics: knowledge preservation (retention of prior claims), knowledge acquisition (uptake of new claims), and knowledge projection (anticipation of future claims). The authors construct a dataset of 15,444 papers and 30,888 synthetic claims across 10 domains, arrange papers into prior/new/future trajectories by citation and publication dates, and evaluate five update methods on LLaMA-3.1-8B and OLMo2-32B using claim judgment and claim generation tasks. The main empirical claim is that no knowledge update method achieves all three objectives, with the best observed values being 85.9% preservation, 71.7% acquisition, and 37.7% projection. The paper also reports cross-domain correlations with citation counts and pretraining occurrence, and evaluates a domain-adapted scientific LLM in Materials Science.

Significance. If the framework is valid, this would be a useful and much-needed benchmark for evaluating scientific knowledge updates in LLMs, an area where most existing benchmarks focus on factoid or Wikipedia-style knowledge. The paper has clear strengths: code and data are linked; the dataset is large and multi-domain; synthetic claims receive expert validation; a comparison with author-annotated claims is reported; metric formulas are explicit; and a specialized scientific LLM is included in the evaluation. These are genuine contributions. The main obstacle is construct validity of the projection metric, which directly underpins the 37.7% headline number; until that is repaired or relabeled, the central claim is not supported as stated.

major comments (3)
  1. [2.2, Appendix D, Table 8] The projection metric does not measure what it is named, and this issue is load-bearing for the headline 37.7% figure. In Section 2.2, the future-knowledge task defines the ground-truth label y(c) as whether the associated future paper supports or refutes the claim, with no title or paper provided to the model. Appendix D, Table 8, however, prompts the model with 'Is the claim correct?' rather than 'Would the future paper support this claim?' Meanwhile, Section 2.1 defines REFUTE claims as 'relevant but non-supporting,' not as false. A claim can therefore be true and still labeled REFUTE if one particular future paper does not support it, and a claim can be false yet labeled SUPPORT if that paper endorses it. The resulting score measures how often the model predicts a single paper's stance, not how well it anticipates correct scientific knowledge. The paper's own caveat in Section 2.3 that 'the true magnitude of Knowledge Projection is likely higher' acknowledges label unreliability but does not fix the mismatch between the prompt, the label, and the construct. A concrete fix is to (i) revise the prompt and label so that the task is explicitly to predict whether the named future paper would support the claim, and rename the metric accordingly, or (ii) build future labels from genuinely confirmed or refuted claims using later consensus or human verification.
  2. [3.2] The claim-generation results in Table 2 rely on GPT-4o as both claim generator and factual-accuracy judge, but the paper does not report human validation of this judge. The synthetic-claim generation is validated by experts (Table 6) and the linguistic-confidence judge is validated against human raters (75.9%), but the statement in Section 3.2 that 'we assess the factual accuracy of the generated claim by inviting GPT-4o to determine whether y(ĉ,t)=SUPPORT' has no such check. Because the reference claims were also produced by GPT-4o, a systematic bias in the judge would directly affect the generation-task columns of Table 2, which are averaged into the headline preservation, acquisition, and projection numbers. Please add a human sample evaluation of the GPT-4o judge on generated claims, or report agreement statistics and failure cases.
  3. [2.3, Table 10] Knowledge Acquisition and Knowledge Projection are conditional on the claim being unknown to the model before the update. The formulas in Table 10 are explicit about this, but the abstract and Section 3.4 state that the best methods 'acquire 71.7% of new knowledge' and 'project 37.7% of future knowledge' without the conditional qualifier. Because the denominator excludes claims the model already answered correctly, a method that already knows many of the new or future claims can appear to have low acquisition or projection even if it updates correctly. The percentages are therefore not rates over all new or future claims, and the headline statement is potentially misleading. Please state the conditional interpretation prominently in the abstract and Section 3.4, or additionally report unconditional rates.
minor comments (4)
  1. [Appendix C.2 and D] The prompt templates contain grammatical errors, for example 'a atomic verifiable statements' in Tables 5, 8, and 9; these should be corrected before publication.
  2. [Table 2] The column headers 'PresDist LossAcquDist LossProjLoss' are compressed and hard to parse; the color-coding described in the text is not visible in a monochrome printout. Consider using separate labeled subcolumns or a clearer grouping.
  3. [4.1, Figure 4] The cross-domain correlation analysis is based on only 10 domains, but the paper reports Pearson r values without p-values or confidence intervals. Adding these, or a nonparametric alternative, would make the 'strong relationship' claim more credible.
  4. [3.3] The notation in the INFER method description, 'g(LMf(Ptestnew), p)=g(LM|pnew,p)', is confusing and should be rewritten with a cleaner definition of how the new paper is inserted into the prompt.

Circularity Check

0 steps flagged · score 2.0 of 10

ScienceMeter's headline 85.9/71.7/37.7 results are holdout measurements against externally sourced future-paper abstracts and human-validated claims, not reductions to fitted inputs; the projection metric's stance-versus-truth label mismatch is a disclosed construct-validity caveat rather than circularity.

full rationale

The three headline numbers are empirical aggregates, not derived quantities: Knowledge Preservation, Acquisition, and Projection are conditional-accuracy formulas in Table 10 evaluated on model responses against fixed external labels, and the p_future papers used for projection were published after the training cutoff (Table 4) and are never shown to the model at update time, so projection is a genuine forward holdout rather than a fitted input renamed as a prediction. Claim ground truths are generated by GPT-4o but are externally anchored: expert evaluation reports at least 80% strict and 95% broad compliance (Section 2.1, Table 6), author-annotated claims reproduce the results with no statistically significant difference (Section C.3, Table 7), and the GPT-4o factual-accuracy and linguistic-confidence judging (Section 3.2) is validated against three human evaluators at 75.9% agreement and combined with two rule-based methods by majority voting; because all five update methods are scored through the same pipeline, GPT-4o's involvement cannot force the comparative conclusion that no method satisfies all three objectives simultaneously. No fitted parameter is later re-reported as a prediction, and no uniqueness or ansatz result is imported from the authors' prior work. Same-lab citations appear but are not load-bearing: [10, 12] are provenance for the 'need more information' confidence probe, [58] only glosses correlation coefficients computed from data (r = -0.709 and -0.736 in Figure 4), and the SciFact inspiration [54] for atomic claims is external provenance with independent benchmarks. The strongest concern is construct validity, not circularity: the future-knowledge label y(c) is defined as whether the associated future paper supports or refutes the claim (Section 2.2), while the prompt asks "Is the claim correct?" (Table 8), so the 37.7% projection figure measures prediction of a specific paper's stance and may understate truth-oriented projection; the paper itself discloses this in Section 2.3 ("the true magnitude of Knowledge Projection is likely higher than our current estimate") and Appendix A ("Contradictory Claims"), which is an honest caveat about label reliability rather than a circular deduction. Accordingly, the derivation chain is self-contained against external benchmarks and no circular step is identified.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical entities or latent variables. The main extra assumptions are about how to turn raw papers into ground-truth claims, how to define future knowledge, and how to measure confidence. The most consequential is the conflation of claim correctness with paper stance in the future projection task.

free parameters (5)
  • claim length target = ~15 words
    Synthetic claims are constrained to approximately 15 words, affecting claim granularity and task difficulty.
  • three-month buffer = 3 months
    Used to account for discrepancies between online availability and official publication dates when assigning papers to prior, new, or future sets.
  • majority voting threshold = majority of 3 confidence methods
    Confidence is determined by majority voting among More Information, Consistency, and Linguistic Confidence, with no validation set to calibrate.
  • number of least frequent tokens = 100
    Domain-specific token proxy uses the 100 least frequent tokens in abstracts, an arbitrary choice that yields noisy token lists.
  • recent cutoff date = varies by model (e.g., 2024.12.1)
    Defines which citing papers count as future knowledge; chosen by the authors and varied across models.
assumptions (5)
  • domain assumption GPT-4o generates atomic, verifiable scientific claims from paper abstracts that can serve as ground truth.
    The dataset's quality rests on this; expert validation on 120 claims shows 80 to 95 percent adherence, but GPT-4o is both generator and evaluator, and the validation sample is small. See Section 2.1 and Appendix C.2.
  • ad hoc to paper A claim labeled REFUTE is 'relevant but not supported by the given paper', and this label is used as ground truth for correctness in future projection.
    For future knowledge, the prompt asks 'Is the claim correct?' but the label is whether the future paper supports or refutes the claim. This conflates truth with paper stance. See Section 2.2 and Appendix D Table 8.
  • ad hoc to paper Papers citing the same original paper, ordered by publication date, represent prior, new, and future knowledge trajectories.
    Citation relationships and date cutoffs are used as a proxy for scientific advancement, which the authors acknowledge is a simplification. See Section 2.1, Appendix C.1, and the Limitations section.
  • domain assumption Factual accuracy of generated claims can be judged by GPT-4o against paper abstracts.
    No human evaluation of generated claims is reported; GPT-4o is used as judge, which may introduce systematic bias. See Section 3.2.
  • domain assumption Model confidence can be measured by three rule-based methods and majority voting.
    The correct/incorrect/unknown categorization depends on this confidence estimate, which is not calibrated on a validation set. See Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScienceMeter: Tracking Scientific Knowledge Updates in Language Models." pith.science (2026). https://pith.science/paper/DQAN7I2D

@misc{pith2026250524302,
  author       = {Pith},
  title        = {Pith review of: ScienceMeter: Tracking Scientific Knowledge Updates in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQAN7I2D}},
  note         = {Machine review of arXiv:2505.24302}
}
read the original abstract

Large Language Models (LLMs) are increasingly used to support scientific research, but their knowledge of scientific advancements can quickly become outdated. We introduce ScienceMeter, a new framework for evaluating scientific knowledge update methods over scientific knowledge spanning the past, present, and future. ScienceMeter defines three metrics: knowledge preservation, the extent to which models' understanding of previously learned papers is preserved; knowledge acquisition, how well scientific claims from newly introduced papers are acquired; and knowledge projection, the ability of the updated model to anticipate or generalize to related scientific claims that may emerge in the future. Using ScienceMeter, we evaluate the scientific knowledge of LLMs through claim judgment and generation tasks on a curated dataset across ten domains. We evaluate five representative knowledge update approaches and find that the best-performing knowledge update methods can preserve only 85.9% of existing knowledge, acquire 71.7% of new knowledge, and project 37.7% of future knowledge, underscoring that developing robust scientific knowledge update mechanisms is both crucial and challenging.

Figures

Figures reproduced from arXiv: 2505.24302 by the authors.

Figure 1
Figure 1. We propose an evaluation framework, SCIENCEMETER, along with novel metrics to quantify the reliability and usefulness of scientific knowledge updates in LLMs: preservation of existing scientific claims and their linkage to existing literature, acquisition of new scientific claims from emerging research, and projection of future scientific claims. For example, when we update an LLM with a new paper introducing the ef… view at source ↗
Figure 2
Figure 2. An overview of SCIENCEMETER: (1) We curate chronologically organized datasets of scientific papers and claims across 10 rapidly evolving domains; (2) define claim judgment and generation tasks to evaluate scientific knowledge, incorporating both factual accuracy and model confidence; and (3) introduce metrics for evaluating scientific knowledge updates. 2 The SCIENCEMETER Evaluation Framework To systematically evalu… view at source ↗
Figure 3
Figure 3. Performance of Standard Instruction-tuning on [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The correlation between Preservation, Acquisition, Projection and average citation count, as well as pretraining occurrence. The strength of the correlation is reflected in how closely the data points cluster around the best-fit line [PITH_FULL_IMAGE:figures/full_fig_…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 11 canonical work pages

  1. [1]

    Litllm: A toolkit for scientific literature review

    Shubham Agarwal, Issam H Laradji, Laurent Charlin, and Christopher Pal. Litllm: A toolkit for scientific literature review. arXiv preprint arXiv:2402.01788, 2024

    arXiv 2024

  2. [2]

    The transformative impact of large language models on medical writing and publishing: current applications, challenges and future directions

    Sangzin Ahn. The transformative impact of large language models on medical writing and publishing: current applications, challenges and future directions. The Korean journal of physiology & pharmacology: official journal of the Korean Physiological Society and the Korean Society of Pharmacology, 2024

    work page 2024

  3. [3]

    The impact of large lan- guage models on scientific discovery: a preliminary study using gpt-4

    Microsoft Research AI4Science and Microsoft Azure Quantum. The impact of large lan- guage models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361, 2023

    arXiv 2023

  4. [4]

    Construction of the literature graph in semantic scholar

    Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et al. Construction of the literature graph in semantic scholar. arXiv preprint arXiv:1805.02262, 2018

    arXiv 2018

  5. [5]

    Openscholar: Synthesizing scientific literature with retrieval-augmented lms

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, et al. Openscholar: Synthesizing scientific literature with retrieval-augmented lms. arXiv preprint arXiv:2411.14199, 2024

    arXiv 2024

  6. [6]

    A general language assistant as a laboratory for alignment

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021

    arXiv 2021

  7. [7]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  8. [8]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

Show all 63 references
  1. [9]

    In praise of stubbornness: The case for cognitive-dissonance-aware knowledge updates in llms

    Simone Clemente, Zied Ben Houidi, Alexis Huet, Dario Rossi, Giulio Franzese, and Pietro Michiardi. In praise of stubbornness: The case for cognitive-dissonance-aware knowledge updates in llms. arXiv preprint arXiv:2502.04390, 2025

    2025 arXiv

  2. [10]

    Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models

    Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models. arXiv preprint arXiv:2305.09955, 2023

    2023 arXiv

  3. [11]

    When one llm drools, multi-llm collaboration rules

    Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, Tomas Pfister, Yejin Choi, and Yulia Tsvetkov. When one llm drools, multi-llm collaboration rules. arXiv preprint arXiv:2403.17852, 2024

    2024 arXiv

  4. [12]

    Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00367, 2024

    2024 arXiv

  5. [13]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 10

    2024 arXiv

  6. [14]

    Don’t stop pretraining: Adapt language models to domains and tasks

    Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020

    2004 arXiv

  7. [15]

    Knowledge updating? no more model editing! just selective contextual reasoning

    Guoxiu He, Xin Song, and Aixin Sun. Knowledge updating? no more model editing! just selective contextual reasoning. arXiv preprint arXiv:2503.05212, 2025

    2025 arXiv

  8. [16]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  9. [17]

    Chime: Llm-assisted hierarchical organization of scientific studies for literature review support

    Chao-Chun Hsu, Erin Bransom, Jenna Sparks, Bailey Kuehl, Chenhao Tan, David Wadden, Lucy Lu Wang, and Aakanksha Naik. Chime: Llm-assisted hierarchical organization of scientific studies for literature review support. arXiv preprint arXiv:2407.16148, 2024

    2024 arXiv

  10. [18]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arxiv 2021. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  11. [19]

    Benchmarking large language models as ai research agents

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Benchmarking large language models as ai research agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023

    2023

  12. [20]

    Towards continual knowledge learning of language models

    Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. arXiv preprint arXiv:2110.03215, 2021

    2021 arXiv

  13. [21]

    Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models

    Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. arXiv preprint arXiv:2204.14211, 2022

    2022 arXiv

  14. [22]

    Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents

    Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. Advances in Neural Infor...

    2025

  15. [23]

    Learning to edit: Aligning llms with knowledge editing

    Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, et al. Learning to edit: Aligning llms with knowledge editing. arXiv preprint arXiv:2402.11905, 2024

    2024 arXiv

  16. [24]

    Instruction-tuned language models are better knowledge learners

    Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, and Srinivasan Iyer. Instruction-tuned language models are better knowledge learners. arXiv preprint arXiv:2402.12847, 2024

    2024 arXiv

  17. [25]

    Overcoming catastrophic forgetting in neural networks

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of scien...

    2017

  18. [26]

    Thomas S. Kuhn. The Structure of Scientific Revolutions. University of Chicago Press, 1962

    1962

  19. [27]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computa...

    2019

  20. [28]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  21. [29]

    Memorization vs

    Aochong Oliver Li and Tanya Goyal. Memorization vs. reasoning: Updating llms with new knowledge. arXiv preprint arXiv:2504.12523, 2025

    2025 arXiv

  22. [30]

    Mapping the increasing use of llms in scientific papers

    Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, et al. Mapping the increasing use of llms in scientific papers. arXiv preprint arXiv:2404.01268, 2024. 11

    2024 arXiv

  23. [31]

    Infini- gram: Scaling unbounded n-gram language models to a trillion tokens

    Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini- gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377, 2024

    2024 arXiv

  24. [32]

    Mitigating heterogeneous token overfitting in llm knowledge editing

    Tianci Liu, Zihan Dong, Linjun Zhang, Haoyu Wang, and Jing Gao. Mitigating heterogeneous token overfitting in llm knowledge editing. arXiv preprint arXiv:2502.00602, 2025

    2025 arXiv

  25. [33]

    Codeupdatearena: Bench- marking knowledge editing on api updates

    Zeyu Leo Liu, Shrey Pandit, Xi Ye, Eunsol Choi, and Greg Durrett. Codeupdatearena: Bench- marking knowledge editing on api updates. arXiv preprint arXiv:2407.06249, 2024

    2024 arXiv

  26. [34]

    Llm4sr: A survey on large language models for scientific research

    Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. Llm4sr: A survey on large language models for scientific research. arXiv preprint arXiv:2501.04306, 2025

    2025 arXiv

  27. [35]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359–17372, 2022

    2022

  28. [36]

    Mass- editing memory in a transformer

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022

    2022 arXiv

  29. [37]

    Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022

    2022 arXiv

  30. [38]

    Arxivdigestables: Synthesizing scientific literature into tables using language models

    Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S Weld, Joseph Chee Chang, and Kyle Lo. Arxivdigestables: Synthesizing scientific literature into tables using language models. arXiv preprint arXiv:2410.22360, 2024

    arXiv

  31. [39]

    2 olmo 2 furious

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  32. [40]

    GPT-4o system card

    OpenAI. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  33. [41]

    Fine-tuning or retrieval? comparing knowledge injection in llms

    Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934, 2023

    2023 arXiv

  34. [42]

    Note on regression and inheritance in the case of two parents

    Karl Pearson. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240–242, 1895. doi: 10.1098/rspl.1895.0041

  35. [43]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. ArXiv, abs/2210.03350, 2022

    2022 arXiv

  36. [44]

    Large language models are zero shot hypothesis proposers

    Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. Large language models are zero shot hypothesis proposers. arXiv preprint arXiv:2311.05965, 2023

    2023 arXiv

  37. [45]

    Learning multiple visual domains with residual adapters

    Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017

    2017

  38. [46]

    Replug: Retrieval-augmented black-box language models.arXiv preprint arXiv:2301.12652, 2023

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models.arXiv preprint arXiv:2301.12652, 2023

    2023 arXiv

  39. [47]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024

    2024 arXiv

  40. [48]

    Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Rus- sell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas ...

    2024

  41. [49]

    Honeybee: Progressive instruction finetuning of large language models for materials science

    Yu Song, Santiago Miret, Huan Zhang, and Bang Liu. Honeybee: Progressive instruction finetuning of large language models for materials science. arXiv preprint arXiv:2310.08511, 2023

    2023 arXiv

  42. [50]

    Paperbench: Evaluating ai’s ability to replicate ai research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025

    2025 arXiv

  43. [51]

    How new data permeates llm knowledge and how to dilute it

    Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, and Mark Sandler. How new data permeates llm knowledge and how to dilute it. arXiv preprint arXiv:2504.09522, 2025

    2025 arXiv

  44. [52]

    Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  45. [53]

    Evowiki: Evaluating llms on evolving knowledge

    Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yugang Jiang, et al. Evowiki: Evaluating llms on evolving knowledge. arXiv preprint arXiv:2412.13582, 2024

    2024 arXiv

  46. [54]

    Fact or fiction: Verifying scientific claims

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. arXiv preprint arXiv:2004.14974, 2020

    2004 arXiv

  47. [55]

    Lekube: A legal knowledge update benchmark

    Changyue Wang, Weihang Su, Hu Yiran, Qingyao Ai, Yueyue Wu, Cheng Luo, Yiqun Liu, Min Zhang, and Shaoping Ma. Lekube: A legal knowledge update benchmark. arXiv preprint arXiv:2407.14192, 2024

    2024 arXiv

  48. [56]

    Wise: Rethinking the knowledge memory for lifelong model editing of large language models

    Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. Advances in Neural Information Processing Systems , 37: 53764–53797, 2024

    2024

  49. [57]

    Knowledge editing for large language models: A survey

    Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. Knowledge editing for large language models: A survey. ACM Computing Surveys, 57(3):1–37, 2024

    2024

  50. [58]

    Resolving knowledge conflicts in large language models

    Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Resolving knowledge conflicts in large language models. arXiv preprint arXiv:2310.00935, 2023

    2023 arXiv

  51. [59]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  52. [60]

    History matters: Temporal knowl- edge editing in large language model

    Xunjian Yin, Jin Jiang, Liming Yang, and Xiaojun Wan. History matters: Temporal knowl- edge editing in large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19413–19421, 2024

    2024

  53. [61]

    Self-tuning: Instructing llms to effectively acquire new knowledge through self-teaching

    Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Yipeng Zhang, Haitao Mi, and Helen Meng. Self-tuning: Instructing llms to effectively acquire new knowledge through self-teaching. arXiv preprint arXiv:2406.06326, 2024

    2024 arXiv

  54. [62]

    A comprehensive survey of scientific large language models and their applications in scientific discovery

    Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. A comprehensive survey of scientific large language models and their applications in scientific discovery. arXiv preprint arXiv:2406.10833, 2024

    2024 arXiv

  55. [63]

    knowledge

    Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah A Smith. Set the clock: Temporal alignment of pretrained language models.arXiv preprint arXiv:2402.16797, 2024. 13 A Limitations Real Scientific Advancement is Far More Complex In this work, we model sci...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.