Pith. sign in

REVIEW 5 major objections 5 minor 71 references

ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that mainstream LLMs consistently exhibit temporal leakage—using post-cutoff knowledge even when explicitly told to reason from a cutoff date—and introduces a four-task benchmark, ExAnte, to measure this failure.

desk verdict A genuinely useful benchmark for a real failure mode, but the Wikipedia task's unvalidated LLM judge and the stock threshold's missing false-positive analysis mean the cross-task consistency claim is one validation short. read the letter →

arxiv 2505.19533 v1 pith:Y7YXPM2A submitted 2025-05-26 cs.LG

Yachuan Liu , Xiaochun Wei , Lin Shi , Xinnuo Li , Bohan Zhang , Paramveer Dhillon , Qiaozhu Mei This is my paper
classification cs.LG
keywords ex-anteinferencetemporalleakagecutoffLLMbenchmarkpromptingstrategiesreasoningknowledgememorizationrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ExAnte, a benchmark that asks whether large language models can reason from a stated cutoff date without drawing on knowledge of later events. It defines temporal leakage as the use of post-cutoff information, and measures it across four tasks: predicting stock prices, recalling the dates of unexpected events, generating Wikipedia-style facts, and listing research publications. Tests of GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro under five prompting strategies find that leakage persists across models, tasks, and prompts, with no strategy that reliably enforces the cutoff. The paper argues that this makes ex-ante inference a distinct capability that simple prompting cannot provide, and proposes leakage rate plus a quality measure as the evaluation protocol for future work.

What carries the argument

The load-bearing machinery is the pair of reference sets $R_{pre}(x, t_c)$ and $R_{post}(x, t_c)$, the facts about a topic $x$ verifiable before and after the cutoff $t_c$, together with the leakage rate defined over them. For single-event prediction, a response leaks when it matches a post-cutoff event, written as an indicator $\hat{y} \in R_{post}$; for multi-event generation, leakage is the fraction of generated atomic claims that appear in $R_{post}$, and dataset-level leakage is the proportion of queries containing any such claim. A companion quality measure—accuracy for Wikipedia and Publication, negative mean absolute error for Stock—keeps models from escaping leakage detection with vacuous outputs, and a response counts as valid only if it satisfies zero leakage and quality above a task-specific threshold. Memorization checks are used to exclude cases where the model never knew the post-cutoff fact, so the leakage rate measures failure to suppress known future knowledge rather than ignorance of it.

What would settle it

Have human annotators judge a random sample of the same Wikipedia claims and stock predictions; if agreement with the LLM judge is low, or if a trivial baseline such as always predicting the most recent pre-cutoff price crosses the 3-percent threshold as often as the models do, then the reported leakage rates would not support the claim that LLMs specifically leak future knowledge.

Watch

Extended reading notes

Core claim

The central claim is that mainstream LLMs systematically fail at ex-ante inference: when told to act as if they are at a cutoff date, they still produce outputs that rely on facts that only became verifiable after that date. The paper supports this by constructing tasks in which pre- and post-cutoff knowledge can be separated, verifying that the model actually knows the post-cutoff facts before calling leakage, and then measuring how often those facts appear despite the cutoff. Across all four benchmark tasks, leakage appears in every model under at least some prompting strategy, and no single prompting strategy eliminates it in all settings; for example, instruction-based prompting helps on stock prediction but not on publication listing, and self-verification helps most tasks but fails on the Wikipedia generation task. The paper further reports that leakage increases with shorter cutoff gaps and with higher memorization accuracy, and that higher response quality is positively correlated with leakage, meaning the most factually correct outputs are often the ones that violate the temporal constraint.

Load-bearing premise

The entire leakage measurement rests on the assumption that its classifiers are unbiased: the LLM judge that attributes Wikipedia claims to pre-cutoff versus post-cutoff snapshots, and the fixed 3-percent price-distance threshold that flags stock predictions as leaked, with no human agreement study or false-positive baseline reported.

Editorial extensions

If this is right

  • No prompting strategy tested enforces a temporal cutoff reliably, so time-sensitive applications such as backtesting, historical simulation, and research-trend prediction cannot assume prompt instructions are enough.
  • Self-verification is the strongest prompt-level mitigation but fails specifically on open-ended Wikipedia generation, showing that self-checking alone does not generalize as a safety mechanism.
  • Because leakage rises when the model's memory is stronger and when the cutoff gap is short, the benchmark implies that better factual recall can make temporal adherence harder rather than easier.
  • The ExAnte protocol of leakage rate plus quality measure gives future systems a concrete target: reduce leaked claims without lowering pre-cutoff accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper tests only prompting; a natural extension is to test retrieval-augmented models with explicit pre-cutoff document snapshots, which would show whether grounding in $R_{pre}$ reduces leakage or whether the model still overrides retrieved context with memorized post-cutoff facts.
  • The positive accuracy–leakage correlation suggests an implicit design tension for training: objectives that reward factual precision in generation may actively reward temporal leakage, so temporal adherence may need to be a separate training objective.
  • Because the dataset covers only English-language, high-resource domains, the same leakage protocol could be applied to non-English and specialized-domain knowledge to test whether leakage is a general property or an artifact of pretraining distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces ExAnte, a benchmark for evaluating whether large language models can perform ex-ante inference, i.e., reason using only information available before a specified temporal cutoff. It defines a leakage-rate metric and a quality measure, and constructs four datasets: Stock (numerical price prediction), QA (exact-date prediction of future events), Wikipedia (multi-claim generation with pre/post-cutoff snapshots), and Publication (listing notable papers with publication-date verification). The authors evaluate GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro under five prompting strategies (zero-shot, instruction-based, chain-of-thought, one-shot, self-verification). The central empirical claim is that models consistently exhibit temporal leakage across all four tasks and that no single prompting strategy reliably enforces a temporal cutoff; secondary claims concern the effects of cutoff gap, memorization, and a positive correlation between leakage and quality.

Significance. If the findings are robust, ExAnte addresses a real and underexplored problem: the inability of LLMs to dynamically adhere to arbitrary temporal cutoffs, which matters for historical simulation, financial backtesting, and research-trend forecasting. The benchmark design has notable strengths: the QA and Stock tasks anchor leakage to external ground truth (exact dates and stock prices), the memorization checks prevent leakage evaluation on examples the model does not know, the dataset and code are publicly released, and the paper includes detailed prompt templates and model configurations. The QA task in particular provides clean, exact-match evidence of leakage. However, the cross-task consistency claim depends on the Wikipedia task, whose leakage labels are produced by an unvalidated LLM judge, and on the Stock task, whose leakage threshold lacks a false-positive analysis. These issues, combined with the absence of error bars or significance tests, mean the benchmark is promising but currently provides only partial support for its strongest claims.

major comments (5)
  1. [Section 3.3, Table 8, Appendix A.2.1] The Wikipedia leakage and accuracy labels are produced entirely by an LLM judge that is not validated: there is no human agreement study, no error analysis, and no report of which model serves as the judge or of the judge's own temporal-leakage behavior. Since the judge is asked to decide whether a generated claim is supported by Rpre or Rpost, and paraphrases and partial support require subjective inference, the 10-23% leakage rates in Table 4 may be systematically biased. Additionally, the truth table in Table 8 treats claims found in neither Rpre nor Rpost as 'irrelevant or unverifiable' and therefore not leakage, which can mask leaked claims that the judge simply fails to locate. The authors should provide human-annotation agreement (e.g., Cohen's kappa) on a representative sample, an error analysis, and a sensitivity analysis that re-labels 'neither' cases as leakage.
  2. [Section 3.1, Eq. (6)] The Stock leakage threshold delta=0.03 assumes that a prediction within 3% of the true future price cannot arise from pre-cutoff information, but no false-positive analysis is provided. A simple pre-cutoff baseline (e.g., predicting the last observed price, the historical mean, or a random draw from the historical price distribution) could land within 3% of the actual price for low-volatility stocks at a non-negligible rate, inflating the measured leakage. The authors should report leakage rates for such baselines under Eq. (6) and justify the threshold relative to those baselines. They should also specify what 'correctly recalls the price' means in the memorization filter (exact match, tolerance, or 3% band), since leakage is computed only on memorized examples.
  3. [Checklist item 7 and Section 4.1] The paper reports no error bars, confidence intervals, or significance tests for any leakage or accuracy figure. Many headline comparisons in Tables 2-5 are separated by only a few percentage points (e.g., GPT-4o Instruction 7.15% vs. CoT 5.37% vs. Self-Verification 5.42% in Table 2), and without variance estimates it is impossible to tell whether these differences are meaningful. The authors state in the checklist that repeating API experiments is expensive, but the appendix already contains repeated self-verification runs and memorization counts with standard deviations, so some repeated-measures analysis is feasible. At minimum, bootstrap confidence intervals over the query set should be reported for the key leakage rates that support the cross-task and cross-prompt conclusions.
  4. [Section 4.2, Tables 4 and 5] The claim of a 'strong positive correlation between leakage and accuracy' in the Wikipedia and Publication datasets is reported without any correlation coefficient, scatter plot, or statistical test. This relationship is particularly fragile because, in the Wikipedia task, both the leakage label and the accuracy score are derived from the same LLM judge, so the correlation may be an artifact of judge behavior rather than a property of the models. For Publication, accuracy is defined as 'cited at least once in the same calendar year as the cutoff' (Section 3.4), which is a weak proxy that may be noisy for recent years. The authors should quantify the correlation with appropriate statistics and discuss how judge reliability affects the result.
  5. [Tables 5 and 15, Section 4.2] There is an apparent internal inconsistency in the Publication leakage rates: Table 5 reports zero-shot leakage values of 82.02% (GPT-4o), 66.23% (Claude), and 78.41% (Gemini), while Appendix C.2 (Table 15) reports query-level zero-shot leakage rates of 41.84%, 32.65%, and 35.05% for the same models. Section 3.4 defines leakage as the proportion of valid publications with an earliest accessible date on or after tc, which suggests claim-level leakage, but the main text does not explain the relationship between Table 5 and Table 15. This discrepancy must be clarified, because the conclusion that 'all models show high leakage rates' in the Publication task is calibrated differently in the two tables.
minor comments (5)
  1. [Section 3.2 vs. Table 1] The QA dataset size is given as 304 in Table 1 but as 300 in Section 3.2 and again in the memorization-rate description; these numbers should be reconciled.
  2. [Section 4.1] The text says four models are evaluated, listing GPT-4o-mini, GPT-4o, Gemini 1.5 Pro, and Claude Sonnet 3.5, but the results tables only report three models and GPT-4o-mini never appears again; either its results should be included and discussed, or it should be removed from the setup.
  3. [Appendix D.2] The sentence describing GPT-4o's independent self-verification says leakage 'drops from 3.41% to 4.12%' between in-conversation and independent settings, but 4.12% is larger than 3.41%, so this is not a drop; the direction of the claimed effect is misstated.
  4. [Section 5 and Appendix A.1] The One-Shot Stock prompt compares the task to recalling lottery numbers, which frames the prediction as arbitrary guessing; this may encourage low-quality predictions that trivially avoid leakage, and the paper should discuss how this prompt choice affects the measured leakage and MAE.
  5. [Throughout] There are several typos and formatting issues: '⊮' in Eq. (5) should be an indicator function, 'Figure 2 (Appendix 2)' should reference Appendix C.1, and the 'MR' column header in Table 2 would benefit from an explicit definition in the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: leakage labels are anchored to external ground truth, and the unvalidated Wikipedia LLM judge is a measurement-validity risk rather than a derivation-equals-input circularity.

full rationale

The paper's central claim, that mainstream LLMs consistently exhibit temporal leakage, is an empirical benchmark result rather than a derived quantity. Leakage is operationalized against external ground truth in three of the four tasks: Yahoo Finance prices for Stock (Eq. 6), exact Wikipedia event dates for QA, and Google Scholar/ArXiv/ACM records for Publication. The Wikipedia task uses an LLM judge to decide whether a claim is supported by the pre-cutoff or post-cutoff snapshot (Section 3.3, Appendix A.2.1, Table 8), and this judge is not validated against human agreement; GPT-4o was also used in topic curation and keyword generation while being an evaluated model. That is a genuine validity and calibration concern, and Appendix F honestly notes that factual correctness is not assessed, but none of these issues makes a reported result equal to its input by construction. No free parameter is fitted to model outputs, no load-bearing self-citation appears, and no uniqueness theorem is imported from the authors' prior work. The Stock threshold delta=0.03 is a fixed assumption rather than a fitted value, and the leakages for Stock, QA, and Publication are anchored to independently verifiable records, so the benchmark's cross-task conclusion does not reduce to a self-referential definition.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The benchmark's central numbers depend on several unstated or lightly stated operational assumptions: the accuracy of an unvalidated LLM judge, the sufficiency of two Wikipedia snapshots, the tolerance of the memorization check, and the reliability of the publication-date pipeline. The free parameters delta, tau, and the memorization tolerance are all chosen without calibration evidence.

free parameters (3)
  • delta (stock leakage threshold) = 0.03
    Equation 6 flags a prediction as leaked if its relative error to the true future price is below delta; the paper says delta=0.03 balances sensitivity and specificity but provides no ROC or false-positive analysis.
  • tau (quality threshold) = unspecified
    Equation 5 defines a valid response as one with quality Q >= tau, but the paper never states tau's value for any task, leaving the validity criterion unoperationalized.
  • memorization recall tolerance = unspecified
    Section 3.1 says a stock price is 'correctly recalled' but does not define the tolerance used in the memorization check, which determines whether examples are included in leakage analysis.
assumptions (5)
  • domain assumption The archived Wikipedia snapshot closest to the cutoff captures all pre-cutoff facts, and the latest version captures all post-cutoff facts.
    Section 3.3 states that using only two versions provides 'a comparable effect' to all intermediate revisions; this is asserted, not demonstrated.
  • domain assumption The LLM judge correctly classifies generated claims as supported by pre- or post-cutoff Wikipedia content.
    Section 3.3 and the judgment prompt in A.2.1 are used without validation against human labels.
  • domain assumption A model that fails the memorization check does not possess the post-cutoff fact.
    Section 2.1 excludes examples from leakage evaluation unless the model can recall the future value; if the recall prompt under-triggers, the leakage rate is measured on a biased subset.
  • domain assumption Human annotators correctly filtered QA events to those that were not predictable in advance.
    Section 3.2 relies on two human annotators with no reported inter-annotator agreement.
  • domain assumption The publication date verification pipeline (Google, ArXiv, ACM) gives the earliest accessible date for each generated title.
    Section 3.4 automates existence and date checks, but the accuracy of this pipeline is not audited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models." pith.science (2026). https://pith.science/paper/Y7YXPM2A

@misc{pith2026250519533,
  author       = {Pith},
  title        = {Pith review of: ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y7YXPM2A}},
  note         = {Machine review of arXiv:2505.19533}
}
read the original abstract

Large language models (LLMs) face significant challenges in ex-ante reasoning, where analysis, inference, or predictions must be made without access to information from future events. Even with explicit prompts enforcing temporal cutoffs, LLMs often generate outputs influenced by internalized knowledge of events beyond the specified cutoff. This paper introduces a novel task and benchmark designed to evaluate the ability of LLMs to reason while adhering to such temporal constraints. The benchmark includes a variety of tasks: stock prediction, Wikipedia event prediction, scientific publication prediction, and Question Answering (QA), designed to assess factual knowledge under temporal cutoff constraints. We use leakage rate to quantify models' reliance on future information beyond cutoff timestamps. Experimental results reveal that LLMs struggle to consistently adhere to temporal cutoffs across common prompting strategies and tasks, demonstrating persistent challenges in ex-ante reasoning. This benchmark provides a potential evaluation framework to advance the development of LLMs' temporal reasoning ability for time-sensitive applications.

Figures

Figures reproduced from arXiv: 2505.19533 by the authors.

Figure 1
Figure 1. Illustration of temporal reasoning and benchmark task structure. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. GPT-4o’s historical stock price memorization pattern for AAPL. The blue line represents [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 38 canonical work pages

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Anthropic

    A. Anthropic. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3(6), 2024

    work page 2024

  3. [3]

    J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024

    arXiv 2024

  4. [4]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  5. [5]

    W. Chen, X. Wang, and W. Y . Wang. A dataset for answering time-sensitive questions.arXiv preprint arXiv:2108.06314, 2021

    arXiv 2021

  6. [6]

    Cheng, M

    J. Cheng, M. Marone, O. Weller, D. Lawrie, D. Khashabi, and B. Van Durme. Dated data: Tracing knowledge cutoffs in large language models. arXiv preprint arXiv:2403.12958, 2024

    arXiv 2024

  7. [7]

    Z. Chu, J. Chen, Q. Chen, W. Yu, H. Wang, M. Liu, and B. Qin. TimeBench: A comprehensive evaluation of temporal reasoning abilities in large language models. In L.-W. Ku, A. Martins, and V . Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1204–1228, Bangkok, Thailand...

    doi:10.18653/v1/2024.acl-long.66 2024

  8. [8]

    P. Das, S. Chaudhury, E. Nelson, I. Melnyk, S. Swaminathan, S. Dai, A. Lozano, G. Kollias, V . Chenthama- rakshan, S. Dan, et al. Larimar: Large language models with episodic memory control. arXiv preprint arXiv:2403.11901, 2024

    arXiv 2024

Show all 71 references
  1. [9]

    T. de Kok. Chatgpt for textual analysis? how to use generative llms in accounting research. Management Science, 2025

    2025

  2. [10]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Fatemi, M

    B. Fatemi, M. Kazemi, A. Tsitsulin, K. Malkan, J. Yim, J. Palowitch, S. Seo, J. Halcrow, and B. Perozzi. Test of time: A benchmark for evaluating llms on temporal reasoning. arXiv preprint arXiv:2406.09170, 2024

    2024 arXiv

  4. [12]

    Goyal, J

    T. Goyal, J. J. Li, and G. Durrett. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356, 2022

    2022 arXiv

  5. [13]

    X. Hu, D. Li, B. Hu, Z. Zheng, Z. Liu, and M. Zhang. Separate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18252–18260, 2024

    2024

  6. [14]

    Z. Ji, T. Yu, Y . Xu, N. Lee, E. Ishii, and P. Fung. Towards mitigating llm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827–1843, 2023

    2023

  7. [15]

    Kasai, K

    J. Kasai, K. Sakaguchi, R. Le Bras, A. Asai, X. Yu, D. Radev, N. A. Smith, Y . Choi, K. Inui, et al. Realtime qa: what’s the answer right now? Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [16]

    J. Lee, N. Stevens, and S. C. Han. Large language models in finance (finllms). Neural Computing and Applications, pages 1–15, 2025

    2025

  9. [17]

    J. Li, H. Bu, and J. Wu. Sentiment-aware stock market prediction: A deep learning method. In 2017 international conference on service systems and service management, pages 1–6. IEEE, 2017

    2017

  10. [18]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  11. [19]

    S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  12. [20]

    S. Lin, W. Hua, L. Li, C.-J. Chang, L. Fan, J. Ji, H. Hua, M. Jin, J. Luo, and Y . Zhang. Battleagent: Multi-modal dynamic emulation on historical battles to complement historical analysis. arXiv preprint arXiv:2404.15532, 2024. 10

    2024 arXiv

  13. [21]

    Liska, T

    A. Liska, T. Kocisky, E. Gribovskaya, T. Terzi, E. Sezener, D. Agrawal, D. Cyprien De Masson, T. Scholtes, M. Zaheer, S. Young, et al. Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models. In International Conference on Machine Learni...

    2022

  14. [22]

    Z. Liu, G. Dou, Z. Tan, Y . Tian, and M. Jiang. Machine unlearning in generative ai: A survey. arXiv preprint arXiv:2407.20516, 2024

    2024 arXiv

  15. [23]

    X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y . Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems , 35: 27591–27609, 2022

    2022

  16. [24]

    Mallen, A

    A. Mallen, A. Asai, V . Zhong, R. Das, D. Khashabi, and H. Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022

    2022 arXiv

  17. [25]

    Marketbeat: Stock market news and research tools

    MarketBeat. Marketbeat: Stock market news and research tools. https://www.marketbeat.com/,

  18. [26]

    S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023

    2023 arXiv

  19. [27]

    Pawelczyk, S

    M. Pawelczyk, S. Neel, and H. Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023

    2023 arXiv

  20. [28]

    Petroni, T

    F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y . Wu, A. H. Miller, and S. Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019

    1909 arXiv

  21. [29]

    Pochinkov and N

    N. Pochinkov and N. Schoots. Dissecting language models: Machine unlearning via selective pruning. arXiv preprint arXiv:2403.01267, 2024

    2024 arXiv

  22. [30]

    S. Qiao, Y . Ou, N. Zhang, X. Chen, Y . Yao, S. Deng, C. Tan, F. Huang, and H. Chen. Reasoning with language model prompting: A survey. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistic...

    2023 doi

  23. [31]

    Z. Su, J. Li, J. Zhang, T. Zhu, X. Qu, P. Zhou, Y . Bowen, Y . Cheng, et al. Living in the moment: Can large language models grasp co-temporal reasoning? arXiv preprint arXiv:2406.09072, 2024

    2024 arXiv

  24. [32]

    Q. Tan, H. T. Ng, and L. Bing. Towards benchmarking and improving the temporal reasoning capability of large language models. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume ...

    2023 doi

  25. [33]

    Q. Tan, H. T. Ng, and L. Bing. Towards benchmarking and improving the temporal reasoning capability of large language models. arXiv preprint arXiv:2306.08952, 2023

    2023 arXiv

  26. [34]

    Thorne, A

    J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355, 2018

    2018 arXiv

  27. [35]

    A. Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  28. [36]

    Q. Wang, D. Downey, H. Ji, and T. Hope. Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259, 2023

    2023 arXiv

  29. [37]

    W. Wang, L. Gu, L. Zhang, Y . Luo, Y . Dai, C. Shen, L. Xie, B. Lin, X. He, and J. Ye. Scipip: An llm-based scientific paper idea proposer. arXiv preprint arXiv:2410.23166, 2024

    2024 arXiv

  30. [38]

    Wang and Y

    Y . Wang and Y . Zhao. TRAM: Benchmarking temporal reasoning for large language models. In L.-W. Ku, A. Martins, and V . Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024, pages 6389–6415, Bangkok, Thailand, Aug. 2024. Association for Comput...

    2024 doi

  31. [39]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 11

    2022

  32. [40]

    Xiong, A

    S. Xiong, A. Payani, R. Kompella, and F. Fekri. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.06853, 2024

    2024 arXiv

  33. [41]

    C. Yuan, Q. Xie, J. Huang, and S. Ananiadou. Back to the future: Towards explainable temporal reasoning with large language models. In Proceedings of the ACM on Web Conference 2024, pages 1963–1974, 2024

    2024

  34. [42]

    J. Yuan, L. Pan, C.-W. Hang, J. Guo, J. Jiang, B. Min, P. Ng, and Z. Wang. Towards a holistic evaluation of llms on factual knowledge recall. arXiv preprint arXiv:2404.16164, 2024

    2024 arXiv

  35. [43]

    Z. Yuan, E. Chamoun, R. Aly, C. Whitehouse, and A. Vlachos. Probelm: Plausibility ranking evaluation for language models. arXiv preprint arXiv:2404.03818, 2024

    2024 arXiv

  36. [44]

    Zhang, F

    T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, and T. B. Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57, 2024

    2024

  37. [45]

    + Event +

    Y . Zhuang, Y . Yu, K. Wang, H. Sun, and C. Zhang. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36:50117–50143, 2023. 12 A Prompts Used for Benchmarking Experiment A.1 The Five Prompting Strategies Table 6...

    2023

  38. [47]

    + Event +

    Assume you are on {cutoff date}. Now please answer the following question. Just leave the best answer without explanation. What was the exact date when" + Event + "? Please answer in the format YYYY-MM-DD. You should answer this question honestly. If you don’t know, please ans...

  39. [48]

    Otherwise, you ONLY return ’No leakage’

    Do you think this answer relied on information beyond {cutoff date}? If you believe your response used future knowledge, please revise your answer based only on information available on or before {cutoff date}. Otherwise, you ONLY return ’No leakage’. Wikipedia Zero-Shot Regar...

    1946

  40. [49]

    Each fact should be a short, clear statement representing a distinct piece of information

    Regarding your knowledge about “[title]”, generate 5 atomic facts. Each fact should be a short, clear statement representing a distinct piece of information. Only use information from before December 31st, [cutoff_year-1]

  41. [50]

    title":

    You are a knowledgeable person living just before [cutoff_year]. Evaluate if this fact was known before [cutoff_year]. Fact: claim Answer in this format: [Known Before [cutoff_year]: yes/no [Explanation]: Brief explanation of your decision If the model replies with no” in the ...

  42. [51]

    title":

    Assume you are on {cutoff date}. Please list the most notable publications in the field of {subfield} in computer science published in {year} that you consider impressive. Just list the publications without explanation. For each publication, please provide title. Your output i...

  43. [52]

    What was the stock closing price of [Company Name] ([Ticker]) on [Event Date]? Please return a float number only. Example: ’143.16’

    Do you think this answer relied on information beyond {cutoff date}? If you believe your response used future knowledge, please revise your answer based only on information available on or before {cutoff date}. Otherwise, you ONLY need return ’No leakage’. A.2 Other Prompts Us...

    2010

  44. [53]

    Post-2010 Evolution: Must have significant development/changes after 2010 (new research, technological advances, shifts in discourse)

    2010

  45. [54]

    Identifiable Cutoff: Must have a clear time point where the topic saw a noticeable shift in development/methodology/adoption

  46. [55]

    If you are at 2021-12-30, what is Apple’s stock price on 2022-12-30? Your answer should just be a num- ber, e.g., ’143.16’

    Distinct Phases: Must have considerable discussion both before and after the cutoff, with clear differences in understanding Respond in this exact format: Classification: VALID or INVALID Cutoff Year: [YYYY] (only if VALID) Reason: Brief explanation including key developments ...

    2012

  47. [56]

    • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope Guideline...

  48. [57]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] 26 Justification: We provided a limitation section in Appendix. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that th...

  49. [58]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Although our paper is not theoretical work, we provide the full set of assump- tions of our ...

  50. [59]

    Guidelines: 27 • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  51. [60]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the open access ...

  52. [61]

    • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We specify all the training a...

  53. [62]

    Instead of reporting error bars, we tested each method on multiple datasets

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: As the experiments rely on the API calls to multipl...

  54. [63]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide the computer...

  55. [64]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethicshttps://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We follow the NeurIPs Code of Ethics. Guidelines: • The answer NA means that the...

  56. [65]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have a section dicuss the potential impaces of our work. Guidelines: • The answer NA means that there ...

  57. [66]

    30 Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We describe th...

  58. [67]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The creators or ...

  59. [68]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The new assets introduced in the paper are documented. Guidelines: • The answer NA means that the paper does not rel...

  60. [69]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 3...

  61. [70]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  62. [71]

    Answer: [Yes] Justification: We describe the usage of LLM

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  63. [2025]

    Accessed: 2025-05-15

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.