Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that temporal leakage in LLM backtests is measurable at claim level and that an inference-time architecture, TimeSPEC, can reduce Shapley-DCLR by 75–99% while preserving performance where pre-cutoff evidence suffices.

desk verdict Useful claim-level leakage audit for LLM backtesting, but the metric's decision-critical claim rests on an unvalidated faithfulness assumption and the mitigation results are partly built into the evaluation. read the letter →

arxiv 2602.17234 v2 pith:VNQV46QI submitted 2026-02-19 cs.AI cs.LG

Zeyu Zhang , Ryan Chen , Bradly C. Stadie This is my paper · ORCID
classification cs.AIcs.LG
keywords temporalleakagebacktestingShapleyvaluesclaimdecompositionknowledgecutoffcontaminationdetectiontime-supervisedpredictionLLMforecasting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that temporal knowledge leakage in LLM backtesting—when a model uses information from after the reference cutoff to make a prediction—can be detected, quantified, and mitigated at the level of individual claims in the model's rationale. The proposed metric, Shapley-DCLR, weights each claim by its Shapley-value contribution to the prediction, so it measures what fraction of decision-driving reasoning is contaminated, not just how many leaked claims appear. The paper further claims that TimeSPEC, a five-phase pipeline that filters search results by date, extracts and verifies claims programmatically, regenerates when violations are found, and forces final aggregation to use only validated claims, can cut Shapley-DCLR by 75% on a salary-prediction task, 99% on a stock-ranking task, and effectively to zero on legal prediction. A sympathetic reader would care because this could turn backtesting from an unverifiable exercise into an auditable one, separating genuine forecasting ability from hindsight-masked memory.

What carries the argument

The key machinery is Shapley-DCLR, defined as Σ|φᵢ|·ℓ(cᵢ) / Σ|φᵢ|, where φᵢ is the Shapley value of claim cᵢ computed via Monte Carlo sampling, and ℓ(cᵢ) is a leakage indicator. The leakage indicator is made efficient by a claim taxonomy: outcome claims (A4) and consequential claims (A5) are always leaked by definition, background (B1) and definitional (B2) claims are never leaked, and only temporally verifiable claims (A1–A3) require external search for their determination date. The mitigation architecture, TimeSPEC, interleaves generation, claim supervision, regeneration, re-supervision, and aggregation, with a closed-world constraint that the final prediction may use only task input and v

What would settle it

Take a TimeSPEC stock-ranking output (which reports zero Shapley-DCLR) and ask the same backbone model to re-rank the same stocks from memory without providing any rationale or claims. If the model reproduces the post-cutoff ranking that baselines achieved, then the leaked knowledge lives in parametric memory, not in the filtered claims, and the reported 99% reduction would be about text rather than cognition.

Watch

Extended reading notes

Core claim

The central discovery is that temporal leakage in LLM predictions can be decomposed into atomic claims, each with a determinable leakage status, and that weighting those claims by their Shapley-value contribution yields Shapley-DCLR—the fraction of decision-critical reasoning that derives from post-cutoff information. The paper further shows that a programmatic verification pipeline, TimeSPEC, which retrieves only pre-cutoff documents, extracts claims, checks each against a taxonomy (always-leaked outcome claims, never-leaked background claims, and searchable dated claims), and regenerates until clean, can reduce Shapley-DCLR by 75–99% across three tasks. Notably, the salience of leakage var

Load-bearing premise

The entire measurement stands on the assumption that a model's prediction is genuinely a function of the rationale it writes, so that contamination detected in extracted claims is contamination in the decision process itself.

Editorial extensions

If this is right

  • A backtest that reports accuracy without a leakage audit can be misleading: on leakage-sensitive tasks, high baseline performance is a signal of contamination, not forecasting skill.
  • Prompt-based temporal constraints ('only use information before this date') are measurably insufficient; programmatic verification with explicit date comparison is required for reliable temporal control.
  • The cost of enforcing temporal validity scales with a task's reliance on post-cutoff information: legal prediction loses almost nothing, salary prediction loses moderate accuracy, and stock ranking loses most of its apparent performance.
  • Deterministic leakage status for certain claim types makes leakage detection cheap: only a subset of claims (A1–A3) require external verification, reducing computational overhead.
  • TimeSPEC's closed-world final aggregation provides a principled estimate of what a genuinely pre-cutoff-informed predictor can achieve, giving a 'leakage-free baseline' for future comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Shapley-DCLR is adopted as a standard audit metric, backtest validity becomes a continuous quantity rather than a binary flag, letting practitioners trade acceptable leakage against performance per task.
  • Because TimeSPEC blocks rather than removes leaked parametric knowledge, an equivalent training-time or post-training intervention (e.g., unlearning or RL-based suppression) could complement inference-time filtering and reduce the performance cost on leakage-sensitive tasks.
  • The taxonomy's deterministic rules for outcome and background claims suggest that similar deterministic structure could be exploited for other contamination types, such as benchmark leakage or memorized verbatim examples.
  • The stock-ranking result implies that any LLM evaluation spanning a period with a black-swan event should require a claim-level leakage audit before interpreting performance as predictive ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a claim-level framework for detecting temporal contamination in LLM backtesting. It decomposes a model's prediction rationale into atomic claims, categorizes them by temporal verifiability (A1–A5/B1–B2), computes Shapley values to weight each claim's contribution to the prediction, and aggregates these into Shapley-DCLR, an importance-weighted leakage rate. It also introduces TimeSPEC, an inference-time architecture that alternates temporally filtered search, claim extraction/verification, and regeneration so that final predictions are grounded only in pre-cutoff evidence. Experiments on Supreme Court case prediction, NBA salary estimation, and stock return ranking report that standard baselines leak substantially, while TimeSPEC reduces Shapley-DCLR by 75–99%, with performance preserved where pre-cutoff evidence suffices and degraded on the deliberately leakage-sensitive stock task.

Significance. Temporal leakage is a real and important threat to LLM backtesting, and the goal of attributing leakage to individual claims is valuable. The paper is transparent in its prompts and architecture, and the case studies (Appendix C) are informative. The ability to separate 'correct but leaked' predictions from 'correct and clean' ones would be a useful contribution. However, the central quantitative claims currently rest on two unvalidated assumptions: that the extracted claims are the causal drivers of the prediction, and that the characteristic function used to compute Shapley values reflects decision influence rather than accuracy. The evaluation is also partially circular because TimeSPEC's filter and the leakage detector share the same rules. If these issues are addressed with independent validation, the framework could become a practical audit tool.

major comments (5)
  1. [Section 6.3, Eq. (11)] The characteristic functions are not consistent with the stated goal of measuring decision influence. v_class(S) is defined as P(ŷ=y*|S), the probability assigned to the true outcome, and v_rank(S)=1−ρ(r(S), r_N), where r_N is the 'full-information ranking'—which, as written, appears to be the ground-truth ranking. This measures contribution to accuracy, not to the model's decision. A claim that changes the output ranking can receive a very different |φ| depending solely on where the true ranking lies. If r_N is instead the model's own full-rationale ranking, the text should say so; the classification definition still needs reconciling. Please define v(S) as a function of the model's output alone.
  2. [Appendix B.1] The re-prediction validation is not sufficient to establish decision-level faithfulness. Asking the same backbone model to re-predict from the cleaned rationale cannot rule out parametric recall of the outcome; the baselines' high stock-ranking correlation (ρ≈0.54) on a task that is unknowable from pre-cutoff data is direct evidence of such recall. Low re-prediction error is therefore compatible with the rationale playing little or no causal role. A control condition is needed, e.g., re-predicting from scrambled or another model's rationales, or a human fidelity study.
  3. [Sections 5.2 and 7.2] TimeSPEC's Supervisor filters claims using exactly Eq. (3), and the evaluation pipeline labels leakage using the same Eq. (3). Consequently, TimeSPEC's near-zero Shapley-DCLR is partly guaranteed by construction: every claim the Supervisor removes would have been labeled leaked by the evaluator. The reported 75–99% reductions are therefore not an independent empirical finding. An independent audit—human-annotated leakage labels on a sample, or a different detection method—is needed before the headline reductions can be accepted.
  4. [Tables 3 and 4] All results are reported as point estimates with no confidence intervals, standard errors, or significance tests. On datasets of N=98/152/100, differences such as legal BS 0.228 vs 0.201 or salary RE 0.379 vs 0.256 may be within sampling noise. Since the paper draws strong comparative conclusions across agents and tasks, at least bootstrap confidence intervals and paired tests are required.
  5. [Section 3.2 and Phase 1 (Section 4.1)] The taxonomy's exhaustiveness is asserted without proof, and no inter-annotator agreement, human validation, or accuracy numbers are reported for claim extraction and categorization. Leakage labels for A1–A3 also depend on external search and date extraction, whose error rates are unreported. Errors in these stages propagate directly into OLR and Shapley-DCLR. A human-evaluation study on a sample of rationales is needed.
minor comments (5)
  1. [Abstract] Typo: 'illstrates' should be 'illustrates'.
  2. [Table 1] Typo in example: 'Marth 15' should be 'March 15'.
  3. [Section 6.3, Eq. (11)] The notation r_N and y* is not defined explicitly in the text. Please define these symbols when first used.
  4. [Figure 3] The y-axis ranges include negative values down to −0.05, but Shapley-DCLR is defined on [0,1]. Consider restricting the axis or explaining the visual padding.
  5. [Section 6.1] The datasets are author-curated and no download links or data files are provided. For reproducibility, please release the datasets with the paper.

Circularity Check

1 steps flagged · score 4.0 of 10

Shapley-DCLR is not circular by equation, but its decision-level interpretation rests on a circular same-model faithfulness validation.

  1. other [Appendix B.1 (Rationale Faithfulness Validation), Eqs. (12)-(13); see also Section 6.4 ('All agents use Claude-sonnet-4 as the backbone LLM')]
    "For each agent output, we: (1) clean the rationale by removing explicit prediction statements while retaining the reasoning and evidence; (2) prompt the same backbone model (Claude Sonnet 4) with the task input and cleaned rationale to generate a new prediction. If the re-prediction closely matches the original, the prediction faithfully followed the rationale."

    The validation's success condition (MRE < 0.3%) is satisfied whenever the backbone model can reproduce the original output, whether by following the rationale or by recalling the resolved outcome from parametric memory (e.g., Dobbs, Brogdon's contract, COVID-era stock rankings). Since the same Claude-sonnet-4 both makes the original prediction and performs the re-prediction, the test cannot distinguish 'prediction follows rationale' from 'prediction recalls outcome.' It presupposes that the model is not using post-cutoff knowledge—exactly the hypothesis the framework aims to detect. Thus the 'decision-critical' interpretation of Shapley-DCLR is supported only by a circular validation, and the reported 75-99% reductions may reflect cleaned rationale text rather than cleaned decision process

full rationale

The core metric Shapley-DCLR (Eq. 4) is a definition—an importance-weighted sum of leakage indicators—and the deterministic A4/A5 rules in Eq. 3 follow logically from the definition of outcome/consequence claims, so they are not circular by construction. The Shapley computation (Eq. 2) is a standard attribution over the model's own coalition predictions; it is not fitted to the leakage metric. TimeSPEC's supervisor deliberately removes claims that Eq. 3 would mark leaked, so its near-zero OLR/DCLR in Table 4 is a design consequence rather than an independent falsification of the metric. The central validity step is Appendix B.1, which asserts faithfulness from same-model re-prediction. Because the backbone may have memorized the resolved outcomes, low re-prediction error cannot establish that the original prediction used only the rationale claims. The same issue affects Phase 2 Shapley prompts ('Use ONLY provided claims'), which ask a contaminated model to ignore parametric knowledge—an instruction the paper itself argues LLMs cannot reliably follow. These are circular-validation and validity threats, not equation-level reductions, so a moderate score is appropriate rather than a high one.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No physical entities are invented. The framework introduces methodological constructs (taxonomy, Shapley-DCLR, TimeSPEC) and several hand-set engineering thresholds that affect the reported numbers; the main assumptions are the exhaustiveness of the taxonomy, deterministic leakage status for four categories, LLM compliance with claim-only prediction, and reliability of external search dates.

free parameters (4)
  • regeneration_iterations = 1
    Bounds TimeSPEC cost; residual leakage after one regeneration is logged but not removed, so the leakage-reduction numbers depend on this hand-set cap (Section 6.4).
  • min_search_results = 10
    Generator accumulates at least 10 search results; evidence sufficiency and performance depend on this hand-set threshold (Appendix A.2).
  • shapley_monte_carlo_permutations = 100 (seed 42)
    Number of Monte Carlo permutations per instance; controls variance of the Shapley estimates and therefore the reported Shapley-DCLR (Appendix A.1).
  • strict_timestamp_interpretation = period-end (e.g., '2023' -> 2023-12-31)
    Choice to map vague timestamps to latest possible dates avoids false negatives but can mark legitimate claims as leaked if the fact became known earlier (Section 4.3).
assumptions (5)
  • ad hoc to paper Claim taxonomy A1-A5/B1-B2 is exhaustive: every factual assertion either has a determinable time of first knowability or is timeless.
    Stated in Section 3.2 without proof; the deterministic leakage shortcuts for A4/A5 and B1/B2 depend on this partition.
  • domain assumption Outcome claims (A4) and consequential claims (A5) are always leaked by definition; background (B1) and definitional (B2) claims are never leaked.
    Encoded in Eq. 3; reasonable for prediction targets but assumes the LLM extractor labels categories correctly.
  • domain assumption An LLM asked to predict from a subset of claims uses only those claims and not its parametric memory.
    Appendix D.3 instructs 'Use ONLY provided claims - NO external knowledge'; no test verifies compliance, so Shapley values may still be contaminated by the model's own knowledge.
  • domain assumption Public knowledge is monotone, K(t1) ⊆ K(t2), and tau(c) = inf{t: c in K(t) or derivable from K(t)} is well-defined.
    Section 3.1 formalizes the problem; assumes nested knowledge sets and decidability of derivability.
  • domain assumption External search results and extracted publication dates reliably determine when a claim became publicly verifiable.
    Section 4.3 and Appendix A.1 rely on Perplexity API search; no accuracy evaluation of the verification step is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting." pith.science (2026). https://pith.science/paper/VNQV46QI

@misc{pith2026260217234,
  author       = {Pith},
  title        = {Pith review of: All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNQV46QI}},
  note         = {Machine review of arXiv:2602.17234}
}
read the original abstract

Backtesting LLMs on resolved events assumes models reason only from pre-cutoff knowledge, yet pretrained models inevitably leak post-cutoff knowledge. We introduce a claim-level evaluation framework that decomposes prediction rationales into atomic claims and applies Shapley values to quantify each claim's decision impact, yielding \textbf{Shapley-DCLR} (\textbf{Shapley}-weighted \textbf{D}ecision-\textbf{C}ritical \textbf{L}eakage \textbf{R}ate) -- an interpretable metric measuring what fraction of decision-driving reasoning is contaminated. We further propose \textbf{TimeSPEC} (\textbf{Time}-\textbf{S}upervised \textbf{P}rediction with \textbf{E}xtracted \textbf{C}laims), an inference-time architecture that interleaves temporally-filtered retrieval with claim-level supervision, producing predictions grounded entirely in pre-cutoff evidence. Across three LLMs, the ablation experiments confirm retrieval and supervision are jointly necessary; and a three-task probe further illstrates that the performance cost of temporal enforcement scales with each task's reliance on post-cutoff information.

Figures

Figures reproduced from arXiv: 2602.17234 by the authors.

Figure 1
Figure 1. Overview of the temporal leakage evaluation pipeline. Given a prediction rationale [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Architecture of TimeSPEC. Phase 1 (Generator) performs temporally-filtered search retrieving only documents [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Two-dimensional evaluation of prediction agents across three tasks. X-axis: transformed performance (1-BS, 1-RE, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Look-Ahead-Freedom as Temporal Non-Interference: A Verifiable Correctness Property for Backtesting and Agentic Trading Pipelines

    cs.CR 2026-07 accept novelty 7.5 of 10

    Look-ahead-freedom is temporal non-interference; it is undecidable with value-dependent availability but soundly checkable in linear time on the value-independent pipelines practitioners write.

Reference graph

Works this paper leans on

40 extracted references · 5 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Oshin Agarwal and Ani Nenkova. 2022. Temporal effects on pre-trained models for language processing tasks.Transactions of the Association for Computational Linguistics10 (2022), 904–921

    2022

  2. [2]

    Anonymous. 2025. Forecasting with LLMs: A Dataset for Rapid Backtesting Without Temporal Contamination. InICLR 2026 Conference Submission. Under review

    2025

  3. [3]

    Anthropic. 2024. The Claude 3 Model Family: A New Standard for Intelligence. Anthropic Technical Report(2024)

    2024

  4. [4]

    David H Bailey, Jonathan M Borwein, Marcos López de Prado, and Qiji Jim Zhu

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in Neural Information Processing Systems33 (2020), 1877–1901

    2020

  6. [6]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In30th USENIX Security Symposium (USENIX Security 21). 2633–2650

    2021

  7. [7]

    Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, and Swabha Swayamdipta. 2023. REV: Information-theoretic evaluation of free-text rationales. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2007–2030

    2023

  8. [8]

    Shung, and Bradly C

    Ryan Chen, Youngmin Ko, Zeyu Zhang, Catherine Cho, Sunny Chung, Mauro Giuffré, Dennis L. Shung, and Bradly C. Stadie. 2025. LAMP: Extracting Locally Linear Decision Surfaces from LLM World Models. https://arxiv.org/abs/2505. 11772

    2025

Show all 40 references
  1. [9]

    Shuaiyu Chen, T Clifton Green, Huseyin Gulen, and Dexin Zhou. 2024. What does chatgpt make of historical stock returns? extrapolation and miscalibration in llm stock return forecasts.arXiv preprint arXiv:2409.11540(2024)

    2024 arXiv

  2. [10]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. 2024. TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024 doi

  3. [11]

    Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. ERASER: A benchmark to evalu- ate rationalized NLP models. InProceedings of the 58th annual meeting of the association for computational linguistics. 4443–4458

    2020

  4. [12]

    Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022. Time-aware language models as temporal knowledge bases.Transactions of the Association for Computational Linguistics10 (2022), 257–273

    2022

  5. [13]

    Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, and Robert Sim. 2021. Training data leakage analysis in language models.arXiv preprint arXiv:2101.05405(2021)

    2021 arXiv

  6. [14]

    Zheng Ping Jiang, Yining Lu, Hanjie Chen, Daniel Khashabi, Benjamin Van Durme, and Anqi Liu. 2024. RORA: Robust free-text rationale evaluation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1070–1087

    2024

  7. [15]

    Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E Tetlock. 2025. ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. InThe Thirteenth International Conference on Learning Representations

    2025

  8. [16]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing ...

    2020

  9. [17]

    Xiangyu Li, Yawen Zeng, Xiaofen Xing, Jin Xu, and Xiangmin Xu. 2025. Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents.arXiv preprint arXiv:2510.07920(2025)

    2025

  10. [18]

    Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024. An Open- Source Data Contamination Report for Large Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2024. 528–541

    2024

  11. [19]

    Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and San- tiago Zanella-Béguelin. 2023. Analyzing leakage of personally identifiable in- formation in language models. In2023 IEEE Symposium on Security and Privacy (SP). IEEE, 346–363

    2023

  12. [20]

    Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. InAdvances in Neural Information Processing Systems, Vol. 30

    2017

  13. [21]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine- grained Atomic Evaluation of Factual Precision in Long Form Text Generation. InProceedings of the 2023 Conference on Emp...

    2023

  14. [22]

    OpenAI. 2023. GPT-4 Technical Report.arXiv preprint arXiv:2303.08774(2023)

    2023 arXiv

  15. [23]

    Andreas Östling, Holli Sargeant, Huiyuan Xie, Ludwig Bull, Alexander Terenin, Leif Jonsson, Måns Magnusson, and Felix Steffek. 2023. The Cambridge Law Corpus: A Dataset for Legal AI Research. InAdvances in Neural Information Processing Systems, Vol. 36

    2023

  16. [24]

    Piotr Pęzik, Konrad Kaczyński, Maria Szymańska, Filip Żarnecki, Zuzanna Deck- ert, Jakub Kwiatkowski, and Wojciech Janowski. 2025. LLMLagBench: Identify- ing Temporal Training Boundaries in Large Language Models.arXiv preprint arXiv:2511.12116(2025)

    2025

  17. [25]

    Suproteem K Sarkar and Keyon Vafa. 2025. Lookahead Bias in Pretrained Lan- guage Models. InICML 2025 Workshop on Reliable and Responsible Foundation Models

    2025

  18. [26]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems36 (2023), 68539–68551

    2023

  19. [27]

    Lloyd S Shapley. 1953. A value for n-person games.Contributions to the Theory of Games2, 28 (1953), 307–317

    1953

  20. [28]

    Charles Spearman. 1904. The proof and measurement of association between two things.The American Journal of Psychology15, 1 (1904), 72–101

    1904

  21. [29]

    Philip E Tetlock, Barbara A Mellers, Nick Rohrbaugh, and Eva Chen. 2014. Fore- casting tournaments: Tools for increasing transparency and improving the quality of debate.Current Directions in Psychological Science23, 4 (2014), 290–295

    2014

  22. [30]

    James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The Fact Extraction and VERification (FEVER) Shared Task. InProceedings of the First Workshop on Fact Extraction and VERification (FEVER). 1–9

    2018

  23. [31]

    Jonas Wallat, Adam Jatowt, and Avishek Anand. 2024. Temporal blind spots in large language models. InProceedings of the 17th ACM International Conference on Web Search and Data Mining. 683–692

    2024

  24. [32]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models.Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  25. [33]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V Le. 2024. Long-form factuality in large language models.arXiv preprint arXiv:2403.18802(2024)

    2024 arXiv

  26. [34]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebas- tian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann

  27. [35]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InThe Eleventh International Conference on Learning Representations

    2023

  28. [36]

    Haofei Yu, Fenghai Li, and Jiaxuan You. 2025. LiveTradeBench: Seeking Real- World Alpha with Large Language Models.arXiv preprint arXiv:2511.03628 (2025)

    2025

  29. [37]

    Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. 2025. Is Your LLM Outdated? A Deep Look at Temporal Generalization. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hum...

    2025

  30. [38]

    2023”→ 2023-12-31, “Q3 2023

    Zhiyuan Zhu, Yusheng Liao, Zhe Chen, Yuhao Wang, Yunfeng Guan, Yanfeng Wang, and Yu Wang. 2025. EvolveBench: A Comprehensive Benchmark for Assessing Temporal Awareness in LLMs on Evolving Knowledge. InProceedings of the 63rd Annual Meeting of the Association for Computational ...

    2025 doi

  31. [2014]

    Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance.Notices of the American Mathematical Society61, 5 (2014), 458–471

    2014

  32. [2023]

    Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564(2023)

    2023 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.