REVIEW 4 major objections 6 minor 2 cited by
Bench to the Future: A Pastcasting Benchmark for Forecasting Agents
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims Bench to the Future is the first forecasting benchmark that is realistic, hermetic, and repeatable, by having LLM forecasters work on past questions with sealed web snapshots.
desk verdict A careful but under-validated benchmark: the hermetic RetroSearch corpus is genuinely new, but the central fidelity claim rests on 20 visual comparisons, not rigorous evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is RetroSearch, a hermetic pastcasting environment built on per-question web snapshots: for each question, an intelligent crawler explores base rates, identifies influential actors, and performs deep research, storing around 20,000 scraped pages in a database dated to the question's original forecast window. At evaluation time, the agent searches through a live Google interface that is filtered to return only pages already in the snapshot, and is told that today's date is the snapshot date, making a past event feel like a live open question while preventing information leakage from after that date.
What would settle it
A direct test would take a set of resolved questions, run the same agent both on the BTF retro snapshot and on a properly time-boxed live-web search from before the resolution date, and check whether Brier-score differences or rank reversals exceed what would be expected by chance; the paper already documents one such failure, the NFL Tush Push question, where the retro corpus missed the decisive May 21 vote that live forecasters saw, showing that a single missing decisive document can produce a large divergence.
Extended reading notes
Core claim
The paper's central claim is that a pre-crawled snapshot of the web, taken at the time a forecasting question was asked, contains enough of the decision-relevant information that an LLM forecaster can produce forecasts of comparable quality to those made with live internet access at the time. On a curated set of 20 questions, the distribution of retro forecasts was visually similar to live forecasts for all but one question, where the retro corpus missed a decisive May 21 vote that live forecasters saw. The paper further reports that on the full 299-question set, newer models (Claude Sonnet 4, Gemini 2.5 Pro) scored better than older ones (Claude Sonnet 3.5 and 3.7), that agentic ReAct forecasters outperformed a non-agentic fixed-evidence pipeline, and that averaging five runs reduced inter-run forecast variance by roughly two-thirds, making repeated evaluations the recommended protocol.
Load-bearing premise
The benchmark's validity rests on the assumption that a pre-crawled snapshot of about 20,000 pages per question contains the same decision-relevant information a live forecaster would have found.
Editorial extensions
If this is right
- Forecasting capability can be measured and compared across model releases within hours of a new model appearing, rather than waiting months for real-world resolutions.
- Because evaluations are repeatable, benchmark scores can be reported as distributions over runs, and the paper's finding that five-run averaging cuts variance by two-thirds gives a concrete protocol for stable comparisons.
- Agentic forecasters that choose their own evidence outperform pipelines that receive pre-sourced evidence, suggesting that strategic, iterative information gathering is a measurable component of forecasting skill.
- The yes-bias observed across all models and approaches warrants attention, and the benchmark's design makes it possible to test whether that bias is an artifact of binary expansion or a genuine feature of LLM judgment.
- As new models with more recent training cutoffs appear, BTF's living-question design lets the benchmark retire questions a model might have memorized and add newer ones, keeping the hermetic guarantee intact.
Reading between the lines
- If pastcasting fidelity holds generally, the same snapshot-and-seal approach could be applied to other time-sensitive evaluations, such as medical diagnosis from historical records, legal research from past dockets, or market prediction from archived news, wherever a cutoff date is naturally defined.
- The single Tush Push failure suggests a concrete improvement: benchmarks could track whether each question's snapshot contains the specific decisive documents that live forecasters cite, and flag questions where it does not, turning snapshot fidelity into a measured property rather than an assumption.
- The result that agentic forecasters beat fixed-evidence pipelines despite likely seeing fewer pages implies that evaluation cost could be reduced by focusing on the agent's selection behavior, not just its final probability, and that trace analysis of chosen evidence might predict Brier scores better than raw page counts.
- The absence of a human baseline makes absolute Brier scores uninterpretable, but the benchmark's repeatability means a human crowd could be run on the same snapshots in a follow-up study, giving the field a direct human-vs-model comparison on identical information.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Bench to the Future (BTF) is a 'pastcasting' benchmark for evaluating LLM forecasting agents. For each of 299 questions sourced from Metaculus (binary, multiple-choice, and numeric, expanded to 299 binary questions with an effective sample size of 178.8 after down-weighting constructed correlations), the authors provide a hermetic RetroSearch environment: a pre-crawled snapshot of roughly 20,000 web pages per question, with search mediated by a live Google search filtered to the stored pages. The paper evaluates several LLMs (Claude 3.5, 3.7, 4, Gemini 2.5 Pro) with three forecasting approaches (ReAct agent, Fixed Evidence, No Evidence) and reports Brier scores, calibration, inter-run variation, and a live-versus-retro validation on 20 separate questions. The central claims are that BTF is the first realistic, hermetic, and repeatable forecasting benchmark, and that pastcasting with RetroSearch can produce results comparable to live forecasting, enabling fast, contamination-free evaluations and tracking of capability progress.
Significance. If the central fidelity claim holds, BTF would be a valuable contribution: it would allow rapid, repeatable, and contamination-free evaluation of forecasting agents without waiting for resolutions. The methodology is careful in several respects: the authors explicitly weight questions that are correlated by construction (Eq. 2), justify the choice of averaging five runs with a bootstrap analysis, document the filtering of 11 malformed forecasts at a very low rate, and include a No Evidence baseline as a contamination control. The results—improvement across model generations and the advantage of ReAct over fixed-evidence pipelines—are plausible and consistent with prior work. However, the validity of the entire benchmark rests on the fidelity of the snapshots to the live information environment, and that assumption is not yet quantitatively established.
major comments (4)
- [§3.3] The central claim that 'our pastcasting environment can produce results comparable to those based on forecasts using the internet on at-the-time unresolved questions' (Abstract) is supported only by visual inspection of 20 live-versus-retro forecast distributions. No quantitative agreement metric is reported, and the No Evidence control is used only informally. The authors document one concrete failure (the 'Tush Push' question, where the decisive May 21 vote was absent from the May 20 snapshot), demonstrating that snapshot omissions can change forecasts. The paper should report a quantitative measure of live-retro agreement (e.g., a distributional distance such as the Wasserstein or Jensen-Shannon divergence, with the No Evidence baseline as a reference) and, more importantly, provide a retrieval-recall audit on a larger sample: for each question, check whether the pages or facts that live forecasters actually used are present in the RetroSearch corpus.
- [§2.1.2 and §4] The snapshot generation is an 'intelligent web crawl' that 'attempts to exhaustively search over the avenues a forecaster might take,' but the paper concedes in Section 4 that future agents may explore avenues the scraper does not anticipate. If the snapshots omit decisive facts, agent rankings on BTF may reflect the crawler's search heuristics rather than forecasting skill. To establish construct validity, the authors should provide evidence of snapshot completeness relative to the information used by successful live forecasters. Concretely, I would ask for an audit that, on a sample of resolved questions, checks whether the specific documents (or the specific facts) that were decisive for live forecasts are present in the corresponding snapshot; the current 20-question visual check is insufficient.
- [§3.2.1 and Table 5] The claim of a 'steady accuracy improvement between older models and newer models' is based on pairwise t-tests with no correction for multiple testing, and most adjacent-model comparisons are not statistically significant (e.g., Claude Sonnet 3.7 vs Claude Sonnet 4, p=0.1393; Claude Sonnet 3.7 vs Gemini 2.5 Pro, p=0.3536). With an effective sample size of 178.8 weighted questions and Brier-score differences of roughly 0.01-0.02 between adjacent models, the benchmark currently lacks the power to distinguish adjacent generations. This limits the paper's stated goal of 'track[ing] steady forecasting capability progress over time.' The authors should report multiple-testing-adjusted p-values or confidence intervals, and temper or qualify the progress-tracking claim accordingly.
- [§2.2 and §3.2.2] Claude Sonnet 4's official training-data cutoff is March 2025, which overlaps with the resolution window of the BTF question set (February-April 2025). The paper's checks (manual probing and the No Evidence baseline) are sensible, but they do not rule out partial leakage: a model with even fragmentary knowledge of outcomes could have its measured Brier score inflated, confounding the model-comparison results. I would like to see a sensitivity analysis, for example a version of the main benchmark restricted to questions whose resolutions are after the model's cutoff, or a comparison of model rankings after excluding questions where the No Evidence forecaster's accuracy is anomalously high.
minor comments (6)
- [Table 2] The entries 'Crawled URLsq114,390' and 'Crawled URLsq330,850' appear to be formatting errors for the first and third quartiles; please fix.
- [§2.5, Eq. (1)] The index i is used for both the summation and the question index; please clarify the notation and explicitly define the weighted mean Brier score used in the results.
- [§4] 'Not that biases do only invalidate the approach insofar they affect different models differently' appears to contain a typo; it should read 'Note that biases only invalidate the approach insofar as they affect different models differently.'
- [Figure 6] The caption says 'by model and question' but the panel structure is not described; please clarify how the 20 questions are aggregated in each histogram.
- [Abstract] 'hello@futuresearch.aito' should be 'hello@futuresearch.ai'.
- [§2.2] The phrase 'Claude models are instructed to say their training window cutoff is January 2025' is vague; please specify the source of this instruction (e.g., system prompt) and its implications for contamination risk.
Circularity Check
No circular derivation found: the pastcasting-vs-live claim is an empirical live/retro A/B comparison, not an identity or a fitted parameter; the main weakness is unmeasured snapshot completeness, which the paper itself concedes.
full rationale
The paper's central claim is that the RetroSearch pastcasting environment 'can produce results comparable to those based on forecasts using the internet on at-the-time unresolved questions' (Abstract). This is supported empirically in Sections 2.4.2 and 3.3 by running the same ReAct, Variable Evidence, and No Evidence forecasters in live and retro modes on 20 separate questions and visually comparing forecast distributions. This is an A/B test rather than a definitional identity: retro and live distributions could in principle diverge, and the paper reports one clear divergence (the Tush Push question, where scraping on May 20 missed the decisive May 21 vote). The No Evidence Forecaster control provides a noise floor for the live/retro comparison, indicating that agreement is measured rather than enforced by the setup. RetroSearch itself is cited to the authors' prior work [8], but the load-bearing validity evidence in this paper is the Section 3.3 comparison, so the self-citation is not doing the derivational work. No parameter is fitted to a subset and then renamed as a prediction; the Brier scoring (Equation 1) and weighting (Equation 2) are standard and applied uniformly. The main threat to the benchmark is construct validity, not circularity: the snapshot-crawl heuristic in Section 2.1.2 is assumed to contain the decision-relevant information a live forecaster would find, and Section 4 concedes that 'future agents will explore avenues our current scraper does not adequately anticipate.' The single documented Tush Push miss shows this failure mode is real, and the absence of a quantitative retrieval-recall audit weakens the fidelity claim; however, this is missing support for an empirical assumption, not a reduction of the result to its inputs. Under the stated rules, no circular step can be exhibited, so the score is in the 0–2 'no significant circularity' range; I assign 1 to reflect the minor self-citation and the acknowledged, non-circular validation gap.
Assumptions & free parameters
free parameters (4)
- numeric question split points =
unspecified, two sensible points within the overall range per question
- run count for final forecasts =
5
- question correlation weight =
w_i = log2(N_i + 1) / (N_i + 1)
- fixed evidence bundle size =
approximately 30 facts
assumptions (4)
- domain assumption For every question, the pre-crawled snapshot corpus contains the decision-relevant information a live forecaster would have had.
- domain assumption The evaluated LLMs do not have crystallized knowledge of post-cutoff outcomes.
- domain assumption Searching a filtered live Google index over pre-scraped pages introduces no material information leakage through result ordering or index changes.
- standard math Brier score and paired t-test statistics are appropriate tools for comparing forecasters on weighted, partially correlated questions.
invented entities (1)
-
RetroSearch environment
Cite this review
Pith. "Pith review of Bench to the Future: A Pastcasting Benchmark for Forecasting Agents." pith.science (2026). https://pith.science/paper/ABW2GBME
@misc{pith2026250621558,
author = {Pith},
title = {Pith review of: Bench to the Future: A Pastcasting Benchmark for Forecasting Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/ABW2GBME}},
note = {Machine review of arXiv:2506.21558}
}
read the original abstract
Forecasting is a challenging task that offers a clearly measurable way to study AI systems. Forecasting requires a large amount of research on the internet, and evaluations require time for events to happen, making the development of forecasting benchmarks challenging. To date, no forecasting benchmark provides a realistic, hermetic, and repeatable environment for LLM forecasters. We introduce Bench To the Future (BTF), a "pastcasting" benchmark with hundreds of high-quality questions for which the resolution is already known. Each question is accompanied by a large offline corpus of tens of thousands of relevant web pages, enabling a way to elicit realistic "forecasts" on past events from LLMs. Results suggest that our pastcasting environment can produce results comparable to those based on forecasts using the internet on at-the-time unresolved questions. We show results benchmarking agent and chain-of-thought forecasting approaches using several LLMs, including the recently-released Claude 4 models, and demonstrate BTF's ability to track steady forecasting capability progress over time. We intend this to be a living benchmark, with new questions added continually to account for increasing training data cutoff dates. We invite researchers to contact us at hello@futuresearch.ai to utilize our benchmark or tooling for their own research.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
Global Merger-Arbitrage Forecasting with Language Models
Expert-context research agents plus hindsight-guided finetuning cut class-balanced Brier score on merger outcomes to 0.151, beating calibrated market prices, XGBoost, and frontier LLMs.
-
Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts
A position paper advocating large-scale training of event forecasting LLMs, with proposals for label selection, counterfactual training data, auxiliary rewards, and multi-source datasets.
Reference graph
Works this paper leans on
-
[1]
Approaching Human-Level Forecasting with Language Models
Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. “Approaching Human-Level Forecasting with Language Models”. In:arXiv preprint arXiv:2402.18563(2024).URL: https://arxiv.org/abs/2402. 18563
arXiv 2024
-
[2]
Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy
Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, and Philip E. Tetlock. “Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy”. In:arXiv preprint arXiv:2402.19379 (2024).URL:https://arxiv.org/abs/2402.19379
arXiv 2024
-
[3]
Forecast- Bench: A Dynamic Benchmark of AI Forecasting Capabilities
Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, et al. “Forecast- Bench: A Dynamic Benchmark of AI Forecasting Capabilities”. In:International Conference on Learning Representations (ICLR). 2025.URL:https://arxiv.org/abs/2409.19839
arXiv 2025
-
[4]
Bosse.Comparing Two F orecasters in an Ideal World
Nikos I. Bosse.Comparing Two F orecasters in an Ideal World. Metaculus Notebook. 2023.URL: https : //www.metaculus.com/notebooks/19335/comparing-two-forecasters-in-an-ideal-world/
work page 2023
-
[5]
2024.URL: https://drive.google.com/file/d/1Tc_xY1NM-US4mZ4OpzxrpTudyo1W4KsE/view
Liam Phan, Aamir Khoja, Mantas Mazeika, and Dan Hendrycks.LLMs Are Superhuman F orecasters. 2024.URL: https://drive.google.com/file/d/1Tc_xY1NM-US4mZ4OpzxrpTudyo1W4KsE/view
work page 2024
-
[6]
Reasoning and Tools for Human-Level Forecasting
Elvis Hsieh, Preston Fu, and Jonathan Chen. “Reasoning and Tools for Human-Level Forecasting”. In:arXiv preprint arXiv:2408.12036(2024).URL:https://arxiv.org/abs/2408.12036
arXiv 2024
-
[7]
Quarterly forecasting tournaments
Metaculus.Metaculus AI F orecasting Benchmark Series. Quarterly forecasting tournaments. 2024.URL: https: //www.metaculus.com/aib/
work page 2024
-
[8]
FutureSearch, Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, et al.Deep Research Bench: Evaluating AI Web Research Agents. 2025. arXiv: 2506 . 06287 [cs.AI].URL: https : //arxiv.org/abs/2506.06287
arXiv 2025
Show all 11 references
-
[9]
Forecasting platform
Metaculus.Metaculus F orecasting Platform. Forecasting platform. 2025.URL: https://www.metaculus. com/
2025
-
[10]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, et al.ReAct: Synergizing Reasoning and Acting in Language Models. 2023. arXiv: 2210.03629 [cs.CL].URL: https://arxiv.org/ abs/2210.03629. 13 Bench to the Future
2023 arXiv
-
[11]
Verification of Forecasts Expressed in Terms of Probability
Glenn W. Brier. “Verification of Forecasts Expressed in Terms of Probability”. In:Monthly Weather Review 78.1 (Jan. 1950), pp. 1–3.ISSN: 1520-0493, 0027-0644.DOI: 10 . 1175 / 1520 - 0493(1950 ) 078<0001 : VOFEIT>2.0.CO;2. (Visited on 01/21/2022). 14 Bench to the Future A Appen...
1950
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.