REVIEW 8 cited by
ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Forecasts of future events are essential inputs into informed decision-making. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no framework for evaluating the accuracy of ML systems on a standardized set of forecasting questions. To address this gap, we introduce ForecastBench: a dynamic benchmark that evaluates the accuracy of ML systems on an automatically generated and regularly updated set of 1,000 forecasting questions. To avoid any possibility of data leakage, ForecastBench is comprised solely of questions about future events that have no known answer at the time of submission. We quantify the capabilities of current ML systems by collecting forecasts from expert (human) forecasters, the general public, and LLMs on a random subset of questions from the benchmark ($N=200$). While LLMs have achieved super-human performance on many benchmarks, they perform less well here: expert forecasters outperform the top-performing LLM ($p$-value $<0.001$). We display system and human scores in a public leaderboard at www.forecastbench.org.
Forward citations
Cited by 8 Pith papers
-
Predicting Empirical AI Research Outcomes with Language Models
A fine-tuned, retrieval-augmented language model predicts which of two AI research ideas will perform better empirically, beating human experts and frontier models on a new contamination-controlled benchmark.
-
Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities
In two adversarial real-world domains with time-delayed expert ground truth, frontier LLMs over-generate plausible ideas but under-select the specific solutions experts adopted, indicating a filtering and prioritization gap.
-
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
A pre-registered benchmark of 13 LLM systems on all 104 World Cup 2026 matches shows fine-grained predictions expose differences that result accuracy hides.
-
FIFA World Cup 2026 as a Contamination-Free Benchmark for LLM Forecasting Agents: Four Models, a Bookmaker, and 104 Matches
On 104 World Cup matches, four LLM forecasting agents make identical top picks in 92% of matches, none beats the betting market's Brier score, but their betting ROI spans -18% to +10%.
-
Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agents
An explicit selection–sizing–risk trading layer on fixed forecasts yields the only positive ROI and stake-weighted Sharpe among five policies on a Polymarket decision archive.
-
Bench to the Future: A Pastcasting Benchmark for Forecasting Agents
Bench to the Future is a pastcasting benchmark: 299 already-resolved forecasting questions, each paired with a frozen corpus of about 20,000 web pages, on which newer LLMs and agentic search score better.
-
Pitfalls in Evaluating Language Model Forecasters
A systematic critique showing temporal leakage and extrapolation flaws can undermine claims that LLM forecasters match or beat humans.
-
Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematics
A self-evolving curriculum that retrains a language model on variants of problems it can mostly get right lifts AIME pass@1 from 5.6% to 16.5%, beating static augmentation under the same data budget.
Discussion (0). Continue with ORCID to comment.