REVIEW 6 cited by
FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce $\textbf{FutureX}$, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.
Forward citations
Cited by 6 Pith papers
-
Scientific reasoning does not reliably translate into scientific forecasting in frontier AI
Introduces the CUSP benchmark across 4760 events and finds frontier AI models can pick plausible directions but fail to predict whether or when scientific advances will occur, with performance varying by domain and in...
-
Decentralized Aggregation of LLM Predictions via Wagering Mechanisms
A leave-one-out wagering payout makes LLM aggregation weights equal expected score advantage, yielding DSIC predictions, decentralized wager learning, and performance matching centralized routers.
-
Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs
BLF achieves state-of-the-art binary forecasting on ForecastBench by using linguistic belief states updated in tool-use loops, hierarchical multi-trial logit averaging, and hierarchical Platt scaling calibration.
-
AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models
LLMs are unreliable when asked to emit buy/sell/hold actions, so this paper benchmarks them as code-writing quantitative researchers whose generated strategies are backtested deterministically.
-
Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery
DBench-Bio builds a dynamic biology benchmark from post-release abstracts, but LLM-generated gold answers and unverified per-model temporal separation undermine its claim to measure knowledge discovery.
-
FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
FutureOmni, a 919-video, 1,034-question audio-visual future-forecasting benchmark, shows top MLLMs reach only 64.8% accuracy, and OFF tuning improves open models.
Discussion (0). Continue with ORCID to comment.