Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

LLMs Can Teach Themselves to Better Predict the Future

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that language models can improve their own probabilistic forecasting by fine-tuning on their self-generated reasoning traces, ranked only by how close each forecast landed to the realized outcome, and that this lifts two…

desk verdict A genuinely new outcome-ranked self-play + DPO pipeline for forecasting, with the right control — but the headline 7-10% gain is likely inflated by test-set-based epoch selection. read the letter →

arxiv 2502.05253 v1 pith:CPVBP4VG submitted 2025-02-07 cs.CL cs.AI

Benjamin Turtel , Danny Franklin , Philipp Schoenegger This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMforecastingDirectPreferenceOptimizationself-playfine-tuningoutcome-drivenlearningBrierscorepredictionmarketsprobabilisticreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a language model can improve its probabilistic forecasting by fine-tuning on nothing more than its own reasoning traces and the eventual yes/no resolution of prediction-market questions. The authors generate two reasoning-and-forecast traces per question from a 14-billion-parameter model, rank each pair by how close the predicted probability was to the realized outcome, and train with Direct Preference Optimization (DPO) on those ranked pairs. On a held-out set of 2,300 questions that resolve after the training data, the fine-tuned Phi-4 14B and DeepSeek-R1 14B models achieve mean Brier scores of 0.200 and 0.197, beating their base versions and randomized-label controls by 7–10% and statistically matching GPT-4o. If this stands, it means models can keep improving their judgment about future events using only public resolutions and their own generation, without human-curated rationales or crowd forecasts.

What carries the argument

The load-bearing object is the outcome-ranked preference pair: two self-generated reasoning traces attached to the same resolved question, ordered by $r(p,o)=|p-o|$, the absolute distance between each trace's probability $p$ and the binary outcome $o$. Direct Preference Optimization (DPO) turns these ordered pairs into a policy update without any human reward model, so the model is pushed to prefer reasoning that would have produced a probability closer to what actually happened. The same ranking metric also supplies the randomized-label control: shuffling the ordering removes the outcome signal and isolates the effect of training on the reasoning/news exposure alone.

What would settle it

Shuffle the realized outcomes before computing $r(p,o)$ while keeping every other step identical; if the fine-tuned model still beats the base and the randomized-label control, then outcome proximity is not the active ingredient, and the claimed discovery fails.

Watch

Extended reading notes

Core claim

The central discovery is that outcome proximity is a usable preference signal for fine-tuning forecasters. For each of 9,427 training questions, the model generates two reasoning traces and two probabilities $p_1,p_2$; ranking them by $r(p,o)=|p-o|$ against the realized outcome $o\in\{0,1\}$ creates preference pairs, and DPO over those pairs shifts the model toward reasoning patterns that produce probabilities closer to the truth. The authors report that this lowers mean Brier score from 0.221 to 0.200 for Phi-4 14B and from roughly 0.212 to 0.197 for DeepSeek-R1 14B, with the improvement over both base and randomized-label controls significant at $p<0.05$ after correction for multiple comparisons. The fine-tuned models also show no significant difference from GPT-4o ($p>0.7$), so the method is claimed to bring 14B models to frontier-model forecasting quality.

Load-bearing premise

The load-bearing premise is that ranking a reasoning trace by how close its probability happened to land to the realized yes/no outcome is a valid preference signal; for a single event that distance is mostly noise, and the method depends on averaging this signal over 9,427 questions to learn anything.

Editorial extensions

If this is right

  • Phi-4 14B's fine-tuned Brier score of 0.200 beats its base (0.221) and randomized-label control (0.214), with adjusted p-values of 0.002 and 0.027.
  • DeepSeek-R1 14B's fine-tuned Brier score of 0.197 beats its base and control (both near 0.212), with adjusted p-values of 0.027 in each comparison.
  • Neither fine-tuned model differs significantly from GPT-4o (adjusted p > 0.7), so the paper claims 14B models reach frontier-level forecasting accuracy.
  • The gain concentrates in very accurate forecasts: the fine-tuned Phi-4 has 35.7% of Brier scores below 0.05 versus about 21% for base/control, while DeepSeek-R1 has 32.78% versus about 23%.
  • Because the training data are generated by the model itself and resolved outcomes are publicly available, the pipeline can in principle be re-run on new questions as they resolve, without human-written rationales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a concrete scaling recipe the authors did not test: run the same self-play/DPO loop repeatedly on rolling windows of resolved questions, and forecasting accuracy may continue to climb as the model encounters new outcome feedback.
  • The randomized-label control improved Phi-4 over base (0.214 vs 0.221) but not DeepSeek-R1, which the authors attribute to news exposure; an ablation that removes news summaries entirely would test whether outcome ranking alone, rather than fine-tuning on news-bearing prompts, drives the gain.
  • The ranking criterion $|p-o|$ is one of several possible outcome losses; squared error or log loss would emphasize calibration at extreme probabilities differently, and the paper does not compare them, leaving room for a cheap experiment.
  • A practical deployment consequence is that the method inherits any biases in prediction-market question selection and in news summaries, so safeguards would be needed before using it for high-stakes institutional forecasting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an outcome-driven fine-tuning framework in which an LLM generates two reasoning traces per forecasting question, ranks those traces by their absolute distance to the resolved binary outcome, and is then fine-tuned with DPO on the resulting preference pairs. The method is applied to Phi-4 14B and DeepSeek-R1 14B on Polymarket questions, with a temporally held-out test set resolving after the training set. The authors report mean Brier improvements of 7--10% over the base models and over randomized-label DPO controls, and report that the fine-tuned small models perform on par with GPT-4o on this test set.

Significance. If the central result holds, this is a genuinely useful and scalable contribution: it removes the need for human-curated rationales, uses only resolved outcomes as supervision, and includes a randomized-label control that strengthens the causal interpretation. The temporal split between training and test sets is appropriate, and the comparison against both base and control models is the right design. The main risk is statistical: the paper does not describe a validation split, and the per-epoch model selection described in Section 2.5 may have used the test set, which would make the reported p-values conditional on test-set performance. The ranking-metric concern is real but is partially mitigated by the randomized-label control; the evaluation-protocol concern is not addressed.

major comments (3)
  1. [Section 2.5 / Figure 3 / Section 2.1] The paper does not define a validation split. Section 2.1 partitions data into 9,800 training and 2,300 test questions, while Section 2.5 states that "we found a plateau at the fifth epoch" for Phi-4 and "the fourth epoch" for DeepSeek-R1, with Figure 3 plotting per-epoch Brier scores. If the per-epoch scores in Figure 3 are computed on the test set, then the epoch was selected using the test set, and the final reported Brier scores and t-test p-values are the result of a selection procedure not applied to the base or control models. This would bias the comparison in favor of the fine-tuned models and invalidate the reported p-values. The authors must state explicitly which set was used to produce Figure 3 and, if it is the test set, provide a validation-based or pre-specified epoch selection, or re-run the evaluation on a truly held-out set.
  2. [Section 2.4, Eq. (1)] The ranking metric r(p,o)=|p-o| uses the realized binary outcome as the quality signal for a single noisy event. For a single question, a forecast closer to the outcome can be closer by luck rather than by reasoning quality, so the preference pair may encode noise. Averaging over 9,427 questions and the randomized-label control mitigate this, but the paper should provide additional robustness evidence that the preference signal is not dominated by noise, for example by reporting results when only pairs with a minimum probability margin are used, or by showing how often the ranked order changes under slight perturbations of the outcomes. Without such a check, the mechanism claimed for DPO remains a correctness-risk.
  3. [Section 3, Table 3] All models are evaluated on the same 2,300 test questions, so the observations are paired across models. The independent-samples t-tests ignore this pairing. The p-values should be recomputed with paired t-tests or a question-level block bootstrap, and the paper should report effect sizes and confidence intervals for the mean Brier differences. This is not necessarily fatal to the conclusions, but it is required for the reported inferential claims.
minor comments (6)
  1. [Section 2.4] The handling of ties in the ranking metric is not described; for example, with outcome o=1 and forecasts p1=0.6 and p2=0.4, both have r=0.4. The paper should state whether tied pairs are removed, kept with a consistent order, or otherwise handled.
  2. [Section 3, footnote 1] The hypothesis that the Phi-4 control improves over base because of an earlier knowledge cutoff is speculative and is not tested. It should be presented as a conjecture or supported with an analysis.
  3. [Figure 3] Figure 3 does not show which data split the per-epoch Brier scores were computed on, and the y-axis range obscures differences. Adding the split and epoch counts would make the model-selection procedure transparent.
  4. [Table 3] The p-value reported as 0.000 for Phi-4 Fine-Tune vs. Base should be reported as p < 0.001.
  5. [Section 2.3] The text reports 9,427 training questions after removing questions with identical forecasts, while Section 2.1 states 9,800 training questions. The exclusion procedure and resulting number should be clarified.
  6. [Section 2 / References] Minor typo: "Gwen2.5-14B" should be "Qwen2.5-14B" in Section 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: external resolved outcomes provide the preference signal, the test set is temporally disjoint, and a randomized-label control isolates the learning effect.

full rationale

The preference signal in Section 2.4, r(p,o)=|p-o|, is computed from externally resolved Polymarket outcomes rather than from model outputs, and it is applied only to the 9,427 training questions; the 2,300 test questions resolve after the training set's outcome window (Section 2.1), so the reported Brier scores are not defined in terms of the training labels. The inclusion of a randomized-label DPO control (Section 2.4) directly tests whether any improvement is attributable to the outcome-based ranking, and the comparison against base and control models on the same held-out set gives the central claim independent content. The only self-citation, [6] (Schoenegger et al.), appears in a related-work list about ensemble aggregation and is not load-bearing. A real methodological concern remains: Section 2.5 reports per-epoch Brier plateaus without describing a validation split, so if those curves were computed on the test set, the epoch choice would inflate the reported improvement; this is a statistical validity issue about test-set reuse, not a definitional circularity, because no equation in the paper defines the test predictions in terms of the training outcomes.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of outcome-based preference ranking, the accuracy of Polymarket resolutions, and the absence of leakage in the temporal split. The DPO training epoch count is effectively tuned on the test set, which is the main free parameter affecting the reported effect size.

free parameters (3)
  • DPO training epochs = 5 (Phi-4), 4 (DeepSeek-R1)
    Chosen by observing the per-epoch Brier plateau in Figure 3; if evaluated on the test set, this is a test-set-tuned hyperparameter.
  • Maximum self-play re-runs to obtain a second forecast = 4
    Hand-chosen cap; determines which questions enter training and the diversity of preference pairs.
  • News lookback window = 14 days
    Chosen without formal justification; affects the information available in prompts.
assumptions (4)
  • domain assumption Polymarket binary resolutions are accurate ground truth.
    All training and test labels come from Polymarket resolutions; any resolution error propagates into preferences and evaluation. Section 2.1.
  • domain assumption News summaries retrieved 14 days before resolution provide relevant, non-leaking information.
    Test questions resolve after the final training outcome, but the paper does not verify that no test-period news leaked into model pretraining or news retrieval. Sections 2.2 and 2.6.
  • ad hoc to paper Ranking reasoning traces by absolute distance to the realized outcome is a valid preference signal.
    For a single binary event, a forecast closer to the outcome is not necessarily a better forecast; the paper assumes this noise averages out over 9,427 questions. Section 2.4, r(p,o)=|p-o|.
  • standard math Brier score is an appropriate accuracy metric for this claim.
    Brier is a proper scoring rule, but the paper does not report calibration or discrimination, so improvements may reflect prior shifts. Section 2.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs Can Teach Themselves to Better Predict the Future." pith.science (2026). https://pith.science/paper/CPVBP4VG

@misc{pith2026250205253,
  author       = {Pith},
  title        = {Pith review of: LLMs Can Teach Themselves to Better Predict the Future},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CPVBP4VG}},
  note         = {Machine review of arXiv:2502.05253}
}
read the original abstract

We present an outcome-driven fine-tuning framework that enhances the forecasting capabilities of large language models (LLMs) without relying on human-curated reasoning samples. Our method leverages model self-play to generate pairs of diverse reasoning trajectories and probabilistic forecasts for a set of diverse questions that resolve after the models' knowledge cutoff date. We then rank pairs of these reasoning traces by their distance to the actual outcomes before fine-tuning the model via Direct Preference Optimization (DPO). On a separate test set, our approach increases prediction accuracy of Phi-4 14B and DeepSeek-R1 14B by between 7--10\% over a base model and a DPO fine-tuned control model with randomized labels, bringing them on par with forecasting capabilities of much larger frontier models like GPT-4o.

Figures

Figures reproduced from arXiv: 2502.05253 by the authors.

Figure 1
Figure 1. Overview Flowchart Question Generation Polymarket 12,100 questions Binary outcomes NewsCatcher API News Summaries Prompt Creation (Questions + News) LLM Self-Play 2 reasoning traces per question DPO Training Reranking Responses by Accuracy Fine-tuned LLM Notes: This chart outlines the stages of our method. There has been some work explicitly looking to apply and boost the forecasting capabilities of LLMs. Such work … view at source ↗
Figure 2
Figure 2. Accuracy Results for all Models DeepSeek-R1 14B Fine-tuned DeepSeek-R1 14B Control DeepSeek-R1 14B Base Phi-4 14B Fine-tuned Phi-4 14B Control Phi-4 14B Base GPT4o Model 0.15 0.16 0.17 0.18 0.19 0.20 0.21 0.22 0.23 Mean Brier Score Mean Brier Scores with SEM Notes: The bar graph shows mean Brier scores with standard error of the mean (SEM) error bars. The y-axis starts at 0.15. Lower Brier scores indicate higher acc… view at source ↗
Figure 3
Figure 3. Per-Epoch Accuracy. 1 2 3 4 5 6 7 8 9 Epoch 0.200 0.205 0.210 0.215 0.220 Brier Score Per-Epoch Brier Scores for Fine-tuning Phi-4 14B Fine-tune DeepSeek-R1 14B Fine-tune Notes: This plot shows the per-epoch accuracy results for both Phi-4 14B and DeepSeek-R1 14B. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Forecasting Prompts by Model Phi-4 14B: [Question, Question Background, Resolution Criteria, Today’s/Question Close Date, News Summaries] Instructions: 1. Given the above question, rephrase and expand it to help you do better answering. Maintain all information in the …
Figure 5
Figure 5. Figure 5: Ridge Plot of Forecasting Accuracy for each Model. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prompt Engineering Large Language Models' Forecasting Capabilities

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Across two preregistered studies and six LLMs, most prompt engineering variations produced no reliable improvement in forecasting accuracy, while Bayesian-style prompts consistently hurt performance.

  2. Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A position paper advocating large-scale training of event forecasting LLMs, with proposals for label selection, counterfactual training data, auxiliary rewards, and multi-source datasets.

Reference graph

Works this paper leans on

27 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    Karger, H

    E. Karger, H. Bastani, C. Yueh-Han, Z. Jacobs, D. Halawi, F. Zhang, and P. E. Tetlock. Forecastbench: A dynamic benchmark of ai forecasting capabilities, 2024. arXiv preprint arXiv:2409.19839

    arXiv 2024

  2. [2]

    P. E. Tetlock and D. Gardner.Superforecasting: The art and science of prediction . Random House, 2016

    work page 2016

  3. [3]

    A. Kim, M. Muhn, and V. Nikolaev. Financial statement analysis with large language models, 2024. arXiv preprint arXiv:2407.17866

    arXiv 2024

  4. [4]

    X. Wang, M. Feng, J. Qiu, J. Gu, and J. Zhao. From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection, 2024. arXiv preprint arXiv:2409.17515

    arXiv 2024

  5. [5]

    C. Cao, J. Zhuang, and Q. He. Llm-assisted modeling and simulations for public sector decision-making: Bridging climate data and policy insights. InAAAI-2024 Workshop on Public Sector LLMs: Algorithmic and Sociotechnical Design, 2024

    work page 2024

  6. [6]

    Schoenegger, I

    P. Schoenegger, I. Tuminauskaite, P. S. Park, and P. E. Tetlock. Wisdom of the silicon crowd: Llm ensemble prediction capabilities rival human crowd accuracy, 2024. arXiv preprint arXiv:2402.19379

    arXiv 2024

  7. [7]

    Halawi, F

    D. Halawi, F. Zhang, C. Yueh-Han, and J. Steinhardt. Approaching human-level forecasting with language models, 2024. arXiv preprint arXiv:2402.18563

    arXiv 2024

  8. [8]

    Q. Yan, R. Seraj, J. He, L. Meng, and T. Sylvain. Autocast++: Enhancing world event prediction with zero-shot ranking-based context retrieval, 2023. arXiv preprint arXiv:2310.01880

    work page Pith review arXiv 2023

Show all 27 references
  1. [9]

    Q. Lyu, K. Shridhar, C. Malaviya, L. Zhang, Y. Elazar, N. Tandon, and C. Callison-Burch. Calibrating large language models with sample consistency, 2024. arXiv preprint arXiv:2402.13904

    2024

  2. [10]

    Silver, T

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. arXiv preprint arXiv:1712.01815

    2017 arXiv

  3. [11]

    Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. arXiv preprint arXiv:2401.01335

    2024 arXiv

  4. [12]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, volume 36, 2024

    2024

  5. [13]

    S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, and Y. Wu. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024. arXiv preprint arXiv:2404.10719

    2024 arXiv

  6. [14]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, and Y. He. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. arXiv preprint arXiv:2501.12948

    2025 arXiv

  7. [15]

    Abdin, J

    M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, and Y. Zhang. Phi-4 technical report,

  8. [16]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, and I. Kivlichan. Gpt-4o system card, 2024. arXiv preprint arXiv:2410.21276. 9

    2024 arXiv

  9. [17]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. arXiv preprint arXiv:2311.12022

    2023 arXiv

  10. [18]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. arXiv preprint arXiv:2103.03874

    2021 arXiv

  11. [19]

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, and Z. Qiu. Qwen2.5 technical report, 2024. arXiv preprint arXiv:2412.15115

    2024 arXiv

  12. [20]

    M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, and A. Odena. Show your work: Scratchpads for intermediate computation with language models, 2021. arXiv preprint arXiv:2112.00114

    2021 arXiv

  13. [21]

    Munos, M

    R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, C. Fiegel, A. Michi, M. Selvi, S. Girgin, N. Momchev, O. Bachem, D. J. Mankowitz, D. Precup, and B. Piot. Nash learning from human feedback, 2023. arXiv preprint arXiv:2312.00886

    2023 arXiv

  14. [22]

    B. Xu, A. Yang, J. Lin, Q. Wang, C. Zhou, Y. Zhang, and Z. Mao. Expertprompting: Instructing large language models to be distinguished experts, 2023. arXiv preprint arXiv:2305.14688

    2023 arXiv

  15. [23]

    R. H. Pherson and R. J. Heuer.Structured analytic techniques for intelligence analysis . Cq Press, 2019

    2019

  16. [24]

    Benjamini and Y

    Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing.Journal of the Royal Statistical Society: Series B (Methodological) , 57(1):289–300, 1995

    1995

  17. [25]

    O. Zem. Exploring the impact of quantization on llm performance.https://medium.com/@olga.zem/ exploring-the-impact-of-quantization-on-llm-performance-5698e16c5564 , January 3 2024. Ac- cessed: 2024-01-03

    2024

  18. [26]

    Huang, X

    W. Huang, X. Zheng, X. Ma, H. Qin, C. Lv, H. Chen, and M. Magno. An empirical study of llama3 quantization: From llms to mllms.Visual Intelligence, 2(1):36, 2024. 10

    2024

  19. [2024]

    arXiv preprint arXiv:2412.08905

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.