Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Analyzing the Role of Context in Forecasting with Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On a new dataset of 614 resolved binary Metaculus questions, the paper shows that adding news-article summaries to LLM prompts improves forecasting accuracy for all three tested models, while adding few-shot examples reduces it.

desk verdict The news-context gains are probably outcome leakage, but the dataset and systematic prompt ladder are worth a re-run under proper temporal filtering. read the letter →

arxiv 2501.06496 v1 pith:AZMMSTIO submitted 2025-01-11 cs.CL cs.IR

Gerrit Mutschlechner , Adam Jatowt This is my paper · ORCID
classification cs.CLcs.IR
keywords LLMforecastingbinaryquestionsnewsarticlecontextfew-shotpromptingMetaculusdatasetpromptdesigntemporalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether giving large language models more context helps them forecast binary real-world events. Using a new dataset of 614 resolved Metaculus questions, each paired with background information, three news-article summaries, and resolution criteria, it measures accuracy under five prompts of increasing context for three models (GPT-3.5-turbo, Alpaca-7B, Llama2-13B-chat). The central finding is that adding news articles lifts accuracy for every model, while adding few-shot examples lowers accuracy. A sympathetic reader would take this as evidence that current LLMs can be steered toward useful forecasts mainly by supplying timely, relevant news context, and that prompt length and exemplars can hurt.

What carries the argument

The central object is the context ladder: five prompts that start with a bare binary question and add, in order, background text, three news-article summaries, resolution criteria, and two solved few-shot examples. The ingredient that does the work is the news-article summary: each question is linked to at least three news articles published at least five days before the question's resolution date, and each article is condensed by GPT-3.5-turbo into a few question-relevant sentences. These summaries are what shift models away from an almost universal 'no' prediction and toward answers that track the true balance of the dataset. The paper also contributes the dataset itself, a collection of 614 recent, resolved, binary questions with background, resolution criteria, news URLs, and summaries.

What would settle it

Annotate a sample of the article summaries for whether they state the event's outcome; if summaries published after the event date but before resolution reveal the answer, the news-context accuracy gain is partly answer retrieval. A direct experiment would rerun the prompts on a subset where every article predates the event in question and compare the accuracy gains.

Watch

Extended reading notes

Core claim

On a dataset of 614 binary forecasting questions drawn from Metaculus, with questions created after October 2021 and already resolved, the authors evaluate forecasting accuracy under five prompt conditions: question only (Q); question plus background (B); background plus three news-article summaries (NA); background, news, and resolution criteria (R); and all of the above plus two few-shot examples (FS). Accuracy for GPT-3.5-turbo rises from 0.64 with the bare question to 0.68 with background and news; Alpaca-7B rises from 0.49 to 0.63; Llama2-13B-chat rises from 0.57 to 0.68. The prompting stage that includes news articles is the best or tied-best for each model, and adding few-shot examples consistently reduces accuracy by a few points. With only the question, the two larger models answer 'no' for nearly every item; supplying context rebalances the prediction mix toward the dataset's actual yes/no distribution.

Load-bearing premise

The setup assumes that news articles published at least five days before a question's resolution date do not contain information about the actual outcome, even though Metaculus resolves questions after the event has occurred and post-event articles can therefore reveal the answer.

Editorial extensions

If this is right

  • Forecasting systems built on LLMs should supply current news summaries as context: they raised accuracy by 4 to 14 percentage points across the three models tested.
  • Few-shot examples should be omitted or tested carefully: adding them lowered accuracy for all three models.
  • Model scale helps: the 13B-parameter Llama2-chat outperformed the 7B Alpaca in the best-context condition, and GPT-3.5-turbo led overall.
  • The released dataset gives the community a recent binary-forecasting benchmark (614 questions, 2021–2024) with background text, resolution criteria, and news-article links.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The five-day publication gap does not rule out leakage, because Metaculus resolves after the fact; if post-event articles are among those summarized, part of the news-context benefit may be retrieval rather than forecasting. This is my editorial reading, not the paper's claim.
  • The strong default-to-'no' behavior in the bare-question condition suggests these models are not genuinely reasoning about the question as a forecast; context may help mainly by breaking that default, implying that prompt design can dominate model reasoning in this task.
  • Filtering for questions with at least three findable news articles likely selects for media-covered, event-heavy topics, so the measured gains may not transfer to quieter or slower-moving forecasts.
  • A natural extension would vary the publication cutoff (e.g., 30 days before resolution or before the event's start) to test whether accuracy gains persist; if they vanish, the gains are leakage-driven.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces a dataset of 614 resolved binary forecasting questions from Metaculus, each augmented with background text, resolution criteria, and three Google News articles summarized with GPT-3.5-turbo. The authors evaluate five prompt conditions (Q; Q,B; Q,B,NA; Q,B,NA,R; Q,B,NA,R,FS) on GPT-3.5-turbo, Alpaca-7B, and Llama2-13B-chat. They report that adding news articles improves accuracy, adding few-shot examples lowers accuracy, and larger models outperform smaller models.

Significance. The dataset and prompt-ablation design are potentially useful for the forecasting community, and the paper is transparent about model versions and includes confusion matrices and horizon/duration analyses. However, the main empirical claims are not supported by the evidence as presented. The news-article filter in Section 3 permits articles that postdate the event but predate Metaculus's resolution, so the 'news improves forecasting' result may simply reflect outcome retrieval. The statistical analysis lacks baselines and significance tests, and the cross-model comparison is confounded by different question subsets. If the leakage issue were corrected and the analyses redone, the dataset could still be a valuable resource, but the current manuscript's conclusions cannot be accepted.

major comments (4)
  1. [Section 3; Appendix A.1] The dataset-construction rule that news articles must be published at least five days before the resolution date does not prevent information leakage. Metaculus resolves a question only after the outcome is known, and the outcome event can occur well before resolution. A news article published after the event but before the resolution date can therefore report the ground-truth answer. Because prompts 3-5 feed the article summaries directly to the model, the accuracy gains reported in Table 3 (e.g., GPT-3.5-turbo 0.64 to 0.68; Alpaca-7B 0.49 to 0.62; Llama2-13B-chat 0.57 to 0.68) are exactly what answer retrieval would produce. Appendix A.1 records Created time, Publish time, and Resolve time, so the filter should have required article publish dates to precede the question's creation or publish date (or the event window). Section 6 does not mention this leakage window, and the central claim of the abstract depends on it.
  2. [Section 5; Table 3; Figure 1] The claim that news context 'significantly improves' performance is not supported by any statistical evidence. No error bars, confidence intervals, or significance tests are reported. For GPT-3.5-turbo, the Q-only condition achieves 0.64 accuracy while predicting 'no' on almost all questions (recall 0.04, see Figure 1), which is essentially the majority-class baseline (391/614 = 0.64). The improvement to 0.68 is about 4 percentage points on n=614, within roughly two standard errors and not demonstrated to be reliable. For Alpaca-7B and Llama2-13B-chat the sample sizes are 249 and 154, making the reported differences even more uncertain. The word 'significantly' in the abstract should be removed or justified with a proper test.
  3. [Section 5; Appendix B.5] The few-shot result is confounded. Only two fixed examples are used, and no ablation separates the effect of the examples' content from the increased prompt length. The paper attributes the decline to 'input prompts becoming quite large,' but with two hand-picked examples this explanation is not distinguishable from example-specific bias. Moreover, the observed differences (e.g., 0.68 to 0.67 for GPT-3.5-turbo, 0.63 to 0.59 for Alpaca-7B, 0.61 to 0.57 for Llama2-13B-chat) are small relative to the sample sizes, and the conclusion that few-shot examples 'lead to a decline' is not statistically established.
  4. [Section 4; Table 3] The claim that larger models outperform smaller models is not supported by a controlled comparison. Alpaca-7B is evaluated on the 249 questions created after March 2023, while Llama2-13B-chat is evaluated on the 154 questions created after August 2023; the sets differ in time period, topic mix, and difficulty. GPT-3.5-turbo is evaluated on all 614 questions. Thus accuracy differences across models are confounded with data distribution and cannot be attributed to model size. A matched evaluation on the same question subset is needed.
minor comments (6)
  1. [Section 3] The sentence 'The news article must be published at least five days before forecasting questions’s resolution date' has a grammar issue ('questions’s'); please revise and clarify whether the five-day gap is measured relative to question creation or resolution.
  2. [Section 2] The citation 'Yuan et al. Yuan et al. (2023)' is duplicated; it should read 'Yuan et al. (2023)'.
  3. [Table 3] The row labels in the first column run into the numbers (e.g., 'Q,B,NA,R,FS0.67'); add a space or delimiter for readability.
  4. [Appendix A.1] Appendix A.1 lists both a Newspaper3k 'Summary' and a GPT-3.5-turbo 'Summary LLM'; the paper says summaries are generated with GPT-3.5-turbo, but it should state explicitly which field is inserted into Prompts 3-5.
  5. [Section 6] The phrase 'news genre' should likely be 'news articles' or 'the news genre'; please revise for clarity.
  6. [Dataset Availability] The dataset is not made publicly available ('can be obtained after contacting authors'); for reproducibility, the dataset and code should be released.

Circularity Check

1 steps flagged · score 6.0 of 10

News-context gains are answer retrieval because the five-day pre-resolution filter allows articles published after the event to contain the outcome.

  1. other [Section 3, Dataset Creation (news filtering); Appendix A.1 (Created/Publish/Resolve time); Appendix B.3-B.5 (Prompts 3-5); Table 3]
    "The news article must be published at least five days before forecasting questions's resolution date, to prevent information leakage. We choose this gap because questions are resolved by the Metaculus team when sufficient evidence is available, which may take some time."

    The filter is anchored to the resolution date, not to the event date or the question creation date. Since Metaculus resolves questions only after the outcome is known, an article published after the event but before resolution can state the answer. Such articles (or their GPT-3.5-turbo summaries) are inserted into Prompts 3-5, so the LLM is asked to 'forecast' an outcome that is already present in its input. The headline gains—GPT-3.5-turbo 0.64 to 0.68, Alpaca-7B 0.49 to 0.62, Llama2-13B-chat 0.57 to 0.68—are then explainable by retrieval rather than forecasting. The dataset even records Created time and Publish time, so the missing pre-outcome check is a design gap, not an unavailable datum.

full rationale

The paper's contribution is an empirical comparison of prompts; there is no formal derivation and no load-bearing self-citation. The central claim, however, that adding news articles significantly improves accuracy, depends on the dataset rule that news articles must predate the resolution date by five days. Because Metaculus resolution happens only after sufficient evidence exists, this rule does not exclude articles that postdate the event and report its outcome. Prompts 3-5 then contain the ground-truth answer in the supplied context, making the measured improvement consistent with answer retrieval. This is an input-leakage circularity in the empirical sense: the 'prediction' is present in the prompt. The paper's limitation section does not mention this window, so the concern is not acknowledged. Score 6 reflects that the central news-context result partially reduces to leakage, while the dataset and other prompt comparisons (e.g., few-shot decline) still have independent content.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several untested design assumptions rather than on fitted parameters. The most important is that the 5-day-before-resolution rule prevents outcome leakage, which is likely false since Metaculus resolves after the outcome is known. The paper also assumes training-cutoff dates cleanly separate seen from unseen information, that the GPT-3.5-turbo-generated summaries do not leak outcomes, and that the two fixed few-shot examples are neutral. None of these are verified with controls.

assumptions (4)
  • domain assumption News articles published at least five days before the resolution date do not leak the outcome.
    Stated in Section 3 as the leakage-prevention rule; the resolution date occurs after the event outcome, so the rule does not prevent articles from reporting the result. This is load-bearing for interpreting the news-context results as forecasting.
  • domain assumption The training cutoff dates of the models are reliable anchors for ensuring the model does not know the answer.
    Section 4; the paper assumes questions created after the cutoff are unseen, but cutoff dates are approximate and the models may have seen related information.
  • domain assumption GPT-3.5-turbo-generated summaries faithfully condense the articles without adding outcome information.
    Section 3; the summarizer is the same model family evaluated, and the paper assumes no leakage through the summaries.
  • ad hoc to paper The two few-shot examples are representative and their content does not bias responses.
    Appendix B; the examples are hand-written by the authors (COVID hospitalizations and UK by-election), and the paper does not ablate over example choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing the Role of Context in Forecasting with Large Language Models." pith.science (2026). https://pith.science/paper/AZMMSTIO

@misc{pith2026250106496,
  author       = {Pith},
  title        = {Pith review of: Analyzing the Role of Context in Forecasting with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZMMSTIO}},
  note         = {Machine review of arXiv:2501.06496}
}
read the original abstract

This study evaluates the forecasting performance of recent language models (LLMs) on binary forecasting questions. We first introduce a novel dataset of over 600 binary forecasting questions, augmented with related news articles and their concise question-related summaries. We then explore the impact of input prompts with varying level of context on forecasting performance. The results indicate that incorporating news articles significantly improves performance, while using few-shot examples leads to a decline in accuracy. We find that larger models consistently outperform smaller models, highlighting the potential of LLMs in enhancing automated forecasting.

Figures

Figures reproduced from arXiv: 2501.06496 by the authors.

Figure 1
Figure 1. Ratio of questions forecasted as ’no’ across [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of categories. B Prompts We use the subsequent prompts to conduct the ex￾periments with LLMs. In the first prompt, we provide only the forecasting question to the LLM, 9 https://github.com/codelucas/newspaper 10https://github.com/codelucas/newspaper [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Confusion matrices for forecasts with only the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Confusion matrices for forecasts with question [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrices for forecasts with ques [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Confusion matrices for forecasts with ques [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 11
Figure 11. Figure 11: Success rate across forecast horizons of ques [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Question count across forecast horizon (Alpaca-7B) [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 10
Figure 10. Figure 10: Ratio of the top five categories across fore [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 14
Figure 14. Figure 14: Success rate across forecast horizons of ques [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: Question count across forecast horizon (Llama2-13B-chat) [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 16
Figure 16. Figure 16: Ratio of the top five categories across fore [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 20
Figure 20. Figure 20: Success rate across question duration, grouped in 50-day intervals (from 1 to 300 with a step of 50). Question forecasts were made by Alpaca-7B using various input prompts [PITH_FULL_IMAGE:figures/full_fig_p012_20.png]
Figure 24
Figure 24. Figure 24: Question count across question duration (Llama2-13B-chat) [PITH_FULL_IMAGE:figures/full_fig_p012_24.png]
Figure 25
Figure 25. Figure 25: Ratio of the top five categories across ques [PITH_FULL_IMAGE:figures/full_fig_p012_25.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Wisdom of the Crowds in Forecasting: Forecast Summarization for Supporting Future Event Prediction

    cs.LG 2025-02 conditional novelty 5.0 of 10

    A survey of crowd-based future event prediction from text, plus a new eight-component data model for representing individual forecast statements.

Reference graph

Works this paper leans on

16 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Michael F \"a rber, Victoria Burkard, Adam Jatowt, and Sora Lim. 2020. A multidimensional dataset based on crowdsourcing for analyzing and detecting news bias. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 3007--3014

    work page 2020

  4. [4]

    Adam Jatowt and Ching-man Au Yeung. 2011. https://doi.org/10.1145/2063576.2063759 Extracting collective expectations about the future from large text collections . In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM '11, page 1259–1264, New York, NY, USA. Association for Computing Machinery

    work page arXiv 2011

  5. [5]

    Adam Jatowt, Hideki Kawai, Kensuke Kanazawa, Katsumi Tanaka, Kazuo Kunieda, and Keiji Yamada. 2013. https://doi.org/10.1109/CultureComputing.2013.13 Multi-lingual analysis of future-related information on the web . In 2013 International Conference on Culture and Computing, pages 27--32

    work page doi:10.1109/culturecomputing.2013.13 2013

  6. [6]

    Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. 2021. https://doi.org/10.18653/v1/2021.acl-long.357 F orecast QA : A question answering challenge for event forecasting with temporal text data . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati...

    doi:10.18653/v1/2021.acl-long.357 2021

  7. [7]

    Hideki Kawai, Adam Jatowt, Katsumi Tanaka, Kazuo Kunieda, and Keiji Yamada. 2010. Chronoseeker: Search engine for future and past events. In Proceedings of the 4th International Conference on Uniquitous Information Management and Communication, pages 1--10

    work page 2010

  8. [8]

    Ptaszynski, Hirotoshi Honma, and Fumito Masui

    Yoko Nakajima, M. Ptaszynski, Hirotoshi Honma, and Fumito Masui. 2018. https://api.semanticscholar.org/CorpusID:203578888 Future reference sentence extraction in support of future event prediction

    work page 2018

Show all 16 references
  1. [9]

    Yoko Nakajima, Michal Ptaszynski, Fumito Masui, and Hirotoshi Honma. 2020. https://doi.org/10.1016/j.cogsys.2019.09.004 Future prediction with automatically extracted morphosemantic patterns . Cogn. Syst. Res., 59(C):37–62

    2020 doi

  2. [10]

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2024. https://doi.org/10.48550/arXiv.2307.06435 A Comprehensive Overview of Large Language Models . arXiv preprint

    2024 doi arXiv

  3. [11]

    Juwal Regev, Adam Jatowt, and Michael F\" a rber. 2024. https://doi.org/10.1145/3616855.3635693 Future timelines: Extraction and visualization of future-related content from news articles . In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, ...

    2024

  4. [12]

    Jing Su, Chufeng Jiang, Xin Jin, Yuxin Qiao, Tingsong Xiao, Hongda Ma, Rong Wei, Zhi Jing, Jiajun Xu, and Junhong Lin. 2024. https://doi.org/10.48550/arXiv.2402.10350 Large Language Models for Forecasting and Anomaly Detection : A Systematic Literature Review . arXiv preprint

    2024 doi arXiv

  5. [13]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 doi arXiv

  6. [14]

    Yi Yu, Adam Jatowt, Antoine Doucet, Kazunari Sugiyama, and Masatoshi Yoshikawa. 2021. https://doi.org/10.18653/v1/2021.acl-long.32 Multi- T ime L ine summarization ( MTLS ): Improving timeline summarization by generating multiple summaries . In Proceedings of the 59th Annual M...

    2021 doi

  7. [15]

    Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2023. https://doi.org/10.48550/arXiv.2310.01074 Back to the Future : Towards Explainable Temporal Reasoning with Large Language Models . arXiv preprint

    2023 doi arXiv

  8. [16]

    Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. 2022. https://doi.org/10.48550/arXiv.2206.15474 Forecasting Future World Events with Neural Networks . arXiv preprint

    2022 doi arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.