{"id":"967033f2-e74a-44c4-9515-98393fa3ccda","arxiv_id":"2501.06496","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A 614-question Metaculus benchmark shows news context improves LLM binary-forecast accuracy by 2 to 6 points and few-shot examples slightly hurt, but the setup risks leaking outcomes into the news articles.","lead":"Giving language models news articles, background text, and resolution rules alongside binary yes/no forecast questions shifts their accuracy from about 64% to about 68%, while adding lengthy few-shot examples slightly reduces accuracy. The study introduces 614 resolved Metaculus questions, but its news-selection rule may leak the already-known outcomes into the articles, so the 'forecasting' gains may be answer retrieval.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 5-day pre-resolution news filter in Section 3 does not exclude articles that postdate the event but predate Metaculus's resolution, so the reported news-context gains may reflect outcome retrieval rather than forecasting.","rationale":"The reader's weakest assumption identifies exactly the same concern: the news-article selection rule permits articles that contain the outcome, because it keys only on the resolution date rather than the event date or the question's creation date. This is the single most load-bearing threat to the paper's central claim. The accuracy gains from adding news context in Table 3 are modest (4--5 points for GPT-3.5-turbo, 13 points for Alpaca-7B, 11 points for Llama2-13B-chat), and leakage could plausibly account for all of them. I also note that the paper lacks uncertainty estimates, significance tests, and a majority-class baseline, but those are secondary; the leakage concern alone justifies rejection or, at minimum, a major re-analysis. The proposed concrete test—filtering articles to those published before the question was created—would settle whether the improvement persists in a genuinely prospective setting. If the gap survives, the central claim gains credibility; if not, the paper's main finding is an artifact. I therefore endorse the reader's REJECT verdict, with the possibility of reconsideration if the authors re-run the analysis with a strict temporal cutoff and release the data.","tokens_in":10021,"tokens_out":3297,"duration_ms":31440,"concrete_test":"Restrict the evaluation to the subset of questions where every supplied news article has a publish date strictly earlier than the question's 'Created time' (the date the question was authored, before public release). Recompute the accuracy for prompts 1 (Q) and 3 (Q,B,NA) for all three LLMs on this leakage-free subset. If the Q,B,NA minus Q accuracy gap disappears or shrinks materially (e.g., to ≤1 point), the reported news-context gains are explained by outcome leakage, not forecasting ability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that adding news articles significantly improves LLM forecasting accuracy. This claim rests on the dataset construction rule in Section 3: news articles must be published at least five days before the question's resolution date, stated as a measure 'to prevent information leakage.' This filter is insufficient. Metaculus questions resolve only after the outcome is known, and the outcome event can occur weeks or months before resolution. An article published after the event but before the resolution date can therefore report the outcome itself. The dataset records 'Created time', 'Publish time', and 'Resolve time' (Appendix A.1), but the filtering step uses only the resolution date; no check ensures articles predate the question's creation or the event window. Consequently, articles (or their LLM-generated summaries) in prompts 3--5 can contain the answer verbatim, turning the forecasting task into answer retrieval. The accuracy gains reported in Table 3--e.g., GPT-3.5-turbo improving from 0.64 (Q) to 0.68 (Q,B,NA), Alpaca-7B from 0.49 to 0.62, and Llama2-13B-chat from 0.57 to 0.68--are exactly the kind of improvement that leakage would produce. Section 6 lists limitations but does not mention this leakage window. This is the most load-bearing concern because the paper's headline finding, that news context improves forecasting, would be invalid if the context contains the answer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a dataset of 614 resolved binary forecasting questions from Metaculus, each augmented with background text, resolution criteria, and three Google News articles summarized with GPT-3.5-turbo. The authors evaluate five prompt conditions (Q; Q,B; Q,B,NA; Q,B,NA,R; Q,B,NA,R,FS) on GPT-3.5-turbo, Alpaca-7B, and Llama2-13B-chat. They report that adding news articles improves accuracy, adding few-shot examples lowers accuracy, and larger models outperform smaller models.","tokens_in":10266,"tokens_out":7118,"duration_ms":54718,"significance":"The dataset and prompt-ablation design are potentially useful for the forecasting community, and the paper is transparent about model versions and includes confusion matrices and horizon/duration analyses. However, the main empirical claims are not supported by the evidence as presented. The news-article filter in Section 3 permits articles that postdate the event but predate Metaculus's resolution, so the 'news improves forecasting' result may simply reflect outcome retrieval. The statistical analysis lacks baselines and significance tests, and the cross-model comparison is confounded by different question subsets. If the leakage issue were corrected and the analyses redone, the dataset could still be a valuable resource, but the current manuscript's conclusions cannot be accepted.","major_comments":[{"comment":"The dataset-construction rule that news articles must be published at least five days before the resolution date does not prevent information leakage. Metaculus resolves a question only after the outcome is known, and the outcome event can occur well before resolution. A news article published after the event but before the resolution date can therefore report the ground-truth answer. Because prompts 3-5 feed the article summaries directly to the model, the accuracy gains reported in Table 3 (e.g., GPT-3.5-turbo 0.64 to 0.68; Alpaca-7B 0.49 to 0.62; Llama2-13B-chat 0.57 to 0.68) are exactly what answer retrieval would produce. Appendix A.1 records Created time, Publish time, and Resolve time, so the filter should have required article publish dates to precede the question's creation or publish date (or the event window). Section 6 does not mention this leakage window, and the central claim of the abstract depends on it.","section":"Section 3; Appendix A.1"},{"comment":"The claim that news context 'significantly improves' performance is not supported by any statistical evidence. No error bars, confidence intervals, or significance tests are reported. For GPT-3.5-turbo, the Q-only condition achieves 0.64 accuracy while predicting 'no' on almost all questions (recall 0.04, see Figure 1), which is essentially the majority-class baseline (391/614 = 0.64). The improvement to 0.68 is about 4 percentage points on n=614, within roughly two standard errors and not demonstrated to be reliable. For Alpaca-7B and Llama2-13B-chat the sample sizes are 249 and 154, making the reported differences even more uncertain. The word 'significantly' in the abstract should be removed or justified with a proper test.","section":"Section 5; Table 3; Figure 1"},{"comment":"The few-shot result is confounded. Only two fixed examples are used, and no ablation separates the effect of the examples' content from the increased prompt length. The paper attributes the decline to 'input prompts becoming quite large,' but with two hand-picked examples this explanation is not distinguishable from example-specific bias. Moreover, the observed differences (e.g., 0.68 to 0.67 for GPT-3.5-turbo, 0.63 to 0.59 for Alpaca-7B, 0.61 to 0.57 for Llama2-13B-chat) are small relative to the sample sizes, and the conclusion that few-shot examples 'lead to a decline' is not statistically established.","section":"Section 5; Appendix B.5"},{"comment":"The claim that larger models outperform smaller models is not supported by a controlled comparison. Alpaca-7B is evaluated on the 249 questions created after March 2023, while Llama2-13B-chat is evaluated on the 154 questions created after August 2023; the sets differ in time period, topic mix, and difficulty. GPT-3.5-turbo is evaluated on all 614 questions. Thus accuracy differences across models are confounded with data distribution and cannot be attributed to model size. A matched evaluation on the same question subset is needed.","section":"Section 4; Table 3"}],"minor_comments":[{"comment":"The sentence 'The news article must be published at least five days before forecasting questions’s resolution date' has a grammar issue ('questions’s'); please revise and clarify whether the five-day gap is measured relative to question creation or resolution.","section":"Section 3"},{"comment":"The citation 'Yuan et al. Yuan et al. (2023)' is duplicated; it should read 'Yuan et al. (2023)'.","section":"Section 2"},{"comment":"The row labels in the first column run into the numbers (e.g., 'Q,B,NA,R,FS0.67'); add a space or delimiter for readability.","section":"Table 3"},{"comment":"Appendix A.1 lists both a Newspaper3k 'Summary' and a GPT-3.5-turbo 'Summary LLM'; the paper says summaries are generated with GPT-3.5-turbo, but it should state explicitly which field is inserted into Prompts 3-5.","section":"Appendix A.1"},{"comment":"The phrase 'news genre' should likely be 'news articles' or 'the news genre'; please revise for clarity.","section":"Section 6"},{"comment":"The dataset is not made publicly available ('can be obtained after contacting authors'); for reproducibility, the dataset and code should be released.","section":"Dataset Availability"}],"recommendation":"reject","confidential_remarks":"The leakage problem in Section 3 is decisive: the headline result cannot be interpreted as forecasting performance. A resubmission that re-filters articles to predate question creation or event windows, reports significance tests and baselines, and uses matched evaluation across models could address these concerns, but the current paper does not support its conclusions. The dataset itself is potentially useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The reader's rejection is right, and for the right reason. The five-day pre-resolution news filter in Section 3 does not stop leakage: Metaculus resolves only after the outcome is known, and a news article published after the event but before resolution can state the answer. So the reported gains from adding news context are plausibly retrieval, not forecasting. This is the load-bearing flaw, and the authors do not flag it in the limitations.\n\nGive credit where it is due. The dataset is genuinely new: 614 binary Metaculus questions resolved 2021-2024, with background, resolution criteria, and news summaries, filling a gap left by Autocast (ends 2022) and ForecastQA (ends 2019). The prompt ladder - question only, +background, +news, +resolution criteria, +few-shot - is a clean way to isolate context effects, and the GPT-generated summaries keep prompts manageable. Those are useful pieces.\n\nThe soft spots beyond leakage: no error bars or significance tests, and no majority-class baseline, which matters because the Q-only condition nearly collapses to predicting 'no' for two of the three models. The model-size comparison is also confounded by different test subsets (614 vs 249 vs 154 questions). The few-shot decline is a side result, not a problem, but it is underpowered given those subset sizes.\n\nIf the leakage were fixed - restricting articles to before the question's creation or before the event window - the paper could be a modest but honest contribution. As it stands, the central claim does not hold. I would not cite it in its current form, and the dataset is not released, which further limits its immediate value.\n\nThis paper deserves a serious referee, not a desk reject, because the dataset and the ablation design are salvageable and would be useful to the forecasting community if re-run correctly. I would send it to review with the expectation of major revision: redo the news selection with a causal cutoff, add baselines and uncertainty, and release the data. Who is this for? Researchers working on LLM forecasting and temporal leakage in evaluation datasets. They will read the leakage point and the dataset description with interest.","headline":"The news-context gains are probably outcome leakage, but the dataset and systematic prompt ladder are worth a re-run under proper temporal filtering.","tokens_in":695,"tokens_out":1490,"would_cite":false,"duration_ms":22161,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a new dataset of 614 resolved binary Metaculus questions, the paper shows that adding news-article summaries to LLM prompts improves forecasting accuracy for all three tested models, while adding few-shot examples reduces it.","keywords":["LLM forecasting","binary forecasting questions","news article context","few-shot prompting","Metaculus dataset","prompt design","temporal reasoning"],"falsifier":"Annotate a sample of the article summaries for whether they state the event's outcome; if summaries published after the event date but before resolution reveal the answer, the news-context accuracy gain is partly answer retrieval. A direct experiment would rerun the prompts on a subset where every article predates the event in question and compare the accuracy gains.","tokens_in":9766,"feed_emoji":"🔮","tokens_out":7884,"duration_ms":65158,"temperature":0.7,"pith_summary":"The paper asks whether giving large language models more context helps them forecast binary real-world events. Using a new dataset of 614 resolved Metaculus questions, each paired with background information, three news-article summaries, and resolution criteria, it measures accuracy under five prompts of increasing context for three models (GPT-3.5-turbo, Alpaca-7B, Llama2-13B-chat). The central finding is that adding news articles lifts accuracy for every model, while adding few-shot examples lowers accuracy. A sympathetic reader would take this as evidence that current LLMs can be steered toward useful forecasts mainly by supplying timely, relevant news context, and that prompt length and exemplars can hurt.","feed_headline":"News context lifts LLM forecasting; few-shot prompts lower it","feed_subtitle":"Adding news summaries to three LLMs raised forecasting accuracy by up to 14 points on 614 Metaculus questions.","key_machinery":"The central object is the context ladder: five prompts that start with a bare binary question and add, in order, background text, three news-article summaries, resolution criteria, and two solved few-shot examples. The ingredient that does the work is the news-article summary: each question is linked to at least three news articles published at least five days before the question's resolution date, and each article is condensed by GPT-3.5-turbo into a few question-relevant sentences. These summaries are what shift models away from an almost universal 'no' prediction and toward answers that track the true balance of the dataset. The paper also contributes the dataset itself, a collection of 614 recent, resolved, binary questions with background, resolution criteria, news URLs, and summaries.","core_discovery":"On a dataset of 614 binary forecasting questions drawn from Metaculus, with questions created after October 2021 and already resolved, the authors evaluate forecasting accuracy under five prompt conditions: question only (Q); question plus background (B); background plus three news-article summaries (NA); background, news, and resolution criteria (R); and all of the above plus two few-shot examples (FS). Accuracy for GPT-3.5-turbo rises from 0.64 with the bare question to 0.68 with background and news; Alpaca-7B rises from 0.49 to 0.63; Llama2-13B-chat rises from 0.57 to 0.68. The prompting stage that includes news articles is the best or tied-best for each model, and adding few-shot examples consistently reduces accuracy by a few points. With only the question, the two larger models answer 'no' for nearly every item; supplying context rebalances the prediction mix toward the dataset's actual yes/no distribution.","pith_inferences":["The five-day publication gap does not rule out leakage, because Metaculus resolves after the fact; if post-event articles are among those summarized, part of the news-context benefit may be retrieval rather than forecasting. This is my editorial reading, not the paper's claim.","The strong default-to-'no' behavior in the bare-question condition suggests these models are not genuinely reasoning about the question as a forecast; context may help mainly by breaking that default, implying that prompt design can dominate model reasoning in this task.","Filtering for questions with at least three findable news articles likely selects for media-covered, event-heavy topics, so the measured gains may not transfer to quieter or slower-moving forecasts.","A natural extension would vary the publication cutoff (e.g., 30 days before resolution or before the event's start) to test whether accuracy gains persist; if they vanish, the gains are leakage-driven."],"forward_implications":["Forecasting systems built on LLMs should supply current news summaries as context: they raised accuracy by 4 to 14 percentage points across the three models tested.","Few-shot examples should be omitted or tested carefully: adding them lowered accuracy for all three models.","Model scale helps: the 13B-parameter Llama2-chat outperformed the 7B Alpaca in the best-context condition, and GPT-3.5-turbo led overall.","The released dataset gives the community a recent binary-forecasting benchmark (614 questions, 2021–2024) with background text, resolution criteria, and news-article links."],"supporting_citations":[{"why":"Supplies the main prior forecasting dataset (Autocast) and the baseline result that the best evaluated model reached 65% accuracy versus 92% for aggregate human forecasts.","marker":"Zou et al. (2022)"},{"why":"Provides ForecastQA, the prior binary forecasting QA dataset, and the practice of assigning timestamps before article publication to avoid outcome leakage that the paper adapts.","marker":"Jin et al. (2021)"},{"why":"Defines Llama 2 and its training cutoff date, which the paper uses to select the post-cutoff subset of questions for Llama2-13B-chat.","marker":"Touvron et al. (2023)"},{"why":"Identifies GPT-3.5-turbo as an optimized successor of GPT-3, fixing the model lineage referenced in the experiments.","marker":"Su et al. (2024)"}],"fun_headline_variants":["News context boosts LLM forecasting accuracy","Few-shot examples reduce LLM prediction quality","LLMs forecast better with news articles","News helps, few-shot hurts LLM forecasting","Contextual news lifts LLM forecasting performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The setup assumes that news articles published at least five days before a question's resolution date do not contain information about the actual outcome, even though Metaculus resolves questions after the event has occurred and post-event articles can therefore reveal the answer.","fun_headline_variants_meta":{"raw":{"variants":["News context boosts LLM forecasting accuracy","Few-shot examples reduce LLM prediction quality","LLMs forecast better with news articles","News helps, few-shot hurts LLM forecasting","Contextual news lifts LLM forecasting performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2227,"prompt_tokens":831,"completion_tokens":1396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":1331}},"tokens_in":447,"tokens_out":1396,"duration_ms":9859,"temperature":1.0,"reasoning_tokens":1331,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:58:59.132508+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a sample of the article summaries for whether they state the event's outcome; if summaries published after the event date but before resolution reveal the answer, the news-context accuracy gain is partly answer retrieval. A direct experiment would rerun the prompts on a subset where every article predates the event in question and compare the accuracy gains.","supporting_citations":[],"review_version":1}