{"id":"58e2e6a9-5abc-40d6-afad-f8a2376fe9e9","arxiv_id":"2501.03928","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Transformer models fine-tuned on dyad-level newswire digests forecast conflict escalation better than a fatality-history baseline at one-month horizons, but lose skill at six months.","lead":"Researchers combined 2.16 million newswire articles with UCDP conflict event data to train transformer models that forecast whether armed conflicts escalate or de-escalate at the level of warring dyads. The work matters because it promises earlier, actor-level warnings for conflict dynamics, a task where classical fatality-history models have been weak.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation target leaks future information: validation labels come from a GP fit through March 2023, so test-month 'escalation states' are not observable at forecast time and reported skill does not measure real-time prediction.","rationale":"I read the paper as claiming that newswire text alone can forecast conflict escalation/de-escalation momentum at the dyad-month level, with the key evidence being the DeBERTa models beating a strong baseline at nowcast and 1-month horizons. For that evidence to support the claim, the labels used in evaluation must be knowable at forecast time. Section 3.5 explicitly builds the validation target from a GP fitted through March 2023, the end of the evaluation window, so each test-month label incorporates future fatalities. This is the same load-bearing flaw the reader identified. It is also directly visible in the manuscript: the stated purpose is to 'prevent data leakage,' but fitting the validation smoother through the end of the test window does the opposite. I agree with the reader's REJECT verdict, so I recommend no verdict change. The concrete test I propose would settle the issue cleanly by rebuilding labels with a causal smoother and re-running the comparison; if the skill persists, the textual-signal claim is credible, and if it collapses, the current evaluation cannot support it. The six-month overclaim in the abstract is secondary but reinforces that the results section, not the conclusion, is the accurate summary of the evidence.","tokens_in":24577,"tokens_out":3645,"duration_ms":38335,"concrete_test":"Recompute the validation target causally: for each test month m, refit the hierarchical GP using only log-fatality observations through month m (or use a one-sided/particle-filter estimate of the first derivative), then re-evaluate the same pre-trained DeBERTa RAG and low-context models plus the bootstrap baseline on these real-time-observable labels. If the nowcast and 1-month-ahead AP/F1 advantages shrink to baseline, the claimed text signal is an artifact of future information in the target.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central out-of-sample claim depends on the validation labels being the real-time escalation state. Section 3.5 defines the target as the discretized first derivative of a Gaussian process and states: 'To prevent data leakage from the training to the validation partition, we estimate two Gaussian processes – one f^train,dyad trained on data until the end of the training window (December 2022) and one used for validation f^val,dyad until the end of the data (March 2023).' Because a GP smoother at month x uses observations from both sides of x, every validation label in the Jan 2022–Mar 2023 window is a function of fatalities observed after the forecast month, including fatalities up to three months later. Thus the 'predictions' are scored against an ex-post smoothed target, not against anything knowable when the forecast would be issued. This is not a minor benchmark quibble: both the DeBERTa models and the bootstrap baseline are compared on these leaky labels, so the reported AP/F1 advantage may largely reflect the models' ability to track a target that has been retroactively smoothed. The same passage gives f^train through December 2022, after the declared end-2021 training split, suggesting leakage may extend into training targets as well if taken literally. The paper's own Figure 4 also contradicts the abstract's 'up to six months' claim: at six months the DeBERTa models are at or near baseline. If the target is made causal, the headline forecasting claim may not survive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end pipeline that forecasts dyad-month conflict dynamics from newswire text. It curates a Factiva corpus using UCDP GED annotations, trains a ConfliBERT-based classifier to assign unlabeled articles to conflict dyads, constructs low- and high-context monthly digests via BERTopic and retrieval-augmented generation, and defines the forecast target as the discretized first derivative of a hierarchical Gaussian process fitted to log fatalities. The authors fine-tune DeBERTa-v3-large and Mistral 7B with step-shifting at nowcasting and 1-, 3-, and 6-month horizons, and compare against a bootstrap baseline. The reported results show DeBERTa outperforming the baseline for nowcasting and one-month-ahead forecasts, attenuating at three months, and near collapse at six months, while Mistral performs poorly.","tokens_in":24854,"tokens_out":7895,"duration_ms":74760,"significance":"If the evaluation were valid, this would be a substantial contribution to conflict forecasting: it moves from incidence/severity targets to escalation dynamics, operates at the actor-dyad level, and demonstrates that text-only inputs can replace manually curated event data at prediction time. The pipeline is transparent and the authors deserve credit for reporting the Mistral failure and the six-month attenuation honestly. However, the central out-of-sample claim is currently undermined by two evaluation problems: the validation target is an ex-post smoothed quantity, and hyperparameter tuning was performed on the test window. Both issues are load-bearing for the headline result and need to be addressed before the forecasting claim can be assessed.","major_comments":[{"comment":"The validation target is not observable at forecast time. The paper states: 'To prevent data leakage from the training to the validation partition, we estimate two Gaussian processes – one f^train,dyad trained on data until the end of the training window (December 2022) and one used for validation f^val,dyad until the end of the data (March 2023).' Because a Gaussian process smoother at month x uses observations from both sides of x, every validation label in the January 2022–March 2023 window is a function of fatalities observed after the forecast month, in the worst case as far ahead as March 2023. Thus the reported AP/F1 scores for nowcasting and one-month-ahead forecasts are scores against an ex-post smoothed target, not against anything knowable when the forecast would be issued. This is not a minor benchmark quibble: both the DeBERTa models and the bootstrap baseline are compared on these leaky labels. Please re-estimate the target in a causal manner (e.g., sequential GP fits using only data up to the forecast month, or a one-sided filter) and re-report all metrics, including the baseline, on the causal labels.","section":"Section 3.5, Eq. (4)"},{"comment":"Hyperparameter tuning for the DeBERTa models was performed on 'a small randomly sampled dataset from the test window.' Since the test window is January 2022–March 2023, using it for model selection means the reported test-set metrics are not strictly out-of-sample; the headline comparison to the baseline is therefore potentially optimistic. Please move all hyperparameter selection to a validation set drawn from the training period, or use a nested cross-validation scheme, and report whether the conclusions change.","section":"Section 3.6, footnote 34"},{"comment":"The abstract and conclusion claim that newswire texts 'can be used to effectively predict the momentum of armed conflicts ... over periods of up to six months,' but this is contradicted by the paper's own results in Section 4 and Figure 4, where at six months the DeBERTa models are described as 'just slightly better in AP and F1 than the baseline, and ... at baseline for AUROC.' The supported claim is for nowcasting and one-month-ahead horizons, with attenuation at three months. Please revise the abstract and conclusion to state the horizons for which the evidence actually holds.","section":"Section 5 and Figure 4"},{"comment":"The description of the training window is internally inconsistent. Section 3.1 states that the data are divided 'in a training set (until the end of 2021) and a test set (starting January 2022),' whereas Section 3.5 says f^train is trained 'until the end of the training window (December 2022).' If the training digests stop in December 2021 while training targets are derived from a GP fitted through December 2022, then the target attached to the final training months encodes up to twelve months of future information. Please clarify the actual training window and, if needed, recompute training targets with point-in-time information.","section":"Section 3.1 vs. Section 3.5"}],"minor_comments":[{"comment":"The text refers to the 'Matérn 3/2 covariate function'; this should be 'covariance function.' Also, the length-scale parameter ℓ is introduced in Eq. (3) but defined only in the following prose; please define it at first use.","section":"Section 3.5, Eq. (3)"},{"comment":"The caption says 'micro-aggregation where probabilities are involved (ROC, AP),' but the figure also reports F1 micro and F1 weighted. Please clarify how each metric is aggregated and how the bootstrap confidence intervals are computed (e.g., resampling units).","section":"Figure 4 caption"},{"comment":"The target definition depends on the threshold τ=0.25 and on an informative length-scale prior (MAP 122.38) taken from von der Maase (2022b), a working paper by the second author. A sensitivity analysis over τ and the prior, or at least a discussion of their influence, would strengthen confidence in the target construction.","section":"Section 3.5, Eq. (4)"},{"comment":"von der Maase (2022b) is cited as the source for the Gaussian-process target definition, but it is a working paper. Please provide a stable reference or an accessible replication appendix, since the evaluation target depends on that method.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The two evaluation problems — the ex-post GP target and the use of the test window for hyperparameter tuning — are serious enough that the paper should not be accepted in its current form. I recommend major revision rather than rejection because the repair path is clear and within scope: rebuild the target causally, re-tune on a proper validation set, and re-report. If the causal re-evaluation eliminates the DeBERTa advantage over the baseline, the paper's central forecasting claim would no longer be supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth taking seriously: use UCDP's human annotations to back-label a large newswire corpus, build dyad-month digests, and fine-tune DeBERTa (and Mistral) to forecast a momentum target rather than incidence. That is a real departure from topic-proportion news models and from earlier actor-centric work. The pipeline is described carefully, with intermediate evaluations, and the authors are honest that Mistral fails and that the six-month results are essentially at baseline. I also appreciate that they pick a strong bootstrap baseline rather than a weak one.\n\nThe problem is the target. Section 3.5 says they fit two GPs—one for training through December 2022 and one for validation through March 2023—and then use the discretized first derivative as the target for both training and validation. The validation labels for a month in the test window are therefore a function of fatalities observed after that month. A GP smoother at time x uses observations on both sides of x, so every test-month label in the Jan 2022–Mar 2023 window is retroactively smoothed with up to three months of future data. That is look-ahead bias in the evaluation. The paper claims this setup prevents leakage, but it only prevents leakage between the two GP fits; it does not make the target causal. The abstract's claim of skill \"up to six months\" is also contradicted by the paper's own Figure 4, where the DeBERTa models are near baseline at six months. And there is an inconsistency: the declared training split ends December 2021, yet the training GP is fit through December 2022, which if taken literally means the training targets also embed future information.\n\nThe consequence is that the reported AP/F1 advantage over the bootstrap baseline measures how well text predicts a retrospectively smoothed state, not how well it predicts what an analyst could have known at forecast time. That is not a minor benchmark quibble; it is the central forecasting claim. What remains is a well-built proof-of-concept that text contains some signal about conflict dynamics, but the headline result as stated is not supported.\n\nMy recommendation: send it to review. The pipeline and the questions it asks are substantial enough to deserve referee time, but the revision path is clear—redefine the target causally, using for each month only data available at that month (e.g., a filtered derivative or a one-sided smoother), and re-run the evaluation. If the results survive that, this becomes a solid contribution. If they do not, the paper is still a useful methodological case study, but the forecasting claim has to go.","headline":"The paper has a genuinely new text-to-forecast pipeline, but the evaluation target is leaky: the Gaussian process that defines escalation states is fitted on data through the end of the test window, so the reported skill is measured against an ex-post smoothed target, not a real-time observable.","tokens_in":25465,"tokens_out":2737,"would_cite":false,"duration_ms":28554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Newswire text alone can forecast when armed conflicts escalate or de-escalate up to six months ahead.","keywords":["conflict forecasting","text-based prediction","actor dyads","escalation and de-escalation","transformer fine-tuning","newswire corpora","Gaussian process target","early warning systems"],"falsifier":"Re-run the same models against a target computed causally, for example a Gaussian process or moving average updated only with fatalities available at the forecast month, and check whether the DeBERTa models still beat the bootstrapped baseline by the reported margins at the one-month and three-month horizons; if the advantage largely vanishes, the headline skill was an artifact of the ex-post smoothed target.","tokens_in":1953,"feed_emoji":"📰","tokens_out":2515,"duration_ms":84456,"temperature":0.7,"pith_summary":"Armed-conflict forecasting has mostly been built on lagged fatality counts and spatial history, which predict the next fatality level better than they predict changes. This paper tries to establish that the raw newswire text behind conflict event data carries an earlier, richer signal about actor-level dynamics: whether a fighting pair is escalating, de-escalating, plateauing, or at peace. It constructs a large corpus by matching manually coded conflict events to news articles, uses a fine-tuned language model to label the rest of the corpus by actor dyad, and builds monthly digests of violent and contextual articles for each dyad. Fine-tuned encoder-based transformers trained on these digests predict the discretized slope of the smoothed fatality trend, and the paper reports that they beat a strong historical baseline substantially for nowcasts and one-month-ahead forecasts, with skill decaying by six months. If right, text-trained models could generate early warning of escalation at the actor level without waiting for manual event-data curation at prediction time.","feed_headline":"Newswire text can forecast conflict escalation months ahead","feed_subtitle":"Text-only dyad-level models beat historical baselines at predicting which conflicts are escalating or winding down.","key_machinery":"The carrying machinery is a pipeline ending in a four-state momentum target plus a fine-tuned transformer. Momentum is defined as the discretized first derivative of a hierarchical Gaussian process fitted to each dyad's monthly log fatalities, with a Matérn 3/2 kernel and a shared prior on the length scale; a threshold of $\\pm 0.25$ on the derivative splits escalation, plateau, and de-escalation. From the text side, a lightweight BERT-class model is fine-tuned on manually labeled event articles and used to assign every remaining article to one of 25 well-reported actor dyads, filtering out irrelevant news. A topic-model and retrieval step augments each violent event's snippet with nearby non-violent context snippets, and the snippets are concatenated into long dyad-month digests. The forecasting model is a fine-tuned DeBERTa-v3-large encoder with a softmax classification head, trained by step-shifting: predicting the target shifted 0, 1, 3, or 6 months forward, with a 7-billion-parameter decoder model, Mistral 7B, included for comparison.","core_discovery":"The central claim is that the textual record of a conflict dyad's month contains usable predictive signal about the direction its violence is heading, and that a model can extract that signal directly from text. The paper's target is not the level of fatalities but the momentum: a Gaussian process is fitted to monthly log-fatality series for each dyad, and the first derivative of that smoothed trend is discretized into four states: peace, escalation, plateau, and de-escalation. Against a baseline that bootstraps the previous year's observed state distribution, two fine-tuned DeBERTa-based models, one reading only violent-event snippets and one reading events augmented with retrieved contextual articles, exceed the baseline on average precision, AUROC, and F1 for nowcasting and one-month-ahead forecasting, remain above it at three months, and fall to near-baseline at six months. The paper further claims this makes the manual curation step unnecessary at prediction time, and it reports that the much larger decoder-based Mistral model performs poorly, below baseline, a result the authors offer as a caution about decoder-only models.","pith_inferences":["A direct extension, not run in the paper, is to recompute the target causally from only the fatalities known at the forecast month; if the reported margins survive, the claim is robust to the target-leakage concern, and if not, part of the reported skill is an artifact of ex-post smoothing.","The authors' own reading leaves open, and we infer, that much of the signal lives in the violent-event text rather than in the added non-violent context, since the RAG augmentation yields only modest gains.","We infer, beyond the paper's dyad-selection caveat, that the margin over baseline should not be extrapolated to rarely reported or newly emerging conflicts, where the news corpus and the training labels are far thinner."],"forward_implications":["Conflict early-warning systems could skip the weeks-long manual event-data annotation step at prediction time and read newswire text directly for escalation signals.","The text-only signal is strongest for escalation and weaker for de-escalation, so users should expect asymmetric skill and weight early-warning products accordingly.","The collapse of skill between three and six months defines a practical forecast horizon for text-based momentum models.","Large decoder-only generative models are not automatically better at this task; smaller encoder-based models can dominate them, so architecture choice matters.","Retrieval-style context augmentation helps at most horizons but not dramatically, suggesting the main predictive gain comes from the violent-event text itself."],"supporting_citations":[{"why":"Defines the UCDP GED event data and its newswire-based annotation process, the source of the manually labeled articles and dyad-level event labels used throughout.","marker":"(Croicu and Sundberg, 2017)"},{"why":"Supplies ConfliBERT, the pretrained conflict-domain language model fine-tuned to assign unlabeled articles to actor dyads.","marker":"(Y Hu et al., 2022)"},{"why":"Supplies DeBERTa-v3-large, the encoder-only transformer that produces the main forecasting results.","marker":"(He et al., 2020)"},{"why":"Supplies Mistral 7B, the decoder-based model whose poor forecasting performance is the central negative result.","marker":"(Jiang et al., 2023)"},{"why":"Introduces the Gaussian-process time-series decomposition whose first derivative becomes the escalation and de-escalation target.","marker":"(von der Maase, 2022b)"},{"why":"Provides the retrieval-augmented generation technique used to attach non-violent context articles to violent events in the high-context digests.","marker":"(Lewis et al., 2020)"},{"why":"Defines the step-shifting forecasting setup used to train separate models for nowcasting and one-, three-, and six-month horizons.","marker":"(Hegre, Bell, et al., 2021)"},{"why":"Provides the strong bootstrapped historical baseline that the text models must beat, drawn from conflict prediction benchmark practice.","marker":"(Hegre, Vesco, Colaresi, et al., 2024)"},{"why":"Represents the topic-proportion text-forecasting approach the paper extends by modeling the text directly and at the dyad level.","marker":"(Mueller, Rauh and Seimon, 2024)"}],"fun_headline_variants":["News text predicts conflict escalation up to three months ahead","AI reads newswire to forecast conflict escalations and lulls","Combining news and events improves conflict forecasting","Actor-level text embeddings forecast conflict momentum","Transformers on news text beat baselines in conflict forecasting"],"cache_read_input_tokens":27392,"weakest_assumption_plain":"The load-bearing premise is that a smoothed escalation state computed from a curve fitted on fatality data spanning the whole test period can stand in for the value a forecaster would have wanted to predict in real time, even though that curve incorporates fatalities that occurred after the forecast month.","fun_headline_variants_meta":{"raw":{"variants":["News text predicts conflict escalation up to three months ahead","AI reads newswire to forecast conflict escalations and lulls","Combining news and events improves conflict forecasting","Actor-level text embeddings forecast conflict momentum","Transformers on news text beat baselines in conflict forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2526,"prompt_tokens":1001,"completion_tokens":1525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":1451}},"tokens_in":617,"tokens_out":1525,"duration_ms":15134,"temperature":1.0,"reasoning_tokens":1451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:43:49.943422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same models against a target computed causally, for example a Gaussian process or moving average updated only with fatalities available at the forecast month, and check whether the DeBERTa models still beat the bootstrapped baseline by the reported margins at the one-month and three-month horizons; if the advantage largely vanishes, the headline skill was an artifact of the ex-post smoothed target.","supporting_citations":[],"review_version":1}