Pith. sign in

REVIEW 5 major objections 6 minor 27 references

Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that conflict forecasting from LLM pretrained knowledge alone captures broad trends but misses fine-grained ones, and that retrieval-augmented context improves results only when the model can integrate it.

desk verdict Useful question, untrustworthy comparison: the RAG-vs-parametric result hinges on an unspecified retrieval date filter and silent Ethiopia fallback, so treat the headline gains as provisional. read the letter →

arxiv 2505.09852 v1 pith:FJZW7LGB submitted 2025-05-14 cs.CL cs.AI

Apollinaire Poli Nemkova , Sarath Chandra Lingareddy , Sagnik Ray Choudhury , Mark V. Albert This is my paper · ORCID
classification cs.CLcs.AI
keywords conflictforecastinglargelanguagemodelsparametricknowledgeretrieval-augmentedgenerationzero-shotpromptingGDELTACLEDearlywarningsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can forecast violent conflict from what they already know, and whether adding fresh external context helps. It compares zero-shot predictions from GPT-4 and LLaMA-2 with a retrieval-augmented condition that supplies summaries of recent news, event tone, and fatality counts. The core claim is that parametric knowledge alone captures broad conflict dynamics but misses fine-grained trends and numeric estimates, while retrieval augmentation helps only when the model can integrate the retrieved context. On most tasks and countries, GPT-4 improves with RAG; LLaMA-2 shows minimal or inconsistent gains. If true, this means early-warning systems should pair LLMs with curated real-time conflict data rather than rely on pretrained knowledge.

What carries the argument

The machinery is a two-condition evaluation comparing zero-shot parametric forecasting with a retrieval-augmented generation pipeline. In the RAG condition, semantic search retrieves relevant news excerpts, a summarizer condenses them, and the prompt is enriched with the average GDELT tone, the average Goldstein scale score (a numeric proxy for cooperation-conflict in an event), and weekly ACLED fatalities over the past twelve weeks. The model then produces a four-label trend (Escalate, De-escalate, Peace/No Conflict, Stable Conflict) and a fatality estimate, scored against ACLED ground truth. This controlled contrast is what separates the contribution of external context from the model's internal knowledge.

What would settle it

Re-run the RAG condition with a strict retrieval cutoff that provably excludes all articles dated after the forecast month, and compare against the paper's pipeline; if the improved macro-F1 and binned-regression scores vanish, the reported RAG gains are inflated by future leakage, and if they persist, the gains are genuine.

Watch

Extended reading notes

Core claim

The central discovery is that non-parametric knowledge does not automatically improve conflict forecasting. In the paper's experiments, zero-shot LLMs recognize peaceful or stable dynamics but are weak at escalation and de-escalation classes and at fatality counts. When the models receive three months of retrieved news summaries, average article tone, Goldstein scale scores, and weekly fatalities, GPT-4's macro-F1 and binned regression improve across most countries, while LLaMA-2 mostly does not benefit and sometimes degrades. The authors interpret this as evidence that effective non-parametric forecasting requires both retrieval and sufficient model capacity to interpret and incorporate the retrieved information.

Load-bearing premise

The argument assumes the data pipeline is temporally clean: retrieved articles and summaries cover only the three months before each forecast, and the models' pretraining did not already contain the outcome being predicted, but the paper never states the retrieval date filters or model training cutoffs.

Editorial extensions

If this is right

  • For high-stakes early warning, zero-shot LLM outputs should be treated as directional priors, not calibrated forecasts.
  • Retrieval augmentation is worth deploying for models with strong instruction-following like GPT-4, but model choice matters more than the mere presence of retrieval.
  • Binned intensity classes improve with context while raw fatality mean absolute error does not, so evaluation metrics that collapse fatality counts into quantile bins can overstate RAG's practical value.
  • Open-weight models such as LLaMA-2 may still be usable in low-resource settings when paired with retrieval, but only in select regions and task formulations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to control the retrieval window explicitly: if the pipeline is rerun with provably pre-forecast-only articles, and scores persist, the RAG gains are real; if they drop, the reported effects partly reflect future leakage.
  • The paper's label-from-fatality mapping may already encode the outcome; a sharper test would ask for direct ordinal fatality predictions and then map to labels, isolating whether the model actually forecasts conflict rather than reproducing recent levels.
  • The model-specific gap hints that retrieval-augmented conflict forecasting is gated by context adherence: instruction-tuned models that follow the supplied numbers benefit, while smaller chat models may ignore or override them, connecting to the knowledge-conflict phenomenon surveyed in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper compares two ways of using LLMs for one-month-ahead conflict forecasting: parametric zero-shot prompting (Experiment 1) and retrieval-augmented prompting with GDELT-derived news summaries and ACLED indicators (Experiment 2). Using GPT-4 and LLaMA-2 across five countries over 2020-2024, it reports that RAG improves several GPT-4 metrics in some countries while LLaMA-2 shows smaller and inconsistent gains. The central claim is modest: non-parametric augmentation helps conflict forecasting only when the model can integrate retrieved context.

Significance. The question is timely, and the two-condition design is a natural way to separate parametric memory from external grounding. The use of real ACLED and GDELT data gives the study ecological validity, and the authors state that prompts and code will be released, which would support replication. However, the empirical evidence as presented is not yet sufficient to establish the central claim: temporal leakage is not ruled out, several reported rows are identical across the two conditions for reasons the paper does not explain, and no uncertainty quantification or baselines are provided. If these issues are addressed, the comparison would be a useful contribution to the growing literature on LLM-based event forecasting.

major comments (5)
  1. [Section 3.2.1 and Section 3.1] The RAG condition is described as providing 'the previous three months' of context, but the paper never states that the FAISS retrieval index is filtered so that only articles dated before the forecast month are returned; Section 3.1 says GDELT was queried 'between 2020 and 2024' as a single corpus. Without an explicit date constraint, retrieved articles from the ground-truth month are semantically close to the query and would leak the outcome into the prompt, inflating exactly the RAG improvements claimed in Section 4. Please specify the date filter on retrieval and, ideally, rerun the experiments with a strict date-before-forecast constraint.
  2. [Section 3.2] The parametric condition assumes that the models' pretraining excludes the forecast months, but no training cutoffs are reported for GPT-4 or LLaMA-2 relative to the 2020-2024 evaluation window. For months before a model's knowledge cutoff, zero-shot 'forecasts' can be memory retrieval rather than prediction. Please state the training cutoffs, restrict the evaluation to months after the later cutoff, or demonstrate that the results are robust when pre-cutoff months are removed.
  3. [Table 1 and Table 2] All GPT-4 metrics for Ethiopia are identical between Experiment 1 and Experiment 2 in Table 1 (e.g., 0.2712 accuracy, 0.2091 macro-F1, MAE 446.59, and identical binned-regression values). This is exactly what one would expect if the RAG branch silently fell back to the parametric prompt or failed to complete. The paper does not acknowledge or explain this, and identical rows also appear for Israel in Table 2 (e.g., class-categorical F1 macro 0.2710 and binned-regression accuracy 0.3443 in both experiments). The cross-condition comparison is not trustworthy until these rows are explained.
  4. [Section 4 and Section 5] The Results and Discussion narrative contradicts Table 1: the text says 'macro F1 scores in Ethiopia and Israel improved (from 0.2091 to 0.2710 and from 0.1471 to 0.2710)', but Table 1 shows Ethiopia's macro-F1 unchanged at 0.2091 in both experiments, and the second value (0.2710) is Israel's Experiment 2 value, not a gain for either country from the stated starting point. This makes it impossible to tell which of the reported gains are real.
  5. [Section 4 and Section 3.3] No confidence intervals, significance tests, or comparison baselines are reported. With 59 country-months, class imbalance, and often marginal differences between conditions, the claim that RAG 'consistently' improves GPT-4 is not supported as presented. At minimum, report a majority-class or persistence baseline and paired significance tests or bootstrap intervals, and disclose the number of repeated generations at temperature 0.2.
minor comments (6)
  1. [Section 3.3] The paper should define how the slope thresholds and quantile bins for the 'Class (From Fatalities)' and binned-regression tasks are computed, and state whether they are fit on training months only; currently the reader cannot tell whether the evaluation uses full-period ground truth to construct labels.
  2. [Table 1 and Table 2 captions] The table captions should report the number of observations per country and the number of repeated generations or standard errors; the current tables show only point estimates.
  3. [Section 2] Several citations are incomplete: 'Yu et al.' has no year or venue, and the TimeRAG citation by 'Yang et al.' is not matched to a complete reference entry.
  4. [Throughout] The name 'LLama' is written inconsistently with 'LLaMA' in the tables and text; please standardize.
  5. [Section 3.1] The abstract mentions ACLED as a source of structured context, but the method section should clarify that ACLED is used only for weekly fatality counts and ground truth, not for retrieved text passages.
  6. [Section 3.4] The GitHub link is anonymous; before publication, please replace it with a stable repository and add a data availability statement describing the exact GDELT query dates and any filtering steps.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical benchmark with no fitted-parameter-as-prediction or self-citation load-bearing chain.

full rationale

The paper does not derive any quantity from another quantity; it runs two prompting conditions (parametric zero-shot and RAG) and reports measured metrics. The only self-citation (Nemkova et al., 2023) appears in the literature review as an example of transformer-based conflict monitoring and is not load-bearing for any claim. The evaluation's class-from-fatalities and binned-regression metrics use thresholds and quantile bins derived from the ACLED ground truth, but those bins define the evaluation metric, not the model's prediction; the model never fits or sees those bins, so this is standard evaluation construction rather than a circular reduction. The limitations section's caveats (small N, threshold oversimplification, noisy RAG summaries) are acknowledged. The strongest correctness concern--possible retrieval-date leakage and unknown GPT-4 training cutoff in the 2020-2024 window--is an external-validity/data-integrity issue, not a circular-derivation issue, and cannot be exhibited as an equation or fitted-parameter reduction. No pattern from the circularity taxonomy applies, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new entities are postulated. The central empirical claims rest on data-source assumptions and on an unstated temporal-integrity assumption for both parametric and RAG conditions.

free parameters (2)
  • Class label thresholds (slope thresholds) = not reported
    Used to convert fatality counts and predicted fatalities into Escalate, De-escalate, Stable Conflict, Peace. Derived from ACLED ground truth and not specified numerically.
  • Fatality quantile bins = not reported
    Used for binned regression evaluation. Computed from the ACLED fatality distribution; if computed over the full period, they can leak information about test labels.
assumptions (4)
  • domain assumption ACLED fatality counts are an accurate ground truth for conflict intensity.
    Ground truth for classification and regression is built from ACLED; errors in ACLED propagate to all labels.
  • domain assumption GDELT article tone and Goldstein scale are meaningful proxies for conflict dynamics.
    These features are fed to the model in the RAG condition without validation of their predictive content for this task.
  • domain assumption Pre-trained LLM weights do not contain the actual ACLED outcome for the forecast month.
    The parametric setting is treated as a forecast, but GPT-4's training data may overlap the 2020-2024 evaluation period. The paper does not enforce or verify a cutoff.
  • domain assumption Retrieved GDELT content is temporally restricted to the three months before each forecast date.
    Section 3.2.1 says context covers the previous three months, but Section 3.1 describes scraping GDELT across 2020-2024 without stating the retrieval date filter, so leakage is possible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting." pith.science (2026). https://pith.science/paper/FJZW7LGB

@misc{pith2026250509852,
  author       = {Pith},
  title        = {Pith review of: Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FJZW7LGB}},
  note         = {Machine review of arXiv:2505.09852}
}
read the original abstract

Large Language Models (LLMs) have shown impressive performance across natural language tasks, but their ability to forecast violent conflict remains underexplored. We investigate whether LLMs possess meaningful parametric knowledge-encoded in their pretrained weights-to predict conflict escalation and fatalities without external data. This is critical for early warning systems, humanitarian planning, and policy-making. We compare this parametric knowledge with non-parametric capabilities, where LLMs access structured and unstructured context from conflict datasets (e.g., ACLED, GDELT) and recent news reports via Retrieval-Augmented Generation (RAG). Incorporating external information could enhance model performance by providing up-to-date context otherwise missing from pretrained weights. Our two-part evaluation framework spans 2020-2024 across conflict-prone regions in the Horn of Africa and the Middle East. In the parametric setting, LLMs predict conflict trends and fatalities relying only on pretrained knowledge. In the non-parametric setting, models receive summaries of recent conflict events, indicators, and geopolitical developments. We compare predicted conflict trend labels (e.g., Escalate, Stable Conflict, De-escalate, Peace) and fatalities against historical data. Our findings highlight the strengths and limitations of LLMs for conflict forecasting and the benefits of augmenting them with structured external knowledge.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 13 canonical work pages

  1. [1]

    Fulvio Attina, Marcello Carammia, and Stefano M Iacus. 2022. Forecasting change in conflict fatalities with dynamic elastic net. International Interactions, 48(4):649--677

    work page 2022

  2. [2]

    He Chang, Chenchen Ye, Zhulin Tao, Jie Wu, Zhengmao Yang, Yunshan Ma, Xianglin Huang, and Tat-Seng Chua. 2024. A comprehensive evaluation of large language models on temporal event forecasting. arXiv preprint arXiv:2407.11638

    arXiv 2024

  3. [3]

    Mihai Croicu and Simon Polichinel von der Maase. 2025. From newswire to nexus: Using text-based actor embeddings and transformer networks to forecast conflict dynamics. arXiv preprint arXiv:2501.03928

    work page Pith review arXiv 2025

  4. [4]

    H vard Hegre, Marie Allansson, Matthias Basedau, Michael Colaresi, Mihai Croicu, Hanne Fjelde, Frederick Hoyles, Lisa Hultman, Stina H \"o gbladh, Remco Jansen, et al. 2019. Views: A political violence early-warning system. Journal of peace research, 56(2):155--174

    work page 2019

  5. [5]

    Zhengbao Jiang. 2024. Towards More Factual Large Language Models: Parametric and Non-parametric Approaches. Ph.D. thesis, Carnegie Mellon University

    work page 2024

  6. [6]

    Ivanova, Giulia Rambelili, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci

    Carina Kauf, Anna A. Ivanova, Giulia Rambelili, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2022. https://doi.org/10.48550/arXiv.2212.01488 Event knowledge in large language models: the gap between the impossible and the unlikely . Cognitive science, 47 11:e13386

    arXiv doi:10.48550/arxiv.2212.01488 2022

  7. [7]

    Kalev Leetaru and Philip A Schrodt. 2013. Gdelt: Global data on events, location, and tone, 1979--2012. In ISA annual convention, volume 2, pages 1--49. Citeseer

    2013

  8. [8]

    Alex Troy Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . pages 9802--9822

    doi:10.18653/v1/2023.acl-long.546 2022

Show all 27 references
  1. [9]

    Charles A McClelland. 1984. World Event/Interaction Survey (WEIS) Project, 1966-1978. Inter-University Consortium for Political and Social Research

    1984

  2. [10]

    Hannes Mueller and Christopher Rauh. 2018. Reading between the lines: Prediction of political violence using newspaper text. American Political Science Review, 112(2):358--375

    2018

  3. [11]

    Hannes Mueller, Christopher Rauh, and Ben Seimon. 2024. Introducing a global dataset on conflict forecasts and news topics. Data & Policy, 6:e17

    2024

  4. [12]

    Poli Nemkova, Solomon Ubani, Suleyman Olcay Polat, Nayeon Kim, and Rodney D Nielsen. 2023. Detecting human rights violations on social media during russia-ukraine war. arXiv preprint arXiv:2306.05370

    2023 arXiv

  5. [13]

    Kanghui Ning, Zijie Pan, Yu Liu, Yushan Jiang, James Y Zhang, Kashif Rasul, Anderson Schneider, Lintao Ma, Yuriy Nevmyvaka, and Dongjin Song. 2025. Ts-rag: Retrieval-augmented generation based time series foundation models are stronger zero-shot forecaster. arXiv preprint arXi...

    2025

  6. [14]

    Rød, Håvard Hegre, and M

    E. Rød, Håvard Hegre, and M. Leis. 2023. https://doi.org/10.1177/00223433231186452 Predicting armed conflict using protest data . Journal of Peace Research

    2023 doi

  7. [15]

    Kutay Tire, Ege Onur Taga, Muhammed Emrullah Ildiz, and Samet Oymak. 2024. Retrieval augmented time series forecasting. arXiv preprint arXiv:2411.08249

    2024 arXiv

  8. [16]

    Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, and Junhua Zhao. 2024. From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection. Advances in Neural Information Processing Systems, 37:58118--58153

    2024

  9. [17]

    Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023. https://doi.org/10.48550/arXiv.2310.05002 Self-knowledge guided retrieval augmentation for large language models . pages 10303--10315

    2023 doi arXiv

  10. [18]

    Jacob Wood and Deepti Joshi. 2024. Conflict-rag: Understanding evolving conflicts using large language models. In 2024 IEEE International Conference on Big Data (BigData), pages 5459--5467. IEEE

    2024

  11. [19]

    Kevin Wu, Eric Wu, and James Y Zou. 2024. Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence. Advances in Neural Information Processing Systems, 37:33402--33422

    2024

  12. [20]

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. Knowledge conflicts for llms: A survey. arXiv preprint arXiv:2403.08319

    2024 arXiv

  13. [21]

    Silin Yang, Dong Wang, Haoqi Zheng, and Ruochun Jin. 2025. Timerag: Boosting llm time series forecasting via retrieval-augmented generation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2025

  14. [22]

    Chenchen Ye, Ziniu Hu, Yihe Deng, Zijie Huang, Mingyu Derek Ma, Yanqiao Zhu, and Wei Wang. 2024. https://doi.org/10.48550/arXiv.2407.01231 Mirai: Evaluating llm agents for event forecasting . ArXiv, abs/2407.01231

    2024 doi arXiv

  15. [23]

    Temporal data meets llm--explainable financial time series forecasting

    X Yu, Z Chen, Y Ling, S Dong, Z Liu, and Y Lu. Temporal data meets llm--explainable financial time series forecasting. arxiv 2023. arXiv preprint arXiv:2306.11025

    2023 arXiv

  16. [24]

    Hao Zhang, Yuyang Zhang, Xiaoguang Li, Wenxuan Shi, Haonan Xu, Huanshuo Liu, Yasheng Wang, Lifeng Shang, Qun Liu, Yong Liu, and Ruiming Tang. 2024 a . https://doi.org/10.48550/arXiv.2405.19010 Evaluating the external and parametric knowledge fusion of large language models . A...

    2024 doi arXiv

  17. [25]

    Ying Zhang, YangPeng Shen, Gang Xiao, and Jinghui Peng. 2024 b . https://doi.org/10.1109/ACCESS.2024.3505433 Leveraging non-parametric reasoning with large language models for enhanced knowledge graph completion . IEEE Access, 12:177012--177027

    2024

  18. [26]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  19. [27]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.