{"id":"7a10de6d-24e2-415a-96ae-18e5cbff7c5b","arxiv_id":"2412.13511","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CEHA is a new expert-annotated dataset of 500 English news event descriptions for classifying violent conflict events and fine-grained conflict types in the Horn of Africa.","lead":"Researchers built a 500-article dataset of conflict events in the Horn of Africa, labeled by experts for relevance and four fine-grained conflict types. It gives NLP models a new benchmark for detecting violent events in a low-resource region.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GDELT relevance test set is selected by the same Mistral-large model that is later benchmarked, creating a circular bias that undermines the relevance-task benchmark.","rationale":"The paper constructs CEHA, a 500-example conflict-event dataset for the Horn of Africa, with expert annotations for relevance and four fine-grained event types. Its stated contributions are the dataset itself and baseline evidence that the tasks are challenging in low-resource settings. The reader's conditional verdict focuses on the Mistral-large pre-filtering of GDELT as a source of representativeness bias. I agree that this is a serious issue, and I want to sharpen it: because Mistral-large is later evaluated as a baseline, the filtering creates a circular benchmark. The filter removed posts it labeled irrelevant with 38% recall; the surviving No examples are therefore enriched for the filter's false negatives and ambiguous cases. When Mistral-large is then scored on this filtered test set, its performance is not an independent assessment, and comparisons against other models are distorted by the model-specific selection. No per-source or per-filter-status breakdown is reported, so the magnitude is unquantified. This directly threatens the central claim that CEHA is a reliable evaluation resource for relevance classification, and it also qualifies the 'challenging' conclusion. That said, the dataset's construction is otherwise transparent, the annotation guidelines are detailed, and the event-type portion (evaluated only on relevant documents) may be less affected by the GDELT filtering. The issue is addressable: the authors could release the pre-filter labels, evaluate on an unfiltered random sample, or at minimum explicitly disclose the circularity and its likely direction. For these reasons I recommend no change to the CONDITIONAL verdict, but the revision should be required to address the filtering-evaluation circularity, not merely the general distributional caveat.","tokens_in":16552,"tokens_out":7909,"duration_ms":74342,"concrete_test":"Annotate a fresh random sample of GDELT event texts from the same period and region (without Mistral filtering) using the same guidelines, and run the same supervised and LLM baselines on it. If model rankings or F1 scores change materially (e.g., Mistral-large improves by more than 5 points on the No class, or a different model tops the leaderboard), the filtering bias is confirmed. Additionally, for the existing CEHA GDELT examples, compare Mistral-large's predictions with those of the filter on the same texts: if the filter's predicted 'No' examples (i.e., removed ones) are substantially easier for Mistral-large than the retained ones, the test set is biased by construction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 describes applying a few-shot Mistral-Large model to remove irrelevant GDELT posts before sampling; the model achieved only 38% recall on the No class (Appendix C). This filtering is not a neutral downsampling: it preferentially removes 'easy' No examples (those Mistral-large confidently rejects) and leaves 'hard' No examples that Mistral-large misclassifies as relevant or finds ambiguous. The final relevance test set therefore has a No distribution shaped by Mistral-large's error pattern. In Section 5.2, Mistral-large is itself evaluated on this filtered test set (zero-shot and six-shot, Tables 8 and 10). Its reported F1 and the apparent difficulty ordering of models are not independent: Mistral-large is measured on examples that were adversarially selected to include its own false negatives, while other models are affected by the same non-random selection. This circularity is not disclosed in the main text, and no per-source or per-filter-status results are provided to quantify the bias. Consequently, the benchmark conclusion that Event-relevance Classification is 'challenging' (Section 5.2) may be an artifact of the filtering procedure, and the dataset's reuse as a fair evaluation resource for relevance classifiers is compromised.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CEHA, a new benchmark dataset of 500 English-language event descriptions from ACLED and GDELT focusing on violent conflict events in the Horn of Africa. The dataset provides binary Event-relevance labels and fine-grained multi-label Event-type annotations (Tribal/Communal/Ethnic Conflict, Religious Conflict, Socio-political Violence Against Women, Climate-Related Security Risks) produced by subject-matter experts. The authors describe an iterative annotation-guideline development process, two pilot studies with inter-annotator agreement measurement, and a filtering step that used a few-shot Mistral-large model to remove irrelevant GDELT posts before human annotation. They then benchmark supervised models (BERT, RoBERTa, T5) and several LLMs (Mixtral, Mistral-large, DBRX, GPT-4o, Llama3) on both tasks in zero-shot, few-shot, and fine-tuned settings, reporting that the tasks are challenging in low-resource conditions and that LLMs benefit from in-context learning.","tokens_in":16789,"tokens_out":3404,"duration_ms":33456,"significance":"If the dataset is sound, it fills a real gap: existing conflict-event resources such as ACLED and GDELT do not provide fine-grained, region-specific event types that are directly relevant to humanitarian-peace-development stakeholders in the Horn of Africa. The paper's strengths include a transparent annotation process with domain experts, two pilot rounds with reported Cohen's kappa, clearly summarized label definitions, and a broadly scoped baseline comparison across multiple model families. The public release of the dataset and code is also a positive contribution. However, the reliability of the Event-relevance benchmark is undermined by the GDELT pre-filtering step, which selected test-set negatives based on the very model that is later evaluated on them. This issue is load-bearing for the central claim that the relevance task is challenging and that the dataset is a reusable evaluation resource.","major_comments":[{"comment":"The GDELT pre-filtering step is not distribution-neutral. The authors report that the Mistral-large filter achieved 38% recall on the No class (Appendix C), meaning it removed only the irrelevant posts it could confidently recognize and retained the harder ones, many of which were its own false negatives. Because the final test set is drawn from this filtered pool, the Event-relevance test distribution is shaped by Mistral-large's error pattern. In Section 5.2 and Table 8, Mistral-large is then evaluated on this exact filtered test set, so its F1 score and the observed difficulty ordering of models are not independent of the data-generation process. This is not circularity in label generation (the gold labels are human), but it is a selection bias that compromises the benchmark's validity for relevance classification. Please provide per-source and per-filter-status performance, or re-sample the test No examples from unfiltered GDELT and show that the main comparative conclusions still hold.","section":"Section 3.3"},{"comment":"The final annotation protocol assigns each item to a single annotator, with no adjudication of disagreements. The measured average pairwise Cohen's kappa for Event-relevance is only 0.63, which is moderate agreement; for a benchmark intended to evaluate models, label noise at this level can materially affect reported scores. The paper should either add an adjudication step for disagreements, report the distribution of labels per annotator in the test split, or explicitly quantify how single-annotator noise affects the reliability of the benchmark conclusions.","section":"Section 3.3"},{"comment":"The claim that 'precision remains relatively low for most LLM model variations, which demonstrates the challenging nature of the Event-relevance Classification task' is directly tied to the filtered test set. Since the filtered No examples are disproportionately those that Mistral-large failed to reject, the low precision observed for many models may be inflated by the sampling procedure rather than reflecting the true difficulty of relevance classification on raw GDELT data. The authors should temper this conclusion or provide evidence that the filtering bias does not drive the result.","section":"Section 5.2"}],"minor_comments":[{"comment":"The paper describes the 4:1:5 split as providing a 'robust benchmark (test) set'; with 250 test examples and only 23 Climate-Related Security Risks events total, the per-class test sizes are small, and the term 'robust' is an overstatement.","section":"Section 3.4"},{"comment":"There is an inconsistent reference to 'GPT4' in the prose discussing Table 10, while the model is elsewhere called 'GPT-4o'; please harmonize the naming.","section":"Table 10"},{"comment":"The ACLED sample covers 2015-2024 while the GDELT sample covers only 2020-2024; this time-period mismatch should be stated as a potential source of distribution shift, and its possible effect on the benchmark results should be discussed.","section":"Section 3.2"},{"comment":"The paper reports the filter's performance only on the No class; reporting precision, recall, and F1 for the Yes class as well would clarify how the filter's behavior differs between classes and help readers assess the selection bias more completely.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The core dataset contribution is potentially valuable, and the reported annotation process is careful in many respects. The main obstacle is the GDELT pre-filtering bias, which directly affects the relevance benchmark's validity. If the authors can provide an analysis on unfiltered data or otherwise demonstrate that the filter does not change the comparative conclusions, the paper could become acceptable. I would also encourage the editor to consider whether the single-annotator labeling with only moderate agreement is sufficiently reliable for a benchmark resource; adjudication or per-annotator analysis would strengthen the paper. The 'first dataset at this level of regionality' claim is plausible but should be checked against any concurrent or emerging resources in the AI4SG conflict-event space."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe CEHA paper is a genuinely useful resource: 500 expert-labeled English event descriptions from ACLED and GDELT for the Horn of Africa, with binary relevance labels and four conflict-cause-oriented event types (ethnic, religious, violence against women, climate-related security risks). That combination is new, and it fills a real gap for a crisis-prone region. The annotation process is careful: two pilot rounds, inter-annotator agreement improved to 0.63 on relevance and 0.72 on event types, and the final labels come from four experts with spot checks. The paper is also honest about limitations: English-only, small size, class imbalance.\n\nThe baseline experiments are standard and show the tasks are challenging, especially for climate-related security risks. The authors release code and data, which makes the resource immediately usable.\n\nThe main soft spot is the GDELT pre-filtering step. To handle the large volume of irrelevant GDELT posts, the authors used a few-shot Mistral-large model to remove obvious 'No' examples before sampling. That filter had 89% precision on the No class but only 38% recall, so it removed only the easy negatives. The remaining GDELT No examples in CEHA are therefore disproportionately the ones Mistral-large got wrong or was unsure about. Since Mistral-large is later evaluated on the same test set (zero-shot and six-shot), its reported F1 is likely biased downward relative to a random GDELT sample, and the overall difficulty of the relevance task is inflated. This is not circular in the training-data sense: the gold labels are human, not model-generated. But it is a selection bias that affects the interpretation of the relevance benchmark. The paper discloses the filter's recall in Appendix C but does not discuss the implications for the test set, and it does not provide per-source or per-filter-status results to let readers quantify the effect. This should be fixed before publication—either by re-annotating a random GDELT sample for comparison or by clearly framing the relevance task as applying to 'filtered' GDELT data.\n\nThe event-type classification results are less affected by this issue because they are evaluated only on relevant documents and the labels are independent of the filter. That part of the benchmark is solid.\n\nOverall, this is a solid resource paper with one addressable methodological caveat. It deserves peer review; a good reviewer should push on the filtering bias and ask for a more transparent evaluation. I'd take it to reading group.","headline":"Useful new conflict-event dataset for a low-resource region, with a genuine but addressable bias in the relevance test set from the GDELT pre-filter.","tokens_in":17283,"tokens_out":3629,"would_cite":true,"duration_ms":33546,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 500-event benchmark maps the causes behind Horn of Africa conflict.","keywords":["conflict event dataset","Horn of Africa","event-type classification","event relevance classification","low-resource NLP","expert annotation","ACLED","GDELT"],"falsifier":"Take a fresh random sample of GDELT conflict-coded Horn-of-Africa articles, have the same annotators label them without any model pre-filter, and compare the relevance distribution and model F1 against CEHA's test set; large differences would show the pre-filter biased the benchmark.","tokens_in":16376,"feed_emoji":"⚔️","tokens_out":8551,"duration_ms":70490,"temperature":0.7,"pith_summary":"The paper proposes CEHA, a 500-item benchmark of English event descriptions from the Horn of Africa, expert-annotated for binary conflict-event relevance and for fine-grained, cause-focused event types: tribal/communal/ethnic conflict, religious conflict, socio-political violence against women, and climate-related security risks. Its central claim is that no existing conflict-event dataset provides this combination of regional focus and causal event-type granularity, so CEHA fills a gap in NLP resources for humanitarian and peacebuilding work. The authors' baseline results support a second claim: both relevance and event-type classification are genuinely hard in this low-resource setting, with the rare climate-related class the hardest, and few-shot large language models are competitive with small fine-tuned models. A reader who cares about conflict early warning and AI-for-social-good applications would care because the dataset offers a shared test bed for models that try to identify why violence happens, not just that it happened.","feed_headline":"New 500-event benchmark maps Horn of Africa conflict causes","feed_subtitle":"Ethnic, religious, gender-based, and climate-linked violence get separate labels for low-resource NLP evaluation.","key_machinery":"The load-bearing object is the CEHA dataset itself: 500 English event descriptions with gold labels. Its distinctive mechanism is the annotation scheme, which first separates relevant violent-conflict events from irrelevant posts using three criteria (Horn of Africa location, violence directed at people, and a specific rather than summary event), then assigns multi-label event types that emphasize the cause of the conflict rather than the action. The four cause-focused types are tribal/communal/ethnic conflict, religious conflict, socio-political violence against women, and climate-related security risks, with an Other category used only when none of the four apply. This scheme is what turns generic conflict event detection into a benchmark that can measure whether a model understands the conflict dynamics stakeholders care about.","core_discovery":"On its own terms, the paper's discovery is CEHA itself and the evidence that the tasks it defines are non-trivial. Each of the 500 descriptions comes from ACLED or GDELT, 250 from each, and was labeled by subject matter experts working from a refined annotation guideline; 310 descriptions are conflict-relevant, and among relevant events the four event types are distributed unevenly, with tribal/communal/ethnic conflict most common at 115 instances and climate-related security risks rarest at 23. On the held-out test set of 250 items, the best supervised model reaches 83.09 F1 on relevance and 74.80 F1 on event-type classification, while the best six-shot LLM reaches 87.16 and 75.80 respectively, and most models show comparatively low precision on relevance. The authors read this as evidence that fine-grained conflict-event identification in the region remains an open challenge rather than a solved one.","pith_inferences":["The automatic pre-filter used to clean GDELT posts achieved only 38 percent recall on the irrelevant class, so CEHA's relevance benchmark likely over-represents easy negative examples; a version built from an unfiltered random sample would test whether the relevance conclusions generalize.","The cause-focused event-type scheme could be transferred to other conflict-prone regions, reusing the four definitions with local expert calibration.","Because CEHA is English-only, its labels could seed cross-lingual annotation or evaluation of Amharic, Somali, and Arabic news sources.","The low F1 for climate-related security risks points to a concrete gap: building larger targeted collections of climate-conflict events may improve model performance more than general-purpose scaling."],"forward_implications":["Event-relevance and event-type classification in the Horn of Africa remain hard: the best relevance F1 is 87.16 and the best event-type F1 is 75.80.","A small expert-annotated set of 500 items is enough to expose meaningful performance differences across models in low-resource settings.","Few-shot LLM prompting is a viable route for low-resource conflict-event classification, often matching or surpassing fine-tuned BERT, RoBERTa, and T5 baselines.","Climate-related security risks are the hardest event type to classify, partly because they are rare and under-represented in existing training resources.","CEHA is positioned as a shared evaluation benchmark for future conflict-event and AI-for-social-good NLP work in the region."],"supporting_citations":[{"why":"Provides the GDELT source data, with CAMEO-coded events, from which half of CEHA is sampled.","marker":"Leetaru and Schrodt, 2013"},{"why":"Provides the ACLED source data and expert-curated conflict event records, from which the other half of CEHA is sampled.","marker":"Raleigh et al., 2023"},{"why":"Defines the CAMEO ontology used to filter GDELT events by violence level.","marker":"Schrodt, 2012"},{"why":"Serves as the closest precedent for a small expert-annotated event dataset and supplies the T5 question-answering formulation used for baselines.","marker":"Ran et al., 2023"},{"why":"Defines ACE05, whose limited conflict event types motivate CEHA's finer-grained ontology.","marker":"Doddington et al., 2004"},{"why":"Provides BERT, one of the supervised baselines for both tasks.","marker":"Devlin et al., 2018"},{"why":"Provides RoBERTa, the best supervised relevance classifier in the experiments.","marker":"Liu et al., 2019"},{"why":"Provides T5, the best supervised event-type classifier, via a question-answering formulation.","marker":"Raffel et al., 2023"},{"why":"Establishes event extraction as question generation and answering, the setup used for the T5 baselines.","marker":"Lu et al., 2023"}],"fun_headline_variants":["CEHA dataset: 500 Horn of Africa conflict events with fine-grained labels","New benchmark tests NLP on low-resource Horn of Africa conflict types","Horn of Africa conflict events remain a hard NLP problem, CEHA shows","CEHA: fine-grained conflict events in Horn of Africa for NLP research"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The sampling of irrelevant examples relies on an automatic language-model filter that removed only 38 percent of the irrelevant posts it should have removed, so the No examples in CEHA may not represent the mix of irrelevant content a live news feed would deliver.","fun_headline_variants_meta":{"raw":{"variants":["CEHA dataset: 500 Horn of Africa conflict events with fine-grained labels","New benchmark tests NLP on low-resource Horn of Africa conflict types","Horn of Africa conflict events remain a hard NLP problem, CEHA shows","CEHA: fine-grained conflict events in Horn of Africa for NLP research"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001231,"raw_usage":{"total_tokens":5045,"prompt_tokens":919,"completion_tokens":4126,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":4046}},"tokens_in":535,"tokens_out":4126,"duration_ms":21726,"temperature":1.0,"reasoning_tokens":4046,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:02:49.819351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of GDELT conflict-coded Horn-of-Africa articles, have the same annotators label them without any model pre-filter, and compare the relevance distribution and model F1 against CEHA's test set; large differences would show the pre-filter biased the benchmark.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ACLED source data and expert-curated conflict event records, from which the other half of CEHA is sampled."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CAMEO ontology used to filter GDELT events by violence level."}],"review_version":1}