{"id":"4104f2eb-ffa7-4f95-bfdf-8d6d51b195f8","arxiv_id":"2412.15060","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A conflict-specific BERT model beats much larger general-purpose LLMs on classifying political violence texts when the larger models are used off the shelf, and is hundreds of times faster.","lead":"This paper benchmarks ConfliBERT, a small conflict-focused language model, against much larger general-purpose models like Gemma, Llama, and Qwen on classifying political violence texts, and reports that ConfliBERT is more accurate and hundreds of times faster. The result matters for conflict researchers because it suggests that fast, cheap event data coding does not require huge generative models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generative baselines in Sections 4.2–4.3 are evaluated without documented prompts; the reported gaps may be prompt artifacts.","rationale":"The reader's weakest assumption correctly flags asymmetric fine-tuning as a fairness concern, but the paper partially addresses this by including ConfLlama, a fine-tuned Llama, in Section 5. A more fundamental, unresolved issue is the absence of any prompt specification for the zero-shot generative baselines in the two headline comparisons (Tables 1 and 2). The appendix prompt only covers the GTD multi-label task, not binary classification or NER. This is a concrete reproducibility gap: the reported gaps could shrink or disappear with different prompts, independent of fine-tuning. The reader's focus on fine-tuning asymmetry is related but not identical; hence partial agreement. The conditional verdict remains appropriate because the concern is real but does not invalidate the paper's core evidence; it requires additional documentation or experiments to confirm.","tokens_in":82,"tokens_out":4278,"duration_ms":43445,"concrete_test":"Contact the authors for the exact zero-shot prompts used for Tables 1 and 2, or if unavailable, independently re-run Gemma 2 (9B) and Llama 3.1 (8B) on the same BBC and re3d test splits with at least three reasonable prompt variants (e.g., label definitions, few-shot demonstrations, output format constraints). Compare weighted F1 against Table 1/2 values. If any variant lifts Gemma/Llama weighted F1 by more than 0.10 (e.g., from 0.40 to >0.50 on NER), the reported superiority is not robust to prompt choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central superiority claim rests on Tables 1 and 2, comparing fine-tuned ConfliBERT to zero-shot Gemma 2 and Llama 3.1. However, the only prompt disclosed (Appendix A) is for the GTD multi-label task; no prompts are given for the BBC binary classification or re3d NER tasks. Without knowing the instructions used, the comparison is not reproducible and may be unfair: for example, Gemma 2's zero recall on the conflict class in Table 1 is consistent with a prompt that omitted the conflict/non-conflict label definition or asked for a different output format. The reader's concern about asymmetric fine-tuning is partially mitigated by ConfLlama in Section 5, but the missing prompting information is a separate, unresolved gap affecting the two primary comparative tables. Section 5's own admission that fine-tuning could improve generative models does not excuse the lack of prompt documentation for the zero-shot baselines. This is load-bearing because the entire 'superior accuracy/precision/recall' claim for the two core tasks (BC and NER) depends on these unreported experimental conditions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reviews ConfliBERT, a BERT-style language model pretrained on a curated 33.7 GB corpus of conflict and political violence texts, and compares it against generative LLMs (Gemma 2, Llama 3.1, Qwen 2.5) on three tasks: binary conflict classification on BBC news, named entity recognition on re3d, and multi-label attack-type classification on Global Terrorism Dataset (GTD) texts. The central claim is that a fine-tuned domain-specific encoder, ConfliBERT, achieves higher accuracy, precision, and recall than much larger general-purpose LLMs on these tasks while being hundreds of times faster. The paper also introduces ConfLlama, a fine-tuned Llama 3.1 8B baseline, and shows that ConfliBERT retains an advantage on the GTD multi-label task even against this fine-tuned generative model.","tokens_in":22167,"tokens_out":5166,"duration_ms":43092,"significance":"If the comparison were fully controlled and reproducible, the result would be practically important: it would show that a small, open, domain-specific encoder can outperform or match much larger generative models at substantially lower cost for event-coding tasks in conflict research. The paper deserves credit for releasing the model publicly, using external benchmark data (BBC, re3d, GTD), including a fine-tuned generative baseline (ConfLlama), and presenting ROC and precision-recall curves rather than relying only on point accuracies. However, the strength of the stated conclusion currently exceeds the evidence: the two primary comparative tables rest on undocumented zero-shot prompts, an unexplained dataset/sample mismatch, and the speed comparison contains an acknowledged hardware incomparability.","major_comments":[{"comment":"The only prompt disclosed in Appendix A is for the GTD multi-label task. The binary classification (BBC) and NER (re3d) evaluations, which anchor the abstract's superiority claim, are reported without the exact instruction strings, output format constraints, or parsing procedures used for Gemma 2 and Llama 3.1. Without these, Tables 1 and 2 are not reproducible, and the zero recall of Gemma 2 for the conflict class could be a prompt artifact rather than a model limitation. Please provide the full prompts, decoding settings, and any post-processing code for all zero-shot evaluations.","section":"§4.2-4.3, Appendix A"},{"comment":"Section 4.1 describes the BBC test set as 735 articles, but Table 1 reports support values of 269 non-conflict and 53 conflict (total 322). Section 4.2 further says Table 1 covers \"the BBC News and re3d texts,\" although re3d is used elsewhere as an NER corpus. This discrepancy makes it impossible to know which dataset, sample, and preprocessing produced the reported numbers. Please state the exact dataset, class distribution, and any subsampling or filtering steps.","section":"§4.1 and Table 1"},{"comment":"The footnote to Table 4 states that ConflLlama timing measurements were performed on Delta HPC resources and are \"not directly comparable\" to the other models, yet the text immediately after Table 4 claims that \"Processing times were measured on identical hardware configurations to ensure fair comparison.\" This contradiction affects the relative document speed column for ConflLlama and leaves the speed ranking ambiguous for that model. Please specify which rows are hardware-comparable, or remove ConflLlama from the speed comparison entirely.","section":"§5.1 and Table 4"},{"comment":"The comparison is asymmetric: ConfliBERT and ConfLlama are fine-tuned on task-specific data, while Gemma 2, Llama 3.1, and Qwen are evaluated zero-shot with a single prompt. The paper itself admits in Section 5 that \"some pre-training of the generative LLMs could bring their performance up to or exceeding the performance of ConfliBERT.\" The abstract nevertheless states the superiority claim without this qualifier. Either add fine-tuned versions of the generative models for the BBC and re3d tasks (as ConfLlama provides for GTD), or explicitly restrict the claim to zero-shot comparisons.","section":"Abstract, §1, §4.2-4.3, §5"}],"minor_comments":[{"comment":"The phrase \"ConfliBERT is a LLM\" and similar constructions should be \"an LLM\" for grammatical correctness.","section":"Abstract, §1"},{"comment":"The fine-tuned Llama model is named both \"ConflLlama\" and \"ConfLlama\" in different places (e.g., §4.1 vs. Table 4); please unify the spelling.","section":"Various"},{"comment":"Section 3 says ConfliBERT was trained on a 33.7 GB corpus, while Section 6 says about 200 GB of combined training data are invested in ConfliBERT and its Spanish and Arabic variants; clarify the per-model training corpus size.","section":"§3 vs. §6"},{"comment":"The reported maximum memory values for ConfliBERT, Llama 3.1, and Gemma 2 are all near 950 MB, which seems implausible for 8B/9B parameter models; describe how memory was measured and whether this is peak GPU memory per batch rather than model memory.","section":"Table 3"},{"comment":"Figure 4 is labeled 2017-2021, but the text says the test period is 2017-2020; make the time span consistent.","section":"Figure 4 and §5.3"},{"comment":"The sentence \"the the original GTD structure\" contains a duplicated article; also, the Brown (2020) reference is incomplete and should be replaced with the full NeurIPS citation.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is closer to a review/position paper than a new-method paper; its main empirical novelty is the GTD comparison and the ConfLlama baseline. The heavy self-citation pattern is understandable given the model's provenance, but the contribution would be stronger with an explicit statement that the generative baselines are zero-shot. I recommend major revision rather than rejection because the missing prompt documentation, dataset clarification, and hardware-comparability statement are obtainable within the manuscript's scope, and the fine-tuned ConfLlama experiment partially addresses the fairness concern for one task."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: that stress-test note is right. Appendix A only documents the prompt for the GTD multi-label task. For the BBC binary classification (Table 1) and re3d NER (Table 2) there is no disclosed prompt for Gemma 2 or Llama 3.1, so the zero-shot numbers are not reproducible. Gemma's exact-zero performance on the conflict class in Table 1 looks like an output-format or label-definition failure, not a real incapacity. That is load-bearing because the abstract's 'superior accuracy, precision, recall' claim rests on these two tables.\n\nWhat is actually new: the head-to-head benchmark against Gemma 2, Llama 3.1, and Qwen 2.5, the ConfLlama fine-tune, and the GTD multi-label evaluation with label cardinality and Hamming loss. The paper ships the model on HuggingFace and GitHub, describes the event-coding problem clearly, and includes a sensible validity check in Figure 4 showing how misclassification distorts cumulative event counts. The GTD results with fine-tuned ConfLlama give the core claim partial cover: fine-tuning a Llama on the task narrows but does not close the gap. That is real evidence, not just a zero-shot artifact.\n\nSoft spots, in proportion. First, the missing prompts are a major reproducibility gap for two of the three tasks. Second, there are no error bars or significance tests, so the Table 1 and 2 gaps could be within noise. Third, the speed comparison in Table 4 is caveated for ConfLlama, but the abstract's 'hundreds of times faster' doesn't reflect the fine-tuned comparison. Fourth, the abstract should say 'zero-shot' when claiming superiority over Gemma/Llama/Qwen for BC and NER. These are all fixable. Adding the prompts, fine-tuning the generative baselines for the same tasks, and reporting uncertainty would make the central claim solid.\n\nWho is this for: conflict researchers and political-methodology people deciding whether a small domain model can replace a big generative model for event coding. It is a useful benchmark and a good teaching case for evaluation design. I would send it to peer review with a major revision request, not desk reject. The reader's conditional verdict is about right, though I weight the prompt gap more heavily than the fine-tuning asymmetry.","headline":"A useful benchmark with a real GTD result, undercut by missing prompts for the zero-shot baselines on two of three tasks; fixable, but the headline as written overstates the evidence.","tokens_in":22709,"tokens_out":4138,"would_cite":true,"duration_ms":27629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small conflict-trained model beats much larger generalist LLMs on political violence text classification and named entity recognition.","keywords":["ConfliBERT","political conflict","event data","named entity recognition","text classification","domain-specific language models","large language models","computational efficiency"],"falsifier":"Fine-tune Gemma 2, Llama 3.1, and Qwen 2.5 on the same BBC, re3d, and GTD training splits used for ConfliBERT, using similar epochs and adaptation methods, and evaluate on the same test splits; if their F1, AUC, and multi-label metrics reach or exceed ConfliBERT's, the paper's central claim fails.","tokens_in":21726,"feed_emoji":"⚡","tokens_out":5905,"duration_ms":34810,"temperature":0.7,"pith_summary":"The paper argues that a small, domain-specific language model trained on political conflict texts can outperform far larger general-purpose large language models on the tasks conflict researchers actually need: filtering relevant reports, classifying attack types, and extracting named entities. It reports that fine-tuned ConfliBERT beats Gemma 2, Llama 3.1, and Qwen 2.5 on accuracy, precision, recall, and F1 on BBC, re3d, and GTD benchmarks, and is hundreds of times faster on the tested hardware. A sympathetic reader should care because this suggests that for structured event-data extraction, domain knowledge and fine-tuning matter more than raw model scale, and that cheap, open models can sustain near-real-time conflict monitoring.","feed_headline":"Conflict-trained model beats big LLMs, runs hundreds of times faster","feed_subtitle":"For classifying political violence and extracting actors, domain-trained models beat raw scale on accuracy and speed.","key_machinery":"The load-bearing object is ConfliBERT, a BERT-style transformer trained on conflict-specific text; BERT-style encoders process whole inputs in parallel and can be fine-tuned cheaply, in contrast to autoregressive generative LLMs that decode sequentially. Fine-tuning with a weighted multi-task loss on binary classification, multi-class attack labels, and NER transfers domain knowledge from the pretraining corpus into the specific task labels. The paper's comparisons also introduce ConfLlama, a Llama-3-8B model fine-tuned on GTD via QLoRA, as a same-scale generative check on whether fine-tuning rather than architecture drives the result.","core_discovery":"ConfliBERT is a BERT-style encoder pretrained on a curated 33.7 GB corpus of conflict and political-violence text. The paper's central claim is that after task-specific fine-tuning, this relatively small model is more accurate than the much larger generative models Gemma 2 (9B), Llama 3.1 (8B), and Qwen 2.5 (14B) on binary conflict classification, nine-way attack-type classification, and NER on terrorism reports. On the binary BBC task, ConfliBERT's weighted F1 was 0.8706 versus 0.7605 for Gemma 2 and 0.7662 for Llama 3.1, and the larger models essentially failed to detect conflict texts at all. On NER using re3d, ConfliBERT's weighted F1 was 0.5981 versus 0.3987 and 0.3809 for the larger models. On GTD multi-label attack classification, ConfliBERT reached 79.38 percent subset accuracy with the lowest Hamming loss, while taking about 3.5 seconds for classification and 1.4 seconds for NER compared to hundreds of seconds for the generative models.","pith_inferences":["One extension the paper leaves implicit: if the larger generative LLMs were fine-tuned on the same task data with comparable effort, the accuracy gap could narrow or reverse, so a natural next experiment is LoRA fine-tuning of Gemma, Llama, and Qwen on the GTD training split.","The speed and accuracy pattern likely generalizes beyond conflict: for high-volume text-as-data tasks in other social-science domains, a small domain-pretrained encoder trained on curated text may beat a much larger general model.","A stronger test of the method would measure downstream inference quality, such as whether changepoint analyses differ when event counts come from ConfliBERT versus human-coded GTD, rather than only token-level accuracy.","Because the model and evaluation data are public, the comparison can be extended to new languages and event ontologies without the licensing and cost barriers of closed generative models."],"forward_implications":["Conflict researchers can deploy ConfliBERT on conventional hardware to filter and code large news corpora in near-real time, at a small fraction of the compute cost of prompting 8B-14B generative models.","Event-data pipelines that currently rely on prompting general LLMs can expect higher F1 and better class balance on rare attack types by switching to fine-tuned domain encoders.","Downstream time-series analyses of terrorism are less likely to be distorted by systematic over- or under-classification of event types, since ConfliBERT's predicted counts track GTD's human-coded counts more closely.","The comparison provides a replicable baseline for future event-classification models: same datasets, same metrics, and a documented prompt for the generative baselines."],"supporting_citations":[{"why":"Introduces ConfliBERT and the original comparisons to BERT baselines that this paper builds on.","marker":"Hu et al., 2022"},{"why":"Provides the BERT transformer architecture that ConfliBERT is based on.","marker":"Devlin et al., 2018"},{"why":"Supplies the BBC News dataset used for the binary conflict classification task.","marker":"Greene and Cunningham, 2006"},{"why":"Provides the re3d dataset used for named entity recognition evaluation.","marker":"DSTL, 2018"},{"why":"Supplies the Global Terrorism Database texts and human-coded labels used for attack-type classification.","marker":"LaFree and Dugan, 2007"},{"why":"Is the Llama 3.1 baseline model compared against ConfliBERT.","marker":"Dubey et al., 2024"},{"why":"Is the Gemma 2 baseline model compared against ConfliBERT.","marker":"Team Gemma et al., 2024"},{"why":"Is the Qwen 2.5 baseline model compared against ConfliBERT.","marker":"Hui et al., 2024"},{"why":"Supports the general claim that domain-adaptive pretraining improves downstream task performance.","marker":"Gururangan et al., 2020"}],"fun_headline_variants":["Conflict BERT beats big LLMs on accuracy and speed","Small conflict model outperforms large LLMs, 100x faster","Domain-trained BERT excels over giant LLMs on conflict","ConfliBERT: faster, more precise than huge language models","Tiny model tops 14B-param LLMs for violence classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is fair: ConfliBERT and ConfLlama were fine-tuned on the task data, while Gemma 2, Llama 3.1, and Qwen 2.5 were evaluated off the shelf with a single hand-written JSON prompt, and if those larger models were fine-tuned with comparable effort the accuracy and speed gaps could narrow or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Conflict BERT beats big LLMs on accuracy and speed","Small conflict model outperforms large LLMs, 100x faster","Domain-trained BERT excels over giant LLMs on conflict","ConfliBERT: faster, more precise than huge language models","Tiny model tops 14B-param LLMs for violence classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000569,"raw_usage":{"total_tokens":2694,"prompt_tokens":947,"completion_tokens":1747,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1659}},"tokens_in":563,"tokens_out":1747,"duration_ms":10280,"temperature":1.0,"reasoning_tokens":1659,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:39:50.555774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune Gemma 2, Llama 3.1, and Qwen 2.5 on the same BBC, re3d, and GTD training splits used for ConfliBERT, using similar epochs and adaptation methods, and evaluate on the same test splits; if their F1, AUC, and multi-label metrics reach or exceed ConfliBERT's, the paper's central claim fails.","supporting_citations":[{"cited_title":"and Cunningham, P","cited_arxiv_id":null,"evidence_quote":"Supplies the BBC News dataset used for the binary conflict classification task."},{"cited_title":"Relationship and entity extraction evaluation dataset","cited_arxiv_id":null,"evidence_quote":"Provides the re3d dataset used for named entity recognition evaluation."},{"cited_title":"and Dugan, L","cited_arxiv_id":null,"evidence_quote":"Supplies the Global Terrorism Database texts and human-coded labels used for attack-type classification."}],"review_version":1}