{"id":"dd2a8d09-0e79-4317-ba95-cca382719ef9","arxiv_id":"2506.00980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LEMONADE is a new 20-language, expert-annotated conflict event dataset for abstractive event extraction, and ZEST, a zero-shot retrieval entity linker, beats prior zero-shot baselines but trails supervised models.","lead":"This paper presents LEMONADE, a dataset of nearly 40,000 conflict events in 20 languages, built from expert-annotated ACLED data. It also defines abstractive event extraction and introduces ZEST, a zero-shot retrieval system for linking event participants to a domain database.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval database leaks test evidence: Appendix A.2 says entity descriptions are generated from all annotated news articles, including Apr 2024–Jan 2025 test-period articles, so ZEST's 45.7 AEL F1 and 58.3 E2E F1 may be inflated; the zero-shot claims need a leakage-free re-evaluation.","rationale":"The paper's main novelty is LEMONADE and the AEE/AEL formulation; ZEST is presented as a strong zero-shot baseline. The reader's weakest_assumption matches the key evaluation vulnerability: Appendix A.2 documents that entity descriptions are formed from all news articles annotated for each entity. Because the temporal split is described in Section 4.1 (train Jan–Mar 2024; dev/test Apr 2024–Jan 2025) and descriptions are generated over the full 13-month collection, test-period articles can influence the descriptions. ZEST explicitly generates queries to approximate gold entity descriptions, so the leaked descriptions can boost retrieval precisely where it matters. The effect is not speculative: sample descriptions in Appendix D contain event-specific and date-specific details (e.g., anti-HTS protests in 2024, named leaders) that could only come from the very articles being evaluated. This makes the 'zero-shot' label misleading and the ZEST-versus-OneNet gap unreliable. However, the flaw is contained to the zero-shot evaluation: supervised models (Aya Expanse, Llama) are trained on gold annotations and evaluated without the description database, and the dataset statistics themselves are independent. Thus the paper's contribution stands, but its central abstract claim needs to be re-derived under a non-leaking construction. The concrete test would settle the magnitude of the effect; if the drop is small, the concern fails. I therefore keep the reader's CONDITIONAL recommendation: the paper is publishable in principle, but the zero-shot results should be regenerated or explicitly caveated with a train-only entity-description protocol.","tokens_in":43248,"tokens_out":5323,"duration_ms":53966,"concrete_test":"Regenerate all entity descriptions using only ACLED articles dated January 1–March 31, 2024 (the training period). Rerun ZEST(GPT-4o) and OneNet(GPT-4o) on the existing test split with identical prompts, candidate counts, and embedding models, and recompute AEL F1 and E2E F1. If ZEST's AEL F1 drops materially (or the gap to OneNet narrows substantially), the headline zero-shot numbers are contaminated. As a secondary check, count the fraction of test entities whose new description contains a distinctive 5-gram (or named entity) that also appears in the paired test article; a high fraction confirms the original descriptions encode test evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim is ZEST's zero-shot AEL F1 of 45.7% and the 58.3% end-to-end F1. ZEST's stage-1 retrieval embeds entity descriptions, and its LLM is prompted to generate queries that 'closely match the descriptions of the gold entities.' Appendix A.2 states: 'These descriptions are generated by GPT-4o using the news articles that are annotated to have involved each entity.' LEMONADE spans January 2024–January 2025, and the test split covers April 2024–January 2025. Consequently, for any entity involved in a test article, its stored description may contain facts, names, or wording drawn from that exact test article. This artificially raises query–description similarity in Stage 1, and Stage 2's filtering prompt also shows the candidate description alongside the test article, compounding the leakage. OneNet receives the same leaked descriptions, but its bottleneck is span extraction plus disambiguation from spans; the reported 22-point gap could substantially reflect one-sided retrieval advantages rather than true zero-shot linking ability. The supervised numbers and the dataset itself do not depend on these descriptions and remain credible. The evaluation protocol is therefore internally inconsistent with the advertised zero-shot property: the system has effectively seen each test document, through the entity descriptions, before inference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LEMONADE, a multilingual document-level abstractive event extraction dataset derived from ACLED expert annotations, covering 39,786 events in 20 languages with a curated entity database of 10,707 entities. It defines abstractive event extraction (AEE) and its subtask abstractive entity linking (AEL), and presents ZEST, a retrieval-based zero-shot entity linker. Experiments compare zero-shot and supervised models on event detection, argument extraction, entity linking, and end-to-end extraction, reporting ZEST AEL F1 of 45.7 and a best zero-shot end-to-end F1 of 58.3, versus 78.4 for the best supervised model. The paper's central performance claims depend on ZEST's zero-shot evaluation, which is compromised by entity descriptions generated from test-period articles.","tokens_in":43502,"tokens_out":6722,"duration_ms":63491,"significance":"If the zero-shot results survive a leakage-free re-evaluation, the paper would provide a substantial resource: expert-annotated multilingual event data with temporal splits, a domain-specific entity database, and a new abstractive task formulation. The dataset construction is transparent, the temporal split is a good design choice, and the supervised benchmarks are carefully adapted from existing models. However, the central zero-shot claim is currently not supported because the retrieval database contains entity descriptions generated from test-period news articles. This issue does not affect the dataset itself or the supervised results, but it invalidates the headline comparison between ZEST and OneNet as a zero-shot comparison. The contribution is therefore potentially significant but needs a corrected evaluation.","major_comments":[{"comment":"The entity descriptions stored in the retrieval database are generated by GPT-4o \"using the news articles that are annotated to have involved each entity\" (Appendix A.2), and the corpus spans January 2024–January 2025 with the test split covering April 2024–January 2025 (Section 4.1). Consequently, descriptions for entities appearing in test articles can contain facts, names, or wording drawn from those exact test documents. ZEST's Stage 1 prompt explicitly instructs the LLM to generate queries that \"closely match the descriptions of the gold entities\" (§5.1), so the retrieval step is effectively given test-document content through the descriptions; Stage 2 then presents each candidate description alongside the test article (Table 17). This makes the reported zero-shot AEL F1 of 45.7 and the end-to-end F1 of 58.3 inflated, and it breaks the advertised zero-shot property. The comparison with OneNet is also one-sided, because ZEST's query-generation can exploit the leaked descriptions more directly than OneNet's span-based pipeline. The supervised results are not affected, but the zero-shot claims in the abstract and Section 7.3 require re-evaluation with entity descriptions generated only from pre-test (January–March 2024) sources, plus a report of how many test entities have descriptions that cite test-period articles.","section":"Appendix A.2, §5.1, Tables 3 and 5"},{"comment":"The Seen/Unseen entity breakdown is directly affected by the same leakage. An entity that is unseen in training (44.3% of validation and test entities, per Section 4.1) may still have a description generated from its test-period article, so ZEST's 20.0 F1 on unseen entities does not measure generalization to genuinely new entities. The paper's claim that zero-shot methods outperform supervised methods on unseen entities should be re-computed under a leakage-free protocol, and the discussion of the \"Unseen\" category should be revised accordingly.","section":"Table 4 and §7.3"}],"minor_comments":[{"comment":"The text states ZEST surpasses OneNet by 20.0 percentage points, but Table 3 shows 45.7 − 23.7 = 22.0; the arithmetic should be corrected.","section":"Section 7.3"},{"comment":"OneNet is called a \"state-of-the-art zero-shot baseline\" in the abstract but a \"few-shot entity linking model\" in §6.1; the paper should clarify the adaptation and terminology.","section":"Abstract vs §6.1"},{"comment":"The sentence \"with OneNet performing even better in this regard\" is ambiguous with respect to Table 4, since OneNet's Specific score (37.2) is not a decline from its Generic score (10.5); rephrase to state the intended comparison.","section":"Section 7.3"},{"comment":"The sentence \"each argument field ai,j in in domain Di,j\" contains a duplicated \"in\"; please fix the typo.","section":"Section 3"},{"comment":"The entity-description generation procedure should state explicitly whether any filtering was applied to remove test-period content; currently the description is silent on this, which is the source of the leakage concern.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a valuable contribution, and the supervised evaluation appears sound. The key uncertainty is whether ZEST remains competitive when entity descriptions are built without test-period information; I would like the editor to ensure the revision includes this re-evaluation before the zero-shot claims are promoted. The paper may still be acceptable after such a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the dataset is a genuine resource, but the zero-shot entity linking evaluation is not clean. The entity descriptions in the retrieval database were generated from the full collection period, including test articles (Appendix A.2), which leaks test evidence into ZEST's candidate retrieval and filtering. That undermines the headline AEL F1 of 45.7% and the 58.3% end-to-end number, and it makes the comparison against OneNet one-sided. The stress-test note holds up on reading the paper.\n\nWhat is actually new and good: the abstractive event extraction formulation is a sensible reformulation for real-world aggregation, and LEMONADE is the first event extraction dataset to cover 20 languages, including Burmese, Indonesian, Hebrew, Somali, and Nepali. The expert annotations from ACLED, with QA, are a real step up from crowdsourced labels. The construction process is transparent: temporal splits, location reannotation, schema standardization, and entity database construction are all documented. The supervised baselines look careful and the supervised results are credible. ZEST is a reasonable retrieval-based design, but its reported zero-shot performance cannot be taken at face value.\n\nThe soft spot is central, not minor: Appendix A.2 says entity descriptions are generated by GPT-4o using the news articles annotated to involve each entity. LEMONADE spans January 2024 to January 2025, and the test split covers April 2024 to January 2025. So a description for an entity that appears in a test article may well contain facts or wording from that exact article. ZEST's stage-one queries are explicitly designed to match entity descriptions, and stage-two filtering shows the candidate description alongside the test article. That is test evidence flowing into the retrieval database. The gap between ZEST and OneNet might partly reflect this one-sided advantage, not just better zero-shot linking. The paper discloses the construction, but it does not flag the leakage as a limitation.\n\nThe dataset itself is not compromised: the supervised numbers, the entity database, and the multilingual coverage stand. The fix is straightforward: regenerate entity descriptions using only training-period articles (or exclude test-article entities from the description source), then re-run the zero-shot evaluations. Without that, the zero-shot results in the abstract and tables are not reliable.\n\nWho this is for: anyone building multilingual event understanding systems, especially for conflict and protest monitoring. The paper deserves a serious referee, but the evaluation protocol needs a major revision before acceptance. I would not cite the zero-shot numbers as they stand, but I would cite the dataset and the AEE task definition.","headline":"Valuable dataset and task, but the zero-shot entity linking numbers are inflated by test-time leakage in the entity descriptions; supervised results and the resource itself still merit serious peer review.","tokens_in":44057,"tokens_out":2074,"would_cite":true,"duration_ms":24583,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 20-language dataset reframes event extraction as abstractive, structured document understanding, with zero-shot systems reaching 58.3% end-to-end F1 and a retrieval-based entity linker doubling the zero-shot baseline.","keywords":["abstractive event extraction","LEMONADE dataset","conflict events","multilingual","entity linking","ZEST","zero-shot"],"falsifier":"Rebuild the ZEST retrieval database using only the training split's article set to generate or prune entity descriptions, then recompute the AEL F1 and end-to-end F1 on the existing test split. If the gap between ZEST (45.7%) and OneNet (23.7%) shrinks substantially or vanishes, the zero-shot claim is inflated; if it persists, the retrieval-based approach is validated under information-safe conditions.","tokens_in":43068,"feed_emoji":"📰","tokens_out":10596,"duration_ms":76161,"temperature":0.7,"pith_summary":"The paper introduces LEMONADE, a dataset of 39,786 expert-annotated conflict events across 20 languages and 171 countries, reframing event extraction as an abstractive task. It defines abstractive event extraction (AEE), where systems read whole documents and output structured event records with entities normalized to a fixed database, numbers, categories, and booleans, rather than pinpointing text spans. It evaluates zero-shot and supervised models, contributes a zero-shot retrieval pipeline, ZEST, for linking entities without training, and reports that zero-shot systems reach 58.3% end-to-end F1 while supervised models reach 78.4%.","feed_headline":"Rethinking event extraction for multilingual conflict reporting","feed_subtitle":"Reformulates conflict-event extraction as structured document understanding in 20 languages; zero-shot systems hit 58.3% F1.","key_machinery":"The load-bearing object is the abstractive event annotation: a Python-class event signature with typed arguments — lists of entity IDs from a curated 10,707-entity database, boolean enum flags, numerical values, and location fields — rather than text spans. The benchmark's event types number 25 in the socio-political domain. The entity database, accompanied by one-paragraph descriptions, is the second mechanism: it defines the universe for abstractive entity linking (AEL) and allows ZEST's retrieve-filter-assign pipeline to work, generating content-based queries for vector retrieval, filtering candidates by evidence in the document, and finally assigning entities to role arguments.","core_discovery":"The central claim is that a large, real-world conflict event dataset can be built by transforming expert-annotated ACLED records into document-level abstractive annotations, and that abstractive event extraction is the right formulation for studying real-world phenomena because it captures information not stated as text spans — such as that targeted women were government officials, or that no fatalities occurred. The paper further claims that its zero-shot retrieval-based linker, ZEST, achieves 45.7% AEL F1, almost twice the 23.7% of the zero-shot span-based baseline OneNet, while supervised models reach 82.7%. The authors conclude that existing span-based models such as GoLLIE and OneNet transfer poorly to the abstractive setting, and that zero-shot multilingual LLMs can approach but not yet match supervised quality for event extraction.","pith_inferences":["Inference beyond the paper: the abstractive formulation decouples system output from surface form; if it becomes the standard for event data, it would obsolete the intermediate stages of trigger and mention detection that dominated MUC/ACE-style extraction, and datasets could be built directly from humanitarian codebooks.","Inference beyond the paper: because the entity descriptions are generated from the same corpus that contains test documents (Appendix A.2), a critical external test would be to regenerate the entity database using only training-period documents and rerun the reported zero-shot comparisons. If ZEST's advantage persists under that regime, the claim of retrieval-based zero-shot entity linking would b","Inference beyond the paper: LEMONADE's single-event-per-document format with 25 event types could be extended to multi-event and cross-document aggregation. The paper itself notes cross-document event coreference is left out of scope; combining AEE with the existing coreference methods mentioned in the Limitations section is a natural next test."],"forward_implications":["If the abstractive formulation is adopted more broadly, event extraction systems could directly serve analytical queries such as 'fatalities from violence against female government officials in 2024' without a separate pipeline of span detection, coreference resolution and entity linking. Adopting abstractive event extraction would mean re-tooling evaluation from span matching to exact-match on ty","The 20-language, 171-country coverage with region-specific tail entities would enable more truly global studies of civil conflict and protest — the paper's results support the feasibility of monitoring dozens of languages with LLMs and suggest that low-resource languages such as Burmese, Somali and Nepali are now tractable for event extraction, though still the hardest cases. Supervised models are","Zero-shot entity linking, as measured by ZEST, can beat span+link pipelines by a wide margin, so the dominant bottleneck is no longer entity resolution but unseen-entity linking and explicit span detection. On unseen entities, the best supervised model reaches only 30.4% AEL F1, showing a fundamental gap in generalizing to new participants.","Because the dataset is temporally split (train Jan–Mar 2024; dev/test Apr 2024–Jan 2025) and 44.3% of dev/test entities never appear in training, LEMONADE can measure temporal generalization, not just language transfer.","The dataset and code are released, so future work can use LEMONADE to train multilingual AEE models and to improve unseen-entity linking, a direction the paper explicitly highlights for future research."],"supporting_citations":[{"why":"Source of the underlying ACLED conflict event data that LEMONADE is constructed from.","marker":"Raleigh et al., 2010"},{"why":"Describes the ACLED codebook and organization whose expert annotation process is the origin of LEMONADE labels.","marker":"ACLED, 2023"},{"why":"DocEE establishes the document-level, single-event configuration that LEMONADE adopts.","marker":"Tong et al., 2022"},{"why":"Zeshel, which supplies the zero-shot entity-linking research design of aligning mentions with entity descriptions; LEMONADE's entity-description approach parallels it.","marker":"Logeswaran et al., 2019b"},{"why":"OneNet, the state-of-the-art few-shot LLM entity linker that serves as LEMONADE's zero-shot AEL baseline and is adapted with GPT-4o.","marker":"Liu et al., 2024c"},{"why":"Code4Struct, whose schema-to-code conversion LEMONADE adopts for representing event annotations as Python classes, and whose Code4Struct baseline is adapted into AC4S.","marker":"Wang et al., 2023"},{"why":"GoLLIE, the instruction-tuned extraction model used as a zero-shot ED/AEAE/AEL span baseline.","marker":"Sainz et al., 2024"},{"why":"mGTE, the multilingual embedding model used for entity retrieval in both ZEST and the adapted OneNet baseline.","marker":"Zhang et al., 2024b"},{"why":"XLM-R, the multilingual encoder whose RetroMAE extension XLM-RRM is fine-tuned for the event-detection subtask.","marker":"Conneau et al., 2020"}],"fun_headline_variants":["Zero-shot abstractive event extraction: 58.3% F1 in 20 languages","ZEST nearly doubles zero-shot entity linking F1","39K conflict events, 20 languages: new benchmark for event extraction","LLMs outdo span-based models on abstractive event tasks","Abstractive event extraction: zero-shot still 20% behind supervised"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that the zero-shot evaluation is information-safe: the entity descriptions in the retrieval database are generated from all 13 months of articles, including test-period articles, so any test article that mentions a database entity could have its own content fed back into that entity's description, making zero-shot linking look better than it would without that leakage.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot abstractive event extraction: 58.3% F1 in 20 languages","ZEST nearly doubles zero-shot entity linking F1","39K conflict events, 20 languages: new benchmark for event extraction","LLMs outdo span-based models on abstractive event tasks","Abstractive event extraction: zero-shot still 20% behind supervised"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001124,"raw_usage":{"total_tokens":4697,"prompt_tokens":986,"completion_tokens":3711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":3618}},"tokens_in":602,"tokens_out":3711,"duration_ms":27597,"temperature":1.0,"reasoning_tokens":3618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:53:10.006062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the ZEST retrieval database using only the training split's article set to generate or prune entity descriptions, then recompute the AEL F1 and end-to-end F1 on the existing test split. If the gap between ZEST (45.7%) and OneNet (23.7%) shrinks substantially or vanishes, the zero-shot claim is inflated; if it persists, the retrieval-based approach is validated under information-safe conditions.","supporting_citations":[],"review_version":1}