{"id":"350ca207-d82d-4c35-b32a-528207d8257d","arxiv_id":"2508.19359","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"ARIS combines self-mixture-of-agents LLM decoding with a RoBERTa sequence tagger, consensus detection, confidence filtering, and LLM reflection to improve event extraction F1, but its headline SOTA claim is not supported on all datasets.","lead":"A new event extraction pipeline, ARIS, combines several copies of a fine-tuned LLM with a precision-focused sequence tagger, then uses agreement checks, confidence thresholds, and an LLM reflection step to merge their outputs. The authors report state-of-the-art scores on three event extraction benchmarks, but the paper's own tables show the system does not beat the tagger on trigger extraction for one dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central SOTA claim contradicted by Table 1: ARIS underperforms TagPrime on CASIE trigger metrics, so 'outperforms ... across three benchmarks' is false as stated.","rationale":"The reader's strongest_claim correctly identifies that the abstract's SOTA claim requires ARIS to beat prior methods on the three benchmarks. The reader's weakest_assumption focuses on validation-tuned confidence thresholds and the possibility that some thresholds exceed 1.0, which is a legitimate reproducibility and overfitting concern. However, the most decisive problem is simpler and prior: the paper's own Table 1 shows ARIS below TagPrime on CASIE trigger identification and classification, and also below on MLEE triggers for the Llama variant. This directly contradicts the abstract's across-three-datasets superiority claim without needing any assumption about threshold generalization. I therefore agree with the REJECT verdict, but for a different primary reason than the reader's weakest_assumption, hence 'partial'. The algorithmic inconsistency (Algorithm 1's single threshold vs. Section 3.4's three thresholds) and threshold overfitting are additional serious flaws, but the empirical contradiction is the most load-bearing. A focused re-evaluation of the CASIE/MLEE trigger numbers would settle whether the headline claim as written is false; the reported numbers already make that determination, so the paper needs substantial revision before it could be considered for acceptance.","tokens_in":17133,"tokens_out":3993,"duration_ms":39003,"concrete_test":"Run the exact-match F1 evaluation on the CASIE (and MLEE) test splits for ARIS and TagPrime using the same TextEE split1 protocol and released code, reproducing the rows of Table 1. Then compute a per-metric bootstrap or paired test with confidence intervals. If the reproduced CASIE trigger F1s for ARIS remain below TagPrime's, the abstract's 'outperforms existing SOTA across three benchmark datasets' is not supported and must be revised to e.g. 'competitive on triggers and stronger on arguments on CASIE' or the claim restricted to a defined aggregate metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that ARIS outperforms existing state-of-the-art event extraction methods across three benchmark datasets. The paper itself identifies TagPrime as the current SOTA (Sec. 4.3). Yet Table 1 shows ARIS-Llama on CASIE at Trg-I=70.78, Trg-C=70.27 and ARIS-Phi at Trg-I=69.08, Trg-C=68.39, both below TagPrime's Trg-I=72.00, Trg-C=71.60. ARIS-Llama also trails TagPrime on MLEE trigger metrics (Trg-I=73.80 vs 74.61; Trg-C=70.33 vs 72.38). Thus, under the natural reading of 'outperforms existing SOTA across three benchmark datasets,' the paper's own reported results refute the claim. Section 4.3 implicitly retreats to 'competitive with TagPrime for trigger detection' on CASIE, which is inconsistent with the abstract and conclusion. This is more load-bearing than threshold-tuning concerns because it concerns the final reported F1 numbers, not a potential confound. Even if every threshold were perfectly calibrated, the headline superiority claim would still fail on two of the three datasets for at least one model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ARIS, a hybrid event-extraction pipeline that combines a fine-tuned LLM ensemble (Self-MoA, ten agents) with a RoBERTa-based sequence tagger (TagPrime). Predictions are merged by overlap-based agreement detection, confidence filtering with dataset- and temperature-specific thresholds, and an LLM reflection module that verifies ambiguous triggers and arguments. The LLM is fine-tuned with a decomposed instruction curriculum (13 variants) covering trigger and argument subtasks. Experiments on CASIE, M2E2, and MLEE report micro-F1 for trigger/argument identification and classification, along with ablations against TagPrime and LLM baselines. The paper claims state-of-the-art results on all three benchmarks.","tokens_in":17424,"tokens_out":6073,"duration_ms":63915,"significance":"The decomposed instruction fine-tuning and the explicit reporting of per-dataset/per-temperature thresholds are useful contributions; the ablation separating Self-MoA, TagPrime, and their combination is informative. The central claim, however, is not supported by the paper's own Table 1: on CASIE both ARIS variants are below TagPrime on trigger metrics, and on MLEE the Llama-based ARIS also trails TagPrime on trigger metrics. The architecture may still offer value for argument extraction and for stabilizing performance across sampling temperatures, but the advertised 'outperforms existing state-of-the-art across three benchmark datasets' claim is false as stated. Because the headline contribution rests on this claim, the result cannot be accepted in its current form.","major_comments":[{"comment":"The central claim 'outperforms existing state-of-the-art event extraction methods across three benchmark datasets' is contradicted by Table 1. TagPrime is identified as the current SOTA in §4.3, yet on CASIE ARIS-Llama reports Trg-I=70.78, Trg-C=70.27 and ARIS-Phi reports Trg-I=69.08, Trg-C=68.39, both below TagPrime's 72.00/71.60. On MLEE, ARIS-Llama also underperforms TagPrime on trigger metrics (73.80/70.33 vs 74.61/72.38). Section 4.3 retreats to saying the method is 'competitive' with TagPrime for CASIE trigger detection, which is inconsistent with the abstract and conclusion. The authors must either qualify the claim to specific datasets/metrics or revise the headline; as written, the paper's own results refute it.","section":"Abstract / §4.3 / Table 1"},{"comment":"Algorithm 1 is inconsistent with the method description. Section 3.4 defines three thresholds, θS, θ+SMoA, and θ−SMoA, with three outcomes (retain, drop, reflect), whereas Algorithm 1 uses a single threshold τ and only two branches (high-confidence vs ambiguous), with combined_conf = (conftagger + confSMoA)/#models. The main text never defines combined_conf or τ. This makes the actual aggregation procedure irreproducible. Additionally, Tables 4/5 list θ+SMoA values greater than 1.0 (e.g., 1.10 for Phi-3 on CASIE and MLEE), outside the valid range for a proportion/softmax score; this means the high-confidence Self-MoA branch is never exercised in those configurations. The authors should reconcile the algorithm with the prose and explain or correct the out-of-range thresholds.","section":"Appendix E (Algorithm 1) vs §3.4–3.5"},{"comment":"DEBATE-EE and MMUTF are reported using F1 scores 'provided in their original papers' rather than re-evaluated under the same TextEE split1 preprocessing. Without a shared evaluation protocol, the claimed superiority over these two systems is not established. This matters because the SOTA claim depends on comparing against these baselines, not only TagPrime. The authors should either re-run those baselines on their exact splits and metrics or clearly mark the comparison as cross-paper and not directly comparable.","section":"§4.3 baseline comparisons"}],"minor_comments":[{"comment":"The tuple notation is inconsistent: (t, p) is defined as an ordered trigger representation, but the agreement lines use (t, p, _). The algorithm also handles triggers only, while §3.6 includes arguments; an argument-level algorithm or a reference to the argument aggregation in Appendix D would improve clarity.","section":"Appendix E, Algorithm 1"},{"comment":"Table 2 is described as presenting averaged results across all datasets, but no per-dataset break-down is given in the main text. Since the paper emphasizes dataset-specific behavior elsewhere, a per-dataset version or a note on variance would help the reader assess the temperature-stability claim.","section":"§4.4 / Table 2"},{"comment":"There are minor typographical issues (e.g., 'togeneratetrigger' in Appendix F.1, 'Event Decomposed Fine-Tuning' capitalization) and no statement of code or data availability. Providing a public repository would substantially aid reproducibility.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim is directly contradicted by its own Table 1 on two of the three datasets for at least one model. This is not a mere presentation issue: the advertised SOTA contribution fails. If the editors are willing to consider a substantially reframed submission that limits the contribution to argument extraction and stability, a future revision might be viable, but as submitted the manuscript does not support its headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, know this: the paper builds a genuinely interesting pipeline—Self-MoA plus a RoBERTa sequence tagger, with agreement detection, confidence thresholds, and an LLM reflection step—and the decomposed instruction-tuning curriculum (thirteen task variants) is a real contribution. But the abstract's claim that it 'outperforms existing state-of-the-art event extraction methods across three benchmark datasets' is not supported by the paper's own Table 1. On CASIE, both ARIS variants score below TagPrime on trigger identification and classification; on MLEE, the Llama-based ARIS trails TagPrime on trigger metrics. The authors implicitly acknowledge this in Section 4.3, where they say 'competitive with TagPrime' on CASIE triggers, which contradicts the abstract and conclusion.\n\nWhat's actually good: the ablation study shows the full ARIS beats both Self-MoA alone and ARIS without TagPrime on every metric, which demonstrates the hybrid design is doing real work. The instruction-tuning recipe is detailed enough to reproduce, and the appendix gives concrete prompt templates. The three-stage consensus/confidence/reflection idea is sensible and worth testing in other IE tasks.\n\nThe soft spots are real. Algorithm 1 defines a single combined confidence with one threshold τ, while Section 3.4 and Appendix B describe three separate thresholds (θS, θ+SMoA, θ−SMoA). That inconsistency prevents exact replication. Worse, many fitted thresholds exceed 1.0 (e.g., θ+SMoA=1.10 for CASIE and MLEE), meaning certain branches of the mechanism never fire in those configurations—so the claimed reflective behavior wasn't actually exercised there. Thresholds are tuned per dataset, temperature, and model on validation, and no error bars or significance tests are reported, so the argument-F1 gains over TagPrime might be noise. The paper is honest about computational cost in the limitations section, which I appreciate.\n\nWho should read it: people working on hybrid generative/discriminative systems for event extraction, and anyone building decomposed instruction sets for structured prediction. I'd bring it to a reading group, but with the Table 1 discrepancy on the table.\n\nRecommendation: deserves peer review, not desk rejection—the components and ablation are valuable. But the authors need to rewrite the abstract/conclusion to match the actual numbers, reconcile the algorithm pseudocode with the prose, and add variance estimates. I would not cite it in its current form.","headline":"Hybrid pipeline and decomposed instruction tuning are genuinely useful, but the abstract's SOTA claim is contradicted by the paper's own Table 1.","tokens_in":17962,"tokens_out":2933,"would_cite":false,"duration_ms":28852,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM ensemble plus tagger tops event extraction on three benchmarks","keywords":["event extraction","self mixture of agents","sequence tagging","reflective inference","instruction fine-tuning","confidence filtering","hybrid models","LLM"],"falsifier":"Run ARIS on the same test sets while varying each confidence threshold across the fitted range and compare F1; if some settings produce identical results to settings where the reflection branch is never reached, that would show reflection is not the operative component. A direct check is to disable the reflection module, keeping only agreement detection and confidence filtering; if F1 does not drop measurably on instances actually routed to reflection, the reflection claim is not supported.","tokens_in":16998,"feed_emoji":"🤝","tokens_out":4966,"duration_ms":51361,"temperature":0.7,"pith_summary":"This paper claims that event extraction improves if you stop asking a single model to do everything. ARIS runs several instances of the same fine-tuned LLM in parallel (Self-MoA), has a discriminative sequence tagger predict the same spans, and then routes every prediction by agreement: both models agree, keep it; one model is confident, keep it; both are shaky, discard or reflect. Disagreements that survive filtering are sent to an LLM reflection module trained with decomposed instructions that teach trigger detection, classification, argument extraction, and role assignment as separate skills. The paper reports state-of-the-art F1 on CASIE, M2E2, and MLEE, with the largest gains on argument extraction, and argues that structured consensus plus targeted reflection is a better reconciliation mechanism than unstructured multi-agent debate.","feed_headline":"LLM ensemble plus tagger tops event extraction on three benchmarks","feed_subtitle":"ARIS keeps agreed predictions, filters low-confidence clashes, and reflects on the rest to lift argument extraction.","key_machinery":"The Reflective Agreement mechanism: the consensus set is the intersection of Self-MoA and tagger predictions; the disagreement set is filtered by two dataset/temperature-specific confidence thresholds; the residual ambiguity set goes to a reflection module that prompts the LLM to verify triggers and arguments with binary classifications. The decomposed instruction-tuning curriculum (thirteen task variants) is the enabling training signal that gives the reflection module its event-structure competence.","core_discovery":"ARIS's central claim is that the disagreement set between a generative ensemble and a discriminative tagger is not homogeneous and should not be handled with one strategy. The paper partitions disagreements by confidence—tagging confidence from softmax scores, generative confidence from the fraction of agents making the same prediction—and shows that keeping high-confidence single-source predictions, dropping low-confidence ones, and reflecting only on the middle band yields higher F1 than either model alone or a debate-based alternative. A second claim is that decomposed instruction fine-tuning, a curriculum of thirteen tasks from trigger/non-trigger discrimination to full event-structure c","pith_inferences":["The routing logic—consensus, high-confidence keep, low-confidence drop, reflect on the middle—could transfer to other structured prediction tasks such as relation extraction or semantic role labeling, where a cheap discriminative anchor and an expensive generative model disagree; the paper does not test this.","Several fitted thresholds in Tables 4 and 5 exceed 1.0 (for example, CASIE and MLEE trigger θ+SMoA = 1.10), meaning the high-confidence Self-MoA keep branch is never exercised in those configurations; the reflection module's contribution should be interpreted only where thresholds actually route instances to it.","The reported gains carry a large inference-cost multiplier: ten parallel LLM decodes plus reflection calls. The paper acknowledges the overhead but does not quantify latency or cost, so practical deployment value depends on trade-offs left unmeasured.","A reader wanting to deploy this approach should first check whether validation-tuned confidence thresholds transfer to their own test distribution, since the entire disagreement-routing mechanism is threshold-driven."],"forward_implications":["If the claim holds, hybrid event extraction no longer needs unstructured debate loops; agreement plus selective reflection is a cheaper, more controllable reconciliation mechanism.","Argument extraction, the subtask with the lowest baseline F1, is where the hybrid pays off most—gains of roughly 3 to 12 F1 points over TagPrime across the three datasets.","ARIS's F1 varies by less than 2 points across sampling temperatures, so the consensus and confidence pipeline absorbs decode diversity.","Decomposed instruction fine-tuning outperforms end-to-end fine-tuning for the same LLM, suggesting curriculum-style event-extraction training is a reusable recipe beyond ARIS itself."],"supporting_citations":[{"why":"Supplies TagPrime, the RoBERTa-based sequence tagger that serves as the discriminative half of ARIS and the strongest baseline.","marker":"Hsu et al. (2023)"},{"why":"Provides the TextEE benchmark standardization and split1 partitions used for CASIE, M2E2, and MLEE evaluation.","marker":"Huang et al. (2024)"},{"why":"Defines Self-Mixture of Agents, the multi-decoding ensemble strategy ARIS adapts to event extraction.","marker":"Li et al. (2025)"},{"why":"Introduces Mixture of Agents, the ensemble concept Self-MoA builds on.","marker":"Wang et al. (2025)"},{"why":"Defines the CASIE cybersecurity event extraction dataset.","marker":"Satyapanich et al. (2020)"},{"why":"Defines the M2E2 multimedia event extraction dataset, used in text-only form.","marker":"Li et al. (2020)"},{"why":"Defines the MLEE biomedical event extraction dataset.","marker":"Pyysalo et al. (2012)"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method applied to both LLMs.","marker":"Hu et al. (2022)"},{"why":"DEBATE-EE, the debate-based hybrid baseline ARIS compares against.","marker":"Wang and Huang (2024)"}],"fun_headline_variants":["Hybrid LLM-tagger system beats state of the art in event extraction","Confidence-based filtering lifts event extraction beyond LLM or tagger alone","ARIS: Reflecting only on uncertain events improves extraction F1","Event extraction: Keep confident, drop low, reflect on middle","Agreement between agents and tagger refines event predictions"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The entire disagreement-routing mechanism assumes that the validation-tuned confidence thresholds—and the comparability of tagger softmax scores with agent-vote frequencies—carry over to the test distribution; if those thresholds are miscalibrated, the keep, drop, and reflect split silently changes and the claimed gains can vanish.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid LLM-tagger system beats state of the art in event extraction","Confidence-based filtering lifts event extraction beyond LLM or tagger alone","ARIS: Reflecting only on uncertain events improves extraction F1","Event extraction: Keep confident, drop low, reflect on middle","Agreement between agents and tagger refines event predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2646,"prompt_tokens":662,"completion_tokens":1984,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":1893}},"tokens_in":406,"tokens_out":1984,"duration_ms":13098,"temperature":1.0,"reasoning_tokens":1893,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:47:17.502562+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ARIS on the same test sets while varying each confidence threshold across the fitted range and compare F1; if some settings produce identical results to settings where the reflection branch is never reached, that would show reflection is not the operative component. A direct check is to disable the reflection module, keeping only agreement detection and confidence filtering; if F1 does not drop measurably on instances actually routed to reflection, the reflection claim is not supported.","supporting_citations":[],"review_version":1}