{"id":"346d27a0-8917-4af9-a4c5-4fa32c738975","arxiv_id":"2412.12733","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EventFull is an annotation tool that jointly, completely, and consistently annotates temporal, causal, and coreference relations over a given set of event mentions, with a small pilot showing high annotator agreement.","lead":"EventFull is a web-based tool for annotating temporal, causal, and coreference relations between event mentions in text, with automatic checks that all event pairs are covered and no relations conflict. It matters because building complete event-relation datasets is expensive, and this tool is designed to let non-expert annotators produce them faster and more consistently.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EventFull's 'complete' guarantee applies only to an externally supplied event mention set, and the pilot further restricts annotations to 16-18 salient events, so the unqualified completeness claim overstates what the tool ensures.","rationale":"The reader's weakest_assumption correctly identifies the scope of completeness as the load-bearing concern. Other possible concerns—such as the soundness of the transitive closure rules or the lack of a baseline comparison—are less central: the transitive rules follow MATRES start-time semantics, and Table 3 measures reduction against the all-pairs baseline. The external mention set, however, directly qualifies the paper's central promise of 'complete' annotation. The pilot's saliency filter makes this concrete: an average of 17 of 35 anchorable events are kept, so more than half of the possible mention pairs are excluded from the reported completeness and efficiency numbers. A user expecting text-level completeness would be misled. The tool itself is a genuine contribution with a coherent design and a released implementation, but the completeness guarantee is conditional on the input event set, and the pilot evaluates only a curated subset. The reader's CONDITIONAL verdict already captures this caveat, so no verdict change is needed.","tokens_in":11753,"tokens_out":14601,"duration_ms":144355,"concrete_test":"Re-run the pilot's annotation on the same six documents using the full set of anchorable event mentions (no 16–18 saliency filter), keeping all other instructions fixed, and record annotation time, number of steps, and inter-annotator agreement for temporal, coreference, and causal relations. If the full-set annotation remains feasible and yields IAA comparable to the reported values, the completeness claim is materially supported; if the time or disagreements rise substantially, the paper's efficiency and completeness results are specific to the curated subset and must be reported as such.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that event mention detection and selection are left orthogonal to EventFull, with an optional step allowing annotators to filter out any mentions. Consequently, the tool's completeness guarantee is entirely relative to the input mention set: if that set omits events, the output is complete only in the bookkeeping sense that every pair among the supplied mentions is classified. The central claim in §3 ('complete and consistent annotation ... between events mentioned in an input text') invites the reading that the annotation covers all events in the text, but the pilot (§4.1) explicitly instructs annotators to keep only the 16–18 most salient events out of an average of 35 anchorable mentions per document (itself a filtered subset of ~60 detected mentions). The reported 816 pairs therefore cover a curated subset, not the document's events. This matters because the paper's contribution is advertised as solving the 'complete annotation' problem for event relation datasets; users who apply EventFull without curating the mention set will get a complete annotation only over whatever mentions they provide. The concern is not that the tool's algorithms are unsound—the transitive closure and consistency checks are coherent—but that the headline claim overstates the scope of the guarantee.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EventFull, a web-based annotation tool for inter-annotating temporal, coreference, and causal relations between event mentions. The tool takes a text with a pre-marked set of targeted event mentions and leads annotators through three stages: temporal relation annotation with transitive closure, depth-first-style pair prioritization, and inconsistency checks; coreference annotation restricted to temporally equal mentions; and causal annotation over preceding event clusters using an EventStoryLine-style flow. The central claim is that this process guarantees complete and consistent annotation for the three relation types over the targeted event set while minimizing manual effort. A pilot study with three non-expert annotators, six news documents, and about 17 events per document reports inter-annotator agreement comparable to MAVEN-ERE (temporal kappa 0.72, coreference B3 0.96, causal kappa 0.78) and large reductions in the number of manually judged pairs. The paper includes appendices describing the transitive closure, prioritization, consistency checking, annotation screenshots, and customizable guidelines.","tokens_in":11987,"tokens_out":10175,"duration_ms":93988,"significance":"If the claims hold, EventFull addresses a real bottleneck in event-relation dataset creation: exhaustive pairwise annotation is quadratic, and prior datasets either restrict the pairs considered or lack a systematic protocol guaranteeing completeness. The tool's design is internally coherent and grounded in known constraints: temporal transitivity, coreference requiring temporal co-occurrence, and causation requiring temporal precedence. The paper ships the tool and code publicly, the algorithms contain no fitted parameters, and the transitivity rules are taken from prior work (Allen, Verhagen, Ning, Caselli and Vossen), so the central mechanism is not circular. The pilot's agreement numbers are encouraging and compare favorably with MAVEN-ERE. The main weaknesses are evaluation-related: the efficiency evidence is indirect, one reduction metric uses an inconsistent baseline, and the event-selection reconciliation is not specified. The completeness guarantee is also stated in places more strongly than the input-dependent design warrants.","major_comments":[{"comment":"The 41% reduction reported for causal annotation is computed against 136 event-mention pairs, but causal relations are annotated over coreference clusters, not over individual mentions (§3.2.3). The appropriate baseline is the number of cluster pairs (or a full causal protocol such as RED over the same units), so the reported reduction is not directly interpretable and may be overstated. In addition, the paper only reports reductions in judged pairs and absolute times, with no control condition (e.g., the same tool with auto-completion disabled, or a generic annotation tool); this does not by itself substantiate the 'accelerates and simplifies' claim in the abstract and §4. Please report the cluster-level baseline and either add a comparison condition or soften the efficiency claim.","section":"Section 4.2, Table 3"},{"comment":"The opening sentence of §3 says EventFull 'guarantees complete and consistent annotation ... between events mentioned in an input text,' and the abstract uses 'complete annotation' without qualification. However, §3.1 explicitly leaves event-mention detection and selection orthogonal to EventFull, and the pilot further restricts the set to the '16–18 most salient events' (§4.1). The guarantee is therefore complete only over the externally supplied targeted mention set, not over all events a reader might identify. The paper should consistently phrase the guarantee as 'complete over the given targeted event mention set' and explicitly state that the tool does not evaluate or ensure completeness of the mention set itself.","section":"Section 3, opening paragraph; §3.1; §4.1"},{"comment":"The paper does not explain how the three annotators' individual selections of 16–18 salient events were reconciled into the final set of 102 event mentions (17 per document). If annotators annotated different event sets, the pairwise agreement values in Table 1 are not computed over the same pair universe, which would complicate their interpretation. Please specify whether the event set was unified across annotators (e.g., by consensus, by one annotator's selection, or by intersection/union) and, ideally, report agreement on the event mention selection step.","section":"Section 4.1, annotation procedure"}],"minor_comments":[{"comment":"The caption cites 'Fleiss and Cohen, 1973' for the kappa coefficient, but the pairwise kappa values between two annotators are Cohen's kappa; please cite Cohen (1960) or clarify that the Fleiss and Cohen reference pertains to weighted kappa or the intraclass correlation.","section":"Table 1 caption"},{"comment":"The annotation label is called 'uncertain' in the main text but 'VAGUE' in Table 4 and Appendix B; please unify the terminology across the paper.","section":"§3.2.1, Table 4, Appendix B"},{"comment":"The algorithm's name is misspelled as 'Warsheall's algorithm'; it should be Warshall's algorithm, matching the cited Warshall (1962) reference.","section":"Appendix B"},{"comment":"The heading 'Printscreens of EventFull' should be 'Screenshots of EventFull'; also, the reference list contains several instances of 'V ossen' with an internal space that should be 'Vossen'.","section":"Appendix F heading"},{"comment":"The phrase 'around 0.2kappa' should read 'around 0.2 kappa' for readability.","section":"Appendix A"},{"comment":"The claim that EventFull is 'the first tool' supporting these features is hard to verify and may be read as too strong given prior tools such as NarrativeTime; consider writing 'the first tool we are aware of' or specifying the exact combination of features being claimed.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":"This is a tool/demo paper whose algorithmic core is sound and whose public release is a useful contribution. The main risks are in the evaluation and in the wording of the completeness guarantee; both are fixable. The small pilot is acceptable for a tool paper, but the causal reduction metric and the event-set reconciliation need to be corrected before publication. I do not see grounds for rejection if the authors address these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: EventFull is a real tool, not a toy. It is the first, as far as I can tell, to put temporal, coreference, and causal annotation in one workflow, and the engineering is sensible: temporal relations first, transitive closure auto-fills many pairs, consistency checks catch conflicts, and the coreference and causal steps leverage temporal equal/precedes to cut the pair space. The code is public, and the pilot shows decent IAA (temporal kappa 0.72, coref B3 0.96, causal kappa 0.78), comparable to MAVEN-ERE. Table 3's reduction in judged pairs (56% temporal, 96% coref, 41% causal) is a real consequence of the algorithm, not a rhetorical trick. Credit where due: this is a useful contribution to the annotation toolbox.\n\nThe soft spots are about scope and evidence, not about the algorithm. The 'complete' guarantee is explicitly relative to the event mention set supplied to the tool. The paper says this in §3.1, and it is true that for any given set of targeted events, every pair gets classified. But the abstract and intro drop the qualifier, and the pilot adds another layer: annotators were told to keep only 16–18 'most salient' events out of an average of 35 anchorable mentions per document. So the reported 816 pairs cover a curated subset of the events in the text. That is transparent in §4.1, but it means the headline claim reads stronger than what the tool actually ensures. If someone feeds EventFull an incomplete or biased mention list, they get a complete annotation only over that list.\n\nSecond, the efficiency claim—'accelerates and simplifies'—has no control condition. The reduction in steps is measured against the total 136 pairs, not against annotators doing the same task without the transitive closure support. Three annotators and six documents is a pilot, and that's fine for a tool paper, but the time numbers (44 min temporal, 69 total) need a baseline to mean anything. The pilot data and analysis scripts are not released, which is a bit unfortunate given the code is.\n\nBottom line: this deserves a serious referee. It is a well-scoped tool paper with a clear contribution, and the main fixes are phrasing (qualify 'complete' from the abstract onward) and evidence (a controlled comparison, released pilot data). I'd bring it to a reading group and would cite it if we do event-relation annotation work. Send it to review with the expectation of moderate revision.","headline":"A genuinely useful annotation tool whose 'completeness' guarantee is scoped to the supplied mention set—the paper should say so more loudly.","tokens_in":12473,"tokens_out":3284,"would_cite":true,"duration_ms":29652,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EventFull is a web-based annotation tool that guarantees complete and consistent temporal, causal, and coreference labels for every pair of targeted event mentions, while cutting manual judgments by 41–96 percent.","keywords":["event relation annotation","temporal relations","coreference","causal relations","annotation tool","transitive closure","inter-annotator agreement","NLP dataset creation"],"falsifier":"Annotate a three-event text with A before B and B before C, then export the result: if the pair A-C does not already carry the label before, or if a conflicting label can be saved without a warning, the completeness and consistency guarantee fails.","tokens_in":11571,"feed_emoji":"🏷️","tokens_out":6069,"duration_ms":50226,"temperature":0.7,"pith_summary":"Event relation datasets usually omit many pairs because annotating every pair of events costs quadratic time and effort. The paper introduces EventFull, a web-based annotation tool that claims to guarantee complete and consistent annotation of temporal, causal, and coreference relations over any given set of targeted event mentions. EventFull orders the work so each relation type narrows the next: temporal labels are assigned first, coreference is then judged only for temporally co-occurring mentions, and causality is judged only for events that lie before the target. Automated transitive closure fills in deducible temporal links and consistency checks catch contradictions, so fewer pairs need manual judgment. A pilot with three non-expert annotators on six news documents reports high inter-annotator agreement and a 41–96 percent reduction in the number of pairs that annotators must actively judge.","feed_headline":"Tool guarantees a complete label for every event pair","feed_subtitle":"Temporal-first workflow and transitive closure cut manual pair judgments by 41-96 percent.","key_machinery":"The load-bearing mechanism is the layered temporal-first workflow combined with an automated transitive closure and consistency checking over a four-label temporal scheme (before, after, equal, uncertain). Because temporal order constrains the other relations, completing the temporal graph first lets the tool restrict coreference judgements to temporally equal mentions and causal judgements to preceding events, and the transitive closure (computed with Warshall's algorithm over Allen-style interval constraints) auto-annotates pairs whose relation is logically forced from already-annotated paths. This is what converts the quadratic pair count into a smaller set of genuinely informative judgments.","core_discovery":"On its own terms, the paper's central claim is that a dataset creator who supplies any set of targeted event mentions can use EventFull to obtain an annotation in which every mention pair is classified for coreference, every pair of coreference clusters is classified for temporal and causal relations, and no transitive contradiction survives. The tool achieves this by annotating temporal relations first over all mention pairs, applying a Warshall-style transitive closure with Allen-style interval constraints so that any pair whose relation is already implied is auto-labeled, and flagging any direct annotation that conflicts with an inferred path. The subsequent coreference step only considers mentions whose temporal relation is equal, which follows from the earlier step, and the causal step only considers events that temporally precede the target, following the EventStoryLine flow. The pilot reports agreement comparable to MAVEN-ERE, with an average total annotation time of roughly 69 minutes per 500-word document.","pith_inferences":["The completeness guarantee is relative to the input event mention set; EventFull does not solve event detection, so if a mention set misses or mis-selects events the annotation is complete only within that set.","A temporal labeling error early in the workflow could cascade: because coreference candidates are pruned to equal-time mentions and causal candidates to preceding ones, a wrong 'after' label could hide a coreference or cause link that a human would see.","The pilot's 16-18 salient events per document leave the tool's efficiency claims untested at the 35 anchorable-event scale; at roughly 595 pairs per document, the reduction factor might shrink.","The same prioritized transitive-closure interface could be repurposed for other transitive relation types, such as containment or part-of hierarchies, in annotation tools outside temporal reasoning."],"forward_implications":["Dataset builders can produce relation annotations that cover all pairs of their selected mentions, without restricting to adjacent sentences or heuristic subsets.","Models trained or evaluated on EventFull-produced datasets can be tested on long-range and cross-sentence relations that current datasets omit.","Because the same workflow handles temporal, coreference, and causal relations together, the output supports joint training of multi-relation models without alignment mismatches.","The relatively simple scheme and built-in guidelines make non-expert annotators usable, so new datasets in other genres or languages are easier to create."],"supporting_citations":[{"why":"Supplies the interval-algebra temporal constraints that the transitive closure and consistency checking rely on.","marker":"Allen 1984"},{"why":"Provides the base transitive-closure algorithm the tool adapts for temporal relations.","marker":"Warshall 1962"},{"why":"Supplies the joint-reasoning temporal constraint logic that EventFull's consistency checker mirrors.","marker":"Ning et al. 2018a"},{"why":"Defines the MATRES temporal relation scheme (before/after/equal/vague) and the event-selection methodology used in the pilot.","marker":"Ning et al. 2018b"},{"why":"Supplies the event-detection method that generates the initial set of event mentions in the pilot study.","marker":"Cattan et al. 2021"},{"why":"Provides the event-coreference definition and the temporal co-occurrence constraint that restricts coreference candidates to equal-time mentions.","marker":"Cybulska and Vossen 2014"},{"why":"Provides the EventStoryLine causal-annotation flow (focal event plus preceding events) that EventFull adopts.","marker":"Caselli and Vossen 2017"},{"why":"Motivates annotation completeness and is the prior work that covers only temporal relations, with an expert-only scheme.","marker":"Rogers et al. 2024"}],"fun_headline_variants":["One tool, all event relations, zero contradictions","EventFull: complete annotation in half the time","EventFull cuts manual event-pair checks by up to 96%","No more missing event relations: EventFull does it all","Temporal-first closure yields complete event annotation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that completeness is defined with respect to the event-mention list supplied to the tool, not with respect to all events a reader would identify; the pilot compounds this by keeping only 16-18 salient events per document.","fun_headline_variants_meta":{"raw":{"variants":["One tool, all event relations, zero contradictions","EventFull: complete annotation in half the time","EventFull cuts manual event-pair checks by up to 96%","No more missing event relations: EventFull does it all","Temporal-first closure yields complete event annotation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000896,"raw_usage":{"total_tokens":3794,"prompt_tokens":814,"completion_tokens":2980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":2903}},"tokens_in":430,"tokens_out":2980,"duration_ms":19342,"temperature":1.0,"reasoning_tokens":2903,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:46:58.548861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a three-event text with A before B and B before C, then export the result: if the pair A-C does not already carry the label before, or if a conflicting label can be saved without a warning, the completeness and consistency guarantee fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the event-coreference definition and the temporal co-occurrence constraint that restricts coreference candidates to equal-time mentions."}],"review_version":1}