{"id":"e9821ea8-2382-4cc1-8e76-7ba7af570727","arxiv_id":"2501.14112","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An event-based planning pipeline with content selection improves faithfulness and coherence in legal case summarization across four datasets.","lead":"This paper introduces CoPERLex, a system that first picks the most important sentences from a legal judgment, then creates an outline of key events, and finally writes a summary from that outline. The authors report that this step-by-step event-based approach beats several standard summarization systems on four legal datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The event-vs-entity advantage in the abstract rests on an oracle-plan comparison (Table 4); under the actual predicted-plan pipeline no event-vs-entity comparison exists, so the headline planning-representation claim is not yet demonstrated.","rationale":"The reader's conditional verdict is appropriate. I agree with the limitations concern about automatic metrics, but the sharper problem is that the central event-vs-entity comparison never touches the actual inference-time representation. Table 4 uses gold-derived plans and full source inputs; the real pipeline uses noisy Memsum-selected content plus Unlimiformer-predicted plans. The gap between oracle and predicted plan quality is large enough to change conclusions: Table 6 plan-generation ROUGE is low, yet Table 4 oracle plans yield R-1 59.65 versus 50.24 in the final system. A test with predicted entity versus predicted event plans would settle whether the representation choice matters in practice. Until then, the paper supports a modular-pipeline benefit under automatic metrics but not the specific event-over-entity superiority claimed in the abstract. The proposed check is feasible with the paper's own components and would directly decide the claim. I therefore keep the reader's conditional verdict rather than accepting or rejecting.","tokens_in":19321,"tokens_out":4279,"duration_ms":41891,"concrete_test":"Run the full CoPERLex pipeline on MLS-Long and CanLII with two plan generators trained identically: one producing Plumber SVO triples and one producing the entity-chain representation used in Table 4. Keep content selection and summary generation fixed, and compare final summaries on AlignScore, UniEval coherence, and ROUGE with the same significance test. If event-plan final summaries do not significantly outperform entity-plan final summaries, the abstract's event-centric-advantage claim should be restricted to oracle plans or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: (i) content selection plus planning improves faithfulness and coherence, and (ii) event-centric plans outperform entity-centric plans. Part (i) is supported by Tables 2 and 3 under automatic metrics. Part (ii), however, is supported only by Table 4, which feeds oracle plans extracted from the gold summary to the summarizer, on MLS-Long only. The deployed system instead uses plans predicted by Unlimiformer, and Table 6 shows those predicted plans are far from gold (R-1 34.96, R-2 15.65, R-L 21.44). There is no end-to-end comparison of predicted event plans versus predicted entity plans; the ablations in Table 3 remove whole components rather than swapping plan representation. Because oracle plans contain surface tokens from the reference summary, all oracle-conditioned results are inflated, and the event advantage could shrink, disappear, or reverse once plans are predicted. Since the paper's abstract and conclusion explicitly assert the advantage of event-centric representations, this missing experiment is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoPERLex, a three-stage framework for legal case summarization: MemSum performs extractive content selection; an Unlimiformer model generates an event-centric plan, represented as Subject-Verb-Object tuples extracted from gold summaries with Plumber; and a Longformer Encoder-Decoder generates the final summary conditioned on the selected content and the predicted plan. The system is evaluated on four legal summarization datasets (MultiLexSum-Long/Short, CanLII, and SuperSCOTUS) against long-document abstractive baselines, with ablations, a plan-representation comparison, a content-selection comparison, and case studies. The central claim is that integrating content selection and planning improves faithfulness and coherence, and that event-centric plans outperform entity-centric plans in legal summarization.","tokens_in":19525,"tokens_out":7576,"duration_ms":67364,"significance":"If validated, the paper would provide a concrete demonstration that modular planning helps long legal summarization, and that event-based representations are better suited to the narrative structure of legal texts than entity chains. The strengths of the submission include evaluation on four datasets, statistical significance tests in Table 2, an oracle-plan representation comparison in Table 4, ablations for each pipeline component, and a case study illustrating plan controllability. The main weakness is that the headline event-versus-entity claim is supported only under oracle-plan conditions, not under the actual predicted-plan pipeline, so the central advantage claimed in the abstract is not yet demonstrated.","major_comments":[{"comment":"The abstract and conclusion claim that event-centric plans outperform entity-centric representations, but the only evidence for this is Table 4, which feeds oracle plans extracted from the reference summary to the summarizer. In the deployed pipeline, plans are predicted by Unlimiformer, whose agreement with gold plans is low (Table 6: R-1 34.96, R-2 15.65, R-L 21.44), and the summarizer receives MemSum-selected content rather than the full source document. Because oracle plans leak surface content from the reference summary, the event advantage may shrink, disappear, or reverse under plan-prediction error. An end-to-end comparison using predicted event plans versus predicted entity plans is needed to support the paper's central claim.","section":"§4.1, Table 4"},{"comment":"The ablation study removes whole components (e.g., 'w/o Planning') but never holds the pipeline fixed while swapping the plan representation. Consequently, the ablations show that having some plan helps, but not that event-centric plans specifically are responsible for the gains, especially for coherence and faithfulness. Additionally, Table 3 reports no significance tests, and several differences are numerically small (e.g., CanLII ROUGE-1: 49.19 vs 49.06 for 'w/o Content Sel.'). I request an inference-time comparison with predicted entity and event plans, and significance testing for the ablations.","section":"Table 3 and §4.1"},{"comment":"The global claim of 'significant improvements in faithfulness and coherence' is stronger than the evidence. On MLS-Short, ROUGE-2 (23.04) and coherence (69.27) are not marked as statistically significant; on CanLII, ROUGE-1, ROUGE-2, BERTScore, and fluency are not significant. The paper should report which metrics are significant on which datasets and qualify the 'consistently outperforms' phrasing in Section 4.1. The Limitations section already concedes that AlignScore and UniEval 'are limited in their ability to fully capture the unique complexities of legal texts' and that no legal experts were available; this caveat should be reflected in the abstract and conclusion, which currently state improvements in faithfulness and coherence without qualification.","section":"Table 2 and Limitations"},{"comment":"The text states that Unlimiformer 'consistently outperforms' LED for plan generation, but Table 6 reports a lower ROUGE-1 for Unlimiformer (34.96 vs 35.18 for LED). This is a factual contradiction in a result used to justify the choice of Unlimiformer as the plan generator. The claim should be corrected and the comparison verified; if the corrected comparison changes the architecture choice, downstream results may be affected.","section":"§4.1, Table 6"}],"minor_comments":[{"comment":"The term 'UCREAT' appears in Table 4 without being defined in the main text; please introduce the acronym or describe the event-extraction method explicitly.","section":"Table 4"},{"comment":"The caption 'Sum. Gen. with OP' is ambiguous: it should state explicitly whether the summarizer receives the full source document or the selected content together with the oracle plan, since this differs from the deployed setting.","section":"Table 6"},{"comment":"The caption abbreviates content selection as 'CS' while the text refers to 'Con. Sel.'; please align the notation and define all abbreviations in the caption.","section":"Table 3"},{"comment":"There are several typos, e.g., 'sentenece' in Section 3.1, 'dimenssion' in Appendix B, and 'generaiton' in the ablation-study description in Section 4.1.","section":"Throughout"},{"comment":"The sentence saying SLED 'outperforms' the long-range models on CanLII is imprecise because the differences are small and vary by metric; please reword to say it is competitive or specify the exact metrics.","section":"§4.1, Results"}],"recommendation":"major_revision","confidential_remarks":"For the editor: I would require the end-to-end event-versus-entity comparison before acceptance; the oracle-only Table 4 is the main gap. The paper otherwise reports a reasonable amount of experimental detail, and I did not find circularity in the way gold plans are obtained. The self-citations in related work do not affect the central result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoPERLex is a straightforward three-stage pipeline—MemSum for content selection, Unlimiformer to generate an SVO event plan, LED to write the summary—applied to four legal summarization datasets. The combination is new for legal text, and the paper does a fair amount of work to justify each piece. The ablations are the strongest part: removing content selection or planning by itself hurts on MLS-Long and CanLII, and dropping hybrid training consistently degrades results. On most metrics across the four datasets the full system beats the long-document baselines, and several differences are marked significant. Those are real, if incremental, results.\n\nThe soft spot is exactly what the stress-test flags. The paper's central selling point, that event-centric plans outperform entity-centric ones, comes only from Table 4, where the summarizer is fed oracle plans extracted from the reference summary. That tells you what the model could do if it had perfect plans, but the deployed system uses plans generated by Unlimiformer, and Table 6 shows those predicted plans are far from gold (ROUGE-1 around 35). There is no end-to-end comparison of predicted event plans against predicted entity plans. It is entirely possible the event advantage shrinks or vanishes once the plans are noisy. The abstract and conclusion state the event advantage as a finding, which overstates what the experiments actually show. This is a missing experiment rather than a flawed one; the pipeline-level claim holds up.\n\nA couple of smaller issues. Table 2 shows some non-significant differences (e.g., R-2 on MLS-Short, several CanLII metrics), so the \"significant improvements in faithfulness and coherence\" in the abstract is too broad. No code or artifacts are released, which makes the numbers hard to verify. To the paper's credit, the limitations section is honest about the automatic metrics and the lack of legal expert evaluation.\n\nOverall, this is a competent domain-application paper, not a paradigm shift. The referee should push for the predicted-plan event-vs-entity comparison and a softer claim if it isn't run. I would send it to peer review, and I'd probably bring it to a reading group on planned summarization, but I wouldn't cite the event-representation result until that missing experiment exists.","headline":"A solid modular pipeline for legal summarization whose headline event-vs-entity claim is only tested with oracle plans, not the deployed one.","tokens_in":20057,"tokens_out":2651,"would_cite":false,"duration_ms":23700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that legal case summarization improves when content selection and an event-based plan are explicit pipeline stages rather than implicit steps.","keywords":["legal case summarization","content planning","event-centric representation","subject-verb-object triples","extract-then-abstract","faithfulness","coherence","long-document summarization"],"falsifier":"Shuffle the order of the subject-verb-object triples in a fixed plan and regenerate summaries on a held-out set; if faithfulness and coherence scores do not worsen, then the structured ordering of events is not what produces the reported gains.","tokens_in":19113,"feed_emoji":"⚖️","tokens_out":9752,"duration_ms":79773,"temperature":0.7,"pith_summary":"CoPERLex claims that summarizing legal judgments should be decomposed into three explicit stages rather than left to a single end-to-end model: first pick the sentences that matter, then write an ordered plan of what happened in the form of subject-verb-object triples, and only then generate the summary from the selected content plus the plan. The authors argue that legal texts are narratives, so an event-centric plan supplies the structure that end-to-end models lack. On four legal summarization datasets, they report that this pipeline improves faithfulness and coherence over strong long-document baselines, and that event-based plans outperform entity-based plans. If the claim holds, legal summarization becomes more reliable and more controllable, because a user can inspect and edit the plan before the summary is written.","feed_headline":"Event plans beat entity plans for legal summaries","feed_subtitle":"A three-stage pipeline — extract, plan, then write — boosts faithfulness and coherence across four legal datasets.","key_machinery":"The load-bearing object is the content plan: an ordered list of events, each written as a subject-verb-object triple, extracted from reference summaries and generated from the selected salient sentences. This plan is what connects the content-selection stage to the summary-generation stage; it tells the generator what happened, in what order, so the final text follows the case's narrative rather than drifting or inventing details. The pipeline also uses a hybrid training scheme that mixes oracle-selected content and model-selected content to reduce exposure bias.","core_discovery":"On the paper's own terms, the discovery is that explicit content selection and explicit content planning are not optional scaffolding but the main sources of quality in legal summarization, and that the plan should be event-centric. The authors build a three-stage system whose intermediate representation is an ordered sequence of subject-verb-object triples: a reward-driven extractor chooses salient sentences, a long-input encoder-decoder generates the triple plan from them, and a second encoder-decoder writes the final summary conditioned on both. They report consistent gains over five long-document summarization baselines on four datasets, with the largest gains in faithfulness and coherence, and their ablations show that removing either the selection stage or the planning stage degrades performance. In a direct comparison of plan representations, the event-triple plans beat entity-chain plans, which the authors attribute to the narrative, action-driven nature of legal cases.","pith_inferences":["Not tested here: the event-plan bottleneck could act as a cross-lingual bridge, allowing a summary to be generated in a different language from the source by translating or relexicalizing the plan.","The framework's gains may come partly from extractive pre-filtering and partly from the event structure; an experiment that fixes content selection and varies only the plan type would separate these contributions more cleanly than the current ablations.","The same planning idea could transfer to other narrative-dense document genres, such as medical records or financial rulings, if their summaries also follow event sequences.","Because the plan is inspectable, a practical error-correction loop becomes possible: automatically spot an unfaithful triple, replace it, and regenerate the summary without retraining the generator."],"forward_implications":["Summarization systems for long legal documents can be built as transparent pipelines, with content selection acting as a filter that keeps the generator from being overwhelmed by irrelevant material.","Event plans give users a handle on output: editing or deleting a subject-verb-object triple changes the final summary in a predictable way, enabling customized and more concise summaries.","Because the plan representation is a generic triple rather than a jurisdiction-specific taxonomy, the same machinery should transfer to new legal systems and languages without re-engineering the plan format.","Faithfulness checking can be automated at the plan level by verifying that each event in the summary matches an event in the source, rather than comparing full texts.","The approach is most valuable where summaries must condense long, multi-source cases; on shorter-input datasets the gap over end-to-end models is smaller, so the planning overhead earns its keep mainly in the long-document regime."],"supporting_citations":[{"why":"Supplies the reinforcement-learning extractive summarizer that performs the content-selection stage.","marker":"Gu et al., 2022"},{"why":"Provides the long-input encoder-decoder architecture used to generate the event plan from selected content.","marker":"Bertsch et al., 2023"},{"why":"Provides the long-document encoder-decoder used to realize the final summary and also serves as a baseline.","marker":"Beltagy et al., 2020"},{"why":"Extracts subject-verb-object triples from reference summaries to create the target plans used for training.","marker":"Jaradeh et al., 2021"},{"why":"Defines the entity-chain planning representation that is the main competing plan type in the comparison.","marker":"Narayan et al., 2021"},{"why":"Supplies the dependency-based event extraction approach adapted for event-centric legal planning.","marker":"Joshi et al., 2023"},{"why":"Provides the alignment-based AlignScore metric used to measure faithfulness of generated summaries.","marker":"Zha et al., 2023"},{"why":"Provides the UniEval scores used to measure coherence and fluency.","marker":"Zhong et al., 2022"},{"why":"Contributes the MultiLexSumm long and short datasets on which the main experiments run.","marker":"Shen et al., 2022"},{"why":"Contributes the SuperSCOTUS dataset used as a fourth evaluation setting.","marker":"Fang et al., 2023"}],"fun_headline_variants":["Event-based plans beat entity plans in legal summarization","Plan legal summaries with subject-verb-object triples","Three-stage pipeline: extract, plan, write for better legal summaries","Why event triples beat entity chains for legal case summaries","Legal AI: content planning with events beats entities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic metrics used, lexical overlap, alignment-based faithfulness, and learned coherence and fluency scores, genuinely capture the quality that matters in legal summaries, since no legal experts judged the outputs.","fun_headline_variants_meta":{"raw":{"variants":["Event-based plans beat entity plans in legal summarization","Plan legal summaries with subject-verb-object triples","Three-stage pipeline: extract, plan, write for better legal summaries","Why event triples beat entity chains for legal case summaries","Legal AI: content planning with events beats entities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000593,"raw_usage":{"total_tokens":2728,"prompt_tokens":843,"completion_tokens":1885,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":1806}},"tokens_in":459,"tokens_out":1885,"duration_ms":12020,"temperature":1.0,"reasoning_tokens":1806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:21:25.496478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the order of the subject-verb-object triples in a fixed plan and regenerate summaries on a held-out set; if faithfulness and coherence scores do not worsen, then the structured ordering of events is not what produces the reported gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Extracts subject-verb-object triples from reference summaries to create the target plans used for training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the entity-chain planning representation that is the main competing plan type in the comparison."}],"review_version":1}