{"id":"31b79a54-9c2f-4a19-bc15-26c38c002826","arxiv_id":"2509.04182","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Jointly modeling entities and discourse relations improves coherence assessment accuracy over text-only and single-feature models on GCDC, CoheSentia, and TOEFL.","lead":"The paper tests whether combining entity references and discourse relations improves automated coherence assessment. Across three benchmarks, both a fusion Transformer and an LLM prompting method beat text-only baselines, with joint models performing best.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline claim of 'significantly enhances' is never backed by a statistical test; several key margins are within overlapping standard deviations, so the central empirical claim is not yet established.","rationale":"The reader's stated weakest assumption is the restriction to adjacent-sentence discourse relations. I agree that is a scope limitation, but it is not the most load-bearing issue: both the full model and the TextRel/TextEnty ablations share that restriction, so the internal comparison still supports the joint-modeling benefit within that scope. The more direct threat to the paper's strongest claim is the absence of any statistical significance testing. The abstract and Section 1 explicitly claim that integrating both feature types 'significantly enhances' performance, yet the tables only show means and standard deviations across folds. Several of the key margins are smaller than the reported standard deviations, so the claim of significance is not backed by evidence. This is an internal evidential gap, not a disagreement with the field's consensus. I therefore keep the reader's CONDITIONAL verdict: the paper is reasonable but should be accepted only if the significance claim is verified or weakened. I set agreement_with_reader to 'partial' because the reader mentions significance as a condition in the rationale, but their designated weakest assumption is the adjacent-relations scope.","tokens_in":21171,"tokens_out":7118,"duration_ms":74454,"concrete_test":"Re-run the same fold splits and compute a paired bootstrap over folds (or Wilcoxon signed-rank) for each dataset, comparing Method I vs. TextRel and TextEnty, and Method II vs. TextRel and TextEnty, using per-fold accuracy. Report 95% confidence intervals and p-values. If any main comparison (GCDC, CoheSentia, TOEFL) has a CI including 0 or p>0.05, the headline 'significantly enhances' must be weakened to 'directionally improves.' As a secondary check, include Liu and Strube (2025) as a baseline in Tables 1 and 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: integrating entities and discourse relations 'significantly enhances' coherence assessment (Abstract, Section 1). The only support is mean accuracy and standard deviation over cross-validation folds; the paper reports no paired significance test, no confidence intervals, and no p-values anywhere. Several decisive contrasts are small relative to the reported noise. On GCDC (Table 1, RoBERTa), Method I achieves 63.24 vs. TextRel 62.48 and TextEnty 61.83, with stds of 1.6-2.0. On TOEFL fine-tuned Prompt (Table 2), Method II is 78.39 vs. TextRel 77.63 and TextEnty 77.70, with stds around 1.5. These differences are within one standard deviation; without a paired test over the same folds, the word 'significantly' is unsupported. Since the abstract and conclusions rest on that word, the core claim is not yet established. A secondary concern: Liu and Strube (2025), the authors' own ACL 2025 discourse-relation coherence model, is cited but omitted from related work and baselines, so the 'strong baselines' claim is also incomplete.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Liu and Strube propose two methods for jointly modeling entity-based and discourse relation-based coherence: a fusion Transformer that operates on a flat sequence of sentences, entities, and discourse relations with position-aware attention and a visible matrix, and a prompting method that serializes the entity/relation graph into triples and feeds them to Llama in zero-shot or LoRA fine-tuned settings. The methods are evaluated on GCDC, CoheSentia, and TOEFL against TextOnly, TextEnty, and TextRel ablations and prior SOTA systems, with additional analyses including macro-F1, per-label accuracy, cross-domain transfer, and case studies. The central claim is that integrating both feature types significantly improves coherence assessment.","tokens_in":21440,"tokens_out":6513,"duration_ms":59374,"significance":"If supported, the paper would provide a useful demonstration that entity and discourse-relation cues are complementary for coherence modeling, and it offers two practical architectures for combining them. Strengths include the breadth of evaluation (three benchmarks, two encoders, zero-shot and fine-tuned LLMs), the inclusion of macro-F1 and cross-domain experiments, and the release of code. However, the headline improvements are modest (often 1-2 accuracy points) and are not backed by statistical significance tests, so the central empirical claim is currently only suggestive. The omission of the authors' own closely related ACL 2025 work from the baselines further weakens the claim of outperforming strong baselines.","major_comments":[{"comment":"The claim that joint modeling 'significantly enhances' performance is not supported by any statistical test. For example, Table 1 (GCDC, Fusion RoBERTa) shows Method I at 63.24 vs. TextRel 62.48 and TextEnty 61.83, with standard deviations of 1.6-2.0; Table 2 (TOEFL, fine-tuned Prompt) shows Method II at 78.39 vs. TextRel 77.63 and TextEnty 77.70, with stds around 1.5. These differences are within one standard deviation. Because the central claim rests on these comparisons, the authors should report paired significance tests (e.g., Wilcoxon signed-rank or bootstrap over the cross-validation folds) or confidence intervals for the key contrasts, and adjust the wording in the Abstract, Section 1, and Section 4.1 accordingly.","section":"Abstract; Section 4.1; Tables 1-2"},{"comment":"Liu and Strube (2025), a discourse relation-enhanced neural coherence model, is cited in the references and Appendix B.2 but is omitted from Related Work and is not included as a baseline. This work is directly relevant and appears to be the closest prior model; its omission makes the 'strong baselines' claim incomplete. The authors should discuss this work in Related Work and, if possible, compare against it or justify why a direct comparison is not feasible.","section":"Section 2 (Related Work); Section 4 (Baselines)"},{"comment":"The discourse parser extracts relations only between adjacent sentences ('we focus on identifying both explicit and implicit discourse relations between adjacent sentences'). This restricts the method to local discourse relations; long-range discourse relations (e.g., between sentence 1 and sentence 4) are not captured. While this limitation applies equally to all compared models and thus does not invalidate the internal comparisons, it limits the generalizability of the claim that discourse relations contribute to coherence assessment. The manuscript should acknowledge this restriction explicitly in the Limitations section and discuss its potential impact.","section":"Appendix A"}],"minor_comments":[{"comment":"Standard deviations are displayed as concatenated numbers (e.g., '67.600.5'), which is hard to read. Use an explicit plus/minus sign or parenthetical error bars.","section":"Tables 1, 2, 7, 8"},{"comment":"The Conclusions repeat the 'significantly outperform' claim without qualification. If significance tests cannot be provided, the claims should be softened to 'numerically improve' or 'consistently improve'.","section":"Section 6 (Conclusions)"},{"comment":"The grouping of CoheSentia scores (1-2 as low, 3-4 as medium, 5 as high) is a preprocessing choice that could affect results. It is not mentioned in the Limitations or as a potential free parameter. A brief note on its robustness would be helpful.","section":"Section 4 (CoheSentia preprocessing)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript omits the authors' own closely related ACL 2025 work (Liu and Strube 2025) from Related Work and baselines, which is a significant novelty and completeness concern. The lack of significance testing is the more fundamental issue: the headline claim of 'significant' improvement is not supported by the reported statistics. If the authors cannot establish significance after running paired tests, they should reframe the contribution accordingly. The code release and breadth of evaluation are positive aspects."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one if you work on coherence assessment. It's a pragmatic paper that does a lot of things right, and the central result - jointly modeling entities and discourse relations helps more than either alone - looks credible. The two methods are new: a fusion Transformer with 2D positions and a visible matrix, and a prompt-based approach that turns the entity/relation graph into triples for LLMs. The evaluation is genuinely thorough: three benchmarks, cross-validation, ablations, macro-F1, per-label analysis, case studies, cross-domain transfer, and even a 70B zero-shot consistency check. The features come from external tools, so there's no circularity. Code is linked. That's real evidence, and I take the empirical trend seriously. The soft spots are real but not fatal. First, the paper says 'significantly enhances' in the abstract and throughout, but there is no significance test anywhere, and several key margins (1-2%) sit inside one standard deviation. That's a legitimate referee ask: add paired tests over the folds, or soften 'significant' to 'consistently improves.' Second, the authors' own ACL 2025 discourse-relation coherence model is cited but not included in the related work or the baseline table. If your prior model already uses discourse relations, then a fair comparison to that model is exactly what we need to verify that the joint modeling is the active ingredient. Third, the adjacent-sentence restriction on discourse relations is stated in Appendix A but never discussed as a scope condition. For long texts, relations spanning several sentences might be exactly the kind of signal you want. It doesn't sink the paper, but it should be acknowledged as a limitation. The reader's conditional verdict seems about right. This is not a take-down case; the empirical work is honest and broad, and the conclusion is plausible. The missing significance tests are a fixable reporting issue, not a contradiction. The missing prior-work benchmark is an oversight, not a fraud. I'd send this to peer review because it deserves referee time, and a revision could make it a nice practical contribution for the coherence-scoring community.","headline":"A solid, well-scoped empirical study on combining entity and discourse features for coherence scoring, but the headline 'significantly' is not backed by significance tests and the authors' own related ACL 2025 model is missing from the comparison.","tokens_in":617,"tokens_out":720,"would_cite":true,"duration_ms":24771,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that coherence assessment improves when entity continuity and discourse relations are modeled jointly, and demonstrates it with two methods across three benchmarks.","keywords":["coherence assessment","discourse relations","entity grid","fusion transformer","large language models","prompt-based evaluation","PDTB","cross-domain generalization"],"falsifier":"Construct a test set of coherent texts whose key discourse relation spans non-adjacent sentences (e.g., sentence 1 states a claim and sentence 4 gives the reason, with no adjacent relation conveying it) and check whether adding the adjacent-only relation triples still improves over text-only baselines. If it does not, the adjacent-sentence restriction is the limiting factor.","tokens_in":21066,"feed_emoji":"🧩","tokens_out":5441,"duration_ms":48397,"temperature":0.7,"pith_summary":"Coherence—what makes a text hang together—is usually modeled with only one kind of linguistic clue: either how entities recur across sentences or how discourse relations (cause, contrast, instantiation, etc.) link them. This paper argues that the two cue types are complementary and that assessing coherence with both at once works better than either alone. It proposes two ways to do this: a fusion Transformer that reads a flat sequence of sentences, entities, and discourse relations with position-aware attention, and a prompt method that feeds LLMs the connections between sentences as natural-language triples. Across three benchmarks, the joint models outperform text-only and single-cue baselines, and the gains show up especially on low-coherence texts and in cross-domain transfer.","feed_headline":"Entities plus discourse relations improve coherence scoring","feed_subtitle":"Two joint models beat text-only and single-cue baselines on three benchmarks, with the largest gains on low-coherence texts.","key_machinery":"The carrying mechanism is a graph of sentences linked by two edge types—'entity' edges (shared nouns or coreference) and discourse-relation edges such as cause, contrast, and instantiation—which is then flattened for Transformer input. Each element gets a two-dimensional position encoding (start and end positions in the sentence sequence), and attention is modulated by a visible matrix so sentences can only attend to entities and relations that linguistically involve them. For the prompt method, the same graph is linearized into left-to-right triples (s_i, r, s_j) that preserve both order and connection information.","core_discovery":"The paper's central claim is that integrating entity continuity and discourse-relation structure improves automated coherence assessment. It supports this with two models: Method I builds a flat sequence containing sentences, their entities, and the discourse relations between sentences, then applies a fusion Transformer whose position-aware attention sees each element's start and end position in the text and whose visible matrix restricts attention to linguistically plausible connections. Method II rewrites the same graph as ordered triples—(s_i, entity, s_j), (s_i, reason, s_j), etc.—and prompts a large language model to judge coherence from the sentences plus the triples. On GCDC, CoheSen","pith_inferences":["The paper restricts discourse relations to adjacent sentences; a natural extension would be to include long-range relations (e.g., between sentence 1 and sentence 4), which could either strengthen the joint model or reveal where the parser is the bottleneck.","The triple-based prompt format is a generalizable trick: the same linearization could be tested for other structural judgments such as factual consistency, readability, or narrative flow, where linguistic relations matter.","The fusion Transformer's 2D-position-aware attention with a visible matrix may transfer to other graph-structured text tasks where order plus connection structure must both be respected."],"forward_implications":["If this holds, coherence scorers that combine entity tracking with discourse parsers will systematically beat single-cue systems on benchmark suites.","The label-bias analysis implies these joint features make coherence models safer for real-world, imbalanced text collections where most documents cluster in one or two score bands.","The zero-shot prompt gains suggest that explicit entity and relation triples can inject structural linguistic knowledge into LLM evaluation without any training.","Cross-domain transfer results imply that entity and discourse cues are domain-agnostic coherence signals, useful when training and test genres differ."],"supporting_citations":[{"why":"Defines the entity-grid model, the standard entity-based coherence framework the paper's entity features derive from.","marker":"Barzilay and Lapata, 2008"},{"why":"First uses PDTB discourse relations as features for coherence; the paper follows this PDTB-based relation-as-feature approach.","marker":"Lin et al., 2011"},{"why":"Supplies the state-of-the-art implicit discourse relation classifier that the paper's parser relies on.","marker":"Liu and Strube, 2023"},{"why":"Provides the discopy shallow discourse parser that the paper updates for relation extraction.","marker":"Knaebel, 2021"},{"why":"A strong entity-based neural coherence model used as a baseline and prior state of the art.","marker":"Jeon and Strube, 2022"},{"why":"Introduces GCDC, one of the three evaluation benchmarks for the coherence claims.","marker":"Lai and Tetreault, 2018"},{"why":"Provides CoheSentia, the second benchmark used to test joint modeling on generated text.","marker":"Maimon and Tsarfaty, 2023"},{"why":"A graph-convolutional coherence model used as another baseline; its graph framing motivates the paper's graph-based joint approach.","marker":"Liu et al., 2023b"}],"fun_headline_variants":["Fusing entity and discourse features boosts coherence scoring","Joint entity-relation modeling sharpens coherence assessment","Coherence models gain from combining entity and discourse cues","Two joint methods improve coherence, strongest on weak texts"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that discourse relations between adjacent sentences capture enough of the discourse structure needed for coherence; if coherence depends substantially on relations between non-adjacent sentences, the model's discourse features are incomplete.","fun_headline_variants_meta":{"raw":{"variants":["Fusing entity and discourse features boosts coherence scoring","Joint entity-relation modeling sharpens coherence assessment","Coherence models gain from combining entity and discourse cues","Two joint methods improve coherence, strongest on weak texts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1649,"prompt_tokens":599,"completion_tokens":1050,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":343,"completion_tokens_details":{"reasoning_tokens":1001}},"tokens_in":343,"tokens_out":1050,"duration_ms":7652,"temperature":1.0,"reasoning_tokens":1001,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:19:01.964625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set of coherent texts whose key discourse relation spans non-adjacent sentences (e.g., sentence 1 states a claim and sentence 4 gives the reason, with no adjacent relation conveying it) and check whether adding the adjacent-only relation triples still improves over text-only baselines. If it does not, the adjacent-sentence restriction is the limiting factor.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"First uses PDTB discourse relations as features for coherence; the paper follows this PDTB-based relation-as-feature approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the discopy shallow discourse parser that the paper updates for relation extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A strong entity-based neural coherence model used as a baseline and prior state of the art."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces GCDC, one of the three evaluation benchmarks for the coherence claims."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides CoheSentia, the second benchmark used to test joint modeling on generated text."}],"review_version":1}