Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reasoning-optimized LLMs like OpenAI-o1, DeepSeek-R1, and QwQ-32B do not improve dialogue summarization and, across four benchmarks, consistently produce more verbose and less faithful summaries than their non-reasoning counterparts.

desk verdict A useful empirical sweep showing reasoning LLMs underperform on dialogue summarization, but the causal story about reasoning instruction tuning outruns the identification. read the letter →

arxiv 2507.02145 v1 pith:YMYEFO3T submitted 2025-07-02 cs.CL cs.AI

Keyan Jin , Yapeng Wang , Leonel Santos , Tao Fang , Xu Yang , Sio Kei Im , Hugo Gonçalo Oliveira This is my paper · ORCID
classification cs.CLcs.AI MSC 68T50
keywords dialoguesummarizationreasoningLLMschain-of-thoughtLLMevaluationLLM-as-a-judgeSAMSumQMSumfactualconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether "reasoning" large language models, which explicitly emit step-by-step chains of thought before answering, produce better dialogue summaries than their non-reasoning counterparts. It compares three reasoning models (OpenAI-o1, DeepSeek-R1, QwQ-32B) against three non-reasoning models (GPT-4o, DeepSeek-V3, Qwen2.5-32B) across four benchmarks covering generic, role-oriented, and query-oriented summarization, in English and Chinese, from short chats to long meetings. The result is consistently negative: non-reasoning models win on nearly all automatic metrics and in LLM-judge rankings. The paper argues that explicit stepwise reasoning leads to verbosity, factual inconsistencies, and less concise summaries in dialogue summarization, a task that rewards direct abstraction over extended deliberation. If this finding holds, it would temper expectations that reasoning-scaled LLMs generalize their gains from math and logic to summarization-heavy applications.

What carries the argument

The engine of the argument is a pairwise comparison design: each reasoning model is matched with a non-reasoning counterpart (OpenAI-o1 with GPT-4o, DeepSeek-R1 with DeepSeek-V3, QwQ-32B with Qwen-2.5-32B), and the two members of each pair receive identical prompt templates under a zero-shot or few-shot protocol. Outputs are then scored through two parallel channels: a battery of standard automatic metrics (ROUGE, BLEU, CHRF, BERTScore, MoverScore, BARTScore, COMET) and an LLM-as-a-judge protocol in which three independent evaluator LLMs rank summaries on relevance, consistency, fluency, coherence, and overall quality, with the rankings converted to Bradley–Terry scores. The paper additionally analyzes the explicit reasoning traces of DeepSeek-R1 on a sample of 100 dialogue summaries, scoring them on relevance, validity, coherence, utility, and depth, which lets it connect the quality of the reasoning process to the quality of the final summary.

What would settle it

A controlled experiment that isolates the reasoning variable would settle the claim: take one base model, produce two fine-tuned variants that differ only in whether they are trained with an explicit chain-of-thought objective, and compare them on SAMSum and QMSum using the paper's metrics. If the reasoning-tuned variant matches or exceeds the base on ROUGE and LLM-judge rankings, the paper's central claim that reasoning tuning fails for dialogue summarization is falsified. A weaker falsifier would be finding any dialogue benchmark where reasoning models consistently beat their non-reasoning counterparts under the paper's own prompting protocol.

Watch

Extended reading notes

Core claim

The central claim is that reasoning instruction tuning does not transfer to dialogue summarization, and may actively hurt it. Across SAMSum and DialogSum, the non-reasoning GPT-4o and DeepSeek-V3 outperform their reasoning counterparts on ROUGE, BLEU, CHRF, BERTScore, MoverScore, and COMET. The same pattern appears on CSDS for Chinese customer-service dialogues, where the gap holds for user-specific, agent-specific, and overall summaries, and on QMSum for long meeting transcripts under both query-specific and generic summary conditions. LLM judges, using five dimensions of summary quality and a Bradley–Terry ranking model, place DeepSeek-V3 and GPT-4o above their reasoning counterparts, and the paper's case study shows a reasoning model inserting a hallucinated detail and redundant reasoning. The paper concludes that the conciseness and fidelity required by dialogue summarization are better served by strong base models than by models tuned for long chains of stepwise reasoning.

Load-bearing premise

The load-bearing premise is that the performance gap between each reasoning model and its non-reasoning counterpart is caused by the reasoning instruction tuning itself, rather than by uncontrolled differences between the two models, such as different base versions, unconfirmed architecture sharing (explicitly so for OpenAI-o1 versus GPT-4o in footnote 12), and differing training data.

Editorial extensions

If this is right

  • Practitioners choosing a model for dialogue summarization should default to strong non-reasoning LLMs such as GPT-4o or DeepSeek-V3 rather than o1-class reasoning models, since the latter provide no summarization benefit and add latency and cost.
  • Evaluation studies of reasoning models should include conciseness and faithfulness measures, not just accuracy-style metrics, because verbosity and hallucination are the observable failure modes.
  • Reasoning models may need explicit length constraints or summarization-specific inference recipes (for example, draft-then-shorten) before they can compete on summary quality.
  • The finding suggests that the benefits of explicit chain-of-thought are task-dependent, and that the field should identify which task characteristics make reasoning beneficial before assuming universal gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The result likely generalizes to other dialogue-generation tasks that reward concise, fact-bound output, such as meeting action-item extraction or customer-ticket triage, where the same over-reasoning failure may appear; this would be a straightforward extension to test.
  • A direct test of the paper's attribution would be to fine-tune a single base model with and without an explicit reasoning objective while holding all other data fixed; if the reasoning-tuned version converges on or beats the base, the causal claim would need revision.
  • The paper's observed low correlation between automatic metrics and LLM-judge scores implies that future dialogue-summarization leaderboards may need to weight human-aligned judgments more heavily than n-gram overlap, a position the paper states as a challenge without fully resolving.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper compares three reasoning-oriented LLMs (OpenAI-o1, DeepSeek-R1, QwQ-32B) with three non-reasoning counterparts (GPT-4o, DeepSeek-V3, Qwen2.5-32B-Instruct) on dialogue summarization across SAMSum, DialogSum, CSDS, and QMSum, covering generic, role-oriented, and query-oriented paradigms. Using prompting only, it reports a large set of automatic metrics, intrinsic quality measures (length, compression, novelty, coverage), LLM-as-judge rankings with Bradley-Terry scores, reasoning-trace quality scoring, and a case study. The paper's central claim is that explicit stepwise reasoning does not consistently improve dialogue summarization quality and often leads to verbosity, factual inconsistencies, and lower conciseness.

Significance. If read as an observational comparison of currently available model families, the paper is a useful empirical contribution: it spans multiple languages, domains, and summarization paradigms, uses a broad automatic-metric suite, and provides bootstrap confidence intervals for the LLM-judge analysis. The directional finding that non-reasoning models obtain higher automatic-metric scores in most settings is valuable for practitioners. However, the causal interpretation placed on the result—that reasoning instruction tuning itself harms summarization—is not identified by the experimental design, and the QMSum aggregate table contains an apparent data error. The contribution would be substantially strengthened by reframing the claims as comparing existing reasoning and non-reasoning models rather than isolating the effect of reasoning training.

major comments (4)
  1. [§4.1, §4.3.2–4.3.4, footnote 12] The paired comparisons do not isolate the effect of explicit reasoning training. OpenAI-o1 and GPT-4o are different model versions whose shared foundation is explicitly unconfirmed (footnote 12); per the cited DeepSeek-R1 report, DeepSeek-R1 is RL-trained from DeepSeek-V3-Base rather than from the DeepSeek-V3 chat model evaluated here; and QwQ-32B is a separate Qwen release from Qwen2.5-32B-Instruct. Each pair therefore differs in base training, alignment, model version, and API configuration. Statements such as 'explicit reasoning instruction tuning does not generally benefit dialogue summarization' (§4.3.2) and 'the additional instruction tuning for step-by-step reasoning does not benefit—and may even hinder' (§4.3.4) are stronger than the evidence supports. Please either reframe the central conclusion as an observational comparison of currently available reasoning versus non-reasoning model families, or add a controlled condition (e.g., prompting the same base model to produce or suppress stepwise reasoning) that can support causal attribution.
  2. [Table 4 (ALL rows), §4.3.4] The ALL rows are identical to the Committee rows for every model (e.g., OpenAI-o1 ALL R1/R2/R-L = 27.89/28.08, 6.19/6.22, 16.03/15.86; GPT-4o ALL = 29.84/30.40, 8.63/8.60, 18.39/18.34), whereas ALL should aggregate the Academic, Committee, and Product partitions. As reported, the aggregate QMSum results are invalid, and the Section 4.3.4 claim that non-reasoning models win on the overall set is not supported by the table. Please correct the table and re-run any analysis that relies on the ALL partition.
  3. [Tables 2–4, §4.3.2–4.3.4] The main automatic-metric tables report no variance or significance information. Several differences are small relative to the claimed 'clear and consistent advantage'—for example, SAMSum BERTScore F1 is 90.70 (OpenAI-o1) versus 91.18 (GPT-4o), and DialogSum BARTScore H→R differs by about 0.08 between DeepSeek-R1 and DeepSeek-V3. Without paired bootstrap confidence intervals or significance tests over the sampled dialogues, the strength of the comparative conclusions is not statistically grounded. Add such intervals/tests for the automatic metrics, or soften the corresponding statements.
  4. [§5.2, Figures 5–6] DeepSeek-V3, GPT-4o, and Qwen2.5-32B serve both as evaluated candidates and as LLM judges, creating a self-preference risk. Because the evaluators are the same three non-reasoning models that the paper concludes are superior, the ranking results in Figures 5 and 6 may reflect judge identity rather than summary quality. The reported Krippendorff's alpha values (~0.55) indicate moderate agreement, which by itself does not remove this concern. Please report per-evaluator rankings, add an external judge outside the candidate set, or apply a debiasing analysis before using these results to support the central claim.
minor comments (5)
  1. [§5.1, Table 5] The statement that 'reasoning LLMs generally produce longer summaries, resulting in lower compression rates' for English datasets is contradicted by the DeepSeek pair on QMSum: DeepSeek-R1 produces shorter summaries than DeepSeek-V3 (206.58 vs 225.15 tokens) with a slightly higher compression ratio (53.86 vs 53.39). Please qualify the generalization or discuss this per-pair exception.
  2. [Footnote 12] Since the o1/GPT-4o foundational-architecture hypothesis is explicitly unconfirmed, the abstract and conclusion should hedge claims that rely specifically on this pairing, rather than treating it as a matched comparison.
  3. [References] The in-text citation '(Hayes and and Krippendorff, 2007)' contains a doubled 'and', and the reference entry formats the author name inconsistently as 'Hayes, A.F., and Krippendorff, K.' Please correct.
  4. [Figure 4, §4.3.1] The in-context example counts are selected using GPT-4o alone and then applied to all models. A brief justification or a sensitivity check showing that the chosen counts are stable across models would strengthen the configuration choice.
  5. [§5.2] Krippendorff's alpha around 0.55 is usually characterized as moderate rather than 'satisfactory'; please describe the agreement level more precisely and discuss what it implies for the reliability of the LLM-judge rankings.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the headline result rests on external reference-based metrics, with only a minor background self-citation.

full rationale

This paper is an empirical evaluation rather than a derivation chain, so there is no equation-level self-definition or fitted parameter renamed as a prediction. The central comparative claim that non-reasoning LLMs outperform reasoning LLMs on dialogue summarization is carried by Tables 2-4, which report ROUGE, BLEU, CHRF, BERTScore, MoverScore, BARTScore, and COMET computed against human reference summaries; these metrics are external to the models being compared and are not fitted from the model outputs. The few-shot example counts were tuned on validation sets using GPT-4o in Section 4.3.1 and then fixed, which is a standard configuration choice rather than a prediction forced by construction. The only self-citation, Jin et al. (2025), appears in the related-work survey and is not load-bearing for any conclusion. The LLM-as-judge analysis in Section 5.2 does use DeepSeek-V3, GPT-4o, and Qwen2.5-32B as both evaluated candidates and evaluators, creating a self-preference risk, but this is an evaluation-bias and validity concern rather than a definitional circularity: the ranking outcome is not defined in terms of the same models' outputs, and the paper's headline conclusion is independently supported by the reference-based metrics in Tables 2-4 and by the intrinsic length and coverage statistics in Table 5. The causal attribution to 'reasoning instruction tuning' in Sections 4.3.2-4.3.4 is threatened by uncontrolled model-version differences, including the explicitly unconfirmed OpenAI-o1 versus GPT-4o pairing in footnote 12, but that is an identification threat, not circularity. Accordingly, no step reduces to its own input by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical assumptions about reference quality, API comparability, LLM-judge validity, and the comparability of paired models. The only hand-chosen numeric configuration is the per-dataset in-context example count. No new entities are invented.

free parameters (1)
  • per-dataset in-context example count = SAMSum=10, DialogSum=20, CSDS=5, QMSum=0
    Chosen by GPT-4o validation performance in the ablation in Figure 4, then applied to all models. These counts are configuration choices that may favor GPT-4o and affect cross-model comparisons.
assumptions (4)
  • domain assumption Reference summaries are treated as ground-truth quality signal.
    Automatic metrics compare against human-written references, a standard but imperfect proxy for summarization quality.
  • domain assumption Default API inference settings are comparable across models.
    Section 4.2 uses default API parameters without pinning model versions, temperature, or seeds; API behavior may drift and affect reproducibility.
  • domain assumption LLM judges provide valid preference judgments aligned with humans.
    Section 5.2 relies on three LLM judges with Krippendorff alpha around 0.55, but no human validation of these judges is reported in this paper.
  • ad hoc to paper Paired non-reasoning models are valid counterfactuals for reasoning training.
    Section 4.1 pairs OpenAI-o1 with GPT-4o, DeepSeek-R1 with DeepSeek-V3, and QwQ-32B with Qwen2.5-32B, but only the latter two share a base family; the o1/GPT-4o pairing is explicitly unconfirmed, so the comparison does not cleanly isolate reasoning ability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization." pith.science (2026). https://pith.science/paper/YMYEFO3T

@misc{pith2026250702145,
  author       = {Pith},
  title        = {Pith review of: Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMYEFO3T}},
  note         = {Machine review of arXiv:2507.02145}
}
read the original abstract

Dialogue summarization is a challenging task with significant practical value in customer service, meeting analysis, and conversational AI. Although large language models (LLMs) have achieved substantial progress in summarization tasks, the performance of step-by-step reasoning architectures-specifically Long Chain-of-Thought (CoT) implementations such as OpenAI-o1 and DeepSeek-R1-remains unexplored for dialogue scenarios requiring concurrent abstraction and conciseness. In this work, we present the first comprehensive and systematic evaluation of state-of-the-art reasoning LLMs and non-reasoning LLMs across three major paradigms-generic, role-oriented, and query-oriented dialogue summarization. Our study spans diverse languages, domains, and summary lengths, leveraging strong benchmarks (SAMSum, DialogSum, CSDS, and QMSum) and advanced evaluation protocols that include both LLM-based automatic metrics and human-inspired criteria. Contrary to trends in other reasoning-intensive tasks, our findings show that explicit stepwise reasoning does not consistently improve dialogue summarization quality. Instead, reasoning LLMs are often prone to verbosity, factual inconsistencies, and less concise summaries compared to their non-reasoning counterparts. Through scenario-specific analyses and detailed case studies, we further identify when and why explicit reasoning may fail to benefit-or even hinder-summarization in complex dialogue contexts. Our work provides new insights into the limitations of current reasoning LLMs and highlights the need for targeted modeling and evaluation strategies for real-world dialogue summarization.

Figures

Figures reproduced from arXiv: 2507.02145 by the authors.

Figure 1
Figure 1. An illustrative example of three dialogue summarization tasks explored in this study: Generic Summarization, Role-oriented Summarization, and Query-oriented Summarization. widely recognized paradigms include generic summarization, which involves summarizing the entire dialogue; role￾oriented summarization, focusing specifically on the perspectives of distinct dialogue participants; and query-oriented summarization, … view at source ↗
Figure 2
Figure 2. Overview of our experimental framework. We evaluate multiple reasoning and non-reasoning LLMs across three dialogue summarization paradigms using diverse datasets. during inference (Chen et al., 2025c). Other studies have noted that longer reasoning chains do not always improve accuracy and may actually increase the likelihood of self-contradictions and unnecessary self-revisions (Zeng et al., 2025). The evaluation … view at source ↗
Figure 3
Figure 3. Prompt templates for generic, role-oriented, and query-based dialogue summarization tasks used in our experiments. Note that for role-oriented summarization, since the evaluated dataset is in Chinese, we directly use Chinese prompts as shown. The English translations in the figure are provided for clarity. 2021), CSDS3 (Lin et al., 2021), and QMSum4 (Zhong et al., 2021). Each dataset possesses unique characteristics… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance comparison under varying numbers of in-context examples across different datasets using gpt-4o. The best-performing number of in-context examples for each dataset is highlighted by a vertical red line and annotated as “Best performance.” For CSDS, we report…
Figure 5
Figure 5. Figure 5: Distribution of Model Rankings across Evaluation Dimensions. The heatmap illustrates the percentage distribution of each model’s rank (1st–6th) for all five evaluation criteria, as assessed by LLM judges. non-reasoning LLMs generate summaries closer in length and extra…
Figure 6
Figure 6. Figure 6: Bradley-Terry scores with 95% confidence intervals across five evaluation dimensions. Overall A comprehensive judgment considering all of the above aspects. The evaluator models were explicitly instructed to rank candidate summaries from best to worst, with the flexibi…
Figure 7
Figure 7. Figure 7: Distribution of DeepSeek-R1 Reasoning Processes across Evaluation Dimensions. To better understand the relationship between automatic and LLM-based evaluation, we also report the correlation between these metrics in Appendix B.1. This analysis reveals that the correlat…
Figure 8
Figure 8. Figure 8: Spearman correlations between reasoning process evaluation dimensions and summarization quality metrics (LLMs evaluation: Relevance–Overall; automatic metrics: ROUGE-1–COMET). Asterisks (*) denote statistically significant correlations (𝑝 < 0.05). synthesis. Additional…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization

    cs.CL 2026-06 conditional novelty 6.0 of 10

    OmniCSEval evaluates 28 LLMs on 1,800 conversations across six scenarios using bidirectional fact-checking, finding that reasoning improves completeness and conciseness but not faithfulness, and that scale still matters.

  2. Context-Adaptive Synthesis and Compression for Enhanced Retrieval-Augmented Generation in Complex Domains

    cs.CL 2025-08 conditional novelty 4.0 of 10

    CASC uses a fine-tuned Llama-2-7B to extract, de-conflict, and structure retrieved contexts, reporting higher F1 and lower hallucination than RAG baselines on the new SciDocs-QA benchmark.

Reference graph

Works this paper leans on

37 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    Block,J.,Chen,Y.P.,Budharapu,A.,Anthony,L.,Dorr,B.,2023.SummaryCycles:ExploringtheImpactofPromptEngineeringonLargeLanguage Models’ Interaction with Interaction Log Information, in: Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, Keyan Jin et al.:Preprint submitted to Elsevier Page 17 of 21 Evaluation of Reasoning LLMs for Dial...

    work page doi:10.18653/v1/2023.eval4nlp-1.7 2023

  2. [4]

    doi:10.48550/arXiv.2005.14165

    Language Models are Few-Shot Learners. doi:10.48550/arXiv.2005.14165. arXiv:2005.14165 [cs]. Chen, A., Song, Y., Zhu, W., Chen, K., Yang, M., Zhao, T., zhang, M., 2025a. Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysis. doi:10.48550/arXiv.2502.11544. Chen, J., Wei, Z., Ren, Z., Li, Z., Zhang, J., 2025b. LR$^2$Ben...

    arXiv doi:10.48550/arxiv.2005.14165 2005

  3. [9]

    Neural Text Summarization: A Critical Evaluation, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics. pp. 540–551. doi:10.18653/v1/D19-1051. Laban, P., Kryscinski, W., Agarwal, D., Fabb...

    doi:10.18653/v1/d19-1051 2019

  4. [11]

    doi:10.48550/arXiv.2502.12289, arXiv:2502.12289

    Evaluating Step-by-step Reasoning Traces: A Survey. doi:10.48550/arXiv.2502.12289, arXiv:2502.12289. Lee, Y., Yun, T., Cai, J., Su, H., Song, H.,

    doi:10.48550/arxiv.2502.12289

  5. [13]

    doi:10.48550/arXiv.1910.13461

    BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. doi:10.48550/arXiv.1910.13461. arXiv:1910.13461 [cs]. Liang, X., Bian, C., Wu, S., Li, Z.,

    arXiv doi:10.48550/arxiv.1910.13461 1910

  6. [16]

    On Improving Summarization Factual Consistency from Natural Language Feedback, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada. pp. 15144–15161. doi:10.18653/v1/2023.acl-long.844. Lu, Y.J., Hu, T.Y., Koppula, H.S., Pouransari, H....

    doi:10.18653/v1/2023.acl-long.844

  7. [17]

    Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization, in: Findings of the Association for Computational Linguistics: NAACL 2025, Association for Computational Linguistics, Albuquerque, New Mexico. pp. 7237–

    work page 2025

  8. [18]

    LLM aided semi-supervision for efficient Extractive Dialog Summarization, in: Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore. pp. 10002– 10009. doi:10.18653/v1/2023.findings-emnlp.670. Murugadoss, B., Poelitz, C., Drosos, I., Le, V., McKenna, N., Negreanu, C.S., Parnin, C., Sarkar, A.,

    doi:10.18653/v1/2023.findings-emnlp.670 2023

Show all 37 references
  1. [19]

    Proceedings of the AAAI Conference on Artificial Intelligence 39, 19589–19597

    Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions. Proceedings of the AAAI Conference on Artificial Intelligence 39, 19589–19597. doi:10.1609/aaai.v39i18.34157. Okadome,Y.,Yuguchi,A.,Fukui,R.,Matsumoto,Y.,2024. PromptDesignUsingPastDialogueSum...

    doi

  2. [22]

    Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends,in:Proceedingsofthe62ndAnnualMeetingoftheAssociationforComputationalLinguistics(Volume1:LongPapers),Associationfor Computational Linguistics, Bangkok, Thailand. pp. 12549–12561. doi...

    doi

  3. [23]

    COMET: A Neural Framework for MT Evaluation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online. pp. 2685–2702. doi:10. 18653/v1/2020.emnlp-main.213. Reiter, E.,

    2020

  4. [24]

    Computational Linguistics 44, 393–401

    A Structured Review of the Validity of BLEU. Computational Linguistics 44, 393–401. doi:10.1162/coli_a_00322. Rush,A.M.,Chopra,S.,Weston,J.,2015.ANeuralAttentionModelforAbstractiveSentenceSummarization.doi: 10.48550/arXiv.1509.00685. arXiv:1509.00685 [cs]. See,A.,Liu,P.J.,Mann...

    2015 doi arXiv

  5. [25]

    doi:10.48550/ arXiv.2407.00908

    FineSurE: Fine-grained Summarization Evaluation using LLMs. doi:10.48550/ arXiv.2407.00908. Sun, S., Yuan, R., Cao, Z., Li, W., Liu, P.,

    doi arXiv

  6. [26]

    Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization, in: Findings of the Association for Computational Linguistics: ACL 2024, Association for Computational Linguistics. pp. 7551–7558. doi:10.18653/v1/2024. findings-acl.449. Tang, L., Shalyminov, I., Wong, A., ...

    2024 doi

  7. [27]

    TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization, in: Proceedings of the 2024 ConferenceoftheNorthAmericanChapteroftheAssociationforComputationalLinguistics:HumanLanguageTechnologies(Volume1:Long Papers), Association for Computational Linguis...

    2024 doi

  8. [28]

    Dialogue Summarization with Mixture of Experts based on Large Language Models, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Keyan Jin et al.:Preprint submitted to...

    doi

  9. [29]

    Zero-Shot Cross-Lingual Summarization via Large Language Models, in: Proceedings of the 4th New Frontiers in Summarization Workshop, Association for Computational Linguistics, Singapore. pp. 12–23. doi:10.18653/v1/2023.newsum-1.2. Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z.,...

    2023 doi

  10. [30]

    Neurocomputing 639, 130193

    Recursively summarizing enables long-term dialogue memory in large language models. Neurocomputing 639, 130193. doi:10.1016/j.neucom.2025.130193. Xiao,J.,Chen,Y.,Ou,Y.,Yu,H.,Shu,K.,Xiao,Y.,2024. Baichuan2-Sum:InstructionFinetuneBaichuan2-7BModelforDialogueSummarization, in: 20...

    2025

  11. [31]

    doi:10.48550/arXiv.2505.09388

    Qwen3 Technical Report. doi:10.48550/arXiv.2505.09388. Ye,Y.,Fu,B.,Huang,C.,Chen,Y.,Shi,X.,2025.HowWellDoLargeReasoningModelsTranslate?AComprehensiveEvaluationforMulti-Domain Machine Translation. doi:10.48550/arXiv.2505.19987. Yuan, W., Neubig, G., Liu, P.,

    2025 doi arXiv

  12. [32]

    BARTScore: Evaluating Generated Text as Text Generation, in: Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 27263–27277. Zeng,Z.,Cheng,Q.,Yin,Z.,Zhou,Y.,Qiu,X.,2025. RevisitingtheTest-TimeScalingofo1-likeModels:DotheyTrulyPossessTest-TimeScalin...

    2025 doi arXiv

  13. [33]

    Extractive Summarization via ChatGPT for Faithful Summary Generation, in: Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore. pp. 3270–3278. doi:10.18653/v1/2023. findings-emnlp.214. Zhang*, T., Kishore*,...

    2023 doi

  14. [34]

    Zhao,W.,Peyrard,M.,Liu,F.,Gao,Y.,Meyer,C.M.,Eger,S.,2019

    BERTScore: Evaluating Text Generation with BERT, in: International Conference on Learning Representations. Zhao,W.,Peyrard,M.,Liu,F.,Gao,Y.,Meyer,C.M.,Eger,S.,2019. MoverScore:TextGenerationEvaluatingwithContextualizedEmbeddingsand Earth Mover Distance, in: Proceedings of the ...

    2019

  15. [36]

    QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online. pp....

    2021 doi

  16. [37]

    Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi- DomainDialogueSummarization,in:FindingsoftheAssociationforComputationalLinguistics:EMNLP2023,AssociationforComputational Linguistics, Singapore. pp. 6893–6908. doi:10.18653/v1/2023.findings-emnlp.460. Zhou,...

    doi

  17. [311]

    Popović, M.,

    doi:10.3115/1073083.1073135. Popović, M.,

  18. [578]

    Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Awadallah, A.H., Celikyilmaz, A., Liu, Y., Qiu, X., Radev, D.,

    doi:10.18653/v1/D19-1053. Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Awadallah, A.H., Celikyilmaz, A., Liu, Y., Qiu, X., Radev, D.,

    doi

  19. [1952]

    The Method of Paired Comparisons

    Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39, 324–345. doi:10.2307/2334029, arXiv:2334029. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S.,...

    doi

  20. [2007]

    Communication Methods and Measures 1, 77–89

    Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures 1, 77–89. doi:10.1080/19312450709336664. Jin, K., Wang, Y., Yang, X., Im, S.K.,

    doi

  21. [2015]

    chrF: Character n-gram F-score for automatic MT evaluation, in: Proceedings of the Tenth Workshop on Statistical Machine Translation, Association for Computational Linguistics, Lisbon, Portugal. pp. 392–395. doi:10.18653/v1/W15-3049. Ramprasad, S., Ferracane, E., Lipton, Z.,

    doi

  22. [2018]

    Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computat...

    2018 doi arXiv

  23. [2019]

    SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization, in: Proceedings of the 2nd Workshop on New Frontiers in Summarization, Association for Computational Linguistics, Hong Kong, China. pp. 70–79. doi:10.18653/v1/D19-5409. Grusky, M., Naaman, M., Artzi, Y.,

    doi

  24. [2020]

    Intrinsic Evaluation of Summarization Datasets, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online. pp. 8075–8096. doi:10.18653/v1/2020. emnlp-main.649. Bradley, R.A., Terry, M.E.,

    2020 doi

  25. [2021]

    CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization, in: Proceedings of the 2021 Conference onEmpirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic. pp. 4436–4451. doi...

    2021 doi

  26. [2022]

    Towards Modeling Role-Aware Centrality for Dialogue Summarization, in: Proceedings of the 2nd ConferenceoftheAsia-PacificChapteroftheAssociationforComputationalLinguisticsandthe12thInternationalJointConferenceonNatural Language Processing (Volume 2: Short Papers), Association ...

    doi

  27. [2023]

    SummEdits: Measuring LLM Ability at Factual Reasoning ThroughTheLensofSummarization,in:Proceedingsofthe2023ConferenceonEmpiricalMethodsinNaturalLanguageProcessing,Association Keyan Jin et al.:Preprint submitted to Elsevier Page 18 of 21 Evaluation of Reasoning LLMs for Dialogu...

    doi

  28. [2024]

    doi:10.48550/arXiv.2409.19898

    UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs. doi:10.48550/arXiv.2409.19898. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.,

    doi arXiv

  29. [2025]

    Expert Systems 42, e70043

    HiSum: Hierarchical Topic-Driven Approach for Role-Oriented Dialogue Summarisation. Expert Systems 42, e70043. doi:10.1111/exsy.70043. Kryscinski, W., Keskar, N.S., McCann, B., Xiong, C., Socher, R.,

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.