Pith. sign in

REVIEW 3 major objections 6 minor 46 references

Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Human-written plain language summaries lead to higher reader comprehension than LLM-generated summaries.

desk verdict A large-scale, useful evaluation of LLM plain language summaries whose central human-vs-LLM comparison needs one missing detail disclosed (the input text for the MCQ generator) before the result can be trusted. read the letter →

arxiv 2505.10409 v1 pith:HPCCXUSE submitted 2025-05-15 cs.CL

Yue Guo , Jae Ho Sohn , Gondy Leroy , Trevor Cohen This is my paper · ORCID
classification cs.CL
keywords plainlanguagesummariesLLMgenerationcomprehensionevaluationcrowdsourcedhealthcommunicationautomatedmetricsmultiple-choicereadabilitymeasures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can write plain language summaries of medical research that lay readers actually understand, not merely summaries that look understandable. Across 50 biomedical abstracts, crowd workers rated six LLM-generated summaries per abstract as similar to the human-written versions in simplicity, coherence, informativeness, and faithfulness, yet the same readers scored significantly lower on identical multiple-choice comprehension questions after reading the LLM versions. Ten automated metrics were checked against comprehension, and most failed; only a question-answering-based metric tracked reader understanding. The paper's point is that perceived quality and actual comprehension diverge, so plain language summaries should be evaluated by what readers can do with the text, not by how fluent it appears.

What carries the argument

The controlled within-abstract comparison is the load-bearing mechanism: each abstract has one human-written PLS and six LLM-generated variants, and the same three multiple-choice questions are attached to every version, so any difference in accuracy can be attributed to the summary text itself. The comprehension questions were generated by a separate LLM and kept identical across versions; recall was scored by token overlap with the original summary. The statistical core is a set of paired t-tests between human and LLM versions plus linear mixed-effects models with random intercepts for participants and abstracts, used to test which human ratings and automated metrics predict multiple-choice accuracy.

What would settle it

Generate the comprehension questions from a neutral common source, such as the scientific abstract alone or alternating randomly between versions, keep raters blind to which summary they read, and measure multiple-choice accuracy again. If the human-written advantage disappears, the reported comprehension gap is an artifact of question provenance rather than of summary quality.

Watch

Extended reading notes

Core claim

The central claim is that human-written plain language summaries convey biomedical content more effectively than LLM-generated summaries, even though the two are subjectively indistinguishable. The study compares human-written PLSs to six LLM-generated variants—one unoptimized and five optimized for simplification, informativeness, coherence, faithfulness, or all combined—using the same 50 scientific abstracts. All versions were paired with the same three multiple-choice questions about the study's motivation, methods, and results, and participants answered significantly more accurately after reading the human versions. The paper also reports that lexical-overlap metrics (ROUGE, BLEU, METEOR, SARI), perplexity-based fluency, and most model-based metrics do not significantly predict comprehension, while the QA-based metric QAEval does.

Load-bearing premise

The main result assumes the three fixed multiple-choice questions do not favor the human-written summaries, but the paper never states what text was used to generate those questions.

Editorial extensions

If this is right

  • Comprehension questions should become a standard part of plain language summary evaluation, since Likert ratings in this study did not reveal the comprehension gap.
  • Lexical and perplexity-based automated metrics should not be used as proxies for PLS quality; QA-based metrics such as QAEval align better with reader understanding.
  • LLM generation strategies should be optimized for comprehension, for example by rewarding summaries that support correct answers to questions, rather than for readability or n-gram similarity.
  • Including necessary background information seems to help lay readers, because background ratings were among the strongest predictors of multiple-choice accuracy.
  • Diverse lay audiences matter in evaluation: moderately familiar participants outperformed self-identified experts, so expert panels may misjudge what is understandable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to use answerability as a training signal: generate summaries, ask a QA model questions about the source abstract, and reward summaries from which those questions can be answered correctly. If the comprehension gap closes, answerability is a useful objective.
  • The non-significant recall difference suggests the human advantage may be specific to retrieving targeted facts about motivation, methods, and results, not to general memorability. A direct test would score recall at the level of individual claims rather than token overlap.
  • All six LLM variants behaved similarly regardless of optimization prompt, which leaves open that current prompt-level optimization does not steer comprehension-relevant content. A stricter comparison would vary decoding or add retrieval-augmented background information.
  • Since only 50 abstracts were sampled, a stratified replication across topics and reader familiarity levels could show whether the human advantage is driven mainly by the background information human authors add.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper reports a crowdsourced evaluation of plain language summaries (PLSs) generated by GPT-4 against human-authored PLSs from the CELLS corpus. Using 50 abstract–PLS pairs, six LLM-generated PLS variants (criteria-agnostic, simplification, informativeness, coherence, faithfulness, and all combined), and 150 Amazon Mechanical Turk workers, the authors collect subjective Likert ratings, multiple-choice comprehension questions, and recall responses. They find that LLM-generated PLSs receive subjective ratings comparable to human-written PLSs, but that readers answer multiple-choice questions significantly more accurately after reading human-written PLSs. They also report that most automated evaluation metrics are not significantly associated with comprehension, with QAEval being the only significant predictor in their rigorous subset. The paper argues for comprehension-centered evaluation of PLSs rather than reliance on surface-level metrics or subjective ratings.

Significance. If the central finding is valid, this is a valuable contribution to health communication and NLP evaluation. The study is unusually large for PLS evaluation, uses both subjective and objective measures, and makes a concrete attempt to avoid model self-preference by generating comprehension questions with LLaMA-3 rather than GPT-4. The mixed-effects modeling framework and the comparison of ten automated metrics against human comprehension are also useful. The main claim—that perceived quality does not imply actual comprehension—is important and testable. However, the validity of the headline result rests entirely on the multiple-choice instrument, and a key detail about how that instrument was constructed is missing from the manuscript.

major comments (3)
  1. [Section 4.2] The comprehension-question generation procedure does not disclose the input text. The attention-check prompt explicitly refers to "the text", but the comprehension prompt says only "Create three multiple-choice questions in plain language that assess (1) the motivation of the study, (2) the methods used, and (3) the main results" and shows no input field. Because the same three questions are used across all PLS versions, the source of these questions is load-bearing for the central claim in Section 2.4. If LLaMA-3 was prompted with the human-written PLS, the correct answers would be keyed to the structure, vocabulary, and possibly added background of the human text, making the human condition artificially easier. Since recall differences were not significant, the MCQ is the only objective measure supporting the claim that human-written PLSs lead to significantly better comprehension. The authors must state what text was supplied to LLaMA-3 when generating the questions, or otherwise demonstrate source-neutrality; without this, the main finding may be a measurement artifact.
  2. [Section 4.4] The statistical comparison in Section 2.4 uses paired t-tests, but the pairing unit is unspecified. Participants were told that each person evaluated only one version, and the study design assigns each participant to one batch; it is unclear whether the pairing is across abstracts (e.g., averaged ratings per abstract per version), across participants, or across annotation pairs. The validity of the reported p-values depends on the correct pairing structure, and the description in Section 4.4 does not define it. Please specify the exact pairing and explain how the design supports it.
  3. [Section 2 and Tables 2–3] The "rigorous evaluation subset" is constructed by removing responses whose completion time falls outside the 25th to 75th percentile, but no justification is given for this percentile filter, and the filter is applied after other exclusions. This arbitrary choice affects the secondary analyses in Tables 2 and 3, even though the full-dataset results in Appendix Tables 4 and 5 are directionally consistent. The authors should justify the filter, state whether it was applied per participant or per batch, and report whether the conclusions of Section 2.5 and 2.6 are sensitive to the specific percentile threshold.
minor comments (6)
  1. [Section 2.1] There is a typo: "the our Institutional Review Board" should be "our Institutional Review Board".
  2. [Section 4.2] The word "comperehensively" in the last sentence of Section 4.2 is misspelled; it should be "comprehensively".
  3. [Section 4.4] In the description of mixed-effects models, "differences in abstract difficult" should be "differences in abstract difficulty".
  4. [Section 2.5] The text says subjective ratings "predict" comprehension, but the models are contemporaneous associations within the same reading episode; consider using "are associated with" or "are predictive of" only if a temporal or out-of-sample analysis is provided.
  5. [Figure 2 caption] The caption mentions "factuality" as a rated dimension, while the text consistently uses "faithfulness"; please align the terminology.
  6. [Section 2.2] The discussion of readability metrics says human-written scientific abstracts received higher scores than PLSs "but the difference was not statistically significant", while Figure 1 asterisks indicate only comparisons of LLM-generated PLSs to human PLSs; please clarify whether the abstract comparison is included in the multiple-comparison adjustment.

Circularity Check

0 steps flagged · score 1.0 of 10

No demonstrated circularity; the central human-vs-LLM comprehension comparison rests on external crowdsourced data, with only minor non-load-bearing self-citations.

full rationale

The paper's central claim—that human-written PLSs support comprehension significantly better than LLM-generated PLSs—is supported by objective multiple-choice accuracy and recall data collected from MTurk participants, which are external to the modeling pipeline and not constructed from the claimed conclusion. The mixed-effects models in Sections 2.5 and 2.6 estimate associations between ratings or automated metrics and comprehension outcomes; they do not fit a parameter to a subset and then rename it as a prediction. Self-citations to the CELLS dataset [3], the APPLS metric suite [18], and the coherence metrics of [13] supply inputs and evaluation tools that are then tested against new human judgments, so they are not used to make the conclusion true by construction. One validity concern is that Section 4.2 quotes the LLaMA-3 comprehension-question prompt ('Create three multiple-choice questions in plain language that assess (1) the motivation of the study, (2) the methods used, and (3) the main results') without stating which text was supplied as input; if the questions were generated from the human-written PLS, the headline comparison could be biased. However, the paper nowhere states this, and an omitted input detail is an unreported validity risk rather than a demonstrated circular reduction. No load-bearing step in the derivation reduces to its own inputs by definition or by self-citation, so the score reflects only the presence of minor self-citations that are not load-bearing.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper contributes no new theoretical entities or fitted physical constants. Its load-bearing assumptions are the validity of the crowd sample, the construction of the comprehension test, and the gold-standard status of the CELLS human PLSs. The only hand-chosen numerical choice that materially affects the conclusions is the completion-time filter.

free parameters (2)
  • Completion-time percentile filter = 25th to 75th percentile
    Responses outside this hand-chosen window are removed to form the rigorous subset (N=459) used for the automated-metric alignment analysis. Changing this threshold changes which metrics appear significant (compare Table 3 with Appendix Table 5), so the headline 'metrics fail' claim depends on this choice.
  • Number of abstracts sampled = 50
    The sample size is a design choice, not a fitted constant, but it limits topical diversity and statistical power; the authors acknowledge this limitation in Section 3.
assumptions (4)
  • domain assumption MTurk workers who self-report English proficiency, US residence, and no medical training or advanced biology coursework are valid proxies for lay readers.
    Used in Section 2.1 to define the evaluation population; if the screening is inaccurate, comprehension results could reflect the sample rather than the summaries.
  • domain assumption The multiple-choice questions generated by LLaMA-3 measure comprehension of the summary in a version-independent way.
    Section 4.2 relies on identical MCQs across all PLS versions; the source text for generation is not stated, so the assumption is incomplete.
  • domain assumption Human-authored PLSs in the CELLS dataset are a suitable gold standard baseline.
    Section 4.1 samples 50 pairs from CELLS (ref [3], authored by the same group) and uses the human PLS as the reference for both generation prompts and evaluation.
  • standard math Linear mixed-effects model assumptions hold for the rating, comprehension, and automated metric analyses.
    Section 4.4 specifies random intercepts for participants and abstracts but does not check distributional assumptions or report model diagnostics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation." pith.science (2026). https://pith.science/paper/HPCCXUSE

@misc{pith2026250510409,
  author       = {Pith},
  title        = {Pith review of: Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPCCXUSE}},
  note         = {Machine review of arXiv:2505.10409}
}
read the original abstract

Plain language summaries (PLSs) are essential for facilitating effective communication between clinicians and patients by making complex medical information easier for laypeople to understand and act upon. Large language models (LLMs) have recently shown promise in automating PLS generation, but their effectiveness in supporting health information comprehension remains unclear. Prior evaluations have generally relied on automated scores that do not measure understandability directly, or subjective Likert-scale ratings from convenience samples with limited generalizability. To address these gaps, we conducted a large-scale crowdsourced evaluation of LLM-generated PLSs using Amazon Mechanical Turk with 150 participants. We assessed PLS quality through subjective Likert-scale ratings focusing on simplicity, informativeness, coherence, and faithfulness; and objective multiple-choice comprehension and recall measures of reader understanding. Additionally, we examined the alignment between 10 automated evaluation metrics and human judgments. Our findings indicate that while LLMs can generate PLSs that appear indistinguishable from human-written ones in subjective evaluations, human-written PLSs lead to significantly better comprehension. Furthermore, automated evaluation metrics fail to reflect human judgment, calling into question their suitability for evaluating PLSs. This is the first study to systematically evaluate LLM-generated PLSs based on both reader preferences and comprehension outcomes. Our findings highlight the need for evaluation frameworks that move beyond surface-level quality and for generation methods that explicitly optimize for layperson comprehension.

Figures

Figures reproduced from arXiv: 2505.10409 by the authors.

Figure 1
Figure 1. Comparison of linguistic and readability metrics across human-written and LLM-generated PLS under various optimization strategies (N = 1346). Metrics include paragraph length, vocabulary size, word familiarity, and three standard readability scores (Flesch-Kincaid, SMOG, Automated Readability Index). The red dashed line indicates the mean score for human-written PLS. Asterisks (*) denote statistically significant di… view at source ↗
Figure 2
Figure 2. Subjective and objective evaluation of human-written and LLM-generated PLS. The left panel (”Subjective Evaluation”) shows mean human Likert-scale ratings across five quality dimensions: simplicity, informativeness, coherence, faithfulness, and providing necessary background knowledge (N = 1346). The right panel (”Objective Evaluation”) presents performance on comprehension tasks: multiple-choice questions (MCQ) acc… view at source ↗
Figure 3
Figure 3. Heatmaps of mean task time grouped by multiple-choice question (MCQ) accuracy (left) and recall rate (right), with rows representing self-identified familiarity (N = 1346). Color indicates average work time (in minutes). Each cell shows both the count of participants and the percentage within the corresponding familiarity group. which capture paragraph-level and discourse-level structure respectively, showed compara… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 34 canonical work pages

  1. [1]

    & Singh, J

    Ravinetto, R. & Singh, J. A. Responsible dissemination of health and medical research: some guidance points. BMJ evidence-based medicine 28, 144–147 (2023)

    work page 2023

  2. [2]

    Clearly communicating research results across the clinical trials continuum (2025)

    National Institutes of Health. Clearly communicating research results across the clinical trials continuum (2025). Accessed: March 10, 2025

    work page 2025

  3. [3]

    & Cohen, T

    Guo, Y ., Qiu, W., Leroy, G., Wang, S. & Cohen, T. A. Retrieval augmentation of large language models for lay language generation. J. biomedical informatics 104580 (2022)

    work page 2022

  4. [4]

    Luhn, H. P. The automatic creation of literature abstracts. IBM J. research development 2, 159–165 (1958)

    work page 1958

  5. [5]

    Lewis, M. et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)

    arXiv 2019

  6. [6]

    & Cohen, T

    Guo, Y ., Qiu, W., Wang, Y . & Cohen, T. Automated lay language summarization of biomedical scientific reviews. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 160–168 (2021)

    2021

  7. [7]

    Devaraj, A., Marshall, I., Wallace, B. C. & Li, J. J. Paragraph-level simplification of medical texts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4972–4984 (2021)

    work page 2021

  8. [8]

    Touvron, H. et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    arXiv 2023

Show all 46 references
  1. [9]

    & Ananiadou, S

    Luo, Z., Xie, Q. & Ananiadou, S. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.15621 (2023)

    2023 arXiv

  2. [10]

    & Kilicoglu, H

    You, Z., Radhakrishna, S., Ming, S. & Kilicoglu, H. Uiuc_bionlp at biolaysumm: an extract-then-summarize approach augmented with wikipedia knowledge for biomedical lay summarization. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, 132–143 (2024)

    2024

  3. [11]

    & Lin, C

    Goldsack, T., Scarton, C., Shardlow, M. & Lin, C. Overview of the biolaysumm 2024 shared task on the lay summarization of biomedical research articles. arXiv preprint arXiv:2408.08566 (2024)

    2024 arXiv

  4. [12]

    & Shukla, A

    Leroy, G., Kauchak, D., Harber, P., Pal, A. & Shukla, A. Text and audio simplification: Human vs. chatgpt. AMIA Summits on Transl. Sci. Proc. 2024, 295 (2024)

    2024

  5. [13]

    & Leroy, G

    Cohen, T., Xu, W., Guo, Y ., Pakhomov, S. & Leroy, G. Coherence and comprehensibility: Large language models predict lay understanding of health-related content. J. Biomed. Informatics 161, 104758 (2025)

    2025

  6. [14]

    Jain, D., Borah, M. D. & Biswas, A. Summarization of legal documents: Where are we now and the way forward. Comput. Sci. Rev. 40, 100388 (2021)

    2021

  7. [15]

    & Demner-Fushman, D

    Ondov, B., Attal, K. & Demner-Fushman, D. A survey of automated methods for biomedical text simplification. J. Am. Med. Informatics Assoc. 29, 1976–1988 (2022)

    2022

  8. [16]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y . Rouge: A package for automatic evaluation of summaries. InText summarization branches out, 74–81 (2004)

    2004

  9. [17]

    & Rappoport, A

    Sulem, E., Abend, O. & Rappoport, A. Bleu is not suitable for the evaluation of text simplification. arXiv preprint arXiv:1810.05995 (2018)

    2018 arXiv

  10. [18]

    Guo, Y ., August, T., Leroy, G., Cohen, T. A. & Wang, L. L. Appls: Evaluating evaluation metrics for plain language summarization. In Conference on Empirical Methods in Natural Language Processing (2023). 12/14

    2023

  11. [19]

    & Venditti, J

    Hirschberg, J., Liscombe, J. & Venditti, J. Experiments in emotional speech. In ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 1–7 (2003)

    2003

  12. [20]

    & Martin, M

    Wiebe, J., Wilson, T., Bruce, R., Bell, M. & Martin, M. Learning subjective language. Comput. linguistics 30, 277–308 (2004)

    2004

  13. [21]

    G., Nettle, D

    Haselton, M. G., Nettle, D. & Andrews, P. W. The evolution of cognitive bias. The handbook evolutionary psychology 724–746 (2015)

    2015

  14. [22]

    & Cohen, T

    Guo, Y ., Qiu, W., Leroy, G., Wang, S. & Cohen, T. Cells: A parallel corpus for biomedical lay language generation.arXiv preprint arXiv:2211.03818 (2022)

    2022 arXiv

  15. [23]

    Quality aspects of annotated data: A research synthesis

    Beck, J. Quality aspects of annotated data: A research synthesis. AStA Wirtschafts-und Sozialstatistisches Arch. 17, 331–353 (2023)

    2023

  16. [24]

    Leroy, G. et al. Next-generation metrics for monitoring genetic erosion within populations of conservation concern. Evol. Appl. 11, 1066–1083 (2018)

    2018

  17. [25]

    Srikanth, N. & Li, J. J. Elaborative simplification: Content addition and explanation generation in text simplification. arXiv preprint arXiv:2010.10035 (2020)

    2020 arXiv

  18. [26]

    Matz, S. C. et al. The potential of generative ai for personalized persuasion at scale. Sci. Reports 14, 4692 (2024)

    2024

  19. [27]

    X., Park, J

    Karinshak, E., Liu, S. X., Park, J. S. & Hancock, J. T. Working with ai to persuade: Examining a large language model’s ability to generate pro-vaccination messages. Proc. ACM on Human-Computer Interact. 7, 1–29 (2023)

    2023

  20. [28]

    & Baram-Tsabari, A

    Rakedzon, T., Segev, E., Chapnik, N., Yosef, R. & Baram-Tsabari, A. Automatic jargon identifier for scientists engaging with the public and science communication educators. PloS one 12, e0181742 (2017)

    2017

  21. [29]

    & Gardner, J

    Martínez Silvagnoli, L., Shepherd, C., Pritchett, J. & Gardner, J. How can we optimize the readability and format of plain language summaries for medical journal articles? a cross-sectional survey study. J Med Internet Res 22122 (2020)

    2020

  22. [30]

    Brehaut, J. C. et al. Presenting evidence to patients online: what do web users think of consumer summaries of cochrane musculoskeletal reviews? J. Med. Internet Res. 13, e1532 (2011)

    2011

  23. [31]

    Buljan, I. et al. Comparison of blogshots with plain language summaries of cochrane systematic reviews: a qualitative study and randomized trial. Trials 21, 1–10 (2020)

    2020

  24. [32]

    Buljan, I. et al. Framing the numerical findings of cochrane plain language summaries: two randomized controlled trials. BMC medical research methodology 20, 1–9 (2020)

    2020

  25. [33]

    Santesso, N. et al. A summary to communicate evidence from systematic reviews to the public improved understanding and accessibility of information: a randomized controlled trial. J. Clin. Epidemiol. 68, 182–190, DOI: 10.1016/j.jclinepi. 2014.04.009 (2015). Epub 2014 Jul 14

    2015 doi

  26. [34]

    & Simon, S

    Bredbenner, K. & Simon, S. M. Video abstracts and plain language summaries are more effective than graphical abstracts and published abstracts. PloS one 14, e0224697 (2019)

    2019

  27. [35]

    Buljan, I. et al. No difference in knowledge obtained from infographic or plain language summary of a cochrane systematic review: three randomized controlled trials. J. clinical epidemiology 97, 86–94 (2018)

    2018

  28. [36]

    Kirkpatrick, E. et al. Understanding plain english summaries. a comparison of two approaches to improve the quality of plain english summaries in research reports. Res. involvement engagement 3, 1–14 (2017)

    2017

  29. [37]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. https://openai.com/research/gpt-4 (2024). Accessed: February 12, 2024

    2024

  30. [38]

    & Feng, S

    Panickssery, A., Bowman, S. & Feng, S. Llm evaluators recognize and favor their own generations. Adv. Neural Inf. Process. Syst. 37, 68772–68802 (2024)

    2024

  31. [39]

    & Zhu, W.-J

    Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318 (2002). 13/14

    2002

  32. [40]

    & Lavie, A

    Banerjee, S. & Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 65–72 (2005)

    2005

  33. [41]

    & Callison-Burch, C

    Xu, W., Napoles, C., Pavlick, E., Chen, Q. & Callison-Burch, C. Optimizing statistical machine translation for text simplification. Transactions Assoc. for Comput. Linguist. 4, 401–415 (2016)

    2016

  34. [42]

    Zhang, T., Kishore, V ., Wu, F., Weinberger, K. Q. & Artzi, Y . Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  35. [43]

    Maddela, M., Dou, Y ., Heineman, D. & Xu, W. Lens: A learnable evaluation metric for text simplification.arXiv preprint arXiv:2212.09739 (2022)

    2022 arXiv

  36. [44]

    & Roth, D

    Deutsch, D., Bedrax-Weiss, T. & Roth, D. Towards question-answering as an automatic metric for evaluating the content quality of a summary. Transactions Assoc. for Comput. Linguist. 9, 774–789 (2021)

    2021

  37. [45]

    Fabbri, A. R. et al. Summeval: Re-evaluating summarization evaluation. Transactions Assoc. for Comput. Linguist. 9, 391–409 (2021)

    2021

  38. [46]

    & Specia, L

    Alva-Manchego, F., Martin, L., Scarton, C. & Specia, L. EASSE: Easier automatic sentence simplification evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi...

    2019 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.