Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper finds that n-gram metrics correlate poorly with human judgments of summaries in fusional languages, while neural evaluation-trained metrics like COMET correlate better, and tokenization can rescue n-gram metrics.

desk verdict Valuable 20k-annotation resource, but the headline typology claim is an artifact of pooling Arabic and Hebrew; per-language data show only Hebrew drives the negative correlation. read the letter →

arxiv 2507.08342 v1 pith:5VIDZHBF submitted 2025-07-11 cs.CL

Itai Mondshine , Tzuf Paz-Argaman , Reut Tsarfaty This is my paper
classification cs.CL
keywords multilingualsummarizationautomaticevaluationmetricsROUGEBLEUCOMETtypologicalfamiliesfusionalmorphologylow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Summarization quality for English is usually measured by n-gram overlap metrics such as ROUGE, but it is unclear whether those numbers mean the same thing in other languages. This paper tests ten evaluation metrics against about 20,000 human ratings of generated summaries in eight languages spanning four typological families (isolating, agglutinative, low-fusional, high-fusional), with both high- and low-resource members in each family. It tries to establish that n-gram metrics under-correlate with human judgment in fusional languages, that adapting tokenization or lemmatizing can substantially improve (sometimes reverse) those correlations, and that the neural metric COMET consistently outperforms other metrics in low-resource settings. If true, the paper would imply that multilingual leaderboards built on ROUGE/BLEU are unreliable for fusional languages and that evaluation practice should shift toward neural metrics trained for the evaluation task.

What carries the argument

The load-bearing mechanism is the evaluation-correlation suite: for each language, human annotators rate model-generated summaries on coherence and completeness, and each candidate metric's score is then Pearson-correlated with the averaged human ratings, with p-values reported. The typological axis is the paper's central experimental design: eight languages grouped into four families, each containing a high- and a low-resource member, so that metric behavior can be attributed to morphological type rather than mere language identity. Within this design, the decisive technical objects are the metric adaptations: ROUGE with mBERT BPE tokenization, ROUGE with language-specific tokenizers, and BLEU over lemmatized text, which change what counts as a 'word' and thereby change the overlap signal; and COMET, a neural regression metric trained on human quality judgments, whose weights are reused for summarization by omitting the source side.

What would settle it

Run the same annotation-correlation protocol on additional fusional languages with rich morphology (e.g., Russian, Polish, or Czech); if space-delimited ROUGE and BLEU show positive correlations with human judgments in several such languages, the claim that n-gram metrics systematically under-correlate in fusional languages would be contradicted.

Watch

Extended reading notes

Core claim

The paper reports that the reliability of automatic summarization metrics depends on the typological family of the language being evaluated. For isolating and agglutinative languages, ROUGE-family metrics show modest positive correlations with human coherence and completeness judgments; for fusional languages these correlations weaken, and for high-fusional languages (Arabic, Hebrew) they become negative for standard space-delimited ROUGE, BLEU, and CHRF. Replacing the default tokenizer with an mBERT or language-specific tokenizer, or applying BLEU to lemmatized text, improves the correlations and can turn negative values positive in the high-fusional group. Among neural metrics, COMET — a regression model trained for machine translation evaluation and adapted here by omitting the source — shows the strongest and most consistent correlations with human judgments, and in low-resource languages it beats both n-gram and other neural metrics. The authors conclude that n-gram metrics are ill-suited to fusional languages and advocate for investment in neural metrics trained specifically for evaluation.

Load-bearing premise

The paper assumes that two languages per typological family are enough to represent how that entire family behaves in evaluation; if Hebrew and Arabic are not typical of high-fusional languages, the family-level negative correlations may not generalize.

Editorial extensions

If this is right

  • ROUGE-, BLEU-, and CHRF-based leaderboards for abstractive summarization in fusional languages (e.g., Arabic, Hebrew) will misalign with human judgments, as these metrics show negative or near-zero correlations in the paper's high-fusional group.
  • Using subword tokenization (mBERT or language-specific tokenizers) before computing ROUGE, or applying BLEU to lemmatized text, can raise correlations substantially and even reverse negative trends in high-fusional languages.
  • COMET should be preferred over other neural metrics and n-gram metrics for evaluating summaries in low-resource languages, since it correlates best with human judgments there.
  • Evaluation studies should report per-language correlations with p-values and use at least about 400 samples per language, the sample size the paper's power analysis finds necessary for detecting effects at p ≤ 0.05.
  • Multilingual model development should not treat English-derived n-gram metrics as a universal yardstick; the choice of metric needs to be language-aware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The per-language results in the paper's appendix suggest the family-level negative correlation for high-fusional languages is driven mainly by Hebrew; Arabic's correlations are positive or near zero, so the typological conclusion may be about Hebrew's specific morphology or data rather than high-fusional morphology as a whole.
  • A direct test would rerun the protocol on additional fusional languages (e.g., Russian, Polish, Czech); if negative n-gram correlations do not replicate, the family-level claim needs refinement.
  • Because COMET was trained only on machine-translation quality judgments yet transfers to summarization, evaluation-specific training may transfer across generation tasks, possibly to dialogue or long-form QA in low-resource languages.
  • The tokenization results imply that standardizing subword tokenization across languages could make n-gram metrics more comparable, but the paper's own data show it can hurt isolating languages like Chinese, so any standardization must be typology-aware.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies automatic evaluation metrics for abstractive summarization in eight non-English languages spanning four typological families (isolating, agglutinative, low-fusional, and high-fusional) at both high- and low-resource levels. The authors collect roughly 20,000 human annotations of coherence and completeness for GPT-3.5 and Gemini generated summaries, and compute Pearson correlations with n-gram metrics (ROUGE, BLEU, CHRF), adapted variants (mBERT tokenization, language-specific tokenizers, lemmatization), and neural metrics (BERTScore, MoverScore, COMET, Gemini-as-a-judge). The paper claims that n-gram metrics correlate less with human judgments in fusional languages, that tokenization and lemmatization can mitigate or reverse these negative correlations, and that COMET consistently outperforms other metrics in low-resource settings.

Significance. If the headline claims are valid, the paper would provide actionable guidance for choosing evaluation metrics in multilingual summarization and would strengthen the case for neural, task-specific metrics over n-gram metrics. The paper's concrete strengths are the substantial annotation resource (~20,000 human judgments, eight languages, four families), the released code and data, the inclusion of both n-gram and neural metric families, and the explicit reporting of significance values. These assets are real regardless of the analysis issues detailed below. However, the central typological and resource-level claims are not statistically identified by the current pooled analyses, so the paper's conclusions, as stated, require substantial revision.

major comments (4)
  1. [Section 6, Table 5] The family-level correlations are computed by pooling all languages within a family, but Table 11 shows opposite correlation signs within the high-fusional family: for coherence, Arabic ROUGE1 is 0.10 while Hebrew ROUGE1 is -0.31; Arabic BLEU is 0.06 while Hebrew BLEU is -0.16; Arabic ROUGE-L is 0.06 while Hebrew ROUGE-L is -0.26. Table 4 reports large between-language differences in mean human scores (Hebrew coherence mean 3.8 vs. Arabic 2.6), so the pooled negative correlations in Table 5 can be driven entirely by between-language mean shifts rather than by any within-language association. The claim that n-gram metrics show lower correlation with human judgments in fusional languages is therefore not identified. Please report per-language correlations as primary results, or model language as a random effect and test whether family membership explains correlation magnitude.
  2. [Section 6, Table 6] The statement that COMET 'consistently outperform[s]' other neural metrics in low-resource languages is not supported by the per-language correlations in Table 11. For coherence, the low-resource COMET correlations are Yoruba 0.01, Turkish 0.21, Hebrew 0.38, and Ukrainian 0.11, while the pooled low-resource value in Table 6 is 0.18. This heterogeneity is substantial, and the paper provides no test of whether COMET's correlation is significantly higher than other metrics within the low-resource group. Please present per-language results for each resource level and add a formal interaction or bootstrap test to substantiate the resource-level claim.
  3. [Section 6, Table 5, line 16] The claimed reversal of negative correlations by applying BLEU to lemmatized text is confounded with the corruption protocol. Section A.5 states that coherence corruption is implemented by replacing verbs with their lemma forms, producing ungrammatical summaries. When BLEU is computed on lemmatized text, those corrupted summaries are normalized by the same transformation that was used to corrupt them, which can preferentially increase their metric scores and artificially inflate the correlation with human coherence judgments. The paper therefore does not establish a general benefit of lemmatization for fusional languages. Please re-evaluate the mitigation on uncorrupted data or with a corruption method that does not overlap with the tested transformation.
  4. [Section 5 and Section 6] The paper compares many metric-by-language-by-criterion correlations but reports no confidence intervals, no tests of differences between correlations (e.g., Fisher's z), and no multiple-comparison correction. Verbal claims such as 'consistently showing a better correlation' and 'the advantage of agglutinative languages' rest on point estimates that may not be statistically distinguishable. Please add formal comparisons, such as bootstrap confidence intervals or a mixed-effects model with metric and language as factors, to support the qualitative conclusions.
minor comments (5)
  1. [Table 3] In the country-of-residence list, 'mibya' appears to be a typo for 'Libya'; please correct it and also clarify entries such as 'West Africa' that are regions rather than countries.
  2. [Section 5.1] The term 'Abstactness' should be 'Abstractness' in the description of the generated-summaries analysis.
  3. [Limitations] The Limitations section says the criteria are 'coherence and consistency,' but the body of the paper consistently uses 'coherence and completeness'; please align the terminology.
  4. [Section 3, Table 8] Arabic is classified as high-resource despite being below the 0.1% token threshold stated in Section 3; the footnote justifies this by worker availability, but this exception should be discussed as a sensitivity analysis because it affects the high- versus low-resource comparisons in Table 6.
  5. [Section 4.3] The number of annotators varies considerably across languages, with Japanese having only one worker; the paper should explicitly discuss how this imbalance affects the reliability of the aggregated human scores used as the gold standard.

Circularity Check

1 steps flagged · score 6.0 of 10

The lemma-based mitigation claim is circular: the coherence corruption is defined as replacing words with lemma forms, and the proposed fix is to lemmatize before scoring, so the reversal is built into the test construction.

  1. self definitional [Section 4.1 and Appendix A.5 (coherence corruption); Section 6, Table 5 lines 6 vs 16 (BLEU lemmatized mitigation).]
    ""For coherence, we replaced nouns and verbs with their lemma forms, creating ungrammatical sentences." (Sec 4.1); "Coherence: All verbs were replaced with their lemma forms, resulting in ungrammatical sentences." (App A.5); "applying BLEU to the lemmatized text shows a significant improvement for fusional languages, with the correlation increasing from -0.10 to 0.40 for high-fusional languages (line 6 vs. 16).""

    The low-coherence examples are created by replacing surface inflected forms with lemma forms, and the proposed mitigation computes BLEU on lemmatized text, applying the same surface-to-lemma transformation to both summary and reference. Corrupted summaries are already in the representation that lemmatized BLEU maps the reference into, so any overlap gain for corrupted items is an artifact of the corruption definition. The headline reversal from negative to positive correlation for high-fusional languages is therefore a designed property of the evaluation protocol, not an independent discovery about lemmatization; the test construction and the mitigation share the same transformation. Other tokenization results (e.g., mBERT-tokenized ROUGE) are not affected by this specific circularity.

full rationale

Most of the paper is a direct, non-circular measurement: the authors collected new human judgments and correlated them with off-the-shelf n-gram and neural metrics, so the core descriptive claim that n-gram metrics correlate less in fusional languages rests on new data and external metrics. The COMET advantage and the mBERT-tokenizer improvements are also evaluated against independent human annotations and are not constructed from the same operation as the test corruption. The one genuinely circular step is the lemmatized-BLEU mitigation: the coherence corruption explicitly replaces nouns and verbs with lemma forms, and the mitigation then applies lemmatization before scoring. That makes the reported reversal from -0.10 to 0.40 a consequence of the corruption design rather than evidence about morphological typology. The family-level aggregation that pools Arabic and Hebrew into one 'high-fusional' correlation is a serious statistical identification problem (per-language signs differ sharply in Table 11), but that is a validity/correctness concern, not circularity, and is not counted in the score. The paper's self-citations (HeSum, prior Hebrew ROUGE results) are used as external data or motivation, not as load-bearing derivations, so they do not raise the circularity score. Overall, the central measurement survives, but one headline mitigation result reduces by construction, giving partial circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No mathematical derivation or fitted model is central here; the empirical claims depend on annotation reliability, the representativeness of artificially corrupted summaries, and the assumption that two languages represent a typological family. The paper partially addresses the first with Krippendorff alpha but does not address the second or third.

free parameters (1)
  • corruption_rate = 1/3 of summaries
    Section 4.1: one-third of generated summaries were artificially corrupted to create score dispersion. This hand-chosen rate shapes the quality distribution and affects correlation magnitudes, though it is not fitted to the outcome.
assumptions (3)
  • domain assumption Human Likert ratings averaged over three annotators are an interval-scale gold standard for summary quality.
    Section 4.2 and 4.3 treat the 1-4 scores as continuous and use Pearson correlation; differences in inter-annotator agreement across languages are not modeled.
  • domain assumption The artificial corruption protocol produces quality variation representative of real model errors.
    Section 4.1 and Appendix A.5 construct degraded summaries by lemmatizing, reordering, swapping entities, and inserting unrelated sentences; the conclusions about metric validity assume these corruptions resemble natural failures.
  • domain assumption Languages within each typological family are representative of that family.
    Two languages per family are pooled in Table 5; the paper assumes within-family homogeneity, which Table 11 shows is violated for high-fusional languages (Arabic versus Hebrew).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization." pith.science (2026). https://pith.science/paper/5VIDZHBF

@misc{pith2026250708342,
  author       = {Pith},
  title        = {Pith review of: Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VIDZHBF}},
  note         = {Machine review of arXiv:2507.08342}
}
read the original abstract

Automatic n-gram based metrics such as ROUGE are widely used for evaluating generative tasks such as summarization. While these metrics are considered indicative (even if imperfect) of human evaluation for English, their suitability for other languages remains unclear. To address this, we systematically assess evaluation metrics for generation both n-gram-based and neural based to evaluate their effectiveness across languages and tasks. Specifically, we design a large-scale evaluation suite across eight languages from four typological families: agglutinative, isolating, low-fusional, and high-fusional, spanning both low- and high-resource settings, to analyze their correlation with human judgments. Our findings highlight the sensitivity of evaluation metrics to the language type. For example, in fusional languages, n-gram-based metrics show lower correlation with human assessments compared to isolating and agglutinative languages. We also demonstrate that proper tokenization can significantly mitigate this issue for morphologically rich fusional languages, sometimes even reversing negative trends. Additionally, we show that neural-based metrics specifically trained for evaluation, such as COMET, consistently outperform other neural metrics and better correlate with human judgments in low-resource languages. Overall, our analysis highlights the limitations of n-gram metrics for fusional languages and advocates for greater investment in neural-based metrics trained for evaluation tasks.

Figures

Figures reproduced from arXiv: 2507.08342 by the authors.

Figure 1
Figure 1. Elo score distribution of human annotations [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Participant Interface in a closed mode: The interface includes three drop-down sections: Instructions, Qualification and the Annotation task [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. The Participant Instructions Interface: The participant has general steps and a detailed explanation and examples of each tested criteria [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The Participant Qualification Interface: The human summary is displayed at the top (the example is in Arabic), while the Qualification section below requires the participant to fill in their home state and answer a question generated by Gemini based on the human summar…
Figure 5
Figure 5. Figure 5: The Participant Annotation Interface: Two summaries are displayed side by side. Each criterion includes a slider ranging from 1 to 4, along with an info hover feature providing a reminder of the criterion’s definition [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: The Participant Annotation Interface: displayed in Spanish Coherence Completeness Typological Family Isolating Agglutinative High Fusional Low Fusional Isolating Agglutinative High Fusional Low Fusional Language Code ZH YOR JA TR AR HE ES UKR ZH YOR JA TR AR HE ES UKR …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 34 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Gulinigeer Abudouwaili, Wayit Ablez, Kahaerjiang Abiderexiti, Aishan Wumaier, and Nian Yi. 2023. Strategies to improve low-resource agglutinative languages morphological inflection. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 508--520

    work page 2023

  4. [4]

    Anton Alexandrov, Veselin Raychev, Dimitar I Dimitrov, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024. Bggpt 1.0: Extending english-centric llms to other languages. arXiv preprint arXiv:2412.10893

    arXiv 2024

  5. [5]

    Abdulaziz Alhamadani, Xuchao Zhang, Jianfeng He, and Chang-Tien Lu. 2022. Lans: Large-scale arabic news summarization corpus. arXiv preprint arXiv:2210.13600

    work page Pith review arXiv 2022

  6. [6]

    Giorgio Francesco Arcodia et al. 2007. Chinese: A language of compound words. Selected proceedings of the 5th D \'e cembrettes: Morphology in Toulouse , pages 79--90

    work page 2007

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2004. Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments. Proceedings of ACL-WMT, pages 65--72

    2004

  8. [8]

    Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1--34

    work page 2008

Show all 57 references
  1. [9]

    Anouschka Bergmann, KC Hall, and SM Ross. 2007. Language files. In Materials for an Introduction to Language and Linguistics. The Ohio State University Press

    2007

  2. [10]

    Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075--8096

    2020

  3. [11]

    Georgie Botev, Arya D McCarthy, Winston Wu, and David Yarowsky. 2022. Deciphering and characterizing out-of-vocabulary words for morphologically rich languages. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5309--5326

    2022

  4. [12]

    Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit, and Kemal Oflazer. 2014. A human judgement corpus and a metric for arabic mt evaluation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 207--213

    2014

  5. [13]

    Svitlana Budzhak-Jones. 1998. Against word-internal codeswitching: Evidence from ukrainian-english bilingualism. International Journal of Bilingualism, 2(2):161--182

    1998

  6. [14]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116

    2019 arXiv

  7. [15]

    Etienne Denoual and Yves Lepage. 2005. Bleu in characters: towards automatic mt evaluation in languages without word delimiters. In Companion Volume to the Proceedings of Conference including Posters/Demos and tutorial abstracts

    2005

  8. [16]

    Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. 2021. A checklist to combat cognitive biases in crowdsourcing. In Proceedings of the AAAI conference on human computation and crowdsourcing, volume 9, pages 48--59

    2021

  9. [17]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  10. [18]

    Arpad E Elo and Sam Sloan. 1978. The rating of chessplayers: Past and present. (No Title)

    1978

  11. [19]

    Ori Ernst, Ori Shapira, Ido Dagan, and Ran Levy. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.924 Re-examining summarization evaluation across multiple quality criteria . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13829--13838, Sin...

    2023 doi

  12. [20]

    Alexander R Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391--409

    2021

  13. [21]

    Jessica Forde, Ruochen Zhang, Lintang Sutawika, Alham Aji, Samuel Cahyawijaya, Genta Indra Winata, Minghao Wu, Carsten Eickhoff, Stella Biderman, and Ellie Pavlick. 2024. Re-evaluating evaluation for multilingual summarization. In Proceedings of the 2024 Conference on Empirica...

    2024

  14. [22]

    Daniela Gerz, Ivan Vuli \'c , Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. 2018. Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. Transactions of the Association for Computational Linguistics, 6:451--465

    2018

  15. [23]

    Ziwei Gong, Lin Ai, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Zehui Wu, Ahmad Emami, and Julia Hirschberg. 2024. Cream: Comparison-based reference-free elo-ranked automatic evaluation for meeting summarization. arXiv preprint arXiv:2409.10883

    2024 arXiv

  16. [24]

    Rilyn Han, Jiawen Chen, Yixin Liu, and Arman Cohan. 2024. Rethinking efficient multilingual text summarization meta-evaluation. In Findings of the Association for Computational Linguistics ACL 2024, pages 15739--15746

    2024

  17. [25]

    Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. arXiv preprint arXiv:2106.13822

    2021 arXiv

  18. [26]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  19. [27]

    Ozlem Istek and Ilyas Cicekli. 2007. A link grammar for an agglutinative language. Proceedings of Recent Advances in Natural Language Processing (RANLP 2007), pages 285--290

    2007

  20. [28]

    Farayi Kambarami, Scott McLachlan, Bojan Bozic, Kudakwashe Dube, and Herbert Chimhundu. 2021. Computational modeling of agglutinative languages: the challenge for southern bantu languages. Arusha Work. Pap. Afr. Linguist, 3(1):52--81

    2021

  21. [29]

    Marvin Kaster, Wei Zhao, and Steffen Eger. 2021. Global explainability of bert-based evaluation metrics by disentangling along linguistic factors. arXiv preprint arXiv:2110.04399

    2021 arXiv

  22. [30]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1. Minneapolis, Minnesota

    2019

  23. [31]

    Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Evaluating the efficacy of summarization evaluation across languages. arXiv preprint arXiv:2106.01478

    2021 arXiv

  24. [32]

    Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability

    2011

  25. [33]

    Sandeep Kumar and Arun Solanki. 2023. Rouge-ss: A new rouge variant for evaluation of text summarization. Authorea Preprints

    2023

  26. [34]

    Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint arXiv:2304.05613

    2023 arXiv

  27. [35]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \'e , et al. 2023. Bloom: A 176b-parameter open-access multilingual language model

    2023

  28. [36]

    Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology

    1932

  29. [37]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  30. [38]

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668

    2021 arXiv

  31. [39]

    a ubener, Sophie Fellenz, Asja Fischer, Thomas G \

    Laura Manduchi, Kushagra Pandey, Robert Bamler, Ryan Cotterell, Sina D \"a ubener, Sophie Fellenz, Asja Fischer, Thomas G \"a rtner, Matthias Kirchler, Marius Kloft, et al. 2024. On the challenges and opportunities in generative ai. arXiv preprint arXiv:2403.00025

    2024 arXiv

  32. [40]

    Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745

    2018 arXiv

  33. [41]

    Jolaade Okanlawon. 2016. An analysis of the yoruba language with english. Phonetics, Phonology, Morphology and Syntax. NorthEastern University

    2016

  34. [42]

    Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. arXiv preprint arXiv:2104.13346

    2021 arXiv

  35. [43]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  36. [44]

    Tzuf Paz-Argaman, Itai Mondshine, Asaf Achi Mordechai, and Reut Tsarfaty. 2024. Hesum: a novel dataset for abstractive text summarization in hebrew. arXiv preprint arXiv:2406.03897

    2024 arXiv

  37. [45]

    Maja Popovi \'c . 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392--395

    2015

  38. [46]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025

    2020 arXiv

  39. [47]

    Masayoshi Shibatani and Taro Kageyama. 2015. Introduction to the handbooks of japanese language and linguistics. Kubozono, Haruo (Hg.): Handbook of Japanese Phonetics and Phonology. Berlin Ua: De Gruyter, S. Vii--Xxxiii

    2015

  40. [48]

    Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew. arXiv preprint arXiv:2308.16687

    2023 arXiv

  41. [49]

    Otakar Smr z . 2007. Functional arabic morphology. The Prague Bulletin of Mathematical Linguistics, (88):5--30

    2007

  42. [50]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  43. [51]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  44. [52]

    Reut Tsarfaty, Amit Seker, Shoval Sadde, and Stav Klein. 2019. What's wrong with hebrew nlp? and how to make it right. arXiv preprint arXiv:1908.05453

    2019 arXiv

  45. [53]

    AA Vetrov and EA Gorn. 2022. A new approach to calculating bertscore for automatic assessment of translation quality. arXiv preprint arXiv:2203.05598

    2022 arXiv

  46. [54]

    Danqing Wang, Jiaze Chen, Xianze Wu, Hao Zhou, and Lei Li. 2021. Cnewsum: a large-scale chinese news summarization dataset with human-annotated adequacy and deducibility level. arXiv preprint arXiv:2110.10874

    2021 arXiv

  47. [55]

    Oreen Yousuf, Gongbo Tang, and Zeying Jin. 2024. Improving bertscore for machine translation evaluation through contrastive learning. IEEE Access

    2024

  48. [56]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  49. [57]

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622

    2019 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.