Pith. sign in

REVIEW 4 major objections 4 minor 62 references

Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For translation and summarization in six Indian languages, LLM-as-judge metrics—especially DeepSeek-V3—agree with human ratings more strongly than lexical, embedding, or learned neural metrics.

desk verdict Useful benchmark; headline ranking of DeepSeek-V3 is plausible but statistically fragile. read the letter →

arxiv 2510.07061 v2 pith:OESFPGXV submitted 2025-10-08 cs.CL

Amir Hossein Yari , Kalmit Kulkarni , Ahmad Raza Khan , Fajri Koto This is my paper
classification cs.CL
keywords IndianlanguagesmachinetranslationevaluationsummarizationLLM-as-judgemetricreliabilityhumanjudgmentsoutliersensitivityrobustnesstoperturbations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ITEM, a benchmark for testing how well automatic evaluation metrics match human judgments in machine translation and text summarization across six major Indian languages. It claims that LLM-based evaluators, with DeepSeek-V3 the strongest, align most closely with human ratings at the segment level, outperforming traditional lexical and embedding metrics and even earlier learned neural metrics. It also claims that the reliability of metrics depends heavily on the quality dimension being scored: summarization metrics track content fidelity well but coherence poorly, while translation metrics track fluency well but adequacy less reliably. A reader should care because these languages are used by over a billion people, yet existing metrics were almost entirely validated on English and a few high-resource languages; if the paper is right, the practical choice of metric for evaluating Indian-language systems changes markedly.

What carries the argument

The key machinery is ITEM, a benchmark built from sampled article-summary and sentence-translation pairs for Hindi, Bengali, Tamil, Telugu, Marathi, and Gujarati, scored by two native speakers per language on a five-point scale across two or four fine-grained dimensions. The benchmark then computes Pearson correlations between human scores and a large set of 26 automatic metrics—lexical n-gram metrics, embedding-based metrics, learned neural metrics, and LLM-as-judge prompts—with outlier removal via a median-based robust z-score, and a controlled perturbation suite that tests paraphrasing sensitivity, entity-level noise, negation, synonym/antonym replacement, shuffling, and content truncatio

What would settle it

Take the same ITEM samples and recompute segment-level correlations after replacing the two-annotator averages with adjudicated scores from five independent annotators per item (or after switching from Pearson to Spearman or Kendall correlations): if DeepSeek-V3 no longer ranks at the top for both summarization and translation, then the paper's central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that, within ITEM, the ordering of automatic metrics by correlation with human judgments is not the same as in English-centric studies, and LLM judges dominate at the fine-grained segment level. Specifically, DeepSeek-V3 achieved the highest correlations for both summarization and translation across all six languages and quality dimensions, and native-language prompts helped for translation while English prompts often helped for summarization. The paper also establishes that outliers strongly distort metric-human correlations, that summarization metrics capture content-preservation dimensions better than discourse-level coherence, that translation metrics capture fluency

Load-bearing premise

The entire ranking of metrics rests on human scores, averaged from only two annotators per item, being trustworthy ground truth—yet those annotators agree only modestly on some quality dimensions, so the reported metric-human correlations inherit that noise.

Editorial extensions

If this is right

  • Practitioners evaluating translation or summarization systems in these six languages should prefer LLM-based judges, especially DeepSeek-V3, over BLEU, ROUGE, and embedding metrics when they need fine-grained segment-level quality signals.
  • Metric choice must be task- and dimension-aware: in summarization, metrics are more trustworthy for faithfulness and coverage than for coherence; in translation, they are more trustworthy for fluency than for adequacy.
  • Outlier handling should become standard in metric evaluations, since outlier removal changes correlations substantially for some metrics (e.g., LASER improves by tens of percent for some dimensions) while leaving others unchanged.
  • At the system level, most metrics correlate near-perfectly with human rankings, so system-level recommendations are less affected, but segment-level differences are large enough to change which metric a practitioner should deploy.
  • Robustness is not uniform across languages: Hindi is the most vulnerable and Gujarati the most resilient to perturbations, and negation affects Tamil and Telugu much more strongly than the other four languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's rankings may be more fragile than they appear because human scores are the average of only two annotators, and inter-annotator correlations for some dimensions (coherence in Bengali at 0.4, coverage in Hindi at 0.48) suggest substantial noise; using five or more adjudicated judgments per item could shift the apparent metric ordering.
  • LLM judges correlate highly with each other (r≈0.7–0.9) but only moderately with traditional metrics, which could indicate a shared systematic bias in LLM judgment rather than superior access to an independent human truth; a useful extension would compare LLM judges against a third, qualitatively different annotation protocol.
  • The perturbation suite—entity edits, negation, shuffling, off-topic insertion, and key-sentence truncation—could be reused as a standard stress test for any new metric claiming Indian-language support, and a metric's resilience across these perturbations could be reported as a single robustness score.
  • Because ITEM uses only three generated outputs per source (two large LLMs plus one or two open Indian-language models), a natural extension is to expand the candidate-output pool, which would let the metric rankings be tested across a wider range of quality and prevent overfitting to the outputs of particular systems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ITEM, a public benchmark for evaluating automatic MT and summarization metrics against human fine-grained judgments in six Indian languages. It samples 150 source–output pairs per task/language from XLSum and FLORES-200, generates outputs with three models, and collects two native-speaker ratings per sample across four TS aspects (faithfulness, focus, coverage, coherence) and two MT aspects (adequacy, fluency). Twenty-six metrics (including many ROUGE, BERTScore, and BLEU configurations) are correlated at the segment level, with additional analyses of outliers, language-specific reliability, inter-metric correlation, system-level ranking, and robustness under controlled perturbations. The central claim is that LLM-based evaluators—DeepSeek-V3 in particular—show the strongest alignment with human judgments, with DeepSeek-V3 designated 'the most reliable metric overall'.

Significance. If the central result holds, ITEM would be a valuable, reusable multilingual evaluation resource: it covers six Indian languages, includes fine-grained quality dimensions, ships a public dataset, and examines outlier sensitivity and perturbation robustness more broadly than prior Indic-language meta-evaluations. The paper also presents a useful comparison of lexical, embedding-based, neural-learned, and LLM-based metrics. However, the headline ranking claim is currently statistically unsupported: optimal configurations are selected in-sample, no confidence intervals accompany the correlations, and the human gold standard has non-negligible annotator noise. These issues are fixable and do not undermine the value of the benchmark itself, but they must be addressed before the 'dominance' finding can be accepted.

major comments (4)
  1. [§4.3.1, Table 6] The 'optimal configurations' for ROUGE, BERTScore, and BLEU are selected from the same segment-level correlation table in which they are then reported. This is an in-sample argmax over 48 ROUGE, 144 BERTScore, and 4 BLEU configurations, so the Table 6 rows for these metrics reflect positive selection bias. LLM-based metrics are not selected by the same procedure, making the comparison between, e.g., ROUGE-2-N-R (overall 0.227) and DeepSeek-V3 (0.29) unfair as a comparison of representative metric families. Please select configurations on a held-out split, or report all configurations and correct for multiple comparisons, or explicitly state that the reported row is the best of many on the test set.
  2. [§4.3.1, Table 6] The central claim that DeepSeek-V3 is 'the most reliable metric overall' rests on very small differences: DeepSeek-V3 overall r = 0.29 vs. GPT-4.1 (English prompt) 0.275, Gemini 2.5 Flash 0.274, and COMET 0.253. The paper reports no standard errors, bootstrap confidence intervals, or significance tests for any row. Given that human scores are the mean of two annotators and inter-annotator correlations are as low as 0.40 in Table 2, gold-standard noise alone could shift correlations by several hundredths. The gaps among top metrics are within a plausible noise range. Please provide bootstrap CIs (ideally resampling over both annotators and segments) and a significance test for the top-metric ordering.
  3. [§3.4, Table 2] Human gold-standard reliability is not propagated into the metric rankings. Table 2 reports post-refinement inter-annotator Pearson correlations as low as 0.40 (Bengali Coherence), 0.48 (Hindi Coverage), and 0.53 (Hindi Faithfulness). The paper filters samples with high disagreement but does not quantify residual noise in the averaged human scores or propagate it into Table 6. This is load-bearing: the entire paper ranks metrics against these scores. Please report per-aspect reliability (e.g., ICC or Krippendorff's alpha) and perform a sensitivity analysis using alternative human-score aggregates (single annotator, median, QC-reference scores) to show that the metric ordering is stable.
  4. [Appendix C.1, Table 7] The system-level claim is underpowered. Only three model outputs per task are ranked, so most entries in Table 7 are exactly 1.00 or -0.33. With three systems, Kendall's tau cannot meaningfully separate metric families, and the statement that most metrics achieve 'perfect correlations' at the system level is largely an artifact of the tiny system set. Please either add more systems or substantially temper the system-level conclusion.
minor comments (4)
  1. [Abstract] The abstract says '29 automatic metrics' while the full text consistently says '26 automatic metrics' (e.g., §1 contributions and Table 6). This inconsistency should be fixed.
  2. [§4.1 / Table 6] The configuration label 'ROUGE-2-N-R' is not defined in the table or text. Since 'N' and 'R' are explained only in prose, add a footnote or table note explaining that 'N' stands for no-stopwords and 'R' for recall.
  3. [§4.4.1] The paraphrasing section describes prompt design and validation but, in the main text, does not report metric sensitivity to the validated paraphrases; Figure 6 only shows counts. Please state explicitly where the paraphrase results appear (e.g., Figures 7–8) or add a dedicated analysis.
  4. [Table 6] Several rows have formatting inconsistencies such as '0.258 /0.258' (missing space) and inconsistent alignment. Please normalize the table formatting.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central metric–human correlations are measured against externally collected human judgments; the only self-citations are incidental and non-load-bearing.

full rationale

The paper's central claim is an empirical ranking: segment-level Pearson correlations (Table 6) between 26 established automatic metrics and human scores, where human scores are defined as the average of two annotators (§3.4) and the human annotations were collected for this benchmark. There is no equation in the paper that derives a predicted correlation from a fitted parameter, and no prediction is reported as if independent of the data used to make it. The candidate self-reference is the adoption of the four TS evaluation aspects from Koto et al. (2022), a published framework by a co-author; this supplies the annotation rubric but does not by construction force the reported outcome that LLM-based evaluators correlate highest. The in-sample selection of the best ROUGE/BERTScore/BLEU configuration (§4.3.1) is a selection-bias concern, not a circularity: the selected configurations are then reported on the same benchmark, but the main LLM-vs-other-metric comparison is not an artifact of that selection. The low inter-annotator correlations in Table 2 are a validity limitation of the gold standard, not a circular step. No load-bearing step reduces to its own input, so the score is at the low end.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The empirical rankings rest on two-annotator human scores and on the assumption that reported metric configurations are not cherry-picked on the same data. Several configuration choices are in-sample, and the annotator-disagreement filter threshold is unspecified.

free parameters (4)
  • ROUGE configuration (ROUGE-2, recall, stopwords removed) = ROUGE-2-N-R
    Selected as the best of 48 ROUGE variants using the same segment-level correlations reported in Table 6 (§4.3.1); in-sample selection can inflate reported performance.
  • BERTScore configuration (IndicBERT, layer 8, recall) = IndicBERT layer 8 recall
    Selected as the best of 144 BERTScore configurations on the same test data (§4.3.1); this is post-hoc selection rather than a held-out choice.
  • BLEU n-gram order = BLEU-3
    Selected as the best among BLEU 1-4 based on the same data (§4.3.1).
  • Annotator-disagreement filtering cutoff = unspecified
    Samples with 'high annotator disagreement' were filtered out (§3.4), reducing the dataset from 5,400 to 5,175 samples, but the exact threshold is not reported and could affect correlations.
assumptions (4)
  • domain assumption Human scores are the average of two annotators per language and are treated as reliable gold standard.
    §3.3-3.4: QC checks against a 5-expert subset are reported, but inter-annotator correlations in Table 2 are as low as 0.48, so the ground-truth signal is noisy.
  • domain assumption The evaluation aspects (adequacy, fluency, faithfulness, focus, coverage, coherence) are valid, separable dimensions of MT/TS quality.
    Adopted from Koehn & Monz 2006 and Koto et al. 2022 (§3.2); the paper does not independently validate their separability for these Indian languages.
  • domain assumption Pearson correlation on segment-level scores, with MAD-based outlier removal, is an appropriate measure of metric-human alignment.
    Used throughout §4.3; Pearson is sensitive to distribution shape, and the reported correlations lack confidence intervals.
  • domain assumption The three generation models (Cohere Command R+, GPT-4o mini, AI4Bharat models) provide a sufficient diversity of output quality for benchmarking.
    §3.1: only these three model families generate the evaluated outputs; system-level conclusions may not generalize to other systems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages." pith.science (2026). https://pith.science/paper/OESFPGXV

@misc{pith2026251007061,
  author       = {Pith},
  title        = {Pith review of: Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OESFPGXV}},
  note         = {Machine review of arXiv:2510.07061}
}
read the original abstract

While automatic metrics drive progress in Machine Translation (MT) and Text Summarization (TS), existing metrics have been developed and validated almost exclusively for English and other high-resource languages. This narrow focus leaves Indian languages, spoken by over 1.5 billion people, largely overlooked, casting doubt on the universality of current evaluation practices. To address this gap, we introduce ITEM, a large-scale benchmark that systematically evaluates the alignment of 29 automatic metrics with human judgments across six major Indian languages, enriched with fine-grained annotations. Our extensive evaluation, covering agreement with human judgments, sensitivity to outliers, language-specific reliability, inter-metric correlations, and resilience to controlled perturbations reveals four central findings: (1) LLM-based evaluators show the strongest alignment with human judgments at both segment and system levels; (2) outliers exert a significant impact on metric-human agreement; (3) In TS, metrics are more effective at capturing content fidelity, whereas in MT, they better reflect fluency; and (4) Metrics differ in their robustness and sensitivity when subjected to diverse perturbations. Collectively, these findings offer critical guidance for advancing metric design and evaluation in Indian languages.

Figures

Figures reproduced from arXiv: 2510.07061 by the authors.

Figure 1
Figure 1. End-to-End process of dataset creation. Language Summarization Translation Fa. Fo. Cov. Coh. Avg Ad. Fl. Avg Hindi 0.53 0.53 0.48 0.55 0.74 0.54 0.63 0.61 Bengali 0.8 0.82 0.84 0.4 0.93 0.54 0.47 0.64 Tamil 0.45 0.57 0.68 0.71 0.81 0.65 0.61 0.72 Telugu 0.74 0.69 0.72 0.73 0.89 0.78 0.76 0.87 Marathi 0.58 0.57 0.55 0.58 0.79 0.59 0.49 0.44 Gujarati 0.85 0.83 0.86 0.65 0.91 0.75 0.75 0.77 [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figure 2
Figure 2. Distribution of human evaluation scores across evaluation aspects. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Pearson correlation network of human evalua [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Language-specific Pearson correlations of top [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Pearson correlation matrices of automatic metrics. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Distribution of high-quality paraphrased sum [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Relative COMET score drops (%) across languages under diverse perturbations (top-right: TS; bottom-left: [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Relative Pearson correlation drops (%) across evaluation aspects under perturbations [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: represents the relationship between sum￾mary compression (ratio of article tokens to human summary tokens) and average human evaluation scores. Individual points represent observations, the red line shows the linear regression, and the shaded region indicates the 95% c…
Figure 10
Figure 10. Figure 10: shows the relative changes in Pearson correlation between human and COMET scores under different types of entity noise. In MT, en￾tity perturbations substantially reduce correlations, with entity removal causing the steepest declines in both Adequacy and Fluency. In c…
Figure 11
Figure 11. Figure 11: Relative ROUGE score drops (%) across languages under diverse perturbations (top-right: TS; bottom [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Relative Pearson correlation drops (%) across evaluation aspects under perturbations [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Relative BERTScore score drops (%) across languages under diverse perturbations (top-right: TS; [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Relative Pearson correlation drops (%) across evaluation aspects under perturbations [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 10 canonical work pages

  1. [1]

    Sweta Agrawal, Ant \'o nio Farinhas, Ricardo Rei, and Andre Martins. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.802 Can automatic metrics assess high-quality translations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14491--14502, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.802 2024

  2. [2]

    Duarte Alves, Ricardo Rei, Ana C Farinha, Jos \'e G. C. de Souza, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.43/ Robust MT evaluation with sentence-level multilingual augmentation . In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 469--478, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com...

    2022

  3. [3]

    Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the association for computational linguistics, 7:597--610

    2019

  4. [4]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

    2005

  5. [5]

    Vic Barnett, Toby Lewis, and 1 others. 1994. Outliers in statistical data, volume 3. Wiley New York

    1994

  6. [6]

    Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the ninth workshop on statistical machine translation, pages 362--367

    2014

  7. [7]

    Xiaoyu Chen, Daimeng Wei, Hengchao Shang, Zongyao Li, Zhanglin Wu, Zhengzhe Yu, Ting Zhu, Mengli Zhu, Ning Xie, Lizhi Lei, Shimin Tao, Hao Yang, and Ying Qin. 2022. https://aclanthology.org/2022.wmt-1.46/ Exploring robustness of machine translation metrics: A study of twenty-two automatic metrics in the WMT 22 metric task . In Proceedings of the Seventh C...

    2022

  8. [8]

    Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur Parikh. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.584 SEAHORSE : A multilingual, multifaceted dataset for summarization evaluation . In Proceedings of the 2023 Conference on Empirical Methods...

    work page doi:10.18653/v1/2023.emnlp-main.584 2023

Show all 62 references
  1. [9]

    Cohere Labs . 2024. https://doi.org/10.57967/hf/3135 c4ai-command-r-plus-08-2024

    2024 doi

  2. [10]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jian...

    2025 arXiv

  3. [11]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  4. [12]

    Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, and Pratyush Kumar. 2022. https://doi.org/10.18653/v1/2022.findings-acl.145 I ndic BART : A pre-trained model for indic natural language generation . In Findings of the Association for Computati...

    2022 doi

  5. [13]

    Debtanu Datta, Shubham Soni, Rajdeep Mukherjee, and Saptarshi Ghosh. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.321 MILDS um: A novel benchmark dataset for multilingual summarization of I ndian legal case judgments . In Proceedings of the 2023 Conference on Empirical Me...

    2023 doi

  6. [14]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. https://arxiv.org/abs/24...

    2025 arXiv

  7. [15]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  8. [16]

    George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138--145

    2002

  9. [17]

    Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://arxiv.org/abs/2007.12626 Summeval: Re-evaluating summarization evaluation . Preprint, arXiv:2007.12626

    2021 arXiv

  10. [18]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.62 Language-agnostic BERT sentence embedding . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2022 doi

  11. [19]

    Gabriel Forgues, Joelle Pineau, Jean-Marie Larchev \^e que, and R \'e al Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2, page 168

    2014

  12. [20]

    Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. https://doi.org/10.18653/v1/2024.wmt-1.2 Ar...

    2024 doi

  13. [21]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. https://doi.org/10.18653/v1/2024.naacl-long.365 GPTS core: Evaluate as you desire . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2024 doi

  14. [22]

    Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M

    Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. https://arxiv.org/abs/2305.16307 Indictrans2...

    2023 arXiv

  15. [23]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  16. [24]

    Yue Guo, Tal August, Gondy Leroy, Trevor Cohen, and Lucy Lu Wang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.519 APPLS : Evaluating evaluation metrics for plain language summarization . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proce...

    2024 doi

  17. [25]

    Aaron L. F. Han, Derek F. Wong, and Lidia S. Chao. 2012. https://aclanthology.org/C12-2044/ LEPOR : A robust evaluation metric for machine translation with augmented factors . In Proceedings of COLING 2012: Posters , pages 441--450, Mumbai, India. The COLING 2012 Organizing Committee

    2012

  18. [26]

    Rilyn Han, Jiawen Chen, Yixin Liu, and Arman Cohan. 2024. https://doi.org/10.18653/v1/2024.findings-acl.930 Rethinking efficient multilingual text summarization meta-evaluation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 15739--15746, Bangko...

    2024 doi

  19. [27]

    Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M

    Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. https://doi.org/10.18653/v1/2021.findings-acl.413 XL -sum: Large-scale multilingual abstractive summarization for 44 languages . In Fin...

    2021 doi

  20. [28]

    Yichen Huang and Timothy Baldwin. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.340 Robustness tests for automatic machine translation metrics with adversarial attacks . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5126--5135, Singapo...

    2023 doi

  21. [29]

    Boris Iglewicz and David C Hoaglin. 1993. Volume 16: how to detect and handle outliers. Quality Press

    1993

  22. [30]

    Khapra, and Pratyush Kumar

    Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.445 I ndic NLPS uite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual langua...

    2020 doi

  23. [31]

    Philipp Koehn and Christof Monz. 2006. https://aclanthology.org/W06-3114/ Manual and automatic evaluation of machine translation between E uropean languages . In Proceedings on the Workshop on Statistical Machine Translation, pages 102--121, New York City. Association for Comp...

    2006

  24. [32]

    Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022. FFCI : A framework for interpretable automatic evaluation of summarization. Journal of Artificial Intelligence Research, 73:1553--1607

    2022

  25. [33]

    Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. https://doi.org/10.18653/v1/2021.findings-acl.71 Evaluating the efficacy of summarization evaluation across languages . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 801--812, Online. As...

    2021 doi

  26. [34]

    Raghvendra Kumar, Mohammed Salman S A, Aryan Sahu, Tridib Nandi, Pragathi Y P, Sriparna Saha, and Jose G Moreno. 2025. https://doi.org/10.18653/v1/2025.acl-long.427 COSMMIC : Comment-sensitive multimodal multilingual I ndian corpus for summarization and headline generation . I...

    2025 doi

  27. [35]

    Thomas K Landauer and Susan T Dumais. 1997. A solution to plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211

    1997

  28. [36]

    Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of experimental social psychology, 49(4):764--766

    2013

  29. [37]

    Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. 2025. Language ranker: A metric for quantifying LLM performance across high and low-resource languages. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28186--28194

    2025

  30. [38]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  31. [39]

    Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. https://doi.org/10.18653/v1/2020.acl-main.448 Tangled up in BLEU : Reevaluating the evaluation of automatic machine translation evaluation metrics . In Proceedings of the 58th Annual Meeting of the Association for Computat...

    2020 doi

  32. [40]

    Khapra, Pratyush Kumar, Rudra Murthy, and Anoop Kunchukuttan

    Arnav Mhaske, Harshit Kedia, Sumanth Doddapaneni, Mitesh M. Khapra, Pratyush Kumar, Rudra Murthy, and Anoop Kunchukuttan. 2023. https://doi.org/10.18653/v1/2023.acl-long.582 Naamapadam: A large-scale named entity annotated data for I ndic languages . In Proceedings of the 61st...

    2023 doi

  33. [41]

    Andrew Mutton, Mark Dras, Stephen Wan, and Robert Dale. 2007. https://aclanthology.org/P07-1044/ GLEU : Automatic evaluation of sentence-level fluency . In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 344--351, Prague, Czech Rep...

    2007

  34. [42]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024....

    2024 arXiv

  35. [43]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  36. [44]

    Kevin Patel, Suraj Agrawal, and Ayush Kumar. 2024. https://doi.org/10.18653/v1/2024.trustnlp-1.14 Tweak to trust: Assessing the reliability of summarization metrics in contact centers via perturbed summaries . In Proceedings of the 4th Workshop on Trustworthy Natural Language ...

    2024 doi

  37. [45]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. Association for Computational Linguistics

    2015 doi

  38. [46]

    Maja Popovi \'c . 2017. https://doi.org/10.18653/v1/W17-4770 chr F ++: words helping character n-grams . In Proceedings of the Second Conference on Machine Translation, pages 612--618, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  39. [47]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  40. [48]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

    2020 doi

  41. [49]

    Vasile Rus and Mihai Lintean. 2012. https://aclanthology.org/W12-2018/ A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics . In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP ...

    2012

  42. [50]

    Khapra, and Raj Dabre

    Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, and Raj Dabre. 2023. https://doi.org/10.18653/v1/2023.acl-long.795 I ndic MT eval: A dataset to meta-evaluate machine translation metrics for I ndian languages . In Proceedings ...

    2023 doi

  43. [51]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  44. [52]

    Anushka Singh, Ananya Sai, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh Khapra. 2024. https://doi.org/10.18653/v1/2024.acl-short.58 How good is zero-shot MT evaluation for low resource I ndian languages? In Proceedings of the 62nd Annual Meeting of the Associat...

    2024 doi

  45. [53]

    Juhi Tandon and Dipti Misra Sharma. 2017. https://aclanthology.org/W17-6529/ Unity in diversity: A unified parsing strategy for major I ndian languages . In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 255--265, Pisa, Italy...

    2017

  46. [54]

    NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...

    2022 arXiv

  47. [55]

    Ashok Urlana, Pinzhen Chen, Zheng Zhao, Shay Cohen, Manish Shrivastava, and Barry Haddow. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.777 PMI ndia S um: Multilingual and cross-lingual headline summarization for languages in I ndia . In Findings of the Association for...

    2023 doi

  48. [56]

    Guojun Wu, Shay B Cohen, and Rico Sennrich. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.169 Evaluating automatic metrics with incremental machine translation systems . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2994--3005, Miami, ...

    2024 doi

  49. [57]

    Amir Hossein Yari and Fajri Koto. 2025. https://doi.org/10.18653/v1/2025.acl-long.987 Unveiling cultural blind spots: Analyzing the limitations of m LLM s in procedural text comprehension . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguist...

    2025 doi

  50. [58]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://arxiv.org/abs/2106.11520 Bartscore: Evaluating generated text as text generation . Preprint, arXiv:2106.11520

    2021 arXiv

  51. [59]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  52. [60]

    Meyer, and Steffen Eger

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. https://doi.org/10.18653/v1/D19-1053 M over S core: Text generation evaluating with contextualized embeddings and earth mover distance . In Proceedings of the 2019 Conference on Empirical ...

    2019 doi

  53. [61]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  54. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.