A Measure of the System Dependence of Automated Metrics

Jan Deriu; Mark Cieliebak; Pius von D\"aniken

REVIEW 2 major objections 5 minor 28 references

A Measure of the System Dependence of Automated Metrics

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A translation metric can show high segment-level agreement with humans and still rank systems wrongly, because the mapping from metric score to human quality changes from system to system.

desk verdict A genuinely useful measure of metric/system dependence, but the empirical ED/SysDep numbers rest on a NaN-handling choice that could bias the sign pattern; worth careful revision, not a desk reject. read the letter →

arxiv 2412.03152 v2 pith:XSLZGXNF submitted 2024-12-04 cs.CL cs.AI

Pius von Däniken , Jan Deriu , Mark Cieliebak This is my paper · ORCID

classification cs.CLcs.AI

keywords machinetranslationevaluationautomatedmetricssystemdependenceExpectedDeviationSysDepscoremetriccalibrationMQMratingsXCOMET

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a translation metric can pass the standard test, namely high correlation with human judgments at the segment level, and still be unfair when used to rank entire systems. The reason is formalized as a per-system calibration curve: for each system, the expected human rating given a metric score can differ, even when the pooled curve across all systems is monotonic. The authors introduce Expected Deviation, how much the global curve over- or under-predicts a system's human quality, and SysDep, the spread of those deviations across systems, as measures of this system dependence. On WMT23 Chinese-to-English data, XCOMET shows a SysDep of 2.816, with individual deviations ranging from $-0.82$ to $1.996$, enough to invert the order of systems compared with human rankings. The paper concludes that metrics should be evaluated for consistency across systems, not only for correlation.

What carries the argument

The load-bearing object is the system-specific conditional expectation $f_k(m) = \mathbb{E}[h \mid m]$, estimated by bootstrap-averaged isotonic regression on paired human and metric ratings for each system. It converts metric scores to the human scale for one system, and the analogous pooled function $f_G$ converts them on average. From these, Expected Deviation is $ED(k) = \frac{1}{N}\sum_j f_G(m_k^{(j)}) - \frac{1}{N}\sum_j f_k(m_k^{(j)})$, and the SysDep score is the range $\max_k ED(k) - \min_k ED(k)$; the range captures the worst-case pair of systems that could be misordered.

What would settle it

Re-run the SysDep computation on a fresh, larger set of human MQM ratings for the same WMT23 systems, or on held-out halves of the existing ratings: if the expected deviations shrink to near zero, flip sign, or vary as much within a single system as between systems, the paper's claim of systematic system dependence would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a metric's per-system mapping $f_k(m) = \mathbb{E}[h \mid m]$, the expected human rating for a given metric score on system $k$, is not the same across systems, and that this heterogeneity, not noise, explains why metric and human system rankings disagree. Using isotonic regression with bootstrap averaging on WMT23 data, the authors estimate each $f_k$ and a global $f_G$; the difference between remapping system scores through $f_G$ and through $f_k$ defines the Expected Deviation. For XCOMET on zh-en, the deviations span 2.816 points on the MQM scale, which moves NLLB-Greedy from 15th to 12th place and leaves the metric ranking inconsistent with the human ranking despite XCOMET's high global segment-level correlation of 0.65. The authors therefore state that monotonic correlation to humans is insufficient to certify a metric for use in ranking systems.

Load-bearing premise

The per-system calibration curves, fitted from only 1,177 human-rated segments per system, truly reflect how the metric maps to human quality instead of random noise.

Editorial extensions

If this is right

A metric can report a high segment-level correlation with humans and still produce system rankings that contradict human rankings, as XCOMET does on WMT23 zh-en.
SysDep gives a single number for comparing metrics on fairness: on zh-en, GEMBA-MQM has the lowest range among WMT23 metrics while XCOMET's range is 2.816.
System-level evaluation should report remapped scores through the global calibration function, not raw metric averages, because raw averages implicitly assume all per-system curves coincide.
Adding system dependence to standard metric evaluation would change which metrics are considered best: metrics that win on correlation will not necessarily be the most consistent across systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same calibration-curve argument should apply to any automated evaluator that ranks systems by averaged scores, such as summarization or image captioning, so SysDep could become a generic fairness diagnostic for automatic evaluation.
With only about 1,177 human-rated segments per system, the differences between ED values for neighboring systems may be within bootstrap noise; reporting confidence intervals on ED and testing pairwise ED differences would sharpen the paper's central claim.
A direct extension would be to decompose SysDep by score range to identify where a metric is unfair and to use that information to recalibrate per system, although the paper explicitly stops at measurement.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: SysDep is a genuinely useful addition to MT metric evaluation. The formalism is straightforward, the code ships, and the paper demonstrates a real phenomenon—a metric can have high segment-level correlation yet disagree with human system rankings because the human-to-metric mapping differs across systems. That is worth saying clearly.

What is new: the SysDep summary and its application to WMT23. The building blocks are properly attributed to Wu and Resnick (2024) and Posocco and Bonnefoy (2021), and the derivation of Eq. 1 is sound. The definition of ED as the difference between global and per-system remapped means matches the intuition behind system-dependent calibration. The limitations section is honest, and the intra-system control in Appendix D is a good attempt to address noise. Shipping the code in Appendix E helps a lot.

The soft spots are empirical, not conceptual. First, there are no confidence intervals on ED or SysDep; the bootstrap is used only for the f curves in Figure 1. That makes it hard to know whether the 2.8-point SysDep is distinguishable from sampling noise. Second, the causal phrasing in the introduction ("the cause... is due to") overstates what a correlational study can show. Third, and most important, the code evaluates f_G on all NM metric ratings with out_of_bounds='nan', so any score outside the pooled training range is silently dropped from mu_G_k. Because mu_G_k averages over NM while mu_H_k averages over NH, the dropped segments are exactly the unrated ones—and they are not necessarily a random subset. The paper does not report how many scores are dropped per system or check a clipping/extrapolation variant. Without that, the sign pattern of ED (negative for top systems, positive for bottom systems) could in principle be an artifact of dropping extreme scores. I would not call it a proven fatal flaw—the pooled global training set is large, so the dropped fraction may be small—but the authors need to show that.

The intra-system control in Appendix D does not settle this, because it inherits the same code path.

This paper is for MT evaluation researchers and anyone choosing among metrics at the system level. The conceptual contribution should survive; the empirical claims need revision. I would send it to peer review, with the expectation that the authors add sensitivity analyses, report dropped counts, and temper the causal language.

Referee Report

2 major / 5 minor

Summary. The paper proposes a new evaluation dimension for automatic MT metrics: how consistently a metric maps metric scores to human quality across different MT systems. It formalizes the system-specific calibration function f_k and a global function f_G, defines the Expected Deviation ED(k) as the difference between the global predicted mean and the system-specific human mean, and defines SysDep as the range of ED across systems. On WMT23 zh-en data with XCOMET, the paper reports SysDep = 2.816, with top systems underrated and bottom systems overrated, and argues this explains ranking inconsistencies despite high segment-level correlation. Additional results cover other language pairs and metrics, plus an intra-system variability control.

Significance. The core idea is timely and practically relevant: correlation with human judgments alone does not guarantee consistent system-level rankings, and the proposed measure directly quantifies a form of metric bias. The derivation of Eq. (1) is sound, the estimation via isotonic regression with bootstrap is standard, and the code is provided, which supports reproducibility. The intra-system control in Appendix D is a valuable sanity check. However, the empirical support for the central claim is currently weakened by a silent data-dropping mechanism in the implementation and by the absence of uncertainty quantification; these issues are addressable within the scope of the paper.

major comments (2)

[Section 3, Appendix E, Table 1] The code in Appendix E (Listing 1, line 43) sets out_of_bounds='nan' for each isotonic regression model, and Listing 2 computes remapped expectations with np.nanmean. Any metric rating outside the pooled human-rated support is therefore silently discarded when computing mu_G_k. Since mu_G_k is averaged over NM=1976 metric ratings while mu_H_k is averaged over NH=1177 human ratings, and the 799 unrated segments are likely to contain extreme metric scores, the dropped segments are not random: top systems lose high scores and bottom systems lose low scores. This mechanically produces the observed ED pattern of negative values for top systems and positive values for bottom systems, thereby inflating SysDep. The paper does not report the number of dropped segments per system, and the text in Section 3 says the ranking discrepancy is 'attributed to the uncertainty introduced by bootstrapping and extrapolating to the unpaired metric ratings,' although the code does not extrapolate. Please report drop counts and rerun the analysis with clipping or extrapolation to show that the main claim is robust to the choice of handling out-of-range metric scores.
[Section 3, Table 1] ED and SysDep are reported without any uncertainty estimates. Given that each system's f_k is fitted on only 1177 human-rated segments, the ED point estimates have substantial sampling error, and SysDep, being a max-min over 15 noisy EDs, is particularly sensitive to extreme observations. The bootstrap machinery already used for the calibration curves could be extended to produce bootstrap or permutation confidence intervals for the per-system ED values and for SysDep, together with a test of whether the observed between-system SysDep exceeds the intra-system baseline in Appendix D. Without such intervals, the reader cannot tell whether SysDep = 2.816 is statistically distinguishable from zero or from the intra-system control. Please provide these intervals and a formal comparison.

minor comments (5)

[Section 2] The illustrative example with f1(m) = f2(m) + C claims that equal metric means imply 1/N sum f2(m1) > 1/N sum f2(m2). This inequality does not follow from monotonicity alone; it requires an additional condition such as convexity of f2 or a stochastic dominance assumption. Please revise the example to state the needed assumption or provide explicit distributions.
[Section 3] The phrase 'extrapolating to the unpaired metric ratings' is inconsistent with the code, which uses out_of_bounds='nan' and therefore drops rather than extrapolates. Either implement a bounded extrapolation (e.g., constant or linear extension) or reword the sentence to describe dropping.
[Section 3, Table 1] The exact XCOMET variant used in the main experiment is not specified. Appendix C lists XCOMET-Ensemble, XCOMET-XL, and XCOMET-XXL, but Table 1 labels the metric simply as 'XCOMET.' Please identify which checkpoint produced the reported numbers.
[References] The Lommel et al. (2014) reference is malformed: it reads 'Arle. Language Technology Lab) Lommel, Hans. Language Technology Lab) Uszkoreit, and Aljoscha. Language Technology Lab) Burchardt.' and needs to be corrected to a standard author list.
[Section 2, Eq. (2)] Equation (2) uses a single N for both sums, but Section 3 distinguishes NH and NM. Please clarify whether the definition assumes equal sample sizes or whether the estimator should explicitly account for the different numbers of human-rated and metric-rated segments.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: SysDep is a directly computed empirical statistic, and the self-citations are background, not load-bearing evidence.

full rationale

The central quantity SysDep is not a prediction that hides fitted values. Section 2 defines ED(k) = (1/N) Σ f_G(m_j^{(k)}) − (1/N) Σ f_k(m_j^{(k)}) (Equation 2), which the paper identifies with μG_k − μH_k; Section 3 then computes μH_k directly from observed human MQM ratings, μG_k from the isotonic estimate of f_G, and takes SysDep = max_k ED(k) − min_k ED(k) (Equation 3). The isotonic fits are explicit inputs to the measurement, not a parameter fitted to the outcome being predicted. The paper claims only to quantify the defined quantity on WMT23 data, not to predict held-out behavior, and its ranking-based conclusion is supported by the directly observable ranking differences in Table 1 (e.g., Lan-BridgeMT is rank 1 by humans but rank 2 by XCOMET, while NLLB-Greedy is rank 15 by humans but rank 12 by XCOMET). The derivation in Appendix A is a standard iterated-expectation identity, and f_k = E[h|m] is definitional. The self-citations (Deriu et al. 2023; von Däniken et al. 2022, 2024) appear only in Related Work as prior context and as a suggested explanation; they are not invoked as evidence for the new empirical claim, and no uniqueness theorem or load-bearing premise is imported from them. The out_of_bounds='nan' behavior in Appendix E is a legitimate robustness concern about dropped extreme metric scores, but it is a statistical bias criticism, not a circularity: nothing in that code path makes Equation 2 or Equation 3 equivalent to its own output by construction. The Limitations section also explicitly frames the results as an initial empirical study needing larger samples, consistent with the paper treating SysDep as a measured statistic rather than a derived prediction. Overall, the definition-to-measurement chain is self-contained and externally grounded in WMT23 human annotations; no circular step was identified.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central mathematical claim rests on the law of total expectation, which is standard. The empirical claim rests on several domain assumptions about human ratings, sample representativeness, and the reliability of isotonic regression. The fairness interpretation of deviations from a pooled calibration curve is an added assumption, not a theorem.

free parameters (1)

Bootstrap resample count B = 200
Chosen by hand for estimating f_k and confidence intervals in Figure 1; not fitted to data and not part of the SysDep definition itself.

assumptions (5)

standard math Law of total expectation: E[h] = E_m[E[h|m]]
Used in Section 2, Equation 1, and Appendix A to express expected human rating via metric ratings.
domain assumption Human MQM ratings are a valid scalar gold standard for translation quality and higher values mean better quality
Section 2 assumes human ratings can be averaged and used as ground truth for ranking systems.
domain assumption The 1,177 human-rated segments per system are representative of the 1,976 metric-rated segments, so f_k estimated on the former applies to the latter
Section 3 uses NH paired ratings to fit f_k and NM metric ratings to compute remapped scores; out_of_bounds='nan' in Appendix E shows the fit is not extrapolated.
domain assumption Isotonic regression provides a reliable monotonic estimate of the conditional expectation
Section 3 fits all f_k and f_G as bootstrap-averaged isotonic regressions; the paper gives no consistency proof or error analysis.
ad hoc to paper The pooled global curve f_G is the correct reference for judging fairness across systems
The SysDep definition treats deviations from the pooled curve as metric bias; this interpretive choice is asserted in Section 2, not derived.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Measure of the System Dependence of Automated Metrics." pith.science (2026). https://pith.science/paper/XSLZGXNF

@misc{pith2026241203152,
  author       = {Pith},
  title        = {Pith review of: A Measure of the System Dependence of Automated Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSLZGXNF}},
  note         = {Machine review of arXiv:2412.03152}
}

read the original abstract

Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect.

Figures

Figures reproduced from arXiv: 2412.03152 by the authors.

**Figure 1.** Average Human Ratings associated with XCOMET scores on Chinese to English (zh-en) WMT 23 data. We show scores for all system in aggregate (global) and two individual systems. is a monotonic function between metric ratings and human ratings) and whether the system-level ratings can reproduce the same ranking as human ratings (Kocmi et al., 2021; von Däniken et al., 2024). In this paper, we argue that this evaluation … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 9 canonical work pages

[1]

R. E. Barlow and H. D. Brunk. 1972. https://doi.org/10.1080/01621459.1972.10481216 The isotonic regression problem and its dual . Journal of the American Statistical Association, 67(337):140--147

arXiv 1972
[2]

Anja Belz and Eric Kow. 2010. https://aclanthology.org/W10-4201 Comparing rating scales and preference judgements in language evaluation . In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics

work page 2010
[3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

2020
[4]

Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-1060 The price of debiasing automatic metrics in natural language evalaution . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643--653, Melbourne, Australia. Association for Computational Linguistics

doi:10.18653/v1/p18-1060 2018
[5]

Jan Deriu, Pius von D \"a niken, Don Tuggener, and Mark Cieliebak. 2023. https://doi.org/10.18653/v1/2023.findings-acl.404 Correction of errors in preference ratings from automated metrics for text generation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 6456--6474, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.findings-acl.404 2023
[6]

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. https://doi.org/10.18653/v1/2023.wmt-1.51 Results of WMT 23 metrics shared task: Metrics might be guilty but references are not innoc...

doi:10.18653/v1/2023.wmt-1.51 2023
[7]

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.2 Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics are better and more robust . In Proceedings of the Seventh Conference on Machine...

2022
[8]

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ond r ej Bojar. 2021. https://aclanthology.org/2021.wmt-1.73 Results of the WMT 21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain . In Proceedings of the Sixth Conference on Machine Translation, pages 73...

work page 2021

Show all 28 references

[9]

Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F

Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. https://arxiv.org/abs/2310.10482 xcomet: Transparent machine translation evaluation through fine-grained error detection . Preprint, arXiv:2310.10482

2023 arXiv
[10]

Harris, K

Charles R. Harris, K. Jarrod Millman, St \' e fan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jai...

2020
[11]

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. https://arxiv.org/abs/2302.09210 How good are gpt models at machine translation? a comprehensive evaluation . Preprint, arX...

2023 arXiv
[12]

Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.wmt-1.63 M etric X -23: The G oogle submission to the WMT 2023 metrics shared task . In Proceedings of the Eighth Conference on Machin...

2023 doi
[13]

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, To...

2023 doi
[14]

Tom Kocmi and Christian Federmann. 2023. https://doi.org/10.18653/v1/2023.wmt-1.64 GEMBA - MQM : Detecting translation quality error spans with GPT -4 . In Proceedings of the Eighth Conference on Machine Translation, pages 768--775, Singapore. Association for Computational Linguistics

2023 doi
[15]

Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. https://aclanthology.org/2021.wmt-1.57 To ship or not to ship: An extensive evaluation of automatic metrics for machine translation . In Proceedings of the ...

2021
[16]

Language Technology Lab) Lommel, Hans

Arle. Language Technology Lab) Lommel, Hans. Language Technology Lab) Uszkoreit, and Aljoscha. Language Technology Lab) Burchardt. 2014. https://doi.org/10.5565/rev/tradumatica.77 Multidimensional quality metrics (mqm) : a framework for declaring and describing translation qua...

2014 doi
[17]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...

2011
[18]

Nicolas Posocco and Antoine Bonnefoy. 2021. Estimating expected calibration errors. In Artificial Neural Networks and Machine Learning -- ICANN 2021, pages 139--150, Cham. Springer International Publishing

2021
[19]

Brian Thompson and Matt Post. 2020 a . https://doi.org/10.18653/v1/2020.emnlp-main.8 Automatic machine translation evaluation in many languages via zero-shot paraphrasing . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages...

2020 doi
[20]

Brian Thompson and Matt Post. 2020 b . https://aclanthology.org/2020.wmt-1.67 Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity . In Proceedings of the Fifth Conference on Machine Translation, pa...

2020
[21]

Gomez, ukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000–6010, ...

2017
[22]

Pius von D \"a niken, Jan Deriu, Don Tuggener, and Mark Cieliebak. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.108 On the effectiveness of automated metrics for text generation systems . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages ...

2022 doi
[23]

Pius von D \"a niken, Jan Deriu, Don Tuggener, and Mark Cieliebak. 2024. https://doi.org/10.18653/v1/2024.acl-long.243 Favi-score: A measure for favoritism in automated preference ratings for generative AI evaluation . In Proceedings of the 62nd Annual Meeting of the Associati...

2024 doi
[24]

Johnny Wei and Robin Jia. 2021. https://doi.org/10.18653/v1/2021.acl-long.533 The statistical advantage of automatic NLG metrics at the system level . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co...

2021 doi
[25]

Siqi Wu and Paul Resnick. 2024. https://doi.org/10.1609/icwsm.v18i1.31414 Calibrate-extrapolate: Rethinking prevalence estimation with black box classifiers . Proceedings of the International AAAI Conference on Web and Social Media, 18(1):1634--1647

2024 doi
[26]

Yangjian Wu and Gang Hu. 2023. https://doi.org/10.18653/v1/2023.wmt-1.15 Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings . In Proceedings of the Eighth Conference on Machine Translation, pages 166--169, Singap...

2023 doi
[27]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[28]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

R. E. Barlow and H. D. Brunk. 1972. https://doi.org/10.1080/01621459.1972.10481216 The isotonic regression problem and its dual . Journal of the American Statistical Association, 67(337):140--147

arXiv 1972

[2] [2]

Anja Belz and Eric Kow. 2010. https://aclanthology.org/W10-4201 Comparing rating scales and preference judgements in language evaluation . In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics

work page 2010

[3] [3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

2020

[4] [4]

Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-1060 The price of debiasing automatic metrics in natural language evalaution . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643--653, Melbourne, Australia. Association for Computational Linguistics

doi:10.18653/v1/p18-1060 2018

[5] [5]

Jan Deriu, Pius von D \"a niken, Don Tuggener, and Mark Cieliebak. 2023. https://doi.org/10.18653/v1/2023.findings-acl.404 Correction of errors in preference ratings from automated metrics for text generation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 6456--6474, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.findings-acl.404 2023

[6] [6]

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. https://doi.org/10.18653/v1/2023.wmt-1.51 Results of WMT 23 metrics shared task: Metrics might be guilty but references are not innoc...

doi:10.18653/v1/2023.wmt-1.51 2023

[7] [7]

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.2 Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics are better and more robust . In Proceedings of the Seventh Conference on Machine...

2022

[8] [8]

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ond r ej Bojar. 2021. https://aclanthology.org/2021.wmt-1.73 Results of the WMT 21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain . In Proceedings of the Sixth Conference on Machine Translation, pages 73...

work page 2021