Pith. sign in

REVIEW 4 major objections 5 minor 37 references

Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Segment-level translation quality estimation is structurally unable to serve as a standalone gate for production translation workflows, and a 104,762-segment empirical test confirms it.

desk verdict A valuable, hard-nosed empirical case against TAUS EPIC QE as a triage gate, but the paper's class-level conclusion outruns the single-model evidence. read the letter →

arxiv 2608.03577 v1 pith:X4UG2NB3 submitted 2026-08-04 cs.CL

Serge Gladkoff , Angelika Vaasa , Sue Ellen Wright , Ingemar Strandvik , Lifeng Han This is my paper · ORCID
classification cs.CL
keywords translationqualityestimationsegment-levelQEMQMmachineevaluationproductionworkflowscorecalibrationdistributionshifthuman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that automatic translation quality estimation (QE) at the level of isolated segments is structurally unable to serve as a reliable standalone instrument in real translation production. It combines a theoretical critique—translation quality depends on context that a segment-level score cannot see—with a large empirical study of 104,762 MQM-annotated segments scored by a commercial QE model. The results show weak-to-moderate ranking at best, calibration consistently worse than a constant base-rate predictor, and threshold behavior that either flags most of the file or misses many errors. The paper concludes that segment-level QE scores should not be used for routing, release, or review bypass, and that future work should focus on automating human evaluation grounded in MQM.

What carries the argument

The load-bearing distinction is between ranking and triage: a ranker tells a reviewer which segments to inspect first, while a triage gate decides which segments can safely bypass review. The paper formalizes translation quality as $Q(s,t,c)$, where $s$ is the source segment, $t$ the target translation, and $c$ the contextual parameters that are unrecoverable from the segment alone, and it shows that a score can be a weak ranker while failing as a triage gate. Usability is therefore not a property of the model alone but of the model, corpus, language pair, error base rate, threshold, and cost structure. The argument is operationalized with a strict screen (review at most 50 percent of a file, catch at least 80 percent of errors, make the flagged set at least 1.5 times richer in errors than random) and with calibration measured by the Brier score relative to a no-skill base-rate predictor.

What would settle it

Run the paper's own operational screen—review at most half the file, catch at least 80 percent of MQM-annotated errors, and make the flagged set at least 1.5 times richer in errors than random selection—with a different sentence-level quality-scoring system on held-out production data across several domains and language pairs; if any system passes consistently and is better calibrated than a constant base-rate predictor, the paper's categorical conclusion is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quality of a translated segment is not a property of the segment alone: it is a function of source, target, and a context set $c$ that includes discourse, terminology, register, and communicative intent. Because a segment-level QE model must score without $c$, it operates under incomplete information, and its output is at best a corpus-dependent proxy for error risk rather than a calibrated measure of quality. The empirical section tests this against 104,762 MQM-annotated segments spanning very poor machine output, unedited output from a strong engine, and clean human-edited text. The commercial model tested shows weak-to-moderate ranking (median AUC 0.591 on clean text, 0.631 on unedited output), calibration worse than a no-skill base-rate predictor in every slice, and no threshold that simultaneously reviews at most half the file, catches 80 percent of errors, and makes the flagged set at least 1.5 times richer in errors than random selection. The paper therefore concludes that segment-level QE scores should not be used as a standalone basis for routing, release, or review bypass, and that future work should focus on automating human evaluation grounded in MQM (Multidimensional Quality Metrics), a structured human error-annotation standard.

Load-bearing premise

The empirical case is built on one commercial sentence-scoring system, and the paper itself notes that major-error thresholds were unstable in its cleanest corpus because such errors were sparse; since the conclusion is stated for the whole class of segment-level QE systems, the blanket warning would collapse if that one system is unrepresentative, leaving the theoretical context-dependence argument as the only support.

Editorial extensions

If this is right

  • Production workflows that currently let segment-level QE scores decide which translations can ship without human review should treat that practice as unsupported; at most the scores can order a review queue after local validation on the same domain, language pair, MT source, and error-rate regime.
  • No single QE threshold is portable: the same model and language pair can behave differently across corpora (the paper reports an EC–EP AUC correlation of only 0.379), so any deployment must re-validate on its own material.
  • Because the score is not calibrated (the Brier score was worse than a constant base-rate predictor in every slice), QE outputs should not be read as probabilities of error or as absolute quality scores, even when they carry significant rank correlation.
  • The realistic use of segment-level QE is audit enrichment or prioritization, not autonomous triage: under a screen requiring review of at most 50 percent of a file while catching at least 80 percent of errors, no threshold scenario in the study passed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the structural argument is right, more data or larger models cannot fix segment-level QE by themselves, because the missing contextual variables are absent from the input; this redirects attention to document-level or context-aware evaluation, a step the paper gestures toward but does not itself test.
  • The same ranking-versus-triage distinction may apply to other sentence-level quality predictors, such as machine-generated summaries or code snippets, where correctness depends on intent outside the unit being scored; one could test it by varying surrounding context while holding the unit fixed and checking whether any fixed-threshold gate stays safe.
  • The paper's 'Tiene razón' example suggests a concrete falsification-style benchmark: assemble identical source–target segment pairs placed in two contexts that force different verdicts, and show whether any segment-level QE model can avoid assigning both the same score; a model that cannot is structurally blind to the decisive variable.
  • In low-error regimes, even a good ranker yields very low precision (for example, AUC near 0.69 with roughly 10 percent precision at peak F1), so any useful deployment would need a separate base-rate estimate; the paper's analysis implies this but leaves the design of such an estimator to future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper challenges the use of segment-level automatic translation quality estimation (QE) as a standalone decision instrument in production workflows. It offers a theoretical argument that translation correctness depends on extra-segmental context, c, which is not recoverable from an isolated segment pair (Q(s,t,c)), and it reviews literature on generalization failure, annotation misalignment, systematic biases, and error-detection limitations. The original empirical component evaluates TAUS EPIC QE against MQM annotations on three corpora (DFKI, EP, EC; 104,762 scored segments) and reports that the score has weak-to-moderate ranking power, poor calibration, threshold instability, and low cross-corpus portability. The paper concludes that segment-level QE scores should not be used as a standalone basis for routing, release, or review bypass.

Significance. If the strong conclusion were established, the paper would have immediate practical consequences for translation production systems. The study makes a useful and often-missed distinction between ranking and triage, introduces an explicit operational screen (flag at most 50% of segments, recall at least 80% of errors, flagged set at least 1.5x enriched), and reports results on a large real-world dataset spanning three very different quality regimes. The empirical analysis is internally coherent and the stated interim conclusions at the end of Section 5 are appropriately cautious. However, the paper's abstract and Section 6 draw a categorical conclusion about 'segment-level QE scores' from evidence on a single commercial model, and the threshold screen is applied to any-error labels rather than to the severe-error condition that matters for production safety. These gaps mean the manuscript currently supports a qualified claim about TAUS EPIC QE, not the class-level ban announced in the abstract.

major comments (4)
  1. [§5 (interim conclusions) and §6 vs. §5 data] The empirical evaluation tests one commercial model, TAUS EPIC QE, on three corpora. The interim conclusions in §5 are explicitly restricted to TAUS EPIC QE, yet the abstract and the first paragraph of §6 generalize to 'segment-level QE scores' as a class. A single-model study cannot license a categorical ban on all segment-level QE systems unless the model is shown to be representative or the theoretical argument is shown to rule out any safe operating point for any model. The theoretical argument in §2 demonstrates that segment-level QE cannot be perfectly correct because context is missing, but it does not by itself rule out a calibrated, locally validated threshold with adequate severe-error recall. Please either re-scope the title and conclusion to 'TAUS EPIC QE' or add a representativeness argument.
  2. [§5 'Threshold behavior and triage'] The operational screen (review at most 50%, catch at least 80% of actual errors, 1.5x enrichment) is evaluated against the any-MQM-error binary label, whereas the production-safety discussion in §2 and §4 is about serious errors that can leak into the unreviewed tail. The paper itself notes that major-plus thresholds were sparse and unstable in EC slices. As reported, the screen does not establish that severe-error gating is unsafe; it shows that no threshold in the saved strategies achieves these operating points on the any-error condition. Please analyze the major-plus severity condition where feasible, or explicitly justify why the any-error condition is the appropriate risk measure for the routing-safety conclusion.
  3. [§5, Table 1] The Brier/no-skill ratio of 11.139 for the DFKI corpus is implausible if a proper Brier score on a [0,1] probability scale is used, because with a base rate near 90% this would imply a Brier score near 1.0, which would be an extremely poor calibration that the reported AUC (0.679) would not suggest. This value suggests that raw model scores were plugged into a Brier-type formula without a probability transformation. Since the calibration-failure claim is a central piece of the argument, the definition of the Brier score (what exactly is entered as the predicted probability, and how it is bounded) must be stated, and every ratio in Table 1 should be recomputed accordingly.
  4. [§5 'Threshold behavior and triage' and 'Non-portability'] The threshold scenarios are selected on the same data on which the operational screen is applied. Reported precision, recall, and triage-gain figures are therefore in-sample estimates and are likely optimistic relative to a deployment where thresholds are fixed in advance on a held-out or previous dataset. The absence of any cross-validation or train/test split for threshold selection weakens the quantitative claim that 'no saved threshold scenario satisfied all three conditions.' Please clarify whether the threshold strategies were tuned on the same slices and, if so, add a validation scheme or state the limitation.
minor comments (5)
  1. [§5 'Data and the gold standard'] The manuscript reports that bootstrap confidence intervals and permutation p-values were computed, but Table 1 reports only point medians. At least the headline median AUCs (EC 0.591, EP 0.631, DFKI 0.679) should be accompanied by confidence intervals so the reader can judge the stability of the 'modest' ranking claim.
  2. [§5 'QE model and analysis instrument'] The Revix export contains 186 snapshots; the duplication into 72 unique slices and the use of all 186 for threshold analysis is described, but it is not stated whether the 14 duplicate snapshots per slice (186 vs. 72) affect the distribution of reported medians. Clarify how multiple snapshots per slice are aggregated in Table 1.
  3. [Figure 1] The figure is reproduced from prior work (Gladkoff, 2025); it should be labeled as such and the model name and error severity categories should be defined in the caption.
  4. [Data availability] No data or code availability statement is provided. Given the paper's own emphasis on reproducibility in QE research, the Revix-derived statistics or an anonymized aggregate dataset should be made available to allow independent verification of Table 1 and the threshold screen.
  5. [References] Several citations are to unpublished or preprint sources (e.g., Liang and Han, 2026; Rosenbaum et al., 2026; Siani et al., 2026). Please mark these as forthcoming/preprint in the reference list, and provide DOIs or archive IDs where available.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical argument is independent and the new 104k-segment experiment measures TAUS EPIC QE against external MQM annotations; the class-level conclusion is an empirical generalization, not a construction.

full rationale

The paper's central claim has two independent supports. First, the theoretical argument formalizes translation quality as Q(s,t,c), where c denotes unrecoverable extra-segmental context, and infers that segment-level QE, which lacks c, is incomplete. This is an inference from a stated model, not a result whose conclusion was assumed in the definition; the Spanish pronoun and 'Apply the patch' examples illustrate the mechanism concretely. Second, Section 5 reports a new empirical evaluation: 104,762 segments scored by TAUS EPIC QE and compared against independently produced MQM human annotations. The QE scores and human labels are separate measurements, so the comparison is not circular. The 'no safe threshold' screen (at most 50% flagged, at least 80% of errors caught, at least 1.5x enrichment) is an operational criterion applied to observed threshold scenarios; no fitted parameter is renamed as a prediction. The main self-citations (Gladkoff et al. 2022, Gladkoff 2025, Lommel et al. 2024) provide background and an earlier case study, but the paper's own Section 5 independently reproduces the core threshold-failure pattern on a different model and data. The gap between the single-model finding and the class-level conclusion about 'segment-level QE scores' is a representativeness and overgeneralization concern, not circularity: the conclusion may be under-supported by one commercial model, and the any-error label may not match severe-error production risk, but these are correctness limitations outside the circularity definition.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a hand-chosen operational screen, a binary MQM error label, and the representativeness of one commercial QE model. No new theoretical entities are introduced.

free parameters (3)
  • Operational review screen thresholds = review <= 50% of file, catch >= 80% of errors, flagged set >= 1.5x richer than random
    Hand-chosen decision rule in Section 5; the central negative result ('no saved threshold satisfies all three conditions') depends on these exact values. A different screen (e.g., 70% recall or 2x enrichment) could change the verdict.
  • Binary error label = any MQM error (vs. major-plus)
    The main analysis uses 'any error' as the positive class. The paper notes major-error counts were sparse and unstable, so the choice of this label materially shapes AUC and precision numbers.
  • Threshold strategies explored = max F1, F2, F0.5, recall>=90%, Youden's J, max triage gain, manual/unspecified
    The claim that no threshold passes the screen is limited to these saved strategies; an exhaustive threshold sweep might find a different result.
assumptions (5)
  • domain assumption MQM human annotations are a valid, noise-free gold standard for translation quality.
    Used as ground truth for all AUC/Brier analyses in Section 5; the paper does not report inter-annotator agreement or annotation uncertainty, despite recommending that others do so.
  • domain assumption TAUS EPIC QE is representative of segment-level QE systems as a class.
    The empirical generalization from one commercial model to 'segment-level QE scores' in the conclusion rests on this; only weak literature support is given.
  • domain assumption The three corpora (DFKI, EP, EC) span the practical range of production translation workflows.
    DFKI (very poor MT), EP (unedited MT), EC (human-edited) cover three quality regimes, but they are convenience samples; no sampling design or representativeness argument is provided.
  • domain assumption Extra-segmental context is not recoverable from isolated segments.
    The theoretical critique in Section 2 depends on this; the examples (Tiene razon, Apply the patch) illustrate ambiguity but do not prove that QE models cannot infer some context from the segment or system metadata.
  • standard math The standard mathematical tools used (AUC, Brier, bootstrap) are appropriate.
    Assumed background; not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation." pith.science (2026). https://pith.science/paper/X4UG2NB3

@misc{pith2026260803577,
  author       = {Pith},
  title        = {Pith review of: Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4UG2NB3}},
  note         = {Machine review of arXiv:2608.03577}
}
read the original abstract

Automation of Translation Quality Estimation (QE) has emerged as a widely discussed approach to managing translation quality at scale, and a growing number of tools and technologies have been released in pursuit of this goal. However, the proliferation of new QE systems has not always been accompanied by robust, transparent, and reproducible research and testing. This gap deserves critical scrutiny. This paper examines some fundamental limitations of the QE technology from both theoretical and empirical perspectives, arguing that current QE systems are structurally ill-equipped to serve as reliable standalone tools in real-world translation workflows. The reviewed evidence suggests that QE suffers from a range of interrelated and largely unresolved limitations. Most fundamentally, the evaluation of the quality of translation at the level of isolated segments is problematic because it tends to miss out on cohesion, coherence, and stylistic and rhetorical text features. In addition, empirical research documents several other limitations and flaws, including failure to generalize, systematic biases, overfitting and distribution collapse, performance gaps, error annotation challenges, and data scarcity. These are structural limitations arising from the complexity of human language and translation as a cognitive and communicative act - limitations that more data and better architectures have so far not overcome. Consequently, segment-level QE scores should not be used as a standalone basis for routing, release, or review bypass in production; we argue future work should focus on automating human evaluation grounded in MQM.

Figures

Figures reproduced from arXiv: 2608.03577 by the authors.

Figure 1
Figure 1. Automatic translation quality prediction scores vs. actual error distributions. (COMET model wmt20- [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Identical QE scores can lead to different [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 29 canonical work pages

  1. [1]

    Proceedings of the 9th International Conference on Natural Language and Speech Processing (ICNLSP 2026) , year =

    Gladkoff, Serge and Vaasa, Angelika and Wright, Sue Ellen and Strandvik, Ingemar and Han, Lifeng , title =. Proceedings of the 9th International Conference on Natural Language and Speech Processing (ICNLSP 2026) , year =

    work page 2026

  2. [2]

    2026 , eprint=

    Semi-Synthetic Parallel Data for Translation Quality Estimation: A Case Study of Dataset Building for an Under-Resourced Language Pair , author=. 2026 , eprint=

    work page 2026

  3. [3]

    Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages=

    Translating into morphologically rich languages with synthetic phrases , author=. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages=

    work page 2013

  4. [4]

    Intelligent Translation Memory Matching and Retrieval with Sentence Encoders

    Intelligent Translation Memory Matching and Retrieval with Sentence Encoders , author =. arXiv preprint arXiv:2004.12894 , year =

    work page Pith review arXiv 2004

  5. [5]

    2004 , school =

    Segmentation in Translation and Translation Memory Systems: An Empirical Investigation of Cognitive Segmentation , author =. 2004 , school =

    work page 2004

  6. [6]

    Frontiers in Digital Health , volume=

    Neural machine translation of clinical text: an empirical investigation into multilingual pre-trained language models and transfer-learning , author=. Frontiers in Digital Health , volume=. 2024 , publisher=

    2024

  7. [7]

    and Espl \`a -Gomis, Miquel and Specia, Lucia

    Scarton, Carolina and Forcada, Mikel L. and Espl \`a -Gomis, Miquel and Specia, Lucia. Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality. Proceedings of the 16th International Conference on Spoken Language Translation. 2019

    work page 2019

  8. [8]

    Introducing Quality Estimation to Machine Translation Post-editing Workflow: An Empirical Study on Its Usefulness

    Liu, Siqi and Dai, Guangrong and Li, Dechao. Introducing Quality Estimation to Machine Translation Post-editing Workflow: An Empirical Study on Its Usefulness. Proceedings of Machine Translation Summit XX: Volume 1. 2025

    2025

Show all 37 references
  1. [9]

    ITALERT : Assessing the Quality of LLM s and NMT in Translating I talian Emergency Response Text

    Staiano, Maria Carmen and Han, Lifeng and Monti, Johanna and Chiusaroli, Francesca. ITALERT : Assessing the Quality of LLM s and NMT in Translating I talian Emergency Response Text. Proceedings of Machine Translation Summit XX: Volume 1. 2025

    2025

  2. [10]

    1965 , publisher=

    A linguistic theory of translation , author=. 1965 , publisher=

    1965

  3. [11]

    On Translation , editor =

    On Linguistic Aspects of Translation , author =. On Translation , editor =. 1959 , pages =

    1959

  4. [12]

    Findings of the Association for Computational Linguistics: ACL 2023 , pages=

    Rethinking the word-level quality estimation for machine translation from human judgement , author=. Findings of the Association for Computational Linguistics: ACL 2023 , pages=

    2023

  5. [13]

    MetaHOPE: Metaphor Translation Evaluation Framework Investigating Open-Source LLMs and State-of-the-Art Neural Translation Models , author=. , year=

  6. [14]

    Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors

    Mehandru, Nikita and Agrawal, Sweta and Xiao, Yimin and Gao, Ge and Khoong, Elaine and Carpuat, Marine and Salehi, Niloufar. Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors. Proceed...

    2023 doi

  7. [15]

    Proceedings of the Thirteenth Language Resources and Evaluation Conference , year =

    Gladkoff, Serge and Sorokina, Irina and Han, Lifeng and Alekseeva, Anastasia , title =. Proceedings of the Thirteenth Language Resources and Evaluation Conference , year =

  8. [16]

    The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control

    Lommel, Arle and Gladkoff, Serge and Melby, Alan and Wright, Sue Ellen and Strandvik, Ingemar and Gasova, Katerina and Vaasa, Angelika and Benzo, Andy and Marazzato Sparano, Romina and Foresi, Monica and Innis, Johani and Han, Lifeng and Nenadic, Goran. The Multi-Range Theory ...

    2024

  9. [17]

    MTQE .en-he: Machine Translation Quality Estimation for E nglish- H ebrew

    Rosenbaum, Andy and Siani, Assaf and Kernerman, Ilan. MTQE .en-he: Machine Translation Quality Estimation for E nglish- H ebrew. Proceedings of the Second Workshop on Language Models for Low-Resource Languages ( L o R es LM 2026). 2026. doi:10.18653/v1/2026.loreslm-1.48

    2026 doi

  10. [18]

    2025 , note =

    Gladkoff, Serge , title =. 2025 , note =

    2025

  11. [19]

    Mathematics , volume =

    Jung, Dongjun and Park, Chanjun and Eo, Sugyeong and Lim, Heuiseok , title =. Mathematics , volume =. 2023 , doi =

    2023

  12. [20]

    Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems , year =

    Qian, Shenbin and Or. Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems , year =. 2503.16158 , archivePrefix =

    arXiv

  13. [21]

    Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement , booktitle =

    Sarti, Gabriele and Zouhar, Vil. Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement , booktitle =. 2025 , doi =

    2025

  14. [22]

    Proceedings of Machine Translation Summit XIX , year =

    Dinh, Tu Anh and Niehues, Jan , title =. Proceedings of Machine Translation Summit XIX , year =

  15. [23]

    Findings of the Association for Computational Linguistics: ACL 2022 , year =

    Kocyigit, Mehmet and Lee, James and Wijaya, Derry , title =. Findings of the Association for Computational Linguistics: ACL 2022 , year =

    2022

  16. [24]

    Tailoring Domain Adaptation for Machine Translation Quality Estimation , booktitle =

    Sharami, Javad Pourmostafa Roshan and Shterionov, Dimitar and Blain, Fr. Tailoring Domain Adaptation for Machine Translation Quality Estimation , booktitle =. 2023 , url =

    2023

  17. [25]

    and Rei, Ricardo and van Stigt, Daan and Coheur, Luisa and Colombo, Pierre and Martins, Andr

    Guerreiro, Nuno M. and Rei, Ricardo and van Stigt, Daan and Coheur, Luisa and Colombo, Pierre and Martins, Andr. xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection , year =. 2310.10482 , archivePrefix =

    arXiv

  18. [26]

    2025 , eprint =

    Zhang, Yilin and Xu, Wenda and Liu, Zhongtao and Nakagawa, Tetsuji and Freitag, Markus , title =. 2025 , eprint =

    2025

  19. [27]

    Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation , booktitle =

    Zaranis, Emmanouil and Attanasio, Giuseppe and Agrawal, Sweta and Martins, Andr. Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation , booktitle =. 2025 , url =

    2025

  20. [28]

    Transactions of the Association for Computational Linguistics , year =

    Sarti, Gabriele and Zouhar, Vil. Transactions of the Association for Computational Linguistics , year =

  21. [29]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

    Geng, Xiang and Lai, Zhejian and Chen, Jiajun and Yang, Hao and Huang, Shujian , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

  22. [30]

    Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Huang, Fei and Xu, Jian-Ming and Ittycheriah, Abraham and Roukos, Salim , title =. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2014 , doi =

    2014

  23. [31]

    Turchi, Marco and Negri, Matteo and Federico, Marcello , title =. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages =. 2015 , doi =

    2015

  24. [32]

    The Role of Machine Translation Quality Estimation in the Post-Editing Workflow , journal =

    B. The Role of Machine Translation Quality Estimation in the Post-Editing Workflow , journal =. 2021 , doi =

    2021

  25. [33]

    Lee, Dongjun and Ahn, Jaehui and Park, Hwanhee and Jo, Jihun , title =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations , pages =. 2021 , doi =

    2021

  26. [34]

    Proceedings of Machine Translation Summit XVIII: Users and Providers Track , year =

    Alva-Manchego, Fernando and Specia, Lucia and Szoc, Sara and Vanallemeersch, Tom and Depraetere, Heidi , title =. Proceedings of Machine Translation Summit XVIII: Users and Providers Track , year =

  27. [35]

    Estimation vs Metrics: is QE Useful for MT Model Selection?

    Zaretskaya, Anna and Concei c \ a o, Jos \'e and Bane, Frederick. Estimation vs Metrics: is QE Useful for MT Model Selection?. Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. 2020

    2020

  28. [36]

    Uncertainty-Aware Machine Translation Evaluation , booktitle =

    Glushkova, Taisiya and Zerva, Chrysoula and Rei, Ricardo and Martins, Andr. Uncertainty-Aware Machine Translation Evaluation , booktitle =. 2021 , doi =

    2021

  29. [37]

    Findings of the

    Lavie, Alon and Hanneman, Greg and Agrawal, Sweta and Kanojia, Diptesh and Lo, Chi-Kiu and Zouhar, Vil. Findings of the. Proceedings of the Tenth Conference on Machine Translation , month = nov, year =. doi:10.18653/v1/2025.wmt-1.24 , url =

    2025 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.