Preliminary Ranking of WMT25 General Machine Translation Systems

Anton Dvorkovich; Barry Haddow; Eleftherios Avramidis; Howard Lakougna; Jessica Lundin; Kenton Murray; Konstantin Dranch; Lisa Yankovskaya; Lorenzo Proietti; Maja Popovi\'c

REVIEW 4 major objections 5 minor 4 references

Preliminary Ranking of WMT25 General Machine Translation Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Shy-hunyuan-MT tops WMT25's automatic ranking in every pair

desk verdict A useful, honestly hedged preliminary WMT25 ranking that deserves review, but the undisclosed overlap between the LLM judges and the evaluated systems is a real confound, especially for systems whose automatic ranking becomes the official result. read the letter →

arxiv 2508.14909 v2 pith:DKPAYGQX submitted 2025-08-11 cs.CL

Tom Kocmi , Eleftherios Avramidis , Rachel Bawden , Ondřej Bojar , Konstantin Dranch , Anton Dvorkovich , Sergey Dukanov , Natalia Fedorova

show 20 more authors

Mark Fishel Markus Freitag Thamme Gowda Roman Grundkiewicz Barry Haddow Marzena Karpinska Philipp Koehn Howard Lakougna Jessica Lundin Kenton Murray Masaaki Nagata Stefano Perrella Lorenzo Proietti Martin Popel Maja Popović Parker Riley Mariya Shmatova Steinthór Steingrímsson Lisa Yankovskaya Vilém Zouhar

This is my paper · ORCID

classification cs.CL

keywords machinetranslationWMT25automaticevaluationLLM-as-a-judgequalityestimationAUTORANKGEMBA-ESAsharedtask

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WMT25's General Machine Translation shared task has too many submissions to wait for human evaluation before participants start writing system descriptions, so this paper issues a preliminary ranking computed entirely by automatic metrics. The ranking method, named AUTO RANK, combines two reference-based metrics, one quality-estimation model, and two LLM judges, then rescales and averages them to produce one order per language pair. For the lowest-resource pairs, Bhojpuri and Maasai, the method falls back to the surface metric chrF++. The authors stress that the result is provisional: it may inflate systems that re-rank outputs with metrics or MBR, and the official WMT25 ranking, based on human Error Span Annotation, will replace it. Read as an early signal, the tables indicate which systems are worth studying; they are not final findings.

What carries the argument

The central object is AUTO RANK, a named aggregation procedure. It converts each metric's system scores $x_s^{(m)}$ into robust z-scores $z_s^{(m)}=(x_s^{(m)}-\tilde{x}^{(m)})/D^{(m)}$, where $\tilde{x}^{(m)}$ is the median and $D^{(m)}=\max(\varepsilon,Q_{100}^{(m)}-Q_{25}^{(m)})$, averages the $z_s^{(m)}$ with equal weights to get $\bar{z}_s$, and maps $\{\bar{z}_s\}$ linearly onto $1,\dots,N$ so lower is better. The construction is continuous and monotonic, so no system is discarded, within-metric order is preserved, extreme outliers do not dominate, and the relative spacing between systems survives the final remap. It is the piece of machinery that turns five heterogeneous metric columns

What would settle it

Once WMT25 releases the human Error Span Annotation scores, compare the human ordering per language pair with AUTO RANK: if the two disagree badly for high-resource pairs (e.g., Spearman rank correlation below roughly 0.5), the claim that this metric ensemble is a meaningful quality ordering fails. A separate probe for the judge-participant overlap: have GPT-4.1 and Command A score a blinded set containing their own outputs under different system names and compare with scores from an independent judge; systematic self-inflation would invalidate their columns.

Watch

Extended reading notes

Core claim

The paper establishes the AUTO RANK protocol as the provisional official ranking mechanism for WMT25. For each typical language pair it scores every submitted system with GEMBA-ESA run by two LLM judges (GPT-4.1 and Command A), MetricX-24-Hybrid-XL, XCOMET-XL, and CometKiwi-XL, averaging paragraph-level scores and combining them by median-interpercentile scaling followed by an equal-weight mean and a linear remap to ranks 1 through N. Bhojpuri and Maasai, where learned metrics are unvalidated, are ranked by chrF++ alone. The resulting tables put Shy-hunyuan-MT, a 7B constrained system, first on every language pair shown, ahead of large proprietary models. The authors frame the output as prel

Load-bearing premise

The load-bearing premise is that the five automatic metrics—including two LLM judges that are themselves among the ranked systems—behave like human quality judgments at paragraph level across all 30 language pairs, including low-resource pairs where they were never validated.

Editorial extensions

If this is right

If AUTO RANK reflects quality, participants can use these tables immediately to position their system descriptions while human evaluation is still running.
For the 16 language pairs without human evaluation, AUTO RANK is the final official ranking, so its reliability there is not merely preliminary.
Systems that use Quality Estimation or Minimum Bayes Risk re-ranking may be systematically over-ranked, so the tables should not be read as pure model-quality ordering.
The repeated first-place of a 7B constrained system implies small open models can beat much larger proprietary ones on this metric mix, a pattern the human results can confirm or refute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A blinded swap test could isolate judge self-bias: if GPT-4.1 and Command A are removed as judges and replaced by an independently judged LLM, their system ranks may shift, revealing whether judge-participant overlap inflates their positions.
Because the normalization keeps every system and preserves spacing, AUTO RANK is sensitive to which systems enter the pool; adding or removing a strong outlier changes everyone's scaled distance, so cross-language comparisons of rank gaps are not meaningful.
The low-correlation appendix suggests Marathi, whose learned metrics disagree most, is the weakest point of the multilingual subtrack; recomputing Marathi's ranking with chrF++ if references were available would test how much the order depends on metric choice.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This is a straightforward preliminary evaluation report for WMT25. The rankings are new for this year, the test sets and system outputs are public, and the aggregation scheme in Eqs. (1)-(2) is clearly specified and correctly implemented as far as I can tell from the tables. The paper is also unusually candid: it says outright that automatic rankings can be biased toward re-ranking, that human evaluation will supersede them, and that some participants optimized for GEMBA-ESA. That honesty is worth crediting.

The main thing I'd flag is the judge-participant overlap. GPT-4.1 and Command A serve as the two GEMBA-ESA judges, and both also appear as submitted systems in the same tables. The Limitations section discusses metric optimization and unsupported languages, but never mentions this. If either judge has any systematic self-preference, its own AUTO RANK position is inflated through two of the five metric slots. The released per-segment scores allow a direct check: compare how GPT-4.1-as-judge scores GPT-4.1-as-system versus how Command-A-as-judge scores it, and vice versa, after normalizing. That test should be run before anyone takes the fine-grained rankings seriously. The paper's own caveat that human evaluation will supersede these results softens the problem, but not fully: for any system not selected for human evaluation, the AUTO RANK is the official final result.

The other soft spot is the absence of uncertainty quantification. The paper gives a deterministic rank ordering with no confidence intervals or significance testing, and the final linear remapping to 1..N amplifies small mean shifts in GEMBA-ESA. That's a minor issue for a preliminary report, but it interacts with the judge-overlap problem.

The low-resource chrF++ decision is reasonable given the cited evidence, though the paper admits chrF++ correlates poorly with humans. The duplicated sentence in the abstract is a trivial editorial slip.

Overall, the central claim – that these are preliminary, automatically computed rankings – holds up. The methodology is inherited, but the dataset and results are new. I'd send this to peer review with a request for the authors to disclose the overlap and either run the self-preference check or qualify the affected ranks. The paper is a useful community resource and the thinking is transparent.

Referee Report

4 major / 5 minor

Summary. The paper presents preliminary rankings of WMT25 General Machine Translation systems, computed exclusively from automatic metrics and released to help participants write system description papers. The AUTO RANK procedure combines GEMBA-ESA with two LLM judges (GPT-4.1 and Command A), MetricX-24-Hybrid-XL, XCOMET-XL, and CometKiwi-XL (excluded for language pairs without human references), with chrF++ used alone for English-Bhojpuri and English-Maasai. System-level scores are averaged over ~100-word segments, normalized by median-interpercentile scaling (Eq. 1), averaged with equal weights (Eq. 2), and linearly rescaled to a 1..N rank. The authors explicitly warn that automatic rankings may favor re-ranking systems and that human evaluation will supersede them. Data, code, and per-segment scores are publicly released.

Significance. If the ranking is taken at face value, it provides a timely, transparent snapshot of system performance across 30+ language pairs and is a useful service to the WMT community. Strengths include the public release of test data, system outputs, per-segment scores, and LaTeX source; the aggregation procedure is clearly specified and easily reproducible; and the manuscript is unusually candid about metric bias, paragraph-level reliability, and low-resource limitations. However, the paper's central claim—that AUTO RANK produces a meaningful ordering—is weakened by an undisclosed judge-participant overlap (GPT-4.1 and Command A act both as GEMBA-ESA judges and as evaluated systems), by the absence of any uncertainty quantification, and by a tension between 'preliminary' framing and the statement that AUTO RANK is the official final result for systems not selected for human evaluation. These issues are testable with the released data and should be addressed before the ranking is used as a basis for system-description papers.

major comments (4)

[Automatic Ranking / Table 1]
[Eq. (2) and per-LP tables] No uncertainty quantification is provided for the reported ranks. Adjacent systems often differ by small fractions of an AutoRank unit (e.g., English-Czech rows: 3.5, 3.9, 4.4, 5.0; English-Russian rows: 4.2, 4.3, 4.4, 4.5). Given test sets of roughly 37K words per language pair and known segment-level metric noise, such differences may be statistically indistinguishable. The paper should provide bootstrap confidence intervals or a minimum-rank-difference threshold, and should avoid presenting one-decimal AUTO RANK values as if they represent a reliably ordered ranking. As written, the precision is misleading and the central claim of a meaningful ordering is not yet supported.
[Appendix A / Eq. (2)] The paper justifies equal-weight averaging by combining 'complementary failure modes,' but Appendix A shows that GEMBA-ESA-CMdA and GEMBA-ESA-GPT4.1 correlate above 0.8 for most language pairs (e.g., en-fa: 0.852; en-lt: 0.828; en-sv: 0.780), and the LLM-judge family is 2 of 5 metrics (2 of 4 when CometKiwi is excluded). Equal weighting therefore gives the LLM-judge family more effective weight than the nominal 2/5, and if the overlap concern above is real, the bias compounds. A robustness check—leave-one-metric-out or inverse-correlation reweighting—would show whether the ranking is stable under alternative aggregation schemes.
[Human Evaluation] The Abstract and Introduction describe the rankings as 'preliminary,' but the Human Evaluation section states that 'for any system not selected for human evaluation, the automatic metric ranking (AUTO RANK) serves as the official final result.' Many systems in the per-LP tables are not marked for human evaluation (blank Humeval column), so a substantial portion of the reported ranking is final, not preliminary. This tension should be resolved: either the abstract should say that the automatic ranking is final for a subset of systems, or the official status should be qualified for those systems. This matters because finality raises the stakes of the judge-overlap and uncertainty issues discussed above.

minor comments (5)

[Abstract and Introduction] The sentence 'The official WMT25 ranking will be based on human evaluation, which is more reliable and will supersede these results.' is duplicated verbatim in both the abstract and the introduction.
[Table 1] Several rows contain the LaTeX artifact '— /times' (e.g., Claude-4, Gemini-2.5-Pro, GPT-4.1, Mistral-Medium); these should render as '—' or '✗' consistently.
[Prompts] The repository link is malformed: '/githubgithub.com/wmt-conference/wmt-collect-translations' should be a proper URL.
[Low-resource exception] Minor typo: 'i.e.,Bhojpuri' should have a space after the comma. Also, the table header 'LP Supported' is ambiguous; consider renaming to 'LP officially supported by system' or similar.
[Limitations] Capitalization inconsistency: 'Gemba-ESA' appears in the Limitations section; the established acronym elsewhere is 'GEMBA-ESA'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the AUTO RANK is explicitly defined as the aggregate of the metric scores, and the paper disclaims any final quality claim.

full rationale

The paper's central output is the AUTO RANK, and it is transparently constructed from the metric scores it reports: Equation (2) defines \bar{z}_s as the average of robust-scaled metric scores, and the final ranking is a monotonic remapping of that average. The abstract explicitly says the rankings are 'as determined by automatic evaluation metrics' and that human evaluation 'will supersede these results.' Thus the ranking is a direct report of the metric inputs, not a derived prediction of an independent quantity, so it cannot reduce to its inputs by construction. The judge-participant overlap (GPT-4.1 and Command A both serve as GEMBA-ESA judges and appear in Table 1 as evaluated systems) is a genuine evaluation confound and an omitted limitation, but it is not circular in the sense defined here: no equation defines a system's score as that system's own judgment, and no fitted parameter is renamed as a prediction. Demonstrating self-preference would require an external check against other judges or human scores, which the paper does not perform; the reader's proposed per-segment comparison is an empirical test, not a reduction. The paper's Limitations section explicitly acknowledges that 'some models have been optimized for the very metrics we employ in AUTO RANK' and that GEMBA-ESA itself 'may have been leveraged to optimize machine translation models.' Those admissions weaken the ranking's validity as a quality estimate, but they are external gaming effects on the metric inputs, not circularity inside the derivation. Self-citations are present (GEMBA-ESA cited to Kocmi and Federmann 2023; ESA protocol to Kocmi et al. 2024), but the metric choice is also supported by internal cross-metric correlation analysis and by the inclusion of independent trained metrics (MetricX-24-Hybrid-XL, XCOMET-XL, CometKiwi-XL). The reasoning therefore does not collapse into a self-citation chain. The low-resource chrF++ exception is an acknowledged fallback for languages where the main metrics are unvalidated, again a stated methodological choice rather than a definitional loop. Overall, the analysis is self-contained as a metric-aggregation report; its weaknesses are validity and disclosure concerns, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new theoretical entities or fit parameters. The central claim rests on the validity of existing automatic metrics and the evaluation protocol, particularly the assumption that the LLM judges are unbiased even when they are also evaluated systems. The chrF++ low-resource exception is a pragmatic choice, not a fitted parameter.

assumptions (4)

domain assumption The selected automatic metrics (GEMBA-ESA, MetricX-24-Hybrid-XL, XCOMET-XL, CometKiwi-XL) are valid proxies for human translation quality at the paragraph level.
The entire AUTO RANK depends on this; the paper itself notes metrics are usually reliable at sentence level and that learned metrics struggle with low-resource languages. Invoked throughout the Automatic Ranking section.
domain assumption GPT-4.1 and Command A provide unbiased LLM-as-a-judge scores across all 30 language pairs, despite Command A officially supporting only 23 languages and GPT-4.1's language coverage not being documented.
Stated in the Limitations section: 'Command A officially supports only 23 languages... we retained them as judges for all 30 language pairs.' This is load-bearing for the GEMBA-ESA component.
domain assumption The test sets are representative and the reference translations are of sufficient quality for the trained metrics.
The paper describes the domains and segmentation but does not verify reference quality beyond noting human annotators had multimodal context. Acknowledged in Limitations with the reference bias discussion.
domain assumption Systems labeled as constrained or unconstrained are correctly categorized.
The ranking tables separate these categories and the human evaluation selection prioritizes constrained systems, so misclassification could affect which systems get human evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Preliminary Ranking of WMT25 General Machine Translation Systems." pith.science (2026). https://pith.science/paper/DKPAYGQX

@misc{pith2026250814909,
  author       = {Pith},
  title        = {Pith review of: Preliminary Ranking of WMT25 General Machine Translation Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DKPAYGQX}},
  note         = {Machine review of arXiv:2508.14909}
}

read the original abstract

We present the preliminary rankings of machine translation (MT) systems submitted to the WMT25 General Machine Translation Shared Task, as determined by automatic evaluation metrics. Because these rankings are derived from automatic evaluation, they may exhibit a bias toward systems that employ re-ranking techniques, such as Quality Estimation or Minimum Bayes Risk decoding. The official WMT25 ranking will be based on human evaluation, which is more reliable and will supersede these results. The official WMT25 ranking will be based on human evaluation, which is more reliable and will supersede these results. The purpose of releasing these findings now is to assist task participants with their system description papers; not to provide final findings.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages

[2]

Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets

Mitigating metric bias in minimum Bayes risk decoding. In Proceedings of the Ninth Conference on Machine Translation, pages 1063–1094, Miami, Florida, USA. Association for Computational Lin- guistics. Nikita Moghe, Arnisa Fazla, Chantal Amrhein, Tom Kocmi, Mark Steedman, Alexandra Birch, Rico Sen- nrich, and Liane Guillou. 2024. Machine translation meta e...

work page Pith review arXiv 2024
[4]

Association for Com- putational Linguistics

How good is zero-shot MT evaluation for low resource Indian languages? In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (V olume 2: Short Papers), pages 640–649, Bangkok, Thailand. Association for Com- putational Linguistics. Jiayi Wang, David Ifeoluwa Adelani, and Pontus Stene- torp. 2024. Evaluating WMT 2024 m...

work page 2024
[191]

Ricardo Rei, Nuno M

Association for Computational Linguistics. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. 2023. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation , pages 841–848. Associa...

work page 2023
[2024]

Transac- tions of the Association for Computational Linguis- tics, 12:979–995

xcomet: Transparent machine translation eval- uation through fine-grained error detection. Transac- tions of the Association for Computational Linguis- tics, 12:979–995. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on...

arXiv 2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [2]

Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets

Mitigating metric bias in minimum Bayes risk decoding. In Proceedings of the Ninth Conference on Machine Translation, pages 1063–1094, Miami, Florida, USA. Association for Computational Lin- guistics. Nikita Moghe, Arnisa Fazla, Chantal Amrhein, Tom Kocmi, Mark Steedman, Alexandra Birch, Rico Sen- nrich, and Liane Guillou. 2024. Machine translation meta e...

work page Pith review arXiv 2024

[2] [4]

Association for Com- putational Linguistics

How good is zero-shot MT evaluation for low resource Indian languages? In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (V olume 2: Short Papers), pages 640–649, Bangkok, Thailand. Association for Com- putational Linguistics. Jiayi Wang, David Ifeoluwa Adelani, and Pontus Stene- torp. 2024. Evaluating WMT 2024 m...

work page 2024

[3] [191]

Ricardo Rei, Nuno M

Association for Computational Linguistics. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. 2023. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation , pages 841–848. Associa...

work page 2023

[4] [2024]

Transac- tions of the Association for Computational Linguis- tics, 12:979–995

xcomet: Transparent machine translation eval- uation through fine-grained error detection. Transac- tions of the Association for Computational Linguis- tics, 12:979–995. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on...

arXiv 2024