A comparison of translation performance between DeepL and Supertext

Alex Fl\"uckiger; Chantal Amrhein; Florian Schottmann; Fr\'ed\'eric Odermatt; Martin P\"omsl; Philippe Schl\"apfer; Samuel L\"aubli; Tim Graf

REVIEW 4 major objections 4 minor 15 references

A comparison of translation performance between DeepL and Supertext

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Document-level ratings flip DeepL–Supertext ranking in three of four language directions.

desk verdict Small, honest A/B comparison of DeepL and Supertext whose headline document-level claim is not supported by its own statistics; the released data is worth reanalyzing. read the letter →

arxiv 2502.02577 v3 pith:GEXUVRI7 submitted 2025-02-04 cs.CL

Alex Flückiger , Chantal Amrhein , Tim Graf , Frédéric Odermatt , Martin Pömsl , Philippe Schläpfer , Florian Schottmann , Samuel Läubli This is my paper · ORCID

classification cs.CL

keywords machinetranslationevaluationdocument-levelhumanA/BtestingLLM-basedconsistencyDeepLSupertext

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares two commercial machine-translation services, DeepL and Supertext, under conditions designed to use each system's full document context: professional translators read entire unsegmented texts and rated sentence-by-sentence which system's translation was better. At the segment level the two systems are near ties in three of four language directions. When per-segment preferences are aggregated per document, Supertext is preferred in three of four directions, which the authors attribute to better consistency across longer texts; the exception is English-to-German (en → de-CH), where DeepL is preferred at both levels. The study matters because real-world translation use is document-level, so segment-only benchmarks may misrank systems.

What carries the argument

The central mechanism is a blind A/B test with document-level presentation and document-level aggregation. Professional translators see the full source text and both translations side-by-side in original order, so context is available; each segment gets a three-way judgment (A better, B better, equal). A document is then scored by majority vote of its segment judgments. This two-step construction — segment judgment under full context, then per-document majority — is what produces the reversal from a segment-level tie to a document-level preference, and it is the object the paper argues should become standard practice.

What would settle it

Re-run the same 80-document evaluation with several independent professional translators per document and ask each for an explicit document-level preference in addition to per-segment ratings; if the majority-aggregated Supertext preference in three of four directions does not reproduce, or does not correlate with the explicit whole-document preference, the paper's headline finding collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that the unit of measurement changes the comparison: pooling all segment judgments shows no strong preference, but counting, for each document, which system was preferred on more segments yields a Supertext preference in three of four language directions (de → en-GB, de → fr-CH, de → it-CH) and a DeepL preference in en → de-CH. The authors interpret the document-level result as evidence that Supertext maintains consistency across longer texts, giving an example where DeepL renders the German word Startseite as 'start page', 'home page', and 'Home page' in one document while Supertext stays consistent. The study is presented as a case for context-sensitive evaluation methodology in the LLM era of machine translation.

Load-bearing premise

The load-bearing assumption is that counting which system was preferred on more segments within a document is a faithful measure of document-level translation quality; the paper does not validate this aggregation against an independent whole-document judgment or report inter-annotator agreement.

Editorial extensions

If this is right

If document-level aggregation reflects real-world quality, segment-only benchmarking of commercial MT systems can produce misleading rankings for users who translate full documents.
The Supertext preference in three of four directions supports the paper's proposal that smaller LLM-based providers can compete with dominant closed systems, especially on long-text consistency.
A terminology-consistency explanation predicts that Supertext's advantage will be most visible in documents with repeated terms and cross-paragraph references; the Startseite example illustrates the pattern.
For en → de-CH, users should expect DeepL to remain preferable even with full context, possibly because of within-sentence errors or target-variant mixing in Supertext.
Future benchmarking campaigns should let the systems segment the input themselves and evaluate with full document context rather than feeding pre-split sentences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's design cannot distinguish 'better consistency' from 'aggregation artifact': a document with one clearly better segment may outweigh several ties, so a preference-strength-weighted score (for example, MQM severity) could rank the systems differently.
If context-window utilisation is the driving factor, a direct test is to translate the same documents again after forcing sentence-by-sentence segmentation; the prediction is that Supertext's document-level advantage shrinks or disappears.
The en → de-CH exception suggests that target-language variant control (Swiss German versus standard German) may dominate perceived quality; comparing de-CH with de-DE outputs for the same source text could separate consistency from variant handling.
The public release of data and scripts allows re-aggregation with different rules, so the three-of-four result can be checked against a per-document significance test or a per-rater analysis.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a readable, transparent, small-scale A/B comparison, not a deep methodological advance. The genuinely new bit is the head-to-head with full document context between two commercial systems, and the release of the raw data and scripts. That counts for something; the authors do the field a service by showing how a segment-level tie can become a document-level gap, if their aggregation means what they think.

What the paper does well: the setup is sensible and honestly reported. They used professional translators, blind A/B, kept system assignments consistent within a document, and published everything. The limitations section admits the big holes; most papers would hide them. The example in Table 2 is a nice concrete illustration of the consistency phenomenon.

Where it goes soft: the central claim in the abstract—'document-level analysis reveals a preference for Supertext in three out of four language directions'—is not backed by any significance test or confidence interval. For de→it-CH, the strongest case, the count is 7 DeepL, 3 equal, 10 Supertext; a two-sided binomial test against p=0.5 gives p≈0.63, so chance is not ruled out. For two of the four directions, the document counts appear only as figure bars, not numbers, so the reader cannot check the arithmetic. The paper's own Limitations section concedes there is no inter-annotator agreement and no significance testing. That is not a minor footnote; it is a load-bearing gap, because the whole point of the paper is the discrepancy between segment-level and document-level rankings. With 20 documents per direction and a single rater per document, majority voting over segments is a fragile proxy for document-level quality, and the authors do not validate it against independent document-level judgments or a second rater.

One more thing: the authors work for Supertext. They disclose this and the A/B is randomized and blind, so I don't infer deliberate bias. But the design means the vendor's own employees are the only raters, which is a genuine independence limitation for a comparison meant to inform buyers.

The stress-test note is right, and the reader's CONDITIONAL verdict is about right. This is not a rejection-level paper; the research question is timely, the data release is real, and the finding, if it survives proper analysis, would matter for MT benchmarking practice. A serious referee should engage, but the authors need to add significance testing, report document-level counts for all four directions, and get external ratings or at least a second rater for a subset. As it stands, the headline sentence overclaims what the statistics can support.

Referee Report

4 major / 4 minor

Summary. The paper reports a bilingual A/B evaluation of two commercial machine translation systems, DeepL and Supertext, on 80 documents and 1033 segments across four language directions. Professional translators rated the outputs with full document context, and the authors derive document-level preferences by aggregating segment-level preferences per document. They report that segment-level ratings show no strong preference in most directions, while document-level aggregation favors Supertext in three of four language directions, which they attribute to superior consistency across longer texts. The evaluation data and scripts are released publicly.

Significance. If the document-level aggregate were statistically robust, the paper would be a useful empirical contribution: it demonstrates that the unit of evaluation can change the ranking of two commercial systems, and it provides a reusable dataset for context-aware MT evaluation. The blind design, the use of professional translators, and the public release of data and scripts are concrete strengths. However, as detailed below, the central document-level claim is currently under-supported because the reported counts do not reach statistical significance, no inter-annotator agreement is measured, and the document-level measure is an untested aggregation of segment-level preferences.

major comments (4)

[Section 5.2] The document-level conclusion is not supported by the reported statistics. The only numeric document count given in the text is de→it-CH: 7 DeepL, 3 equal, 10 Supertext. Among the 17 non-tied documents, a two-sided exact binomial test against p=0.5 gives p≈0.63, so this count does not reject chance. Document counts for de→en-GB and de→fr-CH are not reported numerically, only as bars in Figure 2, and no confidence intervals or significance tests are provided. Even the direction with the largest apparent imbalance (en→de-CH: 13 DeepL, 2 equal, 5 Supertext) has a two-sided binomial p≈0.096 over 18 non-tied documents. The abstract's statement that document-level analysis 'reveals a preference for Supertext in three out of four language directions' therefore overstates what the data show.
[Sections 4.3 and 5.2] The 'document-level preference' is defined as a majority vote over one rater's segment-level preferences, not as an independent holistic judgment of document quality. Since each document is assigned to a single rater, there is no inter-annotator agreement estimate, and the Limitations section explicitly acknowledges this. The central claim depends on this aggregation rule, but the paper does not validate it against a direct document-level rating or against any measure of cross-segment consistency. Please report per-document counts for all directions, provide evidence on rater reliability (even a small re-rating subset), and justify or validate the majority-vote aggregation.
[Section 6] The attribution of the aggregated pattern to 'consistency across longer texts' is not supported by the evidence. The paper states that it has 'yet to conduct a systematic qualitative comparison' and offers one anecdotal example in Table 2. Because the document-level measure is derived from segment-level preferences, the observed pattern is equally compatible with small per-segment asymmetries or with the distribution of identical segments (9.5% of all segments, rising to 26.1% in the FAQ subset of de→en-GB). The authors should either provide a direct analysis of terminology consistency across segments or explicitly weaken the causal interpretation.
[Section 5.1] Segment-level claims also lack uncertainty quantification. For example, the en→de-CH segment counts (88 DeepL vs. 57 Supertext among 145 non-tied segments) may be statistically distinguishable, while the claim of 'no strong preference' in the other three directions rests solely on raw counts. Adding exact binomial tests or confidence intervals to the segment-level counts would make the contrast between segment- and document-level results interpretable, and is necessary before the paper can claim that the two units of measurement lead to different conclusions.

minor comments (4)

[References] The reference list contains typographical errors that should be corrected, e.g., 'W A' for 'WA', 'V olk' for 'Volk', and 'V .' for 'V.'.
[Figure 2] The caption does not state the tie-handling rule or the denominators for the aggregated document counts, and only one direction is reported numerically in Section 5.2. The numeric counts should be included in the text or in the figure itself.
[Table 2] Some glyphs in the source text (e.g., '?' and '≡') appear to be rendering artifacts; the example should be checked to ensure it faithfully reproduces the original source text.
[Section 3.1] The claim that the selected texts are 'unlikely to be contained in the training data' is stated without verification; phrasing this as an assumption rather than a fact would be more accurate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the comparison is an empirical A/B evaluation whose document-level result is a transparent aggregation of segment-level ratings, not a fitted input or self-citational reduction.

full rationale

The paper's derivation chain is empirical rather than formal. It collects source texts, produces translations with two commercial systems under default settings, has professional translators assign blind segment-level preferences with full document context, and then defines document-level preference as a majority-vote aggregation of those segment ratings (Section 5.2: 'We derive document-level preferences by aggregating the segment-level ratings of each evaluated document'). This is an explicit, non-circular operationalization, not a hidden reuse of the conclusion. The segment-level data are the measured input; the document-level counts are computed from them by a stated rule. There is no fitted parameter that is later renamed as a prediction, no equation in which the target quantity appears on both sides, and no uniqueness theorem imported from the authors' prior work. The only self-citation is to Laeubli et al. (2018) in the introduction, used to motivate document-level evaluation as a general methodological concern; that cited work is external published research and the present paper's own preference data are independent of it. The paper also explicitly discloses its limitations, including the absence of inter-annotator agreement and significance testing, which bears on statistical strength but does not create circularity. The released data and scripts further make the evaluation self-contained. Accordingly, the manuscript exhibits no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical evaluation rather than a derivation, so there are no fitted free parameters and no invented entities. The central claim rests on assumptions about text novelty, single-rater reliability, rater impartiality, and the validity of segment-vote aggregation as a document-level measure.

assumptions (4)

domain assumption The sampled NYT and NZZ news and FAQ texts are unlikely to be contained in the training data of either system.
Stated in Section 3.1 without verification. If false, one system could benefit from memorized source-target pairs and the comparison would be biased.
domain assumption A single professional rater per document provides reliable preference data, and rater-specific biases do not drive the aggregate results.
Section 4.3 assigns each document to one rater and counts preferences, but no repeated ratings or inter-annotator agreement measure is reported.
domain assumption Raters employed by Supertext are unbiased because assignments were randomized and anonymized.
The Limitations section asserts no bias from randomization and anonymization. Randomization reduces ordering effects but does not eliminate interest bias without external validation.
ad hoc to paper Majority-vote aggregation of segment-level preferences yields a meaningful document-level preference.
Section 5.2 defines document-level preference as counting which system was preferred on more segments. This operationalization is specific to the paper and is not validated against an independent document-level judgment.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A comparison of translation performance between DeepL and Supertext." pith.science (2026). https://pith.science/paper/GEXUVRI7

@misc{pith2026250202577,
  author       = {Pith},
  title        = {Pith review of: A comparison of translation performance between DeepL and Supertext},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEXUVRI7}},
  note         = {Machine review of arXiv:2502.02577}
}

read the original abstract

As strong machine translation (MT) systems are increasingly based on large language models (LLMs), reliable quality benchmarking requires methods that capture their ability to leverage extended context. This study compares two commercial MT systems -- DeepL and Supertext -- by assessing their performance on unsegmented texts. We evaluate translation quality across four language directions with professional translators assessing segments with full document-level context. While segment-level assessments indicate no strong preference between the systems in most cases, document-level analysis reveals a preference for Supertext in three out of four language directions, suggesting superior consistency across longer texts. We advocate for more context-sensitive evaluation methodologies to ensure that MT quality assessments reflect real-world usability. We release all evaluation data and scripts for further analysis and reproduction at https://github.com/supertext/evaluation_deepl_supertext.

Figures

Figures reproduced from arXiv: 2502.02577 by the authors.

**Figure 2.** Aggregated segment-level ratings per text. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

[1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate . In Proceedings of ICLR , San Diego, CA

work page 2015
[2]

Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for Computational Linguistics, 9:1460--1474

doi:10.1162/tacl_a_00437 2021
[3]

Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. https://arxiv.org/abs/1803.05567 Achieving human parity on automatic chinese to english news translation . arXiv preprint arXiv:1803.05567

arXiv 2018
[4]

Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models . In Proceedings of EMNLP , pages 1700--1709, Seattle, WA

work page 2013
[5]

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovi \'c , Mariya Shmatova, Steinth \'o r Steingr \'i msson...

doi:10.18653/v1/2024.wmt-1.1 2024
[6]

Samuel L \"a ubli, Rico Sennrich, and Martin Volk. 2018. https://doi.org/10.18653/v1/D18-1512 Has machine translation achieved human parity? a case for document-level evaluation . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791--4796, Brussels, Belgium. Association for Computational Linguistics

doi:10.18653/v1/d18-1512 2018
[7]

Matt Post and Marcin Junczys-Dowmunt. 2023. https://api.semanticscholar.org/CorpusID:258309151 Escaping the sentence-level paradigm in machine translation . ArXiv, abs/2304.12959

arXiv 2023
[8]

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks . In Proceedings of NIPS , pages 3104--3112, Montreal, Canada

work page 2014

Show all 15 references

[9]

Diane Tang, Ashish Agarwal, Deirdre O'Brien, and Mike Meyer. 2010. https://doi.org/10.1145/1835804.1835810 Overlapping experiment infrastructure: more, better, faster experimentation . In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Da...

2010
[10]

Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. https://doi.org/10.18653/v1/W18-6312 Attaining the unattainable? reassessing claims of human parity in neural machine translation . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages ...

2018 doi
[11]

Guojun Wu, Shay B Cohen, and Rico Sennrich. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.169 Evaluating automatic metrics with incremental machine translation systems . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2994--3005, Miam...

2024 doi
[12]

Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. 2024 b . https://arxiv.org/abs/2401.06468 Adapting large language models for document-level machine translation . Preprint, arXiv:2401.06468

2024 arXiv
[13]

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. https://arxiv.org/abs/2309.11674 A paradigm shift in machine translation: Boosting translation performance of large language models . Preprint, arXiv:2309.11674

2024 arXiv
[14]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[15]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate . In Proceedings of ICLR , San Diego, CA

work page 2015

[2] [2]

Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for Computational Linguistics, 9:1460--1474

doi:10.1162/tacl_a_00437 2021

[3] [3]

Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. https://arxiv.org/abs/1803.05567 Achieving human parity on automatic chinese to english news translation . arXiv preprint arXiv:1803.05567

arXiv 2018

[4] [4]

Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models . In Proceedings of EMNLP , pages 1700--1709, Seattle, WA

work page 2013

[5] [5]

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovi \'c , Mariya Shmatova, Steinth \'o r Steingr \'i msson...

doi:10.18653/v1/2024.wmt-1.1 2024

[6] [6]

Samuel L \"a ubli, Rico Sennrich, and Martin Volk. 2018. https://doi.org/10.18653/v1/D18-1512 Has machine translation achieved human parity? a case for document-level evaluation . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791--4796, Brussels, Belgium. Association for Computational Linguistics

doi:10.18653/v1/d18-1512 2018

[7] [7]

Matt Post and Marcin Junczys-Dowmunt. 2023. https://api.semanticscholar.org/CorpusID:258309151 Escaping the sentence-level paradigm in machine translation . ArXiv, abs/2304.12959

arXiv 2023

[8] [8]

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks . In Proceedings of NIPS , pages 3104--3112, Montreal, Canada

work page 2014