Pith. sign in

REVIEW 5 major objections 5 minor 33 references

Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Literary-domain tuned systems beat general-purpose baselines on document-level BLEU in WMT 2024, but human judgments disagreed with the automatic ranking for one system.

desk verdict Useful shared-task report with a real data-release contribution, but the abstract claims a constrained-track result that the paper never shows. read the letter →

arxiv 2412.11732 v1 pith:HVTR4256 submitted 2024-12-16 cs.CL

Longyue Wang , Siyou Liu , Chenyang Lyu , Wenxiang Jiao , Xing Wang , Jiahao Xu , Zhaopeng Tu , Yan Gu
show 6 more authors
Weiyu Chen Minghao Wu Liting Zhou Philipp Koehn Andy Way Yulin Yuan
This is my paper · ORCID
classification cs.CL
keywords literarytranslationdiscourse-levelmachinedocument-levelBLEUhumanevaluationGuoFengWebnovelCorpusWMTsharedtasklargelanguagemodelsChinese-to-English
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the results of the second WMT shared task on discourse-level literary translation, covering Chinese-to-English, Chinese-to-German, and Chinese-to-Russian. Its central claim is empirical: after literary-domain enhancements, most participating systems outperform the Baseline systems (Llama-MT, Google Translate, and GPT-4) in terms of document-level BLEU (d-BLEU); for one system, human and automatic evaluations disagree sharply; and the only Constrained Track system performed close to the best Unconstrained Track system. The paper also releases the GuoFeng Webnovel Corpus V2, along with system outputs and a leaderboard, so the claims can be checked and extended. The result matters because it provides a public, reproducible benchmark for whether discourse-aware literary machine translation is actually improving rather than merely producing superficially fluent sentences.

What carries the argument

The central mechanism is the evaluation design. The document-level automatic score, d-BLEU, concatenates every sentence in a document into a single line and computes n-gram matches with sacreBLEU against two references. The human rubric gives each window of neighboring sentences two scores from 0 to 5, one for general quality (fluency and adequacy) and one for discourse awareness (consistency, word choice, anaphora, tone); the official ranking is built from these human scores rather than from d-BLEU. The newly added language directions, Chinese-to-German and Chinese-to-Russian, have no sentence-level alignment and only one reference, and because only two teams participated in each direction, those directions were evaluated automatically only.

What would settle it

Using the released Test_final outputs and references, recompute d-BLEU per book and compare it with new independent human scores on the same passages: if the human/automatic disagreement disappears or the top ranking flips under a different set of annotators, the paper's central findings would not replicate.

Watch

Extended reading notes

Core claim

The paper's findings, stated in the abstract and supported by the shared-task results, are threefold. First, most participating teams' models, after enhancements aimed at the literary domain, outperform the Baseline systems (Llama-MT, Google Translate, and GPT-4) in terms of d-BLEU, the document-level automated metric computed by concatenating each document's sentences into one line before applying sacreBLEU. Second, for a particular system the conclusions drawn from human evaluation diverge significantly from those drawn from automatic metrics. Third, the only Constrained Track submission was close to the best Unconstrained Track submission under both types of evaluation. The official ranking is based on the overall human judgments, which were collected only for the Chinese-to-English direction using a rubric that scores general quality and discourse awareness separately.

Load-bearing premise

The load-bearing premise is that the human scores, collected by three annotators for the Chinese-to-English direction on a custom 0–5 rubric, are a reliable measure of translation quality; if those scores are noisy or unrepresentative, the official ranking and the claimed human-automatic gap lose their foundation.

Editorial extensions

If this is right

  • Literary-domain adaptation on web-novel data yields measurable d-BLEU gains over general-purpose translation baselines in Chinese-to-English.
  • Document-level and human evaluation are not interchangeable: the same system can appear strong under one and weak under the other.
  • A constrained system can be competitive with unconstrained systems in this task, so access to external data or models is not a decisive advantage.
  • For language pairs without sentence-level alignment, document-level automatic scoring is the only evaluation used, so conclusions about Chinese-to-German and Chinese-to-Russian rest on d-BLEU alone, where the Google baseline ranked first.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The human/automatic disagreement suggests that d-BLEU alone would have produced a different leaderboard; a direct test is to recompute the ranking using only automatic metrics and compare it with the official human-based ranking.
  • Because the human evaluation covers only one language direction and a small number of annotators, the claimed gap between human and automatic conclusions should be read as a hypothesis about literary MT evaluation rather than a universal property; re-running the rubric on more languages and more annotators would show whether the gap is systematic.
  • The new Chinese-to-German and Chinese-to-Russian references were created by machine translation with human post-editing, so systems that stay close to Google's n-gram style may be favored by d-BLEU; a human evaluation on those directions could reveal whether Google's top d-BLEU score reflects real perceived quality.
  • The released system outputs would allow a per-book breakdown to test whether the d-BLEU advantage of primary systems over baselines holds outside books represented in the training data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper presents the findings of the WMT 2024 shared task on Discourse-Level Literary Translation, covering three language directions (Chinese→English, Chinese→German, Chinese→Russian). The organizers describe the second edition of the GuoFeng Webnovel Corpus, the evaluation protocol (automatic metrics plus human evaluation), the five participating teams, and their systems. The abstract states three headline findings: (1) most participating systems outperform baselines (Llama-MT, Google Translate, GPT-4) in d-BLEU; (2) for a certain system, there is a significant gap between conclusions drawn from human and automatic evaluations; and (3) the only Constrained Track system performed close to the best Unconstrained Track system in both automatic and human evaluation.

Significance. Findings papers for WMT shared tasks serve as community reference points for state of the art in a given translation direction. This paper contributes a new corpus extension (Zh-De, Zh-Ru), baseline comparisons, and system summaries, and it makes the data, outputs, and leaderboard publicly available. If the reported claims were fully supported, the paper would be a useful record for researchers working on discourse-level and literary MT. However, several of the central claims are currently not backed by the presented evidence, and the human evaluation protocol is described with unresolved inconsistencies. The paper's value depends on correcting these issues.

major comments (5)
  1. [Abstract and Section 5 / Table 2] Abstract finding (3) claims that the only Constrained Track system performed close to the best Unconstrained Track system, but no constrained system appears anywhere in the paper. Table 2 lists all five participating teams with the flag 'N' (unconstrained); no row uses the flag 'J', and every system in Tables 3 and 4 is marked with the '⋆' symbol defined as unconstrained. Section 5 describes each team's methods without mentioning any constrained submission. Because one of the three headline findings cannot be reconstructed from the reported evidence, the abstract overstates the paper's demonstrated results.
  2. [Section 3.2, Human Evaluation] The human evaluation protocol is incompletely specified and internally inconsistent. The text contains the literal placeholder 'For each chapter, we assessed xxx sentences', leaving the evaluation unit undefined. Immediately afterward, the paper says 'We employed two professional evaluators' and then 'We employed three professional evaluators', while Table 1 lists three evaluators (A, B, C). Since the official ranking is said to be based on these human judgments, the number of judgments per system, the evaluation unit, and the consistent evaluator count are load-bearing details that must be reported accurately.
  3. [Abstract and Section 6.2] Abstract finding (2) states that there is a 'significant gap' between human and automatic evaluation conclusions for a certain system, but no such system is identified and no statistical evidence is provided. Section 6.2 reports human scores without any significance test, confidence interval, or explicit comparison between the human and automatic rankings. The word 'significant' implies quantitative support that is absent from the paper, so this headline claim is currently unsubstantiated.
  4. [Table 5 and Section 6.2] The ranking information in Table 5 is ambiguous and contradicts the text. The Rank column shows '1 / 2' for NLP2CT-UM and '2 / 1' for SJTU-LoveFiction, without specifying which rank corresponds to General quality and which to Discourse awareness. Section 6.2 states that NLP2CT-UM 'achieved the highest scores in General (3.96) and Discourse (4.00)', but SJTU-LoveFiction has a Discourse score of 4.13, which is higher than 4.00. This inconsistency affects the official human-based ranking, which is the paper's central product.
  5. [Section 6.1, Tables 3 and 4] The claim in Section 6.1 that 'the primary systems outperform the baselines' in d-BLEU is not supported by the reported data. In Table 3, NTU has a d-BLEU of 34.6, well below Google's 47.3, and in Table 4 both NLP2CT-UM and SJTU-LoveFiction score below the Google baseline for Zh-De and Zh-Ru. Abstract finding (1) says 'most' systems outperform baselines, which may be salvageable, but the unqualified wording in Section 6.1 is contradicted by the tables and should be corrected to reflect the actual comparisons.
minor comments (5)
  1. [Section 3.1] The text says sacreBLEU, chrF, TER, and d-BLEU are calculated 'with two references', but the Zh-De and Zh-Ru test sets are described in Section 2.2 as having only one reference; the metric configuration for each language direction should be clarified.
  2. [Figure 1] The Zh-Ru row shows 'Test 122 13 15.6K'; the book count is presumably 12 rather than 122, which appears to be a typo.
  3. [Sections 5.1 and 5.4] Both sections contain 'introduced 1 translation systems'; the noun should be singular.
  4. [Section 2.3 and References] The in-text citation 'Chinese-Llama-2 7B (Zefeng Du, 2023)' does not match any reference entry; the closest entry is 'Longyue Wang Zefeng Du, Minghao Wu. 2023. Chinese-llama-2.' This should be reconciled in the reference list.
  5. [Table 2] The flag legend includes 'J' for constrained systems, but no row uses this flag; the paper should state explicitly that no constrained systems participated in the final evaluation, or if one did, it must be added to the table and described.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the shared-task findings rest on external submissions and standard metrics; one abstract claim lacks in-body support, but that is an evidence gap, not a circular derivation.

full rationale

The paper's central findings are empirical reports about externally submitted systems (Cloudsheep, HW-TSC, NLP2CT-UM, NTU, SJTU-LoveFiction) measured against external baselines (Google Translate, GPT-4) and standard metrics (sacreBLEU, chrF, TER, COMET, d-BLEU) plus human annotation. No parameter is fitted and then renamed as a prediction; the ranking is produced by human judgments collected under a rubric described in Table 6, with 0.86 Cohen's kappa reported. The dataset is released by the organizers and previous organizers' papers are cited for continuity (e.g., Wang et al., 2023c), but these citations are contextual rather than load-bearing: the numerical claims in Tables 3-5 would stand or fall on the submitted outputs and evaluation procedures, not on the cited prior work. The only concern of note is not circularity: Abstract finding (3), about the sole Constrained Track system performing close to the best Unconstrained system, is unreconstructable from the body, because Table 2 marks all five submissions with flag N and no constrained row appears in Tables 3-5; Section 1 merely says 'most teams choose Unconstrained Track.' This is an internal-support/consistency gap that weakens that particular claim, but it does not constitute a self-referential derivation. Similarly, the placeholder 'xxx' in Section 3.2 and the inconsistent evaluator count (two vs three) are reporting flaws, not circularity. Accordingly, the circularity score is minimal.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. Its claims depend on standard evaluation assumptions: automatic metrics such as d-BLEU approximate document-level quality, and the custom human rubric is a reliable measure of translation quality. These assumptions are standard in MT evaluations but are not independently validated here.

assumptions (2)
  • domain assumption Human evaluation scores (0-5) for general and discourse quality are a valid ground truth for ranking systems.
    The paper bases the official ranking on human judgments, yet only a small number of evaluators scored Zh-En outputs, and the scoring window size is unspecified (Section 3.2).
  • domain assumption d-BLEU is a meaningful automatic proxy for document-level translation quality.
    The paper uses d-BLEU as the headline document-level metric, and the Zh-De and Zh-Ru rankings rely entirely on it (Section 3.1, Table 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation." pith.science (2026). https://pith.science/paper/HVTR4256

@misc{pith2026241211732,
  author       = {Pith},
  title        = {Pith review of: Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HVTR4256}},
  note         = {Machine review of arXiv:2412.11732}
}
read the original abstract

Following last year, we have continued to host the WMT translation shared task this year, the second edition of the Discourse-Level Literary Translation. We focus on three language directions: Chinese-English, Chinese-German, and Chinese-Russian, with the latter two ones newly added. This year, we totally received 10 submissions from 5 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. We release data, system outputs, and leaderboard at https://www2.statmt.org/wmt24/literary-translation-task.html.

Figures

Figures reproduced from arXiv: 2412.11732 by the authors.

Figure 1
Figure 1. Data statistics of the GuoFeng Webnovel Corpus V2 in Chinese [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 22 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Li An, Linghao Jin, and Xuezhe Ma. 2023. Max-isi system at wmt23 discourse-level literary translation task. In Proceedings of the Eighth Conference on Machine Translation, pages 282--286

    work page 2023

  4. [4]

    Kechen Li, Yaotian Tao, Hongyi Huang, and Tianbo Ji. 2024. Linchance-ntu for unconstrained wmt2024 literary translation. In Proceedings of the Ninth Conference on Machine Translation, pages 987--992

    work page 2024

  5. [5]

    Lisa Liu, Ryan Liu, Angela Tsai, and Jingbo Shang. 2024 a . Cloudsheep system for wmt24 discourse-level literary translation. In Proceedings of the Ninth Conference on Machine Translation, pages 960--966

    work page 2024

  6. [6]

    Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726--742

    2020

  7. [7]

    Yuchen Liu, Yutong Yao, Runzhe Zhan, Yuchu Lin, and Derek F Wong. 2024 b . Noveltrans: System for wmt24 discourse-level literary translation. In Proceedings of the Ninth Conference on Machine Translation, pages 980--986

    work page 2024

  8. [8]

    Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradum \`a tica , (12):0455--463

    2014

Show all 33 references
  1. [9]

    Wang Longyue. 2019. Discourse-aware neural machine translation. Ph.D. thesis, Dublin City University

    2019

  2. [10]

    Fabien Lopez, Gabriela Gonz \'a lez-S \'a ez, Damien Hansen, Mariam Nakhl \'e , Behnoosh Namdarzadeh, Marco Dinarelli, Emmanuelle Esperan c a-Rodier, Sui He, Sadaf Mohseni, Caroline Rossi, et al. 2023. The make-nmtviz system description for the wmt23 literary task

    2023

  3. [11]

    Yuanchang Luo, Jiaxin Guo, Daimeng Wei, Hengchao Shang, Zongyao Li, Zhanglin Wu, Zhiqiang Rao, Shaojun Li, Jinlong Yang, and Hao Yang. 2024. Context-aware and style-related incremental decoding framework for discourse-level literary translation. In Proceedings of the Ninth Con...

    2024

  4. [13]

    Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023 b . New trends in machine translation using large language models: Case examples with chatgpt. arXiv preprint arXiv:2305.01181

    2023 arXiv

  5. [14]

    Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276--282

    2012

  6. [15]

    Lingfeng Ming, Bo Zeng, Chenyang Lyu, Tianqi Shi, Yu Zhao, Xue Yang, Yefeng Liu, Yiyu Wang, Linlong Xu, Yangyang Liu, et al. 2024. Marco-llm: Bridging languages via massive multilingual training for cross-lingual enhancement. arXiv preprint arXiv:2412.04003

    2024 arXiv

  7. [16]

    Jianhui Pang, Fanghua Ye, Longyue Wang, Dian Yu, Derek F Wong, Shuming Shi, and Zhaopeng Tu. 2024. Salute the classic: Revisiting challenges of machine translation in the age of large language models. arXiv preprint arXiv:2401.08350

    2024 arXiv

  8. [17]

    Maja Popovi \'c . 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation

    2015

  9. [18]

    Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers

    2018

  10. [19]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET : A neural framework for MT evaluation. In EMNLP

    2020

  11. [20]

    Mathew Snover, Bonnie Dorr, Richard Schwartz, John Makhoul, Linnea Micciulla, and Ralph Weischedel. 2006. A study of translation error rate with targeted human annotation. In Proceedings of the Association for Machine Transaltion in the Americas (AMTA)

    2006

  12. [21]

    Haoxiang Sun, Tianxiang Hu, Ruize Gao, Jialong Tang, Pei Zhang, Baosong Yang, and Rui Wang. 2024. Sjtu lovefiction’s system for wmt24 discourse-level literary translation. In Proceedings of the Ninth Conference on Machine Translation

    2024

  13. [22]

    Longyue Wang, Zefeng Du, Wenxiang Jiao, Chenyang Lyu, Jianhui Pang, Leyang Cui, Kaiqiang Song, Derek Wong, Shuming Shi, and Zhaopeng Tu. 2024. Benchmarking and improving long-text translation with large language models. In Findings of the Association for Computational Linguist...

    2024

  14. [23]

    Longyue Wang, Zefeng Du, Donghuai Liu, Cai Deng, Dian Yu, Haiyun Jiang, Yan Wang, Leyang Cui, Shuming Shi, and Zhaopeng Tu. 2023 a . Disco-bench: A discourse-aware evaluation benchmark for language modelling. arXiv preprint arXiv:2307.08074

    2023 arXiv

  15. [24]

    Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023 b . Document-level machine translation with large language models. arXiv preprint arXiv:2304.02210

    2023 arXiv

  16. [25]

    Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Liting Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan, and Shuming Shi. 2023 c . https://doi.org/10.18653/v1/2023.wmt-1.3 Findings o...

    2023 doi

  17. [26]

    Longyue Wang, Zhaopeng Tu, Xing Wang, Li Ding, Liang Ding, and Shuming Shi. 2020. Tencent ai lab machine translation systems for wmt20 chat translation task. In Proceedings of the fifth conference on machine translation, pages 483--491

    2020

  18. [27]

    Longyue Wang, Zhaopeng Tu, Xing Wang, and Shuming Shi. 2019. One model to learn both: Zero pronoun prediction and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua...

    2019

  19. [28]

    Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In EMNLP

    2017

  20. [29]

    Minghao Wu, Jiahao Xu, and Longyue Wang. 2024 a . Transagents: Build your translation company with language agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 131--141

    2024

  21. [30]

    Minghao Wu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. 2024 b . (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts. arXiv preprint arXiv:2405.11804

    2024 arXiv

  22. [31]

    Yuhao Xie, Zongyao Li, Zhanglin Wu, Daimeng Wei, Xiaoyu Chen, Zhiqiang Rao, Shaojun Li, Hengchao Shang, Jiaxin Guo, Lizhi Lei, et al. 2023. Hw-tsc’s submissions to the wmt23 discourse-level literary translation shared task. In Proceedings of the Eighth Conference on Machine Tr...

    2023

  23. [32]

    Longyue Wang Zefeng Du, Minghao Wu. 2023. Chinese-llama-2. https://github.com/longyuewangdcu/Chinese-Llama-2

    2023

  24. [33]

    Anqi Zhao, Kaiyu Huang, Hao Yu, and Degen Huang. 2023. Dutnlp system for the wmt2023 discourse-level literary translation. In Proceedings of the Eighth Conference on Machine Translation, pages 296--301

    2023

  25. [34]

    Shaolin Zhu and Deyi Xiong. 2023. Tjunlp: System description for the wmt23 literary task in chinese to english translation direction. In Proceedings of the Eighth Conference on Machine Translation, pages 307--311

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.