Pith. sign in

REVIEW 5 major objections 6 minor 27 references

DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Model

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read A single fine-tuned LVLM, trained with multi-task learning and a two-stage perceptual chain-of-thought, can handle both OCR-based and OCR-free document image translation, with MBR decoding and post-processing adding consistent BLEU gains.

desk verdict A sensible competition system report with a useful unified DIMT recipe; the numbers are plausible but the headline MTL-PCOT gain is under-supported on the OCR-based track. read the letter →

arxiv 2504.17315 v1 pith:OCVAPGCB submitted 2025-04-24 cs.CV cs.AI

classification cs.CVcs.AI
keywords documentimagemachinetranslationlargevision-languagemodelmulti-tasklearningperceptualchain-of-thoughtminimumBayesianriskdecodingBLEUcomplexlayoutsOCR-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper describes a competition system for end-to-end document image machine translation on complex layouts. It claims that fine-tuning an open large vision-language model with multi-task learning plus a perceptual chain-of-thought training format makes one model handle both OCR-based and OCR-free translation, and that minimum Bayesian risk decoding and lightweight post-processing add further BLEU gains. A sympathetic reader would care because a unified OCR/OCR-free document translation model removes the need for separate OCR, layout-analysis, and translation pipelines, which is a practical bottleneck for translating manuals, reports, and archival documents.

What carries the argument

Perceptual chain-of-thought (PCOT) is a two-stage training and prompting format: the model first produces a sorted OCR result from the document image, then produces the translation conditioned on that OCR, so layout perception and cross-lingual transfer are separated explicitly. Multi-task learning mixes these staged examples with OCR-free translation examples in one SFT session. At inference, minimum Bayesian risk (MBR) decoding generates beam-search outputs and diverse sampled candidates, scores them by pairwise BLEU similarity, and selects the highest-scoring one; post-processing then collapses runs of more than ten repeated special symbols, skips overly complex tables, and normalises spaces in Chinese output.

What would settle it

Train an otherwise identical model on the same data but replace the two-stage PCOT prompt with a direct single-prompt 'translate this document image' format. If the direct model matches or beats MTL-PCOT BLEU on the DIMT25 test sets, the perceptual chain-of-thought stage is not carrying the claimed layout-comprehension advantage.

Watch

Extended reading notes

Core claim

The paper's central claim is that combining multi-task learning (MTL) with a perceptual chain-of-thought (PCOT) during supervised fine-tuning, on top of an open large vision-language model, produces a single system that competes in both DIMT25 tracks: the OCR-based track and the OCR-free track. In the reported BLEU results, MTL-PCOT SFT beats single-task direct SFT on validation sets, MBR decoding and post-processing each add further gains on the test sets, and the larger 8-billion-parameter model outperforms the 1-billion-parameter model under identical methods.

Load-bearing premise

The reported gains rest on the assumption that the DIMT25 benchmark's reference translations and BLEU scoring faithfully measure document image translation quality on complex layouts, because the model is fine-tuned and selected entirely on that benchmark.

Editorial extensions

If this is right

  • A single fine-tuned LVLM can serve both OCR-based and OCR-free document translation, eliminating the need to build and maintain separate OCR-to-MT pipelines.
  • MTL-PCOT combined SFT is reported to improve BLEU over single-task direct SFT on the DIMT25 validation and test sets.
  • MBR decoding and simple rule-based post-processing give complementary, repeatable BLEU gains on top of the SFT checkpoint.
  • Model capacity matters: the 8-billion-parameter model outperforms the 1-billion-parameter model under the same training and decoding setup.
  • The framework is presented as a reproducible technical recipe, covering model choice, data format, training configuration, and decoding strategy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if PCOT's two-stage format is what enables layout comprehension, the same separation of a perceptual pass from a linguistic pass could transfer to other document-based generation tasks such as form filling, diagram captioning, or table summarisation.
  • Editorial inference: because MBR selection scores candidates by BLEU, the chosen output may be the one most stylistically similar to the model's own generations; a reference-free metric or human rating could rank the candidate sets differently and might change which decoding strategy is best.
  • Editorial inference: the post-processing rule that truncates long runs of special symbols suggests a testable failure mode, namely that the model over-generates table decorations, so training on augmented examples that enforce table-format fidelity could make the heuristic unnecessary.
  • Editorial inference: a direct test of the PCOT claim is to feed the model its own OCR output as text-only input; if translation quality does not drop, the perceptual stage is not doing the load-bearing work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper is the system report of HW-TSC for the ICDAR 2025 DIMT25 competition, which consists of two tracks: an OCR-based track (DIMT-WebDoc-300K) and an OCR-free track (DIMT-arXiv-124K). The authors start from the open-source LVLMs InternVL2.5-1B-MPO and InternVL2.5-8B-MPO and perform full-parameter SFT with a multi-task training set in which prompts either ask the model to sort a provided OCR result and translate it (Track 1) or to extract text from the image and translate it (Track 2), each in a two-stage perceptual chain-of-thought (PCOT) format. At inference they apply minimum Bayes-risk-style decoding with BLEU as the utility metric over a beam-search output plus 10 sampled candidates, followed by rule-based post-processing (cap repeated special symbols, drop outputs for overly complex tables, normalize spaces). The experiments report BLEU on the competition's validation and test sets; the paper claims that (1) the combined MTL-PCOT SFT improves over single-task SFT, (2) MBR decoding and post-processing give complementary gains, and (3) the 8B model outperforms the 1B model.

Significance. The paper is a competently written engineering report whose strengths are its concrete reproducibility details — the exact HuggingFace checkpoints, training hyperparameters (learning rate 4e-5, 5 epochs, batch size 4 with gradient accumulation 4, DeepSpeed zero3), the data format listing in Fig. 1, and a clear compliance statement — and the fact that all results are measured on an external competition test set, which avoids the circularity of fitting a private benchmark. If the reported improvements were statistically established, the unified two-track training recipe and the PCOT prompt format would be a useful baseline for practitioners building end-to-end document image translation systems. However, the scientific significance is limited: the central comparisons are single-run BLEU deltas of roughly 1.5–3.6 points with no uncertainty quantification, the single-task SFT baseline is absent for the OCR-based track and for all test-set columns, and the MTL and PCOT components are never ablated separately, so the specific attribution claims in Section 4.2 are not yet supported.

major comments (5)
  1. [§4.2, Table 1] The headline finding (1) — that the MTL-PCOT combined SFT method outperforms single-task direct SFT — is not established for Track 1, and is not established on any test set. In Table 1 the single-task SFT rows report only Valid-MT values (67.21 and 72.74 for the 1B and 8B models on Track 1; 59.81 and 64.24 on Track 2), and every SFT cell for Test-OCR and Test-MT is marked '/'. The only SFT-versus-MTL-PCOT comparisons available are therefore four Valid-MT deltas (3.60, 2.98, 2.36, and 1.53 BLEU), and no comparison of MTL-PCOT against single-task SFT exists on the test sets of either track. Because finding (1) is stated for 'the DIMT25 task' as a whole, the paper should either add the missing Track-1 SFT runs and test-set SFT baselines or explicitly restrict the claim to Track-2 validation.
  2. [§4.2, Table 1] All BLEU scores in Table 1 come from a single run and are reported without error bars, confidence intervals, or significance tests, while the effects claimed are small: the Track-2 MTL-PCOT advantage is 1.53–2.36 BLEU on validation, and the largest post-processing gain is about 2.22 BLEU on Track 1. On a 1,000-document test set such differences are plausibly within evaluation noise, especially since the same systems drop by 7–10 BLEU between validation and test (e.g., 8B Track 1: 75.72 to 65.32; 8B Track 2: 65.77 to 58.57), indicating substantial distribution shift. The authors should provide document-level bootstrap confidence intervals or a paired significance test for the SFT-versus-MTL-PCOT, MBR, and post-processing comparisons before drawing the conclusions stated in Section 4.2.
  3. [§2.1, §4.2] The design of the comparison does not permit the claimed attribution to the 'MTL-PCOT combined' method. The single-task SFT and the proposed training differ simultaneously in the number of task formats in the data, in the prompt templates, and in the addition of the two-stage PCOT output format, and no ablation varies MTL and PCOT independently. Since Section 2.1 motivates both components as load-bearing parts of the method, the paper should present at least minimal ablations — for example, MTL without PCOT and PCOT without MTL — to show which ingredient (or their interaction) produces the Table 1 gains.
  4. [§2.2, Table 1] MBR decoding in this system selects the final hypothesis using BLEU as the pairwise utility (with t=0.7, p=0.95, and 10 sampled candidates fixed without sensitivity analysis), and BLEU is also the metric used to report every result in Table 1. The reported MBR gains — for instance Track 1, 8B, Test-MT rising from 65.32 to 68.26 — may therefore reflect selection toward the evaluation metric rather than an improvement in translation quality. A concrete check would be to re-evaluate the selected hypotheses with at least one auxiliary metric not used in selection (e.g., COMET or chrF), and to vary the sampling temperature and candidate count to show the choice is not a tuned artifact.
  5. [§2.2, Table 1] The post-processing rule that 'removes translation outputs for overly complex tables' is ambiguous in a way that bears directly on the reported 1.5–2.2 BLEU gains: if the model's output for a table is deleted while the reference still contains that table, BLEU would normally be penalized, so the observed gains imply either that removed documents are excluded from scoring or that the reference side is filtered identically. The paper should state how dropped outputs interact with the official evaluation script; otherwise the post-processing contribution in finding (2) cannot be interpreted.
minor comments (6)
  1. [§2.2] The term 'minimum Bayesian decoding' should be 'minimum Bayes risk (MBR) decoding', and reference [9] is a paper on translation hypothesis ensembling with LLMs rather than the primary source of the MBR method; a canonical citation (e.g., the MBR decoding work of Kumar et al., 2023, or Eikema and Aziz, 2020) is needed so the method definition can be located precisely.
  2. [§3.2] Reference [27] is cited as the source of the MOTBench benchmark, but the reference as printed is the authors' own menu-OCR and translation paper, which does not define MOTBench; the correct benchmark source should be cited, or the sentence should be reworded to describe what [27] actually reports.
  3. [§1, Table 1, Fig. 1] The mapping between the two tracks and the 'OCR-based/OCR-free' terminology is never stated explicitly: Table 1 shows OCR columns only for Track 1, and the two prompts in Fig. 1 differ in whether an unsorted OCR result is provided, but the text does not connect these facts. A sentence tying Track 1/Track 2 to the TASK1/TASK2 prompts would remove the ambiguity.
  4. [References] References 21–23 and 27 are all self-citations of the same team; adding independent references for MBR in machine translation and for the MOTBench claim would make the related-work coverage more balanced.
  5. [Fig. 1] Figure 1 is nearly illegible because the prompt text is compressed into the two-column layout; a full-width figure or a separate listing of the exact prompts and outputs is needed for reproducibility, especially since the paper's central claims depend on the PCOT format.
  6. [Table 2] Table 2 reports dataset sizes but not the language directions or document categories of the two tracks; a sentence describing the layout complexity of DIMT-WebDoc-300K relative to DIMT-arXiv-124K would help readers interpret why Track 1 and Track 2 scores differ so markedly, and why validation-to-test transfer is so weak.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical results on an external DIMT25 test set, and MBR's internal BLEU-based selection does not reduce to the evaluation metric.

full rationale

The paper makes no first-principles derivation whose conclusion is already contained in its inputs. The central claims are empirical: fine-tuning InternVL2.5 models with MTL-PCOT, then applying MBR and post-processing, is reported to improve BLEU on the DIMT25 competition's held-out test splits. Model selection and evaluation use the competition's reference translations, which are external to the training procedure. The MBR step uses BLEU to compute pairwise similarity among model-generated candidate hypotheses, not to compare against the test references, so it is an inference-time objective rather than a restatement of the evaluation metric. The post-processing rules are fixed heuristics, not fitted parameters renamed as predictions. The prior self-citations [21-23, 27] support the general usefulness of MBR and related benchmarks, but the load-bearing evidence for this paper's improvement claims is its own Table 1 on the external DIMT25 test set, so these citations are not circularly relied upon. Concerns about missing Track 1 single-task SFT baselines, unrepeated BLEU deltas, and lack of error bars are evidentiary weaknesses, not circularity. The method description is self-contained relative to the claims made.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The report makes no mathematical derivation. Its central claims are empirical improvements on a competition benchmark, so the load-bearing inputs are the pretrained base model, the competition data and BLEU metric, and the assumed positive effects of MTL and PCOT. The hand-selected MBR and post-processing parameters are the only numerical knobs introduced by the authors, and none are sensitivity-tested.

free parameters (6)
  • MBR sampling temperature = 0.7
    Used for diverse candidate generation in Section 2.2; no sensitivity analysis is reported.
  • MBR nucleus sampling probability = 0.95
    Used with the temperature in Section 2.2; chosen by hand without sensitivity analysis.
  • MBR candidate count = 10
    Ten diverse samples are generated for MBR selection in Section 2.2; no justification is given for the count.
  • Post-processing special-symbol limit = 10
    The system reduces runs of repeated special symbols longer than 10 to 10; this threshold was set after inspecting model outputs in Section 2.2.
  • Initial learning rate = 4e-5
    SFT hyperparameter listed in Section 4.1; not fitted and no sweep is reported.
  • Training epochs = 5
    SFT schedule listed in Section 4.1; no early stopping or epoch sensitivity is reported.
assumptions (5)
  • domain assumption InternVL2.5-MPO base models supply sufficient OCR and layout understanding for document translation.
    Section 3.2 selects these models on the basis of MOTBench and no ablation against other LVLM bases is performed.
  • domain assumption The DIMT25 competition data and BLEU metric faithfully measure document translation quality.
    Sections 3.1 and 4.2 rely exclusively on this benchmark; no human evaluation or second metric is reported.
  • domain assumption Multi-task training on OCR-sorting and translation tasks yields positive transfer rather than task interference.
    Section 2.1 asserts MTL benefits, but the paper never ablates MTL, PCOT, or training-data composition separately.
  • domain assumption The perceptual chain-of-thought prompt makes the model first perceive layout and text, then translate, as intended.
    Section 2.1 and Figure 1 describe the two-stage prompt; no intermediate-output or attention analysis verifies the mechanism.
  • domain assumption The open-source InternVL framework and DeepSpeed zero3-offload behave correctly for full fine-tuning.
    Section 4.1 states these tools; no custom implementation or verification is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Model." pith.science (2026). https://pith.science/paper/OCVAPGCB

@misc{pith2026250417315,
  author       = {Pith},
  title        = {Pith review of: DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCVAPGCB}},
  note         = {Machine review of arXiv:2504.17315}
}
read the original abstract

This paper presents the technical solution proposed by Huawei Translation Service Center (HW-TSC) for the "End-to-End Document Image Machine Translation for Complex Layouts" competition at the 19th International Conference on Document Analysis and Recognition (DIMT25@ICDAR2025). Leveraging state-of-the-art open-source large vision-language model (LVLM), we introduce a training framework that combines multi-task learning with perceptual chain-of-thought to develop a comprehensive end-to-end document translation system. During the inference phase, we apply minimum Bayesian decoding and post-processing strategies to further enhance the system's translation capabilities. Our solution uniquely addresses both OCR-based and OCR-free document image translation tasks within a unified framework. This paper systematically details the training methods, inference strategies, LVLM base models, training data, experimental setups, and results, demonstrating an effective approach to document image machine translation.

Figures

Figures reproduced from arXiv: 2504.17315 by the authors.

Figure 1
Figure 1. Training Data Organization Structure for Our Method. B Training Data Size [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 16 canonical work pages

  1. [1]

    LayoutDIT: Layout-aware end-to-end document image translation with multi-step conductive decoder[C]//Findings of the Associa- tion for Computational Linguistics: EMNLP 2023

    Zhang Z, Zhang Y, Liang Y, et al. LayoutDIT: Layout-aware end-to-end document image translation with multi-step conductive decoder[C]//Findings of the Associa- tion for Computational Linguistics: EMNLP 2023. 2023: 10043-10053

  2. [2]

    Liang Y, Zhang Y, Ma C, et al. Document image machine translation with dynamic multi-pre-trained models assembling[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024: 7084-7095

  3. [3]

    Document Image Analysis for Text Extraction and Trans- lation

    Thendral, R., et al. "Document Image Analysis for Text Extraction and Trans- lation." 2023 4th International Conference on Intelligent Technologies (CONIT). IEEE, 2024

  4. [4]

    Deep learning

    LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436-444

  5. [5]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Chen, Zhe, et al. "Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling." arXiv preprint arXiv:2412.05271 (2024)

  6. [6]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Wang, Weiyun, et al. "Enhancing the reasoning ability of multimodal large language models via mixed preference optimization." arXiv preprint arXiv:2411.10442 (2024)

  7. [7]

    A survey on multi-task learning

    Zhang, Yu, and Qiang Yang. "A survey on multi-task learning." IEEE transactions on knowledge and data engineering 34.12 (2021): 5586-5609

  8. [8]

    Chain-of-thought prompting elicits reasoning in large language models

    Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837

Show all 27 references
  1. [9]

    An empirical study of translation hypothesis ensembling with large language models

    Farinhas, António, José GC de Souza, and André FT Martins. "An empirical study of translation hypothesis ensembling with large language models." arXiv preprint arXiv:2310.11430 (2023)

  2. [10]

    Optical character recogni- tion

    Mithe, Ravina, Supriya Indalkar, and Nilam Divekar. "Optical character recogni- tion."Internationaljournalofrecenttechnologyandengineering(IJRTE)2.1(2013): 72-75

  3. [11]

    A survey on evaluation of large language models

    Chang, Yupeng, et al. "A survey on evaluation of large language models." ACM transactions on intelligent systems and technology 15.3 (2024): 1-45

  4. [12]

    ChatGPT for good? On opportunities and challenges of large language models for education

    Kasneci, Enkelejda, et al. "ChatGPT for good? On opportunities and challenges of large language models for education." Learning and individual differences 103 (2023): 102274

  5. [13]

    Continual pre-training of large language models: How to (re) warm your model?

    Gupta, Kshitij, et al. "Continual pre-training of large language models: How to (re) warm your model?." arXiv preprint arXiv:2308.04014 (2023)

  6. [14]

    How abilities in large language models are affected by supervised fine-tuning data composition

    Dong, Guanting, et al. "How abilities in large language models are affected by supervised fine-tuning data composition." arXiv preprint arXiv:2310.05492 (2023)

  7. [15]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint arXiv:2204.05862 (2022)

  8. [16]

    Qwen2. 5-vl technical report

    Bai, Shuai, et al. "Qwen2. 5-vl technical report." arXiv preprint arXiv:2502.13923 (2025)

  9. [17]

    Gemini: a family of highly capable multimodal models

    Team, Gemini, et al. "Gemini: a family of highly capable multimodal models." arXiv preprint arXiv:2312.11805 (2023)

  10. [18]

    Perspectives and prospects on transformer architecture for cross-modal tasks with language and vision

    Shin, Andrew, Masato Ishii, and Takuya Narihira. "Perspectives and prospects on transformer architecture for cross-modal tasks with language and vision." Interna- tional journal of computer vision 130.2 (2022): 435-454

  11. [19]

    Machine learning paradigms

    Lampropoulos, Aristomenis S., and George A. Tsihrintzis. "Machine learning paradigms." Applications in recommender systems. Switzerland: Springer Interna- tional Publishing (2015). 6 Wu et al

  12. [20]

    A comparison of multi-task learning and single-task learning approaches

    Marquet, Thomas, and Elisabeth Oswald. "A comparison of multi-task learning and single-task learning approaches." International Conference on Applied Cryp- tography and Network Security. Cham: Springer Nature Switzerland, 2023

  13. [21]

    HW-TSC’s Submission to the CCMT 2024 Machine Transla- tion Tasks

    Wu, Zhanglin, et al. "HW-TSC’s Submission to the CCMT 2024 Machine Transla- tion Tasks." China Conference on Machine Translation. Singapore: Springer Nature Singapore, 2024

  14. [22]

    ChoosetheFinalTranslationfromNMTandLLMhypotheses Using MBR Decoding: HW-TSC’s Submission to the WMT24 General MT Shared Task

    Wu,Zhanglin,etal."ChoosetheFinalTranslationfromNMTandLLMhypotheses Using MBR Decoding: HW-TSC’s Submission to the WMT24 General MT Shared Task." arXiv preprint arXiv:2409.14800 (2024)

  15. [23]

    ImprovingtheQualityofIWLST2024CascadeOfflineSpeech Translation and Speech-to-Speech Translation via Translation Hypothesis Ensem- bling with NMT models and Large Language Models

    Wu,Zhanglin,etal."ImprovingtheQualityofIWLST2024CascadeOfflineSpeech Translation and Speech-to-Speech Translation via Translation Hypothesis Ensem- bling with NMT models and Large Language Models." Proceedings of the 21st International Conference on Spoken Language Translation...

  16. [24]

    COMET: A Neural Framework for MT Evaluation

    Rei, Ricardo, et al. "COMET: A Neural Framework for MT Evaluation." Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020

  17. [25]

    COMET-22: Unbabel-IST 2022 submission for the met- rics shared task

    Rei, Ricardo, et al. "COMET-22: Unbabel-IST 2022 submission for the met- rics shared task." Proceedings of the Seventh Conference on Machine Translation (WMT). 2022

  18. [26]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, Kishore, et al. "Bleu: a method for automatic evaluation of machine translation." Proceedings of the 40th annual meeting of the Association for Com- putational Linguistics. 2002

  19. [27]

    Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models

    Wu, Zhanglin, et al. "Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models" arXiv preprint arXiv:2504.13945 (2025). DIMT25@ICDAR2025: HW-TSC’s System Report 7 A Training Data Format <image>\nYour task is ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.