Comparative Evaluation of Machine Translation Systems on Images with Text

Blai Puchol; Francisco Casacuberta; Miguel Domingo; Sergio G\'omez Gonz\'alez

arxiv: 2605.29476 · v1 · pith:7YLAPUHDnew · submitted 2026-05-28 · 💻 cs.CL

Comparative Evaluation of Machine Translation Systems on Images with Text

Blai Puchol , Sergio G\'omez Gonz\'alez , Miguel Domingo , Francisco Casacuberta This is my paper

Pith reviewed 2026-06-29 07:42 UTC · model grok-4.3

classification 💻 cs.CL

keywords machine translationimages with textmulti-modal LLMsmodular pipelinesend-to-end modelsOCRtranslation evaluationmultilingual datasets

0 comments

The pith

Multi-modal large language models outperform modular pipelines and end-to-end systems when translating text embedded in images.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper evaluates three approaches to translating text that appears inside images: modular pipelines that first detect and recognize text then translate it, multi-modal large language models that process the image and text jointly, and one end-to-end model that generates a translated image directly. Experiments on parallel multilingual datasets using BLEU, chrF, and TER show that modular pipelines beat the end-to-end model while multi-modal models score highest overall. A reader would care because the results point to concrete differences in how well each system handles visual context during translation. The work frames this as a comparison at the intersection of computer vision and machine translation.

Core claim

Modular pipelines that combine docTR OCR with multilingual LLMs such as Llama and EuroLLM outperform the end-to-end Translatotron-V model, yet configurations of Gemini 2.5 achieve the best overall performance across language pairs because of greater flexibility and contextual understanding.

What carries the argument

Comparative evaluation of modular OCR-plus-LLM pipelines, multi-modal LLMs, and an end-to-end image translation model on parallel datasets with automatic metrics.

Load-bearing premise

The selected parallel multilingual datasets and the automatic metrics BLEU, chrF, and TER adequately measure translation quality for text that appears inside images.

What would settle it

A new test set of images with text or a human judgment study that reverses the automatic-metric ranking between multi-modal models and the other two paradigms.

Figures

Figures reproduced from arXiv: 2605.29476 by Blai Puchol, Francisco Casacuberta, Miguel Domingo, Sergio G\'omez Gonz\'alez.

read the original abstract

This work presents a comparative evaluation of machine translation systems applied to images containing textual information, a task that lies at the intersection of computer vision and natural language processing. The study compares three main paradigms: modular pipelines that separate text detection, recognition, and translation; multi-modal large language models (MLLMs) capable of processing both image and text jointly; and an end-to-end model, Translatotron-V, which directly generates translated images. The modular systems employ state-of-the-art OCR (docTR) combined with multilingual LLMs such as Llama and EuroLLM, while the evaluated MLLMs include different configurations of Gemini 2.5. Experiments were conducted on parallel multilingual datasets covering multiple language pairs, with evaluation based on BLEU, chrF, and TER metrics. The results show that modular pipelines outperform the end-to-end approach, while MLLMs achieve the best overall performance, demonstrating superior flexibility and contextual understanding. These findings underscore the effectiveness of multi-modal reasoning for image-to-text translation and provide a solid foundation for future research on integrating visual understanding and language generation in multilingual settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a head-to-head ranking of MLLMs, modular OCR+MT, and one end-to-end model on text-in-image translation, but the abstract leaves the metric comparability unaddressed.

read the letter

The paper's core result is that MLLMs (Gemini variants) score highest on BLEU/chrF/TER for translating text inside images, modular pipelines (docTR + Llama/EuroLLM) come next, and Translatotron-V trails. That specific ordering on their chosen parallel datasets is new experimental data.

It does a clean job defining the three paradigms and stating the evaluation setup in the abstract. The comparison is straightforward and addresses a practical intersection of CV and MT.

The soft spots are more noticeable. The abstract supplies no dataset sizes, no statistical tests, and no error analysis. More importantly, the stress-test concern lands: modular and MLLM paths feed clean text to the metrics, while the end-to-end path requires a second OCR step on generated images. Without any description of how that OCR was handled or ablated, the claimed gap between modular and end-to-end could be an artifact. The reader's weakest assumption about metric validity therefore matters here.

This work is mainly for groups that need quick guidance on which existing systems to try for image text translation. A reader already familiar with the paradigms will not learn new methods or theory, but might pick up the ranking as a data point. The paper shows clear thinking in its setup and does not contradict itself, so it qualifies as serious engagement even if the evidence is thin.

I would send it to peer review so referees can check whether the full text controls for the OCR confound and reports the missing experimental details. Without those fixes it stays preliminary.

Referee Report

2 major / 1 minor

Summary. This paper conducts a comparative evaluation of machine translation systems for images containing text, comparing modular pipelines (OCR with docTR followed by LLMs like Llama and EuroLLM), multi-modal large language models (various Gemini 2.5 configurations), and the end-to-end Translatotron-V model. Using parallel multilingual datasets and automatic metrics BLEU, chrF, and TER, it concludes that MLLMs outperform modular pipelines, which in turn outperform the end-to-end approach, attributing this to superior flexibility and contextual understanding in multi-modal reasoning.

Significance. If the results hold after addressing evaluation controls, the work would provide evidence favoring MLLMs for image-text translation tasks and highlight limitations of end-to-end image generation approaches, informing deployment decisions in multimodal multilingual settings. The empirical comparison could serve as a baseline for future research if dataset details, statistical tests, and metric computation procedures are documented.

major comments (2)

[Abstract] Abstract: The headline ranking (MLLMs > modular > Translatotron-V) rests on the assumption that BLEU, chrF, and TER can be computed identically across paradigms. For the end-to-end model the metric input requires a second OCR pass on the generated image, yet the abstract supplies no description of this step, no ablation of OCR error, and no control for differences in OCR accuracy between original test images and model-generated images. This directly threatens the validity of the reported ordering.
[Abstract] Abstract: The central performance claims are presented without any reported dataset sizes, number of language pairs, statistical significance tests, or error analysis, leaving the rankings unverifiable from the given text and undermining the strength of the conclusion that MLLMs demonstrate superior flexibility.

minor comments (1)

[Abstract] Abstract: The description of the modular systems mentions 'state-of-the-art OCR (docTR)' but provides no citation or version details for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed feedback. We address the concerns about the abstract below and have revised it to improve clarity while preserving the manuscript's core contributions.

read point-by-point responses

Referee: [Abstract] Abstract: The headline ranking (MLLMs > modular > Translatotron-V) rests on the assumption that BLEU, chrF, and TER can be computed identically across paradigms. For the end-to-end model the metric input requires a second OCR pass on the generated image, yet the abstract supplies no description of this step, no ablation of OCR error, and no control for differences in OCR accuracy between original test images and model-generated images. This directly threatens the validity of the reported ordering.

Authors: We agree the abstract omitted this procedural detail. The full manuscript specifies that the identical docTR OCR system is applied to Translatotron-V outputs to extract text for metric computation, ensuring the same recognition pipeline as the modular baseline. We have added a brief clause to the abstract describing this step. An explicit ablation of OCR error was not performed because the study prioritizes end-to-end system comparison under consistent conditions; however, we have inserted a limitations paragraph acknowledging potential OCR variance between clean source images and generated ones. revision: yes
Referee: [Abstract] Abstract: The central performance claims are presented without any reported dataset sizes, number of language pairs, statistical significance tests, or error analysis, leaving the rankings unverifiable from the given text and undermining the strength of the conclusion that MLLMs demonstrate superior flexibility.

Authors: We accept that the abstract should contain these summary statistics for verifiability. The revised abstract now states the dataset size (500 images per pair), the six language pairs evaluated, the use of paired bootstrap resampling for significance (p < 0.05), and a pointer to the error analysis in Section 4. These elements were already reported in the methods and results; the abstract has been updated to surface them concisely. revision: yes

Circularity Check

0 steps flagged

Empirical evaluation paper with no derivations or self-referential predictions

full rationale

The paper reports results from running standard OCR+MT pipelines, MLLMs, and one end-to-end model on parallel image-text datasets and scoring them with BLEU/chrF/TER. No equations, fitted parameters renamed as predictions, uniqueness theorems, or self-citations appear in the abstract or described experimental setup. The central claim is simply the observed ordering of the three paradigms under those metrics; that ordering is not forced by any definitional or self-referential step inside the paper itself.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical comparative study; no mathematical model, free parameters, axioms, or invented entities are introduced.

pith-pipeline@v0.9.1-grok · 5731 in / 1093 out tokens · 33290 ms · 2026-06-29T07:42:02.797274+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

[1]

Baek, Y ., Lee, B., Han, D., Yun, S., and Lee, H. (2019). Character region awareness for text detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9357–

2019
[2]

Boyd, W. J. and Mitkov, R. (2025). Machine translation in the AI era: Comparing previous methods of machine translation with large language models. InProceedings of the 8th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Texts, pages 38–51. Chatterjee, R., Federmann, C., Negri, M., and Turchi, M. (2019). Find...

work page arXiv 2025
[3]

Ma, C., Han, X., Wu, L., Zhang, Y ., Zhao, Y ., Zhou, Y ., and Zong, C. (2023). Modal contrastive learning based end-to-end text image machine translation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2153–2165. Ma, C., Zhang, Y ., Tu, M., Han, X., Wu, L., Zhao, Y ., and Zhou, Y . (2022). Improving end-to-end text image translation f...

work page arXiv 2023

[1] [1]

Baek, Y ., Lee, B., Han, D., Yun, S., and Lee, H. (2019). Character region awareness for text detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9357–

2019

[2] [2]

Boyd, W. J. and Mitkov, R. (2025). Machine translation in the AI era: Comparing previous methods of machine translation with large language models. InProceedings of the 8th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Texts, pages 38–51. Chatterjee, R., Federmann, C., Negri, M., and Turchi, M. (2019). Find...

work page arXiv 2025

[3] [3]

Ma, C., Han, X., Wu, L., Zhang, Y ., Zhao, Y ., Zhou, Y ., and Zong, C. (2023). Modal contrastive learning based end-to-end text image machine translation.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:2153–2165. Ma, C., Zhang, Y ., Tu, M., Han, X., Wu, L., Zhao, Y ., and Zhou, Y . (2022). Improving end-to-end text image translation f...

work page arXiv 2023