{"id":"ed1529a7-c4eb-4f03-86ca-df3952bc66fa","arxiv_id":"2504.17315","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning an open vision-language model with multi-task learning and perceptual chain-of-thought, plus MBR decoding, achieves strong BLEU scores on end-to-end document image translation at ICDAR 2025.","lead":"The Huawei translation team describes a single vision-language model that reads a document image and directly outputs translated text, handling both OCR-assisted and OCR-free document translation tracks at an ICDAR 2025 competition. A generalist reader can see how a single fine-tuned model replaces the usual two-stage OCR-then-translate pipeline, and how small inference tricks add a few points of quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing Track1 single-task SFT baseline and unrepeated BLEU deltas leave the core MTL-PCOT improvement claim unestablished for the OCR-based track.","rationale":"The reader's weakest_assumption focused on benchmark validity and on whether PCOT induces genuine layout comprehension. I agree those are relevant, but the more immediate, manuscript-internal problem is that the only direct evidence for the headline claim (MTL-PCOT better than single-task SFT) comes from Track2, with Track1 SFT baselines absent from Table 1 and with no statistical quantification. This is a concrete omission rather than a speculation about the benchmark. The paper remains a plausible competition report, and the MBR/post-processing gains are internally consistent, so the appropriate verdict is conditional acceptance pending the missing baseline and variance analysis. My recommendation is therefore unchanged from the reader's CONDITIONAL verdict, but for a sharper and more actionable reason than the benchmark-validity concern alone.","tokens_in":4885,"tokens_out":4184,"duration_ms":40475,"concrete_test":"Re-run the exact single-task SFT and MTL-PCOT SFT configurations described in Section 4.1 on both Track1 and Track2, with at least 5 random seeds each, and report mean BLEU, standard deviation, and paired bootstrap confidence intervals on Test-OCR and Test-MT. Critically, fill in the missing Track1 single-task SFT row for both the 1B and 8B models. If MTL-PCOT does not beat single-task SFT on Track1 by more than the confidence interval, or if the Track2 deltas (2.35 and 1.08 BLEU) overlap zero, the central claim in Section 4.2(1) is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.2 item (1) is that \"Compared to single-task direct SFT, the MTL-PCOT combined SFT method demonstrates more significant performance improvement for LVLM on the DIMT25 task.\" Table 1, however, shows no single-task SFT results for Track1 for either model size (all cells are \"/\"). The only SFT-versus-MTL-PCOT comparison is on Track2, where the Test-MT deltas are 2.35 BLEU for the 1B model and 1.08 BLEU for the 8B model. These are single-run, unrepeated scores with no error bars, significance tests, or confidence intervals; on a 1K-document test set, such small BLEU differences may be within normal evaluation noise. Thus, even granting the benchmark's validity, the paper's headline improvement claim is not fully supported: it lacks evidence on Track1, and the Track2 evidence is statistically fragile. A secondary issue is that no ablation separates MTL from PCOT, so the \"combined\" label cannot be attributed to either component individually, but the primary load-bearing gap is the incomplete and statistically unquantified comparison against single-task SFT.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is the system report of HW-TSC for the ICDAR 2025 DIMT25 competition, which consists of two tracks: an OCR-based track (DIMT-WebDoc-300K) and an OCR-free track (DIMT-arXiv-124K). The authors start from the open-source LVLMs InternVL2.5-1B-MPO and InternVL2.5-8B-MPO and perform full-parameter SFT with a multi-task training set in which prompts either ask the model to sort a provided OCR result and translate it (Track 1) or to extract text from the image and translate it (Track 2), each in a two-stage perceptual chain-of-thought (PCOT) format. At inference they apply minimum Bayes-risk-style decoding with BLEU as the utility metric over a beam-search output plus 10 sampled candidates, followed by rule-based post-processing (cap repeated special symbols, drop outputs for overly complex tables, normalize spaces). The experiments report BLEU on the competition's validation and test sets; the paper claims that (1) the combined MTL-PCOT SFT improves over single-task SFT, (2) MBR decoding and post-processing give complementary gains, and (3) the 8B model outperforms the 1B model.","tokens_in":5122,"tokens_out":13508,"duration_ms":111254,"significance":"The paper is a competently written engineering report whose strengths are its concrete reproducibility details — the exact HuggingFace checkpoints, training hyperparameters (learning rate 4e-5, 5 epochs, batch size 4 with gradient accumulation 4, DeepSpeed zero3), the data format listing in Fig. 1, and a clear compliance statement — and the fact that all results are measured on an external competition test set, which avoids the circularity of fitting a private benchmark. If the reported improvements were statistically established, the unified two-track training recipe and the PCOT prompt format would be a useful baseline for practitioners building end-to-end document image translation systems. However, the scientific significance is limited: the central comparisons are single-run BLEU deltas of roughly 1.5–3.6 points with no uncertainty quantification, the single-task SFT baseline is absent for the OCR-based track and for all test-set columns, and the MTL and PCOT components are never ablated separately, so the specific attribution claims in Section 4.2 are not yet supported.","major_comments":[{"comment":"The headline finding (1) — that the MTL-PCOT combined SFT method outperforms single-task direct SFT — is not established for Track 1, and is not established on any test set. In Table 1 the single-task SFT rows report only Valid-MT values (67.21 and 72.74 for the 1B and 8B models on Track 1; 59.81 and 64.24 on Track 2), and every SFT cell for Test-OCR and Test-MT is marked '/'. The only SFT-versus-MTL-PCOT comparisons available are therefore four Valid-MT deltas (3.60, 2.98, 2.36, and 1.53 BLEU), and no comparison of MTL-PCOT against single-task SFT exists on the test sets of either track. Because finding (1) is stated for 'the DIMT25 task' as a whole, the paper should either add the missing Track-1 SFT runs and test-set SFT baselines or explicitly restrict the claim to Track-2 validation.","section":"§4.2, Table 1"},{"comment":"All BLEU scores in Table 1 come from a single run and are reported without error bars, confidence intervals, or significance tests, while the effects claimed are small: the Track-2 MTL-PCOT advantage is 1.53–2.36 BLEU on validation, and the largest post-processing gain is about 2.22 BLEU on Track 1. On a 1,000-document test set such differences are plausibly within evaluation noise, especially since the same systems drop by 7–10 BLEU between validation and test (e.g., 8B Track 1: 75.72 to 65.32; 8B Track 2: 65.77 to 58.57), indicating substantial distribution shift. The authors should provide document-level bootstrap confidence intervals or a paired significance test for the SFT-versus-MTL-PCOT, MBR, and post-processing comparisons before drawing the conclusions stated in Section 4.2.","section":"§4.2, Table 1"},{"comment":"The design of the comparison does not permit the claimed attribution to the 'MTL-PCOT combined' method. The single-task SFT and the proposed training differ simultaneously in the number of task formats in the data, in the prompt templates, and in the addition of the two-stage PCOT output format, and no ablation varies MTL and PCOT independently. Since Section 2.1 motivates both components as load-bearing parts of the method, the paper should present at least minimal ablations — for example, MTL without PCOT and PCOT without MTL — to show which ingredient (or their interaction) produces the Table 1 gains.","section":"§2.1, §4.2"},{"comment":"MBR decoding in this system selects the final hypothesis using BLEU as the pairwise utility (with t=0.7, p=0.95, and 10 sampled candidates fixed without sensitivity analysis), and BLEU is also the metric used to report every result in Table 1. The reported MBR gains — for instance Track 1, 8B, Test-MT rising from 65.32 to 68.26 — may therefore reflect selection toward the evaluation metric rather than an improvement in translation quality. A concrete check would be to re-evaluate the selected hypotheses with at least one auxiliary metric not used in selection (e.g., COMET or chrF), and to vary the sampling temperature and candidate count to show the choice is not a tuned artifact.","section":"§2.2, Table 1"},{"comment":"The post-processing rule that 'removes translation outputs for overly complex tables' is ambiguous in a way that bears directly on the reported 1.5–2.2 BLEU gains: if the model's output for a table is deleted while the reference still contains that table, BLEU would normally be penalized, so the observed gains imply either that removed documents are excluded from scoring or that the reference side is filtered identically. The paper should state how dropped outputs interact with the official evaluation script; otherwise the post-processing contribution in finding (2) cannot be interpreted.","section":"§2.2, Table 1"}],"minor_comments":[{"comment":"The term 'minimum Bayesian decoding' should be 'minimum Bayes risk (MBR) decoding', and reference [9] is a paper on translation hypothesis ensembling with LLMs rather than the primary source of the MBR method; a canonical citation (e.g., the MBR decoding work of Kumar et al., 2023, or Eikema and Aziz, 2020) is needed so the method definition can be located precisely.","section":"§2.2"},{"comment":"Reference [27] is cited as the source of the MOTBench benchmark, but the reference as printed is the authors' own menu-OCR and translation paper, which does not define MOTBench; the correct benchmark source should be cited, or the sentence should be reworded to describe what [27] actually reports.","section":"§3.2"},{"comment":"The mapping between the two tracks and the 'OCR-based/OCR-free' terminology is never stated explicitly: Table 1 shows OCR columns only for Track 1, and the two prompts in Fig. 1 differ in whether an unsorted OCR result is provided, but the text does not connect these facts. A sentence tying Track 1/Track 2 to the TASK1/TASK2 prompts would remove the ambiguity.","section":"§1, Table 1, Fig. 1"},{"comment":"References 21–23 and 27 are all self-citations of the same team; adding independent references for MBR in machine translation and for the MOTBench claim would make the related-work coverage more balanced.","section":"References"},{"comment":"Figure 1 is nearly illegible because the prompt text is compressed into the two-column layout; a full-width figure or a separate listing of the exact prompts and outputs is needed for reproducibility, especially since the paper's central claims depend on the PCOT format.","section":"Fig. 1"},{"comment":"Table 2 reports dataset sizes but not the language directions or document categories of the two tracks; a sentence describing the layout complexity of DIMT-WebDoc-300K relative to DIMT-arXiv-124K would help readers interpret why Track 1 and Track 2 scores differ so markedly, and why validation-to-test transfer is so weak.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this manuscript is a competition system report, and its evaluation is thin relative to the standard of a research paper: single-run BLEU differences of about 1.5–3.6 points, a missing Track-1 SFT baseline, and no MTL/PCOT ablation. The reference list contains a dense cluster of self-citations (21–23, 27), and one of them (ref 27) appears to be cited for a benchmark it does not define, which should be checked before publication. I also note that the paper's novelty is incremental engineering; if the intended venue is a formal journal rather than workshop proceedings, the authors should be asked to justify the contribution level. None of this indicates bad faith or data problems — the compliance statement is explicit and the experimental setup is transparent — but the evidence for the paper's three stated findings needs to be strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a competition system report, not a discovery paper. What is actually new is the specific PCOT prompt format and the multi-task SFT arrangement that lets one InternVL model handle both OCR-based and OCR-free document image translation. The paper is honest about its scope and the numbers in Table 1 are internally consistent. The gains from MBR decoding and post-processing show up consistently on all Test-MT cells, and the training data format is clearly described. That is a legitimate engineering contribution for the DIMT community.\n\nThe soft spots are real, but none of them are disqualifying. The stress-test note is right: the claim that MTL-PCOT beats single-task SFT is only supported on Track2, because all Track1 single-task SFT cells are missing. On Track2 the deltas are 2.35 and 1.08 BLEU, with no error bars or significance tests, so they could be noise. There is also no ablation separating MTL from PCOT, so the label \"combined\" is doing a lot of work. The MBR selection uses BLEU as the utility metric, the same metric used for evaluation, which biases the reported scores upward; that is a minor methodological wart, common in this literature. The self-citations for MBR are fine; they are prior work by the same group and MBR is standard. The absence of code and trained weights is the bigger reproducibility issue, but for a competition report it is not a fatal flaw.\n\nWho is this for? Practitioners working on document image translation who want a starting point for unified OCR/translation models. It deserves a serious referee because it is a useful system report with credible numbers. A good reviewer can push for the missing Track1 baseline, statistical uncertainty, and an ablation that teases apart MTL and PCOT. With those additions it would be a solid contribution to the DIMT literature. My recommendation: send it to peer review, but expect the authors to do a revision round rather than accept it as-is.","headline":"A sensible competition system report with a useful unified DIMT recipe; the numbers are plausible but the headline MTL-PCOT gain is under-supported on the OCR-based track.","tokens_in":5689,"tokens_out":1991,"would_cite":true,"duration_ms":20596,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single fine-tuned LVLM, trained with multi-task learning and a two-stage perceptual chain-of-thought, can handle both OCR-based and OCR-free document image translation, with MBR decoding and post-processing adding consistent BLEU gains.","keywords":["document image machine translation","large vision-language model","multi-task learning","perceptual chain-of-thought","minimum Bayesian risk decoding","BLEU","complex layouts","OCR-free translation"],"falsifier":"Train an otherwise identical model on the same data but replace the two-stage PCOT prompt with a direct single-prompt 'translate this document image' format. If the direct model matches or beats MTL-PCOT BLEU on the DIMT25 test sets, the perceptual chain-of-thought stage is not carrying the claimed layout-comprehension advantage.","tokens_in":4679,"feed_emoji":"📄","tokens_out":6528,"duration_ms":54748,"temperature":0.7,"pith_summary":"This paper describes a competition system for end-to-end document image machine translation on complex layouts. It claims that fine-tuning an open large vision-language model with multi-task learning plus a perceptual chain-of-thought training format makes one model handle both OCR-based and OCR-free translation, and that minimum Bayesian risk decoding and lightweight post-processing add further BLEU gains. A sympathetic reader would care because a unified OCR/OCR-free document translation model removes the need for separate OCR, layout-analysis, and translation pipelines, which is a practical bottleneck for translating manuals, reports, and archival documents.","feed_headline":"One model handles both OCR-based and OCR-free document translation","feed_subtitle":"Multi-task training plus perceptual chain-of-thought lifts BLEU, and MBR decoding adds further gains.","key_machinery":"Perceptual chain-of-thought (PCOT) is a two-stage training and prompting format: the model first produces a sorted OCR result from the document image, then produces the translation conditioned on that OCR, so layout perception and cross-lingual transfer are separated explicitly. Multi-task learning mixes these staged examples with OCR-free translation examples in one SFT session. At inference, minimum Bayesian risk (MBR) decoding generates beam-search outputs and diverse sampled candidates, scores them by pairwise BLEU similarity, and selects the highest-scoring one; post-processing then collapses runs of more than ten repeated special symbols, skips overly complex tables, and normalises spaces in Chinese output.","core_discovery":"The paper's central claim is that combining multi-task learning (MTL) with a perceptual chain-of-thought (PCOT) during supervised fine-tuning, on top of an open large vision-language model, produces a single system that competes in both DIMT25 tracks: the OCR-based track and the OCR-free track. In the reported BLEU results, MTL-PCOT SFT beats single-task direct SFT on validation sets, MBR decoding and post-processing each add further gains on the test sets, and the larger 8-billion-parameter model outperforms the 1-billion-parameter model under identical methods.","pith_inferences":["Editorial inference: if PCOT's two-stage format is what enables layout comprehension, the same separation of a perceptual pass from a linguistic pass could transfer to other document-based generation tasks such as form filling, diagram captioning, or table summarisation.","Editorial inference: because MBR selection scores candidates by BLEU, the chosen output may be the one most stylistically similar to the model's own generations; a reference-free metric or human rating could rank the candidate sets differently and might change which decoding strategy is best.","Editorial inference: the post-processing rule that truncates long runs of special symbols suggests a testable failure mode, namely that the model over-generates table decorations, so training on augmented examples that enforce table-format fidelity could make the heuristic unnecessary.","Editorial inference: a direct test of the PCOT claim is to feed the model its own OCR output as text-only input; if translation quality does not drop, the perceptual stage is not doing the load-bearing work."],"forward_implications":["A single fine-tuned LVLM can serve both OCR-based and OCR-free document translation, eliminating the need to build and maintain separate OCR-to-MT pipelines.","MTL-PCOT combined SFT is reported to improve BLEU over single-task direct SFT on the DIMT25 validation and test sets.","MBR decoding and simple rule-based post-processing give complementary, repeatable BLEU gains on top of the SFT checkpoint.","Model capacity matters: the 8-billion-parameter model outperforms the 1-billion-parameter model under the same training and decoding setup.","The framework is presented as a reproducible technical recipe, covering model choice, data format, training configuration, and decoding strategy."],"supporting_citations":[{"why":"Provides the open large vision-language model (InternVL2.5-MPO) that the system fine-tunes.","marker":"[6]"},{"why":"Supplies the multi-task learning methodology used to combine OCR, layout, and translation objectives.","marker":"[7]"},{"why":"Supplies the chain-of-thought reasoning approach that is extended into perceptual chain-of-thought.","marker":"[8]"},{"why":"Supplies the minimum Bayesian risk decoding method adapted for candidate selection at inference.","marker":"[9]"},{"why":"Supplies BLEU, the metric used both for reporting results and for computing pairwise similarity in MBR decoding.","marker":"[26]"}],"fun_headline_variants":["MTL plus perceptual CoT propels one model to both DIMT tracks","Unified LVLM system lifts BLEU on OCR-based and OCR-free tracks","Multitask + perceptual CoT makes one translator for both DIMT tasks","MBR decoding and MTL-PCOT lift BLEU in both document translation tracks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the assumption that the DIMT25 benchmark's reference translations and BLEU scoring faithfully measure document image translation quality on complex layouts, because the model is fine-tuned and selected entirely on that benchmark.","fun_headline_variants_meta":{"raw":{"variants":["MTL plus perceptual CoT propels one model to both DIMT tracks","Unified LVLM system lifts BLEU on OCR-based and OCR-free tracks","Multitask + perceptual CoT makes one translator for both DIMT tasks","MBR decoding and MTL-PCOT lift BLEU in both document translation tracks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000497,"raw_usage":{"total_tokens":2378,"prompt_tokens":832,"completion_tokens":1546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1457}},"tokens_in":448,"tokens_out":1546,"duration_ms":11030,"temperature":1.0,"reasoning_tokens":1457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:43:23.223133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an otherwise identical model on the same data but replace the two-stage PCOT prompt with a direct single-prompt 'translate this document image' format. If the direct model matches or beats MTL-PCOT BLEU on the DIMT25 test sets, the perceptual chain-of-thought stage is not carrying the claimed layout-comprehension advantage.","supporting_citations":[{"cited_title":"A survey on multi-task learning","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-task learning methodology used to combine OCR, layout, and translation objectives."}],"review_version":1}