{"id":"65552587-7582-4c91-8618-158eac483d09","arxiv_id":"2504.21681","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Machine-translated task data is the best average parallel-data source for cross-lingual transfer of vision-language encoders, but authentic caption-like data beats it in some languages, and multilingual training helps on average up to a point.","lead":"This paper studies how to make English vision-language models work in other languages by swapping in a multilingual text encoder trained on parallel data. It compares machine-translated task data, translated captions, generic parallel text, and authentic caption-like data, finding that task-specific machine translations win on average.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Task MT best' result conflates translation status with domain: Task MT is drawn from NLVR2, the same task used to fine-tune BridgeTower, so the ranking may reflect in-domain text rather than parallel-data type.","rationale":"The reader's weakest_assumption identifies exactly the domain confound between Task MT and the downstream fine-tuning task. My stress-test confirms that this is the load-bearing issue: the strongest claim about Task MT being best on average rests on a comparison in which the in-domain condition is also the only task-derived MT condition. Without an in-domain authentic condition or a cross-task MT condition, the paper cannot separate the effect of translation from the effect of domain. This does not invalidate the whole study: the comparison between Caption MT and Caption-like is more controlled, and the multilingual vs. bilingual comparison in Table 2 at least partially supports the second claim. However, the headline overgeneralizes, and the lack of error bars makes small differences unreliable. The reader's CONDITIONAL verdict remains appropriate; no verdict change is needed, but the conditions should explicitly require a deconfounded Task MT condition and variance estimates.","tokens_in":10658,"tokens_out":6097,"duration_ms":61924,"concrete_test":"Re-run Experiment 1 replacing Task MT with machine translations of English training sentences from a different VL task (e.g., VQAv2 or SNLI-VE) that is not used for fine-tuning, while keeping BridgeTower fine-tuned on NLVR2. If the cross-task MT condition no longer beats Caption MT on the MARVL average (or the gap shrinks substantially), the headline ranking is driven by domain overlap with the fine-tuning task, not by parallel-data type. Repeat with at least three random seeds to bound variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the Abstract is that 'even machine-translated task data are the best on average.' Section 4.2 defines Task MT as machine translations of NLVR2 sentences, while the downstream model was fine-tuned on NLVR2 (Section 4.1, Figure 1). This means the English side of Task MT consists of the exact in-domain sentences whose hidden states the English RoBERTa encoder was fine-tuned to produce. The alignment loss in Equation 1 therefore teaches XLM-R to mimic task-specialized representations on the evaluation-relevant text distribution. Caption MT (MSCOCO) and Caption-like (classifier-filtered OPUS-100) both come from the caption domain, so the comparison between those two conditions partially controls for domain; Task MT does not. Consequently, the reported advantage of Task MT over Caption MT could be fully explained by domain overlap with the fine-tuning task rather than by any property of machine-translated task data. The secondary claim that 'caption-like authentic parallel data outperformed it in some languages' is less affected, since it compares two caption-domain conditions. The absence of error bars or multiple seeds compounds the risk: several differences in Tables 1-3 are within a few points, so they may not be stable even if the confound were removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies cross-lingual transfer for vision-language encoders by replacing a fine-tuned English text encoder with a multilingual encoder trained to mimic the English encoder's hidden states on parallel data. The authors extend CliCoTea with a subword-level alignment loss and a bottleneck projection, and they evaluate four parallel-data strategies (machine-translated task data, machine-translated captions, generic parallel data, and caption-like authentic parallel data) plus scaling in the number of languages. Experiments on MARVL, M5-VGR, and XVNLI lead to the claims that machine-translated task data is best on average, caption-like authentic data is better in some languages, and most languages benefit from multilingual training.","tokens_in":10860,"tokens_out":7984,"duration_ms":72213,"significance":"If the claims hold, the paper gives practical guidance for building multilingual VL systems: task-domain machine-translated data is a strong transfer signal, and pooling multiple languages helps in low-resource settings. The paper also demonstrates that a lightweight encoder-transfer method can match CliCoTea with less data, and it provides a useful comparison across several multilingual benchmarks. These contributions are of interest to practitioners in multilingual multimodal learning. However, the significance is conditional on ruling out the confounds described below and on establishing that the reported differences are not within run-to-run variance.","major_comments":[{"comment":"The Task MT condition uses machine-translated sentences from NLVR2, while the VL model in the same pipeline is fine-tuned on English task data, including NLVR2 (Section 4.1, Figure 1). The alignment loss in Eq. (1) therefore encourages XLM-R to reproduce the representations of exactly the in-domain task text, whereas Caption MT (MSCOCO) and Caption-like (OPUS-100) come from other domains. This confounds 'translation method' with 'domain overlap with the fine-tuning task,' so the reported advantage of Task MT over the other conditions could be due to the latter rather than to any property of machine-translated task data. Please add a control (e.g., task MT from a task other than the fine-tuning task, or authentic task-domain data) or re-frame the headline conclusion.","section":"§4.2, Table 1"},{"comment":"All results are from single runs without error bars, confidence intervals, or significance tests. Many differences that carry the conclusions are small, for example Table 3's averages 66.3, 66.5, 66.7, 66.7, 66.3 across language counts, and Table 2's Biling. 5k average (65.9) versus Multi. 25k average (66.3). Without multiple seeds and a paired significance test across the five MARVL languages or across benchmarks, the claims 'task MT is best on average' and 'most languages benefit from multilingual training' are not established.","section":"Tables 1-4"},{"comment":"The bilingual-versus-multilingual comparison and the language-scaling experiment both confound the number of languages with the total amount of training data. In Table 2, 'Biling. 5k' uses 5k pairs total for one language while 'Multi. 25k' uses 25k pairs total (5k per language for five languages); in Table 3, the language count grows from 5 to 40 while keeping 5k pairs per language, so the total data grows from 25k to 200k. The improvements attributed to multilingual training or to adding languages could therefore be driven by larger training data rather than by language diversity. Please include matched-total-data controls (or state the data budget explicitly per condition) before drawing the Section 6 conclusion.","section":"§4.3, Tables 2-3"}],"minor_comments":[{"comment":"The caption refers to 'M5B-VGR' while the rest of the paper uses 'M5-VGR'; please unify the notation.","section":"Table 1 caption"},{"comment":"The phrase 'cross-signal transfer' appears to be a typo for 'cross-lingual transfer'; please correct it.","section":"§4.3"},{"comment":"The manuscript does not state the number of sentence pairs used for the Task MT and Caption MT conditions; please specify the per-language data budget for these conditions so that the comparison with Generic and Caption-like (5k pairs) is explicit.","section":"§4.2"},{"comment":"The label 'Multi. 25k' is unclear because the Section 4.3 text says 5k caption-like pairs per language; please clarify whether 25k is the total across five languages.","section":"Table 2"},{"comment":"References [13] and [14] appear to refer to the same paper (Visually Grounded Reasoning across Languages and Cultures); please deduplicate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely question and the overall direction is useful, but the two headline claims rest on comparisons that do not isolate the stated factors. I would ask for multiple seeds with significance tests and a domain-matched control for the Task MT condition, or a careful re-framing of the conclusions. If these can be provided, the paper could be suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version first: this paper's headline result—that machine-translated task data is the best parallel data for cross-lingual VL encoder transfer—is undercut by a domain confound. The 'Task MT' condition uses machine-translated NLVR2 sentences, and the model is fine-tuned on NLVR2, so the English side is exactly in-domain. That makes the comparison with caption-domain data (MSCOCO, OPUS-100) unfair. The paper is still worth reading, though, because the language-count experiments are clean and the finding that caption-like authentic parallel data can beat machine-translated captions in some languages is a genuinely useful data point.\n\nWhat's new: the systematic comparison of four parallel data types and the scaling of languages (5, 10, 20, 30, 40) for encoder transfer. CliCoTea only reported bilingual transfer with one data type. The method is a modest extension—a learnable weighted combination of layers plus a mean-pooling loss—but the ablation shows it helps. The results on multilingual training are consistent: more languages generally help until ~20, then plateau or dip. That's a useful practical finding.\n\nWhere it's soft: the confound I mentioned. The paper doesn't isolate translation status from domain. If you want to claim machine-translated task data is best, you need a control—for example, task MT from a different task, or fine-tune on captions and compare. The caption-like vs caption-MT comparison is cleaner, and that's where the 'authentic data can win' claim lives. Second, there are no error bars or multiple seeds. Several differences in the tables are within a few points; with single runs, I wouldn't bet on them. Third, no code or data released, so reproducibility is limited.\n\nWho it's for: people actually building multilingual vision-language systems and deciding what parallel data to mine. They'll get the language-count guidance, but they should not treat the Task-MT-is-best ranking as solid until the confound is addressed.\n\nMy recommendation: send it to review, but tell the authors to redo or re-frame the Task MT comparison and add variance estimates. The paper has a real contribution buried under an over-claimed headline.","headline":"Useful empirical study of parallel data types for cross-lingual VL encoder transfer, but the headline 'Task MT best' result is confounded by domain overlap with the downstream task.","tokens_in":11433,"tokens_out":3669,"would_cite":false,"duration_ms":37402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For cross-lingual transfer of vision-language encoders, machine-translated task data is the best all-round parallel data, though authentic caption-like data wins in some languages and most languages benefit from multilingual pooling.","keywords":["cross-lingual transfer","vision-language models","parallel data selection","multilingual training","visual reasoning","machine-translated data","encoder transfer","low-resource languages"],"falsifier":"Repeat the transfer experiments with machine-translated data from a different task (e.g., translated VQA or image-captioning data) and authentic parallel data from that same task. If the translated task data no longer beats authentic task data on average, the claim that machine-translated task data are the best parallel data type is not general.","tokens_in":10412,"feed_emoji":"🌐","tokens_out":8654,"duration_ms":72191,"temperature":0.7,"pith_summary":"This paper investigates how to extend a fine-tuned English vision-language model to other languages by replacing its English text encoder with a multilingual one and adapting the new encoder on parallel data. It compares four kinds of parallel data — machine-translated task sentences, machine-translated captions, generic parallel text, and authentic caption-like sentences — and finds that machine-translated task data is best on average, while authentic caption-like data wins in several individual languages. It also compares bilingual transfer with transfer on a single encoder trained on many languages, finding that multilingual training helps most languages in very low-resource settings, with the benefit peaking around 20 languages. These results offer concrete guidance for building multilingual vision-language systems without large multilingual pre-training.","feed_headline":"Machine-translated task data wins for multilingual VL transfer","feed_subtitle":"Authentic caption data beats it in some languages; pooling several languages helps most targets.","key_machinery":"The method replaces BridgeTower's English RoBERTa text encoder with XLM-R and trains XLM-R to reproduce RoBERTa's hidden states. The training signal is a subword-aligned imitation loss: Eflomal word alignments are projected to subword level, and the mean squared error between aligned hidden states is averaged with a mean-pooled state-matching loss over the last six layers that feed the cross-modal encoder. A learnable weighted mixture of layers, followed by a bottleneck projection with GELU and layer normalization, gives XLM-R the flexibility to match RoBERTa's representation space. This machinery is what makes the transfer possible, and the paper varies the parallel data and the number of languages while keeping it fixed.","core_discovery":"The paper's central empirical claim is that for cross-lingual transfer of a vision-language encoder via parallel data, the data's source matters more than whether it is machine-translated or authentic. On average across MARVL, M5-VGR, and XVNLI, machine-translated sentences from the same downstream task (NLVR2) give the highest accuracy, beating machine-translated COCO captions, generic OPUS-100 parallel sentences, and caption-like filtered OPUS-100 data. But the caption-like authentic data outperform machine-translated data for several languages, so translation quality alone does not determine the ranking. In addition, a single XLM-R encoder trained on parallel data from all five MARVL languages beats separate bilingual encoders when each language has only 5k pairs, and adding up to 20 languages continues to improve average performance before diminishing returns set in.","pith_inferences":["The task-MT advantage may partly reflect domain match rather than translation: the machine-translated task data come from NLVR2, the same task used for fine-tuning; a test on a different task domain would separate these factors.","The caption-like selection method suggests that authentic parallel data filtered for task similarity could reduce dependence on machine translation services, which is a practical route for low-resource settings where MT engines are unavailable.","The language-synergy curve (help up to 20 languages, then decline) echoes the known trade-off in multilingual models between shared representation and per-language capacity; the optimal language count likely shifts with per-language data size and language distance."],"forward_implications":["When extending a vision-language encoder to many languages, machine-translated data from the target task is a strong default: it gives the highest average accuracy in the paper's three benchmarks.","Authentic caption-like parallel data, selected by a caption classifier, can replace machine-translated captions and outperform them on several languages, offering a cheaper and sometimes better alternative.","Pooling several languages into a single multilingual encoder helps most languages in the very low-resource case (5k parallel pairs per language), compared with separate bilingual encoders.","The multilingual benefit grows as languages are added but peaks around 20 languages; beyond that, average accuracy slightly declines.","The paper's approach matches CliCoTea's reported performance using a fraction of the parallel data (25k pairs), suggesting data selection quality matters as much as data quantity."],"supporting_citations":[{"why":"supplies the base encoder-transfer method and the comparison point for training with fewer parallel samples.","marker":"[10]"},{"why":"provides BridgeTower, the vision-language encoder whose English text encoder is replaced.","marker":"[29]"},{"why":"introduces the mean-pooled state-matching loss and the prior mCLIP cross-lingual transfer approach.","marker":"[3]"},{"why":"provides XLM-R, the multilingual text encoder trained to mimic the English encoder.","marker":"[5]"},{"why":"provides RoBERTa, the English text encoder whose hidden states are imitated.","marker":"[15]"},{"why":"provides the Eflomal word aligner used to compute the subword alignments for the loss.","marker":"[17]"},{"why":"supplies OPUS-100, the source of authentic parallel data and the negative examples for the caption classifier.","marker":"[32]"},{"why":"provides MARVL, one of the native-speaker-annotated multilingual visual reasoning test sets.","marker":"[13]"},{"why":"supplies the IGLUE benchmark, including XVNLI and the English task sentences used in the transfer.","marker":"[1]"},{"why":"provides M5-VGR, the second multilingual visual reasoning benchmark used for evaluation.","marker":"[23]"}],"fun_headline_variants":["Parallel data source outranks translation quality for VL transfer","Machine-translated task data tops multilingual VL transfer","Authentic captions beat MT for some languages in VL transfer","Multilingual parallel data boosts cross-lingual VL encoders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison of parallel data types assumes the type itself is the explanatory variable, but the datasets also differ in domain and filtering: Task MT comes from the same NLVR2 task used for fine-tuning, so its average advantage may be an in-domain effect rather than a property of machine-translated task data.","fun_headline_variants_meta":{"raw":{"variants":["Parallel data source outranks translation quality for VL transfer","Machine-translated task data tops multilingual VL transfer","Authentic captions beat MT for some languages in VL transfer","Multilingual parallel data boosts cross-lingual VL encoders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2013,"prompt_tokens":822,"completion_tokens":1191,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":1124}},"tokens_in":438,"tokens_out":1191,"duration_ms":9497,"temperature":1.0,"reasoning_tokens":1124,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:56:01.289626+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the transfer experiments with machine-translated data from a different task (e.g., translated VQA or image-captioning data) and authentic parallel data from that same task. If the translated task data no longer beats authentic task data on average, the claim that machine-translated task data are the best parallel data type is not general.","supporting_citations":[{"cited_title":"In: Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI","cited_arxiv_id":null,"evidence_quote":"provides BridgeTower, the vision-language encoder whose English text encoder is replaced."},{"cited_title":"In: Proceedings of the Thirteenth Language Resources and Evaluation Conference","cited_arxiv_id":null,"evidence_quote":"introduces the mean-pooled state-matching loss and the prior mCLIP cross-lingual transfer approach."},{"cited_title":"The Prague Bulletin of Mathematical Linguistics106(1), 125 (2016)","cited_arxiv_id":null,"evidence_quote":"provides the Eflomal word aligner used to compute the subword alignments for the loss."},{"cited_title":"In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"provides MARVL, one of the native-speaker-annotated multilingual visual reasoning test sets."},{"cited_title":"IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages","cited_arxiv_id":"2201.11732","evidence_quote":"supplies the IGLUE benchmark, including XVNLI and the English task sentences used in the transfer."},{"cited_title":"In: Findings of the Association for Computational Linguis- tics: EMNLP 2024","cited_arxiv_id":null,"evidence_quote":"provides M5-VGR, the second multilingual visual reasoning benchmark used for evaluation."}],"review_version":1}