{"id":"e09d1501-b029-4b9a-85e8-4c1c3fefb3c1","arxiv_id":"2412.17787","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Cross-lingual text-rich visual question answering drops roughly 30% in accuracy across LVLMs, and a distillation method narrows the gap but is evaluated on its own training set.","lead":"New tests show vision-language AI drops about 30% in accuracy when the text in an image and the question asked about it are in different languages. A proposed training fix shrinks the gap, but its gains are measured on the same data used to train it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MVCL-MI is fine-tuned and evaluated on the same XPaperQA data with no reported held-out split, so the central claim that it reduces cross-lingual disparity while preserving monolingual performance is not actually supported.","rationale":"The reader's verdict of REJECT is well grounded. The strongest empirical claim, that MVCL-MI reduces the cross-lingual gap while preserving monolingual performance, rests entirely on Table 3, which reports results on XPaperQA. The fine-tuning procedure in Section 5 and the training details in the appendix do not mention any held-out subset, and the benchmark construction section describes XPaperQA as a single final set of 3,870 QA pairs with 1,039 paper images. Without a train/test split or evaluation on an external cross-lingual text-rich VQA set, the observed improvements could be caused by memorizing or overfitting to the exact questions and answers used in training. This is the most load-bearing problem because it attacks the central mitigation claim directly, independent of any specific information-theoretic derivation. The reader's weakest_assumption focuses more on the Eq. 7 Gaussian-noise equivalence and correlation-to-causation steps, but the same-data evaluation is also mentioned in the rationale and is arguably the more decisive issue. I therefore partially agree with the reader: the diagnostic concerns matter, but the same-data evaluation alone is sufficient to withhold acceptance of the method claim. The paper does provide a plausible benchmark and documents a real cross-lingual accuracy drop across many LVLMs, so the dataset contribution has value; nevertheless, the headline MVCL-MI result needs a held-out test before the central claim can be supported.","tokens_in":19648,"tokens_out":3343,"duration_ms":32371,"concrete_test":"Obtain or reconstruct the XPaperQA training set used for MVCL-MI and require a disjoint held-out split, for example holding out 20% of the 1,039 paper images together with all QA pairs from those pages. Retrain MVCL-MI on the remaining 80% and report Table 3 on the held-out split. In addition, evaluate the trained MVCL-MI on the cross-lingual versions of ChartQA, TextVQA, OCRVQA, and DocVQA that XT-VQA already includes, since those were not used for fine-tuning. If the cross-lingual accuracy improvements and gap reductions persist on held-out data, for example the Chinese-paper overall gain stays positive and the gap reduction remains above noise, the central claim is supported; if the gains shrink to near zero or reverse, the method is overfitting to the evaluation set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main positive claim is that MVCL-MI \"effectively reduces the visual-text cross-lingual performance disparity while preserving the inherent capabilities of LVLMs\" (Abstract, Table 3, Conclusion). The evidence for this claim is Table 3, which reports MVCL-MI on XPaperQA-en and XPaperQA-zh. But MVCL-MI is trained on XPaperQA: the training objective in Eq. 10 uses cross-lingual QA pairs, the method is described as \"deploy our training method on MiniCPM-Llama3-V\" for one epoch, and the hyperparameter table gives only training settings, with no train/test split described. Nowhere is a held-out partition of XPaperQA (by paper, by page, or by QA pair) specified. Therefore the reported gains on XPaperQA (e.g., +5.3 overall on Chinese papers, gap reduction from 29.3% to 21.3%) may reflect fitting the evaluation set rather than a genuine reduction of the cross-lingual gap. Because this is the only evaluation of MVCL-MI, the central mitigation claim is currently unsupported. The Eq. 7 sign inconsistency and the 100-example correlation analysis would matter only after this evaluation issue is resolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces XT-VQA, a cross-lingual text-rich visual question answering benchmark that extends four existing VQA datasets with translated questions and adds a newly constructed bilingual academic-paper dataset, XPaperQA. It evaluates eight open and two closed LVLMs and reports a substantial accuracy drop when the question language differs from the image text language. The paper then proposes MVCL-MI, a distillation-based fine-tuning method that is claimed to maximize vision-language cross-lingual mutual information and to reduce the cross-lingual performance gap while preserving monolingual capability. The method is applied to MiniCPM-Llama3-V and evaluated on XPaperQA, with reported gains in cross-lingual accuracy and reductions in the gap.","tokens_in":19986,"tokens_out":8576,"duration_ms":79145,"significance":"If the central claims are established, XT-VQA and XPaperQA would be useful resources for studying a realistic and under-explored failure mode of LVLMs, and the reported cross-lingual gap across ten models is an important observation. The paper's strengths include the release of code and a new bilingual benchmark, the breadth of the model evaluation, and the explicit attempt to connect an information-theoretic diagnostic to a mitigation method. However, the current evidence for the method's effectiveness is undermined by the absence of any held-out evaluation split, and the mutual information derivation contains sign errors and unsupported equivalence claims. These issues prevent the paper from supporting its main conclusions as written.","major_comments":[{"comment":"The paper never specifies a held-out split between the data used for MVCL-MI training and the XPaperQA data used for evaluation in Table 3. The training objective in Eq. (10) uses cross-lingual QA pairs, the method is described as deploying training on MiniCPM-Llama3-V for one epoch, and Table 5 reports only hyperparameters, with no mention of a train/test partition by paper, page, or QA pair. If the model is trained on XPaperQA and evaluated on the same XPaperQA, the reported gains (for example +1.9 on XPaperQA-en and +5.3 on XPaperQA-zh) and the claimed gap reductions from 19.4% to 13.1% and 29.3% to 21.3% are consistent with fitting the evaluation set rather than with genuine cross-lingual transfer. The central mitigation claim is therefore unsupported by the evidence presented.","section":"§6, Table 3 and Training Details"},{"comment":"The mutual information derivation contains sign and conditioning errors. In Eq. (5), the expansion of I(Y;V|Q) = H(Y|Q) − H(Y|V,Q) writes the first term as −Σ P(y|V) log P(y|V), which depends on V rather than Q, and the second term as −Σ P(Y|V,Q) log P(Y|V,Q), so the displayed expression equals H(Y|V) + H(Y|V,Q) rather than the stated difference. In Eq. (7), the expansion of H(Y|Vε,Q) − H(Y|V,Q) is written as Σ pθ(yi|Vε,Q) log pθ(yi|Vε,Q) − Σ pθ(yi|V,Q) log pθ(yi|V,Q), which equals −H(Y|Vε,Q) − H(Y|V,Q), not the claimed mutual information. These errors affect the numerical MI values, the correlation reported in Figure 4, and the diagnostic conclusion that cross-lingual questions insufficiently activate visual information.","section":"§4, Eqs. (5) and (7)"},{"comment":"The paper claims a 'strong correlation' between accuracy and mutual information based on 100 randomly selected ChartQA examples and 'eight different languages,' but it reports no correlation coefficient, confidence interval, or significance test, and it does not list the eight languages or explain how they relate to the earlier description of XT-VQA as English/Chinese/French. The 100-example subset and the missing statistics are not sufficient to support the causal statement that the cross-lingual gap stems from insufficient activation of visual information. A concrete quantitative analysis with variance estimates is needed before this diagnostic claim can be accepted.","section":"§4.3, Figures 3-4 and Mutual Information Analysis"},{"comment":"The paper asserts that minimizing the KL divergence in Eq. (9) is equivalent to maximizing the mutual information I(Y;V|Q) from Eq. (7), but no derivation or formal argument is given. Minimizing DKL(P_tgt || P_src) aligns the output distributions for target- and source-language questions; it does not directly optimize the difference between conditional entropies with and without visual tokens as defined in Eq. (7). Without a proof or an empirical verification that the KL objective increases the Eq. (7) quantity, the information-theoretic justification for MVCL-MI remains an unsupported equivalence claim.","section":"§5, Eqs. (9)-(10)"},{"comment":"The assumption that adding Gaussian noise to the image is equivalent to removing the image, H(Y|Q) ≈ H(Y|Vε,Q), is asserted without validation. A heavily noised image is not the same input as an absent image, and residual text-like patterns or layout information could still influence the model's output distribution. This assumption is load-bearing for the entire MI analysis and for the motivation of MVCL-MI; the authors should provide an empirical check, such as comparing output entropy and accuracy under true no-image conditions versus heavy-noise conditions, before relying on it.","section":"§4, Eq. (7) and following paragraph"}],"minor_comments":[{"comment":"The text says 'Table 2 shows LVLM performance on XT-VQA,' but the performance table is labeled Table 1; Table 2 contains XPaperQA data statistics. The cross-reference should be corrected.","section":"§3, Evaluation of LVLMs on XT-VQA"},{"comment":"The dataset name is inconsistent: the construction section uses 'XPaperVQA,' while elsewhere the paper uses 'XPaperQA.' One consistent name should be used throughout.","section":"§3, Dataset Construction"},{"comment":"Figure references are swapped in places: the entropy distributions are shown in Figure 3 and the accuracy-MI statistics in Figure 4, but the text attributes them the other way around in two sentences.","section":"§4, Mutual Information Analysis across Languages"},{"comment":"The appendix refers to 'TeXT-VQA' in the dataset description; this should be 'TextVQA.'","section":"Appendix, Model Details"},{"comment":"The phrase 'w/o KL-Loss ... shown in F. 10' should read 'Eq. 10,' and the same correction applies to the w/o Cross-CE description.","section":"§5, Ablation Study"},{"comment":"The overall column of Table 4 shows values such as '26.0 +0.7' and '22.5 +1.9,' but the text reports an increase of 1.9 on XPaperQA-en; please clarify whether the reported number refers to overall accuracy or to the cross-lingual subset, and make the table and text consistent.","section":"§6, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the cross-lingual gap finding are potentially valuable, but the current version cannot support the method's central claim because of the missing held-out split. The sign errors in the MI equations and the unsupported equivalence between KL minimization and MI maximization are fixable in revision, but they require substantive re-analysis and re-evaluation rather than copy-editing. I would encourage the editor to request a revised version with a proper train/test separation and corrected information-theoretic derivations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the benchmark is a real contribution, the drop they document is credible, and the paper deserves a serious look. But the central claim that MVCL-MI closes the cross-lingual gap is not supported in this version, because the model is fine-tuned on XPaperQA and tested on the same XPaperQA with no held-out split.\n\nWhat's new: XT-VQA combines five existing text-rich VQA datasets with translated questions and adds XPaperQA, a bilingual (English/Chinese) academic paper QA dataset. That fills a real gap; most prior multilingual VQA benchmarks keep the image text and question in the same language. Their evaluation of ten LVLMs shows a consistent, large drop (up to about 33%) when the question language does not match the image text. That is a reproducible finding the community will use.\n\nSoft spots, in order of severity. First, the mitigation experiment: MVCL-MI is trained on XPaperQA for one epoch and the only reported results (Table 3, ablation Table 4) are on XPaperQA. No train/test split is described. That means the +1.9/+5.3 gains could be memorization, not a generalizable cross-lingual alignment. That is a load-bearing flaw in the paper's main positive claim. Second, the mutual information derivation has sign errors: Eq. 6 defines entropy as sum p log p, missing the minus sign, and Eq. 7 defines I as H(noise) - H(image) but writes the sums so it is actually H(image) - H(noise), the negative of the claimed MI. This flips the interpretation of their higher MI equals better activation story. Third, the strong correlation between accuracy and MI is based on 100 ChartQA examples, with no correlation coefficient or significance test, and the causal claim (cross-lingual questions fail to activate visual information) is inferred from that correlation.\n\nThe benchmark itself deserves to be published, and the documented drop is important. But as it stands, the method results are not validated. I would send this to peer review with a request for major revision: add a proper held-out split (by paper or by page) for XPaperQA and re-run; fix the entropy/MI signs; and either strengthen or soften the information-theory claims. If those are addressed, XT-VQA would be a solid contribution.\n\nRecommendation: engage with it, but do not take the mitigation numbers at face value.","headline":"Useful cross-lingual VQA benchmark and a credible documented performance drop, but the mitigation results are not supported because MVCL-MI is fine-tuned and evaluated on the same XPaperQA data with no held-out split.","tokens_in":20526,"tokens_out":2710,"would_cite":true,"duration_ms":24779,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Question language mismatches cost vision-language models up to 32.6% accuracy on text-rich images, and a mutual-information-guided distillation narrows the gap.","keywords":["cross-lingual VQA","text-rich images","mutual information","knowledge distillation","KL divergence","multimodal benchmark","LVLM","visual activation"],"falsifier":"Recompute the paper's mutual-information estimate on the same ChartQA questions with a blank or black image in place of Gaussian noise; if the accuracy–mutual-information correlation across the eight languages disappears, the diagnostic and MVCL-MI's motivation are falsified. Running the same correlation on a sample of 1,000 or more examples rather than 100 would settle whether the effect is stable.","tokens_in":19480,"feed_emoji":"🌐","tokens_out":12281,"duration_ms":99841,"temperature":0.7,"pith_summary":"Large vision-language models (LVLMs) read text-rich images well when asked in English, but their accuracy drops sharply when the question is in a different language from the text in the image. The paper builds XT-VQA, a benchmark that combines existing text-rich VQA data with a newly collected bilingual paper-image set, XPaperQA, and measures drops averaging up to 32.6% across eight open and two closed models. An information-theoretic diagnostic computes mutual information between model outputs and image tokens conditioned on the question; accuracy tracks this quantity across languages, suggesting that cross-lingual questions under-activate the visual input. To fix this, the paper proposes MVCL-MI, which fine-tunes a model by distilling monolingual output distributions into cross-lingual answers with KL divergence while also training on cross-lingual question-answer pairs. On XT-VQA the method reduces the cross-lingual gap on XPaperQA from 19.4% to 13.1% (English paper) and 29.3% to 21.3% (Chinese paper) with little or no loss in monolingual accuracy.","feed_headline":"Asking in another language cuts vision-language QA accuracy by a third","feed_subtitle":"New benchmark XT-VQA measures the gap, traces it to weak visual activation, and a distillation fix narrows it.","key_machinery":"The load-bearing objects are the XT-VQA benchmark and a mutual-information identity. XT-VQA merges OCRVQA, TextVQA, ChartQA, and DocVQA with XPaperQA, a bilingual English–Chinese paper-image QA set built from QASPER-style questions plus a Gemini-based generation pipeline with confidence, similarity, and consistency filters. The identity is $I(Y;V|Q) = H(Y|V_\\epsilon,Q) - H(Y|V,Q)$, where $V_\\epsilon$ is a Gaussian-noised image standing in for \"no image\" and the quantity is meant to measure how strongly the image drives the answer under a given question language. The mitigation mechanism is MVCL-MI, whose objective combines cross-entropy on source- and target-language answers with two KL-divergence terms $D_{\\mathrm{KL}}(P^{\\mathrm{tgt-src}}\\|P^{\\mathrm{src-src}})$ and $D_{\\mathrm{KL}}(P^{\\mathrm{src-tgt}}\\|P^{\\mathrm{tgt-tgt}})$ that pull cross-lingual output distributions toward the corresponding monolingual ones.","core_discovery":"On the paper's own terms, the central discovery is that cross-lingual text-rich VQA fails because the question in a second language does not sufficiently activate the visual information needed to answer it, and this can be measured and repaired. The authors quantify activation as conditional mutual information $I(Y;V|Q)$ between the model's output tokens $Y$ and the image tokens $V$ given the question $Q$, using a Gaussian-noised image to approximate the no-image baseline. Across eight languages on ChartQA, accuracy and this mutual-information estimate are strongly correlated, with the source-language question delivering the most mutual information. MVCL-MI then maximizes the target-language mutual information indirectly: rather than minimizing target entropy directly (which invites degenerate sharp distributions), it minimizes KL divergence between cross-lingual output distributions and the monolingual output distribution, using the monolingual logits as teacher. Experimentally this narrows the cross-lingual gap while keeping monolingual scores essentially intact, and ablations show both the cross-lingual cross-entropy term and the KL term are needed.","pith_inferences":["Editorial inference: the same mutual-information diagnostic could be applied to monolingual hallucination or OCR-failure cases, replacing \"question language\" with question phrasing to detect when a model ignores image content.","Editorial inference: if the Gaussian-noise equivalence holds, MVCL-MI should transfer to language pairs outside English–Chinese–French; the untested, harder case is a low-resource target language with a different script.","Editorial inference: because the method is fine-tuned and evaluated on the same XPaperQA data, an out-of-distribution benchmark (for example, foreign street signs or tables from a new domain) is the next check on whether the mutual-information fix generalizes.","Editorial inference: the accuracy–MI correlation suggests mutual information could serve as an inference-time confidence score that triggers translation or re-asking, though the paper does not propose this."],"forward_implications":["Cross-lingual accuracy should become a standard reporting axis for LVLMs, since even models trained with multilingual data retain a measurable gap.","Questions that demand faithful extraction or abstraction from image text degrade more than yes/no questions, so the cross-lingual failure is worst in the settings that need image comprehension most.","MVCL-MI can be layered onto an existing LVLM with a small fine-tuning budget and no need for paired parallel images, only for translated questions and answers.","The mutual-information estimate gives a training-free signal for when a model is answering from linguistic priors rather than from the image.","Ablation evidence shows the KL terms, not just extra cross-lingual training data, are what preserve monolingual behavior while closing the gap."],"supporting_citations":[{"why":"supplies the QASPER questions reconstructed into XPaperQA-en document images.","marker":"(Dasigi et al. 2021)"},{"why":"ChartQA is the dataset used for the eight-language mutual-information analysis.","marker":"(Masry et al. 2022)"},{"why":"motivates using entropy and mutual information to analyze model output uncertainty.","marker":"(Farquhar et al. 2024)"},{"why":"justifies treating Gaussian-noised images as approximating the image-free baseline in the mutual-information estimate.","marker":"(Leng et al. 2024)"},{"why":"DocVQA is one of the text-rich benchmarks extended into XT-VQA.","marker":"(Mathew, Karatzas, and Jawahar 2021)"},{"why":"OCRVQA is one of the text-rich benchmarks extended into XT-VQA.","marker":"(Mishra et al. 2019)"},{"why":"TextVQA is one of the text-rich benchmarks extended into XT-VQA.","marker":"(Singh et al. 2019)"},{"why":"defines the KL divergence used in the MVCL-MI distillation objective.","marker":"(Kullback and Leibler 1951)"}],"fun_headline_variants":["Cross-lingual VQA gap traced to weak visual activation","Mutual information pinpoints why cross-lingual visual QA degrades","Distillation fix restores cross-lingual VQA by realigning vision and text","Why foreign-language charts trip up LVLMs: missing visual spark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that adding heavy Gaussian noise to an image makes the model behave as if the image were absent, so the mutual-information estimate really measures visual activation; the paper's causal story also depends on a correlation observed on 100 ChartQA examples.","fun_headline_variants_meta":{"raw":{"variants":["Cross-lingual VQA gap traced to weak visual activation","Mutual information pinpoints why cross-lingual visual QA degrades","Distillation fix restores cross-lingual VQA by realigning vision and text","Why foreign-language charts trip up LVLMs: missing visual spark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":3094,"prompt_tokens":1104,"completion_tokens":1990,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":720,"completion_tokens_details":{"reasoning_tokens":1911}},"tokens_in":720,"tokens_out":1990,"duration_ms":13774,"temperature":1.0,"reasoning_tokens":1911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:08:42.050893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the paper's mutual-information estimate on the same ChartQA questions with a blank or black image in place of Gaussian noise; if the accuracy–mutual-information correlation across the eight languages disappears, the diagnostic and MVCL-MI's motivation are falsified. Running the same correlation on a sample of 1,000 or more examples rather than 100 would settle whether the effect is stable.","supporting_citations":[],"review_version":1}