Pith. sign in

REVIEW 5 major objections 6 minor 47 references

Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Question language mismatches cost vision-language models up to 32.6% accuracy on text-rich images, and a mutual-information-guided distillation narrows the gap.

desk verdict Useful cross-lingual VQA benchmark and a credible documented performance drop, but the mitigation results are not supported because MVCL-MI is fine-tuned and evaluated on the same XPaperQA data with no held-out split. read the letter →

arxiv 2412.17787 v1 pith:SBXSA5X4 submitted 2024-12-23 cs.CV cs.CL

Xinmiao Yu , Xiaocheng Feng , Yun Li , Minghui Liao , Ya-Qi Yu , Xiachong Feng , Weihong Zhong , Ruihan Chen
show 5 more authors
Mengkang Hu Jihao Wu Dandan Tu Duyu Tang Bing Qin
This is my paper · ORCID
classification cs.CVcs.CL
keywords cross-lingualVQAtext-richimagesmutualinformationknowledgedistillationKLdivergencemultimodalbenchmarkLVLMvisualactivation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large vision-language models (LVLMs) read text-rich images well when asked in English, but their accuracy drops sharply when the question is in a different language from the text in the image. The paper builds XT-VQA, a benchmark that combines existing text-rich VQA data with a newly collected bilingual paper-image set, XPaperQA, and measures drops averaging up to 32.6% across eight open and two closed models. An information-theoretic diagnostic computes mutual information between model outputs and image tokens conditioned on the question; accuracy tracks this quantity across languages, suggesting that cross-lingual questions under-activate the visual input. To fix this, the paper proposes MVCL-MI, which fine-tunes a model by distilling monolingual output distributions into cross-lingual answers with KL divergence while also training on cross-lingual question-answer pairs. On XT-VQA the method reduces the cross-lingual gap on XPaperQA from 19.4% to 13.1% (English paper) and 29.3% to 21.3% (Chinese paper) with little or no loss in monolingual accuracy.

What carries the argument

The load-bearing objects are the XT-VQA benchmark and a mutual-information identity. XT-VQA merges OCRVQA, TextVQA, ChartQA, and DocVQA with XPaperQA, a bilingual English–Chinese paper-image QA set built from QASPER-style questions plus a Gemini-based generation pipeline with confidence, similarity, and consistency filters. The identity is $I(Y;V|Q) = H(Y|V_\epsilon,Q) - H(Y|V,Q)$, where $V_\epsilon$ is a Gaussian-noised image standing in for "no image" and the quantity is meant to measure how strongly the image drives the answer under a given question language. The mitigation mechanism is MVCL-MI, whose objective combines cross-entropy on source- and target-language answers with two KL-divergence terms $D_{\mathrm{KL}}(P^{\mathrm{tgt-src}}\|P^{\mathrm{src-src}})$ and $D_{\mathrm{KL}}(P^{\mathrm{src-tgt}}\|P^{\mathrm{tgt-tgt}})$ that pull cross-lingual output distributions toward the corresponding monolingual ones.

What would settle it

Recompute the paper's mutual-information estimate on the same ChartQA questions with a blank or black image in place of Gaussian noise; if the accuracy–mutual-information correlation across the eight languages disappears, the diagnostic and MVCL-MI's motivation are falsified. Running the same correlation on a sample of 1,000 or more examples rather than 100 would settle whether the effect is stable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that cross-lingual text-rich VQA fails because the question in a second language does not sufficiently activate the visual information needed to answer it, and this can be measured and repaired. The authors quantify activation as conditional mutual information $I(Y;V|Q)$ between the model's output tokens $Y$ and the image tokens $V$ given the question $Q$, using a Gaussian-noised image to approximate the no-image baseline. Across eight languages on ChartQA, accuracy and this mutual-information estimate are strongly correlated, with the source-language question delivering the most mutual information. MVCL-MI then maximizes the target-language mutual information indirectly: rather than minimizing target entropy directly (which invites degenerate sharp distributions), it minimizes KL divergence between cross-lingual output distributions and the monolingual output distribution, using the monolingual logits as teacher. Experimentally this narrows the cross-lingual gap while keeping monolingual scores essentially intact, and ablations show both the cross-lingual cross-entropy term and the KL term are needed.

Load-bearing premise

The load-bearing premise is that adding heavy Gaussian noise to an image makes the model behave as if the image were absent, so the mutual-information estimate really measures visual activation; the paper's causal story also depends on a correlation observed on 100 ChartQA examples.

Editorial extensions

If this is right

  • Cross-lingual accuracy should become a standard reporting axis for LVLMs, since even models trained with multilingual data retain a measurable gap.
  • Questions that demand faithful extraction or abstraction from image text degrade more than yes/no questions, so the cross-lingual failure is worst in the settings that need image comprehension most.
  • MVCL-MI can be layered onto an existing LVLM with a small fine-tuning budget and no need for paired parallel images, only for translated questions and answers.
  • The mutual-information estimate gives a training-free signal for when a model is answering from linguistic priors rather than from the image.
  • Ablation evidence shows the KL terms, not just extra cross-lingual training data, are what preserve monolingual behavior while closing the gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same mutual-information diagnostic could be applied to monolingual hallucination or OCR-failure cases, replacing "question language" with question phrasing to detect when a model ignores image content.
  • Editorial inference: if the Gaussian-noise equivalence holds, MVCL-MI should transfer to language pairs outside English–Chinese–French; the untested, harder case is a low-resource target language with a different script.
  • Editorial inference: because the method is fine-tuned and evaluated on the same XPaperQA data, an out-of-distribution benchmark (for example, foreign street signs or tables from a new domain) is the next check on whether the mutual-information fix generalizes.
  • Editorial inference: the accuracy–MI correlation suggests mutual information could serve as an inference-time confidence score that triggers translation or re-asking, though the paper does not propose this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces XT-VQA, a cross-lingual text-rich visual question answering benchmark that extends four existing VQA datasets with translated questions and adds a newly constructed bilingual academic-paper dataset, XPaperQA. It evaluates eight open and two closed LVLMs and reports a substantial accuracy drop when the question language differs from the image text language. The paper then proposes MVCL-MI, a distillation-based fine-tuning method that is claimed to maximize vision-language cross-lingual mutual information and to reduce the cross-lingual performance gap while preserving monolingual capability. The method is applied to MiniCPM-Llama3-V and evaluated on XPaperQA, with reported gains in cross-lingual accuracy and reductions in the gap.

Significance. If the central claims are established, XT-VQA and XPaperQA would be useful resources for studying a realistic and under-explored failure mode of LVLMs, and the reported cross-lingual gap across ten models is an important observation. The paper's strengths include the release of code and a new bilingual benchmark, the breadth of the model evaluation, and the explicit attempt to connect an information-theoretic diagnostic to a mitigation method. However, the current evidence for the method's effectiveness is undermined by the absence of any held-out evaluation split, and the mutual information derivation contains sign errors and unsupported equivalence claims. These issues prevent the paper from supporting its main conclusions as written.

major comments (5)
  1. [§6, Table 3 and Training Details] The paper never specifies a held-out split between the data used for MVCL-MI training and the XPaperQA data used for evaluation in Table 3. The training objective in Eq. (10) uses cross-lingual QA pairs, the method is described as deploying training on MiniCPM-Llama3-V for one epoch, and Table 5 reports only hyperparameters, with no mention of a train/test partition by paper, page, or QA pair. If the model is trained on XPaperQA and evaluated on the same XPaperQA, the reported gains (for example +1.9 on XPaperQA-en and +5.3 on XPaperQA-zh) and the claimed gap reductions from 19.4% to 13.1% and 29.3% to 21.3% are consistent with fitting the evaluation set rather than with genuine cross-lingual transfer. The central mitigation claim is therefore unsupported by the evidence presented.
  2. [§4, Eqs. (5) and (7)] The mutual information derivation contains sign and conditioning errors. In Eq. (5), the expansion of I(Y;V|Q) = H(Y|Q) − H(Y|V,Q) writes the first term as −Σ P(y|V) log P(y|V), which depends on V rather than Q, and the second term as −Σ P(Y|V,Q) log P(Y|V,Q), so the displayed expression equals H(Y|V) + H(Y|V,Q) rather than the stated difference. In Eq. (7), the expansion of H(Y|Vε,Q) − H(Y|V,Q) is written as Σ pθ(yi|Vε,Q) log pθ(yi|Vε,Q) − Σ pθ(yi|V,Q) log pθ(yi|V,Q), which equals −H(Y|Vε,Q) − H(Y|V,Q), not the claimed mutual information. These errors affect the numerical MI values, the correlation reported in Figure 4, and the diagnostic conclusion that cross-lingual questions insufficiently activate visual information.
  3. [§4.3, Figures 3-4 and Mutual Information Analysis] The paper claims a 'strong correlation' between accuracy and mutual information based on 100 randomly selected ChartQA examples and 'eight different languages,' but it reports no correlation coefficient, confidence interval, or significance test, and it does not list the eight languages or explain how they relate to the earlier description of XT-VQA as English/Chinese/French. The 100-example subset and the missing statistics are not sufficient to support the causal statement that the cross-lingual gap stems from insufficient activation of visual information. A concrete quantitative analysis with variance estimates is needed before this diagnostic claim can be accepted.
  4. [§5, Eqs. (9)-(10)] The paper asserts that minimizing the KL divergence in Eq. (9) is equivalent to maximizing the mutual information I(Y;V|Q) from Eq. (7), but no derivation or formal argument is given. Minimizing DKL(P_tgt || P_src) aligns the output distributions for target- and source-language questions; it does not directly optimize the difference between conditional entropies with and without visual tokens as defined in Eq. (7). Without a proof or an empirical verification that the KL objective increases the Eq. (7) quantity, the information-theoretic justification for MVCL-MI remains an unsupported equivalence claim.
  5. [§4, Eq. (7) and following paragraph] The assumption that adding Gaussian noise to the image is equivalent to removing the image, H(Y|Q) ≈ H(Y|Vε,Q), is asserted without validation. A heavily noised image is not the same input as an absent image, and residual text-like patterns or layout information could still influence the model's output distribution. This assumption is load-bearing for the entire MI analysis and for the motivation of MVCL-MI; the authors should provide an empirical check, such as comparing output entropy and accuracy under true no-image conditions versus heavy-noise conditions, before relying on it.
minor comments (6)
  1. [§3, Evaluation of LVLMs on XT-VQA] The text says 'Table 2 shows LVLM performance on XT-VQA,' but the performance table is labeled Table 1; Table 2 contains XPaperQA data statistics. The cross-reference should be corrected.
  2. [§3, Dataset Construction] The dataset name is inconsistent: the construction section uses 'XPaperVQA,' while elsewhere the paper uses 'XPaperQA.' One consistent name should be used throughout.
  3. [§4, Mutual Information Analysis across Languages] Figure references are swapped in places: the entropy distributions are shown in Figure 3 and the accuracy-MI statistics in Figure 4, but the text attributes them the other way around in two sentences.
  4. [Appendix, Model Details] The appendix refers to 'TeXT-VQA' in the dataset description; this should be 'TextVQA.'
  5. [§5, Ablation Study] The phrase 'w/o KL-Loss ... shown in F. 10' should read 'Eq. 10,' and the same correction applies to the w/o Cross-CE description.
  6. [§6, Table 4] The overall column of Table 4 shows values such as '26.0 +0.7' and '22.5 +1.9,' but the text reports an increase of 1.9 on XPaperQA-en; please clarify whether the reported number refers to overall accuracy or to the cross-lingual subset, and make the table and text consistent.

Circularity Check

1 steps flagged · score 6.0 of 10

MVCL-MI is fine-tuned and evaluated on the same XPaperQA benchmark with no described held-out split, so the reported cross-lingual gains are train-set measurements rather than independent predictions.

  1. fitted input called prediction [Methodology Eq. (10), Training Details; Experiments Table 3 and 'Experiment Results' paragraph]
    "Finally, our training objective is L = LCE(ysrc−src, ˆysrc) + LCE(ytgt−src, ˆysrc) + LCE(ysrc−tgt, ˆytgt) + LCE(ytgt−tgt, ˆytgt) + αLKL(P src−tgt||P tgt−tgt) + βLKL(P tgt−src||P src−src). ... We trained on 8 A100-sxm4-80gb for 1 epoch ... MVCL-MI effectively mitigates the cross-lingual gap ... with an increase of 1.9 (↑9.3%) on XPaperQA-en and 5.3 (↑16.3%) on XPaper-zh ..."

    The Eq. (10) objective is minimized on cross-lingual text-rich QA pairs; XPaperQA is the only dataset described that supplies such pairs, and no train/test split, held-out paper/page partition, or unseen subset is specified in the training details. Table 3, the sole evaluation of MVCL-MI, reports the same XPaperQA-en and XPaperQA-zh benchmark. The headline numbers (+5.3 on Chinese papers; gap reduction from 29.3% to 21.3%) are therefore measurements on the training distribution, not independent predictions. Since the abstract and conclusion treat these numbers as evidence that MVCL-MI 'effectively reduces the visual-text cross-lingual performance disparity,' the central claim reduces to fitting the benchmark unless a held-out split is supplied.

full rationale

The only clear circularity is the train/eval overlap in the MVCL-MI experiments: the method is fine-tuned with Eq. (10) on the same XPaperQA material used for Table 3, and no held-out split is described. Because the paper's central mitigation claim rests entirely on that table, the positive result is currently a fitted-input re-description rather than an independent prediction. The mutual-information diagnosis (Eq. 7 and the 100-example ChartQA correlation) is an empirical estimate, not circular, though it is fragile; the asserted equivalence between the KL objective and mutual-information maximization is an unproven formal claim rather than a circular reduction. Self-citations (e.g., Texthawk for OCR-tuned models) are peripheral and not load-bearing. Thus the circularity score is 6: the method's headline evaluation reduces, by construction, to training-set performance, but the benchmark construction and diagnostic analyses are not themselves circular.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The ledger captures the hand-chosen hyperparameters and assertions on which the MI diagnosis and MVCL-MI training rest. The alpha and beta weights, noise parameters, and data filtering thresholds are tuned or chosen without reported sensitivity analysis; the noise-equivalence and KL-to-MI equivalence are unproven assumptions.

free parameters (6)
  • alpha (Eq. 10 KL weight) = not reported
    Controls the strength of the KL distillation term from source-language anchor to target-language output; chosen by hand and not listed in the appendix hyperparameter table.
  • beta (Eq. 10 KL weight) = not reported
    Controls the reverse KL direction; same treatment as alpha.
  • Gaussian noise parameters mu, sigma for V_epsilon = not reported
    The noisy image in Eq. 7 is assumed equivalent to removing the image; the exact noise level affects every MI number in the paper.
  • XPaperQA confidence filter threshold = 7
    Discards Gemini answers with self-rated confidence below 7; changes the dataset composition.
  • XPaperQA similarity filter threshold = 0.1
    Jaccard similarity above 0.1 removes redundant QA pairs.
  • BERT similarity threshold for translated questions = unspecified predefined threshold
    Controls which machine-translated questions are manually corrected in the multilingual extension.
assumptions (4)
  • domain assumption Adding Gaussian noise to an image approximates making the model blind to the image, yielding H(Y|Q) approximately H(Y|V_epsilon, Q).
    Introduced in Eq. 7 and justified by reference to visual contrastive decoding; no quantitative validation is provided.
  • standard math The LVLM output distribution factorizes as p_theta(y|v,x) = product p_theta(y_i|v,x,y<i).
    Used in Eq. 4 and Eq. 6 to decompose the entropy over tokens.
  • ad hoc to paper KL divergence minimization between source- and target-language output distributions maximizes vision-language mutual information.
    Stated as 'this objective is equivalent to maximizing the mutual information' in the Methodology; the equivalence is asserted, not proven.
  • domain assumption The observed correlation between accuracy and mutual information reflects a causal mechanism: cross-lingual questions fail to activate visual information.
    The paper moves from the Figure 4 correlation to a causal diagnosis without a controlled test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective." pith.science (2026). https://pith.science/paper/SBXSA5X4

@misc{pith2026241217787,
  author       = {Pith},
  title        = {Pith review of: Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBXSA5X4}},
  note         = {Machine review of arXiv:2412.17787}
}
read the original abstract

Recent Large Vision-Language Models (LVLMs) have shown promising reasoning capabilities on text-rich images from charts, tables, and documents. However, the abundant text within such images may increase the model's sensitivity to language. This raises the need to evaluate LVLM performance on cross-lingual text-rich visual inputs, where the language in the image differs from the language of the instructions. To address this, we introduce XT-VQA (Cross-Lingual Text-Rich Visual Question Answering), a benchmark designed to assess how LVLMs handle language inconsistency between image text and questions. XT-VQA integrates five existing text-rich VQA datasets and a newly collected dataset, XPaperQA, covering diverse scenarios that require faithful recognition and comprehension of visual information despite language inconsistency. Our evaluation of prominent LVLMs on XT-VQA reveals a significant drop in performance for cross-lingual scenarios, even for models with multilingual capabilities. A mutual information analysis suggests that this performance gap stems from cross-lingual questions failing to adequately activate relevant visual information. To mitigate this issue, we propose MVCL-MI (Maximization of Vision-Language Cross-Lingual Mutual Information), where a visual-text cross-lingual alignment is built by maximizing mutual information between the model's outputs and visual information. This is achieved by distilling knowledge from monolingual to cross-lingual settings through KL divergence minimization, where monolingual output logits serve as a teacher. Experimental results on the XT-VQA demonstrate that MVCL-MI effectively reduces the visual-text cross-lingual performance disparity while preserving the inherent capabilities of LVLMs, shedding new light on the potential practice for improving LVLMs. Codes are available at: https://github.com/Stardust-y/XTVQA.git

Figures

Figures reproduced from arXiv: 2412.17787 by the authors.

Figure 1
Figure 1. An example of the LVLM answering unfaith [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The XPaperQA dataset construction pipeline consists of three parts: (1) Converting PDF papers into metadata using [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The entropy distribution of 100 randomly selected examples on the ChartQA dataset in 8 different languages, where [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Statistics of accuracy and mutual information over [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Data example from XT-VQA [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: An example of extractive QA from XPaperQA-en dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: An example of abstractive QA from XPaperQA-zh dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Data Visualization of XPaperQA, Chinese paper instruction was translated to English for clearer statistic. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 14 canonical work pages

  1. [1]

    Andersland, M. 2024. Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages. arXiv:2403.06354

    arXiv 2024

  2. [2]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

    arXiv 2023

  3. [3]

    Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D. M.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, ...

    arXiv 2020

  4. [4]

    Chen, Y.; Zhang, H.; Zhou, Y.; Bai, X.; Wang, Y.; Zhong, M.; Yan, J.; Li, Y.; Li, J.; Zhu, M.; and Zhang, Y. 2023. Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation. arXiv:2307.04018

    work page Pith review arXiv 2023

  5. [5]

    Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500

    arXiv 2023

  6. [6]

    A.; and Gardner, M

    Dasigi, P.; Lo, K.; Beltagy, I.; Cohan, A.; Smith, N. A.; and Gardner, M. 2021. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. arXiv:2105.03011

    arXiv 2021

  7. [7]

    Farquhar, S.; Kossen, J.; Kuhn, L.; and Gal, Y. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017): 625--630

    2024

  8. [8]

    Guha, N.; Nyarko, J.; Ho, D.; R \'e , C.; Chilton, A.; Chohlas-Wood, A.; Peters, A.; Waldon, B.; Rockmore, D.; Zambrano, D.; et al. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36

    work page 2024

Show all 47 references
  1. [9]

    L.; Schneider, F.; Yu, S.; Bhiwandiwalla, A.; Lauscher, A.; Tseng, S.; and Lal, V

    Hinck, M.; Holtermann, C.; Olson, M. L.; Schneider, F.; Yu, S.; Bhiwandiwalla, A.; Lauscher, A.; Tseng, S.; and Lal, V. 2024. Why do LLaVA Vision-Language Models Reply to Images in English? arXiv preprint arXiv:2407.02333

    2024 arXiv

  2. [10]

    Holtzman, A.; West, P.; Shwartz, V.; Choi, Y.; and Zettlemoyer, L. 2021. Surface Form Competition: Why the Highest Probability Answer Isn ' t Always Right. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods ...

    2021

  3. [11]

    Huang, K.; Mo, F.; Li, H.; Li, Y.; Zhang, Y.; Yi, W.; Mao, Y.; Liu, J.; Xu, Y.; Xu, J.; Nie, J.-Y.; and Liu, Y. 2024. A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers. arXiv:2405.10936

    2024 arXiv

  4. [12]

    Jelinek, F. 1985. Markov source modeling of text generation. In The impact of processing techniques on communications, 569--591. Springer

    1985

  5. [13]

    Kullback, S.; and Leibler, R. A. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1): 79--86

    1951

  6. [14]

    Lee, K.; Joshi, M.; Turc, I.; Hu, H.; Liu, F.; Eisenschlos, J.; Khandelwal, U.; Shaw, P.; Chang, M.-W.; and Toutanova, K. 2023. Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding. arXiv:2210.03347

    2023 arXiv

  7. [15]

    Leng, S.; Zhang, H.; Chen, G.; Li, X.; Lu, S.; Miao, C.; and Bing, L. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13872--13882

    2024

  8. [16]

    Li, J.; Galley, M.; Brockett, C.; Gao, J.; and Dolan, B. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Knight, K.; Nenkova, A.; and Rambow, O., eds., Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Co...

    2016

  9. [17]

    Li, J.; and Jurafsky, D. 2016. Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv:1601.00372

    2016 arXiv

  10. [18]

    Li, Z.; Yang, B.; Liu, Q.; Ma, Z.; Zhang, S.; Yang, J.; Sun, Y.; Liu, Y.; and Bai, X. 2024. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26763--26773

    2024

  11. [19]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. arXiv:2304.08485

    2023 arXiv

  12. [20]

    Lu, P.; Bansal, H.; Xia, T.; Liu, J.; Li, C.; Hajishirzi, H.; Cheng, H.; Chang, K.-W.; Galley, M.; and Gao, J. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255

    2023 arXiv

  13. [21]

    Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507--2521

    2022

  14. [22]

    Luo, C.; Shen, Y.; Zhu, Z.; Zheng, Q.; Yu, Z.; and Yao, C. 2024. LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15630--15640

    2024

  15. [23]

    X.; Tan, J

    Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244

    2022 arXiv

  16. [24]

    Mathew, M.; Karatzas, D.; and Jawahar, C. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2200--2209

    2021

  17. [25]

    K.; and Chakraborty, A

    Mishra, A.; Shekhar, S.; Singh, A. K.; and Chakraborty, A. 2019. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), 947--952. IEEE

    2019

  18. [26]

    Mu, Y.; Zhang, Q.; Hu, M.; Wang, W.; Ding, M.; Jin, J.; Wang, B.; Dai, J.; Qiao, Y.; and Luo, P. 2024. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Nandwani, Y.; Kumar, V.; Raghu, D.; Joshi, S.; and Lastras, L. A. 2023. Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs. arXiv:2305.12191

    2023 arXiv

  20. [28]

    Niu, R.; Li, J.; Wang, S.; Fu, Y.; Hu, X.; Leng, X.; Kong, H.; Chang, Y.; and Wang, Q. 2024. ScreenAgent: A Vision Language Model-driven Computer Control Agent. arXiv:2402.07945

    2024 arXiv

  21. [29]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

    2021 arXiv

  22. [30]

    Schneider, F.; and Sitaram, S. 2024. M 5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks. arXiv:2407.03791

    2024 arXiv

  23. [31]

    Shin, D.; Lim, H.; Won, I.; Choi, C.; Kim, M.; Song, S.; Yoo, H.; Kim, S.; and Lim, K. 2024. X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment. arXiv:2403.11399

    2024 arXiv

  24. [32]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards VQA Models That Can Read. arXiv:1904.08920

    2019 arXiv

  25. [33]

    Takayama, J.; and Arase, Y. 2019. Relevant and Informative Response Generation using Pointwise Mutual Information. In Chen, Y.-N.; Bedrax-Weiss, T.; Hakkani-Tur, D.; Kumar, A.; Lewis, M.; Luong, T.-M.; Su, P.-H.; and Wen, T.-H., eds., Proceedings of the First Workshop on NLP f...

    2019

  26. [34]

    Tang, J.; Liu, Q.; Ye, Y.; Lu, J.; Wei, S.; Lin, C.; Li, W.; Mahmood, M. F. F. B.; Feng, H.; Zhao, Z.; et al. 2024. MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering. arXiv preprint arXiv:2405.11985

    2024 arXiv

  27. [35]

    H.; Wieting, J.; Lin, J.; and Cer, D

    Thakur, N.; Ni, J.; Ábrego, G. H.; Wieting, J.; Lin, J.; and Cer, D. 2024. Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval. arXiv:2311.05800

    2024 arXiv

  28. [36]

    Tishby, N.; and Zaslavsky, N. 2015. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), 1--5. IEEE

    2015

  29. [37]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971

    2023 arXiv

  30. [38]

    Tschannen, M.; Mustafa, B.; and Houlsby, N. 2023. Clippo: Image-and-language understanding from pixels only. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11006--11017

    2023

  31. [39]

    Wan, Z.; Liu, C.; Zhang, M.; Fu, J.; Wang, B.; Cheng, S.; Ma, L.; Quilodr \'a n-Casas, C.; and Arcucci, R. 2024. Med-unic: Unifying cross-lingual medical vision-language pre-training by diminishing bias. Advances in Neural Information Processing Systems, 36

    2024

  32. [40]

    Wang, B.; Liu, Z.; Huang, X.; Jiao, F.; Ding, Y.; Aw, A.; and Chen, N. F. 2024. SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning. arXiv:2309.04766

    2024 arXiv

  33. [41]

    Wang, J.; Liang, Y.; Meng, F.; Zou, B.; Li, Z.; Qu, J.; and Zhou, J. 2023. Zero-Shot Cross-Lingual Summarization via Large Language Models. arXiv:2302.14229

    2023 arXiv

  34. [42]

    Xiao, Y.; and Wang, W. Y. 2021. On hallucination and predictive uncertainty in conditional language generation. arXiv preprint arXiv:2103.15025

    2021 arXiv

  35. [43]

    Yang, Y.; Zhou, T.; Li, K.; Tao, D.; Li, L.; Shen, L.; He, X.; Jiang, J.; and Shi, Y. 2024. Embodied multi-modal agent trained by an llm from a parallel textworld. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 26275--26285

    2024

  36. [44]

    A.; and Huang, F

    Ye, J.; Hu, A.; Xu, H.; Ye, Q.; Yan, M.; Xu, G.; Li, C.; Tian, J.; Qian, Q.; Zhang, J.; Jin, Q.; He, L.; Lin, X. A.; and Huang, F. 2023. UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model. arXiv:2310.05126

    2023 arXiv

  37. [45]

    Yu, Y.-Q.; Liao, M.; Wu, J.; Liao, Y.; Zheng, X.; and Zeng, W. 2024. Texthawk: Exploring efficient fine-grained perception of multimodal large language models. arXiv preprint arXiv:2404.09204

    2024 arXiv

  38. [46]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.