Pith. sign in

REVIEW 2 major objections 4 minor 82 references

VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Encoding each page as several precomputed interpretive variants, then letting each query pick the best match, lifts visual document retrieval to state-of-the-art results on ViDoRe V2/V3.

desk verdict A genuinely new variant-representation mechanism for late-interaction document retrieval, but the headline SOTA claim is undercut by test-set hyperparameter tuning; worth refereeing with revisions. read the letter →

arxiv 2608.01211 v1 pith:BWBSCFEB submitted 2026-08-02 cs.CV

Haocheng Wang , Tongkun Guan , Wei Shen , Xiaokang Yang This is my paper · ORCID
classification cs.CV
keywords visualdocumentretrievallateinteractionvariantrepresentationslatentprobingtokensbranchedencodingmultimodallargelanguagemodelsretrieval-augmentedgenerationNDCG@10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual document retrieval systems that scale to millions of pages must encode documents offline, before any query arrives. VaRS-Doc argues that a fixed, query-agnostic document vector is the wrong target: the same page can legitimately be read as a comparison, a trend analysis, or a risk assessment depending on the query, and one vector can only encode the average of those readings. The paper proposes encoding each page as several variant representations, produced by learnable latent probing tokens that read the document and then guide separate encoding branches, and letting each query score the page by its best-matching variant. On the ViDoRe V2 and V3 benchmarks the method raises the overall average NDCG@10 (a standard ranking-quality score) from 55.5 with the ColQwen2.5 baseline to 60.0, improving on all twelve datasets, while keeping index time and query latency close to the single-representation baseline. If correct, the work reframes late-interaction retrieval as preparation, not compression: documents should be pre-encoded for multiple possible readings, not squeezed into one.

What carries the argument

Latent self-probing: K learnable 'interpretation probing tokens' are appended to the document token sequence. In shared contextualization (the first L−n transformer blocks) the probes read the document without influencing it; in branched encoding (the final n blocks) each branch's attention mask exposes the document tokens to exactly one probe, and the branch produces a full document embedding variant. Retrieval uses variant-aware late interaction: a MaxSim score is computed for each variant and the maximum is taken. Training is two-stage: a single-variant warm-up establishes a reliable retrieval space, then multi-variant adaptation applies a hardness-aware InfoNCE loss, a batch-level balanc

What would settle it

Run a matched-capacity control where the K branches use identical attention masks (or frozen probe tokens) so variants cannot diverge; if retrieval stays at the full-model level on ViDoRe V2, then branched interpretation is not the cause. Alternatively, on a sample of several hundred pages, if same-intent variant-choice agreement is statistically no higher than different-intent agreement (Psame ≈ Pcross), the variants are not capturing query intent.

Watch

Extended reading notes

Core claim

The paper's central claim is that the late-interaction paradigm's fixed per-document representation is the limiting factor in visual document retrieval, and that part of query-conditioned interpretation can be recovered offline. Concretely, VaRS-Doc inserts K learnable 'latent interpretation probing tokens' after the document tokens. In the early transformer blocks these probes attend to the document but the document cannot attend to them, so they become document-conditioned silent readers; in the final n blocks the document states gain access to exactly one probe each, creating K parallel branches, and each branch emits a full document embedding set. Retrieval is then variant-aware late int

Load-bearing premise

The load-bearing premise is that the learnable probing tokens discover genuinely distinct and complementary interpretations of a document that generalize to unseen queries, rather than converging to redundant variants whose only real effect is added model capacity; the paper's evidence for this is a 100-query complementarity analysis with no statistical testing.

Editorial extensions

If this is right

  • Retrieval quality improves across all twelve ViDoRe V2/V3 datasets, with the largest gains on Economics Reports (+7.1), Industrial Documents (+7.7), and human-labeled ESG Reports (+7.0), suggesting that multi-interpretable visual documents benefit most from variant representations.
  • The efficiency cost of diversification stays modest: with K=5 the index is 5× larger, but index time grows only 1.14× and per-query latency grows from 22.38 ms to 27.15 ms, so the method remains practical for first-stage retrieval at scale.
  • Branched encoding, not extra projection heads, is what creates the gain: adding K projection heads alone moves NDCG@10 from 61.6 to 61.8, while branched encoding moves it to 65.8 — meaning the differentiation has to happen inside the encoder, not at the output layer.
  • The two-stage training strategy is load-bearing: skipping the single-variant warm-up drops NDCG@10 from 66.5 to 63.7 and makes one branch win 94.2% of samples, showing that without the warm-up the variant mechanism collapses into a single dominant representation.
  • Variant-selection behavior is query-sensitive: same-intent paraphrases pick the same variant 77.6% of the time against 30.8% for different intents, supporting the paper's interpretation that variants are complementary readings rather than random noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We conjecture the same latent-probing recipe transfers to other late-interaction retrievers: any model that indexes documents offline could append probing tokens and branched blocks, so the reported gains may not be specific to ColQwen2.5 — a claim the paper does not test.
  • Because the final score is a max over K variants, the method implicitly learns a query-to-intent routing at retrieval time. An untested extension is to expose the identity of the winning variant as an interpretable label or to train a shallow router that scores only the most promising variant, trading a little accuracy for lower latency.
  • The paper's complementarity evidence rests on 100 sampled pages and no significance test. Scaling that analysis to a larger sample and reporting the association between query intent and variant choice is the direct way to check whether the variants are truly interpreting the document rather than supplying extra capacity.
  • A testable consequence of the hardness-aware weighting is that the variants should specialize most sharply on hard negatives; one could verify this by clustering the negatives each variant ranks highest and checking whether they correspond to different semantic aspects of the evidence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes VaRS-Doc, a visual document retrieval framework built on ColQwen2.5 that precomputes K latent 'interpretation' variants of each document instead of a single fixed representation. Variants are produced by inserting K learnable probing tokens after the document tokens, reading the document in shared contextualization blocks, then allowing each probe to guide a separate branch of the final n transformer blocks via a visibility mask (Eqs. 3–10). At retrieval, the query is encoded once and the document score is the max over the K variants of the late-interaction MaxSim score (Eqs. 13–14). Training is two-stage: a single-variant warm-up (Eq. 15), then multi-variant adaptation with a hardness-weighted InfoNCE loss, a balance loss, and a diversity loss (Eqs. 16–21). Experiments on ViDoRe V2 and V3 report average NDCG@10 of 66.5 and 56.7, an overall average of 60.0, and improvements over the ColQwen2.5 baseline on all 12 datasets. Ablations, efficiency measurements, and a small-scale complementarity analysis are also provided.

Significance. The conceptual direction is timely and interesting: it attacks a real limitation of late-interaction retrieval—fixed offline document representations—without giving up offline indexing. The proposed shared-contextualization / branched-encoding design is simple and plausible, and the two-stage training objective directly addresses the branch-collapse failure mode. The projection-only control in Table 4 is a thoughtful check against the trivial explanation that the gains come merely from extra output capacity. If the reported results are out-of-sample, the improvements are practically meaningful (about 4.5 NDCG@10 points overall). The paper is clearly written and includes efficiency measurements that support the practical claims. However, the empirical protocol has a load-bearing weakness: the final configuration is selected on ViDoRe V2, which is one of the two test benchmarks, and no held-out validation split is described. This compromises the main SOTA claim as a generalization estimate.

major comments (2)
  1. [Appendix B.2, Table 1] Hyperparameter selection is performed on a test benchmark. Figures B.1/B.2 and Tables 4/5 select K, n, lambda_bal, lambda_div, and gamma by measuring NDCG@10 on ViDoRe V2, which is also one of the two benchmarks in Table 1. No held-out validation split is described anywhere in the paper. Consequently, the headline V2 average (66.5) and the combined overall average (60.0) are produced by a configuration explicitly tuned on that test set; the V3 numbers are also affected by these choices. The claim of outperforming ColQwen2.5 on all 12 datasets is therefore not a clean out-of-sample result. Please add a validation split, re-run with fixed defaults, or otherwise demonstrate that the selected hyperparameters are not overfit (e.g., sensitivity curves across random subsets of the training data).
  2. [Table 2 and Appendix C] The complementarity evidence is thin and not statistically grounded. The analysis uses N=100 manually selected regions, and the only numbers reported are Psame=77.6% and Pcross=30.8%. With N=100, the standard errors are substantial, and no confidence intervals or permutation tests are provided. Moreover, the queries are generated from the evidence region itself by an MLLM, which may inflate the apparent alignment between query intent and variant selection. This is the only direct evidence that the variants are semantically distinct 'interpretations' rather than merely extra capacity; Table 4's projection-only control does not rule out capacity added by the branched LoRA layers. Please report statistical significance, confidence intervals, and clarify the query-generation protocol's possible bias.
minor comments (4)
  1. [Table 1] The header formatting is too compressed; the column groups for ViDoRe V2 and ViDoRe V3, including the two 'Avg.' columns, must be visually separated. I parsed the table as internally consistent (V2 avg 66.5 vs baseline 61.6; V3 avg 56.7 vs 52.5), but the compressed header invites misreading (as happened in the reader's report).
  2. [Table 5] The caption says 'averaged over four datasets' but does not state that these are the ViDoRe V2 datasets. Moreover, no per-dataset breakdown or variance estimate is given; for small deltas (e.g., Table 4, 61.8 vs 61.6), it is unclear whether the difference is meaningful.
  3. [Eq. (7)] The indicator notation 1[1≤i≤M, j=M+k] is clear but should be defined in the text. Also specify whether the probe token's own attention to document tokens remains active in the branch blocks (it should, given the shared stage), since this affects the interpretation of the mask.
  4. [Table B.1 / Section B.1] The three aggregation strategies differ not only in aggregation but also in whether query token representations see fused variant tokens. Please clarify the token-level fusion implementation, since the comparison is meant to isolate the aggregation rule.

Circularity Check

1 steps flagged · score 5.0 of 10

Reported ViDoRe V2 gains are partially fitted: hyperparameters were selected on the same benchmark used for the headline result.

  1. fitted input called prediction [Appendix B.2 (Hyperparameter Analysis) and Table 1 (Main Results)]
    "As shown in Figure B.1(a), increasing the number of variants from one to three improves NDCG@10 from 61.6 to 65.1 ... Performance further increases to 66.5 with five variants ... Figure B.2(a) shows ... Increasing λ bal from 0 to 0.1 improves NDCG@10 from 65.8 to 66.5 ... Performance increases from 65.7 with γ = 0 to 66.5 with γ = 0.05 ... We therefore use λ bal = 0.1, λ div = 0.05, and γ = 0.05 in all experiments. ... VaRS-Doc achieves state-of-the-art performance on both ViDoRe V2 and V3, obtaining average NDCG@10 scores of 66.5 and 56.7, respectively, and an overall average of 60.0."

    The hyperparameters K, n, λbal, λdiv, and γ are chosen by maximizing NDCG@10 on ViDoRe V2 in Appendix B.2. The same ViDoRe V2 average (66.5) is then reported in Table 1 as the headline V2 result and contributes to the overall 60.0. Thus the reported V2 score is the maximum of a search over configurations evaluated on the test set, not an independent out-of-sample measurement. The claim of outperforming ColQwen2.5 on all 12 datasets is partly forced for the four V2 datasets, since the configuration was selected to maximize performance there. V3 results are less affected, so the circularity is partial, but the central SOTA claim is not a clean independent evaluation.

full rationale

The architectural derivation of VaRS-Doc is not circular: the shared-contextualization/branched-encoding design, the loss terms, and the variant-aware late interaction follow from the stated problem formulation without relying on a self-citation chain or an imported uniqueness theorem. No ansatz is smuggled in via citation, and no known result is merely renamed. The only substantive circularity is in the evaluation protocol: hyperparameters are tuned directly on ViDoRe V2, one of the two benchmarks used for the final claim, so the V2 and overall averages are partially fitted rather than predicted. V3 provides some independent evidence, and the ablations suggest the components have real effects, but the headline SOTA claim is weakened by this test-set selection.

Assumptions & free parameters 6 free parameters · 3 assumptions · 1 invented entities

The central claim rests on two categories of unverified input: hyperparameters chosen on the test benchmark, and the behavioral assumption that latent probes yield distinct interpretations. There are no external constants or first-principles derivations.

free parameters (6)
  • Number of variants K = 5
    Selected by ablation on ViDoRe V2 (Appendix B.2); the paper notes K=7 gives only marginal gain. This is a hand-chosen capacity parameter affecting the main result.
  • Branched blocks n = 4
    Selected by ablation on ViDoRe V2 (Appendix B.2); branching at 6 or 8 blocks hurts performance.
  • Balance loss weight λ_bal = 0.1
    Chosen from ablation on ViDoRe V2 (Figure B.2).
  • Diversity loss weight λ_div = 0.05
    Chosen from ablation on ViDoRe V2 (Figure B.2).
  • Diversity margin γ = 0.05
    Chosen from ablation on ViDoRe V2 (Figure B.2).
  • LoRA rank and scaling = r=32, α=32
    Standard LoRA configuration, not ablated in the paper.
assumptions (3)
  • domain assumption Late-interaction MaxSim is a valid relevance scoring function.
    Adopted from ColBERT/Khattab and Zaharia (2020) and used in Eq. 13. The whole retrieval design assumes token-level max-similarity is an effective relevance model.
  • ad hoc to paper The latent probing tokens, after conditioning on the document, are sufficient to capture distinct interpretations without explicit natural-language supervision.
    The core hypothesis of the method, stated in the Method section. No external evidence beyond the paper's own experiments supports this directly.
  • domain assumption The ColPali training set (118,695 query-page pairs) is representative and sufficient for training a visual document retriever.
    The paper trains only on this dataset (Appendix A). If the training distribution is misaligned with the ViDoRe test queries, the reported gains may not generalize.
invented entities (1)
  • Latent interpretation probing tokens
    purpose: Learnable vectors appended to document tokens that read the document and then guide each encoding branch to produce a different interpretation.
    These tokens only exist inside the trained model; their effect is measured indirectly through retrieval scores and the small complementarity analysis. There is no external falsifiable prediction that isolates them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval." pith.science (2026). https://pith.science/paper/BWBSCFEB

@misc{pith2026260801211,
  author       = {Pith},
  title        = {Pith review of: VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWBSCFEB}},
  note         = {Machine review of arXiv:2608.01211}
}
read the original abstract

Visual document retrieval has recently become increasingly important in applications such as enterprise search, scientific literature discovery, and retrieval-augmented generation. These applications depend on efficiently identifying query-relevant pages across large collections of visually rich documents. Existing methods commonly adopt late-interaction architectures that encode and index documents offline to enable scalable and low-latency online retrieval. Despite its efficiency, this paradigm requires each document to be encoded into a fixed representation before the query is known. However, the same content in a visual document may induce different interpretations depending on the query intent, which a fixed representation struggles to capture. Yet postponing document encoding until the query arrives would incur prohibitive online retrieval latency. To address this gap, we propose VaRS-Doc, a visual document retrieval framework that diversifies document representations by enabling the model to actively explore variant latent interpretations during document encoding, while preserving efficient late-interaction retrieval in which each query adaptively selects the best-fit representation. We further introduce a two-stage training strategy that encourages the model to capture complementary semantic interpretations and prevents it from falling back to train a single dominant representation. Experiments on visual document retrieval benchmarks show that VaRS-Doc achieves state-of-the-art retrieval performance, offering a practical solution to the mismatch between query-agnostic document encoding and query-specific retrieval needs. Code is available at https://github.com/bokufa/VaRS-Doc.

Figures

Figures reproduced from arXiv: 2608.01211 by the authors.

Figure 1
Figure 1. Motivation of VaRS-Doc. Scalable retrieval re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of VaRS-Doc. During offline indexing, latent interpretation probing tokens silently read the document [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 64 canonical work pages

  1. [1]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; et al. 2022. Flamingo: A Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems, volume 35, 23716--23736

    2022

  2. [2]

    Bai, J.; Bai, S.; Yang, S.; et al. 2023. Qwen-VL : A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

    arXiv 2023

  3. [4]

    Balarini, J. P. 2025. Eager Embed V1: Multimodal Dense Embeddings for Retrieval

    2025

  4. [6]

    Chen, J.; Xiao, S.; Zhang, P.; Luo, K.; Lian, D.; and Liu, Z. 2024 a . M3 -Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. In Findings of the Association for Computational Linguistics: ACL 2024, 2318--2335

    work page 2024

  5. [7]

    Chen, Z.; Wu, J.; Wang, W.; et al. 2024 b . InternVL : Scaling Up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 24185--24198

    work page 2024

  6. [8]

    Faysse, M.; Sibille, H.; Wu, T.; Omrani, B.; Viaud, G.; Hudelot, C.; and Colombo, P. 2025. ColPali : Efficient Document Retrieval with Vision Language Models. In International Conference on Learning Representations

    work page 2025

  7. [9]

    Fu, P.; Guan, T.; Wang, Z.; et al. 2025. Multimodal Large Language Models for Text-Rich Image Understanding: A Comprehensive Review. In Findings of the Association for Computational Linguistics: ACL 2025, 19941--19958

    work page 2025

  8. [10]

    Golchinfar, D. 2025. SauerkrautLM-ColPali: Multi-Vector Vision Retrieval Models

    work page 2025

Show all 82 references
  1. [11]

    Guan, T.; Wang, H.; Shen, W.; and Yang, X. 2026. LightSTAR : Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement. In European Conference on Computer Vision

    2026

  2. [12]

    Guan, T.; Wang, Z.; Fu, P.; et al. 2025. A Token-Level Text Image Foundation Model for Document Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 23210--23220

    2025

  3. [13]

    K.; et al

    G \"u nther, M.; Sturua, S.; Akram, M. K.; et al. 2025. jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval. In Proceedings of the 5th Workshop on Multilingual Representation Learning, 531--550

    2025

  4. [14]

    J.; Shen, Y.; Wallis, P.; et al

    Hu, E. J.; Shen, Y.; Wallis, P.; et al. 2022. LoRA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2022

  5. [15]

    Jiang, T.; Song, M.; Zhang, Z.; et al. 2024. E5-V : Universal Embeddings with Multimodal Large Language Models. arXiv:2407.12580

    2024 arXiv

  6. [16]

    Jiang, Z.; Meng, R.; Yang, X.; Yavuz, S.; Zhou, Y.; and Chen, W. 2025. VLM2Vec : Training Vision-Language Models for Massive Multimodal Embedding Tasks. In International Conference on Learning Representations

    2025

  7. [17]

    Khattab, O.; and Zaharia, M. 2020. ColBERT : Efficient and Effective Passage Search via Contextualized Late Interaction over BERT . In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 39--48

    2020

  8. [18]

    Kim, G.; Hong, T.; Yim, M.; et al. 2022. OCR -Free Document Understanding Transformer. In European Conference on Computer Vision, 498--517

    2022

  9. [19]

    R.; et al

    Lee, K.; Joshi, M.; Turc, I. R.; et al. 2023. Pix2Struct : Screenshot Parsing as Pretraining for Visual Language Understanding. In Proceedings of the 40th International Conference on Machine Learning, 18893--18912

    2023

  10. [20]

    Lewis, P.; Perez, E.; Piktus, A.; et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, 9459--9474

    2020

  11. [21]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022. BLIP : Bootstrapping Language-Image Pre-Training for Unified Vision-Language Understanding and Generation. In Proceedings of the 39th International Conference on Machine Learning, 12888--12900

    2022

  12. [23]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. In Advances in Neural Information Processing Systems, volume 36, 34892--34916

    2023

  13. [24]

    Loison, A.; Mac \'e , Q.; Edy, A.; Xing, V.; Balough, T.; Moreira, G. d. S. P.; Liu, B.; Faysse, M.; Hudelot, C.; and Viaud, G. 2026. V i D o R e V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios. In Liakata, M.; Moreira, V. P.; Z...

    2026

  14. [26]

    Muennighoff, N.; Tazi, N.; Magne, L.; and Reimers, N. 2023. MTEB : Massive Text Embedding Benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2014--2037

    2023

  15. [27]

    Nussbaum, Z.; Duderstadt, B.; and Mulyar, A. 2024. Nomic Embed Vision: Expanding the Latent Space. arXiv:2406.18587

    2024 arXiv

  16. [28]

    Qin, X.; Zhang, P.; Yang, J. J. O.; Zeng, G.; Li, Y.; Wang, Y.; Zhang, W.; and Dai, P. 2025. Clip is almost all you need: towards parameter-efficient scene text retrieval without OCR. In Proceedings of the Computer Vision and Pattern Recognition Conference, 24873--24883

    2025

  17. [29]

    W.; Hallacy, C.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; et al. 2021. Learning Transferable Visual Models from Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, 8748--8763

    2021

  18. [30]

    Robertson, S.; and Zaragoza, H. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4): 333--389

    2009

  19. [31]

    Team, N. 2025. Nomic Embed Multimodal: Interleaved Text, Image, and Screenshots for Visual Document Retrieval

    2025

  20. [32]

    Tschannen, M.; Gritsenko, A.; Wang, X.; et al. 2025. SigLIP 2 : Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv:2502.14786

    2025 arXiv

  21. [33]

    van den Oord, A.; Li, Y.; and Vinyals, O. 2018. Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748

    2018 arXiv

  22. [34]

    Xu, Y.; Li, M.; Cui, L.; Huang, S.; Wei, F.; and Zhou, M. 2020. LayoutLM : Pre-Training of Text and Layout for Document Image Understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1192--1200

    2020

  23. [35]

    Xu, Y.; Xu, Y.; Lv, T.; et al. 2021. LayoutLMv2 : Multi-Modal Pre-Training for Visually-Rich Document Understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2579--2591

    2021

  24. [36]

    Yu, S.; Tang, C.; Xu, B.; et al. 2025. VisRAG : Vision-Based Retrieval-Augmented Generation on Multi-Modality Documents. In International Conference on Learning Representations

    2025

  25. [37]

    Zeng, G.; Zhang, Y.; Wei, J.; Yang, D.; Zhang, P.; Gao, Y.; Qin, X.; and Zhou, Y. 2024. Focus, distinguish, and prompt: Unleashing clip for efficient and flexible scene text retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, 2525--2534

    2024

  26. [38]

    Zeng, G.; Zhang, Y.; Zhou, Y.; and Yang, X. 2021. Beyond ocr+ vqa: Involving ocr into the flow for robust and accurate textvqa. In Proceedings of the 29th ACM international conference on multimedia, 376--385

    2021

  27. [39]

    Zeng, G.; Zhang, Y.; Zhou, Y.; Yang, X.; Jiang, N.; Zhao, G.; Wang, W.; and Yin, X.-C. 2023. Beyond OCR+ VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa. Pattern Recognition, 138: 109337

    2023

  28. [40]

    Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11975--11986

    2023

  29. [41]

    Zhang, J.; Zhang, Q.; Wang, B.; et al. 2025. OCR Hinders RAG : Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation. arXiv:2412.02592

    2025 arXiv

  30. [42]

    Zhu, J.; Wang, W.; Chen, Z.; et al. 2025. InternVL3 : Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv:2504.10479

    2025 arXiv

  31. [43]

    Advances in Neural Information Processing Systems , volume =

    Jean-Baptiste Alayrac and Jeff Donahue and Pauline Luc and others , title =. Advances in Neural Information Processing Systems , volume =

  32. [44]

    2023 , eprint =

    Jinze Bai and Shuai Bai and Shusheng Yang and others , title =. 2023 , eprint =

    2023

  33. [45]

    PaliGemma: A Versatile 3B Vision-Language Model for Transfer , year =

    Lucas Beyer and Andreas Steiner and Andr. PaliGemma: A Versatile 3B Vision-Language Model for Transfer , year =. 2407.07726 , archivePrefix =

    arXiv

  34. [46]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Zhe Chen and Jiannan Wu and Wenhai Wang and others , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  35. [47]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages =

    Jianlyu Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu , title =. Findings of the Association for Computational Linguistics: ACL 2024 , pages =

    2024

  36. [48]

    International Conference on Learning Representations , year =

    Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and C. International Conference on Learning Representations , year =

  37. [49]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages =

    Pei Fu and Tongkun Guan and Zining Wang and others , title =. Findings of the Association for Computational Linguistics: ACL 2025 , pages =

    2025

  38. [50]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

    Tongkun Guan and Zining Wang and Pei Fu and others , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

  39. [51]

    jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval , booktitle =

    Michael G. jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval , booktitle =

  40. [52]

    Hu and Yelong Shen and Phillip Wallis and others , title =

    Edward J. Hu and Yelong Shen and Phillip Wallis and others , title =. International Conference on Learning Representations , year =

  41. [53]

    2024 , eprint =

    Ting Jiang and Minghui Song and Zihan Zhang and others , title =. 2024 , eprint =

    2024

  42. [54]

    International Conference on Learning Representations , year =

    Ziyan Jiang and Rui Meng and Xinyi Yang and Semih Yavuz and Yingbo Zhou and Wenhu Chen , title =. International Conference on Learning Representations , year =

  43. [55]

    Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages =

    Omar Khattab and Matei Zaharia , title =. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages =

  44. [56]

    European Conference on Computer Vision , pages =

    Geewook Kim and Teakgyu Hong and Moonbin Yim and others , title =. European Conference on Computer Vision , pages =

  45. [57]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Kenton Lee and Mandar Joshi and Iulia Raluca Turc and others , title =. Proceedings of the 40th International Conference on Machine Learning , pages =

  46. [58]

    Advances in Neural Information Processing Systems , volume =

    Patrick Lewis and Ethan Perez and Aleksandra Piktus and others , title =. Advances in Neural Information Processing Systems , volume =

  47. [59]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi , title =. Proceedings of the 39th International Conference on Machine Learning , pages =

  48. [60]

    Advances in Neural Information Processing Systems , volume =

    Haotian Liu and Chunyuan Li and Qingyang Wu and Yong Jae Lee , title =. Advances in Neural Information Processing Systems , volume =

  49. [61]

    Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages =

    Niklas Muennighoff and Nouamane Tazi and Lo. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages =

  50. [62]

    2024 , eprint =

    Zach Nussbaum and Brandon Duderstadt and Andriy Mulyar , title =. 2024 , eprint =

    2024

  51. [63]

    2018 , eprint =

    Aaron van den Oord and Yazhe Li and Oriol Vinyals , title =. 2018 , eprint =

    2018

  52. [64]

    Proceedings of the 38th International Conference on Machine Learning , pages =

    Alec Radford and Jong Wook Kim and Chris Hallacy and others , title =. Proceedings of the 38th International Conference on Machine Learning , pages =

  53. [65]

    Robertson and Steve Walker and Susan Jones and Micheline Hancock-Beaulieu and Mike Gatford , title =

    Stephen E. Robertson and Steve Walker and Susan Jones and Micheline Hancock-Beaulieu and Mike Gatford , title =. Text REtrieval Conference , year =

  54. [66]

    Foundations and Trends in Information Retrieval , volume =

    Stephen Robertson and Hugo Zaragoza , title =. Foundations and Trends in Information Retrieval , volume =

  55. [67]

    A Statistical Interpretation of Term Specificity and Its Application in Retrieval , journal =

    Karen Sp. A Statistical Interpretation of Term Specificity and Its Application in Retrieval , journal =

  56. [68]

    2025 , eprint =

    Michael Tschannen and Alexey Gritsenko and Xiao Wang and others , title =. 2025 , eprint =

    2025

  57. [69]

    Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =

    Yiheng Xu and Minghao Li and Lei Cui and Shaohan Huang and Furu Wei and Ming Zhou , title =. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages =

  58. [70]

    Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics , pages =

    Yang Xu and Yiheng Xu and Tengchao Lv and others , title =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics , pages =

  59. [71]

    International Conference on Learning Representations , year =

    Shi Yu and Chaoyue Tang and Bokai Xu and others , title =. International Conference on Learning Representations , year =

  60. [72]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

    Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

  61. [73]

    2025 , eprint =

    Junyuan Zhang and Qintong Zhang and Bin Wang and others , title =. 2025 , eprint =

    2025

  62. [74]

    2025 , eprint =

    Jinguo Zhu and Weiyun Wang and Zhe Chen and others , title =. 2025 , eprint =

    2025

  63. [75]

    arXiv preprint arXiv:2502.13923 , year=

    Qwen2.5-VL Technical Report , author=. arXiv preprint arXiv:2502.13923 , year=

    arXiv

  64. [76]

    and Liu, Bo and Faysse, Manuel and Hudelot, Celine and Viaud, Gautier

    Loison, Ant \'o nio and Mac \'e , Quentin and Edy, Antoine and Xing, Victor and Balough, Tom and Moreira, Gabriel de Souza P. and Liu, Bo and Faysse, Manuel and Hudelot, Celine and Viaud, Gautier. V i D o R e V3: A Comprehensive Evaluation of Retrieval Augmented Generation in ...

    2026 doi

  65. [77]

    arXiv preprint arXiv:2505.17166 , year =

    Mac. arXiv preprint arXiv:2505.17166 , year =

  66. [78]

    arXiv preprint arXiv:2601.04720 , year=

    Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking , author=. arXiv preprint arXiv:2601.04720 , year=

    arXiv

  67. [79]

    2025 , publisher=

    Nomic Embed Multimodal: Interleaved Text, Image, and Screenshots for Visual Document Retrieval , author=. 2025 , publisher=

    2025

  68. [80]

    2025 , publisher=

    Eager Embed V1: Multimodal Dense Embeddings for Retrieval , author=. 2025 , publisher=

    2025

  69. [81]

    SauerkrautLM-ColPali: Multi-Vector Vision Retrieval Models , author=

  70. [82]

    European Conference on Computer Vision , year =

    Tongkun Guan and Haocheng Wang and Wei Shen and Xiaokang Yang , title =. European Conference on Computer Vision , year =

  71. [83]

    Proceedings of the 29th ACM international conference on multimedia , pages=

    Beyond ocr+ vqa: Involving ocr into the flow for robust and accurate textvqa , author=. Proceedings of the 29th ACM international conference on multimedia , pages=

  72. [84]

    Pattern Recognition , volume=

    Beyond OCR+ VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa , author=. Pattern Recognition , volume=. 2023 , publisher=

    2023

  73. [85]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Focus, distinguish, and prompt: Unleashing clip for efficient and flexible scene text retrieval , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=

  74. [86]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Clip is almost all you need: towards parameter-efficient scene text retrieval without OCR , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.