Pith. sign in

REVIEW 3 major objections 7 minor 93 references

Jointly encoding document pages cuts retrieval errors by 16 points

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 20:10 UTC pith:OIFQFPNP

load-bearing objection Solid benchmark + method for contextual multimodal retrieval; main concern is train-eval-filter three-way alignment inflating the gap. the 3 major comments →

arxiv 2607.05927 v1 pith:OIFQFPNP submitted 2026-07-07 cs.IR cs.AIcs.CLcs.CV

CMDR: Contextual Multimodal Document Retrieval

Ryota Tanaka , Taku Hasegawa , Kyosuke Nishida This is my paper
classification cs.IR cs.AIcs.CLcs.CV
keywords documentcontextualmultimodalpagesretrievalcmdr-embedembeddingscmdr
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that encoding multiple document pages together, rather than one at a time, produces substantially better page-level retrieval for queries that require cross-page reasoning. The authors introduce CMDR-Bench, a benchmark of 800 human-annotated queries over multi-page documents (averaging 183.5 pages) where the correct page can only be identified by leveraging context from other pages. They propose CMDR-Embed, which uses a chunk-then-split strategy: consecutive pages are encoded jointly by a vision-language model, then the shared representation is separated back into page-level embeddings. A companion training objective, CMCL, balances contextual modeling against page-level discriminability by introducing two types of hard negatives drawn from the same document. The central empirical result is that CMDR-Embed outperforms the same backbone finetuned on the same data by 16.2 nDCG@5 points (56.6 vs. 40.4), isolating the contextual encoding architecture as the source of improvement rather than training data or model scale.

Core claim

The paper's central discovery is that the independent-page encoding assumption used by current multimodal document retrievers is a substantial performance bottleneck for contextual queries. When pages are encoded jointly in overlapping chunks and then split, the resulting page-level embeddings carry cross-page contextual information that enables retrieval of pages whose relevance is not explicitly stated in the query but must be inferred from surrounding pages. The 16.2-point gain over a matched non-contextual baseline, combined with ablations showing that both the contextual encoding and the CMCL contrastive objective contribute independently, establishes that the architecture of encoding,

What carries the argument

CMDR-Embed uses a chunk-then-split encoding pipeline: a sliding window groups consecutive document pages into chunks of size w with stride s, each chunk is processed jointly by a vision-language model, and the resulting token-level representations are separated back into per-page embeddings. Page-level similarity to a query is computed via Late Interaction, which sums the maximum dot product between each query token vector and the page's token vectors. Training uses CMCL loss, a weighted combination of a standard in-batch InfoNCE loss and a context-aware loss that treats other pages from the same chunk (in-chunk negatives) and other pages from the same document but outside the current chunk

Load-bearing premise

The benchmark's 800 human-annotated queries across four author-defined categories are assumed to adequately represent the full space of contextual retrieval needs in real-world documents. The synthetic training data is generated to match exactly these four categories using LLM prompts that mirror the benchmark's task structure, creating a tight train-eval distribution alignment that could inflate the measured gains relative to what would be observed on naturally occurring检索

What would settle it

If a non-contextual retriever, given the same backbone and training data, could match CMDR-Embed's performance by simply increasing the number or diversity of hard negatives during training, the contextual encoding architecture would not be the causal source of the improvement. Alternatively, if the benchmark's query categories were expanded to include contextual retrieval scenarios not anticipated by the four defined types, and CMDR-Embed's advantage disappeared on those new categories, the gains would be specific to the benchmark's task definition rather than evidence for the general value

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Document retrieval systems that currently encode pages independently could adopt joint encoding with modest computational overhead, since the paper reports indexing latency comparable to the backbone model.
  • The benchmark's four query categories (text completion, coreference resolution, structured understanding, multi-hop reasoning) provide a diagnostic framework for identifying specific failure modes in existing retrievers, particularly in coreference and multi-hop scenarios where all models scored lowest.
  • The finding that training data gains saturate around 20k-40k instances while the contextual architecture continues to outperform suggests that architectural design, not data scaling, is the current limiting factor for contextual document retrieval.
  • The multitask learning result showing that contextual and non-contextual retrieval can coexist without degradation suggests future systems can adopt joint encoding as a default without sacrificing standard retrieval performance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper introduces CMDR-Bench, a benchmark for contextual multimodal document retrieval where queries require cross-page reasoning to identify the relevant page, and CMDR-Embed, a model that jointly encodes multiple pages via a sliding window and derives page-level embeddings from a shared contextual representation. The benchmark comprises 800 human-annotated queries across four categories (Text Completion, Coreference Resolution, Structured Understanding, Multi-hop Reasoning) over 255 long documents. CMDR-Embed is trained with a contextual multimodal contrastive learning objective (CMCL) that balances context-aware hard negatives (in-chunk and in-document) with standard in-batch negatives. Experiments show CMDR-Embed outperforming finetuned non-contextual baselines by 13.5–16.2 nDCG@5 points, with ablations isolating the contribution of each component.

Significance. The paper addresses a genuine gap: existing multimodal document retrieval benchmarks do not evaluate indirect, context-dependent retrieval across pages. The chunk-then-split architecture is a natural and well-motivated adaptation of late-chunking ideas to the visual document setting. The experimental design is thorough: ablations (Table 3) isolate CMCL components, the same training data is used for finetuned baselines to control for data effects, efficiency is analyzed (Fig. 7, Table 4), and generalization to non-contextual retrieval is tested (Table 5). The inclusion of Recall@5 results (Table C.3), a reranker baseline (Table C.4), and OCR-quality analysis (Table C.5) strengthens the evaluation. The qualitative examples (Fig. 11) and error analysis (Figs. C.9–C.11) provide useful insight into failure modes.

major comments (3)
  1. §3.2, Data quality control: The benchmark filtering removes queries where three non-contextual retrievers (including ColPali, a Table 2 baseline) achieve average Recall@1 = 1.0, discarding 16% of queries as 'trivial.' This creates selection on the dependent variable: the test set is explicitly constructed to exclude queries solvable by non-contextual models, and those same models are then evaluated on the remaining harder subset. The 16.2-point gap between CMDR-Embed_Qwen and ColQwen+Finetuned is thus measured on a set optimized to disadvantage exactly the baseline it is compared against. The paper should either (a) report results on the full unfiltered set as well, or (b) explicitly acknowledge this as a scope limitation and reframe the magnitude claims accordingly. The current framing ('significantly outperforms') is defensible but the magnitude is difficult to interpret without the un
  2. §5.1 and Supplementary B: The training data CMDR-Synth is generated by LLM prompts that explicitly target the same four query categories used in the benchmark (step 3 prompt includes '{query type}'). This creates a train-eval distribution alignment that could inflate measured gains. While the paper controls for training data by finetuning baselines on CMDR-Synth (which is good practice), the synthetic data generation pipeline mirrors the benchmark's task structure, so both contextual and non-contextual models trained on CMDR-Synth benefit from this alignment. The paper should discuss this alignment explicitly and, if possible, report results on an out-of-distribution contextual retrieval set (e.g., DAPR or ConTEB) to assess transfer.
  3. Table 2, CR column: CMDR-Embed_Qwen achieves only 38.4 nDCG@5 on Coreference Resolution, a mere 3.6-point improvement over ColQwen+Finetuned (34.8), compared to 18.8 and 16.7 on TC and SU respectively. Since CR is one of the four categories that defines the benchmark and motivates the contextual encoding architecture, this weak result on a core category warrants discussion. The paper notes that CR and MR are 'particularly challenging' (§5.2) but does not analyze why the contextual architecture provides minimal benefit on CR specifically. This affects the central claim that joint encoding captures cross-page contextual relationships.
minor comments (7)
  1. §5.2: The claim of 'minimal computational overhead' is supported by Fig. 7, but the text does not state the specific indexing time numbers. Including the key numbers (e.g., 'X ms/page for ColPali+Finetune vs. Y ms/page for CMDR-Embed') in the text would make the claim more concrete.
  2. Table 5: CMDR-Embed_Qwen drops from 88.9 to 85.5 on ViDoRe non-contextual retrieval. The multitask variant recovers to 89.2 but drops on contextual (56.6→56.1). The trade-off should be discussed more explicitly in the text rather than only in the table.
  3. §4.1: The notation for chunk indexing uses s(t-1)+1 but does not clarify whether pages are 0-indexed or 1-indexed. Clarify.
  4. Figure 6: The x-axis labels '>3' for context distance are ambiguous. Consider using explicit bin labels like '1', '2', '3', '4+'.
  5. §3.2: The annotators are described as 'authors, who hold PhDs in computer science.' The number of annotators and inter-annotator agreement statistics are not reported. Adding these would strengthen the quality claim.
  6. Table C.4: Applying the reranker to CMDR-Embed_Pali actually hurts performance (49.8→41.0). The explanation that 'conventional rerankers are not explicitly trained to capture inter-page dependencies' is plausible but speculative; consider softening.
  7. Reference [29] (Qwen3-VL Embedding) is dated 2026. Confirm this is correct and not a typo for 2025.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a thorough and constructive review. The referee correctly identifies three important concerns: (1) selection bias from filtering trivial queries using non-contextual retrievers, (2) train-eval distribution alignment between CMDR-Synth and CMDR-Bench, and (3) weak CR performance relative to other categories. We address each below and commit to revisions for all three.

read point-by-point responses

  1. Referee: §3.2 Data quality control: filtering removes queries where three non-contextual retrievers achieve Recall@1=1.0, creating selection on the dependent variable. The 16.2-point gap is measured on a set optimized to disadvantage baselines. Should report results on full unfiltered set or acknowledge as scope limitation.

    Authors: The referee raises a valid concern. We acknowledge that the filtering process introduces a form of selection bias: by removing queries solvable by non-contextual retrievers, the benchmark is explicitly scoped to contextual retrieval, and the magnitude of improvement over non-contextual baselines is measured on this harder subset. We agree this should be made transparent. In revision, we will: (a) report results on the full unfiltered set (before removal of the 16% trivial queries) so readers can see the complete picture, (b) explicitly reframe the scope of our magnitude claims, noting that the reported gains apply specifically to queries requiring contextual reasoning rather than to all possible retrieval queries over these documents, and (c) clarify in the benchmark description that CMDR-Bench is designed to evaluate contextual retrieval specifically, not general document retrieval. We note that the filtering was conducted with human oversight—annotators reviewed flagged queries and made revision or removal decisions—but the referee's point about interpretability of the magnitude is well taken. revision: yes

  2. Referee: §5.1 and Supplementary B: CMDR-Synth generation prompts explicitly target the same four query categories as the benchmark, creating train-eval distribution alignment that could inflate gains. Should discuss this alignment and report OOD results (e.g., DAPR or ConTEB).

    Authors: The referee correctly identifies that the synthetic training data generation pipeline mirrors the benchmark's task structure, since the query generation prompt (step 3) includes the query type. This is a legitimate concern. We will address it in two ways. First, we will add an explicit discussion of this alignment in the paper, acknowledging it as a potential limitation on the interpretability of absolute gains. Second, regarding OOD evaluation: DAPR and ConTEB are text-only, single-page benchmarks and thus not directly applicable to our multimodal, multi-page setting. However, we can report results on ViDoRe (Table 5 already provides partial evidence of generalization to non-contextual retrieval). For a contextual OOD test, we note that no existing multimodal benchmark evaluates contextual cross-page retrieval in the same formulation as CMDR-Bench, which is precisely the gap our benchmark fills. We will state this limitation honestly and note that constructing an independent contextual multimodal retrieval benchmark is an important direction for future work. We will also emphasize that the controlled comparison (finetuning baselines on the same CMDR-Synth data) isolates the architectural contribution of contextual encoding from the data contribution, even if both benefit from the train-eval alignment. revision: partial

  3. Referee: Table 2, CR column: CMDR-Embed_Qwen achieves only 38.4 nDCG@5 on Coreference Resolution, a 3.6-point improvement over ColQwen+Finetuned (34.8), compared to 18.8 and 16.7 on TC and SU. Weak result on a core category warrants discussion of why contextual architecture provides minimal benefit on CR.

    Authors: The referee is correct that the CR category shows the smallest gain from contextual encoding, and we agree this warrants deeper analysis. We will add a dedicated discussion in Section 5.2. Our analysis of CR errors reveals two main factors. First, coreference resolution often requires resolving references to entities or terms that appear on pages far from the relevant page—sometimes beyond the chunk window (w=4, s=2). When the context page lies outside the jointly encoded chunk, CMDR-Embed cannot leverage it, and performance degrades to approximately the non-contextual baseline level. This is consistent with Figure 6, which shows that performance drops as the context distance increases, particularly for in-chunk negatives. Second, CR queries frequently involve pronouns or abbreviations that are semantically generic (e.g., 'this,' 'the aforementioned'), making the query-page similarity signal weak regardless of contextualization. The contextual architecture helps when the context page provides disambiguating content that shifts the embedding, but when the referent is a generic token, the late-interaction mechanism has limited signal to exploit. We will add this analysis and note that extending the chunk window and incorporating bidirectional attention (as discussed in our Limitations section) are promising directions for improving CR specifically. revision: yes

Circularity Check

0 steps flagged

No derivation-level circularity found; one methodological concern (benchmark filtering with baseline models) is transparently reported and does not reduce the central claim to its inputs by construction.

full rationale

I examined the paper's derivation chain for circularity patterns: self-definitional reductions, fitted inputs called predictions, load-bearing self-citations, uniqueness imports, ansatz smuggling, and renamed known results. The paper is an empirical systems paper, not a theoretical derivation. Its central claim is that CMDR-Embed (joint multi-page encoding) outperforms non-contextual embeddings on CMDR-Bench. The architecture (chunk-then-split with late interaction) and training objective (CMCL, an InfoNCE variant with context-aware hard negatives) are defined independently of the evaluation results — no equation reduces to its inputs by construction. The authors do cite their own prior work ([47] VDocRAG, [48] SlideVQA, [49] VisualMRC), but these are related-work citations, not load-bearing for any derivation. The one legitimate methodological concern is that Section 3.2's benchmark quality control used three non-contextual retrievers (ColPali [17], VLM2Vec [23], NV-Embed [26]) to filter out 'trivial' queries (average Recall@1 = 1.0), and ColPali also appears as a Table 2 baseline. This creates selection bias that could inflate the contextual-vs-non-contextual gap. However, this is an evaluation methodology concern (selection on the dependent variable), not a circularity in a derivation chain: the paper does not fit a parameter to data and then predict the same quantity, nor does it define a quantity in terms of what it claims to derive. The paper is transparent about the filtering, the comparison controls for training data (ColQwen+Finetuned vs. CMDR-Embed_Qwen both trained on CMDR-Synth), and the ablation studies (Table 3) provide independent evidence for each component's contribution. The training data (CMDR-Synth) does mirror the benchmark's four query categories (Supplementary B, step 3 prompt includes '{query type}'), but this is standard task-aligned training data construction, and the same data is given to baseline models. Score 2 reflects the minor methodological concern without rising to construction-level circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new physical entities, particles, or mathematical objects are invented. The method is an architectural composition of existing components (LVLM, late interaction, sliding window, contrastive learning).

free parameters (5)
  • λ (CMCL loss weight) = 0.5
    Selected from {0.1, 0.5, 0.9} on the benchmark; controls balance between context-aware and in-batch contrastive losses (Section 5.1, Fig. 10).
  • τ (temperature) = 0.02
    Scales the late-interaction score in the contrastive loss; chosen by the authors (Section 5.1).
  • s (stride length) = 2
    Sliding window stride; selected from grid search (Fig. 7).
  • w (window size) = 4
    Sliding window size; selected from grid search (Fig. 7).
  • LoRA α, r = 32, 32
    Standard LoRA hyperparameters for fine-tuning (Section 5.1).
axioms (4)
  • domain assumption Cross-page contextual information improves page-level retrieval for queries that require reasoning across multiple pages.
    Core assumption motivating the entire work; supported by the benchmark design and experimental results (Table 2).
  • domain assumption Late interaction (multi-vector matching) preserves fine-grained token-level representations better than single-vector pooling for document retrieval.
    Adopted from ColPali [17]; confirmed by ablation in Table 3 (26.5-point drop without LI).
  • domain assumption Causal attention in LVLMs is sufficient to capture useful contextual information from preceding pages.
    The model uses causal-attention backbones (PaliGemma, Qwen2-VL); the authors acknowledge this limits subsequent-context understanding (Section 5.4, Limitations).
  • domain assumption Synthetic training data generated by Qwen2.5-VL 72B can effectively train contextual retrieval models.
    CMDR-Synth is fully synthetic (Section 5.1); the quality of generated queries depends on the LLM's ability to produce contextual reasoning examples.

pith-pipeline@v1.1.0-glm · 37755 in / 3748 out tokens · 200902 ms · 2026-07-08T20:10:06.837939+00:00 · methodology

0 comments
read the original abstract

Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing benchmarks primarily evaluate simple lexical or semantic matching, and most methods encode pages independently. Consequently, they overlook the contextual information in the document required to resolve queries that aggregate information across multiple pages. In this paper, we introduce CMDR and CMDR-Bench, a new multimodal document retrieval task and benchmark that require modeling document context. To address this challenge, we propose CMDR-Embed, a contextual multimodal embedding framework that explicitly incorporates document context by jointly encoding multiple pages and deriving page-level embeddings from a shared contextual representation. Furthermore, we introduce CMCL, a contextual multimodal contrastive learning objective that effectively trains CMDR-Embed by balancing contextual modeling with page-level discriminability. Experiments demonstrate that CMDR-Embed significantly outperforms non-contextual embeddings, highlighting the importance of context-aware multimodal embeddings for advancing document retrieval.

Figures

Figures reproduced from arXiv: 2607.05927 by Kyosuke Nishida, Ryota Tanaka, Taku Hasegawa.

Figure 1
Figure 1. Figure 1: Comparison benchmarks and methods. Unlike previous benchmarks that rely on direct keyword or simple semantic matching, CMDR-Bench focuses on indirectly retrieving relevant pages by leveraging document context. In contrast to non-contextual embeddings that encode each page independently, CMDR-Embed jointly encodes multiple pages to capture cross-page contextual relationships. cross-page dependencies and doc… view at source ↗
Figure 4
Figure 4. Figure 4: Dataset distribution [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Our model and training loss. (a) CMDR-Embed encodes multiple pages jointly and then splits them to capture contextual information. (b) CMCL loss optimizes context-aware embeddings while preserving discriminability within the same document. retrieval. While DAPR [53] and ConTEB [11] involve contextual reasoning, they are limited to single-page and text-only retrieval settings. Additionally, existing multimo… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of our model variants using the CMCL loss across different context lengths. This demonstrates that CMDR-Bench requires methods to jointly understand the text, layout, and visual modalities of documents. Even in the text-related cate￾gory (TC), text retrievers underperform vision-based retrievers. This is because many pages contain visual elements, such as charts, tables, and figures, that OCR ca… view at source ↗
Figure 7
Figure 7. Figure 7: Efficiency analysis by varying (stride s, window w). Manuals Research reports & papers Theses & dissertations Project proposal Books & e-Books Meeting minutes & summaries ColQwen + Finetune ColPali + Finetune Qwen 80 60 80 60 Manuals Research reports & papers Theses & dissertations Project proposal Books & e-Books Meeting minutes & summaries CMDR-EmbedPali CMDR-Embed Non-Contextual Contextual Non-Contextua… view at source ↗
Figure 10
Figure 10. Figure 10: Analysis of CMCL loss weight by varying different λ [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qualitative results of CMDR-Embed compared to the non-contextual model. retrieval capability, we finetune CMDR-Embed with a multitask learning strategy on both contextual and non-contextual retrieval tasks, using CMDR-Synth and ViDoRe [17] datasets. This strategy enables the model to jointly learn document contextual reasoning and page-level relevance matching [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 93 canonical work pages · 18 internal anchors

  1. [1]

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

    Abdin, M., Jacobs, S.A., Awan, A.A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al.: Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404.14219 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    GPT-4 Technical Report

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: GPT-4 technical report. arXiv:2303.08774 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    In: NeurIPS

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. In: NeurIPS. pp. 23716–23736 (2022)

    work page 2022

  4. [4]

    Qwen2.5-VL Technical Report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2.5-VL technical report. arXiv:2502.13923 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Computer networks and ISDN systems30(1-7), 107–117 (1998)

    Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Computer networks and ISDN systems30(1-7), 107–117 (1998)

    work page 1998

  6. [6]

    VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding

    Chen, J., Li, M., Kil, J., Wang, C., Yu, T., Rossi, R., Zhou, T., Chen, C., Zhang, R.: Visr-bench: An empirical study on visual retrieval-augmented generation for multilingual long document understanding. arXiv:2508.07493 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

    Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., Liu, Z.: BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation. arXiv:2402.03216 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    In: CVPR

    Chen, J., Xu, D., Fei, J., Feng, C.M., Elhoseiny, M.: Document haystacks: Vision- language reasoning over piles of 1000+ documents. In: CVPR. pp. 24817–24826 (2025)

    work page 2025

  9. [9]

    In: EMNLP

    Chen,X.,Zhao,Z.,Chen,L.,Ji,J.,Zhang,D.,Luo,A.,Xiong,Y.,Yu,K.:WebSRC: A dataset for web-based structural reading comprehension. In: EMNLP. pp. 4173– 4185 (2021)

    work page 2021

  10. [10]

    Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling

    Clavié, B., Chaffin, A., Adams, G.: Reducing the footprint of multi-vector retrieval with minimal performance impact via token pooling. arXiv:2409.14683 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    In: EMNLP

    Conti, M., Faysse, M., Viaud, G., Bosselut, A., Hudelot, C., Colombo, P.: Context is gold to find the gold passage: Evaluating and training contextual document embeddings. In: EMNLP. pp. 22605–22619 (2025)

    work page 2025

  12. [12]

    In: NeurIPS

    Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: FlashAttention: Fast and memory- efficient exact attention with io-awareness. In: NeurIPS. pp. 16344–16359 (2022)

    work page 2022

  13. [13]

    Deng, C., Yuan, J., Bu, P., Wang, P., Li, Z.Z., Xu, J., Li, X.H., Gao, Y., Song, J., Zheng, B., et al.: LongDocURL: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. In: ACL. pp. 1135– 1159 (2025)

    work page 2025

  14. [14]

    In: NAACL-HLT

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirec- tional transformers for language understanding. In: NAACL-HLT. pp. 4171–4186 (2019)

    work page 2019

  15. [15]

    In: EMNLP

    Dong, K., Chang, Y., Deik, D.G.X., Li, D., Tang, R., Liu, Y.: MMDocIR: Bench- marking multimodal retrieval for long documents. In: EMNLP. pp. 30959–30993 (2025)

    work page 2025

  16. [16]

    arXiv:2505.16470 (2025)

    Dong, K., Chang, Y., Huang, S., Wang, Y., Tang, R., Liu, Y.: Benchmark- ing retrieval-augmented multimodal generation for document question answering. arXiv:2505.16470 (2025)

    work page arXiv 2025

  17. [17]

    In: ICLR (2025) CMDR: Contextual Multimodal Document Retrieval 17

    Faysse, M., Sibille, H., Wu, T., Viaud, G., Hudelot, C., Colombo, P.: ColPali: Efficient document retrieval with vision language models. In: ICLR (2025) CMDR: Contextual Multimodal Document Retrieval 17

    work page 2025

  18. [18]

    Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

    Günther, M., Mohr, I., Williams, D.J., Wang, B., Xiao, H.: Late Chunking: con- textual chunk embeddings using long-context embedding models. arXiv preprint arXiv:2409.04701 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    In: ICML

    Guu, K., Lee, K., Tung, Z., Pasupat, P., Chang, M.: Retrieval augmented language model pre-training. In: ICML. pp. 3929–3938 (2020)

    work page 2020

  20. [20]

    In: ICLR (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022)

    work page 2022

  21. [21]

    Unsupervised Dense Information Retrieval with Contrastive Learning

    Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learning. arXiv:2112.09118 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  22. [22]

    E5-V: Universal Embeddings with Multimodal Large Language Models

    Jiang, T., Song, M., Zhang, Z., Huang, H., Deng, W., Sun, F., Zhang, Q., Wang, D., Zhuang, F.: E5-V: Universal embeddings with multimodal large language models. arXiv:2407.12580 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    In: ICLR (2025)

    Jiang, Z., Meng, R., Yang, X., Yavuz, S., Zhou, Y., Chen, W.: VLM2Vec: Training vision-language models for massive multimodal embedding tasks. In: ICLR (2025)

    work page 2025

  24. [24]

    Kamalloo, E., Thakur, N., Lassance, C., Ma, X., Yang, J.H., Lin, J.: Resources for brewing beir: Reproducible reference models and an official leaderboard (2023)

    work page 2023

  25. [25]

    In: SIGIR

    Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via con- textualized late interaction over bert. In: SIGIR. pp. 39–48 (2020)

    work page 2020

  26. [26]

    In: ICLR

    Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catanzaro, B., Ping, W.: Nv- embed: Improved techniques for training llms as generalist embedding models. In: ICLR. pp. 79310–79333 (2025)

    work page 2025

  27. [27]

    In: NIPS

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: NIPS. pp. 9459–9474 (2020)

    work page 2020

  28. [28]

    In: ICML

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: ICML. pp. 12888– 12900 (2022)

    work page 2022

  29. [29]

    Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking

    Li, M., Zhang, Y., Long, D., Chen, K., Song, S., Bai, S., Yang, Z., Xie, P., Yang, A., Liu, D., et al.: Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv:2601.04720 (2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  30. [30]

    Focus Anywhere for Fine-grained Multi-page Document Understanding

    Liu, C., Wei, H., Chen, J., Kong, L., Ge, Z., Zhu, Z., Zhao, L., Sun, J., Han, C., Zhang, X.: Focus anywhere for fine-grained multi-page document understanding. arXiv:2405.14295 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Foundations and Trends® in Information Retrieval3(3), 225–331 (2009)

    Liu, T.Y.: Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval3(3), 225–331 (2009)

    work page 2009

  32. [32]

    Liu, Z., Liu, Z., Liang, Z., Zhou, J., Xiao, S., Gao, C., Zhang, C.J., Lian, D.: Any information is just worth one single screenshot: Unifying search with visualized information retrieval. In: ACL. pp. 19238–19261 (2025)

    work page 2025

  33. [33]

    Decoupled Weight Decay Regularization

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv:1711.05101 (2017)

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  34. [34]

    BM25S: Orders of magnitude faster lexical search via eager sparse scoring

    Lù,X.H.:Bm25s: Ordersof magnitudefasterlexical search viaeager sparsescoring. preprint arXiv:2407.03618 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Unifying Multimodal Retrieval via Document Screenshot Embedding

    Ma, X., Lin, S.C., Li, M., Chen, W., Lin, J.: Unifying multimodal retrieval via document screenshot embedding. arXiv:2406.11251 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Ma, X., Zhuang, S., Koopman, B., Zuccon, G., Chen, W., Lin, J.: Visa: Retrieval augmented generation with visual source attribution. In: ACL. pp. 30154–30169 (2025)

    work page 2025

  37. [37]

    Tanaka et al

    Ma, Y., Zang, Y., Chen, L., Chen, M., Jiao, Y., Li, X., Lu, X., Liu, Z., Ma, Y., Dong, X., et al.: MMLongBench-Doc: Benchmarking long-context document 18 R. Tanaka et al. understanding with visualizations. NeurIPS Datasets and Benchmarks Track pp. 95963–96010 (2024)

    work page 2024

  38. [38]

    arXiv:2505.17166 (2025)

    Macé, Q., Loison, A., Faysse, M.: ViDoRe Benchmark V2: Raising the bar for visual retrieval. arXiv:2505.17166 (2025)

    work page arXiv 2025

  39. [39]

    Manualslib: manualslib.com.https://www.manualslib.com/(2024), accessed on April-May, 2025

    work page 2024

  40. [40]

    In: WACV

    Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., Jawahar, C.: Info- graphicVQA. In: WACV. pp. 1697–1706 (2022)

    work page 2022

  41. [41]

    In: WACV

    Mathew, M., Karatzas, D., Jawahar, C.V.: DocVQA: A dataset for vqa on docu- ment images. In: WACV. pp. 2200–2209 (2021)

    work page 2021

  42. [42]

    Representation Learning with Contrastive Predictive Coding

    Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predic- tive coding. arXiv:1807.03748 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  43. [43]

    co / datasets / pixparse/pdfa-eng-wds(2024)

    Pablo Montalvo, R.W.: pdfa-eng-wds.https : / / huggingface . co / datasets / pixparse/pdfa-eng-wds(2024)

    work page 2024

  44. [44]

    In: ICML

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)

    work page 2021

  45. [45]

    Foundations and Trends®in Information Retrieval3(4), 333–389 (2009)

    Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: BM25 and beyond. Foundations and Trends®in Information Retrieval3(4), 333–389 (2009)

    work page 2009

  46. [46]

    In: ICDAR

    Smith, R.: An overview of the tesseract ocr engine. In: ICDAR. pp. 629–633 (2007)

    work page 2007

  47. [47]

    In: CVPR

    Tanaka, R., Iki, T., Hasegawa, T., Nishida, K., Saito, K., Suzuki, J.: VDocRAG: Retrieval-augmented generation over visually-rich documents. In: CVPR. pp. 24827–24837 (2025)

    work page 2025

  48. [48]

    In: AAAI

    Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T., Saito, I., Saito, K.: SlideVQA: A dataset for document visual question answering on multiple images. In: AAAI. pp. 13636–13645 (2023)

    work page 2023

  49. [49]

    In: AAAI

    Tanaka, R., Nishida, K., Yoshida, S.: VisualMRC: Machine reading comprehension on document images. In: AAAI. pp. 13878–13888 (2021)

    work page 2021

  50. [50]

    arXiv:2510.01149 (2025)

    Teiletche, P., Macé, Q., Conti, M., Loison, A., Viaud, G., Colombo, P., Faysse, M.: Modernvbert: Towards smaller visual document retrievers. arXiv:2510.01149 (2025)

    work page arXiv 2025

  51. [51]

    Pattern Recognition144, 109834 (2023)

    Tito, R., Karatzas, D., Valveny, E.: Hierarchical multimodal transformers for mul- tipage docvqa. Pattern Recognition144, 109834 (2023)

    work page 2023

  52. [52]

    arXiv:2511.20227 (2025)

    Tong, A., Niu, X., Liu, Z., Tian, C., Wei, Y., Shi, Z., Wang, M.: HKRAG: Holistic knowledge retrieval-augmented generation over visually-rich documents. arXiv:2511.20227 (2025)

    work page arXiv 2025

  53. [53]

    Wang, K., Reimers, N., Gurevych, I.: DAPR: A benchmark on document-aware passage retrieval. In: ACL. pp. 4313–4330 (2024)

    work page 2024

  54. [54]

    Text Embeddings by Weakly-Supervised Contrastive Pre-training

    Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F.: Text embeddings by weakly-supervised contrastive pre-training. arXiv:2212.03533 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  55. [55]

    NeurIPS pp

    Wang, Q., Ding, R., Zeng, Y., Chen, Z., Chen, L., Wang, S., Xie, P., Huang, F., Zhao, F.: Vrag-rl: Empower vision-perception-based rag for visually rich informa- tion understanding via iterative reasoning with reinforcement learning. NeurIPS pp. 57133–57160 (2026)

    work page 2026

  56. [56]

    In: CIKM

    Webber, W., Moffat, A., Zobel, J.: Statistical power in retrieval experimentation. In: CIKM. pp. 571–580 (2008)

    work page 2008

  57. [57]

    General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model

    Wei, H., Liu, C., Chen, J., Wang, J., Kong, L., Xu, Y., Ge, Z., Zhao, L., Sun, J., Peng, Y., et al.: General ocr theory: Towards ocr-2.0 via a unified end-to-end model. arXiv:2409.01704 (2024) CMDR: Contextual Multimodal Document Retrieval 19

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  58. [58]

    In: NAACL

    Weller, O., Chang, B., MacAvaney, S., Lo, K., Cohan, A., Van Durme, B., Lawrie, D., Soldaini, L.: Followir: Evaluating and teaching information retrieval models to follow instructions. In: NAACL. pp. 11926–11942 (2025)

    work page 2025

  59. [59]

    In: SIGIR

    Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., Nie, J.Y.: C-pack: Packed resources for general chinese embeddings. In: SIGIR. pp. 641–649 (2024)

    work page 2024

  60. [60]

    In: CVPR

    Yan, H., Liu, Y., Liu, X., Zhang, Y., Liao, M., Wu, J., Chen, W., Bai, X.: Doc- seeker: Structured visual reasoning with evidence grounding for long document understanding. In: CVPR. pp. 41140–41149 (2026)

    work page 2026

  61. [61]

    In: ICLR (2025)

    Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z., Wang, S., Han, X., Liu, Z., et al.: VisRAG: Vision-based retrieval-augmented generation on multi-modality documents. In: ICLR (2025)

    work page 2025

  62. [62]

    In: ICCV

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: ICCV. pp. 11975–11986 (2023)

    work page 2023

  63. [63]

    In: ICCV

    Zhang, J., Zhang, Q., Wang, B., Ouyang, L., Wen, Z., Li, Y., Chow, K.H., He, C., Zhang, W.: OCR Hinders RAG: Evaluating the cascading impact of ocr on retrieval-augmented generation. In: ICCV. pp. 17443–17453 (2025)

    work page 2025

  64. [64]

    In: CVPR

    Zhang, X., Zhang, Y., Xie, W., Li, M., Dai, Z., Long, D., Xie, P., Zhang, M., Li, W., Zhang, M.: Bridging modalities: Improving universal multimodal retrieval by multimodal large language models. In: CVPR. pp. 9274–9285 (2025)

    work page 2025

  65. [65]

    Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook

    Zheng, X., Weng, Z., Lyu, Y., Jiang, L., Xue, H., Ren, B., Paudel, D., Sebe, N., Van Gool, L., Hu, X.: Retrieval augmented generation and understanding in vision: A survey and new outlook. arXiv:2503.18016 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  66. [66]

    No” (i.e., “Not self-contained

    Zhu, F., Lei, W., Feng, F., Wang, C., Zhang, H., Chua, T.S.: Towards complex document understanding by discrete reasoning. In: ACMM. pp. 4857–4866 (2022) 20 R. Tanaka et al. CMDR: Contextual Multimodal Document Retrieval Supplementary Material ACMDR-BenchDetails Table A.1: Main statisticsof docu- ments and queries inCMDR-Bench. Statistics Number Total Doc...

    work page 2022

  67. [67]

    - At first glance, it should seem that the answer could be in Image 1

    The query should appear to be about Image 1, but the correct answer is found only in Image 2. - At first glance, it should seem that the answer could be in Image 1. - Only by reading both pages carefully should it become clear that the answer is actually in Image 2

    work page

  68. [68]

    The query must require contextual understanding across both pages, not a simple keyword search

    work page

  69. [69]

    Query Type: {query type}

    work page

  70. [70]

    - Do not include explicit hints or keywords that would guide the reader to look into Image 2

    Do not quote or copy any text from Image 2 directly into the query. - Do not include explicit hints or keywords that would guide the reader to look into Image 2

    work page

  71. [71]

    Only output the query in the specified format without any additional explanation

    work page

  72. [72]

    Use only one question word and keep the query in a single sentence. Output Format: Query: <Your query here> (4) Filtering low-quality queries.Given the prompt below, the context page image (Image 1), and the relevant page image (Image 2), we feed them into Qwen2.5-VL 72B to identify and filter out low-quality or rule-violating queries. CMDR: Contextual Mu...

    work page

  73. [73]

    - It should be plausible that a reader might expect the answer to be found in Image 1 at first glance

    Apparent Focus on Image 1 - The query should initially appear to ask about Image 1. - It should be plausible that a reader might expect the answer to be found in Image 1 at first glance

    work page

  74. [74]

    - Image 1 alone must be insufficient to answer the query correctly

    Answer Actually Located in Image 2 - The correct answer must be found only in Image 2. - Image 1 alone must be insufficient to answer the query correctly

    work page

  75. [75]

    - The answer should NOT be obtainable via simple keyword matching or surface-level lookup

    Cross-Page Contextual Reasoning Required - The query must require integrating contextual information across both images. - The answer should NOT be obtainable via simple keyword matching or surface-level lookup

    work page

  76. [76]

    - The query must NOT contain explicit hints or keywords that clearly direct the reader to Image 2

    No Leakage from Image 2 - The query must NOT quote or directly copy text from Image 2. - The query must NOT contain explicit hints or keywords that clearly direct the reader to Image 2

    work page

  77. [77]

    - Be written as a single sentence

    Single Question Word & Single Sentence - The query must: - Contain exactly one question word (What, Why, When, Where, Who, How). - Be written as a single sentence

    work page

  78. [78]

    - If it does not match the intended reasoning type, mark it as invalid

    Query Type Consistency - The query must match the specified Query Type: {query_type}. - If it does not match the intended reasoning type, mark it as invalid. Output Format: Valid: Yes or No Reasoning: - Criterion 1: Pass or Fail – brief explanation - Criterion 2: Pass or Fail – brief explanation - Criterion 3: Pass or Fail – brief explanation - Criterion ...

    work page

  79. [79]

    R. E. Fikes, N. J. Nilsson. STRIPS: a new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3-4):189-208

    work page

  80. [80]

    James Firby

    R. James Firby. An investigation into reactive planning in complex domains. Proceedings of the 6th National Conference on AI, Seattle, WA, July 1987, 1987

    work page 1987

Showing first 80 references.