Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker

Abhijeet Kumar; Rachna Saxena; Suresh Shanmugam

REVIEW 4 major objections 6 minor 16 references

Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage pipeline—OpenSearch ANN over flattened patch embeddings, then ColPali re-ranking—matches in-memory ColPali recall@1 on 8 of 10 ViDoRe datasets and cuts late-interaction pages from 70–1000 to 13–53.

desk verdict A useful engineering recipe for scalable ColPali-style retrieval, but parity and speedup claims rest on page counts rather than measured latency or candidate-set recall. read the letter →

arxiv 2507.12378 v1 pith:36CBEUS3 submitted 2025-07-16 cs.IR cs.CL

Rachna Saxena , Abhijeet Kumar , Suresh Shanmugam This is my paper · ORCID

classification cs.IRcs.CL

keywords RetrievalaugmentedgenerationVisionlanguagemodelColPaliLateinteractionApproximatenearestneighborOpenSearchVisualdocumentquestionansweringRe-ranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that high-quality vision document retrieval can be run on ordinary vector databases instead of GPU-heavy in-memory late interaction. It flattens ColPali page embeddings into per-patch vectors, indexes them in OpenSearch with HNSW approximate nearest neighbor, retrieves candidate pages per query token using a 0.90 similarity cutoff, and then re-ranks only those pages with full ColPali late interaction. On the ViDoRe benchmark the pipeline matches the recall@1 of in-memory ColPali 1.2 on 8 of 10 datasets and stays within 1% on the other two, while the number of pages sent to late interaction drops from 70–1000 to 13–53. An end-to-end implementation with Sonnet 3.5 as reader reaches 91.8% mean accuracy on a proprietary ESG survey dataset. This matters because enterprises can adopt visual Q&A over tables, charts, and images without custom multi-vector indexes or full-corpus GPU memory.

What carries the argument

The load-bearing mechanism is flatten-and-threshold patch retrieval: each page's (1030, 128) ColPali embedding is stored as 1030 separate 128-d vectors in OpenSearch with document, page, and patch metadata; a query is matched token-by-token against this index with cosine similarity, and any page with a top-scoring patch or a patch scoring ≥ 0.90 is shortlisted. Those shortlist pages are reconstructed in patch-number order into full page embeddings, then scored by ColPali's late-interaction re-ranker. The HNSW index makes the first pass log(n), and the 0.90 threshold plus top-patch rule is what shrinks the re-ranker's input from thousands of pages to dozens.

What would settle it

Run the OpenSearch first pass alone (without ColPali re-ranking) on a corpus where no query token scores ≥ 0.90 against the ground-truth page's patches; any recall@1 miss in that first pass is unrecoverable, so the full-pipeline recall would drop below the reported parity.

Watch

Extended reading notes

Core claim

The paper claims that the standard ColPali late-interaction retrieval—which compares every query token against all ~1030 patch embeddings of every page—can be replaced by a two-pass approximation without sacrificing answer-finding quality. First pass: each query token searches an OpenSearch index of flattened (128-d) patch embeddings, and pages are shortlisted either by being the top match for that token or by having any patch at cosine similarity ≥ 0.90. Second pass: the shortlisted page embeddings are reassembled from their stored patches and scored by ColPali's late interaction to produce the final page ranking. The experiments show recall@1 equal to in-memory ColPali 1.2 on 8 of 10 ViDoRe tasks, and within 1% on the other two, while using only 13–53 pages per query in the re-ranking step instead of the full corpus. Because the first pass uses HNSW ANN and the second pass sees a few dozen pages, the pipeline reduces both space and time by orders of magnitude while keeping the same end-to-end accuracy on the ESG Q&A survey.

Load-bearing premise

The pipeline assumes that a fixed 0.90 cosine-similarity cutoff on flattened patch embeddings, with only the top matching patch considered per query token, reliably finds every page the ColPali re-ranker would need.

Editorial extensions

If this is right

Enterprises can deploy visual document Q&A on commodity vector databases (OpenSearch or similar) instead of proprietary multi-vector stores.
GPU memory for late interaction drops from linear in corpus size (roughly 256 KB per page) to a few dozen pages per query, so million-page corpora no longer require terabyte-scale GPU memory.
The same recall@1 as full in-memory ColPali is obtainable with drastically lower latency, making interactive visual RAG feasible in production.
The end-to-end pipeline with an MLLM reader achieves 91.8% mean accuracy on ESG sustainability-report questions, with the reader instructed to abstain when the answer is absent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 0.90 threshold and top-patch-per-token rule are tuned on ViDoRe; on a corpus with different patch-score distributions, a score-normalization or percentile-based threshold would likely be needed to preserve recall parity.
The flatten-and-rerank trick transfers beyond ColPali: any late-interaction model whose per-token vectors can be flattened should be able to reuse standard ANN indexes for a cheap first pass.
Because the first pass keeps a page only if at least one query token strongly matches one of its patches, queries whose evidence is spread across many weakly matching patches (e.g., multi-hop visual questions) are the most likely failure mode; a benchmark on such questions would test the limit.
Reconstructing page embeddings from stored patches assumes patch order is recoverable from metadata; storing an explicit patch index is therefore a silent correctness requirement of the pipeline.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the paper. Short version: it's a reasonable two-stage recipe — flatten ColPali patch embeddings into OpenSearch, shortlist pages per query token by top patch plus a 0.9 cosine threshold, then re-rank with ColPali late interaction. On ViDoRe it reports recall@1 within rounding of full late interaction on 8 of 10 sets, within 1% on the other two, and cuts the pages sent to the re-ranker from the full corpus to 13-53. That is a real enabler for enterprise RAG if it holds.

The design itself is not in the cited prior work. Vespa has multi-vector HNSW, Qdrant pools to sparse vectors, Byaldi serializes patch embeddings, but the specific OpenSearch flattening with page-number metadata plus threshold shortlisting is a distinct variant. It is also honest: it states clearly that the best case is parity with in-memory ColPali, and the benchmark is external and reproducible.

Where it gets soft. The parity claim has no error bars or significance tests, and one dataset (Shift) goes up by 1.4 points, which smells like noise. More importantly, the paper never reports candidate-set recall — the fraction of queries where the true top-1 page survives the first stage. That is the load-bearing number for any two-stage pipeline, and it is absent. The 0.9 threshold is a free parameter with no sensitivity analysis, and in a million-page corpus the number of noisy patches above 0.9 could inflate the candidate set, so the claimed speedup may not scale the way Table 2 implies. Table 2 only runs at 70-1000 pages. There are also no latency measurements anywhere; the 'significant speed up' is inferred from page counts, not clock time. And while the authors cite Qdrant, Vespa, and Byaldi, they do not benchmark against them, so 'scalable' is only relative to in-memory ColPali.

The ESG evaluation is fine as a demo but weak as evidence: no baseline, single-expert validation, and accuracy percentages without question counts.

Overall: the core idea is sound, the paper is clearly written, and the authors are not overselling — they call it an approximation and list future ablation work. But the current evidence is not enough to take the parity claim at face value. A serious referee would ask for error bars, candidate-set recall, latency numbers, and a threshold sweep. That is revision work, not a desk-reject.

I'd send it to peer review conditional on those additions. For a reading group, it's a useful case study in engineering around ColPali, but I wouldn't block time for it. I probably won't cite it, but if I were building a production system tomorrow I would read it.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes a two-stage retrieval pipeline for visual document Q&A: an OpenSearch HNSW index over flattened ColPali patch embeddings serves as first-pass retriever, and ColPali's late interaction re-ranks the shortlisted pages before an LLM reader answers. Evaluations on the ViDoRe benchmark report recall@1 parity with full in-memory late interaction (within <1% on 8/10 datasets), with the candidate page count reduced from the full corpus size (70-1000 pages) to 13-53 pages. An additional ESG survey dataset yields 91.8% end-to-end answer accuracy. The paper argues the design is scalable and production-ready while reusing standard vector databases.

Significance. If the parity result holds, the approach is a pragmatic contribution: it lets enterprises adopt late-interaction visual retrieval without multi-vector database support and with substantially smaller GPU memory footprint. The paper uses an external benchmark (ViDoRe) and states clearly that the approximation's best case is equality with in-memory ColPali. The main weakness is that the first-stage candidate recall—the load-bearing quantity for the parity claim—is never measured, and the fixed 0.9 similarity threshold and top-patch rule are not subjected to sensitivity analysis. The absence of error bars and latency measurements further limits the strength of the claims.

major comments (4)

[Section 4.2, Table 1] The parity claim is not statistically supported. The reported recall@1 values have no error bars, no significance tests, and no indication of run-to-run variance. The Shift dataset score increases from 60.6 (ColPali 1.2 + Late Inter.) to 62.0 (proposed), which is inconsistent with an approximation that should be a strict subset of full late interaction and suggests measurement noise; this noise could also affect the 'marginally lower' datasets. Please report multiple runs or bootstrap confidence intervals, and state explicitly whether the 8-of-10 'within rounding' claim survives.
[Section 3.2, retriever pseudocode] The candidate-selection rule is underspecified and its recall is unmeasured. The condition 'if top_patch_score or patch_score >= 0.9' is ambiguous about whether the page containing the top-scoring patch is always included and whether every patch above 0.9 contributes a page; the number of top patches returned per query token is not stated. Crucially, the paper never reports the fraction of queries for which the true top-1 page under full late interaction is contained in the candidate set (first-stage recall at k or recall@candidate-set). The parity in Table 1 could hold only because the ViDoRe corpora are small; without candidate-recall measurements, the claim that the 0.90 threshold and top-patch rule generalize is unsupported. Please add a sensitivity analysis over the threshold (e.g., 0.7-0.99) and the number of top patches per token, and report first-stage recall.
[Section 4.2, Table 2 and scalability discussion] The reduction from 70-1000 pages to 13-53 pages does not establish the million-page scalability claimed earlier in the section. A fixed cosine threshold of 0.90 on flattened patch embeddings over a much larger and more heterogeneous corpus may either miss the true top-1 page (if the query's evidence is spread across moderate-scoring patches) or admit many irrelevant pages for common tokens, eroding the speedup. No latency, throughput, or end-to-end memory measurements are reported, so the 'significant speed up' is not quantified. Please report measured inference time and memory as corpus size grows, and discuss how the threshold would be tuned per corpus.
[Section 4.2, Table 3] The ESG end-to-end accuracy of 91.8% is based on a single human validator with no inter-annotator agreement, no error analysis, and no comparison against a non-visual or full-context baseline. As presented, this result is anecdotal and cannot be compared with the ViDoRe retrieval metric. Please clarify the number of questions per client, the annotation protocol, and whether the accuracy is computed per question or per client.

minor comments (6)

[Section 1] The phrase 'flat vison embeddings' should be 'flat vision embeddings'; also, 'HSNW' and 'HNSW' are used inconsistently (e.g., Section 3.1 twice vs Section 4.2 once).
[Section 3.2 pseudocode] The pseudocode contains a typo, 'corrosponding', and the formatting makes the control flow difficult to follow; consider a more standard algorithm layout.
[Table 1] Two baselines are reported, 'ColPali (+Late Inter.)' and 'ColPali 1.2 (+Late Inter.)', but the difference is not explained; clarify whether this is a version change or an evaluation configuration difference.
[Section 4.1] The ESG dataset is described as '150+ questions', but Table 3 reports only accuracy per company with no question counts; state the number of questions per company.
[Section 5 Conclusion] The conclusion mentions 'max-pooled using built-in metadata fields', but Section 3.2 does not perform max-pooling; align the wording or explain the pooling step explicitly.
[General] The paper would benefit from releasing the evaluation code or a public demo; this is not essential, but it would strengthen the reproducibility of the practical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an external benchmark evaluation (ViDoRe) of an OpenSearch+ColPali pipeline, with no claim derived by construction from its own inputs.

full rationale

The paper's central claim is an empirical parity result: a first-pass OpenSearch retriever over flattened ColPali patch embeddings plus a ColPali late-interaction re-ranker achieves roughly the same Recall@1 as full in-memory late interaction on ViDoRe and strong Q&A accuracy on an ESG dataset. This is not a derivation-from-principles claim, and the evaluation is anchored to an external benchmark (ViDoRe, Ref. [13]) and to the independently released ColPali model (Ref. [7]). The proposed system uses ColPali's own embeddings and re-ranker, which is appropriate for measuring the retrieval architecture and is not circular: the first-pass retriever could in principle drop relevant pages, and the paper's reported parity is an empirical outcome, not a mathematical consequence of the definitions. The 0.90 cosine threshold and HNSW settings are implementation parameters, not outputs of a fit to the measured metric; the paper provides no equation that makes Recall@1 equal to these parameters by construction. The ESG accuracy is manually validated by a sustainability expert against externally available reports, again an external check. The paper's statement that 'the best-case performance achieves equal to in-memory ColPali based late interaction retrieval only' is an honest characterization of an approximate system, not a smuggled equivalence. No load-bearing self-citation, uniqueness argument, or ansatz-via-citation pattern is present. The unmeasured first-stage candidate recall and the fixed threshold are legitimate robustness concerns, but they concern correctness risk and scalability, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard ANN search behavior, a domain-specific relevance assumption about cosine similarity for patch embeddings, OpenSearch tooling behavior, and the hand-picked 0.90 threshold. No new physical or mathematical entities are introduced.

free parameters (3)

Similarity threshold for shortlisting pages = 0.90
Used to shortlist pages for re-ranking in Section 3.2; fixed without ablation across ViDoRe datasets.
Top-k patches per query token = Not reported
The retriever fetches 'top matching patches' but k is unspecified; shortlist size depends on it.
HNSW index parameters (M, efSearch, efConstruction) = Not reported
OpenSearch HNSW settings affect the recall/speed tradeoff but are not stated in Section 3.1.

assumptions (4)

standard math HNSW approximate nearest neighbor search returns high-recall top-k results for high-dimensional embeddings.
Relied on in Sections 3.1 and 5, cited [16], without empirical validation on ColPali embeddings.
domain assumption Cosine similarity between a query token embedding and a flattened page-patch embedding is a meaningful first-pass relevance signal.
The retriever in Section 3.2 assumes this to shortlist pages before late interaction.
domain assumption OpenSearch supports filterable metadata on each knn_vector document and returns all patches for a page in one filtered query.
Assumed by the retriever implementation in Section 3.2; no version or configuration details given.
ad hoc to paper The 0.90 similarity threshold is adequate across all datasets.
Introduced in Section 3.2 with no sensitivity analysis.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker." pith.science (2026). https://pith.science/paper/36CBEUS3

@misc{pith2026250712378,
  author       = {Pith},
  title        = {Pith review of: Developing Visual Augmented Q&A System using Scalable Vision Embedding Retrieval & Late Interaction Re-ranker},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36CBEUS3}},
  note         = {Machine review of arXiv:2507.12378}
}

read the original abstract

Traditional information extraction systems face challenges with text only language models as it does not consider infographics (visual elements of information) such as tables, charts, images etc. often used to convey complex information to readers. Multimodal LLM (MLLM) face challenges of finding needle in the haystack problem i.e., either longer context length or substantial number of documents as search space. Late interaction mechanism over visual language models has shown state of the art performance in retrieval-based vision augmented Q&A tasks. There are yet few challenges using it for RAG based multi-modal Q&A. Firstly, many popular and widely adopted vector databases do not support native multi-vector retrieval. Secondly, late interaction requires computation which inflates space footprint and can hinder enterprise adoption. Lastly, the current state of late interaction mechanism does not leverage the approximate neighbor search indexing methods for large speed ups in retrieval process. This paper explores a pragmatic approach to make vision retrieval process scalable and efficient without compromising on performance quality. We propose multi-step custom implementation utilizing widely adopted hybrid search (metadata & embedding) and state of the art late interaction re-ranker to retrieve best matching pages. Finally, MLLM are prompted as reader to generate answers from contextualized best matching pages. Through experiments, we observe that the proposed design is scalable (significant speed up) and stable (without degrading performance quality), hence can be used as production systems at enterprises.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages

[1]

Smith, An Overview of the Tesseract OCR Engine, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), 23-26 September 2007

R. Smith, An Overview of the Tesseract OCR Engine, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), 23-26 September 2007

work page 2007
[2]

Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, LayoutLM: Pre -training of Text and Layout for Document Image Understanding, 20th August 2020, International Conference on Knowledge Discovery & Data Mining

work page 2020
[3]

Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. Multimodal Table Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9102–9124, Bangkok, Thailand

work page 2024
[4]

Jiang, Ziyan and Meng, Rui and Yang, Xinyi and Yavuz, Semih and Zhou, Yingbo and Chen, Wenhu , VLM2Vec: Training Vision -Language Models for Massive Multimodal Embedding Tasks, International Conference on Learning Representations (ICLR), 2025

work page 2025
[5]

Xiaohua Zhai, Basil Mustafa , Alexander Kolesnikov , Lucas Beyer, Sigmoid Loss for Language Image Pre -Training, arxiv: arXiv:2303.15343

arXiv
[6]

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Xiaohua Zhai, PaliGemma: A versatile 3B VLM for transfer , arXiv:2407.07726v2

arXiv
[7]

Colpali: Efficient document retrieval with vision language models

Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. arXiv:2407.01449

arXiv
[8]

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

Omar Khattab, Matei Zaharia . ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. arXiv: 2004.12832

arXiv 2004

Show all 16 references

[9]

Mengzhao Wang, Xiaoliang Xu, Qiang Yue, Yuxiang Wang , A Comprehensive Survey and Experimental Comparison of Graph -Based Approximate Nearest Neighbor Search, arXiv:2101.12631

arXiv
[10]

PyVespa (2025) *https://pyvespa.readthedocs.io/en/latest/ examples/colpali- document-retrieval-vision-language-models-cloud.html*[Online]

2025
[11]

Available: https://github.com/qdrant/qdrant

Qdrant (2025) *Qdrant* [Source code]. Available: https://github.com/qdrant/qdrant

2025
[12]

AnswerDotAI. (2025). *Byaldi* [Source code]. Available: https://github.com/AnswerDotAI/byaldi

2025
[13]

Illuin-Tech. (2025). *Vidore Benchmark* [Source code]. Available: https://github.com/illuin-tech/vidore-benchmark

2025
[14]

Google 2024 Environmental Report - Google Sustainability, July 2024 [online] Available: https://www.gstatic.com/gumdrop/sustainability/google-2024- environmental-report.pdf

2024
[15]

2023 Sustainability Report Journey to Zero , [online] Available: https://www.apple.com/environment/pdf/Apple_Environmental_Progress_Repo rt_2024.pdf

2023
[16]

Yu. A. Malkov, D. A. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, arXiv:1603.09320 ArxivQ DocQ InfoQ TabF TATQ Shift AI Energy Gov. Health. Number of pages in dataset 500 500 500 70 277 1000 1000 1000 1000 ...

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Smith, An Overview of the Tesseract OCR Engine, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), 23-26 September 2007

R. Smith, An Overview of the Tesseract OCR Engine, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), 23-26 September 2007

work page 2007

[2] [2]

Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, LayoutLM: Pre -training of Text and Layout for Document Image Understanding, 20th August 2020, International Conference on Knowledge Discovery & Data Mining

work page 2020

[3] [3]

Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. Multimodal Table Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9102–9124, Bangkok, Thailand

work page 2024

[4] [4]

Jiang, Ziyan and Meng, Rui and Yang, Xinyi and Yavuz, Semih and Zhou, Yingbo and Chen, Wenhu , VLM2Vec: Training Vision -Language Models for Massive Multimodal Embedding Tasks, International Conference on Learning Representations (ICLR), 2025

work page 2025

[5] [5]

Xiaohua Zhai, Basil Mustafa , Alexander Kolesnikov , Lucas Beyer, Sigmoid Loss for Language Image Pre -Training, arxiv: arXiv:2303.15343

arXiv

[6] [6]

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Xiaohua Zhai, PaliGemma: A versatile 3B VLM for transfer , arXiv:2407.07726v2

arXiv

[7] [7]

Colpali: Efficient document retrieval with vision language models

Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. arXiv:2407.01449

arXiv

[8] [8]

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

Omar Khattab, Matei Zaharia . ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. arXiv: 2004.12832

arXiv 2004