{"id":"9e3679aa-a5d6-430f-8568-f7597cbd38d3","arxiv_id":"2507.05513","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new family of text-image retrieval models, built from Eagle2 with bidirectional attention and ColBERT-style late interaction, reports state-of-the-art NDCG@5 scores on ViDoRe V1 (91.0) and V2 (63.5).","lead":"NVIDIA released two retrieval models, CoEmbed 1B and 3B, that search documents by reading them as images instead of extracted text. The larger model reports top scores on the ViDoRe visual document retrieval benchmarks, which is useful for companies building search or question-answering over PDFs, slides, and reports.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is plausible, but the paper never checks whether ViDoRe V1/V2 queries or corpus documents overlap with its Stage-2 training data; that gap is load-bearing because several evaluated dataset families also appear in the training mixture.","rationale":"I read the paper in good faith and the headline numbers are internally consistent across Tables 3, 4, and the legacy Appendix A results, and the weights are released on HuggingFace, which is real independent support for reproducibility. The reader's weakest_assumption identified both contamination and the proxy ranking metric. I focus on contamination because it attacks the numeric scores directly; the Borda-vs-Avg-NDCG issue affects the framing as 'leading MTEB VDR' but not the reported ViDoRe scores themselves. The absence of any overlap analysis is the most load-bearing gap: the training mixture in Section 3.3 and the evaluation benchmarks plausibly share source dataset families, so the stated 1.1- and 2.8-point margins over baselines could in principle be affected by even modest document-level leakage. The paper's unsupported two-stage transfer claim and missing ablations for max_input_tiles are relevant to the explanation of why the model works, but they are not load-bearing for the SOTA claim. The recommended verdict therefore remains CONDITIONAL, unchanged from the reader: the reported scores may well be correct, but accepting the 'top-performing' claim should require the contamination control described above and, ideally, an independent run of the official ViDoRe/MTEB evaluation code on the released checkpoint.","tokens_in":12545,"tokens_out":7025,"duration_ms":81324,"concrete_test":"Compute exact and near-duplicate overlap (via token n-gram hashing and image perceptual hashing) between every query and positive document in ViDoRe V1/V2 and the Stage-2 training sets from Section 3.3 (ColPali train set, Wiki-SS-NQ, VDR, VisRAG-Ret-Train-Synthetic, VisRAG-Ret-Train-In-domain, Docmatix). Recompute NDCG@5 for the released 3B checkpoint on the subset of queries whose positive document has no overlapping corpus page in training, and recompute the same subset scores for the strongest baselines. If the margin against the best baseline narrows by 0.5 NDCG@5 or more, or the relative ranking changes, the SOTA claim requires a contamination-controlled qualification; if the margin is unchanged, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the 3B model scores 91.0 on ViDoRe V1 and 63.5 on ViDoRe V2 and ranks first on both benchmarks. For this claim to hold, evaluation scores must not be inflated by training/evaluation overlap. Section 3.3 lists Stage-2 training data including the ColPali train set, VDR, VisRAG-Ret-Train-Synthetic, VisRAG-Ret-Train-In-domain, and Docmatix. ViDoRe V1 and V2 are built from dataset families such as ArxivQA, DocVQA, InfoVQA, TabFQuad, TAT-DQA, and the V2 economics/ESG collections, which overlap in source with the ColPali train set and the VDR/VisRAG mixtures. The paper provides no split analysis: no exact or near-duplicate check between training queries/documents and ViDoRe test queries/corpora, and no measurement of how many evaluated queries have positive documents that appeared in training. If the same document pages were seen during contrastive training, the MaxSim late-interaction scores for those pages could be memorization-facilitated, inflating NDCG@5. The margin over the best baseline is 1.1 points on V1 and 2.8 points on V2, so even a small contamination effect could change the ranking. This is an addressable gap, not evidence of inflation; a clean-split or post-hoc exclusion analysis would settle it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces llama-nemoretriever-colembed, a family of text-image retrieval models (1B and 3B parameter variants) built on NVIDIA's Eagle2 vision-language model by replacing causal attention with bidirectional attention and adding a ColBERT-style late-interaction scoring. The models are trained in two stages: first text-only contrastive retrieval, then text-image contrastive fine-tuning with hard negative mining. The central empirical claim is that the 3B model achieves NDCG@5 of 91.0 on ViDoRe V1 and 63.5 on ViDoRe V2, ranking first on both leaderboards as of June 27, 2025, and also performs strongly on the MTEB VDR leaderboard and the MIRACL-VISION benchmark. The paper also provides storage and latency trade-off analyses comparing late-interaction and bi-encoder architectures.","tokens_in":12867,"tokens_out":6953,"duration_ms":68023,"significance":"The model release is practically valuable: it provides strong multimodal retrieval performance with competitive parameter efficiency, and the two-stage training recipe with hard negatives is a useful recipe. The storage/latency analysis is a helpful engineering contribution. However, the headline claims are benchmark numbers that hinge on evaluation protocol comparability and on the absence of train/test overlap; neither is fully established, and one of the benchmark claims (MTEB VDR) uses a substituted metric. If the contamination gap is closed and the metric issue clarified, this would be a solid systems contribution.","major_comments":[{"comment":"The paper does not analyze overlap between the Stage-2 training mixture (ColPali train set, VDR, VisRAG-Ret-Train-In-domain, Docmatix, etc.) and the ViDoRe V1/V2 evaluation sets. Many ViDoRe dataset families (ArxivQA, DocVQA, InfoVQA, TabFQuad, TAT-DQA, and the V2 economics/ESG collections) share source corpora with these training datasets. Because the reported margins over the best baselines are only 1.1 NDCG@5 (V1) and 2.8 NDCG@5 (V2), even a small number of memorized document pages or queries could affect the ranking. Please provide exact and near-duplicate contamination checks at the query and document level, or a clean-split re-evaluation.","section":"§3.3, §4"},{"comment":"The paper substitutes average NDCG@5 for the official MTEB Borda Count, stating that Rank Borda scores are not visible on the MTEB leaderboard, and then claims a 'leading MTEB VDR score of 83.1.' This is not the official MTEB VDR ranking metric, and the phrase could mislead readers into thinking 83.1 is the official score. Please report the official Borda Count values for the models and baselines if available, or explicitly label 83.1 as an unofficial average and avoid calling it the 'MTEB VDR score.' This point is load-bearing because the 'state-of-the-art on MTEB VDR' phrasing appears in the conclusion.","section":"§4, Table 2"},{"comment":"MIRACL-VISION is a benchmark introduced by the same authors (reference [30]), and the paper's claim of 'strong multilingual retrieval capabilities' rests entirely on this self-designed evaluation. The table mixes text-only baselines evaluated on a 'text' subset with visual retrieval models on an 'image' subset, and the set of compared image models is small. Please clarify whether these numbers come from the official MIRACL-VISION evaluation harness, report results for all models with the same protocol, and temper the SOTA claim on this benchmark accordingly.","section":"§4, Table 5"},{"comment":"The headline numbers for the 3B model are inconsistent across tables: Table 3 reports ViDoRe V1 91.0 and Table 4 reports V2 63.5, while Table 6 reports 0.9106 and 0.6357 for the same model and notes that a different code base was used. Because the paper itself warns that results 'can slightly change based on the code base,' the exact evaluation code version, commit hash, tiling parameters, and precision must be specified for the numbers in Tables 2–4, and Table 6 should clearly state which protocol it uses to avoid ambiguity about which numbers constitute the SOTA claim.","section":"§4, §5.2, Table 6"}],"minor_comments":[{"comment":"The Table 5 header is corrupted in the manuscript; the column labels are unreadable, making it impossible to verify which models correspond to which scores.","section":"§4, Table 5"},{"comment":"Section 5.2 and Table 6 refer to 'lama-3_2-nemoretriever-1b-vlm-embed-v1' (missing the first 'l' in 'llama'); please fix this typo.","section":"§5.2, Table 6"},{"comment":"Reference [30] lists 'Radek Osmulsk' as an author; this appears to be a typo for 'Radek Osmulski.'","section":"References"},{"comment":"No error bars or significance tests are reported for any of the NDCG@5 values; given the small margins, a note on variance across query samples would increase confidence.","section":"§4"},{"comment":"Section 2.1 states that increasing max_input_tiles from 2 to 4 'showed no performance gains' without providing the supporting experiment; please add the numbers or remove the claim.","section":"§2.1"},{"comment":"The paper does not describe how the pretrained causal-attention weights are initialized or adapted when changing to bidirectional attention; a brief description would improve reproducibility.","section":"§3.2"},{"comment":"The conclusion claims state-of-the-art results on MIRACL-VISION, but Table 5 includes only a small set of visual retrieval baselines; please either broaden the comparison or qualify the claim.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a corporate technical report from NVIDIA, and one of the supporting benchmarks (MIRACL-VISION) is authored by the same group, which raises a self-evaluation concern. The MTEB Borda Count substitution is also a potential source of overstatement. The contamination gap is the most serious issue and should be closed before publication. The paper seems better suited to a systems/benchmark venue than a core-architectural venue, given that it is primarily an empirical model release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a workmanlike model report with genuinely strong benchmark numbers, but the headline “state of the art” should be tagged conditional until the training/evaluation overlap question is answered. The models are released, so the scores are checkable, and the efficiency analysis is worth having.\n\nWhat is actually new: the specific recipe—Eagle2 backbone, bidirectional attention, ColBERT-style late interaction, two-stage text-then-image contrastive training—produces a new SOTA on ViDoRe V1/V2 and looks strong on MIRACL-VISION. The base model is the main novelty; most architectural pieces are borrowed from ColPali and NV-Retriever. The paper does a good job on the engineering side: it reports per-dataset scores, gives legacy numbers, candidly notes that results move with the evaluation codebase, and provides a useful storage/latency comparison including a bi-encoder-plus-reranker alternative.\n\nSoft spots, in order of significance. First, no contamination analysis. Stage-2 training includes the ColPali train set, VisRAG, and VDR, all of which come from the same document families as ViDoRe (ArxivQA, DocVQA, InfoVQA, TAT-DQA). The margins over the best baseline are 1.1 points on V1 and 2.8 on V2, so even modest overlap could move the ranking. The paper never checks near-duplicates between training pairs and test queries/corpora. That is a load-bearing omission, though likely fixable. Second, the abstract's 91.0/63.5 do not match Table 6's 0.9106/0.6357, and the paper says that is due to a different codebase; the leaderboard claim therefore depends on a protocol that the authors themselves say is codebase-sensitive. Using Avg NDCG@5 as a proxy for the official Borda Count ranking adds another layer of ambiguity. Third, the two-stage training claim has no ablation: no experiment shows Stage 1 alone, Stage 2 alone, or the effect of removing Stage 1. These are all addressable, and I don't see evidence of deliberate inflation.\n\nWho this is for: anyone working on visual document retrieval or RAG over images will want the released models and the trade-off tables. The paper deserves peer review. An editor should send it out; a referee should require the contamination check, a robustness analysis across evaluation codebases, and an ablation of the two-stage recipe.","headline":"Strong released models with plausible SOTA scores, but the missing contamination check means the top-1 claim is conditional until the data split question is answered.","tokens_in":13447,"tokens_out":3448,"would_cite":true,"duration_ms":35228,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces llama-nemoretriever-colembed, a unified text-image retrieval model whose 3B variant reports NDCG@5 scores of 91.0 on ViDoRe V1 and 63.5 on ViDoRe V2, placing first on both leaderboards as of June 27, 2025.","keywords":["text-image retrieval","visual document retrieval","late interaction","ColBERT","contrastive learning","two-stage training","ViDoRe benchmark","multilingual retrieval"],"falsifier":"Search the ViDoRe V1 and V2 test queries and documents against the ColPali train set, Wiki-SS-NQ, VDR, VisRAG-Ret-Train, and Docmatix for near-duplicates; if any are found, retrain excluding them and check whether the 91.0 and 63.5 scores and the two first-place rankings survive. Recomputing the leaderboard with the official MTEB Borda-count metric instead of average NDCG@5 would also show whether the 'first' claim depends on the substitute metric.","tokens_in":12329,"feed_emoji":"🖼️","tokens_out":12211,"duration_ms":116911,"temperature":0.7,"pith_summary":"The paper introduces a family of two text-image retrieval models, llama-nemoretriever-colembed 1B and 3B, and claims the 3B model is the top performer on the ViDoRe V1 and V2 visual document retrieval leaderboards as of June 27, 2025, with NDCG@5 of 91.0 and 63.5. It argues the recipe that gets there is taking the Eagle2 vision-language model, switching causal attention to bidirectional attention, and fine-tuning with a ColBERT-style late-interaction objective in two stages: first on text-only retrieval data, then on text-image data with hard-negative mining. If correct, this shows visual document retrieval can be improved substantially by borrowing text-retrieval pretraining and token-level interaction, not just by scaling the vision model. The paper also quantifies a real cost: late interaction stores thousands of embeddings per image, over 2,700 times more storage than a single-vector bi-encoder, with projection to 512 dimensions reducing that overhead by about 88 percent at a modest accuracy loss.","feed_headline":"A 3B model tops both ViDoRe retrieval leaderboards with 91.0 NDCG@5","feed_subtitle":"Token-level late interaction plus two-stage training tops both ViDoRe leaderboards, storing 10 TB per million images.","key_machinery":"The carrying object is the late-interaction bi-encoder built on Eagle2: a shared vision-language encoder that tiles each document image dynamically, outputs one embedding per token, and scores a query by summing per-query-token MaxSim values against stored document token embeddings. That is ColBERT's mechanism transplanted into a vision-language model whose causal attention has been replaced with bidirectional attention. The training machinery is InfoNCE contrastive loss with hard-negative mining (top-k with a 0.95 positive-similarity threshold, K=2), applied first to text-only pairs and then to text-image pairs.","core_discovery":"The central claim is that a late-interaction multimodal retriever initialized from the bidirectional-attention Eagle2 backbone and trained with a two-stage contrastive recipe outperforms all published visual document retrieval models on the two ViDoRe benchmarks: NDCG@5 91.0 on V1 and 63.5 on V2, with the 1B variant also beating several 3B and 7B baselines. The paper attributes the gain to two design choices: replacing causal attention with bidirectional attention so every image or text token sees full context, and keeping per-token embeddings so query-document relevance is scored by ColBERT-style MaxSim rather than a single pooled vector. A two-stage schedule first builds text-only retrieval ability on large-scale corpora, then aligns text with images on a multimodal mixture, and hard negatives are mined with a threshold of 95 percent of the positive similarity. The paper's own numbers include the trade-off: the 3B model requires 10,311.1 GB of embeddings per million images at full 3072 dimensions, while projecting to 512 dimensions cuts storage by about 88 percent and reduces ViDoRe V1 from 0.9106 to 0.9064.","pith_inferences":["The paper does not report an overlap check between its training mixtures (ColPali train set, Wiki-SS-NQ, VDR, VisRAG-Ret-Train, Docmatix) and the ViDoRe test collections, so the reported margins, 91.0 versus 89.9 on V1 and 63.5 versus 60.7 on V2, are not settled against leakage until such a check is done.","The 512-dimensional projection's near-parity (0.9106 versus 0.9064 on ViDoRe V1) suggests the effective capacity for late-interaction matching may be far below 3072; testing 256 and 128 dimensions under the same recipe would map the storage-accuracy frontier.","The reranker comparison in Table 6 implies the practical production choice is not architectural but budgetary: a bi-encoder with reranking reaches comparable accuracy (0.9064 V1, 0.6214 V2) while storing 3.8 GB per million images instead of thousands of gigabytes."],"forward_implications":["The 3B model is reported to score 91.0 on ViDoRe V1 and 63.5 on V2, first on both leaderboards as of June 27, 2025, and to hold the top MTEB Visual Document Retrieval score of 83.10.","The 2.42B-parameter 1B variant outperforms several 3B and 7B visual retrieval baselines, so a smaller multimodal retriever can beat much larger competitors.","Text-only contrastive pretraining transfers to multimodal retrieval, making the two-stage recipe a substantive part of the result rather than a detail.","Full-dimensional late interaction costs 10,311.1 GB of stored embeddings per million images for the 3B model; projecting to 512 dimensions cuts storage by about 88 percent with a small NDCG@5 drop.","On MIRACL-VISION, the 3B model's mean score of 0.5841 beats prior visual retrieval models on image queries across 18 languages."],"supporting_citations":[{"why":"Introduces the document-as-image retrieval paradigm, the ColPali training set, and the ViDoRe V1 benchmark that supply the evaluation and part of the training data.","marker":"[7]"},{"why":"Defines the late-interaction MaxSim scoring mechanism that the model adopts for fine-grained token-level retrieval.","marker":"[17]"},{"why":"Supplies the Eagle2 vision-language backbone that the model starts from and modifies with bidirectional attention.","marker":"[10]"},{"why":"Provides the hard-negative mining method (top-k with a positive-similarity threshold) and the text-only training methodology used in stage one.","marker":"[2]"},{"why":"Defines the ViDoRe V2 benchmark and the updated evaluation protocol that the second leaderboard claim depends on.","marker":"[15]"},{"why":"Supplies the InfoNCE contrastive loss that drives both stages of training.","marker":"[18]"},{"why":"Defines the MIRACL-VISION multilingual visual retrieval benchmark used for the multilingual generalization results.","marker":"[30]"}],"fun_headline_variants":["3B model tops ViDoRe with 91.0 NDCG@5","Late interaction retriever wins ViDoRe V1 and V2","Bidirectional attention + ColBERT-style MaxSim tops ViDoRe","Two-stage training yields SOTA text-image retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ViDoRe scores are comparable and untainted: none of the training corpora overlap the test queries or documents, and the reported average NDCG@5 mirrors the official leaderboard ordering.","fun_headline_variants_meta":{"raw":{"variants":["3B model tops ViDoRe with 91.0 NDCG@5","Late interaction retriever wins ViDoRe V1 and V2","Bidirectional attention + ColBERT-style MaxSim tops ViDoRe","Two-stage training yields SOTA text-image retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00047,"raw_usage":{"total_tokens":2360,"prompt_tokens":985,"completion_tokens":1375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1299}},"tokens_in":601,"tokens_out":1375,"duration_ms":10744,"temperature":1.0,"reasoning_tokens":1299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:24:08.852315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search the ViDoRe V1 and V2 test queries and documents against the ColPali train set, Wiki-SS-NQ, VDR, VisRAG-Ret-Train, and Docmatix for near-duplicates; if any are found, retrain excluding them and check whether the 91.0 and 63.5 scores and the two first-place rankings survive. Recomputing the leaderboard with the official MTEB Borda-count metric instead of average NDCG@5 would also show whether the 'first' claim depends on the substitute metric.","supporting_citations":[{"cited_title":"Colpali: Efficient document retrieval with vision language models, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces the document-as-image retrieval paradigm, the ColPali training set, and the ViDoRe V1 benchmark that supply the evaluation and part of the training data."},{"cited_title":"Colbert: Efficient and effective passage search via con- textualized late interaction over bert","cited_arxiv_id":null,"evidence_quote":"Defines the late-interaction MaxSim scoring mechanism that the model adopts for fine-grained token-level retrieval."},{"cited_title":"A simple frame- work for contrastive learning of visual representations","cited_arxiv_id":null,"evidence_quote":"Supplies the InfoNCE contrastive loss that drives both stages of training."}],"review_version":1}