{"id":"66ed4d2d-ed1d-4f3e-aa01-28b2453c3881","arxiv_id":"2506.02291","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new benchmark, MMIR, built from Wikipedia and WIT by masking entity names and supplying canonical entity images, tests mixed-modal retrieval with single- and multi-entity queries.","lead":"This paper releases two Wikipedia-derived datasets, Entity Images (EI) and Mixed-Modal Image Retrieval (MMIR), that turn image-caption pairs into retrieval queries mixing entity photos with masked text. The authors argue this fills a gap in benchmarks for combined image-and-text retrieval and show that fine-tuning on MMIR improves scores on the new task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MMIR query construction may leak the target image: WIT targets can coincide with EI canonical entity images, making retrieval solvable by visual matching alone; no deduplication check is reported.","rationale":"The reader identified the absence of leakage analysis and ablations as the weakest assumption. I agree, and I sharpen it: the construction in Section 2.2.2 has a plausible mechanism for direct leakage, because both the EI canonical images and the WIT target images come from Wikipedia/Wikimedia and there is no reported exclusion of cases where the target image is one of the query's entity images. This is not merely a missing adversarial check; it is a structural property of the dataset creation pipeline. If even a modest fraction of test examples leak, the R@1 improvement from fine-tuning on MMIR can be inflated by image-to-image matching, and the benchmark would not measure mixed-modal grounding. The proposed test is cheap and decisive: deduplicate and recompute. I therefore keep the reader's CONDITIONAL verdict rather than upgrading to ACCEPT or moving to REJECT, because the concern is not yet confirmed and the datasets may still be useful if the overlap is small or if filtered performance remains high. Separately, Table 4 contains garbled entries in the MS COCO T→I row (e.g., repeated '63.47' and '27.75' values) and should be corrected, but that is a presentation defect rather than the load-bearing issue. Overall, the paper's contribution is valuable as a dataset resource, but the central 'deep cross-modal contextual understanding' claim requires the leakage test before the benchmark is adopted as rigorous.","tokens_in":9050,"tokens_out":3386,"duration_ms":33689,"concrete_test":"Compute exact-URL and perceptual-hash (e.g., pHash) matches between every MMIR target image and the EI canonical images that appear in the corresponding query, for both the validation and test splits, and report the fraction of examples where the target is identical or near-duplicate to a query entity image. Then rerun the Section 4.2.1 MMIR I+T→I evaluation after removing those examples and compare R@1/R@5 against the full-set numbers. If the filtered R@1 drops substantially, the benchmark can be solved by visual matching and the deep cross-modal claim is unsupported; if the overlap fraction is negligible and filtered scores are stable, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MMIR requires deep cross-modal contextual understanding assumes the supplied entity images do not already contain or duplicate the target image. Section 2.2.2 constructs a query from the EI canonical images for entities in P_i = I_i ∩ T_i and asks the model to retrieve the original WIT image i_i. But EI canonical images are themselves sampled from Wikipedia/Wikimedia pages (Section 2.1), and WIT collects images from Wikipedia pages with their reference descriptions (Srinivasan et al., 2021). Nothing in Section 2.2.2 filters out the case where i_i equals, is a thumbnail variant of, or is visually near-duplicate of one of the canonical entity images. In that case, the query contains the target image, and the task degenerates to image-identity matching; the masked caption and any purported contextual understanding are irrelevant. The evaluation in Section 4.2.1 reports only the full mixed-modal I+T→I and I→I+T scores, with no image-only or text-only baseline, no ablation after removing exact or near-duplicate targets, and no leakage statistics. The reported R@1 gain from 7.32 to 13.17 is therefore not yet evidence for mixed-modal reasoning until the trivial-matching route is quantified and removed. The human evaluation in Section 5.2 also does not resolve this: raters judged semantic coherence between the masked caption plus entity images and the original caption, not whether retrieval is actually hard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two new datasets: the Entity Image Dataset (EI), containing canonical images for roughly 1.8M Wikipedia entities across eight physical-entity categories plus an 'other' category, and the Mixed-Modal Image Retrieval Dataset (MMIR), constructed from WIT by masking entity names in reference descriptions and linking the masked tokens to EI canonical images. MMIR contains 9.06M training examples in 108 languages and supports two query types: single entity-image queries and multi-entity-image queries. The paper also presents a dual-encoder model evaluation showing that fine-tuning on MMIR improves MMIR retrieval Recall@1 from 7.32 to 13.17 over zero-shot, without degrading Flickr30k or MS-COCO performance, and reports crowd-sourced human evaluations of EI image quality (97% majority Good/Excellent) and MMIR semantic coherence (81% majority Good/Excellent).","tokens_in":9360,"tokens_out":3898,"duration_ms":34552,"significance":"If the benchmark is validated as claimed, MMIR would be a large-scale, multilingual resource for mixed-modal image retrieval, filling a genuine gap in the benchmark landscape. The public release under CC-BY-SA and the careful construction pipeline are strengths. The human evaluation is on fresh random samples after category-specific threshold tuning, so the earlier threshold selection does not create a circular validation. The fine-tuning gain on MMIR over zero-shot is a useful signal of task learnability. However, the central claim that MMIR demands 'deep cross-modal contextual understanding' is not yet established, because the evaluation does not rule out trivial visual matching between the supplied entity images and the target image, and no text-only or image-only baselines are reported.","major_comments":[{"comment":"The query construction in Section 2.2.2 does not remove cases where the target WIT image is identical or visually near-duplicate to one of the EI canonical entity images included in the query; because EI images are harvested from Wikipedia pages and WIT images also originate from Wikipedia, this overlap is plausible. If the target image coincides with a canonical entity image, the retrieval task degenerates to image-identity matching and the masked caption is unnecessary. The paper reports no leakage statistics, no image-level deduplication check, and no evaluation on a cleaned subset, so the reported R@1 gain (Table 4: 7.32 to 13.17) does not yet establish that MMIR requires cross-modal contextual understanding. The authors should quantify the overlap and either filter such examples or report results after removing them.","section":"Section 2.2.2 (Masking) and Section 4.2.1 (Evaluation on MMIR Eval Set)"},{"comment":"The evaluation includes only the full mixed-modal queries I+T→I and I→I+T; there is no text-only or image-only query baseline. Without such baselines, the improvement from fine-tuning on MMIR could be attributed to learning a visual entity-to-image matching function that ignores the masked caption. The authors should add ablations that (a) remove the entity images (text-only query), (b) remove the masked caption (image-only query), and (c) use the original unmasked caption, to demonstrate that both modalities contribute and that the masked text is not redundant. This is necessary to support the abstract's claim that the benchmark demands deep cross-modal contextual understanding.","section":"Section 4.2.1 (Evaluation on MMIR Eval Set), Table 4"},{"comment":"The human evaluation for MMIR asks raters to judge semantic coherence between the masked caption plus entity images and the original caption, which is appropriate for checking annotation quality but does not measure whether the retrieval task is challenging or whether the target image is identifiable only through mixed-modal reasoning. The 81% Good/Excellent majority ratings therefore do not address the leakage or trivial-matching concerns raised in Section 4.2.1. An additional human study that presents the query with the target image withheld and asks raters to select the correct image from distractors, or to rate whether the query uniquely determines a target, would more directly validate the benchmark's difficulty.","section":"Section 5.2 (Evaluating the MMIR Dataset)"}],"minor_comments":[{"comment":"The abbreviation for the Mixed-Modal Image Retrieval Dataset is spelled \"MMRI\" in the introduction but \"MMIR\" throughout the rest of the paper; please use a single consistent abbreviation.","section":"Section 1 (Introduction)"},{"comment":"There is a typo in the figure caption: \"as shwon\" should be \"as shown.\"","section":"Section 4.1 (Experimental Setup), Figure 4 caption"},{"comment":"The MS COCO T→I row contains garbled entries for the Combined column (\"63.47 63.47 63.47\"), likely a formatting error; each cell should contain a single Recall@K value.","section":"Table 4 (Evaluation Results)"},{"comment":"The phrase \"approximately ∼ 4.34M\" uses both \"approximately\" and \"∼\"; please choose one to avoid redundancy.","section":"Section 2.1 (Candidate Image Consolidation)"},{"comment":"The reference for \"Conceptual 12M\" is incomplete; it should include the arXiv identifier or a proper venue citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is substantial and the paper is generally well-structured, but the central claim of 'deep cross-modal contextual understanding' is not yet supported without a leakage analysis and modality-ablation experiments. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection. I also note that the threshold-tuning followed by fresh human evaluation is methodologically sound, and the public release of the datasets is a positive aspect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2506.02291. First, it ships two real, large-scale resources: the Entity Image Dataset (1.8M entities with canonical images) and the Mixed-Modal Image Retrieval Dataset (9M training examples, 108 languages), built by masking entity names in WIT captions and linking them to entity images. Second, the paper's headline claim — that MMIR demands deep cross-modal contextual understanding — is not actually supported, because the authors never rule out the obvious shortcut where the target image is identical or near-identical to one of the supplied entity images.\n\nWhat is new: the masked-entity-with-image query formulation at this scale. I don't know of prior work that replaces textual entity names with links to canonical images for retrieval. EI alone is a useful resource, and the human eval for both datasets is decent: 97% good/excellent for EI, 81% for MMIR. Fine-tuning on MMIR does improve MMIR R@1 from 7.32 to 13.17 over zero-shot, which suggests the data carries signal. The release under CC-BY-SA is a plus.\n\nWhere the soft spots are, in order of seriousness. The leakage problem is real. The construction in Section 2.2.2 intersects entity mentions in the image and text, then uses EI canonical images for those entities. Since EI comes from Wikipedia pages and WIT images also come from Wikipedia pages, there is no reason to think the target i_i is never one of those canonical images or a thumbnail variant. The paper reports no dedup check, no leakage statistics, and no image-only or text-only baselines, so the reported gain is consistent with the model learning to match images to each other. The human eval in Section 5.2 asks raters whether the masked caption plus entity images preserve the original caption's meaning; it does not measure whether retrieval is actually hard. Table 4 also has a garbled MS COCO T→I entry (the CC3M column looks corrupted). Missing error bars are minor.\n\nNone of this kills the datasets as resources, but it does kill the abstract's \"deep cross-modal contextual understanding\" claim as stated. The fix is mechanical: deduplicate or measure overlap, run single-modality baselines, and report the fraction of queries where the target is one of the entity images.\n\nWho gets value: anyone building or benchmarking mixed-modal retrieval systems, plus people who want to study shortcut learning, because this paper is a clean example of a benchmark leak. It deserves a serious referee — the resource is useful and the problems are addressable — but it should not be accepted in its current form.","headline":"A genuinely useful dataset resource, but the paper's main claim about forcing deep cross-modal understanding is not backed up — the construction may let models solve MMIR by visual matching alone.","tokens_in":9897,"tokens_out":3065,"would_cite":true,"duration_ms":27605,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper builds a 9.06M-example mixed-modal retrieval benchmark that masks entity names in captions and supplies the entities as images, plus an 1.80M-image entity dataset, and shows fine-tuning lifts Recall@1 from 7.32 to 13.17.","keywords":["mixed-modal retrieval","image retrieval benchmark","entity images","canonical image selection","WIT dataset","multilingual retrieval","masked caption grounding","cross-modal understanding"],"falsifier":"Run two ablations on the MMIR test split: a visual-only model that receives the entity image(s) with the masked caption stripped out, and a text-only model that receives the masked caption with all entity images stripped out. If either approaches Recall@1 of 13.17, the benchmark is solvable without joint cross-modal reasoning; likewise, computing near-duplicate rates between EI canonical images and MMIR target images would reveal whether part of the target image is already visible in the query.","tokens_in":8885,"feed_emoji":"🖼️","tokens_out":15578,"duration_ms":112910,"temperature":0.7,"pith_summary":"The paper's central claim is that image retrieval can and should be evaluated in a mixed-modal setting — the query is not a caption alone or an image alone, but a masked textual description that must be resolved against one or more entity images. To make that claim testable, the authors build two resources: EI, 1.80M canonical images for Wikipedia entities, and MMIR, derived from WIT, which contains 9.06M training queries in 108 languages where entity names are masked out of captions and supplied instead as images. They report that fine-tuning a dual-encoder model on MMIR raises Recall@1 on the benchmark from 7.32 (zero-shot) to 13.17 without hurting standard Flickr30k and MS-COCO retrieval, and that crowd-sourced raters judge over 97% of EI image-entity pairs and over 81% of MMIR masked queries good or excellent. If the benchmark holds up, it closes a gap that the paper argues existing resources leave open: a large-scale multilingual testbed where a model cannot succeed by matching one modality alone, because the textual and visual inputs only make sense together.","feed_headline":"Training on 9M mixed-modal queries nearly doubles retrieval recall","feed_subtitle":"MMIR's 9.06M masked-caption queries across 108 languages lift top-match accuracy from 7.32% to 13.17%.","key_machinery":"The load-bearing machinery is the masking-and-grounding step that defines the MMIR query format. For each WIT pair, the intersection of image-detected and text-detected entities, restricted to entities with an EI canonical image, is masked out of the reference description; the missing names are replaced by [MASK_*] tokens bound to the corresponding canonical entity images, so the query is inherently mixed-modal. On the EI side, the machinery is the canonical-selection pipeline: crawl roughly 124M Wikipedia content pages, filter images below 100 pixels or with non-research licenses, confirm via an image annotation service that the page's entity is depicted, consolidate duplicate image URLs across languages and sections, and rank survivors by section position with a confidence-score tie-break, using category-specific confidence thresholds that raised entity coverage from about 862K to 1.80M.","core_discovery":"The core discovery is a construction that converts ordinary web image-caption pairs into retrieval problems that require joint reasoning. From WIT's 37M image-text pairs, the authors keep the 17.2M with reference descriptions, run image and text entity recognizers on each pair, and compute the intersection of the entities the two recognizers agree on. Each intersecting entity is then replaced in the caption by a [MASK_*] token pointing to its canonical image in EI, and the task becomes: given the masked caption plus the entity images, retrieve the original WIT image. Roughly half the resulting 9M+ queries carry a single entity and over 90% carry three or fewer, giving the benchmark a built-in difficulty gradient. The authors present this as the largest multilingual mixed-modal retrieval dataset available, evidence for its difficulty being that MMIR fine-tuning lifts Recall@1 from 7.32 to 13.17, and evidence for its quality being majority human ratings of excellent or good on over 81% of sampled queries.","pith_inferences":["An open check the paper does not run: a model given only the entity images (caption stripped) or only the masked caption (images stripped) on the MMIR test split — if either approaches the full query's Recall@1 of 13.17, the task is solvable without joint reasoning.","A leakage test worth running: EI canonical images come from the same Wikipedia pages WIT sampled, so an image-level near-duplicate analysis between the two collections would show how much of the retrieval signal could be visual overlap rather than contextual understanding.","A harder stress test: generate variants where the masked entity is deliberately not depicted in the target image, or where the relational text binds more than five entities, to push beyond the current construction's bound that every entity must appear in both image and text.","A further use the paper leaves implicit: EI's Wikidata-linked canonical images form a cross-lingual visual grounding vocabulary that could serve entity-linking or visual question-answering tasks beyond retrieval."],"forward_implications":["Training signal: MMIR adds 9.06M training examples, 46.33K validation examples, and 56.67K test examples with fixed WIT-derived splits, so models can be trained and compared on the same mixed-modal task across 108 languages.","Fine-tuning on MMIR nearly doubles the zero-shot score on the benchmark itself (Recall@1 from 7.32 to 13.17), and combining MMIR with CC3M matches CC3M-only performance on Flickr30k and MS-COCO, so the dataset can be added to existing training recipes without measurable cost.","Difficulty gradient: with roughly half the examples at one entity and over 90% at three or fewer, the benchmark can separate models by how many visual entities they can bind into a single relational query.","Quality assurance: majority human ratings mark over 97% of sampled EI canonical images and over 81% of sampled MMIR masked queries as good or excellent, so the datasets are presented as usable as-is rather than requiring further cleaning.","Maintenance caveat the authors state in the Limitations section: the datasets snapshot Wikipedia at one point in time, and as pages, images, and facts change, the datasets will need regeneration from updated snapshots to stay relevant."],"supporting_citations":[{"why":"Supplies WIT, the source collection: every MMIR image-text pair, its reference description, and the train/validation/test splits are inherited from it.","marker":"(Srinivasan et al., 2021)"},{"why":"Image annotation service used both to verify that candidate images depict their page's entity and to produce the image-side entity set I_i for MMIR masking.","marker":"(Google Cloud, b)"},{"why":"Text annotation service that extracts the text-side entity set T_i from each reference description; the intersection with I_i determines what gets masked.","marker":"(Google Cloud, a)"},{"why":"Defines the PaLI-style visual-language architecture and the WebLI-10B pre-trained model from which all Section 4 evaluations start.","marker":"(Chen et al., 2023)"},{"why":"Supplies mT5-Base, the initialization of the text encoder in the dual-encoder retrieval model.","marker":"(Xue et al., 2021)"},{"why":"Supplies ViT-Large, the initialization of the vision encoder in the retrieval model.","marker":"(Dosovitskiy et al., 2021)"},{"why":"Provides CC3M, the comparison training corpus and the partner dataset in the combined MMIR+CC3M fine-tuning recipe.","marker":"(Changpinyo et al., 2021)"},{"why":"Provides Flickr30k, one of the two standard benchmarks used to check that MMIR fine-tuning does not degrade general image-text retrieval.","marker":"(Young et al., 2014)"},{"why":"Provides MS COCO, the other standard benchmark used to check generalization after MMIR fine-tuning.","marker":"(Lin et al., 2014)"}],"fun_headline_variants":["Mixed-modal retrieval gains 9.06M masked-caption queries","Entity images plus text queries: new benchmark for retrieval","Recall improves from 7.32% to 13.17% on mixed-modal benchmark","Largest multilingual mixed-modal retrieval dataset from WIT","Masked-caption queries with entity images challenge retrievers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on MMIR queries not being solvable by shallow visual matching between the supplied entity images and the target image; the paper reports no leakage analysis, no image-level deduplication between EI canonical images and WIT target images, and no text-only or image-only ablations, so the assumption that joint reasoning is required is asserted rather than demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Mixed-modal retrieval gains 9.06M masked-caption queries","Entity images plus text queries: new benchmark for retrieval","Recall improves from 7.32% to 13.17% on mixed-modal benchmark","Largest multilingual mixed-modal retrieval dataset from WIT","Masked-caption queries with entity images challenge retrievers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2905,"prompt_tokens":922,"completion_tokens":1983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1909}},"tokens_in":538,"tokens_out":1983,"duration_ms":14541,"temperature":1.0,"reasoning_tokens":1909,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:26:23.029554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run two ablations on the MMIR test split: a visual-only model that receives the entity image(s) with the masked caption stripped out, and a text-only model that receives the masked caption with all entity images stripped out. If either approaches Recall@1 of 13.17, the benchmark is solvable without joint cross-modal reasoning; likewise, computing near-duplicate rates between EI canonical images and MMIR target images would reveal whether part of the target image is already visible in the query.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies WIT, the source collection: every MMIR image-text pair, its reference description, and the train/validation/test splits are inherited from it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PaLI-style visual-language architecture and the WebLI-10B pre-trained model from which all Section 4 evaluations start."},{"cited_title":"Lawrence Zitnick","cited_arxiv_id":null,"evidence_quote":"Provides MS COCO, the other standard benchmark used to check generalization after MMIR fine-tuning."}],"review_version":1}