Pith. sign in

REVIEW 3 major objections 5 minor 19 references

Entity Image and Mixed-Modal Image Retrieval Datasets

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper builds a 9.06M-example mixed-modal retrieval benchmark that masks entity names in captions and supplies the entities as images, plus an 1.80M-image entity dataset, and shows fine-tuning lifts Recall@1 from 7.32 to 13.17.

desk verdict A genuinely useful dataset resource, but the paper's main claim about forcing deep cross-modal understanding is not backed up — the construction may let models solve MMIR by visual matching alone. read the letter →

arxiv 2506.02291 v1 pith:G3PS23H4 submitted 2025-06-02 cs.CV cs.IR

Cristian-Ioan Blaga , Paul Suganthan , Sahil Dua , Krishna Srinivasan , Enrique Alfonseca , Peter Dornbach , Tom Duerig , Imed Zitouni
show 1 more author
Zhe Dong
This is my paper · ORCID
classification cs.CVcs.IR
keywords mixed-modalretrievalimagebenchmarkentityimagescanonicalselectionWITdatasetmultilingualmaskedcaptiongroundingcross-modalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that image retrieval can and should be evaluated in a mixed-modal setting — the query is not a caption alone or an image alone, but a masked textual description that must be resolved against one or more entity images. To make that claim testable, the authors build two resources: EI, 1.80M canonical images for Wikipedia entities, and MMIR, derived from WIT, which contains 9.06M training queries in 108 languages where entity names are masked out of captions and supplied instead as images. They report that fine-tuning a dual-encoder model on MMIR raises Recall@1 on the benchmark from 7.32 (zero-shot) to 13.17 without hurting standard Flickr30k and MS-COCO retrieval, and that crowd-sourced raters judge over 97% of EI image-entity pairs and over 81% of MMIR masked queries good or excellent. If the benchmark holds up, it closes a gap that the paper argues existing resources leave open: a large-scale multilingual testbed where a model cannot succeed by matching one modality alone, because the textual and visual inputs only make sense together.

What carries the argument

The load-bearing machinery is the masking-and-grounding step that defines the MMIR query format. For each WIT pair, the intersection of image-detected and text-detected entities, restricted to entities with an EI canonical image, is masked out of the reference description; the missing names are replaced by [MASK_*] tokens bound to the corresponding canonical entity images, so the query is inherently mixed-modal. On the EI side, the machinery is the canonical-selection pipeline: crawl roughly 124M Wikipedia content pages, filter images below 100 pixels or with non-research licenses, confirm via an image annotation service that the page's entity is depicted, consolidate duplicate image URLs across languages and sections, and rank survivors by section position with a confidence-score tie-break, using category-specific confidence thresholds that raised entity coverage from about 862K to 1.80M.

What would settle it

Run two ablations on the MMIR test split: a visual-only model that receives the entity image(s) with the masked caption stripped out, and a text-only model that receives the masked caption with all entity images stripped out. If either approaches Recall@1 of 13.17, the benchmark is solvable without joint cross-modal reasoning; likewise, computing near-duplicate rates between EI canonical images and MMIR target images would reveal whether part of the target image is already visible in the query.

Watch

Extended reading notes

Core claim

The core discovery is a construction that converts ordinary web image-caption pairs into retrieval problems that require joint reasoning. From WIT's 37M image-text pairs, the authors keep the 17.2M with reference descriptions, run image and text entity recognizers on each pair, and compute the intersection of the entities the two recognizers agree on. Each intersecting entity is then replaced in the caption by a [MASK_*] token pointing to its canonical image in EI, and the task becomes: given the masked caption plus the entity images, retrieve the original WIT image. Roughly half the resulting 9M+ queries carry a single entity and over 90% carry three or fewer, giving the benchmark a built-in difficulty gradient. The authors present this as the largest multilingual mixed-modal retrieval dataset available, evidence for its difficulty being that MMIR fine-tuning lifts Recall@1 from 7.32 to 13.17, and evidence for its quality being majority human ratings of excellent or good on over 81% of sampled queries.

Load-bearing premise

The benchmark's validity rests on MMIR queries not being solvable by shallow visual matching between the supplied entity images and the target image; the paper reports no leakage analysis, no image-level deduplication between EI canonical images and WIT target images, and no text-only or image-only ablations, so the assumption that joint reasoning is required is asserted rather than demonstrated.

Editorial extensions

If this is right

  • Training signal: MMIR adds 9.06M training examples, 46.33K validation examples, and 56.67K test examples with fixed WIT-derived splits, so models can be trained and compared on the same mixed-modal task across 108 languages.
  • Fine-tuning on MMIR nearly doubles the zero-shot score on the benchmark itself (Recall@1 from 7.32 to 13.17), and combining MMIR with CC3M matches CC3M-only performance on Flickr30k and MS-COCO, so the dataset can be added to existing training recipes without measurable cost.
  • Difficulty gradient: with roughly half the examples at one entity and over 90% at three or fewer, the benchmark can separate models by how many visual entities they can bind into a single relational query.
  • Quality assurance: majority human ratings mark over 97% of sampled EI canonical images and over 81% of sampled MMIR masked queries as good or excellent, so the datasets are presented as usable as-is rather than requiring further cleaning.
  • Maintenance caveat the authors state in the Limitations section: the datasets snapshot Wikipedia at one point in time, and as pages, images, and facts change, the datasets will need regeneration from updated snapshots to stay relevant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An open check the paper does not run: a model given only the entity images (caption stripped) or only the masked caption (images stripped) on the MMIR test split — if either approaches the full query's Recall@1 of 13.17, the task is solvable without joint reasoning.
  • A leakage test worth running: EI canonical images come from the same Wikipedia pages WIT sampled, so an image-level near-duplicate analysis between the two collections would show how much of the retrieval signal could be visual overlap rather than contextual understanding.
  • A harder stress test: generate variants where the masked entity is deliberately not depicted in the target image, or where the relational text binds more than five entities, to push beyond the current construction's bound that every entity must appear in both image and text.
  • A further use the paper leaves implicit: EI's Wikidata-linked canonical images form a cross-lingual visual grounding vocabulary that could serve entity-linking or visual question-answering tasks beyond retrieval.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces two new datasets: the Entity Image Dataset (EI), containing canonical images for roughly 1.8M Wikipedia entities across eight physical-entity categories plus an 'other' category, and the Mixed-Modal Image Retrieval Dataset (MMIR), constructed from WIT by masking entity names in reference descriptions and linking the masked tokens to EI canonical images. MMIR contains 9.06M training examples in 108 languages and supports two query types: single entity-image queries and multi-entity-image queries. The paper also presents a dual-encoder model evaluation showing that fine-tuning on MMIR improves MMIR retrieval Recall@1 from 7.32 to 13.17 over zero-shot, without degrading Flickr30k or MS-COCO performance, and reports crowd-sourced human evaluations of EI image quality (97% majority Good/Excellent) and MMIR semantic coherence (81% majority Good/Excellent).

Significance. If the benchmark is validated as claimed, MMIR would be a large-scale, multilingual resource for mixed-modal image retrieval, filling a genuine gap in the benchmark landscape. The public release under CC-BY-SA and the careful construction pipeline are strengths. The human evaluation is on fresh random samples after category-specific threshold tuning, so the earlier threshold selection does not create a circular validation. The fine-tuning gain on MMIR over zero-shot is a useful signal of task learnability. However, the central claim that MMIR demands 'deep cross-modal contextual understanding' is not yet established, because the evaluation does not rule out trivial visual matching between the supplied entity images and the target image, and no text-only or image-only baselines are reported.

major comments (3)
  1. [Section 2.2.2 (Masking) and Section 4.2.1 (Evaluation on MMIR Eval Set)] The query construction in Section 2.2.2 does not remove cases where the target WIT image is identical or visually near-duplicate to one of the EI canonical entity images included in the query; because EI images are harvested from Wikipedia pages and WIT images also originate from Wikipedia, this overlap is plausible. If the target image coincides with a canonical entity image, the retrieval task degenerates to image-identity matching and the masked caption is unnecessary. The paper reports no leakage statistics, no image-level deduplication check, and no evaluation on a cleaned subset, so the reported R@1 gain (Table 4: 7.32 to 13.17) does not yet establish that MMIR requires cross-modal contextual understanding. The authors should quantify the overlap and either filter such examples or report results after removing them.
  2. [Section 4.2.1 (Evaluation on MMIR Eval Set), Table 4] The evaluation includes only the full mixed-modal queries I+T→I and I→I+T; there is no text-only or image-only query baseline. Without such baselines, the improvement from fine-tuning on MMIR could be attributed to learning a visual entity-to-image matching function that ignores the masked caption. The authors should add ablations that (a) remove the entity images (text-only query), (b) remove the masked caption (image-only query), and (c) use the original unmasked caption, to demonstrate that both modalities contribute and that the masked text is not redundant. This is necessary to support the abstract's claim that the benchmark demands deep cross-modal contextual understanding.
  3. [Section 5.2 (Evaluating the MMIR Dataset)] The human evaluation for MMIR asks raters to judge semantic coherence between the masked caption plus entity images and the original caption, which is appropriate for checking annotation quality but does not measure whether the retrieval task is challenging or whether the target image is identifiable only through mixed-modal reasoning. The 81% Good/Excellent majority ratings therefore do not address the leakage or trivial-matching concerns raised in Section 4.2.1. An additional human study that presents the query with the target image withheld and asks raters to select the correct image from distractors, or to rate whether the query uniquely determines a target, would more directly validate the benchmark's difficulty.
minor comments (5)
  1. [Section 1 (Introduction)] The abbreviation for the Mixed-Modal Image Retrieval Dataset is spelled "MMRI" in the introduction but "MMIR" throughout the rest of the paper; please use a single consistent abbreviation.
  2. [Section 4.1 (Experimental Setup), Figure 4 caption] There is a typo in the figure caption: "as shwon" should be "as shown."
  3. [Table 4 (Evaluation Results)] The MS COCO T→I row contains garbled entries for the Combined column ("63.47 63.47 63.47"), likely a formatting error; each cell should contain a single Recall@K value.
  4. [Section 2.1 (Candidate Image Consolidation)] The phrase "approximately ∼ 4.34M" uses both "approximately" and "∼"; please choose one to avoid redundancy.
  5. [References] The reference for "Conceptual 12M" is incomplete; it should include the arXiv identifier or a proper venue citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: dataset construction is self-contained; performance claims are measured on held-out splits and external benchmarks.

full rationale

The paper's main chain is empirical dataset construction and evaluation: EI is built from Wikipedia using annotation-service confidence scores and position-based selection with category-specific thresholds; MMIR is built by masking entity names that appear in both the WIT image and its reference text; and the model evaluation fine-tunes on MMIR train and scores on MMIR test plus Flickr30k/COCO. None of these steps fits a parameter to the target metric and then reports that metric as a predicted result. The category-specific thresholds in Section 2.1 are tuned against human annotations, but the human evaluation in Sections 5.1 and 5.2 reports ratings on newly sampled examples, so this is calibration followed by validation rather than circularity. The only self-citations (WIT as source data and Dong et al. for the dual-encoder architecture) are external source material or standard design inspiration and are not load-bearing evidence for a derived claim. The reviewer-flagged risk that MMIR targets may coincide with or be near-duplicates of the supplied entity images is a benchmark-validity and leakage concern, not a circularity of the paper's derivation chain: it does not make the reported R@1 numbers equivalent to the inputs by construction. No equation, fitted parameter, or prediction reduces to its own inputs, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central benchmark construction has one fitted parameter set (category-specific confidence thresholds) plus domain assumptions about commercial annotation quality, Wikipedia page-position semantics, the adequacy of WIT splits, and absence of image leakage. No new physical or semantic entities are postulated.

free parameters (2)
  • category-specific confidence threshold t = not reported
    In Section 2.1, separate thresholds on the image annotation confidence score are chosen for each of the 8 entity categories plus Other, optimized to maximize agreement with human annotations. These thresholds determine which entities receive canonical images and therefore affect both EI and all MMIR examples.
  • maximum number of entities per training example = 5
    Section 4.1 filters the MMIR training set to examples with at most 5 entities because the majority of examples have 5 or fewer. This hand-chosen cutoff changes the training distribution relative to the full dataset and is not evaluated as a variable.
assumptions (4)
  • domain assumption Google Cloud Vision and Natural Language APIs correctly identify entities in images and texts, and their confidence scores correlate with human judgment.
    Used throughout Sections 2.1 and 2.2.2 to build candidate entity images and to compute the intersection Pi = Ii ∩ Ti for masking. If the APIs are noisy, a large fraction of MMIR examples will have incorrect or missing masks.
  • domain assumption Images appearing higher on a Wikipedia page are more likely to be representative of the entity.
    Section 2.1 Candidate Image Consolidation explicitly states this heuristic when prioritizing image URLs by topmost position before confidence-based ranking.
  • domain assumption The WIT reference description is the most salient text for the image, and the original WIT splits are adequate for a retrieval benchmark.
    Section 2.2.1 restricts to reference descriptions only, discarding alt-text and attribution; Section 3.2 reuses WIT random splits for train, validation, and test, without a language-balanced or temporal split. This affects how well results generalize.
  • domain assumption The target images in MMIR are not trivially recoverable from the supplied entity images alone, i.e., no systematic near-duplicate leakage.
    The benchmark's difficulty claim depends on this, but no deduplication or leakage analysis is reported anywhere in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Entity Image and Mixed-Modal Image Retrieval Datasets." pith.science (2026). https://pith.science/paper/G3PS23H4

@misc{pith2026250602291,
  author       = {Pith},
  title        = {Pith review of: Entity Image and Mixed-Modal Image Retrieval Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3PS23H4}},
  note         = {Machine review of arXiv:2506.02291}
}
read the original abstract

Despite advances in multimodal learning, challenging benchmarks for mixed-modal image retrieval that combines visual and textual information are lacking. This paper introduces a novel benchmark to rigorously evaluate image retrieval that demands deep cross-modal contextual understanding. We present two new datasets: the Entity Image Dataset (EI), providing canonical images for Wikipedia entities, and the Mixed-Modal Image Retrieval Dataset (MMIR), derived from the WIT dataset. The MMIR benchmark features two challenging query types requiring models to ground textual descriptions in the context of provided visual entities: single entity-image queries (one entity image with descriptive text) and multi-entity-image queries (multiple entity images with relational text). We empirically validate the benchmark's utility as both a training corpus and an evaluation set for mixed-modal retrieval. The quality of both datasets is further affirmed through crowd-sourced human annotations. The datasets are accessible through the GitHub page: https://github.com/google-research-datasets/wit-retrieval.

Figures

Figures reproduced from arXiv: 2506.02291 by the authors.

Figure 1
Figure 1. Examples of Entity Image Dataset (EI) and Mixed-Modal Image Retrieval Dataset (MMIR). Starting from WIT (Srinivasan et al., 2021) dataset, entity recognition is applied to identify the entities mentioned in the reference description. For each entity, a unique image is identified from Wikipedia, as the canonical entity image, in EI dataset. In MMIR dataset, we remove the entity name from original reference descriptio… view at source ↗
Figure 2
Figure 2. An example Wikipedia page with multiple candidate images. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example features of Entity Image Dataset ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Shared-parameter dual-encoder architecture for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Human evaluation template for EI dataset (on the left) and MMIR dataset (on the right). a robust and effective canonical image selection process across diverse entity types within the EI dataset. In addition to evaluating the semantic match be￾tween the entity and its …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset . arXiv preprint arXiv:1611.09268

    arXiv 2016

  4. [4]

    Henry, Robert Bradshaw, and Nathan Weizenbaum

    Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R. Henry, Robert Bradshaw, and Nathan Weizenbaum. 2010. https://doi.org/10.1145/1809028.1806638 Flumejava: easy, efficient data-parallel pipelines . SIGPLAN Not., 45(6):363–375

    work page arXiv 2010

  5. [5]

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. http://arxiv.org/abs/2102.08981 Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    arXiv 2021

  6. [6]

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Ca...

    work page 2023

  7. [7]

    https://en.wikipedia.org/wiki/Wikipedia:Copyrights

    Wikipedia Copyright. https://en.wikipedia.org/wiki/Wikipedia:Copyrights

    work page

  8. [8]

    Exploring Dual Encoder Architectures for Question Answering

    Zhe Dong, Jianmo Ni, Daniel M. Bikel, Enrique Alfonseca, Yuan Wang, Chen Qu, and Imed Zitouni. 2022. http://arxiv.org/abs/2204.07120 Exploring dual encoder architectures for question answering

    work page Pith review arXiv 2022

Show all 19 references
  1. [9]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. http://arxiv.org/abs/2010.11929 An image is worth 16x16 words:...

    2021 arXiv

  2. [10]

    Google Cloud . a . A nalyzing E ntities | C loud N atural L anguage A P I . https://cloud.google.com/natural-language/docs/analyzing-entities

  3. [11]

    Google Cloud . b . D etect labels in an image by using the C loud V ision A P I . https://cloud.google.com/vision/docs/detect-labels-image-api

  4. [12]

    Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S

    Xintong Han, Zuxuan Wu, Phoenix X. Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. 2017. Automatic spatially-aware fashion concept discovery. In ICCV

    2017

  5. [13]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural Questions: a Benchmark for Question Answering Research . TACL, 7:453--466

    2019

  6. [14]

    https://github.com/google-research-datasets/wit/blob/main/LICENSE

    WIT License. https://github.com/google-research-datasets/wit/blob/main/LICENSE

  7. [15]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context . In Computer Vision -- ECCV 2014, pages 740--755. Springer International Publishing

    2014

  8. [16]

    Noam Shazeer and Mitchell Stern. 2018. http://arxiv.org/abs/1804.04235 Adafactor: Adaptive learning rates with sublinear memory cost

    2018 arXiv

  9. [17]

    Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning . In Proceedings of the 44th international ACM SIGIR conference on research and development in informa...

    2021

  10. [18]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. http://arxiv.org/abs/2010.11934 mt5: A massively multilingual pre-trained text-to-text transformer

    2021 arXiv

  11. [19]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67--78

    2014

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.