MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval

Ho-Joong Kim; Seong-Whan Lee; Yeong-Joon Ju

REVIEW 2 major objections 5 minor 50 references

MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fusion-free retriever that lets text queries attend to image patches while withholding text signals from the visual branch reports higher zero-shot scores than prior fused multimodal retrievers on four benchmarks.

desk verdict A real architectural twist (flipped cross-attention, no text in the trained query vector) backed by a controlled PreFLMR comparison, but with missing error bars, a few gaps in baseline coverage, and one benchmark where the visual component isn't additive. read the letter →

arxiv 2411.08334 v3 pith:F3XHCNJI submitted 2024-11-13 cs.CV cs.AIcs.IRcs.MM

Yeong-Joon Ju , Ho-Joong Kim , Seong-Whan Lee This is my paper · ORCID

classification cs.CVcs.AIcs.IRcs.MM

keywords multimodalretrievallateinteractionvisual-textalignmenttext-dominantissuequery-guidedattentivepoolingresponse-to-passageconversionzero-shotColBERTv2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that multimodal query retrieval suffers from a text-dominant issue: when a visual-text alignment stage lets text features directly influence visual representations, the retriever learns to lean on textual similarity and neglects the image. To counter this, the authors propose MIRe, which lets the textual query attend to patch-level visual embeddings but never feeds text-driven signals back into the visual stream. The two modalities are combined only at scoring time through a late-interaction MaxSim step. With a pretraining dataset built by inserting concise VQA responses into Wikipedia passages, MIRe reports zero-shot gains over prior fused retrievers on OK-VQA, ReMuQ, and E-VQA. The central claim is that fusion-free interaction plus this response-to-passage data recipe yields better multimodal query understanding than direct fusion.

What carries the argument

The load-bearing component is the query-guided attentive pooling module: given textual token embeddings $E_t \in \mathbb{R}^{l_t \times d_t}$ and patch-level visual keys and values $K_m, V_m \in \mathbb{R}^{h \times l_v \times d_t}$, it computes $A = \mathrm{Softmax}(E_t K_m^\top / \sqrt{d_t})$, then returns $E_m = \mathrm{Linear}(\frac{1}{l_t}\sum_i A V_m)$, producing $h$ visual embeddings per image. Because the text embeddings appear only as queries in attention and are never added back into the visual branch, and because $E_t$ is withheld from the query embedding during pretraining, the visual projection must adapt to the linguistic space on its own. The second mechanism is the response-to-passage conversion: simple answer strings are completed with nouns from the question, then a frozen text retriever pulls top-$k$ Wikipedia passages for the response, and the response is spliced between them, creating noisy training passages that force image-text joint understanding rather than answer-snippet matching.

What would settle it

A decisive check is to take the trained MIRe and the same pretraining data, strip the answer sentence out of each synthesized passage (or replace it with a mismatched sentence), and then evaluate zero-shot retrieval on OKVQA-GS and ReMuQ. If recall stays high, the model learned genuine image-text-to-passage matching; if it collapses, the benchmark gains were largely a pasted-answer artifact. A complementary probe is to feed queries whose text is deliberately uninformative (e.g., 'what is shown?') and verify that the visual branch alone still retrieves the right passages.

Watch

Extended reading notes

Core claim

In the authors' own framing, the discovery is that the text-dominant issue in multimodal retrieval can be mitigated by architecture: align the visual embeddings to the text retriever's space while using the textual query only as an attention query over image patches, and exclude textual embeddings from the query representation during alignment. MIRe's query-guided attentive pooling computes attention scores from text token embeddings against projected visual patch keys, mean-pools the resulting visual values, and uses those visual features together with a global image embedding as the query during pretraining. Only after alignment are textual embeddings appended for downstream retrieval, and scoring uses the ColBERT-style late-interaction MaxSim over token embeddings. Across OKVQA-GS, OKVQA-WK11M, ReMuQ, and E-VQA, MIRe reports higher zero-shot MRR and Recall@k than FLMR, ReViz, UniIR, VISTA, and a PreFLMR re-trained with the same data. Ablations show that reintroducing text into alignment (residual connections, concatenating $E_t$) degrades performance, and that the response-to-passage conversion is essential to the gains.

Load-bearing premise

The recipe assumes that inserting the gold response into Wikipedia passages retrieved for that response produces training passages whose relevance can only be determined by combining image and text, and that learning on this synthetic data transfers to real corpora; if the model can instead exploit the pasted answer snippet as a text-only shortcut, the reported gains would come from the data construction rather than from solving the text-dominant issue.

Editorial extensions

If this is right

If MIRe is right, multimodal retrievers can be built by aligning visual embeddings into the text encoder's space while leaving the text encoder untouched, preserving text retrieval strength and adding image grounding without retraining the language backbone.
The ablation showing that feeding $E_t$ into the alignment stage hurts zero-shot transfer suggests that existing token-fusion and cross-attention designs may be leaving multimodal retrieval performance on the table.
The response-to-passage conversion turns VQA and visual-instruction data into retriever training data at scale (1.35M pairs), allowing abundant QA data to be reused for retrieval pretraining.
Because scores are computed by late-interaction MaxSim over token embeddings, the approach inherits ColBERTv2-style indexing and can be plugged into existing late-interaction retrieval stacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own analysis of Infoseek shows that a large part of the gain there comes from adding WiT external knowledge, which the authors flag as a concern about benchmarks relying on background knowledge; a fair reading is that fusion-free alignment helps most where the benchmark tests image-conditioned text understanding rather than purely encyclopedic knowledge.
A natural next test the authors do not run is to replace the frozen text retriever used for response-to-passage conversion with a stronger retriever, and check whether the synthetic passages become harder or easier; if harder passages improve downstream zero-shot recall, the data recipe itself is a trainable component.
The direction suggests a symmetric formulation worth exploring: let image patches attend to text tokens during alignment while withholding visual signals from the text branch, to test whether text-dominance is asymmetric in the other direction.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The thing to know: this paper's central trick is a genuine architectural change. It flips PreFLMR's cross-attention so text tokens attend to visual patches, and it deliberately leaves text embeddings out of the trained query vector during pre-training. That is a real, testable design choice, and the controlled comparison against PreFLMR trained on the identical data is the right kind of experiment. On ReMuQ the gap is huge (83 vs 52 MRR@5), and the ablations back up the mechanism: feeding text embeddings during alignment hurts, and adding a residual connection hurts.

What's also good: the response-to-passage data construction is practical and clearly described, code is released, and the zero-shot evaluations span four benchmarks. The paper earns its claim that the fusion-free alignment is the driver, not just the data.

Where I'd push back: the paper reports no variance or significance numbers, and Table 1 has gaps (no CLIP/ReViz on E-VQA). More substantively, the stress-test note about the R2P data is worth taking seriously, but it lands only on E-VQA. Yes, the text-only inference variant gets 42.48 MRR@5 on E-VQA, and the full model gets 41.88—so visual features actually hurt there. But on ReMuQ, text-only is 71.69 and full is 83.06, so the visual component is strongly additive. The model is text-dominant at inference by design; the paper's claim is about the alignment stage, and the ablations isolate that. Also, the WiT pre-training overlaps with Wikipedia-based evaluation corpora, so a clearer analysis of that contamination is needed.

The paper is not airtight, but the load-bearing argument holds: the architectural choice matters, and the controlled comparison supports it. It deserves a serious referee; the authors should be asked to add error bars, fill the baseline gaps, and run a diagnostic that shows image-only grounding on a subsample.

For you: worth a read for the architecture and the negative result on fusion. I'd cite it if I worked in multimodal retrieval.

Referee Report

2 major / 5 minor

Summary. The paper proposes MIRe, a multimodal retrieval framework that combines a frozen text retriever (ColBERTv2) with a frozen vision encoder (CLIP ViT) via query-guided attentive pooling: textual embeddings attend to visual patch embeddings, but the attended visual representations are aggregated without a residual connection or direct fusion of text features into the visual stream. The authors also construct a pre-training dataset by converting visual question-answer pairs into retrieval passages through a response-to-passage (R2P) conversion, in which the gold response is inserted into Wikipedia passages retrieved by a text retriever. MIRe is evaluated zero-shot and fine-tuned on OK-VQA, ReMuQ, E-VQA, and related benchmarks, reporting strong gains over PreFLMR trained on the same data. The central claims are that the fusion-free interaction mitigates a text-dominant issue and that the R2P data recipe is essential for the gains.

Significance. If the central claims hold, the paper makes a useful contribution: it shows that a late-interaction retriever can be adapted to multimodal queries by attending text to visual tokens while deliberately withholding text features during the visual alignment stage, and it introduces a scalable recipe for constructing multimodal retrieval training data from visual dialogue and VQA sources. The main comparison is well controlled (PreFLMR trained under the same settings), and the ablations isolate the R2P conversion, the WiT component, and individual embedding types. The paper also ships code and reports zero-shot results on four standard benchmarks. The significance is conditional on whether the reported gains actually require image understanding, which is the main open question: the paper's own ablations show that a text-only inference variant already achieves strong performance on ReMuQ and E-VQA, so the evidence that the visual modality drives the improvements is currently incomplete.

major comments (2)

[Section 3.4, Eqs. (5)-(6); Table 3]
[Table 1]

minor comments (5)

[Section 3.3, Eq. (2)]
[Table 3 caption]
[Section 4.3]
[Table 1]
[Section 5, Fig. 5]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are evaluated against external benchmarks and are not definitionally reducible to its inputs.

full rationale

The derivation chain is: (1) R2P data construction uses the external ColBERTv2 retriever to fetch Wikipedia passages for each response and inserts the response into the positive passage (Eqs. 5-6); (2) the model is trained with frozen text and vision encoders via contrastive loss (Eq. 7); (3) zero-shot and fine-tuned results are reported on external benchmarks (OKVQA-GS, OKVQA-WK11M, ReMuQ, E-VQA). No fitted parameter is reused to predict a benchmark score, and no benchmark labels enter the pre-training objective. The text retriever used in R2P is the same model family as the frozen text encoder, but this is a data-generation choice rather than a definitional identification of the predicted quantity with the fitting target; benchmark passages and relevance judgments are independent of the training set construction. The paper's ablations (Table 3) are internally consistent and compare architectural variants under the same data, and the comparison with PreFLMR trained on the same dataset controls for the data recipe. There are no load-bearing self-citations: all cited retrieval backbones (ColBERTv2, PLAID, FLMR/PreFLMR, VISTA) are external prior work. The possible shortcut that positive passages always contain the answer phrase is a legitimate experimental-validity concern about whether the model uses image information, but it does not make the reported benchmark numbers equivalent to the training objective by construction; the central claim remains externally falsifiable. Accordingly, no circular step is exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a set of hand-chosen architecture and data-generation hyperparameters (visual token counts, attention heads, temperature, number of retrieved passages, WiT sample size) and on background assumptions about the frozen ColBERTv2 and CLIP ViT spaces, the validity of the synthetic pre-training passages, and the integrity of the benchmarks. No new physical or mathematical entities are introduced.

free parameters (5)

Number of visual tokens from global embedding (lg) = 16
Set by hand; determines dimension of Eg that contributes to EQ and shapes the visual representation capacity.
Number of attentive pooling heads and tokens (h) = 12
Set by hand; Em has h=12 tokens, affecting the granularity of query-guided visual features.
Temperature tau = 0.3 (pre-training), 0.8 (fine-tuning)
Chosen by hand; controls contrastive loss sharpness and affects embedding calibration.
Number of retrieved passages k in response-to-passage conversion = 3
Set by hand; larger k would make passages noisier; this is a data-generation hyperparameter.
WiT sample size = 0.5M (1.0M in Table 4)
Chosen by hand; the amount of external knowledge data affects performance on Infoseek and, to a lesser extent, other benchmarks.

assumptions (5)

domain assumption ColBERTv2 token embeddings, trained on MS MARCO, provide a suitable fixed linguistic space that visual features can be projected into for late-interaction retrieval.
Section 3.3 and A.1 freeze RT and align visual features to its space; if the spaces are incompatible, the alignment cannot work.
domain assumption CLIP ViT global and patch embeddings retain enough visual information to answer knowledge-based queries.
Vision encoder is frozen; all visual information entering retrieval comes from CLIP ViT features (Section 3.3).
domain assumption The response-to-passage converted QA pairs are a faithful proxy for multimodal query retrieval, with the pasted response acting as a golden span.
Section 3.4 constructs training passages by inserting the gold response into retrieved Wikipedia text; if this makes the retrieval task trivial, the learned alignment would not transfer.
standard math Standard contrastive in-batch negative training with frozen encoders learns meaningful alignment.
Section 3.5 uses in-batch negatives and an InfoNCE-style loss; this is standard practice, but the optimization assumptions are inherited.
domain assumption Benchmark ground truths and knowledge bases (OKVQA-GS, OKVQA-WK11M, ReMuQ, E-VQA) are valid and free of leakage with the WiT pre-training data.
Section 4 uses these to claim zero-shot superiority; if WiT images overlap the WK11M corpus, part of the gain could come from exposure.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval." pith.science (2026). https://pith.science/paper/F3XHCNJI

@misc{pith2026241108334,
  author       = {Pith},
  title        = {Pith review of: MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3XHCNJI}},
  note         = {Machine review of arXiv:2411.08334}
}

read the original abstract

Recent multimodal retrieval methods have endowed text-based retrievers with multimodal capabilities by utilizing pre-training strategies for visual-text alignment. They often directly fuse the two modalities for cross-reference during the alignment to understand multimodal queries. However, existing methods often overlook crucial visual information due to a text-dominant issue, which overly depends on text-driven signals. In this paper, we introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment. Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations. Additionally, we construct a pre-training dataset for multimodal query retrieval by transforming concise question-answer pairs into extended passages. Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries, resulting in strong performance across four multimodal retrieval benchmarks under zero-shot settings. Moreover, our ablation studies and analyses explicitly verify the effectiveness of our framework in mitigating the text-dominant issue. Our code is publicly available: https://github.com/yeongjoonJu/MIRe

Figures

Figures reproduced from arXiv: 2411.08334 by the authors.

**Figure 2.** Overview of the MIRe architecture. This figure illustrates the interaction between the text encoder RT and the vision encoder RV . T and passage D, utilizing multi-vector representations under the late-interaction mechanism. The text encoder generates token-level embeddings Et ∈ R lt×dt , where lt denotes the number of tokens in T and dt represents the embedding dimension. Visual Embeddings. We use ViT (Dosovitsk… view at source ↗

**Figure 3.** Our data construction process. Starting with visual dialogue datasets, our process involves two steps to convert the dialogue tasks to knowledge retrieval tasks. After preprocessing, we transform responses into a passage format by unifying the response and relevant passages retrieved from Wikipedia. for each embedding within Vm. Then, the attended visual output Em ∈ R h×dt is calculated with value vectors Vm ∈ R h×l… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Visualization of multimodal query processing, illustrating the alignment between textual and visual modalities. pooling (w/ Residual), we observe a performance drop, indicating a slight exacerbation of the textdominant issue. Moreover, when text features are allowed…

**Figure 6.** Figure 6: Comparison of Embedding Distribution. (a) with Et during alignment, where query embeddings (Q, orange) remain distinct from passage embeddings (D, blue); (b) our method, where Q (green) is better integrated into the textual space. to 42.61, bringing it on par with FLMR…

**Figure 5.** Figure 5: Training convergence and retrieval performance. All models were trained for only one epoch under the same settings. 5 Discussion Effect of Query-guided Attentive Pooling To demonstrate the effectiveness of MIRe in capturing modality interactions, we visualize the emb…

**Figure 7.** Figure 7: UMAP visualization of embeddings extracted using the Contriever model (Izacard et al., 2022), comparing Wikipedia documents (purple) and LLaVA responses (red). The separation between clusters highlights the structural and semantic differences. Inference. Passages wit…

**Figure 8.** Figure 8: Examples for our dataset. The figure illustrates samples in the dataset, where the red-colored text denotes inserted responses [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 40 canonical work pages

[1]

Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mar c al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291--4301

2019
[2]

D Chen. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051

arXiv 2017
[3]

Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M 3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics (ACL), pages 2318--2335

work page 2024
[4]

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 14948--14968

work page 2023
[5]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics (NAACL) , pages 4171--4186

work page 2019
[6]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR)

work page 2021
[7]

Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. 2022. A thousand words are worth more than a picture: Natural language-centric outside-knowledge visual question answering. arXiv preprint arXiv:2201.05299

work page Pith review arXiv 2022
[8]

Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT : A knowledge augmented transformer for vision-and-language. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 956--968

work page 2022

Show all 50 references

[9]

Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. 2023. Open-domain visual entity recognition: T owards recognizing millions of W ikipedia entities. In Proceedings of the IEEE/CVF International Conference on C...

2023
[10]

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research (TMLR)

2022
[11]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

2020
[12]

Omar Khattab and Matei Zaharia. 2020. ColBERT : Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 39--48

2020
[13]

Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Summarizing retrievals using answer candidates for open-domain QA of LLM s. In The International Conference on Learning Representations (ICLR)

2024
[14]

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 6086--6096

2019
[15]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP : Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning (ICML), pages 12888--12900

2022
[16]

Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. 2023. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems (NeurIPS), 36

2023
[17]

Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne. 2024. P re FLMR : Scaling up fine-grained late-interaction multi-modal retrievers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 5294--5316

2024
[18]

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296--26306

2024
[19]

Siqi Liu, Weixi Feng, Tsu-Jui Fu, Wenhu Chen, and William Wang. 2023. EDIS : Entity-driven image search over multimodal web content. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4877--4894

2023
[20]

Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. 2024. Generative multi-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 18733--18741

2024
[21]

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR)

2019
[22]

Man Luo, Zhiyuan Fang, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2023. End-to-end knowledge retrieval with multi-modal queries. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 8573--8589

2023
[23]

Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. 2021 a . Weakly-supervised visual-retriever-reader for knowledge-based question answering. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6417–--6431

2021
[24]

Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. 2021 b . Weakly-supervised visual-retriever-reader for knowledge-based question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6417--6431

2021
[25]

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-VQA : A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3195--3204

2019
[26]

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, page 861

2018
[27]

Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, Andr \'e Araujo, and Vittorio Ferrari. 2023. Encyclopedic VQA : Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF Internationa...

2023
[28]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO : A human-generated machine reading comprehension dataset

2016
[29]

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Proceedings of the Conference on Empirical Methods in Natural Language Proce...

2022
[30]

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M \"u ller, Joe Penna, and Robin Rombach. 2024. SDXL : Improving latent diffusion models for high-resolution image synthesis. In The International Conference on Learning Representations (ICLR)

2024
[31]

Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W Bruce Croft, and Mohit Iyyer. 2020. Open-retrieval conversational question answering. In Proceedings of the International ACM SIGIR conference on research and development in Information Retrieval (SIGIR), pages 539--548

2020
[32]

Chen Qu, Hamed Zamani, Liu Yang, W Bruce Croft, and Erik Learned-Miller. 2021. Passage retrieval for outside-knowledge visual question answering. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1753--1757

2021
[33]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learni...

2021
[34]

Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , pages 333--389

2009
[35]

Alireza Salemi, Juan Altmayer Pizzorno, and Hamed Zamani. 2023. A symmetric dual encoding dense retrieval framework for knowledge-intensive visual question answering. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval ...

2023
[36]

Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022 a . Plaid: an efficient engine for late interaction retrieval. In Proceedings of the ACM International Conference on Information & Knowledge Management (CIKM), pages 1747--1756

2022
[37]

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022 b . C ol BERT v2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the Conference of the North American Chapter of the Association for Computational...

2022
[38]

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317--8326

2019
[39]

Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. 2021. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information R...

2021
[40]

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER : a large-scale dataset for fact extraction and VER ification. In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics (NAACL) , p...

2018
[41]

Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. 2023. To see is to believe: Prompting GPT-4V for better visual instruction tuning. arXiv preprint arXiv:2311.07574

2023 arXiv
[42]

Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. 2024. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision (ECCV), pages 387--404

2024
[43]

Jialin Wu and Raymond Mooney. 2022. Entity-focused dense passage retrieval for outside-knowledge visual question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8061--8072

2022
[44]

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI), volume 36, pages 3081--3089

2022
[45]

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023. LLaVAR : Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107

2023 arXiv
[46]

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, and Yongping Xiong. 2024 a . VISTA : Visualized text embedding for universal multi-modal retrieval. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 3185--3200

2024
[47]

Tianshuo Zhou, Sen Mei, Xinze Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Ge Yu. 2024 b . MARVEL : Unlocking the multi-modal capability of dense retrieval via visual module plugin. In Proceedings of the Annual Meeting of the Association for Computational Linguisti...

2024
[48]

Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774

2021 arXiv
[49]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[50]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mar c al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291--4301

2019

[2] [2]

D Chen. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051

arXiv 2017

[3] [3]

Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M 3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics (ACL), pages 2318--2335

work page 2024

[4] [4]

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. 2023. Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 14948--14968

work page 2023

[5] [5]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North A merican Chapter of the Association for Computational Linguistics (NAACL) , pages 4171--4186

work page 2019

[6] [6]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR)

work page 2021

[7] [7]

Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. 2022. A thousand words are worth more than a picture: Natural language-centric outside-knowledge visual question answering. arXiv preprint arXiv:2201.05299

work page Pith review arXiv 2022

[8] [8]

Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT : A knowledge augmented transformer for vision-and-language. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 956--968

work page 2022