Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RAMQA, a two-stage framework that turns images into text and then has LLaMA re-rank documents and generate answers, reports state-of-the-art scores on WebQA and MultiModalQA.

desk verdict RAMQA is a genuinely useful two-stage recipe with strong reported numbers, but the headline gains are confounded by a 70B second-stage model, and the paper's own appendix shows the zero-shot image-to-text bottleneck that limits its 'true multi-modal' claim. read the letter →

arxiv 2501.13297 v1 pith:MNYCHH5K submitted 2025-01-23 cs.CL cs.AIcs.IRcs.LG

Yang Bai , Christan Earl Grant , Daisy Zhe Wang This is my paper · ORCID
classification cs.CLcs.AIcs.IRcs.LG
keywords multi-modalquestionansweringretrieval-augmentedgenerationlearning-to-rankgenerativerankingLLaVAMAinstructiontuningdocumentpermutation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAMQA is a two-stage framework for multi-modal retrieval-augmented question answering that tries to make decoder-only generative LLMs work as ranking models. The paper's central claim is that a LLaVA-based pointwise ranker, followed by an instruction-tuned LLaMA model that re-ranks the top documents and produces the answer in one pass, outperforms earlier encoder-based retrieval-QA systems. On WebQA the framework reports a QA score of 48.1 against 44.4 for the previous best system, and on the MultiModalQA dev subset it reports an exact-match score of 70.6 against 62.8. The wider significance is that images can be turned into text descriptions once, after which a large generative model can both select evidence and answer questions, avoiding the need for dense document encoders. If these results hold, the recipe offers a practical path for bringing modern LLMs into multi-modal retrieval.

What carries the argument

The central machinery is the data-unification step plus an autoregressive multi-task generator. A zero-shot LLaVA model converts every image into a sentence-level description, so the second stage, RAMLLaMA, only ever sees text; this keeps the LLaMA-3-70B re-ranker within its input-length budget and lets many documents be considered at once. RAMLLaMA is trained by instruction tuning to generate the relevant document IDs and then the answer from a prompt listing the question and up to fifteen numbered candidate documents, and the candidate order is permuted five times per question to suppress position bias. The supporting piece is RankLLaVA, a LLaVA-based pointwise ranker that feeds the query and one document into the model and uses a linear layer on the last token's hidden representation to produce a relevance score, giving the generative stage a strong shortlist to work from.

What would settle it

Construct a test set of questions whose correct answers depend on fine-grained visual details—horn curvature, breed markings, a landmark's arch—and rerun RAMQA twice, once with its zero-shot LLaVA descriptions and once with human-written captions that are known to contain those details. If the framework cannot answer correctly when the decisive cue is missing from the description, the assumption that text-only re-ranking carries the visual evidence is falsified; if it fails even with the cue present, the bottleneck lies elsewhere.

Watch

Extended reading notes

Core claim

The central claim is that the three pieces work together and beat strong baselines in the distractor setting: RankLLaVA, a LLaVA-1.5-7B model fine-tuned with LoRA as a pointwise ranker, scores each query-document pair from the hidden state of the end-of-sequence token; a zero-shot LLaVA model converts each candidate image into a sentence-level description; and RAMLLaMA, an instruction-tuned LLaMA-3-70B model, takes the top fifteen unified documents and is trained to output both the relevant document IDs and the exact answer. The authors argue that the multi-task objective and the five-fold permutation of document order improve both retrieval and answer quality, and that the first stage's access to image pixels makes the pipeline genuinely multi-modal rather than dependent on pre-extracted captions and object detections. The reported WebQA and MultiModalQA numbers are the evidence offered for this claim.

Load-bearing premise

The load-bearing assumption is that the zero-shot LLaVA image-to-text descriptions preserve every visual cue that matters for the answer, because the second stage never sees image pixels; if a caption omits or distorts a decisive detail such as the shape of a horn or the presence of an arch, no amount of re-ranking can recover it.

Editorial extensions

If this is right

  • If the reported results hold, decoder-only generative LLMs can be used as multi-modal rerankers by normalizing images to text, eliminating the need for new encoder-based dense retrievers.
  • Training a single model to generate both relevant document IDs and the answer improves retrieval F1 and QA score together, suggesting that retrieval and reading should be coupled in one objective.
  • Permuting candidate order during training is a cheap data-augmentation that reduces position bias and lifts both retrieval and QA performance, a recipe that transfers to text-only generative reranking.
  • The best input size of about fifteen candidate documents shows a trade-off between recall and noise for generative rerankers, with twenty documents already hurting performance.
  • On MultiModalQA the largest claimed gains are on image questions, pointing to the pixel-level first-stage ranker as the main source of advantage when text evidence is thin.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test is giving the re-ranker a few image-derived visual tokens in addition to the text description, so that cues lost in captioning can still influence the final answer.
  • Because the benchmarks supply positive evidence and hard negatives rather than a full corpus, the reported gains may not carry over to open-domain retrieval; adding a real first-stage retriever over a large collection would test this directly.
  • The permutation trick is a general recipe for generative rerankers trained on ordered candidates, so applying it to tables or audio after modality-to-text unification would be a cheap test of whether the framework generalizes.
  • The paper's own example of confusing two similarly named Washington Square parks suggests a measurable failure mode: questions whose answer depends on a distinctive visual feature present in the image but absent from the unified text description.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RAMQA, a two-stage framework for distractor-setting multimodal retrieval-augmented QA. The first stage trains RankLLaVA, a pointwise ranker initialized from LLaVA-1.5-7B, to score question-document relevance. Before the second stage, all images are converted to text by a zero-shot LLaVA description generator; RAMLLaMA, an instruction-tuned LLaMA-3-70B model, then receives the top-15 unified candidates and is trained to output relevant document IDs and the answer, using five random permutations per question to reduce position bias. The authors report state-of-the-art QA scores on the WebQA test set (48.1 vs. 44.4 for PERQA) and high Exact Match on the MultiModalQA dev subset (70.6 overall), together with ablations showing the importance of the retrieval objective and document permutations.

Significance. If the empirical claims hold, the paper demonstrates a useful recipe: a fine-tuned LLaVA pointwise ranker, text unification of images, and a LLaMA-based multi-task generative re-ranker with document permutations can outperform earlier encoder-based systems on two standard benchmarks. The ablation study is internally consistent, the code and data are promised, and the ablations report three-seed averages. The main limitations are that the headline comparisons are not matched in model scale, the second stage is text-only and inherits all errors of zero-shot image description, and the central results lack variance and significance information. The contribution is solid engineering, but the 'true multimodal' and 'significant improvement' claims need stronger support.

major comments (3)
  1. [§3.3.1–3.3.2, Appendix B, Figure 8] Section 3.3.1 converts every image into text with a zero-shot LLaVA prompt, and Section 3.3.2 defines RAMLLaMA as a LLaMA model that is trained and evaluated on these unified text candidates only; no pixel information from the first-stage ranker is passed downstream. Any decisive visual cue that the description generator omits or distorts is therefore unrecoverable at both the re-ranking and answer-generation levels. This is not a hypothetical concern: Appendix B, Figure 8, provides a concrete error in which RAMQA confuses the Washington Square parks in New York and Chicago because the distinctive New York arch is not present in the text the model reasons over. The Limitations section does not list this information bottleneck. Please (a) add an oracle experiment in which gold or human-written descriptions replace the LLaVA descriptions, (b) quantify how often the final error is attributable to descriptions missing decisive visual evidence, and (c) state the bottleneck as an explicit limitation when claiming 'true multi-modal' advantages.
  2. [§4.4, Tables 2 and 3] The headline comparisons are not matched in model scale. RAMQA uses LLaVA-1.5-7B and LLaMA-3-70B, whereas the published baselines in Tables 2 and 3 use substantially smaller models for retrieval and reranking (BERT, DeBERTa-large, BART-base, OFA-base, T5-base, with mPLUG-Owl for answer generation in PERQA). The reported improvements of 8.3% on WebQA QA and 9.97% over the second-best MultiModalQA All EM are therefore confounded by model scale and by the fact that baselines are not re-run in the same setting. To support the claim that the framework itself is responsible for the gains, please provide matched-scale baselines (e.g., a LLaMA-3-70B-based pointwise or pairwise ranker, or a LLaVA-only end-to-end QA model) and/or controlled ablations that hold the backbone fixed while varying only the proposed components.
  3. [§4.4, §4.5.1, Table 4] Tables 2 and 3 report single numbers without error bars or significance tests; only Table 4 provides mean±std over three random seeds. The conclusions in Section 4.4 use 'significant improvements' in a statistical sense without support. Please report multiple-seed or bootstrap confidence intervals for the headline results, and perform a significance test against the best baseline. In addition, the MultiModalQA results are on the dev subset (Section 4.1.2), so the abstract and the conclusion should explicitly say 'dev set' rather than implying test-set evaluation.
minor comments (5)
  1. [§4.4, Table 3] The sentence 'RAMQA surpasses the second-best PERQA by 9.9% in EM' misidentifies the baseline. In the All column, SKURG (64.2) is second best and PERQA (62.8) is third; the 9.9% figure matches the relative improvement over SKURG, not over PERQA. Please correct the baseline name.
  2. [§4.3] The sentence 'With LoRA, only the linear layer parameters of the LLM were updated' is ambiguous; please clarify whether the added scoring linear layer, the LoRA adapter parameters, or both are trainable.
  3. [§4.4] Describing RAMQA's Retr-F1 as 'on par with the SOTA model PERQA' is imprecise given Table 2 (88.4 vs. 89.6); please phrase this as slightly lower but within unspecified variance, or provide the variance estimate.
  4. [Figure 4] The figure caption should state whether separate models were trained for each #Doc setting. If a single model trained with 15 documents is evaluated at 20, the degradation may reflect train/test mismatch rather than the effect of document count; the caption should also use 'LLaMA-3' instead of 'LLaMA3'.
  5. [§4.5.1] The retrieval F1 threshold is tuned on the WebQA development set; please state this in the implementation details and note explicitly whether the same threshold is applied to the test set results in Table 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RAMQA's central claims are empirical comparisons on held-out benchmark splits, not derivations from their own inputs.

full rationale

The paper's central claim is an empirical result on held-out test/dev splits of WebQA and MultiModalQA, compared against leaderboard and published baselines. RankLLaVA and RAMLLaMA are fine-tuned on training splits and evaluated on held-out data, so the reported QA scores, EM, and retrieval F1 are not fitted quantities renamed as predictions. No load-bearing self-citation appears: the framework builds on external models LLaVA, LLaMA, and RankLLaMA, none of which are the present authors' prior work. The zero-shot LLaVA image-to-text bottleneck is a genuine correctness and robustness concern, and the paper's own Figure 8 documents a failure caused by the model not prioritizing a distinctive visual feature, but this is not circularity: the second stage's dependence on generated descriptions limits the method's ceiling without defining the measured outcome into existence. The dev-set threshold tuning disclosed in Section 4.5.1 is transparent and standard practice, and it does not convert any central prediction into a construction. The derivation chain is therefore self-contained with respect to circularity, even though generalizability and benchmark-specific engineering remain open empirical risks.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on pretrained model checkpoints, benchmark labels, and the image-to-text unification premise. There are no invented entities. The most load-bearing free choices are the dev-tuned evidence threshold, the top-15 candidate count, and the five-permutation training scheme; all affect the reported scores and are chosen using the same benchmarks.

free parameters (3)
  • Evidence selection threshold = not reported (tuned on WebQA dev set)
    Retrieval F1 in the QA-only ablation is computed by thresholding RankLLaVA's binary confidence; the threshold is tuned on the development set (Section 4.5.1), so part of the reported retrieval score is fit to the benchmark.
  • Top-k candidate count for RAMLLaMA = 15
    The paper selects 15 documents for training and testing after inspecting Figure 4 on the same benchmark; this choice directly affects final QA and retrieval scores (Sections 4.3 and 4.5.2).
  • Number of document permutations = 5
    Each training question is permuted five times (Section 3.3.2); no sensitivity study reports whether this count is sufficient or optimal, and the permutation count is a hand-chosen factor in the 5x training set expansion.
assumptions (3)
  • domain assumption Ground-truth evidence and answer labels in WebQA and MultiModalQA are complete and correct.
    All retrieval F1 and QA scores treat the provided labels as the standard; Appendix B itself shows examples where the golden set is incomplete, so metric validity is load-bearing.
  • domain assumption Zero-shot LLaVA image descriptions preserve enough discriminative visual detail for the LLaMA re-ranker to identify evidence and answer correctly.
    Section 3.3.1 converts every image into text before the second stage. If the description drops a decisive visual cue, the re-ranker cannot recover it and the final answer degrades.
  • domain assumption The reported gains are attributable to the RAMQA framework rather than to the much larger base model (LLaMA-3-70B) alone.
    Baselines are taken from prior publications with smaller backbones; no matched-scale control is run, so the causal claim that the framework is effective rests on this attribution assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering." pith.science (2026). https://pith.science/paper/MNYCHH5K

@misc{pith2026250113297,
  author       = {Pith},
  title        = {Pith review of: RAMQA: A Unified Framework for Retrieval-Augmented Multi-Modal Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNYCHH5K}},
  note         = {Machine review of arXiv:2501.13297}
}
read the original abstract

Multi-modal retrieval-augmented Question Answering (MRAQA), integrating text and images, has gained significant attention in information retrieval (IR) and natural language processing (NLP). Traditional ranking methods rely on small encoder-based language models, which are incompatible with modern decoder-based generative large language models (LLMs) that have advanced various NLP tasks. To bridge this gap, we propose RAMQA, a unified framework combining learning-to-rank methods with generative permutation-enhanced ranking techniques. We first train a pointwise multi-modal ranker using LLaVA as the backbone. Then, we apply instruction tuning to train a LLaMA model for re-ranking the top-k documents using an innovative autoregressive multi-task learning approach. Our generative ranking model generates re-ranked document IDs and specific answers from document candidates in various permutations. Experiments on two MRAQA benchmarks, WebQA and MultiModalQA, show significant improvements over strong baselines, highlighting the effectiveness of our approach. Code and data are available at: https://github.com/TonyBY/RAMQA

Figures

Figures reproduced from arXiv: 2501.13297 by the authors.

Figure 1
Figure 1. An example in WebQA (Chang et al., 2021), a Multi-modal Open-domain Question-Answering bench￾mark. This task requires the system to precisely iden￾tify critical sources from distractors and use these key sources to infer the answers. Yang et al., 2023a,b), which are not fully compati￾ble with modern large generative language models. Although recent generative LLMs trained on mas￾sive datasets have dominated NLP task… view at source ↗
Figure 2
Figure 2. RAMQA Framework Overview. A detailed description of the three main components—RankLLaVA, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Zero-shot image description generation for [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Impact of Input Length on RAMLLaMA Performance on the WebQA test set. The horizontal axis(#Doc) represents the number of candidate docu￾ments from RankLLaVA’s output included in RAML￾LaMA’s input prompt during both training and testing. We ensured that the input prompt…
Figure 5
Figure 5. Figure 5: A Training data example of RAMLLaMA. C Scientific Artifacts The licenses for the resources used in this paper are as follows: MultiModalQA (MIT License), We￾bQA (CC0-1.0 License), LLaVA (Llama 2 Com￾munity License), LLaMA (Llama 3 Community License Agreement), and Hugg…
Figure 6
Figure 6. Figure 6: Prediction Examples of RAMQA vs. MuRAG [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: An Example of RAMQA Mispredictions Caused by Incomplete Document Annotations. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: An Example of RAMQA Misprediction Due to Inability to Distinguish Similarly Named Locations [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey of the State-of-the-Art in Conversational Question Answering Systems

    cs.CL 2025-09 conditional novelty 2.0 of 10

    A review that categorizes ConvQA components, techniques, models, and datasets, with no new experimental result.

Reference graph

Works this paper leans on

55 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Qingyao Ai, Xuanhui Wang, Sebastian Bruch, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2019. https://doi.org/10.1145/3341981.3344218 Learning groupwise multivariate scoring functions using deep neural networks . In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '19, page 85–92, New York, NY, US...

    arXiv 2019

  4. [4]

    Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/cd88d62a2063fdaf7ce6f9068fb15dcd-Paper-Conference.pdf Autoregressive search engines: Generating substrings as document identifiers . In Advances in Neural Information Processing Systems, vo...

    work page 2022

  5. [5]

    Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. https://doi.org/10.1145/1273496.1273513 Learning to rank: from pairwise approach to listwise approach . In Proceedings of the 24th International Conference on Machine Learning, ICML '07, page 129–136, New York, NY, USA. Association for Computing Machinery

    arXiv 2007

  6. [6]

    Yingshan Chang, Mridu Baldevraj Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2021. https://api.semanticscholar.org/CorpusID:237385768 Webqa: Multihop and multimodal qa . 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16474--16483

    work page 2021

  7. [7]

    Le, and Christopher D

    Kevin Clark, Minh - Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://openreview.net/forum?id=r1xMH1BtvB ELECTRA: pre-training text encoders as discriminators rather than generators . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net

    2020

  8. [8]

    David Cossock and Tong Zhang. 2006. https://doi.org/10.1007/11776420_44 Subset ranking using regression . In Proceedings of the 19th Annual Conference on Learning Theory, COLT'06, page 605–619, Berlin, Heidelberg. Springer-Verlag

    work page doi:10.1007/11776420_44 2006

Show all 55 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://api.semanticscholar.org/CorpusID:52967399 Bert: Pre-training of deep bidirectional transformers for language understanding . In North American Chapter of the Association for Computational Linguistics

    2019

  2. [10]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://openreview.net/forum?id=YicbFdNTTy An image is worth 1...

    2021

  3. [11]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. https://api.semanticscholar.org/CorpusID:67855846 Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs . In North American Chapter of the Associat...

    2019

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  5. [13]

    Schapire, and Yoram Singer

    Yoav Freund, Raj Iyer, Robert E. Schapire, and Yoram Singer. 2003. An efficient boosting algorithm for combining preferences. J. Mach. Learn. Res., 4(null):933–969

    2003

  6. [14]

    Ross Girshick. 2015. https://doi.org/10.1109/ICCV.2015.169 Fast r-cnn . In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1440--1448

    2015 doi

  7. [15]

    Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, and Maosong Sun. 2019. https://doi.org/10.18653/v1/D19-3029 O pen NRE : An open and extensible toolkit for neural relation extraction . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processi...

    2019 doi

  8. [16]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://openreview.net/forum?id=XPZIaotutsD Deberta: decoding-enhanced bert with disentangled attention . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2...

    2021

  9. [17]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 a . https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  10. [18]

    Hexiang (Frank) Hu, Pat Verga, Wenhu Chen, William Weston Cohen, and Xi Chen. 2022 b . Murag: Multimodal retrieval-augmented generator

    2022

  11. [19]

    Howard, Hartwig Adam, and Dmitry Kalenichenko

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. 2017. https://api.semanticscholar.org/CorpusID:39867659 Quantization and training of neural networks for efficient integer-arithmetic-only inference ....

    2017

  12. [20]

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. https://api.semanticscholar.org/CorpusID:215737187 Dense passage retrieval for open-domain question answering . In Conference on Empirical Methods in Natu...

    2020

  13. [21]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. https://api.semanticscholar.org/CorpusID:204960716 Bart: Denoising sequence-to-sequence pre-training for natural language generation, trans...

    2019

  14. [22]

    Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2023 a . https://doi.org/10.1145/3600088 Parade: Passage representation aggregation fordocument reranking . ACM Trans. Inf. Syst., 42(2)

    2023 doi

  15. [23]

    H. Li. 2011. https://books.google.com/books?id=y080Vc7-iiQC Learning to Rank for Information Retrieval and Natural Language Processing . Online access: Morgan & Claypool Synthesis Collection Five. Morgan & Claypool Publishers

    2011

  16. [24]

    Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.366 Multiview identifiers enhanced generative retrieval . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2023 doi

  17. [25]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296--26306

    2024

  18. [26]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/6dcf277ea32ce3288914faf369fe6de0-Paper-Conference.pdf Visual instruction tuning . In Advances in Neural Information Processing Systems, volume 36, pages 34...

    2023

  19. [27]

    Tie-Yan Liu. 2009. https://doi.org/10.1561/1500000016 Learning to rank for information retrieval . Found. Trends Inf. Retr., 3(3):225–331

    2009 doi

  20. [28]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692

    2019 arXiv

  21. [29]

    Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2019. https://api.semanticscholar.org/CorpusID:208637516 12-in-1: Multi-task vision and language representation learning . 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page...

    2019

  22. [30]

    Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. https://doi.org/10.1145/3626772.3657951 Fine-tuning llama for multi-stage text retrieval . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGI...

    2024

  23. [31]

    Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. 2021. https://doi.org/10.1145/3476415.3476428 Rethinking search: making domain experts out of dilettantes . SIGIR Forum, 55(1)

    2021

  24. [32]

    Rodrigo Nogueira and Kyunghyun Cho. 2019. https://api.semanticscholar.org/CorpusID:58004692 Passage re-ranking with bert . ArXiv, abs/1901.04085

    2019 arXiv

  25. [33]

    Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy J. Lin. 2019. https://api.semanticscholar.org/CorpusID:207758365 Multi-stage document ranking with bert . ArXiv, abs/1910.14424

    2019 arXiv

  26. [34]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  27. [35]

    Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power

    Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. https://doi.org/10.18653/v1/P17-1161 Semi-supervised sequence tagging with bidirectional language models . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ...

    2017 doi

  28. [36]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

    2018 doi

  29. [37]

    Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J

    Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. https://api.semanticscholar.org/CorpusID:204838007 Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach...

    2019

  30. [38]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  31. [39]

    Girshick, and Jian Sun

    Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. https://api.semanticscholar.org/CorpusID:10328909 Faster r-cnn: Towards real-time object detection with region proposal networks . IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:1137--1149

    2015

  32. [40]

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. https://openreview.net/forum?id=ee6W5UgQLa Multimodal \ qa \ : complex question answering over text, tables and images . In International ...

    2021

  33. [41]

    Yubao Tang, Ruqing Zhang, Jiafeng Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2023. https://doi.org/10.1145/3580305.3599903 Semantic-enhanced differentiable search index inspired by learning strategies . In Proceedings of the 29th ACM SIGKDD Con...

    2023

  34. [42]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  35. [43]

    Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W

    Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Don Metzler. 2022. Transformer memory as a differentiable search index. In NeurIPS 2022

    2022

  36. [44]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cant \'o n Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes,...

    2023 arXiv

  37. [45]

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022 a . https://api.semanticscholar.org/CorpusID:246634906 Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learnin...

    2022

  38. [46]

    Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022 b . A neural corpus indexer for document retrieval. In Proceedings of the 36t...

    2022

  39. [47]

    Yiqi Wu, Xiaodan Hu, Ziming Fu, Siling Zhou, and Jiangong Li. 2024. https://api.semanticscholar.org/CorpusID:270521946 Gpt-4o: Visual perception performance of multimodal large language models in piglet activity understanding . ArXiv, abs/2406.09781

    2024 arXiv

  40. [48]

    Qian Yang, Qian Chen, Wen Wang, Baotian Hu, and Min Zhang. 2023 a . https://doi.org/10.1145/3581783.3611964 Enhancing multi-modal multi-hop question answering via structured knowledge and unified retrieval-generation . In Proceedings of the 31st ACM International Conference on...

    2023

  41. [49]

    Shuwen Yang, Anran Wu, Xingjiao Wu, Luwei Xiao, Tianlong Ma, Cheng Jin, and Liang He. 2023 b . https://api.semanticscholar.org/CorpusID:264146942 Progressive evidence refinement for open-domain multimodal retrieval question answering . ArXiv, abs/2310.09696

    2023 arXiv

  42. [50]

    Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. https://doi.org/10.18653/v1/2021.naacl-tutorials.1 Pretrained transformers for text ranking: BERT and beyond . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguist...

    2021 doi

  43. [51]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. https://doi.org/10.48550/ARXIV.2304.14178 mplug-owl: Modularizatio...

    2023 doi arXiv

  44. [52]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://openreview.net/forum?id=5Ya8PbvpZ9 BARTS core: Evaluating generated text as text generation . In Advances in Neural Information Processing Systems

    2021

  45. [53]

    Smucker, Gordon V

    Haotian Zhang, Mustafa Abualsaud, Nimesh Ghelani, Mark D. Smucker, Gordon V. Cormack, and Maura R. Grossman. 2018. https://doi.org/10.1145/3269206.3271796 Effective user interaction for high-recall retrieval: Less is more . In Proceedings of the 27th ACM International Conferen...

    2018

  46. [54]

    Peitian Zhang, Zheng Liu, Yujia Zhou, Zhicheng Dou, Fangchao Liu, and Zhao Cao. 2024. https://doi.org/10.1145/3626772.3657797 Generative retrieval via term set generation . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information...

    2024

  47. [55]

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. https://api.semanticscholar.org/CorpusID:235692795 Vinvl: Revisiting visual representations in vision-language models . 2021 IEEE/CVF Conference on Computer Vision...

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.