{"id":"c6335276-655b-425b-9e51-9429a273df90","arxiv_id":"2501.13297","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-stage pipeline, RAMQA, using a LLaVA pointwise ranker and a LLaMA multi-task generative re-ranker with document permutations, achieves state-of-the-art QA and retrieval scores on WebQA and MultiModalQA.","lead":"RAMQA combines two stages for question answering over mixed text and images: a ranker built from a vision-language model, then a re-ranker built from a large language model that outputs both the relevant evidence and the answer. The paper reports large gains over earlier systems on WebQA and MultiModalQA, pointing to a simple recipe: turn images into text, rank candidates, then generate the answer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reader's weakest assumption is the right one: the second stage and the final answer both depend on zero-shot LLaVA descriptions, and Figure 8 shows a real failure when a decisive visual cue is not verbalized.","rationale":"After reading the full manuscript, I find the reader's conditional verdict fair and agree with the identified weakest assumption. The ablation in Table 4 is important evidence: the retrieval-generation objective and permutations move WebQA QA from 40.4 to 47.5, while the QA-only 70B generator reaches only 40.4, lower than PERQA's 44.4. This mitigates the model-scale confound that the reader also flags, because a 70B model alone does not dominate the baselines. The reported empirical gains are therefore plausible, and I do not see an internal inconsistency that would overturn them. The remaining place where the argument is least secure is the image-to-text bottleneck in Section 3.3.1. The second-stage re-ranker and the answer generator are text-only, so the architecture's ceiling on image questions is set by what zero-shot LLaVA chooses to verbalize. The paper's own Figure 8 provides direct evidence that a decisive visual cue, the arch, can be absent from the text the model reasons over, and the Limitations section does not acknowledge this failure mode. An oracle recoverability audit would settle whether this is a rare edge case or a systematic bottleneck. Because the concern does not disprove the benchmark scores but does identify a real, testable modeling premise, I recommend keeping the reader's CONDITIONAL verdict rather than changing it.","tokens_in":18731,"tokens_out":14890,"duration_ms":148813,"concrete_test":"Run an oracle recoverability audit. Sample 100 image questions from WebQA test and 100 from MultiModalQA dev, stratified by image-only versus mixed questions. For each, give a human annotator the exact prompt text RAMLLaMA receives, namely the question, document titles/captions, and the zero-shot LLaVA descriptions generated by Section 3.3.1, without access to the original images. Ask whether the correct answer and the correct evidence IDs are determinable from that text alone. Compute the fraction of questions where the descriptions are insufficient. If that fraction is high (for example, above 10-15 percent of image questions), the second-stage information bottleneck is confirmed and the framework's multi-modal advantage is capped by LLaVA's description quality. If it is low, the concern is mitigated and the conditional verdict can be relaxed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim can hold only if the text-only second stage has access to the visual evidence that actually matters. Section 3.3.1 converts every image into text using a zero-shot LLaVA prompt before RAMLLaMA sees it, and Section 3.3.2 describes RAMLLaMA as a text-only LLaMA model. Consequently, both the re-ranking decision and the final answer are functions of the generated descriptions. The first-stage RankLLaVA in Section 3.2 consumes pixels and can rank relevant images, but it cannot pass pixel-level evidence downstream; if the description omits or distorts a decisive visual cue, no later component can recover it. The paper's own Appendix B, Figure 8, documents exactly this failure mode: RAMQA confuses the Washington Square parks in New York and Chicago, and the authors explain that the model should have prioritized the distinctive arch, a visual feature that is not captured in the text the model reasons over. The Limitations section does not list this information bottleneck. Because the benchmarks include many image questions whose answers hinge on such cues, the 'true multi-modal' advantage asserted in Section 4.4 is upper-bounded by the faithfulness of the zero-shot LLaVA descriptions. This does not make the reported numbers implausible, but it is the most load-bearing modeling premise and it is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAMQA, a two-stage framework for distractor-setting multimodal retrieval-augmented QA. The first stage trains RankLLaVA, a pointwise ranker initialized from LLaVA-1.5-7B, to score question-document relevance. Before the second stage, all images are converted to text by a zero-shot LLaVA description generator; RAMLLaMA, an instruction-tuned LLaMA-3-70B model, then receives the top-15 unified candidates and is trained to output relevant document IDs and the answer, using five random permutations per question to reduce position bias. The authors report state-of-the-art QA scores on the WebQA test set (48.1 vs. 44.4 for PERQA) and high Exact Match on the MultiModalQA dev subset (70.6 overall), together with ablations showing the importance of the retrieval objective and document permutations.","tokens_in":18842,"tokens_out":11287,"duration_ms":106075,"significance":"If the empirical claims hold, the paper demonstrates a useful recipe: a fine-tuned LLaVA pointwise ranker, text unification of images, and a LLaMA-based multi-task generative re-ranker with document permutations can outperform earlier encoder-based systems on two standard benchmarks. The ablation study is internally consistent, the code and data are promised, and the ablations report three-seed averages. The main limitations are that the headline comparisons are not matched in model scale, the second stage is text-only and inherits all errors of zero-shot image description, and the central results lack variance and significance information. The contribution is solid engineering, but the 'true multimodal' and 'significant improvement' claims need stronger support.","major_comments":[{"comment":"Section 3.3.1 converts every image into text with a zero-shot LLaVA prompt, and Section 3.3.2 defines RAMLLaMA as a LLaMA model that is trained and evaluated on these unified text candidates only; no pixel information from the first-stage ranker is passed downstream. Any decisive visual cue that the description generator omits or distorts is therefore unrecoverable at both the re-ranking and answer-generation levels. This is not a hypothetical concern: Appendix B, Figure 8, provides a concrete error in which RAMQA confuses the Washington Square parks in New York and Chicago because the distinctive New York arch is not present in the text the model reasons over. The Limitations section does not list this information bottleneck. Please (a) add an oracle experiment in which gold or human-written descriptions replace the LLaVA descriptions, (b) quantify how often the final error is attributable to descriptions missing decisive visual evidence, and (c) state the bottleneck as an explicit limitation when claiming 'true multi-modal' advantages.","section":"§3.3.1–3.3.2, Appendix B, Figure 8"},{"comment":"The headline comparisons are not matched in model scale. RAMQA uses LLaVA-1.5-7B and LLaMA-3-70B, whereas the published baselines in Tables 2 and 3 use substantially smaller models for retrieval and reranking (BERT, DeBERTa-large, BART-base, OFA-base, T5-base, with mPLUG-Owl for answer generation in PERQA). The reported improvements of 8.3% on WebQA QA and 9.97% over the second-best MultiModalQA All EM are therefore confounded by model scale and by the fact that baselines are not re-run in the same setting. To support the claim that the framework itself is responsible for the gains, please provide matched-scale baselines (e.g., a LLaMA-3-70B-based pointwise or pairwise ranker, or a LLaVA-only end-to-end QA model) and/or controlled ablations that hold the backbone fixed while varying only the proposed components.","section":"§4.4, Tables 2 and 3"},{"comment":"Tables 2 and 3 report single numbers without error bars or significance tests; only Table 4 provides mean±std over three random seeds. The conclusions in Section 4.4 use 'significant improvements' in a statistical sense without support. Please report multiple-seed or bootstrap confidence intervals for the headline results, and perform a significance test against the best baseline. In addition, the MultiModalQA results are on the dev subset (Section 4.1.2), so the abstract and the conclusion should explicitly say 'dev set' rather than implying test-set evaluation.","section":"§4.4, §4.5.1, Table 4"}],"minor_comments":[{"comment":"The sentence 'RAMQA surpasses the second-best PERQA by 9.9% in EM' misidentifies the baseline. In the All column, SKURG (64.2) is second best and PERQA (62.8) is third; the 9.9% figure matches the relative improvement over SKURG, not over PERQA. Please correct the baseline name.","section":"§4.4, Table 3"},{"comment":"The sentence 'With LoRA, only the linear layer parameters of the LLM were updated' is ambiguous; please clarify whether the added scoring linear layer, the LoRA adapter parameters, or both are trainable.","section":"§4.3"},{"comment":"Describing RAMQA's Retr-F1 as 'on par with the SOTA model PERQA' is imprecise given Table 2 (88.4 vs. 89.6); please phrase this as slightly lower but within unspecified variance, or provide the variance estimate.","section":"§4.4"},{"comment":"The figure caption should state whether separate models were trained for each #Doc setting. If a single model trained with 15 documents is evaluated at 20, the degradation may reflect train/test mismatch rather than the effect of document count; the caption should also use 'LLaMA-3' instead of 'LLaMA3'.","section":"Figure 4"},{"comment":"The retrieval F1 threshold is tuned on the WebQA development set; please state this in the implementation details and note explicitly whether the same threshold is applied to the test set results in Table 2.","section":"§4.5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical systems contribution with promising results, but the current support for the central claims is incomplete. The image-to-text bottleneck identified in Section 3.3.1 and demonstrated in Figure 8 is a real architectural limitation that the authors should confront directly, and the lack of matched-scale baselines makes the reported gains difficult to attribute to the proposed framework. I would be willing to see a revised version that adds oracle-description experiments, matched-size comparisons, and proper significance testing. The manuscript is not ready for acceptance in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RAMQA is worth a serious look. It combines a LLaVA pointwise ranker with a LLaMA-3-70B generative re-ranker trained to output relevant document IDs and an answer simultaneously, using five document permutations per question to reduce positional bias. That specific combination is new, and it works: on WebQA it reports 48.1 QA score vs PERQA's 44.4, and on MultiModalQA dev it reports 70.6 EM vs 62.8. The ablations are internally consistent and show that the multi-task objective and permutations each contribute real gains. The paper is clearly written and the authors are honest about several limitations.\n\nThe main soft spot is the baseline comparison. RAMLLaMA is a 70B model; almost every baseline uses a much smaller backbone (BART-base, T5-base, mPLUG-Owl, etc.). Double-digit relative gains are exactly what you'd expect from scale alone. Without matched-scale reruns or at least error bars on the headline numbers, the framework's specific contribution is not cleanly identified. The ablation table has mean±std over three seeds, which is good, but the main tables don't.\n\nThe second soft spot is the information bottleneck the stress-test note identifies. The second stage never sees image pixels. Everything is funneled through zero-shot LLaVA descriptions, and the paper's own Figure 8 shows a failure caused by exactly that: the model confuses two Washington Square parks because the distinctive arch is not in the text it reasons over. The Limitations section does not mention this. It's not fatal, because the first-stage ranker does use pixels and can rank relevant images, but it means the 'true multi-modal' claim in Section 4.4 is overstated. The system is better described as pixel-assisted image-to-text for a text-only re-ranker.\n\nReproducibility is plausible but unverified: code and data link, but no commit hash, and hyperparameters like learning rate and number of epochs are omitted. Minor.\n\nWho's this for? Anyone working on multi-modal retrieval, generative ranking, or QA pipelines. It's a solid system paper with a reproducible recipe and clear ablations, and the limitations that exist are the usual ones for this kind of work. I'd send it to peer review, but I'd ask for matched-scale baselines or error bars on the headline results, and a revised discussion of the image-to-text bottleneck.\n\nMy verdict: engage with it. The central claim is plausible, the framework is useful, and the flaws are fixable in revision.","headline":"RAMQA is a genuinely useful two-stage recipe with strong reported numbers, but the headline gains are confounded by a 70B second-stage model, and the paper's own appendix shows the zero-shot image-to-text bottleneck that limits its 'true multi-modal' claim.","tokens_in":19515,"tokens_out":2136,"would_cite":true,"duration_ms":27254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAMQA, a two-stage framework that turns images into text and then has LLaMA re-rank documents and generate answers, reports state-of-the-art scores on WebQA and MultiModalQA.","keywords":["multi-modal question answering","retrieval-augmented generation","learning-to-rank","generative ranking","LLaVA","LLaMA","instruction tuning","document permutation"],"falsifier":"Construct a test set of questions whose correct answers depend on fine-grained visual details—horn curvature, breed markings, a landmark's arch—and rerun RAMQA twice, once with its zero-shot LLaVA descriptions and once with human-written captions that are known to contain those details. If the framework cannot answer correctly when the decisive cue is missing from the description, the assumption that text-only re-ranking carries the visual evidence is falsified; if it fails even with the cue present, the bottleneck lies elsewhere.","tokens_in":18384,"feed_emoji":"🧠","tokens_out":8915,"duration_ms":81651,"temperature":0.7,"pith_summary":"RAMQA is a two-stage framework for multi-modal retrieval-augmented question answering that tries to make decoder-only generative LLMs work as ranking models. The paper's central claim is that a LLaVA-based pointwise ranker, followed by an instruction-tuned LLaMA model that re-ranks the top documents and produces the answer in one pass, outperforms earlier encoder-based retrieval-QA systems. On WebQA the framework reports a QA score of 48.1 against 44.4 for the previous best system, and on the MultiModalQA dev subset it reports an exact-match score of 70.6 against 62.8. The wider significance is that images can be turned into text descriptions once, after which a large generative model can both select evidence and answer questions, avoiding the need for dense document encoders. If these results hold, the recipe offers a practical path for bringing modern LLMs into multi-modal retrieval.","feed_headline":"RAMQA beats top multimodal QA systems by 8.3 percent","feed_subtitle":"A two-stage pipeline leads on WebQA and MultiModalQA by turning images into text and re-ranking with LLaMA.","key_machinery":"The central machinery is the data-unification step plus an autoregressive multi-task generator. A zero-shot LLaVA model converts every image into a sentence-level description, so the second stage, RAMLLaMA, only ever sees text; this keeps the LLaMA-3-70B re-ranker within its input-length budget and lets many documents be considered at once. RAMLLaMA is trained by instruction tuning to generate the relevant document IDs and then the answer from a prompt listing the question and up to fifteen numbered candidate documents, and the candidate order is permuted five times per question to suppress position bias. The supporting piece is RankLLaVA, a LLaVA-based pointwise ranker that feeds the query and one document into the model and uses a linear layer on the last token's hidden representation to produce a relevance score, giving the generative stage a strong shortlist to work from.","core_discovery":"The central claim is that the three pieces work together and beat strong baselines in the distractor setting: RankLLaVA, a LLaVA-1.5-7B model fine-tuned with LoRA as a pointwise ranker, scores each query-document pair from the hidden state of the end-of-sequence token; a zero-shot LLaVA model converts each candidate image into a sentence-level description; and RAMLLaMA, an instruction-tuned LLaMA-3-70B model, takes the top fifteen unified documents and is trained to output both the relevant document IDs and the exact answer. The authors argue that the multi-task objective and the five-fold permutation of document order improve both retrieval and answer quality, and that the first stage's access to image pixels makes the pipeline genuinely multi-modal rather than dependent on pre-extracted captions and object detections. The reported WebQA and MultiModalQA numbers are the evidence offered for this claim.","pith_inferences":["A natural extension the paper does not test is giving the re-ranker a few image-derived visual tokens in addition to the text description, so that cues lost in captioning can still influence the final answer.","Because the benchmarks supply positive evidence and hard negatives rather than a full corpus, the reported gains may not carry over to open-domain retrieval; adding a real first-stage retriever over a large collection would test this directly.","The permutation trick is a general recipe for generative rerankers trained on ordered candidates, so applying it to tables or audio after modality-to-text unification would be a cheap test of whether the framework generalizes.","The paper's own example of confusing two similarly named Washington Square parks suggests a measurable failure mode: questions whose answer depends on a distinctive visual feature present in the image but absent from the unified text description."],"forward_implications":["If the reported results hold, decoder-only generative LLMs can be used as multi-modal rerankers by normalizing images to text, eliminating the need for new encoder-based dense retrievers.","Training a single model to generate both relevant document IDs and the answer improves retrieval F1 and QA score together, suggesting that retrieval and reading should be coupled in one objective.","Permuting candidate order during training is a cheap data-augmentation that reduces position bias and lifts both retrieval and QA performance, a recipe that transfers to text-only generative reranking.","The best input size of about fifteen candidate documents shows a trade-off between recall and noise for generative rerankers, with twenty documents already hurting performance.","On MultiModalQA the largest claimed gains are on image questions, pointing to the pixel-level first-stage ranker as the main source of advantage when text evidence is thin."],"supporting_citations":[{"why":"Supplies the WebQA benchmark, its distractor setting, and the QA score metrics used for the headline result.","marker":"(Chang et al., 2021)"},{"why":"Supplies the MultiModalQA benchmark and the text/image subset on which the dev-set EM comparison is made.","marker":"(Talmor et al., 2021)"},{"why":"Defines LLaVA, the backbone for RankLLaVA and for zero-shot image-to-text unification.","marker":"(Liu et al., 2023)"},{"why":"Defines the LLaMA architecture that the generative re-ranker is built from.","marker":"(Touvron et al., 2023)"},{"why":"Supplies the LLaMA-3-70B model actually fine-tuned for RAMLLaMA.","marker":"(Dubey et al., 2024)"},{"why":"Supplies the instruction-tuning method used to train RAMLLaMA on the multi-task objective.","marker":"(Ouyang et al., 2022)"},{"why":"PERQA is the previous state-of-the-art baseline whose numbers RAMQA claims to beat on both benchmarks.","marker":"(Yang et al., 2023b)"},{"why":"MuRAG is the strongest prior baseline on MultiModalQA image questions and the source of the text/image subset convention.","marker":"(Hu et al., 2022b)"},{"why":"Defines BARTScore, the fluency component of the WebQA QA score.","marker":"(Yuan et al., 2021)"},{"why":"RankLLaMA is the LLaMA-based text ranker that the paper extends to build the RankLLaVA pointwise ranker.","marker":"(Ma et al., 2024)"}],"fun_headline_variants":["RAMQA: LLaVA ranker + LLaMA re-ranker beat WebQA and MultiModalQA","Two-stage RAMQA: LLaVA ranks, LLaMA re-ranks, tops WebQA","RAMQA's LLaVA-LLaMA pair beats multimodal QA by 8.3%","RAMQA lifts multimodal QA with LLaVA pointwise rank and LLaMA re-rank","From pixels to answers: RAMQA's two-stage retrieval wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the zero-shot LLaVA image-to-text descriptions preserve every visual cue that matters for the answer, because the second stage never sees image pixels; if a caption omits or distorts a decisive detail such as the shape of a horn or the presence of an arch, no amount of re-ranking can recover it.","fun_headline_variants_meta":{"raw":{"variants":["RAMQA: LLaVA ranker + LLaMA re-ranker beat WebQA and MultiModalQA","Two-stage RAMQA: LLaVA ranks, LLaMA re-ranks, tops WebQA","RAMQA's LLaVA-LLaMA pair beats multimodal QA by 8.3%","RAMQA lifts multimodal QA with LLaVA pointwise rank and LLaMA re-rank","From pixels to answers: RAMQA's two-stage retrieval wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001108,"raw_usage":{"total_tokens":4608,"prompt_tokens":925,"completion_tokens":3683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":3563}},"tokens_in":541,"tokens_out":3683,"duration_ms":28399,"temperature":1.0,"reasoning_tokens":3563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:17:37.728136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set of questions whose correct answers depend on fine-grained visual details—horn curvature, breed markings, a landmark's arch—and rerun RAMQA twice, once with its zero-shot LLaVA descriptions and once with human-written captions that are known to contain those details. If the framework cannot answer correctly when the decisive cue is missing from the description, the assumption that text-only re-ranking carries the visual evidence is falsified; if it fails even with the cue present, the bottleneck lies elsewhere.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the WebQA benchmark, its distractor setting, and the QA score metrics used for the headline result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MultiModalQA benchmark and the text/image subset on which the dev-set EM comparison is made."}],"review_version":1}