{"id":"d5ffac67-42f1-4db5-8553-f11bd1fe1228","arxiv_id":"2412.10151","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper presents a 300-sample, three-language benchmark with two useful, two partial, and one irrelevant passage per query, together with a training set that improved passage selection in VLMs.","lead":"This paper introduces VLR-Bench, a multilingual visual question answering benchmark in which each question is paired with five passages, only two of which contain the information needed to answer correctly. It also releases VLR-IF, a training set of 32,000 examples that improved passage selection and answer quality in the evaluated vision language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 9's passage-selection gain may reflect instruction/format compliance rather than improved RAG selection; the central claim lacks a format-matched control.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper should not be accepted as-is. The reader's weakest assumption concerns the with-passages vs. without-passages comparison used to claim that VLR-Bench requires external knowledge; that is a real confound but it supports the benchmark's validity claim. The most load-bearing epistemic link for the paper's central contribution is Table 9, the only direct quantitative evidence that VLR-IF trains passage-selection ability. That table lacks a format-matched baseline, so the reported 26-point EM improvement could be an artifact of teaching the model to follow an instruction (output two passage indices) rather than teaching it to identify relevant passages. The VLR-IF training data explicitly teaches valid/invalid passage discrimination, and the evaluation prompt itself asks for a selection, so the comparison is not symmetric. The InfoSeek result provides some external validity, but it is a single run with an unresolved retriever inconsistency and a small absolute gain. These issues do not make the resource unusable; the five-passage design is a genuine structural improvement, and the public release is a positive contribution. But the central claim needs stronger, confound-controlled evidence. The reader's concern and mine are adjacent: both target uncontrolled experimental design, but mine is more central to the claimed training benefit. Thus the conditional verdict stands, and I would not change it without additional experiments.","tokens_in":21582,"tokens_out":4095,"duration_ms":45976,"concrete_test":"Run a controlled passage-selection experiment on VLR-Bench with three conditions: (i) baseline LLaVA-Llama-3 given a few-shot or format-forced prompt that explicitly requests exactly two passage indices; (ii) a control model fine-tuned on an instruction-following dataset that contains no valid/invalid passage pairs but uses the same output template; (iii) the VLR-IF model. Report EM/F1 over at least three seeds with confidence intervals. If the VLR-IF gain over the format-matched baseline falls below roughly 10 EM points, the improvement is substantially format compliance, not RAG selection ability. Also re-run Table 7 with a single specified retriever and multiple runs to confirm the 1.6-point InfoSeek gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core claim—that VLR-IF improves the ability to determine which passages are useful—rests primarily on Table 9 (Section D.2), where EM/F1 for passage selection jump from 2.0/15.9 to 28.0/41.0. The baseline LLaVA-Llama-3 was not trained to output passage selections, while the VLR-IF model was trained on instruction-following data that explicitly contains valid/invalid passage distinctions. The reported gain could therefore be dominated by the model learning to produce the required output format (e.g., 'passage 1, passage 3') rather than by improved relevance judgment. The paper does not specify the selection prompt, how EM/F1 are computed, whether both models receive the same structured-output instruction, or how the baseline was expected to solve the task. No confidence intervals or multiple runs are reported for the 300-item test set. In addition, both VLR-Bench and VLR-IF passages are GPT-4o generated, so a model could exploit generation-style surface cues. The InfoSeek transfer result (Table 7, 42.9->44.5) is external evidence, but it is a single run, and the caption ('search engine as a passage retriever') conflicts with the text ('Oracle was used as the Retriever model'). The central claim would require showing that the Table 9 gain survives a format-matched control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VLR-Bench, a multilingual (English/Chinese/Korean) visual question answering benchmark for retrieval-augmented generation (RAG), comprising 300 image-query-answer instances, each paired with five passages (two gold, two silver, one bronze). It also introduces VLR-IF, a 32K-example instruction-following dataset designed to teach vision-language models to select and use relevant passages. Experiments evaluate several open VLMs and GPT-4o with and without passages, report that VLR-IF training improves performance on VLR-Bench and transfers to InfoSeek, and include a passage-selection experiment in which EM/F1 rise from 2.0/15.9 to 28.0/41.0 after VLR-IF fine-tuning.","tokens_in":21847,"tokens_out":6951,"duration_ms":65049,"significance":"If the results hold, VLR-Bench fills a genuine gap: existing external-knowledge VQA benchmarks provide a single gold passage, whereas real RAG systems present top-k results with distractors. The five-passage design with graded gold/silver/bronze relevance is a useful evaluation instrument, and the multilingual parallel construction is a valuable addition. The release of the benchmark and training data, and the positive InfoSeek transfer, are concrete strengths. However, the main evidence for the central claim that VLR-IF improves passage selection rests on a single experiment (Table 9) that lacks a format-matched control, and other headline numbers come from single runs on 300 items without confidence intervals. The validity correlation (Table 3) is partly by construction because GPT-4o generates both the gold passages and the ground-truth answers. The contribution is promising, but the evaluation currently overclaims the strength of the evidence.","major_comments":[{"comment":"","section":"§4.1, Table 1"},{"comment":"","section":"§D.2, Table 9"},{"comment":"","section":"§3.1, Table 3"},{"comment":"","section":"§4, Tables 1 and 7"}],"minor_comments":[{"comment":"","section":"Table 7 caption"},{"comment":"","section":"Figure 12"},{"comment":"","section":"Table 10"},{"comment":"","section":"§3.1"},{"comment":"","section":"§3.2"},{"comment":"","section":"§D.1, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a timely and potentially useful contribution, and the InfoSeek transfer is encouraging. The main risk is that the central claim about VLR-IF improving passage selection rests on a single, format-confounded experiment (Table 9). If the authors can add a format-matched control and multiple runs, the paper would be substantially stronger. I would not reject on the current evidence, but the evaluation needs to be tightened before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is real: VLR-Bench gives each query five passages (two gold, two silver, one bronze), so a model must select what is useful rather than trust a single retrieved gold passage. That is a genuine departure from InfoSeek and Encyclopedic VQA, and the multilingual parallel construction (EN/ZH/KO) is a practical asset. The data is public, the annotation process is described in reasonable detail, and the InfoSeek transfer result (42.9 to 44.5) is at least external evidence that VLR-IF does something beyond fitting the benchmark's own distribution.\n\nThe soft spots are in proportion. Table 9's passage-selection jump (EM 2.0 to 28.0) is the strongest evidence for VLR-IF, but the baseline LLaVA-Llama-3 was presumably never asked to output passage selections, while the VLR-IF model was trained to do exactly that. Without a format-matched baseline—same instruction, same output schema, no VLR-IF training—the gain could be mostly the model learning to produce 'passage 1, passage 3' rather than learning to judge relevance. That is a load-bearing concern, and the paper does not address it. The 300-item test set, single runs, and no confidence intervals also make the headline numbers fragile. The validity check (with vs. without passages) assumes the performance gap measures external knowledge need, but prompt format and response length are confounded; the paper itself concedes KMS is length-biased, and the R-2 drop for X-LLaVA (37.72%) is reported without control conditions. GPT-4o generates the questions, passages, and training data, and also serves as judge, so distribution matching is a real worry; the InfoSeek transfer mitigates but does not eliminate it.\n\nMinor issues: Table 7's caption says 'search engine as a passage retriever' while the text says Oracle was used, and Table 10's Qualitative/Quantitative columns appear mislabeled. These are fixable.\n\nWho this is for: anyone building or evaluating RAG-capable VLMs, especially in multilingual settings. The benchmark is a useful resource even if the training-data claim needs tightening. I would send it to review; a good referee can push for the format-matched control and multi-run evidence. The paper is honest about its limitations, and the central benchmark contribution is solid enough to warrant that effort.","headline":"A genuinely useful small benchmark with a five-passage design that deserves referee time, but the paper's central claim about VLR-IF's effect on passage selection lacks a format-matched control.","tokens_in":22421,"tokens_out":1077,"would_cite":true,"duration_ms":13320,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a five-passage benchmark for vision-language RAG and a training set that improves passage selection.","keywords":["visual question answering","retrieval augmented generation","benchmark dataset","passage selection","multilingual evaluation","instruction following","external knowledge","vision-language models"],"falsifier":"Run a controlled comparison where the same image-question pairs are answered with no passages but with explicit instructions to produce an equally long, descriptive answer; if the gap between the with-passages and without-passages conditions collapses, the benchmark is measuring response-format effects rather than a need for external knowledge.","tokens_in":21385,"feed_emoji":"🎯","tokens_out":7931,"duration_ms":74380,"temperature":0.7,"pith_summary":"This paper proposes VLR-Bench, a 300-sample multilingual visual question answering benchmark for retrieval-augmented generation, in which each query arrives with five passages: two gold, two silver, and one bronze, so that only two of the five contain information needed to answer. It argues that this tests a capability missing from earlier external-knowledge VQA datasets, which provide a single gold passage: deciding which retrieved passage is actually useful. The paper also presents VLR-IF, 32,000 automatically generated instruction-following examples per language, built by pairing valid passages with invalid passages sampled from other examples, and reports that fine-tuning on VLR-IF improves both passage selection and answer quality. The benchmark is constructed as a parallel corpus in English, Chinese, and Korean, with human review of questions, passages, answers, and keywords.","feed_headline":"Five passages per query test if AI picks the right knowledge","feed_subtitle":"Models can learn to ignore wrong retrieved passages, the core skill real RAG systems need.","key_machinery":"The load-bearing object is the five-passage structure: each VLR-Bench item contains two gold passages that directly support the answer, two silver passages that are topic-related but not central, and one bronze passage that is unrelated, together with two manually selected keywords used for keyword-matching scoring alongside ROUGE, BLEU, and BERTScore. For training, VLR-IF is built by taking an image, generating one valid passage, and randomly drawing invalid passages from other examples to create four contrastive patterns: valid only, invalid only, valid plus one invalid, and valid plus two invalid. This contrastive design is what teaches the model both to use the useful passage and to refuse when no useful passage is present.","core_discovery":"On the paper's own terms, the central discovery is that passage selection for RAG can be isolated as a measurable skill and improved by training. In an English passage-selection task, fine-tuning on VLR-IF raises exact match from 2.0 to 28.0 and F1 from 15.9 to 41.0, and the same training lifts performance on the InfoSeek benchmark from 42.9 to 44.5. Correlation analysis shows that gold passages overlap with ground-truth answers much more than silver or bronze passages, while no passage type is more correlated with the question alone, which the paper reads as evidence that both the image and the passages are needed to answer correctly. The paper also reports that removing the passages drops ROUGE-2 sharply, averaging 37.72% for one English model, and takes this as evidence that the benchmark requires external knowledge.","pith_inferences":["The gold, silver, and bronze passage types could be reused as an automatic evaluation protocol for retrievers: a good retriever should surface gold passages before silver or bronze ones, giving a graded measure of retrieval quality without new annotation.","The VLR-IF construction recipe, which draws invalid passages randomly from other examples, is cheap enough to extend to additional languages and domains, and its refusal behavior when only invalid passages are present is a testable way to measure hallucination resistance.","The observed flat correlation between passage types and questions suggests that passage selection cannot be solved from text alone, which in turn predicts that models with stronger visual grounding should gain more from gold passages than models that rely mostly on language priors.","One could probe whether the reported score gaps reflect true knowledge use by holding answer length and prompt format constant while varying only the content of the passages, a control the paper does not report."],"forward_implications":["If the benchmark measures what it claims, RAG evaluation for vision-language models should include distractor passages rather than a single gold passage, because real retrieval returns a mix of useful and irrelevant results.","Fine-tuning on contrastively constructed valid/invalid passage data measurably improves passage selection, with exact match rising from 2.0 to 28.0 and F1 from 15.9 to 41.0 on the paper's selection task.","The same training transfers to an existing external-knowledge VQA benchmark, raising InfoSeek performance from 42.9 to 44.5, so the effect is not confined to the proposed benchmark's format.","The parallel English, Chinese, and Korean construction allows the same underlying questions to be compared across languages, which is useful for studying how passage selection behaves in multilingual RAG.","The finding that removing passages sharply lowers scores, with one English model dropping an average of 37.72% in ROUGE-2, supports the benchmark's design goal of requiring external knowledge."],"supporting_citations":[{"why":"Defines the external-knowledge visual question answering task that VLR-Bench extends and evaluates against.","marker":"Marino et al., 2019"},{"why":"Identifies the challenge of deciding whether retrieved documents are useful, which is the motivation for the passage-selection design.","marker":"Lin and Byrne, 2022"},{"why":"InfoSeek serves as a single-gold-passage benchmark and as the external dataset used to test transfer of VLR-IF training.","marker":"Chen et al., 2023"},{"why":"Encyclopedic VQA is the structurally closest prior benchmark, providing a single gold passage that VLR-Bench contrasts with its five-passage setup.","marker":"Mensink et al., 2023"},{"why":"BOK-VQA supplies the 150 open-world images used in VLR-Bench and the annotation style for long-form answers.","marker":"Kim et al., 2024"},{"why":"Provides the GPT-based instruction-following data generation method that VLR-IF adapts for constructing queries, answers, and passages.","marker":"Liu et al., 2024"},{"why":"Dense passage retrieval is the retrieval approach that motivates evaluating how VLMs handle top-k retrieved passages.","marker":"Karpukhin et al., 2020"},{"why":"Defines retrieval-augmented generation, the framework in which VLR-Bench and VLR-IF are situated.","marker":"Lewis et al., 2021"}],"fun_headline_variants":["Benchmark tests if vision-language AI picks the right passage","New dataset teaches AI to ignore wrong retrieved passages","VLR-Bench: 32K examples to boost RAG passage selection","Passage selection for VQA: new benchmark and training set"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the score drop when passages are withheld reflects the absence of external knowledge, not a change in prompt format or in the length and phrasing of the responses the models produce.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark tests if vision-language AI picks the right passage","New dataset teaches AI to ignore wrong retrieved passages","VLR-Bench: 32K examples to boost RAG passage selection","Passage selection for VQA: new benchmark and training set"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2827,"prompt_tokens":891,"completion_tokens":1936,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1866}},"tokens_in":507,"tokens_out":1936,"duration_ms":12422,"temperature":1.0,"reasoning_tokens":1866,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:17:07.085080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison where the same image-question pairs are answered with no passages but with explicit instructions to produce an equally long, descriptive answer; if the gap between the with-passages and without-passages conditions collapses, the benchmark is measuring response-format effects rather than a need for external knowledge.","supporting_citations":[{"cited_title":"In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497– 2511, Dublin, Ireland","cited_arxiv_id":null,"evidence_quote":"Identifies the challenge of deciding whether retrieved documents are useful, which is the motivation for the passage-selection design."}],"review_version":1}