{"id":"49b86b4c-1155-4ea4-8e3a-ef2269bdd897","arxiv_id":"2411.12287","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-stage multimodal RAG pipeline with intent refinement, API selection, and relevance/safety filtering reports higher win rates than baseline MLLMs and near-oracle accuracy on Encyclopedic VQA.","lead":"CUE-M is a multimodal search pipeline that first describes or tags an uploaded image, then uses an LLM to refine the user's intent, generate extra search queries, pick external APIs such as shopping or maps, and filter irrelevant or unsafe documents before an MLLM writes the final answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's ablation win rates are unverifiable unless the 2.1k Knowledge-iN queries are shown to be disjoint from the automatic-prompt-tuning set; otherwise 0.639 is a fitted value, not a prediction.","rationale":"The paper's central claim is an engineering claim: that CUE-M's multi-stage pipeline materially improves multimodal search. The public Encyclopedic-VQA result (0.762 with Qwen2.5-7B-Instruct) is strong and is not undermined by the Knowledge-iN split question; it shows that the pipeline generalizes to a public benchmark. However, the real-world Knowledge-iN study is the only evidence for the contribution of individual components (Intention Refiner, Relevance Classifier, Query Generator). Those component claims rest entirely on Table 1, whose win rates are computed with GPT-4o comparisons on a dataset that may also have been used to tune the prompts. The manuscript states that automatic prompt tuning selects final prompts on validation samples, but it does not say whether the 2.1k evaluation queries are part of that validation distribution. Without a documented disjoint split, the reported win rates cannot be interpreted as predictive. The reader's conditional verdict is appropriate: the concern is addressable by disclosing or creating a proper split, and it does not by itself invalidate the public benchmark result. I therefore recommend no change to the reader's conditional verdict, but the split disclosure should be a condition of acceptance.","tokens_in":18298,"tokens_out":10570,"duration_ms":112064,"concrete_test":"Require release of the 2.1k Knowledge-iN query IDs. Randomly split them into 80% tuning/validation and 20% held-out test; run the Sec. 3.7 automatic prompt tuner on the 80% only, freeze all prompts, and recompute the Table 1 win rates on the 20%. If the full-pipeline win rate falls substantially below 0.639 or the component gaps shrink beyond the reported Monte Carlo errors, the reported results are fitted. To address selection bias, additionally compute the win rate on a random unselected sample of multimodal Knowledge-iN posts with the same frozen prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing concern is the evaluation protocol for the real-world Knowledge-iN study (Sec. 4.1). The pipeline's prompts are automatically tuned (Sec. 3.7) using a 'curated dataset derived from user query data', with final prompts selected on validation samples (App. E). The paper never documents a split between the 2.1k queries evaluated in Table 1 and the tuning/validation set. If the evaluation queries overlap with prompt tuning, the full-pipeline win rate (0.639) and the component increments (e.g., removal of Intention Refiner drops win rate to 0.595, removal of Relevance Classifier to 0.547) are in-sample fitted numbers rather than predictive estimates. This matters because the ablation table is the primary evidence that each stage of the claimed novel pipeline contributes. A related selection issue is that the eval set was GPT-4o-filtered into Web/Shopping/Maps categories (App. F) that match the pipeline's available APIs, so the win rate is measured on a curated distribution, not the general multimodal query distribution. The public Encyclopedic-VQA result is not affected by this split issue, but it does not validate the component-level claims. This is a protocol gap, not an allegation of misconduct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CUE-M, a multi-stage multimodal retrieval-augmented generation pipeline that combines image enrichment, intention refinement, contextual query generation, external API selection, relevance-based document filtering, and a layered safety filter. The authors evaluate the system on a curated Naver Knowledge-iN dataset, on Encyclopedic-VQA, and on MM-SafetyBench, reporting that CUE-M reaches a 0.639 win rate over a baseline MLLM on the curated set, 0.762 accuracy with Qwen2.5-7B-Instruct on Encyclopedic-VQA (close to the 0.870 oracle), and an average ASR of 0.43 on MM-SafetyBench. The central claim is that each pipeline stage contributes distinct benefits and that the full system outperforms prior knowledge-augmented multimodal methods without additional training.","tokens_in":18598,"tokens_out":4355,"duration_ms":46572,"significance":"If the reported numbers are predictive, CUE-M is a meaningful engineering contribution: it demonstrates that a modular, search-API-based RAG pipeline can substantially improve knowledge-based multimodal question answering, and the public Encyclopedic-VQA result (Table 2) is notably strong for a zero-/few-shot system. The paper is also unusually transparent about sub-module prompts, failure cases, and the limitations of its own metrics (App. C and App. D). However, the central real-world claim rests on a win-rate evaluation whose tuning/evaluation split is not documented, the evaluation set is curated into API-aligned categories by the same model family that judges the responses, and the safety comparison lacks uncertainty quantification. These concerns are fixable but currently limit the strength of the paper's conclusions.","major_comments":[{"comment":"The paper never states whether the 2.1k Knowledge-iN queries used for the win-rate table are disjoint from the validation samples on which automatic prompt tuning selects final prompts. Since App. E says 'the final prompt is selected after iterative tuning, based on its performance on validation samples,' and Sec. 3.7 says the tuning dataset is 'derived from user query data,' the reported win rates (0.639 full, 0.595 without Intention Refiner, 0.547 without Relevance Classifier) would be in-sample fitted values if the splits overlap. This is load-bearing because Table 1 is the primary evidence that each pipeline component contributes. Please specify the exact split between prompt-tuning validation and final evaluation, and if no disjoint split exists, re-run the ablations on a held-out test set.","section":"Sec. 4.1, Table 1; Sec. 3.7; App. E"},{"comment":"The evaluation set was filtered by GPT-4o into Web, Shopping, and Maps categories that align with the pipeline's available APIs (App. F, Table 14). This makes the win rate a measure on a curated subset of multimodal queries rather than on the general multimodal query distribution. Additionally, GPT-4o is used both to curate the data and to judge the win-rate comparisons, creating a risk of systematic preference for pipeline-style answers. Please report agreement between GPT-4o curation and human raters, provide the category distribution of the unfiltered query pool, and include a human-agreement study for the GPT-4o judgments, ideally with a second judge.","section":"Sec. 4.1; App. F"},{"comment":"The safety evaluation reports point estimates of attack success rate over six scenarios with no confidence intervals or significance tests. Comparisons such as 0.36 vs. 0.38 for the PO scenario are within likely sampling noise, and the scenarios were selected ('six relevant scenarios') rather than using the full benchmark. Please report the number of trials per scenario, compute confidence intervals, and either justify the scenario selection or evaluate on the full MM-SafetyBench split.","section":"Sec. 4.3, Table 3"},{"comment":"The headline comparison against EchoSight and Wiki-LLaVA mixes backbone models, retrieval APIs, and evaluation protocols. CUE-M uses Google Lens plus Google Search restricted to Wikipedia, while EchoSight uses BLIP-2 + Mistral-7B and Wiki-LLaVA uses LLaVA-1.5 with a different hierarchical retrieval scheme; the 'Oracle' rows use ground-truth entities and are not directly comparable zero-/few-shot numbers. Please state the exact retrieval configuration and prompting protocol for every row, report any variance across retrieval runs, and soften the 'state-of-the-art' claim unless the comparison is made under matched conditions.","section":"Sec. 4.2, Table 2 and Table 15"},{"comment":"The win-rate metric is the only headline result for the Knowledge-iN study, and it is defined relative to a baseline MLLM response rather than as an absolute quality measure. The paper should report absolute agreement with the reference answers or include a secondary metric on the same queries; App. C provides ROUGE-L, BERTScore, and NER, but these are presented as auxiliary and are not tied to the win-rate evaluation in Table 1.","section":"Sec. 4.1, Sec. 4.2"}],"minor_comments":[{"comment":"'real-word datasets' should be 'real-world datasets'.","section":"Abstract"},{"comment":"'Vanlia' should be 'Vanilla' in both tables.","section":"Tables 2 and 15"},{"comment":"The prompt templates contain typographical errors such as 'docummnt_summaries' and 'usre_query'; please correct them.","section":"Tables 4-7"},{"comment":"The BERTScore discussion correctly notes a known limitation, but the paragraph should also state whether the differences in Table 11 are statistically significant or merely descriptive.","section":"App. C"},{"comment":"The 'Internal' category is defined by an 'out_of_scope_questions' placeholder; please either show the actual examples or explain that they are masked for policy reasons.","section":"Table 14"},{"comment":"The main text says safety performance is 'comparable,' while App. D documents systematic jailbreak vulnerabilities in multimodal retrieval; please reconcile these statements in the conclusion or abstract so the claims match the evidence.","section":"App. D and Sec. 4.3"},{"comment":"The Maps category is notably small (138 samples) relative to Web and Shopping; the paper should report per-category win rates or note the imbalance as a limitation of the curated set.","section":"Table 13"}],"recommendation":"major_revision","confidential_remarks":"The main concern is methodological transparency rather than evidence of misconduct. I would ask for explicit documentation of the tuning/evaluation split and a human-agreement study for the GPT-4o judgments before considering acceptance. The paper appears to come from an industrial lab (Naver), and internal policy may prevent releasing prompts and user logs; nevertheless, the split can be documented without releasing user data, and confidence intervals can be computed from public benchmarks. The Encyclopedic-VQA result is promising and should be preserved; the revision should focus on making the real-world evaluation protocol credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CUE-M is a serious systems paper. The pipeline—image enrichment, intention refinement, query generation, API selection, relevance filtering, safety filters—is coherent and the integration is genuinely new relative to the cited prior work. The strongest result is on Encyclopedic VQA: 0.762 accuracy with a 7B model, no ground-truth entities, close to the 0.870 oracle. That is a real data point for zero-shot multimodal RAG, and it holds up better under scrutiny than the rest of the paper.\n\nThe soft spot is the real-world evaluation in Sec. 4.1. The authors automatically tune prompts on a curated dataset derived from user query data, and App. E says the final prompt is selected on validation samples. But they never say the 2.1k Knowledge-iN queries evaluated in Table 1 are disjoint from that tuning/validation set. If they aren't, the win rates—0.639 full pipeline, and the ablation deltas—are in-sample fitted numbers, not predictions. That is a load-bearing gap because the ablations are the primary evidence that each pipeline stage contributes. The eval set is also GPT-4o-filtered into Web/Shopping/Maps categories, and GPT-4o serves as both curator and judge, so there is a risk of fitting to the judge's preferences. These are protocol gaps, not evidence of misconduct, but they need to be addressed before the component-level claims are credible.\n\nOther issues are minor by comparison: no error bars on the VQA or safety numbers, no code or data release, and the safety evaluation cherry-picks six scenarios that align with the pipeline's filters (though the authors acknowledge remaining jailbreaks). The public VQA comparison is reasonably fair—they show CUE-M adds value over raw Google Lens retrieval—but the lack of released artifacts makes independent verification hard.\n\nAll that said, the paper is honest and the design is thoughtful. The authors flag their own limitations in App. D, and the metrics discussion in App. C shows they thought about evaluation issues. This deserves a serious referee, not a desk reject. If I were handling it, I'd ask for a clear statement on the train/eval split, error bars, and a commitment to release code and data. Without that, the Knowledge-iN numbers are suggestive but not convincing.","headline":"Solid multimodal RAG system with a strong public-benchmark result; the real-world ablation numbers hinge on an undisclosed split between prompt tuning and evaluation.","tokens_in":19097,"tokens_out":3324,"would_cite":true,"duration_ms":32319,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims CUE-M, a five-stage multimodal search pipeline with intent refinement and relevance filtering, lifts knowledge-based visual question answering on Encyclopedic VQA to 0.762 accuracy with a 7B model, near the 0.870 oracle…","keywords":["multimodal retrieval-augmented generation","multimodal large language model","knowledge-based visual question answering","intent refinement","relevance filtering","safety evaluation","Encyclopedic VQA","MM-SafetyBench"],"falsifier":"A concrete check: split the 2.1k Knowledge-iN queries into disjoint prompt-tuning and held-out test sets, run CUE-M against the no-retrieval baseline on the held-out set, and have human raters judge the responses; if the win rate falls from 0.639 toward 0.5 or disagrees with GPT-4o on the same pairs, the pipeline's claimed effectiveness does not generalize. A second check on Encyclopedic VQA: swap the external similar-image and text-search APIs for an equally capable open retriever; if the 0.762 accuracy collapses, the gain belongs to the search APIs rather than to the pipeline design.","tokens_in":18143,"feed_emoji":"🔍","tokens_out":10048,"duration_ms":89696,"temperature":0.7,"pith_summary":"This paper is trying to establish that multimodal search should be organized as a deliberate sequence of retrieval and reasoning steps, not left to a single model call. CUE-M turns a user's image-plus-text question into enriched image context (captions, tags, similar-image documents), refines it into a structured intent, generates supplementary search queries, routes to general web, shopping, or map APIs, filters the retrieved documents by relevance, and only then lets a multimodal LLM compose the final answer. On 2.1k real queries from a large Q&A service the full pipeline beats a searchless MLLM baseline with a 0.639 win rate, and each ablation step contributes to that number. On the public Encyclopedic VQA benchmark, CUE-M with Qwen2.5-7B-Instruct reaches 0.762 accuracy, close to the 0.870 oracle that uses ground-truth documents, while LLaVA-1.5-7B with the same pipeline reaches 0.467. The paper also claims the attached safety filter does not sacrifice protection, with an average attack success rate of 0.43 on MM-SafetyBench, on par with prior multimodal models.","feed_headline":"Five-stage search pipeline lifts visual Q&A to 76 percent accuracy","feed_subtitle":"CUE-M routes image questions through intent refinement and relevance filtering, nearing the oracle without retraining a 7B model.","key_machinery":"The load-bearing object is the CUE-M pipeline itself. It chains five stages: image context enrichment (an MLLM caption plus similar-image search and image-tag text retrieval), intention refinement (a few-shot prompted LLM that collapses the query and retrieved summaries into one structured sentence of what the user needs), contextual query generation (one to three supplementary searches derived from that intent), application selection (a prompted API Selector that ranks shopping, map, or general text-search APIs by the likelihood of a positive token), and relevance-based filtering (an ELECTRA-based cross-encoder that ranks and keeps the top documents). A multi-stage safety filter runs in parallel, combining lightweight image and text classifiers, a prompted multimodal detector, an instance-level query-answer database, and category-level policy answers. The argument works by showing that removing intent refinement or relevance filtering measurably lowers win rate, so the pipeline's decomposition itself, not any single component, is what carries the result.","core_discovery":"CUE-M's central discovery is that a decomposed retrieval pipeline, whose components are each prompt-tuned rather than trained, can convert a small open MLLM into a competitive knowledge-based visual question answerer. The intent refiner is the piece the ablations single out: translating the raw query plus image-derived summaries into a single structured intent is what lets the query generator form useful supplementary queries, and it also feeds the API selector and the safety detector. The relevance classifier, an ELECTRA-based cross-encoder, then removes low-quality documents before answer generation. With these stages, CUE-M beats previous knowledge-augmented methods on Encyclopedic VQA (Wiki-LLaVA 0.218, DPR*V+T 0.291, EchoSight 0.418) and approaches the ground-truth oracle (PaLM 0.870) without access to ground-truth documents. On the curated real-world set the win rate rises from 0.5 for the no-retrieval baseline to 0.639 for the full pipeline, and the safety experiments show comparable attack-success rates while acknowledging vulnerabilities to multimodal-retrieval-specific jailbreaks.","pith_inferences":["Editorial inference: the 0.639 win rate over a no-retrieval baseline is measured by GPT-4o on a filtered query distribution; a human-preference panel on held-out queries would be a stronger test that the gain is real rather than judge-aligned.","Editorial inference: the intent refiner should matter most when image and text point in different directions; a targeted dataset of mismatched captions and questions would isolate whether the win-rate gain comes from disambiguation or simply from more retrieval budget.","Editorial inference: the safety evaluation covers only six scenarios and the paper admits multimodal retrieval jailbreaks; a stress test that composes harmful intents with the API selector, such as indirect or meme-based queries, would reveal whether the category-level filters generalize beyond the benchmark.","Editorial inference: the modular stages could be reused independently, e.g., the intent refiner and relevance classifier as a standalone multimodal query-understanding and filtering benchmark, decoupled from the answer generator."],"forward_implications":["Open 7B MLLMs can reach near-oracle performance on single-hop knowledge-based VQA without fine-tuning, provided the retrieval context is enriched and filtered through intent refinement.","The pipeline is backbone-agnostic: the same prompts move LLaVA-1.5-7B from 0.169 to 0.467 and Qwen2.5-7B-Instruct from 0.304 to 0.762 on Encyclopedic VQA.","Component-level ablations imply that intent refinement and relevance filtering are the main contributors to answer quality; dropping the relevance classifier lowers win rate from 0.639 to 0.547.","Safety filtering can be layered on without a major accuracy trade-off, at least on the six MM-SafetyBench categories tested, with average ASR 0.43.","Because the API Selector is prompted rather than trained, adding a new external service reduces to adding an API description, making the design extensible to shopping, maps, or other specialized tools."],"supporting_citations":[{"why":"Supplies the Encyclopedic VQA test set and the oracle accuracies (PaLM 0.870, GPT-3 0.821) that frame the 0.762 result.","marker":"Mensink et al. (2023)"},{"why":"Supplies Wiki-LLaVA, the hierarchical retrieval-augmented baseline whose 0.218 accuracy CUE-M must beat on the public benchmark.","marker":"Caffagni et al. (2024)"},{"why":"Supplies EchoSight, the strongest prior knowledge-augmented VQA method at 0.418 that CUE-M exceeds.","marker":"Yan and Xie (2024)"},{"why":"Supplies LLaVA-1.5-7B, one of the two open backbones used to show CUE-M is model-agnostic.","marker":"Liu et al. (2024a)"},{"why":"Supplies Qwen2.5-7B-Instruct, the backbone that reaches the 0.762 accuracy.","marker":"Yang et al. (2024)"},{"why":"Supplies MM-SafetyBench and the comparison attack-success-rate scores used in the safety evaluation.","marker":"Liu et al. (2025)"},{"why":"Supplies the Q&A platform from which the 2.1k real-world multimodal queries are curated.","marker":"Naver (2002)"},{"why":"Supplies HCX-L, the Korean-language backbone used for the real-world win-rate ablations.","marker":"Yoo et al. (2024)"},{"why":"Provides the automatic prompt-tuning procedure used to fit the sub-module prompts.","marker":"Sordoni et al. (2023)"}],"fun_headline_variants":["Staged retrieval pipeline lifts visual Q&A toward oracle-level accuracy","Intent refinement key to CUE-M's multimodal search gains","Decomposed pipeline turns small MLLM into competitive VQA system","CUE-M: Prompt-tuned stages beat larger models on knowledge VQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main load-bearing premise is that the curated 2.1k-query evaluation set, which was filtered toward web, shopping, and map intents and used to tune prompts, is representative enough that GPT-4o's win-rate judgments on it reflect real user preference; if tuning and test queries overlap or the judge diverges from humans, the reported gains are fitted rather than predictive.","fun_headline_variants_meta":{"raw":{"variants":["Staged retrieval pipeline lifts visual Q&A toward oracle-level accuracy","Intent refinement key to CUE-M's multimodal search gains","Decomposed pipeline turns small MLLM into competitive VQA system","CUE-M: Prompt-tuned stages beat larger models on knowledge VQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1391,"prompt_tokens":947,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":563,"tokens_out":444,"duration_ms":5368,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:42:00.430529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: split the 2.1k Knowledge-iN queries into disjoint prompt-tuning and held-out test sets, run CUE-M against the no-retrieval baseline on the held-out set, and have human raters judge the responses; if the win rate falls from 0.639 toward 0.5 or disagrees with GPT-4o on the same pairs, the pipeline's claimed effectiveness does not generalize. A second check on Encyclopedic VQA: swap the external similar-image and text-search APIs for an equally capable open retriever; if the 0.762 accuracy collapses, the gain belongs to the search APIs rather than to the pipeline design.","supporting_citations":[],"review_version":1}