{"id":"ff0d58c0-05e4-4d0b-b023-fbbbe66cfcd1","arxiv_id":"2607.24354","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Giving prompt optimizers failed images only at train time yields reusable text checklists that beat text-only APO by ~2.4 pp mean across 12 VQA datasets and 4 VLMs.","lead":"Text-only prompt optimizers for vision-language models never see the images that caused failures, so they cannot fix visual mistakes. This paper adds a one-time visual diagnosis step during optimization and turns the findings into ordinary text prompts that improve VQA accuracy without extra test-time cost.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The \"ranks first on every target\" headline rests on margins that are within run-to-run noise for at least two of four targets, with significance tests shown only for the three largest single-dataset gains.","rationale":"The reader identified the Stage 1–2 mechanism (label-free description + ≤30-case aggregation producing reusable patterns rather than noise) as the weakest assumption. My concern is adjacent but distinct: not whether the mechanism is sound, but whether the measured headline margins exceed run/selection noise on all four targets as claimed. The two concerns interact — if aggregation output were largely noise, gains would not replicate across seeds — so agreement is partial. I did not find the mechanism assumption itself load-bearing in a damaging way: Stage 1 withholding ŷ and y is a reasonable anti-rationalization design, the token-matched ablation (Table 5) directly separates perception from feedback length, and the monotone channel ordering replicating on three targets is real evidence. The observed weaknesses are about statistical reporting rigor: 3 seeds, no std on the headline mean, significance tests confined to the three most favorable cells, and one missing transfer control. These are fixable with additional runs of the existing pipeline and would at most temper the universality phrasing (\"every target\"), not the central claim that cross-modal failure feedback improves multimodal APO at zero inference cost — which is supported by convergent lines (main table, ablation ordering, CMVF* rewriter swap, generalization-gap analysis). Per the paper's own Eq. 5 framing, the gain pattern concentrating on high-αV benchmarks is also internally consistent. Hence ACCEPT stands; the recommended test is a verification step the authors can run cheaply before release, and I note the promised code/prompts release will make the seed-robustness check trivially reproducible by third parties.","tokens_in":15660,"tokens_out":3605,"duration_ms":119376,"concrete_test":"Extend all four targets to 10 seeds and run a paired bootstrap on the 8-benchmark mean difference (CMVF vs per-target strongest baseline); report 95% CIs. If the CI includes 0 for Qwen2.5-VL-7B (+1.5) or Phi (+2.1, and 0.1 over CMVF*), downgrade \"ranks first on every target\" to \"ranks first on 2 of 4 targets.\" Additionally, report exact McNemar p-values for all 32 (target, dataset) pairs with Holm correction instead of only the three largest-gain cells, and add one control row to Table 4: TextGrad prompts transferred cross-model to Qwen3.5-4B, to confirm the transfer advantage is specific to the visual channel.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim leads with \"CMVF consistently ranks first ... +2.4 points on average ... on every target.\" Decomposing Table 1: the mean gain over the strongest baseline is +3.0 (Qwen3.5-4B), +1.5 (Qwen2.5-VL-7B; the text says 1.6, the table shows 60.1 vs 58.6), +3.1 (LLaVA), and +2.1 (Phi). Per-dataset standard deviations over only 3 runs are routinely ±1–3 points (e.g., Qwen2.5-VL PathVQA 40.2±2.2, X-Ray 75.0±3.0), and no std is reported for the 8-benchmark mean itself. On Phi, CMVF (55.9) beats CMVF* (55.8) by 0.1 — a statistical tie — and CMVF* is per-dataset better on CT and X-Ray, so \"ranks first\" on Phi is fragile. Statistical support is asserted via exact McNemar tests, but only three (target, dataset) pairs are reported, and they appear to be the pairs with the largest raw gains (LLaVA/TextVQA, LLaVA/ChartQA, Qwen2.5/TextVQA) — a selection of the most favorable cells with no multiple-comparison correction across the 32 possible pairs. Compounding this, all methods select validation-best prompts over 20 optimization steps on a 150-example validation set; the shared protocol makes the comparison internally fair, but adaptive selection plus 3 seeds means the +1.5–2.1pp margins on two targets could plausibly be selection-plus-seed noise rather than a stable ordering. A secondary gap: the transfer claim (Table 4) compares transferred CMVF prompts only against *directly optimized* TextGrad, never against transferred TextGrad/RE prompts — so the transfer result does not itself isolate the visual channel (text-only prompts may transfer equally well). This does not undermine the core mechanism claim (Table 5's monotone CM-Blind < CM-Caption < CM-Vis < CMVF progression on three targets is genuinely supportive), but it does mean the headline's \"consistently ranks first on every target\" is stronger than the reported evidence strictly establishes.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript identifies a structural weakness of text-only automatic prompt optimization (APO) applied to multimodal tasks: the optimizer's feedback channel contains the question, prediction, and gold answer, but never the image on which the target VLM failed. The proposed remedy, Cross-Modal Visual Feedback (CMVF), adds a two-stage visual channel used only during optimization: (1) a label-free, question-aware description of each failed image by a stronger optimizer VLM (Eq. 2), and (2) error-aware aggregation of up to 30 failed cases into reusable task-level \"visual blind-spot\" patterns that drive a TextGrad-style (or REVOLVE-style, CMVF*) prompt rewrite (Eqs. 3–4). The deployed artifact is an ordinary text prompt with no added inference cost. Across 12 VQA datasets and 4 target VLMs, CMVF reports the best mean accuracy on every target (+2.4 pp over the strongest baseline on average, up to +6.5 pp on RealWorldQA), token-budget-matched ablations (CM-Blind / CM-Caption / CM-Vis / full) attribute the gain to image-aware perception plus aggregation rather than feedback length, and optimized prompts transfer to a new target model without re-optimization (Table 4).","tokens_in":16088,"tokens_out":3663,"duration_ms":126943,"significance":"If the results hold, the contribution is a practical and cleanly motivated one: it identifies a structural limitation of text-only APO on multimodal tasks, fixes it with a train-time-only visual channel, and ships an artifact (a text prompt) with zero inference-cost overhead — a deployability property most multimodal prompting methods (e.g., MPO) do not have. The paper ships genuinely falsifiable evidence: a monotone, token-budget-matched ablation ladder (Blind < Caption < Vis < full) replicated on three targets, per-dataset standard deviations, exact McNemar tests on key pairs, cross-model transfer measurements, and a promised release of code and optimized prompts. The qualitative finding that the optimizer converges on domain-plausible visual checklists from a deliberately broad Stage-1 prompt is interesting independently of the accuracy numbers. The work extends IPO's recipe to generative VQA with a clearer failure-conditioned design. Significance is solid but incremental: the gain magnitudes (+1.5 to +3.1 pp on target means) are useful rather than dramatic, and the core claim's strength currently exceeds what the reported statistics establish.","major_comments":[{"comment":"Table 1 / §3.2: the headline 'ranks first on every target' is statistically under-supported. On Phi-3.5-vision, CMVF (55.9) ties CMVF* (55.8) within 0.1 and loses per-dataset on CT and X-Ray; on Qwen2.5-VL the margin over the best baseline is +1.5 (the text says 1.6), while per-dataset stds over 3 seeds are ±1–3 (PathVQA ±2.2, X-Ray ±3.0) and no std is reported for the 8-benchmark mean. McNemar tests are shown for only 3 of the 32 (target, dataset) pairs, and those three appear to be the largest-gain cells, with no multiple-comparison correction. Combined with validation-best selection over 20 steps on 150 examples, the 'every target' wording is not established. Please report significance for all pairs (or correct across them), give the std/CI of the mean, and temper the headline accordingly.","section":"§3.2, Table 1"},{"comment":"Table 4 / §3.4: the transfer claim compares transferred CMVF prompts only against TextGrad optimized directly on Qwen3.5-4B (70.3), never against *transferred* TextGrad or REVOLVE prompts. If a text-only optimized prompt also transfers +1–2 pp across models (plausible, since generic 'expert-style' instructions are known to be somewhat model-agnostic), then transfer is a property of optimized prompts generally, not evidence that Stage-2 aggregation surfaces reusable *visual* blind-spot patterns. This comparison is load-bearing for contribution 3 and is cheap to run: the optimized baseline prompts already exist from Table 1, so transferring them requires inference only. Please add transferred-baseline rows to Table 4.","section":"§3.4, Table 4"},{"comment":"The optimizer VLM (Qwen3.5-9B) is a stronger member of the same family as two of the four targets (Qwen3.5-4B, Qwen2.5-VL-7B). Stage-1 diagnosis quality (Eq. 2, I_V in Eq. 5) plausibly depends on the optimizer's own perception ability on the failed images, and same-family pairing could inflate the diagnosis-target alignment. The cross-family LLaVA/Phi results mitigate this, but the paper never analyzes sensitivity to the choice of M_O. At minimum, the text should state this dependence explicitly and discuss how gains scale with optimizer strength; ideally one swap of M_O (e.g., a non-Qwen VLM) would be reported.","section":"§3.1 / §2.3 (Eq. 2)"}],"minor_comments":[{"comment":"Text in §3.2 reports the Qwen2.5-VL-7B gain as +1.6, but Table 1 shows 60.1 vs 58.6 = +1.5. Also the abstract's '+2.4 points on average' appears to be (3.0+1.5+3.1+2.1)/4 = 2.4 only if 1.5 is used inconsistently — please reconcile.","section":"§3.2 vs Table 1"},{"comment":"OPRO on Qwen3.5-4B ChartQA is 25.1 ± 28.4, an enormous variance that suggests protocol instability for that (method, dataset) cell; since OPRO is not the comparator for the headline claims this is not load-bearing, but a one-line remark would help readers trust the table.","section":"Table 1"},{"comment":"Table 5's columns for Qwen2.5-VL-7B and LLaVA-1.6-7B are labeled only 'RWQA Text'; clarify in the caption that these are RealWorldQA and TextVQA accuracies under the official protocol, and state whether they are single-run or 3-seed means.","section":"Table 5"},{"comment":"The Stage-1 meta-prompt π_vis is said to be in supplementary material; given that the 'self-organization' claim in §3.3 rests on π_vis not mentioning axes, small text, or slice levels, the full prompt text should be in the main appendix and quoted where the claim is made.","section":"§2.3 / §3.3"},{"comment":"The phrase 'Table 1 restores the full comparison' (§3.2) reads like a rebuttal artifact; rephrase for the standalone manuscript.","section":"§3.2"}],"recommendation":"minor_revision","confidential_remarks":"A few copy artifacts suggest the manuscript was assembled from a rebuttal/response document (\"Table 1 restores the full comparison...\", \"Table 3 lists representative deployed prompts\"), harmless but worth a cleanup pass. The three reported McNemar pairs coincide exactly with the largest single-dataset gains; I cannot tell from the text whether the remaining 29 pairs were computed, so I have framed this as a reporting request rather than a selection-bias accusation. Venue fit is good for an APO/multimodal audience; promised release of code and optimized prompts would materially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit is simple: text-only prompt optimizers never see the image the target failed on, so they cannot name visual failure modes. CMVF fixes that at train time only—label-free visual descriptions of the wrong set, then aggregation into blind-spot checklists—and ships an ordinary text prompt. Zero extra inference cost is the right product constraint.\n\nWhat is actually new relative to IPO is failure-conditioned inspection, Stage-2 error-aware aggregation into reusable VQA instructions, and the scale (12 datasets, four black-box targets). The paper does the empirical work cleanly: gains concentrate where vision matters (RealWorldQA, ChartQA, TextVQA, PathVQA), the token-matched ladder CM-Blind < Caption < Vis < full is monotone on three targets, and the emergent checklists in Table 3 are concrete rather than hand-waved. Cross-model transfer without re-opt is a nice extra, even if the comparison is only vs direct TextGrad.\n\nSoft spots, in proportion. The “ranks first on every target by +2.4” line oversells Table 1. On Qwen2.5-VL and Phi the mean edges are ~1.5–2.1pp with per-cell stds of 1–3 over three seeds and no std on the mean; Phi is essentially a tie with CMVF*. McNemar is shown only on three large-gain cells with no multiplicity correction. Shared val-best selection over 20 steps on 150 examples is fair across methods but does not make small mean gaps decisive. Transfer does not pit transferred CMVF against transferred text-only prompts, so it does not isolate the visual channel by itself. Eq. (5) is intuition, not a bound—fine if read that way. None of this breaks the core claim that image-aware feedback plus aggregation beats longer text feedback.\n\nCitations are in order (TextGrad, REVOLVE, DSPy, IPO, MPO). Free knobs (cap of 30 wrongs, T, split size) are normal for the genre. Code/prompts promised; until then reproducibility is provisional.\n\nWho it is for: anyone shipping VQA behind an API or running black-box APO. Worth a serious referee. I would engage—read the ablations and the prompt examples, discount the universal ranking language, keep the method.","headline":"Solid systems fix for a real APO blind spot; headline “ranks first everywhere” is a bit stronger than the noise bars support, but the mechanism and ablations hold.","tokens_in":17300,"tokens_out":591,"would_cite":true,"duration_ms":19359,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Text-only prompt optimizers never see the images that caused failures; giving them those images only at train time produces better ordinary text prompts at no extra inference cost.","keywords":["automatic prompt optimization","vision-language models","visual question answering","cross-modal feedback","visual blind spots","text-only deployment","prompt transfer"],"falsifier":"Run the same optimization pipeline with the image channel ablated or replaced by generic captions under a matched token budget: if mean accuracy and the expert-style checklist content no longer improve over strong text-only baselines on perception-heavy sets such as RealWorldQA, TextVQA, and medical VQA, the central claim fails.","tokens_in":16911,"feed_emoji":"👁️","tokens_out":983,"duration_ms":25404,"temperature":0.7,"pith_summary":"Automatic prompt optimization is widely used to adapt vision-language models without changing their weights, but on multimodal tasks the optimizer only ever sees the question, the wrong answer, and the gold label. It never sees the image, so it cannot tell whether the model missed a small axis label, confused an anatomical slice, or leaned on a global scene prior. This paper argues that the bottleneck is missing visual evidence, not the search algorithm. It proposes Cross-Modal Visual Feedback (CMVF): a stronger model inspects each failed image without labels, then aggregates those observations into reusable visual blind-spot patterns that rewrite a plain text prompt. The image is used only during optimization; deployment stays a normal text prompt with unchanged cost. Across twelve VQA benchmarks and four target models the method ranks first, averaging a 2.4-point gain over the strongest baseline and producing expert-style visual checklists that transfer across models.","feed_headline":"Prompt optimizers never saw the failed images","feed_subtitle":"Letting them inspect failures only at train time lifts VQA accuracy with ordinary text prompts and no extra test cost.","key_machinery":"Cross-Modal Visual Feedback (CMVF): a two-stage visual-gradient channel in which a stronger optimizer VLM first produces question-aware descriptions of failed images with predictions and labels withheld, then aggregates those descriptions with error triples into compact reusable visual blind-spot instructions that drive a standard text-prompt rewrite.","core_discovery":"The effectiveness of automatic prompt optimization on multimodal tasks is limited by a blind feedback channel: the optimizer never sees the input image on which the target failed, so it cannot diagnose visually grounded errors. Closing that channel with failure-conditioned, label-free visual diagnosis plus error-aware aggregation into task-level blind-spot patterns produces ordinary text prompts that consistently outperform strong text-only APO baselines across twelve VQA datasets and four target VLMs, with average gains of 2.4 points (up to 6.5) and no added inference cost.","pith_inferences":["The same train-time-only visual channel could be tried on other multimodal settings where prompts are discrete and inference cost is constrained, such as document or chart agents.","If aggregation is what turns noisy per-image notes into reusable checklists, similar compress-then-rewrite steps may help text-only APO that currently overfits isolated error triples.","Weaker target models appear to leave more headroom for the visual channel; the method may be most useful precisely where base accuracy is limited by perception rather than reasoning.","Label-free Stage-1 description is doing double duty as both anti-leakage and anti-rationalization; removing either property would be a sharp follow-up ablation."],"forward_implications":["Multimodal APO should treat visual evidence as first-class feedback rather than relying only on textual error triples.","Deployed artifacts can remain pure text prompts, so existing serving stacks gain accuracy without extra VLM calls at test time.","Optimizers can self-organize into transferable expert-style visual checklists (slice level, axes/legends, small text, local vs global priors) without hand-written rules.","Gains concentrate where failures are visually grounded; tasks with little visual error should see little benefit.","Prompts optimized on one model family can transfer to another without re-optimization when they encode task-level visual strategies."],"fun_headline_variants":["Prompt optimizers never saw the images they failed on","Blind feedback blocked APO until optimizers inspected failed images","Failure-conditioned visual diagnosis lifts VQA with plain text prompts","Optimizers that study failed images write stronger reusable prompts","Cross-modal visual feedback closes the blind channel in prompt optimization"],"cache_read_input_tokens":128,"weakest_assumption_plain":"A stronger vision-language model that looks only at failed images and questions, without seeing answers, can extract recurring task-level visual blind spots rather than noise or model-specific excuses.","fun_headline_variants_meta":{"raw":{"variants":["Prompt optimizers never saw the images they failed on","Blind feedback blocked APO until optimizers inspected failed images","Failure-conditioned visual diagnosis lifts VQA with plain text prompts","Optimizers that study failed images write stronger reusable prompts","Cross-modal visual feedback closes the blind channel in prompt optimization"]},"model":"grok-4.5","effort":"low","cost_usd":0.003968,"raw_usage":{"total_tokens":1253,"prompt_tokens":833,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":39684000,"prompt_tokens_details":{"text_tokens":833,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":357,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":833,"tokens_out":63,"duration_ms":7010,"temperature":1.0,"reasoning_tokens":357,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T17:17:53.736993+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same optimization pipeline with the image channel ablated or replaced by generic captions under a matched token budget: if mean accuracy and the expert-style checklist content no longer improve over strong text-only baselines on perception-heavy sets such as RealWorldQA, TextVQA, and medical VQA, the central claim fails.","supporting_citations":[],"review_version":1}