{"id":"455923dc-e74a-450f-9243-c789103f67be","arxiv_id":"2606.07872","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A paired-image benchmark reveals that many MLLMs fail to update predictions when task-critical visual evidence changes, even when they answer individual images correctly.","lead":"VisualFLIP creates 1,374 paired images where the same question has its correct answer flipped by minimal visual changes across four task types. The benchmark measures whether multimodal models actually depend on the changed visual evidence or repeat answers regardless.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Image perturbation validity is the load-bearing assumption for distinguishing correctness from evidence dependence.","rationale":"The reader's weakest assumption is exactly the premise required for the central distinction to be interpretable. The abstract-only view already marks the claim UNVERDICTED; the same concern remains the single load-bearing point once the full construction details are examined.","tokens_in":1698,"tokens_out":305,"duration_ms":16486,"concrete_test":"Randomly sample 100 pairs (25 per task category), have two independent annotators rate each on a 1-5 scale for (a) minimality of change and (b) isolation to the critical evidence only; also verify the gold flip is deterministic. If >10% of pairs receive a rating <4 on either dimension, recompute pair accuracy and CR on the filtered subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that paired correctness and evidence dependence are distinct (capable models still fail to update) requires that each pair changes only the task-critical evidence, keeps all else fixed, and produces a deterministic gold-label flip. The abstract asserts \"minimally changes the evidence\" and \"deterministically flips\" but supplies no construction protocol, human validation, or automated checks for unintended semantic or visual side-effects (lighting, composition, object co-occurrence, etc.). If such confounders exist, non-flipping behavior could be explained by sensitivity to the wrong cues rather than absence of evidence dependence.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces VisualFLIP, a benchmark of 1,374 images forming same-question perturbation pairs across cardinality, attribute, spatial, and logic tasks. Each pair minimally alters task-critical visual evidence to deterministically flip the gold answer while keeping the question fixed. It evaluates 24 MLLMs using pair accuracy (correct on both sides) and Collapse Rate (CR; tendency to repeat the same answer across the pair), arguing that paired correctness and evidence dependence are related but distinct: capable models can fail to update after evidence changes, with collapse worsening in sequential settings.","tokens_in":1792,"tokens_out":598,"duration_ms":18380,"significance":"If the perturbation validity holds, the work supplies a useful diagnostic for grounding in MLLMs beyond accuracy, highlighting that correct predictions need not reflect dependence on the intended visual evidence. The metrics and paired design offer a concrete way to quantify this gap, and the project page aids reproducibility. This could inform more robust evaluation practices in multimodal reasoning.","major_comments":[{"comment":"Benchmark construction section: the manuscript asserts that perturbations 'minimally change the evidence' and 'deterministically flip' the gold answer, but supplies no construction protocol, no inter-annotator agreement for the gold labels, and no verification (human or automated) that changes are confined to task-critical evidence without confounders such as lighting, composition, or co-occurrence shifts. This assumption is load-bearing for the central claim that non-updating behavior demonstrates absence of evidence dependence rather than sensitivity to unintended cues.","section":"Benchmark construction"},{"comment":"Experiments and results sections: pair accuracy and CR are reported across 24 models and tasks, yet no statistical tests, confidence intervals, or significance assessments are provided for differences (e.g., CR elevation in sequential vs. independent settings). Without these, the robustness of the distinction between correctness and evidence dependence cannot be evaluated.","section":"Experiments and results"},{"comment":"Abstract and evaluation sections: the claim that 'capable models can still fail to update after task-critical visual changes' and that collapse is more severe sequentially rests entirely on the pairs being valid minimal perturbations. Absent explicit checks for deterministic gold flips and absence of side-effects, the reported separation of the two phenomena remains provisional.","section":"Abstract and evaluation"}],"minor_comments":[{"comment":"The Collapse Rate definition would benefit from an explicit formula or pseudocode to clarify how 'repeats the same non-empty answer' is operationalized across answer formats.","section":"Metrics definition"},{"comment":"Figure captions and table headers could more explicitly link each reported number to the pair-accuracy vs. CR distinction for quicker reader parsing.","section":"Figures and tables"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the thoughtful and constructive report. The comments highlight important aspects of benchmark validity and statistical reporting that we will address through targeted revisions. Below we respond point-by-point to the major comments.","responses":[{"response":"We agree that the current manuscript provides insufficient detail on the perturbation protocol. In the revised version we will add a dedicated subsection under Benchmark Construction that (1) describes the step-by-step generation process for each task category, (2) specifies the criteria used to ensure changes are minimal and confined to task-critical evidence, and (3) reports the results of a post-hoc human verification study on a random sample of 200 pairs confirming that gold-label flips are deterministic and that no systematic confounders (lighting, composition, co-occurrence) were introduced. Because gold labels are derived directly from the logical task definitions rather than subjective judgment, we did not compute inter-annotator agreement on the labels themselves; the added verification study will serve as the empirical check on perturbation quality.","revision_made":"yes","referee_comment":"[Benchmark construction] Benchmark construction section: the manuscript asserts that perturbations 'minimally change the evidence' and 'deterministically flip' the gold answer, but supplies no construction protocol, no inter-annotator agreement for the gold labels, and no verification (human or automated) that changes are confined to task-critical evidence without confounders such as lighting, composition, or co-occurrence shifts. This assumption is load-bearing for the central claim that non-updating behavior demonstrates absence of evidence dependence rather than sensitivity to unintended cues."},{"response":"We accept this observation. The revised manuscript will include (1) bootstrap 95% confidence intervals for all reported pair-accuracy and CR values and (2) paired statistical tests (McNemar’s test for accuracy differences and Wilcoxon signed-rank test for CR differences) between the independent and sequential presentation conditions, with p-values and effect sizes. These additions will allow readers to assess the reliability of the observed distinctions.","revision_made":"yes","referee_comment":"[Experiments and results] Experiments and results sections: pair accuracy and CR are reported across 24 models and tasks, yet no statistical tests, confidence intervals, or significance assessments are provided for differences (e.g., CR elevation in sequential vs. independent settings). Without these, the robustness of the distinction between correctness and evidence dependence cannot be evaluated."},{"response":"We agree that the strength of the central claim depends on the validity of the pairs. Once the construction protocol and verification results are added (as outlined in response to the first comment), the abstract and evaluation sections will be updated to reference these checks explicitly. We will also add a short limitations paragraph noting that, while the verification study supports the current pairs, future work could further scale automated checks for confounders.","revision_made":"partial","referee_comment":"[Abstract and evaluation] Abstract and evaluation sections: the claim that 'capable models can still fail to update after task-critical visual changes' and that collapse is more severe sequentially rests entirely on the pairs being valid minimal perturbations. Absent explicit checks for deterministic gold flips and absence of side-effects, the reported separation of the two phenomena remains provisional."}],"tokens_in":1452,"tokens_out":686,"duration_ms":16666,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core contribution is a set of 1,374 image pairs across four task types where the question stays fixed but the critical visual detail is altered so the correct answer flips. They measure pair accuracy (both sides correct) and Collapse Rate (model gives the same answer on both despite the change), then run this on 24 models. The results separate raw correctness from evidence dependence: some models solve one side but do not update when the evidence changes, and the problem worsens in sequential prompting.\n\nThe work is straightforward and fills a gap. Standard accuracy numbers do not catch models that guess or latch onto the wrong cue, and the paired design plus the sequential condition make that distinction concrete. Running the same protocol across many models also gives a useful snapshot of current behavior.\n\nThe main weakness is the perturbation step itself. The abstract states the changes are minimal and produce a deterministic gold-label flip, but the provided details do not include construction protocol, human checks for side effects, or tests that other visual or semantic factors stayed constant. If those checks are missing or weak, the collapse numbers could reflect sensitivity to unintended cues rather than lack of evidence use. That assumption carries the central claim.\n\nThe paper is aimed at researchers who evaluate or deploy multimodal models and want diagnostics beyond accuracy. It is worth sending to peer review because the evaluation scale and the distinction it draws are useful even if the perturbation validation needs more work.","headline":"VisualFLIP gives a practical paired benchmark for checking whether MLLMs actually use the visual evidence they need, but the perturbations need tighter validation to support the main claim.","tokens_in":2258,"tokens_out":366,"would_cite":false,"duration_ms":10781,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Multimodal models often repeat the same answer even after task-critical visual evidence is altered.","keywords":["multimodal large language models","visual reasoning","evidence grounding","paired benchmark","answer collapse","perturbation pairs","visual evidence dependence"],"falsifier":"A model that changes its answer correctly on every VisualFLIP pair while keeping high pair accuracy would show that its predictions track the visual evidence rather than collapse.","tokens_in":2598,"feed_emoji":"🔄","tokens_out":625,"duration_ms":13007,"temperature":0.7,"pith_summary":"Accuracy on visual reasoning questions does not guarantee that a model is actually using the key image details. The paper creates VisualFLIP, a set of 1,374 images formed into same-question pairs where a minimal edit flips the correct answer across four task types. It measures pair accuracy, which requires correct answers on both images, and Collapse Rate, which tracks how often a model that gets at least one side right gives the identical non-empty answer to both. The findings separate paired correctness from evidence dependence: even strong models can ignore the change and stick with one answer, and this repetition grows worse for some models when the edited image appears after the first in sequence.","feed_headline":"Models repeat answers after visual evidence changes","feed_subtitle":"VisualFLIP pairs show that correct predictions often ignore minimal edits that should flip the gold answer.","key_machinery":"VisualFLIP benchmark of same-question perturbation pairs with pair accuracy and Collapse Rate (CR) metrics that together test whether correct answers depend on the changed visual evidence.","core_discovery":"Paired correctness and evidence dependence are related but distinct properties. Capable multimodal large language models can solve one member of a perturbation pair yet fail to update their prediction when the task-critical visual evidence changes, producing the same answer for both images. Collapse becomes more severe for certain models in a sequential setting where the edited image follows the original answer.","pith_inferences":["Training may reward surface patterns more than sensitivity to small evidence shifts.","The benchmark could be adapted to test longer reasoning chains or other modalities.","Developers could add explicit checks that force models to verify whether key visual elements have changed.","Repeated answers across pairs point to a need for new objectives that penalize evidence-insensitive outputs."],"forward_implications":["Accuracy alone is insufficient to confirm that a model grounds its answer in the image.","Sequential presentation of paired images can increase answer repetition for some models.","Evaluation must include both sides of each pair to separate evidence dependence from mere correctness.","Models that pass standard benchmarks can still ignore minimal visual changes that should flip the answer."],"fun_headline_variants":["Models repeat answers despite flipping visual evidence","MLLMs ignore changes in task-critical visual evidence","Paired accuracy hides lack of evidence dependence","Models fail to update predictions after image perturbations","Same answer persists after minimal visual edits"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The image edits are minimal, touch only the task-critical evidence, and produce a deterministic flip in the gold answer without adding other visual or semantic changes.","fun_headline_variants_meta":{"raw":{"variants":["Models repeat answers despite flipping visual evidence","MLLMs ignore changes in task-critical visual evidence","Paired accuracy hides lack of evidence dependence","Models fail to update predictions after image perturbations","Same answer persists after minimal visual edits"]},"model":"grok-4.3","cost_usd":0.005085,"raw_usage":{"total_tokens":2455,"prompt_tokens":627,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":50849500,"prompt_tokens_details":{"text_tokens":627,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1764,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":627,"tokens_out":64,"duration_ms":12027,"temperature":1.0,"reasoning_tokens":1764,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T21:54:46.961505+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A model that changes its answer correctly on every VisualFLIP pair while keeping high pair accuracy would show that its predictions track the visual evidence rather than collapse.","supporting_citations":[],"review_version":1}