{"id":"09e57ccd-73db-4bf1-a5f7-86851553516c","arxiv_id":"2506.07202","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fixed-image, multi-task evaluation framework aims to detect data contamination in multimodal LLMs, but its judge is unvalidated and possibly self-referential, and the claimed harm to generalization is not supported by the reported averages.","lead":"This paper proposes evaluating multimodal AI models with the same image or video across four different task types, arguing that uneven scores reveal memorization of test data rather than true understanding. The idea is useful, but the evidence is weakened by an unvalidated AI judge that may be scoring itself, missing error bars, and a contamination experiment whose own numbers contradict the paper's headline claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Controlled contamination experiment in Table 2 contradicts the central claim: after PEFT on test QA, both models' average scores rise and the 7B model's task-space range/SD decrease, so cross-task sharpness does not reliably isolate contamination.","rationale":"The reader's REJECT verdict is based on the observation that cross-task variance may reflect task difficulty rather than contamination, and that the PEFT experiment does not validate the sharpness measure. My analysis confirms this and strengthens it: the controlled experiment's numbers actually contradict the central hypothesis, with the 7B model showing reduced sharpness after extreme contamination and both models showing increased average performance. This is not a matter of an untested alternative explanation; it is an internal inconsistency between the paper's claims and its own Table 2. The only way the central claim survives is if the observed changes are statistically meaningless or arise from uncontrolled factors, which the paper does not address. Therefore the verdict should remain REJECT. I agree fully with the reader's weakest_assumption, and no additional concern is needed to justify rejection. The concrete test proposed would settle whether the contradiction is robust or an artifact of a single run, but regardless, the current evidence does not support the paper's central claim.","tokens_in":14294,"tokens_out":2654,"duration_ms":23550,"concrete_test":"Re-run the controlled contamination experiment for Qwen2.5-VL (7B) with at least 5 random seeds and multiple PEFT ranks (e.g., LoRA r=8, 16, 32), computing per-sample scores and bootstrap 95% confidence intervals for Rng and SD. Under the paper's hypothesis, Rng and SD should increase monotonically with contamination strength. If the 7B model again shows decreased Rng/SD after PEFT, or if confidence intervals overlap with the pre-PEFT values, the claim that sharpness reflects contamination is falsified. Also run a control where PEFT is applied to a disjoint held-out set (not test data) to verify that any sharpness change is specific to contamination rather than fine-tuning in general.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that task perturbation across QA, captioning, question generation, and verification reveals contamination via a 'sharp' cross-task performance profile (Section 1, Section 3.2). The controlled experiment in Table 2 is meant to validate this. For Qwen2.5-VL (3B), PEFT fine-tuning on test QA makes T0 rise from 58.82 to 96.21, T1 drop from 88.50 to 85.16, T2 drop from 70.78 to 63.40, T3 rise slightly; Rng increases from 29.68 to 32.81, SD from 11.07 to 11.80. For Qwen2.5-VL (7B), T0 rises from 63.27 to 96.21, T1 drops from 91.58 to 90.75, T2 rises from 64.64 to 69.08, T3 drops from 89.43 to 89.18; Rng decreases from 28.31 to 27.12, SD decreases from 13.31 to 10.28. Thus the 'extreme contamination' increases overall average performance (74.63 to 81.55 for 3B; 77.23 to 86.31 for 7B) and for the 7B model actually reduces the sharpness metrics. This directly contradicts the abstract's statement that contamination 'harms overall generalization' and the hypothesis that contaminated models have spiky profiles. The paper gives no error bars or significance tests, so we cannot tell whether the small Rng/SD changes are meaningful. The theoretical motivation in Section 2.1 derives input-space curvature (Eq. 2.1-2.4) and then asserts by analogy that task perturbation follows the same sharp-flat logic, but no formal bridge is provided. The empirical contradiction is the load-bearing weakness: the central claimed ability to distinguish contamination from genuine understanding by cross-task variance is not supported by the paper's own controlled test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'dynamic task perturbation' evaluation framework for multimodal large language models (MLLMs). Instead of perturbing the visual or textual input, the framework holds the input fixed and evaluates each model on four tasks derived from the same visual content: QA (T0), captioning (T1), question generation (T2), and answer verification (T3). A reasoning MLLM judge is used to score the open-ended outputs of T1 and T2. The authors define cross-task 'ability vectors' and use the range and standard deviation of per-task scores as 'task-space sharpness' metrics, arguing that contaminated or overfit models show high sharpness (spiky profiles) while genuinely generalizing models show flat profiles. The framework is instantiated on MME, RealWorldQA, and CVRR-ES across many open and closed MLLMs. The central empirical validation is a controlled contamination experiment (Table 2) in which Qwen2.5-VL models are PEFT fine-tuned on test QA data; the authors claim this sharpens task-specific performance while harming overall generalization.","tokens_in":14669,"tokens_out":2927,"duration_ms":33329,"significance":"The core idea of perturbing the task rather than the input is a natural and potentially valuable complement to existing dynamic evaluation methods. The paper also makes a useful concrete proposal for scoring open-ended generative outputs with a judge model, and it evaluates a broad suite of models across image and video benchmarks, which is a substantial empirical effort. If the central claim were established, the framework could serve as a practical tool for flagging suspicious benchmark scores. However, the paper's own controlled experiment does not support the central claim: the sharpness metrics do not reliably increase under simulated extreme contamination, and the judge-based scoring introduces a circularity problem for at least one of the headline models. Because these issues bear directly on the paper's main contribution, the current version does not yet demonstrate that task-space sharpness distinguishes contamination from genuine generalization.","major_comments":[{"comment":"The controlled contamination experiment contradicts the central claim. After PEFT fine-tuning on test QA, Qwen2.5-VL (7B) shows T0 rising from 63.27 to 96.21, average performance rising from 77.23 to 86.31, but Rng decreasing from 28.31 to 27.12 and SD decreasing from 13.31 to 10.28. Thus for the larger model, the 'extreme contamination' condition actually reduces task-space sharpness as measured by both reported metrics. This directly undercuts the abstract's statement that such fine-tuning 'harms overall generalization' and the hypothesis that contaminated models exhibit spiky cross-task profiles. The 3B model does show an increase in Rng and SD, so the two models move in opposite directions; no error bars or significance tests are provided, so it is unclear whether any of these changes are meaningful. At minimum, the paper does not demonstrate that sharpness isolates contamination.","section":"Section 3.2, Table 2"},{"comment":"The theoretical motivation is not formally connected to task perturbation. Equations (2.1)-(2.4) give a Taylor expansion of the loss under an input-space perturbation δ and argue that contaminated models have higher input-space curvature. The text then asserts, without a derivation, that the same flat-versus-sharp logic applies when 'we fix inputs and stress-test across different tasks.' No formal object is defined for a Hessian or curvature in task space, and no argument shows that per-task score variance (Rng, SD) corresponds to the curvature of an underlying loss surface. The analogy may be heuristic, but it is presented as theoretical grounding for the core contamination-detection claim, so this gap is load-bearing.","section":"Section 2.1, Eqs. (2.1)-(2.4)"},{"comment":"The judge-based scoring for the open-ended tasks T1 and T2 is not validated against human judgments, and for at least one model it is circular. The text says a reasoning MLLM 'e.g., VL-Rethinker' is used as the judge, and VL-Rethinker itself appears in Table 1; consequently VL-Rethinker's T1 and T2 scores are in part self-assessments. Even if a different judge were used for the other models, the paper provides no calibration evidence that judge scores agree with human ratings, and Appendix A acknowledges that such discrepancies are an 'empirical risk.' Because T1 and T2 are two of the four tasks that define the ability vector and the sharpness metrics, an unvalidated and partially circular judge undermines the quantitative claims built on those scores.","section":"Section 2.4 and Table 1"},{"comment":"The paper assumes, without adequate support, that cross-task performance variance is a valid proxy for data contamination or overfitting rather than for intrinsic task difficulty or model capability differences. The four tasks are not shown to be comparable in difficulty, and no task-difficulty normalization is attempted. The controlled PEFT experiment was supposed to validate this assumption, but as noted above it does not: for the 7B model, contamination reduces variance. Consequently, the interpretive claims such as 'models known to likely have seen the benchmark during training show markedly uneven performance' (Section 1) are not established by the data presented.","section":"Section 3.2, Figure 3 and Table 1"}],"minor_comments":[{"comment":"The text describing correlations swaps the task labels: it says 'Question Verification (T2) and Question Generation (T3)', but per Section 2.2, T2 is question generation and T3 is answer verification. This makes the correlation discussion hard to follow.","section":"Figure 4 and Section 3.2"},{"comment":"The notation is confusing: Section 2.3 defines both Sdist (maximum pairwise distance) and Sdist (average pairwise distance) with the same symbol, while Table 1 reports 'SD = Sdist'. Use distinct symbols for the two metrics and clarify which one is reported in each table.","section":"Table 1 and Section 2.3"},{"comment":"The abstract states that fine-tuning on simulated test data 'harms overall generalization', but Table 2 shows the average score increases substantially for both models (74.63 to 81.55 for 3B; 77.23 to 86.31 for 7B). The wording should be qualified to refer to performance on non-contaminated tasks, not overall average performance.","section":"Abstract and Section 3.2"},{"comment":"Reference [38] is listed as a survey on multimodal large language models but is cited in the MME benchmark paragraph in a way that may confuse it with the MME paper (Reference [10]). Please verify that all benchmark citations point to the correct sources.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central contribution is a contamination-detection framework based on task-space sharpness, but the controlled experiment in Table 2 fails to show that sharpness increases under simulated contamination for the 7B model, and the judge-based scoring is circular for a headline model. These are not merely presentation issues; they undermine the main claim. The idea may be worth revisiting with a validated judge, task-difficulty controls, and a contamination experiment that actually produces the predicted sharpness increase, but that would be a substantially different paper. I would not invite a resubmission on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the fixed-input multi-task evaluation idea is genuinely useful: hold the image or video constant, vary the task across QA, captioning, question generation, and verification, and look at the resulting “ability vector.” That is a real, modest extension of standard multi-task benchmarks like MME or MM-Vet, and the four-task family is well chosen. Second, the paper’s central claim—that cross-task variance (sharpness) reveals contamination—is contradicted by its own controlled experiment in Table 2. The 7B model, after PEFT fine-tuning on test QA, sees its range drop from 28.31 to 27.12 and its SD from 13.31 to 10.28, while average performance rises. That is the opposite of the predicted spiky contamination profile. The 3B model moves in the predicted direction on range, but the changes are small, no error bars are given, and average performance still rises for both models. So the abstract’s claim that contamination “harms overall generalization” is not supported by the paper’s own numbers.\n\nWhat the paper does well: the task-perturbation framing is a fresh angle, the PEFT contamination simulation is a sensible controlled test, and the authors honestly report numbers that undermine their own narrative. They also acknowledge the judge-bias risk in Appendix A—good sign. But the soft spots are substantial. The theoretical motivation in Section 2.1 is derived for input perturbations and then asserted by analogy for task perturbations; there is no formal bridge. The judge for open-ended tasks is unvalidated against humans, and the named example, VL-Rethinker, appears in the evaluated model table, making its scores self-assessments. Even if a different judge is used elsewhere, the scoring is uncalibrated.\n\nWho this is for: researchers building dynamic evaluation protocols for MLLMs. The idea could be repurposed as a complementary probe, but not as a reliable contamination detector as presented. It deserves a serious referee—the empirical setup is real and reproducible—but the current version needs heavy revision: validate the judge on human judgments, exclude the judge from the evaluated set, report significance or confidence intervals, and either fix the sharpness metric or recalibrate the claims.\n\nRecommendation: send to peer review, but expect major changes. In its current form, I would not cite it as evidence for contamination detection.","headline":"Worth a serious look, but its own Table 2 undercuts the headline claim that cross-task sharpness detects contamination.","tokens_in":15226,"tokens_out":1507,"would_cite":false,"duration_ms":16704,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fixing the visual input and varying the task across QA, captioning, question generation, and verification separates genuine MLLM understanding from memorized benchmark answers.","keywords":["multimodal large language models","data contamination","dynamic evaluation","task perturbation","cross-task ability vector","task-space sharpness","visual question answering","benchmark leakage"],"falsifier":"Take a model trained on clean data but with an intentionally narrow skill profile—say, strong at captioning and weak at verification—and run the four-task battery on it. If its cross-task range and standard deviation are as large as a model that memorized QA test answers, then the sharpness proxy alone cannot separate inherent task imbalance from contamination.","tokens_in":13977,"feed_emoji":"🕵️","tokens_out":7221,"duration_ms":66350,"temperature":0.7,"pith_summary":"The paper argues that benchmark scores of multimodal large language models can hide data contamination, and it offers a way to see through the hiding: hold the image or video fixed and vary only the task. A model is evaluated on question answering, captioning, question generation, and answer verification for the same visual input, producing a cross-task ability vector. The paper claims that contaminated or overfit models show sharp, spiky profiles—high on the memorized task, low on the others—while genuinely generalizing models have flatter profiles. This matters because static single-task benchmarks cannot tell a memorized answer from real understanding, so high scores may be inflated. The authors demonstrate the idea by fine-tuning models on test-set QA data, which spikes QA accuracy while hurting or not helping other tasks, and they interpret the resulting performance range as task-space sharpness.","feed_headline":"Same image, four tasks: a test that exposes memorized MLLM scores","feed_subtitle":"Holding the image fixed, one battery probes QA, captions, questions, and verification to expose memorized answers.","key_machinery":"The central object is the fixed-input multi-task evaluation pipeline: the same image or video is presented with four different task prompts (T0 visual QA, T1 captioning, T2 question generation, T3 answer verification), and the four resulting scores form a model's ability vector. The load-bearing identity is task-space sharpness, measured as the range and standard deviation of those scores, which the paper maps onto the flat-minima versus sharp-minima distinction from loss-landscape theory. A reasoning MLLM judge scores the open-ended captioning and question-generation outputs using rubrics, and a PEFT fine-tune on test QA data serves as the controlled contamination simulator.","core_discovery":"The central claim is that perturbing the task, rather than the input, reveals whether an MLLM's performance is genuine generalization or narrow memorization. For each visual input, the framework evaluates the model on four tasks—visual QA, captioning, question generation, and answer verification—and measures the spread of scores (range and standard deviation) as a proxy for sharpness of the model's loss minimum. A model that memorized a task sits in a sharp minimum: it scores high on that task and noticeably lower on the related tasks sharing the same image. A model with a robust internal representation occupies a flatter region and degrades gracefully. Applying this to 12 image and 11 video MLLMs across three benchmarks, the paper reports that reasoning-oriented models tend to show flatter profiles, and it uses a PEFT fine-tune on test QA data as a controlled contamination experiment to show task-specific sharpening.","pith_inferences":["Beyond the paper: the same fixed-input task battery could be applied to other modalities such as audio or 3D scenes, where contamination is equally hard to detect, since the method does not depend on image-specific augmentation.","Beyond the paper: combining task perturbation with input perturbation—for example, asking for a caption of an image with a deliberately added distractor—would test whether the two failure modes are independent or compound.","Beyond the paper: the authors' own 7B PEFT result shows the range can shrink after contamination, so a user of this framework would want to look at per-task drops rather than relying on aggregate sharpness alone."],"forward_implications":["If task-space sharpness tracks contamination, benchmark leaders with spiky ability vectors are less trustworthy than their single-task accuracy suggests.","Evaluating a model across related tasks for the same input can be added to existing benchmarks like MME, RealWorldQA, and CVRR-ES without new test data, only new prompts and scoring.","Reasoning-oriented post-training that generalizes across tasks would be expected to flatten the ability vector, making cross-task range a cheap monitoring signal during model development.","Worst-task risk and inter-task distance give developers a specific target: the weakest task for a given image indicates where understanding is missing."],"supporting_citations":[{"why":"Supplies the MME benchmark data that anchors the main cross-task table and sharpness analysis.","marker":"[10]"},{"why":"Supplies the RealWorldQA benchmark used for the image evaluation and the PEFT contamination simulation.","marker":"[36]"},{"why":"Supplies the CVRR-ES video benchmark used to instantiate the framework on video inputs.","marker":"[7]"},{"why":"Provides the PEFT/LoRA method used to fine-tune models on test QA data and simulate extreme contamination.","marker":"[15]"},{"why":"Documents contamination in multimodal benchmarks and motivates the need for dynamic evaluation.","marker":"[2]"},{"why":"Represents the prior input-perturbation dynamic evaluation approach that the paper contrasts with task perturbation.","marker":"[37]"},{"why":"Supplies the flat-minima theory linking broad minima to generalization, which underpins the task-space sharpness analogy.","marker":"[14]"},{"why":"Supplies the sharp-minima and generalization-gap results used to interpret contaminated models as sharp and brittle.","marker":"[16]"},{"why":"Establishes concerns about using language-video models as judges, motivating the calibrated reasoning-MLLM judge for subjective tasks.","marker":"[23]"},{"why":"Provides the SFT-memorizes versus RL-generalizes finding used to interpret the flatter profiles of reasoning-oriented models.","marker":"[5]"}],"fun_headline_variants":["Task switching exposes MLLM memorization","Four tasks, one image: a sharper MLLM test","How to spot an MLLM that just memorized answers","A dynamic test reveals if MLLMs truly generalize"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the spread of a model's scores across the four tasks (range and standard deviation) reflects contamination or overfitting rather than genuine differences in task difficulty or in the skills each model was built to have.","fun_headline_variants_meta":{"raw":{"variants":["Task switching exposes MLLM memorization","Four tasks, one image: a sharper MLLM test","How to spot an MLLM that just memorized answers","A dynamic test reveals if MLLMs truly generalize"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000488,"raw_usage":{"total_tokens":2430,"prompt_tokens":999,"completion_tokens":1431,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1366}},"tokens_in":615,"tokens_out":1431,"duration_ms":9514,"temperature":1.0,"reasoning_tokens":1366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:39:35.924923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model trained on clean data but with an intentionally narrow skill profile—say, strong at captioning and weak at verification—and run the four-task battery on it. If its cross-task range and standard deviation are as large as a model that memorized QA test answers, then the sharpness proxy alone cannot separate inherent task imbalance from contamination.","supporting_citations":[{"cited_title":"Realworldqa","cited_arxiv_id":null,"evidence_quote":"Supplies the RealWorldQA benchmark used for the image evaluation and the PEFT contamination simulation."},{"cited_title":"Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES)","cited_arxiv_id":null,"evidence_quote":"Supplies the CVRR-ES video benchmark used to instantiate the framework on video inputs."},{"cited_title":"Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping","cited_arxiv_id":null,"evidence_quote":"Represents the prior input-perturbation dynamic evaluation approach that the paper contrasts with task perturbation."},{"cited_title":"Flat minima","cited_arxiv_id":null,"evidence_quote":"Supplies the flat-minima theory linking broad minima to generalization, which underpins the task-space sharpness analogy."},{"cited_title":"Is your video language model a reliable judge? In The Thirteenth International Conference on Learning Representations, 2025","cited_arxiv_id":null,"evidence_quote":"Establishes concerns about using language-video models as judges, motivating the calibrated reasoning-MLLM judge for subjective tasks."}],"review_version":1}