{"id":"63a0755c-4eba-43af-851b-4aeafea8354c","arxiv_id":"2412.15652","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An error-driven teacher-student pipeline extracts a student LMM's missing skills from validation mistakes and retrieves targeted samples from a task-agnostic dataset to fine-tune it.","lead":"This paper proposes a teacher-student method for adapting large multimodal models to new tasks: a teacher analyzes the student's mistakes, extracts the missing skills, and retrieves related training samples from a large general dataset. The method reports accuracy gains on several vision-language benchmarks, but the evidence would be stronger with error bars, released artifacts, and a more careful evaluation of the mistake identification step.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mistake-identification accuracy is likely overestimated: δ and λ are tuned on the same 100 labeled ScienceQA error examples used for the Table 5 evaluation, and the prompt leaks the gold answer via the 60% prior.","rationale":"The paper proposes a plausible teacher-student loop, and the ablations show each component helps, but the load-bearing novelty is that mistake localization, not surface similarity or random selection, drives the retrieved tuning data. The only direct measurement of that novelty is contaminated: δ and λ are tuned on the same 100 examples later used for the Table 5 evaluation, and the prompt tells the teacher which option is correct via the 60% prior. This makes the reported 65% accuracy an upper bound on what the method would achieve in a clean evaluation. The concern is not a rejection of the framework: Table 4 shows even random mistake selection yields substantial gains over the pretrained model, so a drop in MI accuracy would weaken, not necessarily invalidate, the headline. However, the paper's central causal story—errors lead to correct skills, which lead to targeted samples—depends on MI being reliable, and the current evidence for that reliability is methodologically weak. The Reader already reached CONDITIONAL with moderate confidence, and I agree; the clean re-estimate described above would either restore confidence or require the authors to soften the claim. No independent formal verification or shipped code is provided, and the reproducibility is plausible but unverified. The appropriate verdict remains conditional pending that check, so no change to the Reader's verdict is needed.","tokens_in":17426,"tokens_out":9323,"duration_ms":80332,"concrete_test":"Split the 100 labeled ScienceQA error examples into disjoint tuning and evaluation folds (or label a fresh set of 100 errors from a different task), select δ and λ inside the tuning fold, and re-estimate mistake-identification accuracy with the prior either removed or attached to a randomly chosen option rather than the gold answer. Then run the 10K and 100K pipelines on ScienceQA and at least two other tasks with those cleanly chosen thresholds, comparing final accuracies to Tables 1 and 5. If MI accuracy falls toward the Prompt-Per-Step baseline (28%) and/or the downstream advantage over Random/INSTA narrows by more than a few points, the central error-driven claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's central mechanism is Step 2: the teacher must locate the reasoning step whose correction would flip the student's error. The paper's evidence for this mechanism is Table 5, where the answer-switch method achieves 65% accuracy. But two features of that evaluation make 65% an unreliable estimate. First, the footnote in Sec 3.3 states that δ and λ were manually tuned on '100 error examples from the validation set of ScienceQA', and Sec 4.4 evaluates the method on '100 error samples from the validation set of ScienceQA'—the same set. This is train/test overlap for the hyperparameters, so the reported accuracy is optimistically biased. Second, the Mistake Identification prompt (Fig 4) injects 'There is a probability of 60% that option B is correct' where B is the gold answer. The teacher is therefore told the correct option before any student reasoning is appended; the answer-switch dynamics are then detecting when the student's rationale disagrees with a gold-answer prior, not independently discovering the faulty step. If the true MI accuracy is much lower on unseen examples or with a neutral prior, the skills fed to retrieval are wrong, and the downstream gains—which the ablation attributes to accurate mistakes—may not transfer to new tasks. This concern is more load-bearing than the absence of error bars or the arithmetic inconsistency in the 7.01% headline, because it targets the novel component that distinguishes the method from blind retrieval.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a three-step, error-driven framework for data-efficient tuning of large multimodal models (LMMs) without task-specific training samples. A student LMM is evaluated on a small validation set; a teacher LMM identifies the first erroneous reasoning step in the student's rationale (Mistake Identification), summarizes the missing skill (Skill Analysis), and then retrieves related samples from the task-agnostic Vision-Flan supporting set for fine-tuning (Targeted Tuning). Experiments are reported for LLaVA-7B and Qwen2-VL-7B as students, GPT-4o-mini and LLaVA-OneVision-72B as teachers, across seven tasks and three data scales, with an abstract-level claim of an average 7.01% performance boost.","tokens_in":17729,"tokens_out":7269,"duration_ms":60215,"significance":"If the claims hold, the framework is practically valuable: it addresses the common situation where task-specific training data is scarce, while providing a transparent mechanism (erroneous-step localization, skill inference, targeted retrieval) that goes beyond simple similarity-based data selection. The paper includes component ablations, two student models, two teacher models, and removes the seven evaluation tasks from the supporting dataset, which are genuine strengths. However, the headline number is inconsistent with Table 1, and the evidence for the novel Mistake Identification component is weakened by hyperparameter tuning on the evaluation set and by a prompt that reveals the gold answer. These issues are load-bearing because Mistake Identification drives the entire skill-retrieval loop, and the reported downstream improvements are the central claim. The framework remains credible, but the evidence as presented needs substantial correction before the contribution can be accepted.","major_comments":[{"comment":"The headline claim of an average performance boost of 7.01% is not supported by Table 1. In the 100K setting, the differences between \"Our Approach\" and \"Pre-trained LMM\" across the seven columns are +0.11, +18.30, +18.70, +14.20, +2.00, +16.00, and +2.68 percentage points, which average to 10.28 percentage points. Please verify the arithmetic and clearly state which baseline and which averaging procedure produce 7.01%; the abstract, Section 4.2, and Table 1 must be made consistent.","section":"Abstract; §4.2, Table 1"},{"comment":"The hyperparameters δ and λ in Mistake Identification are tuned on the same 100 manually labeled ScienceQA error examples that are later used for the Table 5 evaluation. The footnote in §3.3 states this explicitly, and §4.4 says it samples the same 100 error samples. The reported 65.0% accuracy is therefore not a held-out estimate, and the central evidence for the mistake-localization mechanism is optimistically biased. Please evaluate on a separate set or use nested validation.","section":"§3.3, §4.4, Table 5"},{"comment":"The Mistake Identification prompt tells the teacher that there is a 60% probability that the gold option is correct (option B in Figure 4). The teacher is thus given the correct answer before reading the student's rationale, and the answer-switch method is effectively detecting when the student's steps disagree with a gold-answer prior. This makes the 65% accuracy difficult to interpret as evidence of independent mistake identification. Please re-run the comparison with a neutral prior, or include an ablation that holds the prior fixed and uninformative.","section":"§3.3, Figure 4"},{"comment":"No error bars or multiple runs are reported, and several differences are small relative to what would be expected from typical LMM fine-tuning variance (e.g., MM-Bench 64.41 vs. 64.30 in the 100K row of Table 1, or several 0.5–1 point gaps in Table 2). The phrase \"significantly improves\" needs statistical support; please report the mean and standard deviation over at least three random seeds, or state explicitly which independent runs were performed.","section":"§4.2, Tables 1–4"}],"minor_comments":[{"comment":"The heading \"Conclution\" is a typo and should be corrected to \"Conclusion\".","section":"Section 5"},{"comment":"The label \"Ramdom\" is misspelled in several rows and should be \"Random\".","section":"Table 1"},{"comment":"The abstract and introduction say the method works without task-specific training samples, but the framework requires a 1K validation set of the target task; the Limitations section honestly acknowledges this, and the earlier statements should be qualified for precision.","section":"Abstract, Introduction, Limitations"},{"comment":"The hyperparameter search space for learning rate and batch size is given, but the selection criterion is not stated; please clarify whether these hyperparameters were chosen on a held-out split rather than on the evaluation test set.","section":"Appendix E"},{"comment":"The parenthetical remark that non-multiple-choice tasks can be handled by treating the gold answer as option B and the wrong prediction as option A is described for a two-choice setting; please clarify how open-ended or multi-class tasks are converted.","section":"§3.3"},{"comment":"The annotation guidelines allow annotators to label a sample as \"-1\" when no reasoning step is erroneous, but the paper does not describe how such samples are handled by the subsequent Skill Analysis and retrieval steps; please clarify.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the framework is promising and the authors are transparent about several limitations. The problems are central but apparently fixable: the 7.01% headline conflicts with the numbers in Table 1, and the Mistake Identification evaluation is compromised by tuning on the evaluation set and by a gold-answer leak in the prompt. I would not recommend rejection at this stage, because the errors appear correctable within the paper's scope. If the corrected numbers still show clear gains over Random and INSTA baselines, I would view this as a solid contribution. I would require a revised version with a reconciled headline number, a held-out evaluation of Mistake Identification with a neutral or fully ablated prior, and variance reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a genuinely new pipeline for data-efficient LMM adaptation—use a teacher to inspect a student's errors, name the missing skill, retrieve targeted samples from a task-agnostic pool, fine-tune. The components are familiar but the combination, especially the answer-switch mistake identification, is new. The paper is worth reading and probably worth refereeing, but the quantitative case for the novelty is weaker than it looks.\n\nThe honest strengths: the framework is simple and practical; they try two student models and two teachers; they remove eval tasks from the support pool; ablations show each component contributes; the open-source teacher (LLaVA-72B) works about as well as GPT-4o-mini, which makes the method more accessible. The idea of exposing capability gaps through reasoning-step localization and then querying a generic data pool is a real contribution.\n\nNow the soft spots, in order of severity. First, the mistake identification evidence is likely over-optimistic. Section 3.3 says the δ and λ thresholds were tuned on the same 100 ScienceQA error examples that Section 4.4 uses for the Table 5 evaluation. That is train/test overlap for the hyperparameters. More importantly, the prompt in Figure 4 tells the teacher 'There is a probability of 60% that option B is correct' where B is the gold answer. The teacher is handed the correct answer before seeing the student's rationale. So the answer-switch dynamics are partly detecting when the rationale conflicts with a gold-answer prior, not independently locating the faulty step. The 65% accuracy is therefore not a trustworthy estimate of the method's true localization ability. If that number falls on unseen tasks or with a neutral prior, the whole retrieval chain—skills, BM25 queries, downstream gains—is built on shakier ground than the paper claims. This is load-bearing, because mistake identification is the novel piece.\n\nSecond, the headline number is wrong. The abstract and Section 4.2 claim an average performance boost of 7.01% on the 100K setting, but Table 1's numbers give an average absolute improvement of about 10.3 points. I don't see how 7.01% is derived from the table. That should be fixed.\n\nThird, there are no error bars or multiple seeds, and the hyperparameter search for fine-tuning is a bit loose. Minor but worth noting.\n\nThe stress-test note about the label leak and tuning overlap lands. The rest of the empirical claims—that targeted retrieval beats random and INSTA*, and that 100K samples often beat full-data fine-tuning—are plausible and externally evaluated, so not circular. The central idea holds up; it just needs cleaner validation.\n\nWho's this for? Researchers working on data selection, instruction tuning, or test-time adaptation of multimodal models. It deserves a serious referee. I'd send it to review with the expectation of major revision, mainly to fix the mistake-identification evaluation and the arithmetic. The authors can do this by evaluating on held-out error samples, removing the gold-answer prior, and reporting seeds.","headline":"A genuinely new pipeline for data-efficient LMM adaptation with a plausible central idea, but the mistake-identification evidence is compromised by train/test overlap and gold-answer leakage, so the quantitative claims need stronger validation.","tokens_in":18249,"tokens_out":3835,"would_cite":false,"duration_ms":30867,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generic large multimodal model can be adapted to a new task without task-specific training samples: a teacher finds the first wrong step in the student's reasoning, names the missing skill, and retrieves targeted samples from a…","keywords":["error-driven learning","data-efficient tuning","large multimodal models","teacher-student framework","data selection","skill gap analysis","mistake identification","retrieval-based fine-tuning"],"falsifier":"On a held-out task, compare the teacher's mistake-step labels with fresh human annotations (not the 100 ScienceQA examples used to tune the margin thresholds). If localization accuracy does not beat random selection, or if replacing the answer-switch step with a random intermediate step in the retrieval pipeline yields the same downstream accuracy, the central mechanism is not doing the work.","tokens_in":17241,"feed_emoji":"🎯","tokens_out":8321,"duration_ms":62580,"temperature":0.7,"pith_summary":"Large multimodal models still need task-specific fine-tuning data, which is often scarce. This paper claims that a generic model can be adapted to a new task without any task-specific training samples: run the model on a small validation set, have a stronger teacher locate the first wrong step in its reasoning and name the missing skill, then retrieve training samples that exercise that skill from an existing task-agnostic dataset. Across seven tasks and three data scales, the authors report an average 7.01% accuracy gain over the untuned model with 100K retrieved samples, and report that 100K samples reproduce or beat fine-tuning on the full 1.55M-sample supporting dataset on five of seven tasks. The load-bearing idea is that capability gaps, not data similarity, should drive data selection.","feed_headline":"No task data needed: error-driven loop lifts LMM scores 7%","feed_subtitle":"A teacher model names the missing skill; targeted retrieval beats full-data fine-tuning on five of seven tasks.","key_machinery":"The answer-switch mistake identification: a teacher model, deprived of the image, receives the question, a prior hint that the correct option has 60% probability, and an incrementally growing prefix of the student's rationale; the first reasoning step after which the teacher's probability for the wrong answer exceeds the correct answer by a margin δ and stays there for λ steps is labeled the mistake step. That step is then converted by the same teacher into a missing-skill phrase, which BM25 matches against per-sample skill annotations of a supporting dataset to retrieve the tuning samples. The loop is rounded out by using the retrieved samples to fine-tune the student.","core_discovery":"On the paper's own terms, the central discovery is that the error pattern of a pretrained LMM can be converted into a training-data query. The teacher reads the student's chain of reasoning, tracks how the probability of the correct versus the wrong answer shifts as each reasoning sentence is appended, and takes the first step that flips the balance as the mistake step; it then names a skill whose absence explained that step. Retrieving samples from a large human-labeled supporting set by matching that skill name, rather than by surface similarity to the validation examples, produces a smaller, more effective tuning set. The paper reports that this selection beat random selection and INSTA-derived similarity selection at 10K, 30K, and 100K samples, and that the ordering held for two student models and two teacher models.","pith_inferences":["One implication the paper leaves implicit: the skill vocabulary is generated by the same teacher that does the diagnosis, so the framework's success may depend on the teacher's skill annotations being consistent with the retrieval index it builds; a testable extension would use a different model to annotate the supporting set than to diagnose the student.","The paper reports only single-round tuning within one pass; iterating the three steps over multiple rounds, as the framework's description allows, is a straightforward extension that could reveal whether skills accumulate or saturate.","The answer-switch method requires token-level probabilities for candidate options, which are not available for all closed-source teachers; an open-weight teacher with accessible logits would make the method reproducible and allow the δ and λ thresholds to be set without held-out labels.","The gains suggest a reframing of data-efficient tuning: rather than maximizing similarity between retrieved samples and the target task, the right objective is maximizing coverage of the student's named capability gaps; this could be tested by ablating the skill-matching step against direct mistake-step retrieval."],"forward_implications":["At 100K tuning samples, the average gain across seven tasks is 7.01 percentage points over the untuned model.","Using 100K samples (6% of the 1.55M-sample supporting set) reaches at least 94.57% of full-data performance and beats full-data fine-tuning on five of seven tasks.","The selection principle transfers: the same pipeline works with LLaVA-OneVision-72B as teacher and with Qwen2-VL-7B as student, with Qwen2-VL gaining up to 3.80%.","The answer-switch localization (65% accuracy on 100 ScienceQA labels) outperforms per-step verification (28%) and pseudo-rationale matching (59%).","Fine-tuning on the 1K validation samples of the target task underperforms the retrieval-based approach by an average of 5.11 percentage points."],"supporting_citations":[{"why":"Supplies the GPT-4o-mini teacher model that localizes mistakes and names missing skills.","marker":"(Achiam et al., 2023)"},{"why":"Supplies the LLaVA-7B student model used in the main experiments.","marker":"(Liu et al., 2024a)"},{"why":"Supplies the Qwen2-VL-7B student model that shows the framework transfers to a stronger base.","marker":"(Wang et al., 2024)"},{"why":"Supplies Vision-Flan, the task-agnostic supporting dataset from which tuning samples are retrieved.","marker":"(Xu et al., 2024)"},{"why":"Supplies the definition of the mistake step as the first erroneous step, plus the per-step verification baseline.","marker":"(Tyen et al., 2024)"},{"why":"Supplies the skill definition that underlies both skill analysis and skill-set annotation.","marker":"(Chen et al., 2023c)"},{"why":"Supplies BM25, the retrieval function that ranks supporting samples by skill match.","marker":"(Robertson et al., 2009)"},{"why":"Supplies INSTA, the similarity-based data-selection baseline that the method is compared against.","marker":"(Lee et al., 2024a)"},{"why":"Supplies ScienceQA, one of the target tasks and the only task with manually labeled mistake steps.","marker":"(Lu et al., 2022)"}],"fun_headline_variants":["Teacher-driven error loop tunes LMMs without target data","Pinpoint skill gaps, retrieve data: LMM tuning up 7%","No task samples: error-driven selection boosts LMM 7%","Teacher model finds missing skills, LMM gains 7%","Mistake-to-data pipeline lifts LMM scores by 7%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the teacher can reliably locate the first genuinely faulty reasoning step from the shift in its own answer probabilities, and that this localization transfers across tasks.","fun_headline_variants_meta":{"raw":{"variants":["Teacher-driven error loop tunes LMMs without target data","Pinpoint skill gaps, retrieve data: LMM tuning up 7%","No task samples: error-driven selection boosts LMM 7%","Teacher model finds missing skills, LMM gains 7%","Mistake-to-data pipeline lifts LMM scores by 7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1359,"prompt_tokens":898,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":370}},"tokens_in":514,"tokens_out":461,"duration_ms":4919,"temperature":1.0,"reasoning_tokens":370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:13:24.074105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out task, compare the teacher's mistake-step labels with fresh human annotations (not the 100 ScienceQA examples used to tune the margin thresholds). If localization accuracy does not beat random selection, or if replacing the answer-switch step with a random intermediate step in the retrieval pipeline yields the same downstream accuracy, the central mechanism is not doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the definition of the mistake step as the first erroneous step, plus the per-step verification baseline."}],"review_version":1}