{"id":"a1079201-c1df-498f-a149-530fe4c934bb","arxiv_id":"2501.04155","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning Llava-1.5 on MM-GEN-generated task-specific questions improves chart, diagram, and spatial reasoning accuracy by 14-29% absolute over the base model.","lead":"MM-GEN generates task-specific question-answer training data for images using a stronger vision-language model, then fine-tunes a smaller VLM on that data. It reports absolute accuracy gains of up to 29% on chart, diagram, and spatial reasoning benchmarks, suggesting targeted synthetic data can rival human-curated data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains rest on using each benchmark's validation set as reference samples to synthesize the training data; the 15%, 14%, and 29% numbers are not trustworthy until the pipeline is re-run with reference samples from a held-out or training-only source.","rationale":"The reader's weakest assumption correctly identifies the evaluation-protocol risk. The paper's own Section 5 text makes the leak concrete: ChartQA and SpatialMap both use the benchmark's validation set as reference samples. A data-curation method may legitimately use a small task-definition set, but a benchmark's validation split is not a neutral task-definition source when the same benchmark is the evaluation target. The reported gains are large absolute moves on small test sets; without error bars or multiple seeds, the single-run numbers cannot separate leakage from method effect. I do not see an internal logical contradiction in the construction; the pipeline is coherent, and the AI2D result (references from training split) suggests the approach may have real value. The control-task claim in Table 1 is internally inconsistent (base 35.8 vs 33.6 after ChartQA training) and should be corrected, but it is not the load-bearing issue for the main claim. The correct disposition is therefore to keep the CONDITIONAL verdict: the paper should be accepted only after a held-out replication with reference samples from training or external sources, candidate images disjoint from evaluation, and repeated-seed reporting.","tokens_in":17918,"tokens_out":7925,"duration_ms":83053,"concrete_test":"Run a held-out replication of the ChartQA and SpatialMap pipelines: draw reference samples only from the task's official training split (or from an independent chart/map QA source), use candidate images disjoint from the evaluation split, keep all other prompts, filtering, and training hyperparameters identical, and evaluate on the official test split with mean +- std over at least 3 random seeds. If the absolute gains over the base model drop materially below the reported 15% (ChartQA) and 29% (SpatialMap), the validation-set reference conditioning is the driver; if the gains persist, the leakage concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that MM-GEN's task-specific synthetic data produces large transferable gains, requires that reference samples specify the task without leaking the evaluation distribution. Section 5 violates this for two of the three headline tasks. For ChartQA, reference samples are 'the validation set of ChartQA' and candidate images come from the ChartQA training set; for SpatialMap, reference samples are 'the validation set of SpatialMap' and the skyline is 'generated using the code provided to generate the evaluation set.' Accuracy is then reported on those same benchmark sets (Fig. 5). Conditioning the teacher on the benchmark's validation split means the synthetic training distribution is shaped by exactly the question style, answer distribution, and image content that the evaluation will use. For SpatialMap, this is especially severe because the validation set is drawn from the same programmatic distribution as the test set, so the 29% gain may largely be an artifact of training on a synthetic expansion of the evaluation distribution rather than a transferable data-curation effect. The method's underlying idea is not disproven, and AI2D uses training-set references and also shows gains, but the reported magnitudes and the comparison with the skyline are not established under a clean protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MM-GEN, a pipeline for generating task-specific synthetic question-answer annotations for images using a stronger VLM (GPT-4o) as a teacher. The pipeline has three stages: partitioning candidate images and reference samples by image type via CLIP, generating annotations by conditioning the teacher on in-context reference samples, and filtering synthesized examples by middle perplexity using the student VLM. The authors fine-tune LLaVA-1.5 7B and 13B on the resulting synthetic data for ChartQA, AI2D, and SpatialMap, and report large absolute gains over the base model (15%, 14%, and 29% for the 7B model), with comparisons against task-agnostic captions, task descriptions without reference images, and a human-curated or programmatic i.i.d. skyline. They also report ablations and a control experiment on MMMU.","tokens_in":18166,"tokens_out":6752,"duration_ms":66540,"significance":"If the headline results are valid, the paper makes a useful practical contribution: a largely automated, task-targeted synthetic data pipeline with modest human input and a released codebase. The AI2D experiment, which draws reference samples from the training split, is a genuinely positive signal that the method can improve a specialized VLM benchmark, and the perplexity-based filtering idea is simple and potentially reusable. However, as presented, the evaluation protocol for two of the three headline tasks conditions the teacher on the same validation benchmarks used for scoring, so the 15%, 14%, and 29% gains cannot be accepted as evidence of transferable data curation without a re-run under a clean held-out protocol.","major_comments":[{"comment":"The evaluation protocol leaks information from the benchmark used for scoring into the training-data generation process. For ChartQA, the reference samples are 'the validation set of ChartQA' and the evaluation is reported on the same benchmark; for SpatialMap, the reference samples are 'the validation set of SpatialMap' and the skyline is generated with the same code that creates the evaluation set. Because the teacher prompt in Appendix A contains the reference samples verbatim, the synthetic training distribution is conditioned on the exact question/answer style and, for ChartQA, on the exact images that define the evaluation split. This does not make the reported 15%, 14%, and 29% absolute gains valid evidence for transferable data curation. The central claim can be restored by regenerating with reference samples drawn from a disjoint training or held-out split, as already done for AI2D, and reporting evaluation on the official test/holdout set; at minimum, an analysis of the overlap between reference samples and evaluation samples is needed.","section":"Sec. 5, Fig. 5"},{"comment":"The claim in Sec. 5.1 that MM-GEN can 'close the gap with or exceed the skyline performance across all three tasks' is contradicted by the reported numbers on SpatialMap: the filtered MM-GEN result is 0.55 versus a 0.98 skyline, i.e., only 0.4x of the skyline's absolute improvement (0.29 over 0.72). This overstatement should be corrected; the results should be described as a partial gap reduction on ChartQA and SpatialMap and a full closing only on AI2D.","section":"Sec. 5.1, Fig. 5(c)"},{"comment":"The statement that training on MM-GEN data 'does not hurt performance on other tasks' is not supported by Table 1: MMMU accuracy drops from 35.8 (base) to 33.6 when using the ChartQA-generated data and to 34.1 when using the SpatialMap-generated data. Please either report variance and statistical significance, soften the claim to 'does not substantially degrade,' or explain why these drops are within noise.","section":"Table 1"},{"comment":"All reported accuracies appear to be single-run validation numbers with no error bars or seeds. Differences that drive the ranking of filtering and ablation variants (e.g., 33.0 vs. 32.8 in Table 3, and 0.58 vs. 0.55 in Fig. 5c) are within the range that typically varies across fine-tuning runs. Please report the number of runs and standard deviation or confidence intervals for the headline comparisons, or identify which numbers are averaged.","section":"Sec. 5, Fig. 5"}],"minor_comments":[{"comment":"The text says partitioning contributes 'a non-trivial 2%' to the improvement, but Table 3 shows 33.0 - 31.6 = 1.4 percentage points; please correct the number.","section":"Sec. 5.2, Table 3"},{"comment":"The filtering step is described only as selecting 'middle perplexity' examples with 50% retained; please specify the exact percentile interval (e.g., 25th-75th) and how ties or boundary cases are handled, so the pipeline is reproducible.","section":"Sec. 4.2.3"},{"comment":"The prompt text contains typos such as 'understadning' and 'Refererence Sample'; please proofread the appendix.","section":"Appendix A"},{"comment":"There are minor wording and consistency issues, including 'taksks' in the conclusion and inconsistent capitalization of 'Llava/LLava'; please fix these throughout.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The validation-set leakage is serious enough that the headline numbers cannot be accepted as evidence of a transferable data-curation effect. I would not require rejection because the method is plausible and the AI2D experiment, which uses training-set reference samples, is a cleaner positive result; the authors can likely fix the protocol by regenerating data with disjoint reference splits or by evaluating on an unseen test set. For a revision, the evaluation should also include variance across runs, because the claimed ablations and the control-task claim currently hinge on differences of 1-2 points with no error bars."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the pipeline is sensible and clearly described, but the headline numbers for ChartQA and SpatialMap are compromised because the reference samples are drawn from the validation sets of the very benchmarks used for evaluation. That needs to be fixed before the reported magnitudes can be trusted.\n\nWhat's actually new: the three-stage recipe—CLIP-based subgroup partitioning, in-context reference-guided generation with a stronger VLM, and middle-perplexity filtering—is a useful combination. The individual pieces exist in prior work, but I haven't seen them put together this way. The paper also contributes clean baselines and a skyline, which is more than most data-curation papers do. The AI2D result (14% absolute gain) uses reference samples from the training set, so that one is a clean demonstration that the method can work without leakage.\n\nSoft spots, in proportion. First, the validation-set leakage is real. Section 5 says ChartQA reference samples are 'the validation set of ChartQA' and SpatialMap uses 'the validation set of SpatialMap', then Fig. 5 reports accuracy on those same benchmark sets. For SpatialMap, the skyline is also generated with the same code that produced the evaluation set, so the 29% gain is likely inflated by training on a synthetic expansion of the evaluation distribution rather than a transferable effect. Second, there are no error bars anywhere—single-run validation accuracies. That is a minor issue on its own but matters here because the gaps are the whole story. Third, the control-task claim contradicts the paper's own Table 1: base MMMU is 35.8, MM-GEN(ChartQA) is 33.6, MM-GEN(SpatialMap) is 34.1. That is a drop, not 'does not hurt performance on other tasks.' The authors should have caught that.\n\nFor what it's worth, the leakage is not fatal to the core idea. AI2D is clean and still shows a solid gain. The fix is straightforward: use reference samples from a held-out split or from a separate task-specific set, run multiple seeds, and report error bars. That is an afternoon of compute, not a redesign.\n\nWho this is for: people working on synthetic data for VLMs and anyone who cares about evaluation hygiene in data-centric ML. It deserves a serious referee—the idea is good and the flaws are addressable—but the authors need to close the leakage path and correct the control-task claim before publication.","headline":"Sensible synthetic-data pipeline, but headline gains for ChartQA and SpatialMap rest on validation-set leakage; fix the eval protocol and the paper becomes much more credible.","tokens_in":18715,"tokens_out":2346,"would_cite":false,"duration_ms":22801,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthetic task-specific data from a stronger VLM lifts small vision-language models by up to 29% on specialized tasks.","keywords":["vision-language models","synthetic data generation","data curation","task-specific fine-tuning","in-context learning","perplexity filtering","chart understanding","spatial reasoning"],"falsifier":"Re-run the MM-GEN pipeline on ChartQA and SpatialMap with reference samples drawn from the training split or from a separate task-survey dataset, evaluate on a truly held-out test set, and compare the gains to the paper's reported 15% and 29%; if the gains shrink substantially, the validation-set conditioning is the source of the improvement.","tokens_in":17718,"feed_emoji":"🤖","tokens_out":12067,"duration_ms":87018,"temperature":0.7,"pith_summary":"The paper argues that the main reason small vision-language models underperform on specialized tasks such as chart reading, diagram understanding, and spatial reasoning is the training text: generic captions omit the specific details the task requires. It proposes MM-GEN, an automated three-stage pipeline that uses a stronger VLM to generate task-relevant question-answer pairs for candidate images, with a small set of reference examples from the target task specifying what counts as relevant. Fine-tuning Llava-1.5 (7B) on this synthetic data yields absolute accuracy gains of 15% on chart understanding, 14% on diagram understanding, and 29% on spatial reasoning, and on AI2D the gain is 1.6 times that of human-curated i.i.d. training data. If these numbers hold, task-specific synthetic data can substitute for expensive human curation across specialized multimodal benchmarks, at the cost of only a handful of reference examples and a pool of candidate images.","feed_headline":"Synthetic task data lifts small VLMs by 29%","feed_subtitle":"Examples from a stronger model guide it to write task-specific training text, beating human-curated data on benchmarks.","key_machinery":"The load-bearing mechanism is reference-sample-conditioned in-context generation: a stronger VLM is given one image-question-answer triple from the task's reference set and asked to generate similar, diverse question-answer pairs for a new candidate image. The reference example, rather than a hand-written task description, is what communicates which visual details matter, and the text is generated separately for each image-type subgroup obtained via CLIP partitioning. The second mechanism is a midpoint-perplexity filter: using the target VLM's own masked-language-modeling perplexity, MM-GEN discards low-perplexity examples the model already knows and high-perplexity examples that are likely noisy, keeping the middle band that adds new training signal.","core_discovery":"MM-GEN's central discovery is that the target task can be specified to a teacher VLM data-centrically, by showing it one reference image-question-answer example, rather than by writing a natural-language task description. The pipeline first partitions both the reference set and the candidate image pool into subgroups by image type using CLIP zero-shot classification, so that each generated example is conditioned on a reference of the same type. It then prompts a stronger VLM (GPT-4o) with a reference sample and a candidate image to produce a diverse set of task-specific questions and answers, and finally filters the generated pairs by the perplexity of the target VLM, retaining the middle-perplexity band that represents non-trivial but learnable examples. The paper reports that models fine-tuned on the resulting data improve by 15%, 14%, and 29% absolute on ChartQA, AI2D, and SpatialMap respectively, outperform task-agnostic captions and natural-language-description baselines, and match or exceed the gains of human-curated i.i.d. training data on AI2D.","pith_inferences":["The reference samples for ChartQA and SpatialMap are taken from the validation sets of those benchmarks, and accuracy is reported on the same benchmarks; the reported gains may therefore partly reflect conditioning the synthetic training distribution on the evaluation distribution, and a cleaner estimate would draw references from a separate split.","The approach could be tested on other detail-hungry multimodal tasks, such as medical imaging or diagrams in specialized documents, where reference examples exist but human-curated task data is scarce.","The perplexity filter's success suggests that the target model's own prediction confidence is a useful selection signal; this principle could be explored for self-training or curriculum design, where the model generates its own training examples and selects them by its own uncertainty.","Because the teacher is a frontier model (GPT-4o), an open question is how the gains scale as the teacher becomes weaker; if a smaller open-source VLM can serve as teacher, the method becomes far more accessible."],"forward_implications":["If the reported gains are real, any specialized VLM task can be improved automatically with just a small reference set and a pool of candidate images, removing the need for manual task-specific dataset construction.","The perplexity filter cuts the generated data volume by up to 50% with no or minimal loss in accuracy, so half as much fine-tuning compute can deliver essentially the same improvement.","The method also improves the larger Llava-1.5 (13B) model and, when data from all three tasks is combined, improves all three benchmarks without degrading performance on a general multimodal benchmark (MMMU), suggesting it can be applied per-task and merged.","The ablation showing that one in-context reference sample works better than three suggests that current VLMs' multi-image limitations make data-centric task specification both cheaper and more effective than scaling the number of demonstrations.","The comparison with natural-language task descriptions indicates that data-centric task specification reliably captures task-relevant details that text descriptions omit, extending the paper's case study on MS COCO captions."],"supporting_citations":[{"why":"ChartQA is the target benchmark for chart understanding; its validation set provides the reference samples and its test set measures the reported gains.","marker":"[32]"},{"why":"Llava-1.5 is the base VLM fine-tuned in all experiments, providing the model whose performance is being improved.","marker":"[29]"},{"why":"GPT-4o is the stronger VLM used to generate the synthetic question-answer pairs.","marker":"[40]"},{"why":"Supplies the perplexity-based data pruning method that MM-GEN adapts for filtering generated examples.","marker":"[31]"},{"why":"CLIP is used for zero-shot grouping of images into the task's image-type subgroups.","marker":"[43]"},{"why":"AI2D is the target benchmark for diagram understanding and provides the reference samples for that task.","marker":"[23]"},{"why":"SpatialMap is the target benchmark for spatial reasoning on maps and provides its validation set as reference samples.","marker":"[55]"}],"fun_headline_variants":["29% VLM gain from example-driven synthetic data","Example-specified data beats human curation for VLM tasks","Mid-perplexity filter picks learnable VLM examples","Task-specific synthetic text yields 29% VLM gain","Reference-image prompted GPT-4o writes task-specific training data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that conditioning the synthetic training data on reference samples drawn from the evaluation benchmarks' validation sets does not inflate the accuracy later reported on those same benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["29% VLM gain from example-driven synthetic data","Example-specified data beats human curation for VLM tasks","Mid-perplexity filter picks learnable VLM examples","Task-specific synthetic text yields 29% VLM gain","Reference-image prompted GPT-4o writes task-specific training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001448,"raw_usage":{"total_tokens":5840,"prompt_tokens":962,"completion_tokens":4878,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":4797}},"tokens_in":578,"tokens_out":4878,"duration_ms":32884,"temperature":1.0,"reasoning_tokens":4797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:39:48.510452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the MM-GEN pipeline on ChartQA and SpatialMap with reference samples drawn from the training split or from a separate task-survey dataset, evaluate on a truly held-out test set, and compare the gains to the paper's reported 15% and 29%; if the gains shrink substantially, the validation-set conditioning is the source of the improvement.","supporting_citations":[{"cited_title":"Joty, and Enamul Hoque","cited_arxiv_id":null,"evidence_quote":"ChartQA is the target benchmark for chart understanding; its validation set provides the reference samples and its test set measures the reported gains."},{"cited_title":"Visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"Llava-1.5 is the base VLM fine-tuned in all experiments, providing the model whose performance is being improved."},{"cited_title":"Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models","cited_arxiv_id":null,"evidence_quote":"SpatialMap is the target benchmark for spatial reasoning on maps and provides its validation set as reference samples."}],"review_version":1}