{"id":"75c6b037-105e-4ee9-940a-845d2b22ae46","arxiv_id":"2507.21167","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark, ChartM3, shows that multimodal large language models perform worse when editing charts from visual-box guidance than from textual descriptions, and that fine-tuning on 24,000 in-distribution samples substantially raises benchmark scores.","lead":"The paper introduces ChartM3, a benchmark of 1,000 chart-editing tasks that combine text instructions with visual boxes highlighting what to change. It finds current AI models often fail to turn the visual hint into correct code, and training on 24,000 similar tasks improves scores on the benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning gains may be an artifact of training and testing on the same automated generation pipeline; the abstract's 'practical' claim needs an out-of-distribution transfer check.","rationale":"I read the strongest claim as two-part: (1) ChartM3 is a valid benchmark that reveals MLLM limitations, and (2) fine-tuning on ChartM3-Train leads to substantial improvements demonstrating the importance of multimodal supervision. Part 1 is reasonably supported: zero-shot results are consistent across models, the human validation in Table 3 gives high agreement for ΔSSIM and moderate agreement for Compliance, and the paper explicitly acknowledges Matplotlib-only scope and parameter-level instructions. Part 2 is the soft spot. The only evidence for it is Table 2, whose fine-tuning set is generated by the same pipeline as the test set. No held-out transfer, no real-user study, and no paraphrase robustness are reported. The conclusion acknowledges several restrictions, but none of these restrictions is tested empirically as a boundary condition on the fine-tuning gains. Section 3.2 also points to 'Section A of the supplementary material' for details of the data construction pipeline, yet no supplementary material is present in the manuscript, so the pipeline cannot be fully audited. The reader's weakest assumption is on the same axis, although I place more weight on the same-pipeline train/test overlap than on the realism of bounding boxes per se. A single out-of-distribution transfer experiment would settle whether the fine-tuning improvements generalize. This does not change the CONDITIONAL verdict: the benchmark remains a useful synthetic resource, but the practical claim should be conditioned on transfer evidence.","tokens_in":19134,"tokens_out":5009,"duration_ms":52599,"concrete_test":"Collect 100 chart-edit requests from human users, or simulate them by jittering bounding boxes, paraphrasing instructions, and drawing code from sources outside the ChartM3 code pools. Evaluate zero-shot GPT-4o and the fine-tuned Qwen2-VL and Llama-3.2 models from Table 2 on this held-out set with the same ΔSSIM and Compliance metrics. If the fine-tuned models' advantage over zero-shot GPT-4o largely disappears or reverses, the same-pipeline train/test construction is the source of the reported 'substantial improvements,' and the practical-generalization claim should be removed or explicitly restricted to the benchmark distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that multimodal supervision yields substantial improvements for practical chart editing rests on Table 2's fine-tuning results. But Section 3.5 says ChartM3-Train uses 'the same data construction pipeline from Section 3.2' as the ChartM3 test set. Both are produced by the same code pools, modification pools, instruction templates, and GPT-4o filtering. Under this protocol, a model can memorize the pipeline's instruction patterns and code-edit templates; the large gains (Qwen2-VL ΔSSIM 12.59→63.23 text, 6.5→57.88 visual, Table 2) then measure in-distribution pattern learning, not general multimodal chart editing. Section 3.2's automated annotation also derives bounding boxes directly from the code modification, so visual indicators in the benchmark are perfectly tight and unambiguous, another mismatch with real user clicks. The conclusion acknowledges Matplotlib-only and parameter-level instructions, but no experiment shows whether fine-tuned models transfer to paraphrased user requests, noisier clicks, or unfamiliar chart code. Without that transfer evidence, the abstract's 'practical chart editing systems' claim is unsupported; the benchmark itself remains useful as a synthetic diagnostic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ChartM3, a benchmark for multimodal chart editing in which user intent is expressed through natural language plus visual indicators (bounding boxes), alongside a text-only description condition. The dataset contains 1,000 test samples across four complexity levels and 24,000 fine-tuning samples, built from Matplotlib code with automatically generated instruction/annotation pairs. The authors evaluate eight MLLMs zero-shot, report that GPT-4o performs best but still struggles with visual indicators, and show that fine-tuning Qwen2-vl, Llama-3.2-vision, and LLaVA-v1.6 on ChartM3-Train yields large gains on ΔSSIM, Compliance, and Non-intervened Robustness. They also propose ΔSSIM and a GPT-4-based score, validating these against human judgments on 200 samples.","tokens_in":19277,"tokens_out":4147,"duration_ms":37031,"significance":"If the claims hold, ChartM3 provides a useful new task paradigm for chart editing that combines pointing with code generation, and the zero-shot results demonstrate a measurable weakness of current MLLMs in translating visual indicators into code-level edits. The paper's concrete strengths include the public release of datasets, code, and evaluation tools; a multi-metric evaluation framework with explicit human validation (ΔSSIM agreement 0.86); and a large-scale training set that enables a systematic fine-tuning study. The central fine-tuning claim, however, is currently supported only by in-distribution gains on test data generated with the same pipeline, so the 'practical chart editing systems' conclusion needs either additional out-of-distribution evidence or a substantial narrowing of the claim.","major_comments":[{"comment":"Section 3.5 states that ChartM3-Train uses 'the same data construction pipeline from Section 3.2,' and the ChartM3 test set is constructed by that same pipeline. The large fine-tuning gains in Table 2 (e.g., Qwen2-vl ΔSSIM 12.59→63.23 in the textual setting and 6.5→57.88 in the visual setting) therefore measure in-distribution pattern learning rather than generalization to real-world chart editing. The abstract's claim that multimodal supervision is important for 'building practical chart editing systems' requires an out-of-distribution transfer check, such as evaluation on paraphrased user instructions, noisier or less precise bounding boxes, or chart code from a different distribution. Without that evidence, the fine-tuning result should be reported as a within-benchmark diagnostic rather than as evidence of practical capability.","section":"§3.5, Table 2"},{"comment":"The bounding boxes in ChartM3 are generated automatically from the exact code modifications, so the visual indicators are perfectly tight and unambiguous. The footnote in Section 1 says that in practice a bounding box is drawn around the user's click position, but real clicks are noisy and may not align exactly with the intended chart element. The benchmark does not test sensitivity to imperfect indicators, and the conclusion does not acknowledge this mismatch. The authors should either add an experiment that perturbs the bounding boxes (e.g., jitter, partial overlap) and measures the effect on model performance, or explicitly scope the benchmark as a synthetic diagnostic with clean indicators.","section":"§3.2, Automated Annotation"},{"comment":"Table 3 reports human agreement of 0.56 for Non-intervened Robustness, which is substantially lower than the 0.86 for ΔSSIM and 0.70 for Directive Compliance. The text in §5.4 says 'ΔSSIM metric and Compliance score demonstrate high agreement with human judgement' and omits discussion of the Non-intervened dimension, even though the abstract claims a 'multi-perspective evaluation' and the GPT Score includes Non-intervened Robustness. The low agreement suggests that this dimension may not be a reliable automated measure, and the paper should either explain the discrepancy, refine the metric, or de-emphasize it in the evaluation framework.","section":"§5.4, Table 3"}],"minor_comments":[{"comment":"The column header 'iSSIM' in Table 2 does not match the metric name 'ΔSSIM' defined in Section 3.4 and Equation (3); the same typo appears in the text ('highest Compliance score and iSSIM score').","section":"Table 2, Section 3.4"},{"comment":"The abstract describes samples as 'triplets' and says the benchmark contains 1,000 samples, while the Introduction says '1,000 carefully curated quadruplets'; Section 3.1 uses (chart, code, multimodal instruction) in the task definition. Please clarify the exact data structure and terminology.","section":"Abstract and Introduction"},{"comment":"The abstract contains a duplicated release sentence and a second, apparently outdated repository URL (https://github.com/yaolinli/VCE); this should be cleaned up.","section":"Abstract"},{"comment":"The human evaluation uses 200 samples with three assessors but reports no inter-annotator agreement measure; reporting Fleiss' kappa or a similar statistic would strengthen the claim of reliable majority-vote judgments.","section":"§5.4"},{"comment":"The 'Execute Rate' metric is reported in Table 2 but never defined in the metrics section; please specify how execution success is determined (e.g., whether a timeout or exception counts as failure).","section":"Table 2, §3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an ACM Multimedia venue, but the abstract's claim of practical chart editing systems is broader than what the experiments support. The duplicated abstract text and stale URL also suggest a need for a careful proofreading pass before acceptance. The central benchmark contribution is valuable, but the fine-tuning generalization issue should be resolved with additional experiments or a significant claim revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ChartM³ is the first benchmark to treat chart editing as a multimodal instruction-following task—bounding boxes on the chart point at the elements to change while a separate text instruction describes the edit. That task definition is genuinely new relative to the text-only chart editing work the paper cites, and the central zero-shot observation holds up: current MLLMs, including GPT-4o, are measurably worse at acting on visual indicators than on text descriptions of the same targets.\n\nWhat the paper does well: the data pipeline is careful, with ten chart types, four difficulty levels, and an evaluation that pairs a visual metric (ΔSSIM) with GPT-4-based code-correctness checks. The human agreement study (ΔSSIM 0.86 on 200 samples) gives the metric framework real anchor. The zero-shot numbers are internally consistent across eight models.\n\nThe soft spots, in order. First, the headline fine-tuning claim: ChartM³-Train is built with the same automated pipeline as the test set—same code pools, same modification templates, same GPT-4o filtering. The stress-test concern is real; Section 3.5 says so explicitly. The large gains (Qwen2-VL ΔSSIM 12.59→63.23) are therefore evidence of in-distribution pattern learning, not of generalizable multimodal chart editing. The abstract calls these systems 'practical,' but no experiment tests transfer to noisier clicks, paraphrased instructions, or unseen chart code. That is the paper's main gap, and an out-of-distribution transfer set would settle it. Second, the automated bounding boxes derive from the ground-truth code modification, so the visual indicators are perfectly tight and unambiguous—a mismatch with real user pointing that affects the zero-shot conclusions too, in an unknown direction. Minor: there are no variance statistics (each number looks like a single run), and the abstract contains two conflicting GitHub URLs, one pointing at an unrelated repo, so the release is not currently verifiable. The conclusion's limitations section is honest about Matplotlib-only scope and parameter-level instructions; that honesty doesn't fix the abstract's overreach, but it shows the authors know the boundaries.\n\nWho this is for: anyone building or evaluating MLLMs on chart-to-code or visual-grounding-to-code tasks. The benchmark is usable as a synthetic diagnostic today; the fine-tuning generalization story needs another round of work.\n\nMy recommendation: send it to a serious referee. The task definition and zero-shot findings earn referee time, and the concerns are addressable—an OOD holdout, repeated runs with error bars, and a cleaned-up release. With those, the fine-tuning claim can be either confirmed or properly scaled back.","headline":"ChartM³ is a genuinely new benchmark for visually-guided chart editing with solid zero-shot evidence, but the fine-tuning gains are in-distribution until a transfer test says otherwise.","tokens_in":19867,"tokens_out":4130,"would_cite":true,"duration_ms":35314,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that chart editing guided by visual indicators is a distinct task where current multimodal large language models, including GPT-4o, perform poorly, and that fine-tuning on 24,000 curated samples substantially improves…","keywords":["chart editing","multimodal large language models","visual indicators","bounding boxes","benchmark","chart-to-code","fine-tuning","Matplotlib"],"falsifier":"Remove the bounding-box overlay from a random subset of visual-indicator test samples while keeping the same editing instruction, rerun the benchmarked models, and compare $\\Delta\\text{SSIM}$ and Compliance scores; if performance does not drop substantially, the central claim that models fail specifically at interpreting visual indicators is unsupported.","tokens_in":18887,"feed_emoji":"📊","tokens_out":7360,"duration_ms":66944,"temperature":0.7,"pith_summary":"The paper argues that text-only instructions are too ambiguous for fine-grained chart edits, so user intent should be expressed with a combination of language and visual indicators, such as bounding boxes around the elements to change. To support this, it builds ChartM3, a 1,000-sample benchmark of (chart, code, multimodal instruction) triples spanning four difficulty levels and two editing paradigms: text-description-guided and visual-indicator-guided. Evaluated on it, eight multimodal large language models, GPT-4o included, show a consistent weakness in mapping visual indicators to code-level changes. The paper then curates ChartM3-Train, a 24,000-sample training set, and shows that fine-tuning on it produces large gains across both paradigms and reduces the text-visual gap.","feed_headline":"ChartM3: AI models fail at click-guided chart edits","feed_subtitle":"A 1,000-sample benchmark shows MLLMs misread bounding boxes; fine-tuning on 24,000 samples closes much of the gap.","key_machinery":"The load-bearing object is the visual indicator: a bounding box overlaid on the chart image that pinpoints the element to be modified, presented together with a basic task instruction. The task is formalized as $\\hat{c} = M(I, c, T_{desc})$ for text-guided editing and $\\hat{c} = M(I_{vi}, c, T)$ for visual-indicator-guided editing, where $I_{vi}$ is the chart with the indicator drawn on it. Around this, ChartM3 contributes a pipeline that samples chart code, applies one to three modifications, generates dual-format instructions, and validates them with LLM-based quality control. Evaluation is carried by two complementary metrics: $\\Delta\\text{SSIM}$, which normalizes the generated image's structural similarity gain toward the reference by the input's starting similarity, and the GPT Score, which asks an LLM judge for directive compliance and non-intervened robustness. The training counterpart, ChartM3-Train with 24,000 samples, feeds the same dual-format instruction pairs to models through cross-entropy loss on the reference code.","core_discovery":"The central claim is that current multimodal large language models cannot reliably translate a visually indicated target region in a chart into the corresponding code edit, and that this failure is distinct from and larger than the ambiguity of textual description. The benchmark measures this with two parallel tasks on the same underlying samples: textual-description-guided editing, where the model locates targets from phrases like 'the third bar from the left,' and visual-indicator-guided editing, where a bounding box highlights the target. Across zero-shot models, performance on the visual-indicator task is consistently lower, and even GPT-4o, with a 90.3% execution rate on text guidance, drops to 87.2% execution and scores much lower on compliance (76.80 to 63.36). Fine-tuning open models on ChartM3-Train lifts visual-indicator $\\Delta$-SSIM scores from 6.5 to 57.88 in Qwen2-VL and 11.18 to 51.00 in Llama-3.2-vision, surpassing GPT-4o on several metrics. The conclusion is that multimodal supervision, not just more text data, is what builds practical chart editing ability.","pith_inferences":["The authors do not test this, but the same click-to-code paradigm should transfer to other code-rendered graphics, such as SVG, HTML canvas, slide tools, or diagram editors, because any visual element with a programmatic counterpart admits the same bounding-box-to-code mapping.","Real users' clicks and boxes will be noisier and less precisely aligned than the auto-generated ground-truth boxes, so the reported zero-shot gaps and fine-tuning gains are likely optimistic; collecting human click data would quantify that gap.","A testable extension of the paper's asymmetry result is that instruction-tuning datasets for general code generation could cheaply include pointing annotations to boost visual grounding, even when the downstream task is text-only."],"forward_implications":["Chart editing evaluations that rely only on natural-language instructions can hide the visual-to-code failure mode, so benchmarks should include a visual-indicator condition.","Fine-tuning on multimodal supervision improves both text-guided and visual-guided chart editing, with the largest absolute gains in the visual-guided task.","Training on visual indicator data transfers to text-guided tasks better than the reverse, making pointing annotations a high-value training signal.","The proposed $\\Delta\\text{SSIM}$ metric, which measures normalized visual progress toward the target, agrees with human preference at 0.86 and can serve as a visual-progress metric beyond chart editing."],"supporting_citations":[{"why":"Supplies the SSIM metric that Delta-SSIM builds on and normalizes.","marker":"[27]"},{"why":"Provides the strongest zero-shot baseline, GPT-4o, and the GPT-4 judge used for the code-correctness evaluation.","marker":"[18]"},{"why":"ChartCoder is the chart-to-code baseline the benchmark compares against.","marker":"[31]"},{"why":"ChartMimic defines the prior text-guided chart editing and chart-to-code task that the paper extends to visual indicators.","marker":"[30]"},{"why":"Qwen2-VL is the open MLLM used for the main fine-tuning and ablation experiments.","marker":"[26]"},{"why":"Llama-3.2-vision is another open MLLM evaluated zero-shot and after fine-tuning on ChartM3-Train.","marker":"[12]"},{"why":"ChartReformer is an existing natural-language chart image editing approach that motivates the need for visual guidance.","marker":"[29]"}],"fun_headline_variants":["Visual cues stump MLLMs in chart editing","Bounding boxes baffle GPT-4o and friends","ChartM3: Multimodal chart editing benchmark exposes gaps","Fine-tuning fixes chart edit misses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that auto-generated bounding boxes around the exact ground-truth target regions, combined with instructions derived from the same code modifications, represent how real users point at charts and phrase edit requests; if real clicks are noisier or real requests differ, both the measured limitations and the fine-tuning gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Visual cues stump MLLMs in chart editing","Bounding boxes baffle GPT-4o and friends","ChartM3: Multimodal chart editing benchmark exposes gaps","Fine-tuning fixes chart edit misses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1370,"prompt_tokens":1097,"completion_tokens":273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":713,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":713,"tokens_out":273,"duration_ms":3170,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:56:36.116641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove the bounding-box overlay from a random subset of visual-indicator test samples while keeping the same editing instruction, rerun the benchmarked models, and compare $\\Delta\\text{SSIM}$ and Compliance scores; if performance does not drop substantially, the central claim that models fail specifically at interpreting visual indicators is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the strongest zero-shot baseline, GPT-4o, and the GPT-4 judge used for the code-correctness evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ChartReformer is an existing natural-language chart image editing approach that motivates the need for visual guidance."}],"review_version":2}