{"id":"ea048db5-d2fa-4b21-9693-904b0e086522","arxiv_id":"2608.13441","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Edit2TikZ is a 1,548-sample benchmark for instruction-guided scientific figure editing via TikZ code, with eight atomic operations, multi-step edits, visual localization, and human-aligned metrics.","lead":"This paper builds a benchmark of 1,548 scientific figure editing tasks that require AI models to rewrite TikZ drawing code from an image and a text instruction. It evaluates 14 AI models, shows the task remains hard even for the best systems, and demonstrates a two-stage training method that substantially improves small open-source models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's semantic metrics RS/ECS are issued by a closed judge that is itself one of the evaluated models, with only a 100-sample validation and no self-preference test; this needs a targeted bias check before the semantic headline numbers are trusted.","rationale":"Reading in good faith, the objective compilation results independently support the core claim that current models are unreliable and that curriculum training improves compilation (45.35→83.40 for Qwen3.5-4B). I therefore do not think the paper's central claim collapses. However, the paper's real contribution is the edit-correctness measurement, and that measurement is the least secure load-bearing component: every RS/ECS score is produced by a closed judge that is also an evaluated model, with only 100 validation samples, no self-preference test, and no per-model or per-operation agreement analysis. This is exactly the reader's weakest assumption, so I agree with that identification. The proposed concrete test would settle it. If the test shows no bias, the conditional verdict can be lifted; if it shows bias, the semantic rankings and training gains need to be recomputed with a neutral judge. I do not see an additional concern that is more load-bearing: release and error bars are reproducibility issues, but they do not attack the internal validity as directly as judge bias does.","tokens_in":13173,"tokens_out":9848,"duration_ms":108759,"concrete_test":"Take a stratified sample of at least 300 model outputs covering GPT-5.6-Terra, another proprietary model, and the trained Qwen3.5-4B, and score each output twice with the exact §3.5 rubric: once with GPT-5.6-Terra and once with an independent judge (a second strong model or human annotators). Compare per-model mean RS/ECS and the Terra-vs-other-model gap across the two judges. If Terra's own outputs are rated systematically higher by Terra than by the independent judge, or if relative model rankings or the size of the training gain change materially, the biased-judge concern is confirmed and the semantic headline numbers must be recomputed; if the scores agree within the human-human MAE for all groups, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's semantic evaluation (§3.5) is not a minor accessory: RS and ECS are the benchmark's stated contribution, they drive the operation-level conclusions in Figure 4, and the 18.7-point average improvement claim in the abstract is computed from them together with compilation success. All RS/ECS numbers in Tables 3 and 5 come from a single closed judge, GPT-5.6-Terra, selected for the lowest MAE on 100 human-rated samples. That same model is one of the eight proprietary models evaluated in Table 3. The paper never tests whether the judge is invariant to the identity or output style of the model being scored. The validation set is small and noisy: human-human MAE is 19.01 (RS) and 16.18 (ECS) on those same 100 cases, so middle-table differences of a few points are within scoring noise. A self-preference or style bias would not affect the objective compile rates, but it would directly affect the edit-correctness and restoration comparisons, and it could inflate the apparent training gains in Table 5 (e.g., ECS 22.54→37.32, RS 26.10→48.72). Without a bias check, the benchmark's core semantic measurements are unverified at the scale at which they are reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Edit2TikZ, a benchmark of 1,548 instruction-guided scientific-figure editing samples in TikZ, spanning real-world and synthetic cases, with text-only and visual-localization prompts, and step-level annotations over eight atomic edit operations. It proposes two semantic metrics, Restoration Score (RS) and Edit Correctness Score (ECS), claims human alignment based on 100 scored samples, and evaluates 14 MLLMs, finding that proprietary models compile about 75% of outputs on average and that the strongest model still fails on about 12% of samples. The paper then constructs TikZEditMix, a 32,448-sample mixed training set, and reports that two-stage reconstruction-then-editing curriculum training improves Qwen3.5-4B compilation success from 45.35% to 83.40% and raises RS/ECS substantially. Code and data are promised for release.","tokens_in":13466,"tokens_out":3205,"duration_ms":33880,"significance":"If the evaluation framework holds up, Edit2TikZ fills a real gap: existing TikZ benchmarks focus on reconstruction or generation, and chart-editing benchmarks are domain-limited, whereas Edit2TikZ targets diverse scientific-figure editing with compilable TikZ output. The benchmark's objective compilation rate and its separation of non-target preservation (RS) from edit completion (ECS) are useful design choices, and the two-stage curriculum result is a concrete, reproducible training recipe for compact models. The paper also demonstrates the weakness of code-overlap and global-similarity metrics for this task, which is an instructive negative result. The main risk is that the semantic metrics RS and ECS are produced by a closed judge model that is itself one of the evaluated models, with only 100 validation samples and no self-preference check; until that is addressed, the semantic headline numbers should be treated as provisional.","major_comments":[{"comment":"The selected judge, GPT-5.6-Terra, is also one of the eight proprietary models evaluated in Table 3, yet the paper performs no self-preference or style-bias check for the RS/ECS scores. Because these scores drive the abstract's 18.7-point average improvement claim, the operation-level conclusions in Figure 4, and the 'better reconstruction does not yield better editing' finding in §4.2, a targeted bias test is load-bearing. I ask for an experiment showing that judge scores are invariant to the identity or output style of the model being scored—for example, by scoring a fixed set of outputs under blinded versus attributed conditions, or by explicitly comparing judge agreement on GPT-5.6-Terra's own outputs versus other models' outputs at matched human-rated quality.","section":"§3.5, Table 2"},{"comment":"The validation of RS/ECS rests on 100 samples, with human-human MAE of 19.01 (RS) and 16.18 (ECS) and human-AI MAE of 10–13. Table 3 reports differences as small as 2–3 points (e.g., GPT-5.6-Terra vs. GPT-5.6-Sol: 62.29 vs. 63.24 RS; 61.04 vs. 64.10 ECS) without confidence intervals, bootstrap estimates, or multiple judge runs. Given the measured noise, fine-grained rankings among proprietary models and even some open-source comparisons are not statistically supported as reported. The paper should provide uncertainty quantification, and ideally a held-out judge-validation split, before interpreting these point differences.","section":"§3.5, Table 2"},{"comment":"The central training claim—that two-stage curriculum learning yields an 18.7-point average improvement—is computed from compilation success, RS, and ECS on the same 1,548-sample test set with no variance estimate. Since RS and ECS depend on the judge model whose bias is not yet established, and since the test-set numbers are single runs, the magnitude of the claimed improvement is not yet robust. A bootstrap over samples, or at least a repeated-evaluation protocol, would clarify whether the 18.7-point gain is outside the evaluation noise.","section":"§4.4, Table 5"}],"minor_comments":[{"comment":"The text contains a typo: 'figrue editing task' should be 'figure editing task'.","section":"§3.5"},{"comment":"'approximately duplicate' should likely read 'near-duplicate' or 'approximately duplicate samples' for clarity.","section":"§3.2"},{"comment":"The instruction text in the figure contains a duplicated article: 'and a and a horizontal one' should be 'and a horizontal one'.","section":"Figure 6"},{"comment":"The row for vTikZ uses '×' and '3' without explicit column alignment; please clarify the notation for 'Edit types' and 'Scope' so the table is readable without referring back to the caption.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for a benchmark-resource venue, and the objective compile-rate results are solid. My main concern is the evaluation conflict: the semantic judge is itself an evaluated model, and no self-preference test is provided. The authors should be asked to add such a test, to report uncertainty on the semantic scores, and to clarify whether the judge was selected on a held-out split. If those additions are made convincingly, the benchmark and its training findings would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nThe short version: this is a genuinely useful new benchmark for a task nobody has benchmarked properly — instruction-driven editing of general scientific figures into compilable TikZ code — and the training recipe for small models looks effective. The soft spot is exactly where the stress-test note points: the two semantic metrics that carry the headline claims are issued by a closed judge model, GPT-5.6-Terra, which is itself one of the eight models being scored, validated on only 100 samples with no self-preference or style-bias check. That needs fixing before the RS/ECS numbers are trusted.\n\nWhat's actually new: the 1,548-sample benchmark with real and synthetic edits, eight atomic operations, multi-step instructions, and visual localization prompts. DeTikZify, ChartEdit, and the rest either reconstruct or stick to charts. Edit2TikZ covers circuits, geometry, flowcharts, relation graphs. The construction pipeline is described carefully, with pHash dedup, human review, and deterministic checks for synthetic edits. The evaluation framework's intent — separate \"did you preserve the rest\" (RS) from \"did you do the edit\" (ECS) — is the right way to think about editing quality, and DSim's poor correlation with human ratings supports their push for bespoke metrics.\n\nThe training contribution is also real: a 32K-sample TikZEditMix plus reconstruction-then-editing curriculum lifts Qwen3.5-4B compilation from 45% to 83% and improves reconstruction on DaTikZv2. Those numbers are objective and don't depend on the judge.\n\nWhere it's soft. First, the judge. RS and ECS drive the operation-level analysis (Figure 4), the abstract's 18.7-point claim, and the Table 5 training gains. GPT-5.6-Terra scoring its own outputs — and probably other OpenAI models — is an obvious confound. The 100-sample validation is too small to rule out style bias: human-human MAE is 19 points on RS, so a 5-point difference between models is noise. A targeted bias test (e.g., swap judge identity, add a second judge, compare distributions on own vs. foreign outputs) should be a condition for trusting the semantic metrics. Second, no uncertainty estimates anywhere: single runs, no CIs, no multiple seeds for the training. Minor relative to the judge issue. Third, code and data aren't out yet; the comparisons are only as good as the release.\n\nNone of this sinks the paper. The benchmark is a real resource and the objective compile-rate results stand on their own. But the headline \"human-aligned evaluation\" is not yet demonstrated at scale. I'd send it to review with a request for a judge-bias analysis and error bars. Worth a serious referee.","headline":"A genuinely useful benchmark for scientific figure editing, but its two semantic metrics are scored by a closed judge that is itself one of the evaluated models, validated on only 100 samples, so the headline RS/ECS numbers need a bias check before they are trusted.","tokens_in":14011,"tokens_out":2293,"would_cite":true,"duration_ms":22146,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"End-to-end scientific-figure editing in TikZ remains unreliable for all 14 tested multimodal models; a two-stage training recipe lifts a 4B model from 45.35% to 83.40% compilation success.","keywords":["scientific figure editing","TikZ code generation","multimodal large language models","benchmark","edit correctness","restoration score","curriculum learning","image-to-code"],"falsifier":"Take a fresh random sample of model outputs from the benchmark's test set, have independent human annotators score RS and ECS without seeing the AI judge's scores, and compare: if the human-AI mean absolute error is much larger than the reported 10.19 (RS) and 12.82 (ECS), or if swapping the judge model changes model rankings, the claim that the automated scores establish current models' unreliability would be undermined.","tokens_in":12980,"feed_emoji":"📐","tokens_out":9210,"duration_ms":81133,"temperature":0.7,"pith_summary":"This paper argues that editing a scientific figure from a rendered image plus a natural-language instruction, by producing a complete compilable TikZ program, is genuinely harder than existing image-to-TikZ reconstruction or chart-only editing benchmarks capture. It introduces Edit2TikZ, a 1,548-sample benchmark spanning eight atomic edit operations, real and synthetic sources, text-only and visual-localization instructions, and multi-step edits with per-step annotations. Across 14 multimodal models, the paper finds that proprietary models compile only 75.15% of outputs on average and the strongest model still fails on about 12% of samples, while compact models below 9B parameters fail far more often, frequently from incomplete or repetitive programs. The paper also shows that a two-stage curriculum, first reconstructing TikZ from images and then training on editing, lifts Qwen3.5-4B's compilation success from 45.35% to 83.40% and improves its proposed edit-quality metrics by 18.7 points on average. A sympathetic reader would take away that reliable end-to-end figure editing is not yet achieved and that source reconstruction, not instruction understanding alone, is the main bottleneck.","feed_headline":"Benchmark: even the best AI fails 12% of figure edits","feed_subtitle":"1,548 figure-editing tests show no reliable model; staged training lifts a small model from 45% to 83%.","key_machinery":"The load-bearing object is the edit-unit decomposition: every benchmark sample is a tuple $(I_s, q_{\\text{text}}, q_{\\text{vis}}, y^*, I_t, E)$ in which the instruction is broken into a sequence of image-verifiable edit units $E=(e_1,\\ldots,e_m)$, each drawn from eight atomic operation types (style, discrete form, insertion, removal, parametric, reference/binding, spatial/layering, and text/symbol updates). This decomposition makes it possible to score correctness per requested change instead of by whole-image similarity, and it is paired with two complementary scores: Restoration Score, which checks that untouched content is preserved across text/style, objects/relations, and layout/scale, and Edit Correctness Score, which grades each unit on a six-level scale. The same unit structure lets the paper identify which edit operations are hardest and lets the training curriculum separate reconstruction from editing.","core_discovery":"The central discovery is that instruction-guided scientific-figure editing in TikZ is a separate capability from figure reconstruction, and current models do not master it. On 1,548 human-verified samples, the paper measures that eight proprietary models achieve an average 75.15% compilation success, 58.76 Restoration Score, and 59.80 Edit Correctness Score; Gemini-3.1-Pro, the strongest, still fails roughly 12% of samples. Open-source models below 9B parameters are markedly weaker, and their failures are dominated by incomplete document structure and repetition rather than by subtle visual mistakes. The paper further finds that better reconstruction does not automatically yield better editing: on the same successfully rendered samples, scaling Qwen3.5 from 4B to 9B improves Restoration Score but slightly lowers Edit Correctness, and providing the gold source program instead of a reconstructed one raises Edit Correctness by 15 to 27 points across three model sizes. A dedicated two-stage curriculum on a 32,448-sample mixed set (TikZEditMix) improves Qwen3.5-4B compilation success from 45.35% to 83.40% and average scores by 18.7 points, showing that compact models can be substantially improved with targeted training.","pith_inferences":["Because the visual-localization subset overlays red boxes on the source image, the reported numbers likely overstate performance on purely text-specified edits of unlabeled or repeated elements; a text-only version of those 434 samples would probably score lower.","The judge model is itself one of the evaluated models, so its scores could carry a self-preference; re-scoring all outputs with a second judge or a human audit would test whether the ranking and headline gaps hold.","If gold source code improves editing as much as the paper reports, a practical system could combine a strong reconstructor with a separate code editor and compiler feedback; the paper's agentic results suggest this only pays off above a capability threshold.","The fixed point allocation in Restoration Score (30/30/40) is one weighting choice; an extension would be to report an edit-versus-preservation tradeoff curve rather than a single weighted score, since preserving layout may compete with making large structural edits."],"forward_implications":["A model's edit quality on this task cannot be read off code-overlap or global image-similarity metrics; per-edit scoring is needed because cBLEU and TED vary little among proprietary models while Edit Correctness spans nearly 26 points.","Source reconstruction is the main bottleneck: feeding the gold pre-edit program instead of a reconstructed one raises Edit Correctness by 15.04 to 27.23 points across the 4B, 9B, and 27B models, so improving image-to-TikZ reconstruction should be a priority.","Compact models below 9B parameters fail primarily from program-completeness problems, repetition and missing \\end{document}, rather than from isolated TikZ errors; training and inference must address long-program termination.","Two-stage curriculum learning, reconstruct then edit, outperforms mixing all data in one stage on the same 32,448 samples, so staged supervision is itself a transferable recipe for small models.","The hardest edit operations are discrete-form, spatial/layering, reference/binding, and metric/parametric updates; these require coordinated changes to anchors, coordinates, and drawing order, and they retain the largest proprietary-versus-open gap."],"supporting_citations":[{"why":"It supplies the image-to-TikZ reconstruction task and the cBLEU/TED code metrics that Edit2TikZ adopts for comparability.","marker":"(Belouadi, Ponzetto, and Eger 2024)"},{"why":"It supplies the DaTikZ-v3 reconstruction data used in TikZEditMix and the DaTikZv2 external test set for reconstruction evaluation.","marker":"(Belouadi et al. 2025)"},{"why":"It is the chart-editing benchmark whose plotting-library scope Edit2TikZ is designed to go beyond, providing the comparison in Table 1.","marker":"(Zhao et al. 2025a)"},{"why":"It is the geometry-only TikZ reconstruction benchmark that motivates the need for editing across diverse scientific figure types.","marker":"(Sun et al. 2026)"},{"why":"It provides DreamSim, the perceptual similarity used for DSim, which the paper shows correlates poorly with human edit-quality scores.","marker":"(Fu et al. 2023)"},{"why":"It is cited as the basis for the claim that global image metrics cannot assess whether requested edits are completed while non-target content is preserved, motivating RS and ECS.","marker":"(Gao et al. 2026)"},{"why":"It supplies the reconstruct-then-edit protocol that the paper evaluates to isolate reconstruction from editing.","marker":"(Su et al. 2026)"}],"fun_headline_variants":["Best AI still fails 12% of TikZ figure edits","No AI model reliably edits scientific TikZ figures","Staged training lifts small model from 45% to 83%","New 1,548-sample benchmark probes AI figure editing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic judge, GPT-5.6-Terra, scores Restoration Score and Edit Correctness on all 1,548 samples as reliably as it did on the 100 human-validated cases; the paper does not test whether the judge favors itself or whether its agreement holds beyond that validation subset.","fun_headline_variants_meta":{"raw":{"variants":["Best AI still fails 12% of TikZ figure edits","No AI model reliably edits scientific TikZ figures","Staged training lifts small model from 45% to 83%","New 1,548-sample benchmark probes AI figure editing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1709,"prompt_tokens":1114,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":730,"completion_tokens_details":{"reasoning_tokens":526}},"tokens_in":730,"tokens_out":595,"duration_ms":6063,"temperature":1.0,"reasoning_tokens":526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:55:04.579084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of model outputs from the benchmark's test set, have independent human annotators score RS and ECS without seeing the AI judge's scores, and compare: if the human-AI mean absolute error is much larger than the reported 10.19 (RS) and 12.82 (ECS), or if swapping the judge model changes model rankings, the claim that the automated scores establish current models' unreliability would be undermined.","supporting_citations":[{"cited_title":"Evaluating Image Editing with","cited_arxiv_id":null,"evidence_quote":"It is cited as the basis for the claim that global image metrics cannot assess whether requested edits are completed while non-target content is preserved, motivating RS and ECS."}],"review_version":1}