Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read End-to-end scientific-figure editing in TikZ remains unreliable for all 14 tested multimodal models; a two-stage training recipe lifts a 4B model from 45.35% to 83.40% compilation success.

desk verdict A genuinely useful benchmark for scientific figure editing, but its two semantic metrics are scored by a closed judge that is itself one of the evaluated models, validated on only 100 samples, so the headline RS/ECS numbers need a bias check before they are trusted. read the letter →

arxiv 2608.13441 v1 pith:S3EUDRCN submitted 2026-08-13 cs.CV

classification cs.CV
keywords scientificfigureeditingTikZcodegenerationmultimodallargelanguagemodelsbenchmarkeditcorrectnessrestorationscorecurriculumlearningimage-to-code
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that editing a scientific figure from a rendered image plus a natural-language instruction, by producing a complete compilable TikZ program, is genuinely harder than existing image-to-TikZ reconstruction or chart-only editing benchmarks capture. It introduces Edit2TikZ, a 1,548-sample benchmark spanning eight atomic edit operations, real and synthetic sources, text-only and visual-localization instructions, and multi-step edits with per-step annotations. Across 14 multimodal models, the paper finds that proprietary models compile only 75.15% of outputs on average and the strongest model still fails on about 12% of samples, while compact models below 9B parameters fail far more often, frequently from incomplete or repetitive programs. The paper also shows that a two-stage curriculum, first reconstructing TikZ from images and then training on editing, lifts Qwen3.5-4B's compilation success from 45.35% to 83.40% and improves its proposed edit-quality metrics by 18.7 points on average. A sympathetic reader would take away that reliable end-to-end figure editing is not yet achieved and that source reconstruction, not instruction understanding alone, is the main bottleneck.

What carries the argument

The load-bearing object is the edit-unit decomposition: every benchmark sample is a tuple $(I_s, q_{\text{text}}, q_{\text{vis}}, y^*, I_t, E)$ in which the instruction is broken into a sequence of image-verifiable edit units $E=(e_1,\ldots,e_m)$, each drawn from eight atomic operation types (style, discrete form, insertion, removal, parametric, reference/binding, spatial/layering, and text/symbol updates). This decomposition makes it possible to score correctness per requested change instead of by whole-image similarity, and it is paired with two complementary scores: Restoration Score, which checks that untouched content is preserved across text/style, objects/relations, and layout/scale, and Edit Correctness Score, which grades each unit on a six-level scale. The same unit structure lets the paper identify which edit operations are hardest and lets the training curriculum separate reconstruction from editing.

What would settle it

Take a fresh random sample of model outputs from the benchmark's test set, have independent human annotators score RS and ECS without seeing the AI judge's scores, and compare: if the human-AI mean absolute error is much larger than the reported 10.19 (RS) and 12.82 (ECS), or if swapping the judge model changes model rankings, the claim that the automated scores establish current models' unreliability would be undermined.

Watch

Extended reading notes

Core claim

The central discovery is that instruction-guided scientific-figure editing in TikZ is a separate capability from figure reconstruction, and current models do not master it. On 1,548 human-verified samples, the paper measures that eight proprietary models achieve an average 75.15% compilation success, 58.76 Restoration Score, and 59.80 Edit Correctness Score; Gemini-3.1-Pro, the strongest, still fails roughly 12% of samples. Open-source models below 9B parameters are markedly weaker, and their failures are dominated by incomplete document structure and repetition rather than by subtle visual mistakes. The paper further finds that better reconstruction does not automatically yield better editing: on the same successfully rendered samples, scaling Qwen3.5 from 4B to 9B improves Restoration Score but slightly lowers Edit Correctness, and providing the gold source program instead of a reconstructed one raises Edit Correctness by 15 to 27 points across three model sizes. A dedicated two-stage curriculum on a 32,448-sample mixed set (TikZEditMix) improves Qwen3.5-4B compilation success from 45.35% to 83.40% and average scores by 18.7 points, showing that compact models can be substantially improved with targeted training.

Load-bearing premise

The load-bearing premise is that the automatic judge, GPT-5.6-Terra, scores Restoration Score and Edit Correctness on all 1,548 samples as reliably as it did on the 100 human-validated cases; the paper does not test whether the judge favors itself or whether its agreement holds beyond that validation subset.

Editorial extensions

If this is right

  • A model's edit quality on this task cannot be read off code-overlap or global image-similarity metrics; per-edit scoring is needed because cBLEU and TED vary little among proprietary models while Edit Correctness spans nearly 26 points.
  • Source reconstruction is the main bottleneck: feeding the gold pre-edit program instead of a reconstructed one raises Edit Correctness by 15.04 to 27.23 points across the 4B, 9B, and 27B models, so improving image-to-TikZ reconstruction should be a priority.
  • Compact models below 9B parameters fail primarily from program-completeness problems, repetition and missing \end{document}, rather than from isolated TikZ errors; training and inference must address long-program termination.
  • Two-stage curriculum learning, reconstruct then edit, outperforms mixing all data in one stage on the same 32,448 samples, so staged supervision is itself a transferable recipe for small models.
  • The hardest edit operations are discrete-form, spatial/layering, reference/binding, and metric/parametric updates; these require coordinated changes to anchors, coordinates, and drawing order, and they retain the largest proprietary-versus-open gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the visual-localization subset overlays red boxes on the source image, the reported numbers likely overstate performance on purely text-specified edits of unlabeled or repeated elements; a text-only version of those 434 samples would probably score lower.
  • The judge model is itself one of the evaluated models, so its scores could carry a self-preference; re-scoring all outputs with a second judge or a human audit would test whether the ranking and headline gaps hold.
  • If gold source code improves editing as much as the paper reports, a practical system could combine a strong reconstructor with a separate code editor and compiler feedback; the paper's agentic results suggest this only pays off above a capability threshold.
  • The fixed point allocation in Restoration Score (30/30/40) is one weighting choice; an extension would be to report an edit-versus-preservation tradeoff curve rather than a single weighted score, since preserving layout may compete with making large structural edits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Edit2TikZ, a benchmark of 1,548 instruction-guided scientific-figure editing samples in TikZ, spanning real-world and synthetic cases, with text-only and visual-localization prompts, and step-level annotations over eight atomic edit operations. It proposes two semantic metrics, Restoration Score (RS) and Edit Correctness Score (ECS), claims human alignment based on 100 scored samples, and evaluates 14 MLLMs, finding that proprietary models compile about 75% of outputs on average and that the strongest model still fails on about 12% of samples. The paper then constructs TikZEditMix, a 32,448-sample mixed training set, and reports that two-stage reconstruction-then-editing curriculum training improves Qwen3.5-4B compilation success from 45.35% to 83.40% and raises RS/ECS substantially. Code and data are promised for release.

Significance. If the evaluation framework holds up, Edit2TikZ fills a real gap: existing TikZ benchmarks focus on reconstruction or generation, and chart-editing benchmarks are domain-limited, whereas Edit2TikZ targets diverse scientific-figure editing with compilable TikZ output. The benchmark's objective compilation rate and its separation of non-target preservation (RS) from edit completion (ECS) are useful design choices, and the two-stage curriculum result is a concrete, reproducible training recipe for compact models. The paper also demonstrates the weakness of code-overlap and global-similarity metrics for this task, which is an instructive negative result. The main risk is that the semantic metrics RS and ECS are produced by a closed judge model that is itself one of the evaluated models, with only 100 validation samples and no self-preference check; until that is addressed, the semantic headline numbers should be treated as provisional.

major comments (3)
  1. [§3.5, Table 2] The selected judge, GPT-5.6-Terra, is also one of the eight proprietary models evaluated in Table 3, yet the paper performs no self-preference or style-bias check for the RS/ECS scores. Because these scores drive the abstract's 18.7-point average improvement claim, the operation-level conclusions in Figure 4, and the 'better reconstruction does not yield better editing' finding in §4.2, a targeted bias test is load-bearing. I ask for an experiment showing that judge scores are invariant to the identity or output style of the model being scored—for example, by scoring a fixed set of outputs under blinded versus attributed conditions, or by explicitly comparing judge agreement on GPT-5.6-Terra's own outputs versus other models' outputs at matched human-rated quality.
  2. [§3.5, Table 2] The validation of RS/ECS rests on 100 samples, with human-human MAE of 19.01 (RS) and 16.18 (ECS) and human-AI MAE of 10–13. Table 3 reports differences as small as 2–3 points (e.g., GPT-5.6-Terra vs. GPT-5.6-Sol: 62.29 vs. 63.24 RS; 61.04 vs. 64.10 ECS) without confidence intervals, bootstrap estimates, or multiple judge runs. Given the measured noise, fine-grained rankings among proprietary models and even some open-source comparisons are not statistically supported as reported. The paper should provide uncertainty quantification, and ideally a held-out judge-validation split, before interpreting these point differences.
  3. [§4.4, Table 5] The central training claim—that two-stage curriculum learning yields an 18.7-point average improvement—is computed from compilation success, RS, and ECS on the same 1,548-sample test set with no variance estimate. Since RS and ECS depend on the judge model whose bias is not yet established, and since the test-set numbers are single runs, the magnitude of the claimed improvement is not yet robust. A bootstrap over samples, or at least a repeated-evaluation protocol, would clarify whether the 18.7-point gain is outside the evaluation noise.
minor comments (4)
  1. [§3.5] The text contains a typo: 'figrue editing task' should be 'figure editing task'.
  2. [§3.2] 'approximately duplicate' should likely read 'near-duplicate' or 'approximately duplicate samples' for clarity.
  3. [Figure 6] The instruction text in the figure contains a duplicated article: 'and a and a horizontal one' should be 'and a horizontal one'.
  4. [Table 1] The row for vTikZ uses '×' and '3' without explicit column alignment; please clarify the notation for 'Edit types' and 'Scope' so the table is readable without referring back to the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark, evaluation, and training claims are empirical and do not reduce to their own inputs.

full rationale

The paper's central claims are an empirical benchmark, an automatic evaluation framework, and a training recipe. The RS and ECS metrics are explicitly defined measurement protocols with hand-specified weights and score levels; they are not derived from, nor do they determine, the reported model outputs. The 'human-aligned' property is supported by a 100-sample agreement study against human raters, and the choice of GPT-5.6-Terra as judge is a calibration decision, not a fitted parameter that forces any particular model ranking or training gain. The reported scores are outputs of an external closed judge applied to independently generated model outputs, so they are not by construction equal to the benchmark definitions or the training objective. The training improvements are measured through a shared TeX compilation environment, the benchmark's own judge, and the external DaTikZv2 test set; the paper also states that training data have no identifier or source-URI overlap with the 1,548-sample test set. No load-bearing step depends on a self-citation or on a uniqueness theorem imported from the authors. The concern that the judge model is itself one of the evaluated models and might exhibit self-preference or style bias is a measurement-validity threat, not a circularity: no equation or definition reduces a reported result to its own input. Accordingly, the derivation chain is self-contained and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 4 invented entities

The benchmark and its metrics are the paper's main contribution. The free parameters are the rubric weights and score levels, which are hand-chosen. The axioms are the unverifiable assumptions about data extraction, human labeling, and judge generalization.

free parameters (2)
  • RS component weights = 30 (text/style), 30 (objects/relations), 40 (layout/scale)
    Chosen by hand in Section 3.5 to weight preservation dimensions; no optimization against data is described.
  • ECS score levels = {0, 40, 80, 90, 95, 100}
    Discrete quality levels defined in Section 3.5 to map judge ratings to numeric scores.
assumptions (3)
  • domain assumption Extracted TikZ blocks from arXiv are self-contained and compile in the shared TeX Live environment.
    Data collection in Section 3.2 relies on this to build the source pool; blocks that do not compile would be excluded, biasing the pool toward simple, self-contained programs.
  • domain assumption Human reviewers correctly verify instruction quality, edit-unit decomposition, and visual prompt accuracy.
    Sections 3.2 and 3.3 state human review without reporting inter-annotator agreement for edit-unit labels; the ECS gold standard depends on these labels.
  • domain assumption GPT-5.6-Terra's RS and ECS ratings remain as accurate on all 1,548 samples as they were on the 100-sample validation set.
    Section 3.5 validates the judge on 100 cases; the full benchmark scores assume this alignment generalizes.
invented entities (4)
  • Edit2TikZ benchmark
    purpose: Standardized evaluation of instruction-guided scientific figure editing in TikZ
    Data and code are only promised, not released, so no independent check is currently possible.
  • Restoration Score (RS)
    purpose: Measure non-target preservation in edited figures
    The metric is validated only on 100 human-rated samples within the same paper.
  • Edit Correctness Score (ECS)
    purpose: Measure completion of each requested edit unit
    Same internal validation; no external benchmark uses it.
  • TikZEditMix training set
    purpose: Mixed reconstruction and editing data for training compact models
    Not released; claimed but unverifiable from the preprint.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ." pith.science (2026). https://pith.science/paper/S3EUDRCN

@misc{pith2026260813441,
  author       = {Pith},
  title        = {Pith review of: Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3EUDRCN}},
  note         = {Machine review of arXiv:2608.13441}
}
read the original abstract

Although multimodal large language models (MLLMs) have shown substantial potential in visual understanding and graphic code generation, editing scientific figures through code presents a greater challenge: a model must jointly recover visual structure, ground the requested change, generate compilable code, and preserve all unrelated content. While existing TikZ benchmarks mainly focus on figure reconstruction and generation, few systematically evaluate instruction-guided scientific figure editing with compilable code. We introduce Edit2TikZ, a comprehensive benchmark for scientific figure editing tasks, featuring 1,548 diverse and high-quality samples. Edit2TikZ combines real-world and controlled synthetic edit cases, supports both textual and visual localization request, and contains multi-step editing, each with step-level annotations. We further construct a human-aligned evaluation framework to measure whether a requested edit is completed while irrelevant content is preserved. Utilizing Edit2TikZ, we evaluate 14 mainstream MLLMs and find that current systems remain unreliable: on average, proprietary models achieve a compilation success rate of merely 75% and remain limited in both figure restoration and edit correctness, while compact models below 9B struggle further with instruction following and complete figure generation. Therefore, we build a mixed training set TikZEditMix and adopt reconstruction-then-editing curriculum learning for compact models. On Qwen3.5-4B, this training improves the compilation success rate from 45.35% to 83.40% and yields an average improvement of 18.7 points across our proposed evaluation metrics. The code and data will be released at https://github.com/Solunny/Edit2TikZ.

Figures

Figures reproduced from arXiv: 2608.13441 by the authors.

Figure 1
Figure 1. Previous benchmarks mainly focus on figure recon [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples from the real-world, text-only synthetic, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Operation-level ECS for the six open-source and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Compilation success rate, RS, and ECS under [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative examples of successful and failed cases. Models perform well on relatively simple figures with few elements [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 16 canonical work pages

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Detikzify: Synthesizing graphics programs for scientific figures and sketches with tikz , author=. Advances in Neural Information Processing Systems , volume=

  2. [2]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  3. [3]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Chartedit: How far are mllms from automating chart analysis? evaluating mllms’ capability via chart editing , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

  4. [4]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Tikzero: Zero-shot text-guided graphics program synthesis , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  5. [5]

    Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , pages=

    Llm code customization with visual results: A benchmark on tikz , author=. Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , pages=

  6. [6]

    arXiv preprint arXiv:2502.19453 , year=

    SVGEditBench V2: A Benchmark for Instruction-Based SVG Editing , author=. arXiv preprint arXiv:2502.19453 , year=

  7. [7]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  8. [8]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Charteditor: A reinforcement learning framework for robust chart editing , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Show all 38 references
  1. [9]

    arXiv preprint arXiv:2602.12125 , year=

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation , author=. arXiv preprint arXiv:2602.12125 , year=

  2. [10]

    arXiv preprint arXiv:2306.09344 , year=

    Dreamsim: Learning new dimensions of human visual similarity using synthetic data , author=. arXiv preprint arXiv:2306.09344 , year=

  3. [11]

    Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

    Bleu: a method for automatic evaluation of machine translation , author=. Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages=

  4. [12]

    SIAM journal on computing , volume=

    Simple fast algorithms for the editing distance between trees and related problems , author=. SIAM journal on computing , volume=. 1989 , publisher=

  5. [13]

    arXiv preprint arXiv:2604.06079 , year=

    Scientific graphics program synthesis via dual self-consistency reinforcement learning , author=. arXiv preprint arXiv:2604.06079 , year=

  6. [14]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

  7. [15]

    International Conference on Learning Representations , volume=

    Chartmimic: Evaluating lmm's cross-modal reasoning capability via chart-to-code generation , author=. International Conference on Learning Representations , volume=

  8. [16]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Chartcoder: Advancing multimodal large language model for chart-to-code generation , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  9. [17]

    Design2code: Benchmarking multimodal code generation for automated front-end engineering , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

  10. [18]

    arXiv preprint arXiv:2312.11556 , year=

    Starvector: Generating scalable vector graphics code from images and text , author=. arXiv preprint arXiv:2312.11556 , year=

  11. [19]

    arXiv preprint arXiv:2605.11307 , year=

    Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation , author=. arXiv preprint arXiv:2605.11307 , year=

  12. [20]

    2026 , eprint=

    VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing , author=. 2026 , eprint=

  13. [21]

    Advances in Neural Information Processing Systems , volume=

    What matters when building vision-language models? , author=. Advances in Neural Information Processing Systems , volume=

  14. [22]

    arXiv preprint arXiv:2408.12637 , year=

    Building and better understanding vision-language models: insights and future directions , author=. arXiv preprint arXiv:2408.12637 , year=

  15. [23]

    Advances in Neural Information Processing Systems , volume=

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms , author=. Advances in Neural Information Processing Systems , volume=

  16. [24]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    From words to structured visuals: A benchmark and framework for text-to-diagram generation and editing , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  17. [25]

    Proceedings of the 33rd ACM International Conference on Multimedia , pages=

    ChartM3: Benchmarking Chart Editing with Multimodal Instructions , author=. Proceedings of the 33rd ACM International Conference on Multimedia , pages=

  18. [26]

    arXiv preprint arXiv:2602.15758 , year=

    ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Models , author=. arXiv preprint arXiv:2602.15758 , year=

  19. [27]

    Li, Shuo and Sun, Jiajun and Wang, Zhekai and Fan, Xiaoran and Li, Hui and Yang, Dingwen and Xi, Zhiheng and Wang, Yijun and Shan, Zifei and Gui, Tao and others , journal=

  20. [28]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Self-distillation bridges distribution gap in language model fine-tuning , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  21. [29]

    arXiv preprint arXiv:2601.18734 , year=

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author=. arXiv preprint arXiv:2601.18734 , year=

  22. [30]

    arXiv preprint arXiv:2606.10334 , year=

    Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts , author=. arXiv preprint arXiv:2606.10334 , year=

  23. [31]

    arXiv preprint arXiv:2605.03677 , year=

    Uni-opd: Unifying on-policy distillation with a dual-perspective recipe , author=. arXiv preprint arXiv:2605.03677 , year=

  24. [32]

    arXiv preprint arXiv:2607.05184 , year=

    Rethinking On-Policy Self-Distillation for Thinking Models , author=. arXiv preprint arXiv:2607.05184 , year=

  25. [33]

    Evaluating Image Editing with

    Gao, Shiqi and Xu, Zitong and Fu, Kang and Duan, Huiyu and Min, Xiongkuo and Wang, Jia , journal=. Evaluating Image Editing with. 2026 , doi=

  26. [34]

    5-omni technical report , author=

    Qwen3. 5-omni technical report , author=. arXiv preprint arXiv:2604.15804 , year=

  27. [35]

    arXiv preprint arXiv:2607.00248 , year=

  28. [36]

    2026 , howpublished=

  29. [37]

    2025 , howpublished=

    Introducing. 2025 , howpublished=

  30. [38]

    arXiv preprint arXiv:2607.02770 , year=

    Gemma 4 technical report , author=. arXiv preprint arXiv:2607.02770 , year=

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.