Pith. sign in

REVIEW 4 major objections 4 minor 9 references

VectorEdits: A Dataset and Benchmark for Instruction-Based Editing of Vector Graphics

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces VectorEdits, a dataset of 271,306 SVG pairs with natural-language edit instructions, and shows that current large language models fail to produce accurate, valid edits.

desk verdict A genuinely useful large SVG editing dataset with a clear negative LLM result, but the leaky-instruction issue needs a source-dependence check before the benchmark claims fully land. read the letter →

arxiv 2506.15903 v1 pith:KQD5CV6X submitted 2025-06-18 cs.LG

Josef Kuchař , Marek Kadlčík , Michal Spiegel , Michal Štefánik This is my paper
classification cs.LG
keywords vectorgraphicsSVGinstruction-basededitingdatasetbenchmarklargelanguagemodelsvision-languageCLIPsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VectorEdits is a new public dataset for instruction-guided vector graphics editing: 271,306 pairs of SVG images, each paired with a natural-language instruction describing how to transform the source into the target. The paper argues that this resource makes it possible to train and evaluate models that edit vector graphics from text, a task previously limited to narrow benchmarks and raster images. Using the dataset as a benchmark, the authors show that state-of-the-art large language models struggle: none of the tested models produces edits that outperform simply returning the original image unchanged, and many outputs are syntactically invalid SVGs. The practical stake is lowering the barrier to digital design by letting users customize scalable vector art through ordinary language commands.

What carries the argument

The carrying mechanism is the automated VectorEdits construction pipeline: within-collection image pairing by CLIP similarity with a manually tuned threshold, followed by automatic instruction generation with a locally hosted 70-billion-parameter vision-language model. On the evaluation side, the pipeline rasterizes generated and reference SVGs to 512×512 pixels and scores them with CLIP similarity, DINOv2 similarity, mean squared error, and an invalid-SVG count. The pipeline's role is to convert an unstructured SVG repository into supervision signal that ties a textual edit to a concrete change in vector code.

What would settle it

Sample 200 pairs uniformly at random and ask independent human annotators whether each instruction describes a real visual edit and whether the target would be guessable from the instruction alone; if a substantial fraction, say over 10%, are judged not meaningful or reconstructible from the instruction, the benchmark's ground truth is too weak.

Watch

Extended reading notes

Core claim

The central claim is that a large, diverse, publicly available dataset of SVG edit triples—source image, target image, instruction—can support the emerging task of instruction-based vector editing, and that this task is far from solved. The dataset is built by sampling pairs within stylistically consistent collections from an open vector repository, keeping only pairs whose CLIP similarity exceeds a threshold tuned on human judgment, and generating edit instructions with a vision-language model. The paper's evaluation of several pretrained LLMs on a 2,000-pair test set finds that every tested model fails to exceed the 'return the original image' baseline on CLIP and DINOv2 similarity, and produces a non-trivial number of invalid SVG files. The authors therefore present VectorEdits both as a training resource and as a benchmark showing current models lack reliable spatial reasoning and SVG code generation for this task.

Load-bearing premise

The automated pipeline's CLIP similarity threshold and vision-language instructions reliably produce meaningful source-instruction-target triples for all 271,306 pairs, despite the paper conceding that some pairings are noisy and some instructions are detailed enough to generate the target from scratch.

Editorial extensions

If this is right

  • Training or fine-tuning on VectorEdits should give models a large, style-diverse set of source-instruction-target triples to learn spatial reasoning and SVG code edits from.
  • The collection-based train/test split means a model that memorizes styles will not trivially help at test time; measured gains should reflect generalization to unseen visual styles.
  • The no-edit baseline will remain a strong reference: any model claiming progress on this benchmark should beat returning the source unchanged on semantic similarity metrics while maintaining valid SVG syntax.
  • Public release of the data lets the field compare models on the same set of 2,000 test instructions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because instructions are machine-generated, some may be too detailed or noisy; a model trained on them could learn to copy from the instruction rather than truly edit the source, so human-filtered subsets may be needed for reliable training signal.
  • The failure of current LLMs suggests the bottleneck is jointly spatial localization and code generation; progress might come from models that first localize the edited region as a mask over the SVG tree and then generate a minimal patch.
  • One could use the same pairing-with-instruction pipeline on other structured formats, such as CAD drawings or UI layouts, since it is not SVG-specific.
  • A stronger test of the benchmark's validity would be to compare human-written instructions against machine-generated ones on a subsample, quantifying the noise the paper's Limitations concede.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces VectorEdits, a dataset of over 270,000 (source SVG, target SVG, natural-language instruction) triples derived from SVG Repo collections. Pairs are selected by CLIP similarity within style-consistent collections using a threshold tuned on 30 manually reviewed collections; instructions are generated by Qwen2.5-VL 70B after a small human ranking. The paper also benchmarks six LLMs, reporting CLIP, DINOv2, MSE, and invalid-SVG counts, and finds that all models fail to beat a no-edit baseline on the semantic similarity metrics. The limitations section acknowledges noisy pairings, inconsistent instructions, and instructions that may reveal the target from scratch.

Significance. If the dataset is reliable, it would be a valuable public resource: it is large, openly released, derived from diverse vector collections, and addresses a task (instruction-based SVG editing) with only narrow prior benchmarks. The negative LLM result, if valid, would establish that the task is unsolved and provide a useful challenge. The paper has methodological strengths: the pairing threshold is manually calibrated across 30 collections, the metric suite includes DINOv2 and MSE in addition to CLIP, and instruction generation uses a locally hostable model, aiding reproducibility. However, the significance hinges on the source-dependence of the instructions and on the interpretability of the benchmark numbers, both of which are currently unresolved.

major comments (4)
  1. [§3.2 and Limitations] Section 3.2 and the Limitations concede that 'some instructions may contain excessive detail, effectively allowing a model to generate the edited SVG from scratch without relying on the source image.' This is not a minor caveat: if a substantial fraction of instructions are self-contained descriptions of the target, the benchmark measures text-to-SVG generation rather than source-conditioned editing, and the no-edit baseline comparison in Table 2 is misleading. The manual ranking in Table 1 evaluated instruction accuracy and relevance, not source-dependence, so it provides no estimate of the leakage rate. The paper should quantify this: sample 100–200 test pairs, ask human annotators (or a strong VLM) to reconstruct the target from the instruction alone, and report the success rate; then either filter leaky instructions or report metrics separately for source-dependent and leaky subsets. This is load-bearing for the central claim that the dataset enables training and evaluation of models that modify vector graphics based on textual commands.
  2. [§3.1 and §4.1] The dataset construction in Section 3.1 selects source-target pairs so that CLIP similarity exceeds a threshold, while Section 4.1 uses CLIP as the primary evaluation metric. Consequently, the no-edit baseline is favored by construction: the target is by design CLIP-similar to the source. The result in Table 2 that all models have lower CLIP than no-edit is therefore partly an artifact and should not be stated in Section 4.2 as evidence that models failed to surpass the baseline strategy tout court. Moreover, GPT-4o mini achieves lower MSE (8,526) than no-edit (10,488), so the blanket statement is contradicted by the paper's own Table 2. Please analyze the three metrics separately, report the distribution of source-target similarity on the test set, and quantify how many test pairs involve near-identical source and target, making no-edit artificially strong.
  3. [§4.2 and Table 2] Table 2 lacks essential protocol details: the exact prompt template, maximum SVG/token lengths, sampling temperature, and how invalid SVGs are treated in the CLIP/DINOv2/MSE computations (e.g., rasterized as a white image or excluded). Without these, the differences among GPT-4o mini, Gemini 2.0 Flash, Llama 4 Maverick, Gemma 3, and DeepSeek V3 are not interpretable, and the benchmark is not reproducible. Please report per-model prompts, post-processing steps, invalid-output handling, and confidence intervals or bootstrap errors for the 2,000-pair test set.
  4. [§3.1 and Figure 2] The similarity threshold is tuned on 30 collections and set to avoid false positives in that annotated set, but the final 271,306 pairs are not validated. The Limitations acknowledge that the approach may 'occasionally pair images with less meaningful or ambiguous transformations.' The paper should report the distribution of pair counts per collection, the fraction of pairs with similarity near the threshold, and ideally a small human validation of randomly sampled final pairs. Without this, the error rate in the core resource is unknown and the dataset's training utility is not established.
minor comments (4)
  1. [Table 1] The statement that the models were 'ordered from best (1) to worst (3)' is imprecise because Qwen2.5-VL (1.51) and Llama 4 Maverick (1.54) are nearly tied; the tie and the selection rationale should be acknowledged in the text.
  2. [§4.1] The claim that 'Following the findings of the Starvector (Rodriguez et al., 2023), we emphasize that semantic similarity metrics like DINOv2 and CLIP better align with human judgments' should be verified against the cited source or clarified, as the reference does not obviously support a strong claim about DINOv2.
  3. [§3.1 and Figure 2] Figure 2 presents five different similarity measures on different scales (cosine similarity vs. absolute error); the caption should state that lower variance is interpreted as higher reliability only after per-method scale normalization, otherwise the comparison is not directly readable.
  4. [§5 and dataset release] The paper should state the license of the generated instructions and of the assembled dataset on the Hugging Face page, in addition to the licenses of the source SVGs, to make downstream use unambiguous.

Circularity Check

2 steps flagged · score 3.0 of 10

Partial circularity: CLIP similarity is used both to select source–target pairs and as a headline evaluation metric, inflating the no-edit baseline; acknowledged instruction leakage further weakens the editing claim.

  1. self definitional [Section 3.1 (Image Pair Sampling) and Section 4.1 (Metrics), Table 2]
    "To sample sensible image pairs, we compared all images within each collection using CLIP-based (Radford et al., 2021) image similarity on the rendered bitmaps. By computing pairwise similarity scores between images, we selected pairs with a similarity above a defined threshold ... We then use several metrics including Mean Squared Error (MSE), DINOv2 similarity, and CLIP score to compare the generated images against ground truth edited images. ... Baseline – no edit uses the original unedited image as the output."

    The dataset construction selects source–target pairs specifically so that CLIP(source, target) is high, above a manually tuned threshold. The no-edit baseline outputs the source image, so its CLIP similarity to the target is inflated by the same selection criterion. Consequently, the observation that all tested models fail to beat the no-edit baseline on CLIP is partly forced by construction, not purely by model deficiency. The DINOv2 and MSE metrics are not affected by this selection, since DINOv2 was only tested as an alternative and MSE is pixel-based, so the paper's overall conclusion retains independent support.

  2. other [Limitations section]
    "Another limitation is that some instructions may contain excessive detail, effectively allowing a model to generate the "edited" SVG from scratch without relying on the source image."

    The benchmark is intended to measure instruction-guided vector image editing, defined as modifying a source graphic based on a textual instruction. If an instruction is detailed enough to specify the target completely, then the source SVG is superfluous and the instance reduces to text-to-SVG generation rather than source-conditioned editing. The authors explicitly acknowledge this, but it is a construction-level property of the automatically generated instructions: for such pairs, the central editing claim is not actually being measured. This weakens the benchmark's interpretation, though it is a data-leakage concern rather than a fitted-input circularity.

full rationale

I found no load-bearing self-citation or imported uniqueness theorem: all references are external works, and no argument reduces to the authors' prior claims. The principal circularity is the overlap between CLIP as the pair-selection similarity (Section 3.1) and CLIP as a primary evaluation metric (Section 4.1): because target pairs are chosen to score high under CLIP, the no-edit baseline's high CLIP score is partly by construction. This does not alone determine the headline negative result, because DINOv2 similarity also shows a wide gap between the no-edit baseline and all models, and MSE is independent of semantic embedding selection. The acknowledged instruction-leakage limitation is a genuine threat to the editing interpretation, but the paper states it openly rather than hiding it. Overall, the dataset's construction and evaluation share a metric, creating a partial, design-level circularity that inflates one baseline, but the central claim that current LLMs struggle with vector editing retains independent support from non-CLIP metrics.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions about CLIP similarity capturing meaningful edits, VLM instructions being accurate, raster-based semantic metrics reflecting quality, and collection-level splits preventing leakage. The only explicitly fitted number is the CLIP threshold, tuned by hand on 30 collections; no other free parameters are reported. No invented entities are introduced.

free parameters (1)
  • CLIP similarity threshold = not reported
    Tuned manually on 30 collections to select meaningful editing pairs; the final threshold value is not given in the paper.
assumptions (4)
  • domain assumption CLIP similarity between rendered SVGs identifies meaningful editing pairs within a collection.
    Used in Section 3.1 to sample all 271,306 pairs; not independently validated on the full dataset.
  • domain assumption Automatically generated instructions from Qwen2.5-VL accurately describe the transformation between paired images.
    Section 3.2; model chosen by ranking on 100 pairs, but no large-scale human validation; limitations admit noise and overly detailed instructions.
  • domain assumption Raster-based semantic metrics such as CLIP and DINOv2 reflect edit quality better than pixel metrics.
    Section 4.1; adopted from Starvector; no human study on this dataset.
  • domain assumption Splitting by whole collections prevents style leakage between train and test sets.
    Section 3; plausible, but no analysis of remaining semantic overlap is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VectorEdits: A Dataset and Benchmark for Instruction-Based Editing of Vector Graphics." pith.science (2026). https://pith.science/paper/KQD5CV6X

@misc{pith2026250615903,
  author       = {Pith},
  title        = {Pith review of: VectorEdits: A Dataset and Benchmark for Instruction-Based Editing of Vector Graphics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KQD5CV6X}},
  note         = {Machine review of arXiv:2506.15903}
}
read the original abstract

We introduce a large-scale dataset for instruction-guided vector image editing, consisting of over 270,000 pairs of SVG images paired with natural language edit instructions. Our dataset enables training and evaluation of models that modify vector graphics based on textual commands. We describe the data collection process, including image pairing via CLIP similarity and instruction generation with vision-language models. Initial experiments with state-of-the-art large language models reveal that current methods struggle to produce accurate and valid edits, underscoring the challenge of this task. To foster research in natural language-driven vector graphic generation and editing, we make our resources created within this work publicly available.

Figures

Figures reproduced from arXiv: 2506.15903 by the authors.

Figure 1
Figure 1. Examples of instruction-guided vector image [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Histogram of CLIP similarity scores for the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. Distribution of manually selected similarity [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Failed edit attempts by several models on [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 2 canonical work pages

  1. [1]

    https://www.svgrepo.com/

    SVG Repo - free svg vectors and icons --- svgrepo.com. https://www.svgrepo.com/. [Accessed 19-05-2025]

    work page 2025

  2. [2]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2022. https://doi.org/10.48550/ARXIV.2211.09800 Instructpix2pix: Learning to follow image editing instructions . arXiv preprint

    arXiv doi:10.48550/arxiv.2211.09800 2022

  3. [3]

    Kunato Nishina and Yusuke Matsui. 2024. https://doi.org/10.48550/ARXIV.2404.13710 Svgeditbench: A benchmark dataset for quantitative assessment of llm's svg editing capabilities . arXiv preprint

    arXiv doi:10.48550/arxiv.2404.13710 2024

  4. [4]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, and 7 others. 2023. https://doi.org/10.48550/ARXIV.2304.07193 Di...

    arXiv doi:10.48550/arxiv.2304.07193 2023

  5. [5]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. https://doi.org/10.48550/ARXIV.2103.00020 Learning transferable visual models from natural language supervision . arXiv preprint

    arXiv doi:10.48550/arxiv.2103.00020 2021

  6. [6]

    Rodriguez, Abhay Puri, Shubham Agarwal, Issam H

    Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. 2023. https://doi.org/10.48550/ARXIV.2312.11556 Starvector: Generating scalable vector graphics code from images and text . arXiv preprint

    arXiv doi:10.48550/arxiv.2312.11556 2023

  7. [7]

    Bocheng Zou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. 2024. https://aclanthology.org/2024.emnlp-main.213 VGB ench: Evaluating large language models on vector graphics understanding and generation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3647--3659, Miami, Florida, USA. Association for Computational...

    work page 2024

  8. [8]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

Show all 9 references
  1. [9]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.