Pith. sign in

REVIEW 3 major objections 5 minor 19 references

CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CAS2UML pairs 557 hand-drawn UML diagrams with validated PlantUML code, aiming to make sketch-to-UML evaluation reproducible.

desk verdict CAS2UML fills a real gap with a public 557-sample dataset of handwritten UML sketches paired with executable PlantUML, but the 'validated' label is stronger than the single-annotator, code-first pipeline actually supports. read the letter →

arxiv 2608.07036 v1 pith:3FRRI7US submitted 2026-08-07 cs.SE cs.CV

Simon Scholz , Mersedeh Sadeghi This is my paper · ORCID
classification cs.SEcs.CV
keywords datasetUMLhand-drawndiagramsPlantsketchrecognitionbenchmarkclassactivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CAS2UML is a proposed public dataset of 557 hand-drawn UML diagrams—271 class diagrams and 286 activity diagrams—each paired with PlantUML code that the authors manually validated. The paper's aim is to give sketch-to-UML and image-to-model research a shared, executable ground truth, where previous hand-drawn UML resources were small, covered one diagram type, or offered only visual annotations. If the dataset holds up, evaluation stops being a matter of judging image similarity and becomes a check of whether generated UML is syntactically valid, renderable, and reusable in modeling tools. Alongside the data, the paper releases a validation tool and scripts that perform syntax checks and, for class diagrams, XMI export so other researchers can reproduce the same checks.

What carries the argument

The central object is the sketch–PlantUML pair: one handwritten image and one executable text supposed to render to the same diagram structure. The load-bearing mechanism is the validation tool, an interface that shows the sketch next to editable PlantUML code with a live rendered preview, so the annotator can confirm correspondence before release. Machine-checkable support comes from the PlantUML syntax check and, for class diagrams, XMI export, which turn a visual judgment into artifacts other tools can consume.

What would settle it

Render the released PlantUML for every pair and compare the rendered elements with the paired sketch: any class, relation, decision node, fork or join, or swimlane present in one but absent from the other disproves the one-to-one correspondence for that sample. A statistical version is to have independent annotators judge a random subsample of pairs as matching or not; a material non-match rate would refute the claim that the ground truth is validated.

Watch

Extended reading notes

Core claim

The central claim is that CAS2UML is the first public resource combining handwritten class and activity diagrams with manually validated, executable PlantUML ground truth at a scale usable for systematic benchmarking. The pairs were produced by taking reference PlantUML specifications—some derived from existing handwritten images, some newly authored—redrawing them by hand on tablets, plain paper, and checkered paper, and then checking each sketch against its code in a custom side-by-side annotation tool. Every released annotation passes the PlantUML parser's syntax check; class diagrams are also serialized to XMI so they can be opened in UML tools. On the authors' terms, the one-to-one sketch–code correspondence is what makes the dataset a benchmark rather than just a collection of pictures.

Load-bearing premise

The dataset's quality rests on a single annotator's visual judgment that each hand-drawn sketch and its PlantUML code represent the same diagram; there is no second annotator, no inter-annotator agreement, and no metric for semantic equivalence between image and code.

Editorial extensions

If this is right

  • A sketch-to-UML model can be trained or evaluated on 557 paired examples spanning two diagram types, replacing the handful of samples used in earlier studies.
  • Automated syntax and renderability checks make results reproducible: another lab can run the same validation script on any generated PlantUML output.
  • Class-diagram XMI files let generated models be imported into UML modeling tools, so success can be measured by downstream usability rather than image similarity alone.
  • Activity diagrams covering decisions, forks, joins, loops, and swimlanes extend benchmarking from static structure to behavioral control flow.
  • Because images come from tablets, plain paper, checkered paper, and photographed paper, a system that performs well on CAS2UML has been tested across varied acquisition conditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because one annotator created or selected every reference and visually checked every pair, an independent re-annotation of a random subsample could quantify how much of the ground truth is individual judgment.
  • Editorial inference: activity diagrams have no XMI ground truth, so semantic evaluation of behavior diagrams is limited to what PlantUML syntax and rendering can express; a formal control-flow model would allow deeper equivalence checks.
  • Editorial inference: the redraw-from-reference workflow likely produces cleaner, more schematic sketches than spontaneous whiteboard drawings, so a field-collected test split would reveal how well models transfer to messier real-world input.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents CAS2UML, a public dataset of 557 handwritten UML diagrams (271 class diagrams and 286 activity diagrams), each paired with PlantUML code that the authors describe as manually validated and executable. The dataset creation pipeline is described in Section 3: reference PlantUML specifications are created or collected, rendered, then manually redrawn on various media; a Gradio-based validation tool is used for side-by-side visual checks; automated checks include PlantUML syntax parsing and, for class diagrams, XMI serialization. The paper also releases validation scripts, coverage guidelines, and a comparison with prior UML sketch datasets, and it cites the authors' follow-up work as preliminary evidence of utility.

Significance. If the dataset delivers what the abstract promises, it would be a useful resource: it is the first public resource combining handwritten class and activity diagrams with executable, machine-readable ground truth at a scale beyond a handful of examples. The public release on Hugging Face, the companion validation tool, the XMI export for class diagrams, the coverage guideline covering GoF-inspired patterns, and the documented media diversity are concrete strengths that support reproducible benchmarking. However, the central value of the dataset rests on the fidelity of the sketch-to-PlantUML ground truth, and the paper does not currently provide evidence strong enough for the 'manually validated' label. The single-annotator workflow and the absence of any semantic-equivalence metric mean the benchmark's quality is unmeasured. This is a correctness-risk concern for the resource's main advertised property, not a mere presentation issue.

major comments (3)
  1. [Section 3.4 and Section 3.2/3.3] The abstract's claim that each pair is 'manually validated' is stronger than the evidence. The validation workflow is: the first author creates or corrects the reference PlantUML, the same author redraws it by hand, and the same author visually compares sketch and rendered diagram in the Gradio tool; Section 3 states 'The first author performed all manual steps.' The automated checks (-syntax flag and XMI serialization for class diagrams) verify parseability or renderability, but they do not verify that class names, attributes, association types, multiplicities, or activity control flow in the code match the drawing. Because the validator already knows the intended code, a redrawing that drops an attribute or reverses an arrow can be missed. There is no inter-annotator agreement and no semantic-equivalence metric. Please either add an independent second annotator or a documented equivalence-checking protocol with reported agreement, or explicitly downgrade the claim to 'annotated' and 'syntax-checked' in the abstract and throughout the paper.
  2. [Section 3.2 and Section 3.3] The dataset is constructed in reverse relative to real sketch-to-UML use: reference PlantUML is created first and then redrawn by hand. Every sketch is therefore a copy of a rendered diagram, not an independently drawn UML sketch. This likely biases the collection toward cleaner, more schematic drawings and away from the messiness, ambiguity, and layout noise of naturally occurring hand-drawn UML. Since the stated purpose is to enable benchmarking of sketch-to-UML approaches on realistic handwritten input, this representativeness concern is load-bearing. Please provide evidence about visual and structural diversity relative to naturally collected sketches, add a subset of independently drawn diagrams, or clearly state the reverse-pipeline limitation and refrain from generalizing to real-world handwritten UML.
  3. [Section 4 and reference [15]] The only preliminary evidence of utility is the authors' own follow-up paper, which is not independent evidence that CAS2UML works as a benchmark. The sentence 'As preliminary evidence of utility, we use CAS2UML in follow-up work...' should be labeled as a self-citation, and it does not quantify how much of the reported model improvement is attributable to the dataset itself. To support the benchmark claim, the paper should either include a small baseline evaluation on CAS2UML (for example, simple image-to-PlantUML models evaluated with the released validation script) or remove the utility claim and present the resource on its own terms.
minor comments (5)
  1. [Abstract] The link formatting in the abstract is malformed: 'available at:Dataset:https...' and 'Tool and Scripts:https...' lack spaces and punctuation, making the URLs hard to parse.
  2. [Section 3.2] The LLM-assisted authoring workflow names specific model versions (ChatGPT 5.3, Claude Sonnet 4.6, Gemini 3 Flash), but no prompts, temperature settings, or version dates are provided; please include these in the repository for reproducibility.
  3. [Section 3.3] The Lindholmen filtering step is described only as URL-path keyword occurrences; please specify the exact keywords and the selection procedure for the approximately 80 reference templates so that the process is reproducible.
  4. [Section 3.2] The exclusion of seven class diagrams whose PlantUML code could not be serialized to XMI due to unsupported association-class constructs should be documented as a coverage limitation, since it means the released class-diagram subset under-represents association classes.
  5. [Table 2] The coverage guideline lists instance counts for GoF-inspired patterns and constructs, but the counts are not tied to sample identifiers; a coverage-to-sample mapping in the repository would make construct-aware benchmarking easier.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the dataset contribution is self-contained; the only self-reference is a non-load-bearing pointer to the authors' own follow-up work.

full rationale

CAS2UML is a resource paper: it makes no quantitative prediction and derives no result from equations, so there is no fitted parameter, ansatz, or uniqueness theorem that could reduce to its own inputs. The central claim, that 557 hand-drawn diagrams are paired with PlantUML code and publicly released, is verified by the artifact itself and does not depend on any internal derivation. The only self-reference appears in Section 4: "As preliminary evidence of utility, we use CAS2UML in follow-up work to fine-tune a handwritten-UML-to-PlantUML model that, evaluated via automatic metrics and a human ranking study, becomes competitive with proprietary vision-language baselines [15]." This is a non-load-bearing pointer: even if the cited model result were retracted, the existence and content of the released dataset would be unaffected. The validation workflow in Sections 3.2-3.4 is also self-referential in a weaker sense (reference PlantUML is created first, then hand-redrawn, and checked visually by the first author), but this is a disclosed data-fidelity limitation rather than a circular derivation, because the paper does not claim to derive the ground truth from independent evidence. No circular step meets the quote-and-reduction bar.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The dataset claim rests on the assumptions that PlantUML text is an adequate executable ground truth for class and activity diagrams, that the single-annotator visual check establishes semantic correspondence, and that syntax checks plus XMI generation demonstrate validity. These are domain assumptions, not derived facts. No free parameters or invented entities are introduced because this is a resource paper rather than a model or derivation paper.

assumptions (4)
  • domain assumption PlantUML code is an adequate executable representation of UML class and activity diagram semantics.
    The dataset pairs every sketch with PlantUML text and treats that text as ground truth. This is reasonable for a benchmark, but it equates UML validity with PlantUML parseability and renderability, which is an accepted convention rather than a formally proven equivalence.
  • ad hoc to paper A single annotator's visual inspection establishes one-to-one correspondence between each sketch and its PlantUML code.
    Section 3.4 describes the Gradio tool with side-by-side sketch, code, and rendered preview, and Section 3.1 states that 'The first author performed all manual steps.' No second annotator or inter-rater reliability is reported, so the semantic validity of the pairs rests on one person's judgment.
  • ad hoc to paper Syntax checking and XMI serialization are sufficient evidence that the PlantUML code is valid and executable.
    Section 3.2 uses the PlantUML JAR with the -syntax flag and XMI generation for class diagrams. These checks confirm parseability and renderability, but they do not verify that the code semantically matches the sketch, which is part of the 'validated' claim.
  • ad hoc to paper Sketches redrawn from rendered PlantUML references are representative of hand-drawn UML diagrams in real use.
    Sections 3.2 and 3.3 describe a workflow where reference PlantUML is authored first and then redrawn by hand. This produces clean, purpose-made pairs, but may underrepresent ambiguous, partial, or messy sketches found in practice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams." pith.science (2026). https://pith.science/paper/3FRRI7US

@misc{pith2026260807036,
  author       = {Pith},
  title        = {Pith review of: CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FRRI7US}},
  note         = {Machine review of arXiv:2608.07036}
}
read the original abstract

Automated UML generation from sketches and images is gaining renewed attention with the rise of large language models and multimodal AI. However, reproducible evaluation remains difficult due to the lack of public datasets with executable groundtruth models. We present CAS2UML, a public dataset of 557 handdrawn UML diagrams, including 271 class diagrams and 286 activity diagrams, each paired with manually validated PlantUML code. We also provide a PlantUML-based validation tool and reusable scripts for checking the syntactic correctness and renderability of generated UML artifacts, enabling reproducible benchmarking of sketch-to-UML approaches. The dataset, validation tool, processing scripts, documentation, and demonstration video are publicly available at: Dataset: https://huggingface.co/datasets/Seym0n /cas2uml_hand-drawn_to_plantuml_dataset; Tool and Scripts: https://github.com/Seym0n/handwritten-uml-dataset; Video: https://www.youtube.com/watch?v=KQrYeGgT3hs.

Figures

Figures reproduced from arXiv: 2608.07036 by the authors.

Figure 1
Figure 1. The CAS2UML validation tool with side-by-side [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. CAS2UML processing and validation pipeline [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 10 canonical work pages

  1. [1]

    Bilal Al-Ahmad, Anas Alsobeh, Omar Meqdadi, and Nazimuddin Shaikh. 2025. A Student-Centric Evaluation Survey to Explore the Impact of LLMs on UML Modeling.Information16, 7 (2025). doi:10.3390/info16070565

    doi:10.3390/info16070565 2025

  2. [2]

    Kua Chen, Yujing Yang, et al . 2023. Automated domain modeling with large language models: A comparative study. In2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems. IEEE, 162–172. doi:10.1109/MODELS58315.2023.00037

    arXiv 2023

  3. [3]

    Aaron Conrardy and Jordi Cabot. 2024. From image to UML: First results of image based UML diagram generation using LLMs. (2024). doi:10.48550/arXiv.2 404.11376

    doi:10.48550/arxiv.2 2024

  4. [4]

    Daniele De Bari, Giacomo Garaccione, Riccardo Coppola, Marco Torchiano, and Luca Ardito. 2024. Evaluating large language models in exercises of uml class diagram modeling. InProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 393–399. doi:10.1145/3674 805.3690741

    work page doi:10.1145/3674 2024

  5. [5]

    Cecilia Eklund and Tom Jonsson. 2025. BENCHMARKING LARGE LANGUAGE MODELS IN UML DIAGRAM GENERATION FROM INFORMAL NOTATIONS

    work page 2025

  6. [6]

    Regina Hebig, Truong Ho Quang, et al. 2016. The quest for open source projects that use UML: mining GitHub. InProceedings of the ACM/IEEE 19th international conference on model driven engineering languages and systems. 173–183. doi:10.1 145/2976767.2976778

    arXiv 2016

  7. [7]

    John Hunt. 2013. Gang of four design patterns. InScala Design Patterns: Patterns for Practical Reuse and Design. Springer, 135–136. doi:10.1007/978-3-319-02192-8

    work page doi:10.1007/978-3-319-02192-8 2013

  8. [8]

    Bilal Karasneh and Michel RV Chaudron. 2013. Extracting UML models from images. In5th Int. Conf. on Computer Science and Information Technology. IEEE. doi:10.1109/CSIT.2013.6588776

    work page arXiv 2013

Show all 19 references
  1. [9]

    Hatice Koc, Ali Mert Erdoğan, Yousef Barjakly, and Serhat Peker. 2021. UML diagrams in software engineering research: a systematic literature review. In Proceedings, Vol. 74. MDPI, 13. doi:10.3390/proceedings2021074013

    2021 doi

  2. [10]

    Parastoo Mohagheghi, Vegard Dehlen, and Tor Neple. 2009. Definitions and approaches to model quality in model-based software development–A review of literature.Information and software technology51, 12 (2009), 1646–1669. doi:10.1 016/j.infsof.2009.04.004

    2009

  3. [11]

    Mert Ozkaya and Ferhat Erata. 2020. A survey on the practical use of UML for different software architecture viewpoints.Info. and Software Tech.(2020). doi:10.1016/j.infsof.2020.106275

    2020

  4. [12]

    Leticia Piucco. 2024. Handwritten UML Class Diagrams. Kaggle dataset. https: //www.kaggle.com/datasets/leticiapiucco/handwritten-uml-class-diagrams Accessed: 7 May 2026

    2024

  5. [13]

    Mantas Ražinskas, Benas Mili¯unas, Mantas Jurgelaitis, Lina Čeponien˙e, and Lina Bisikirskien˙e. 2024. Transforming sketches of UML use case diagrams to models. IEEE Access12 (2024), 185826–185837. doi:10.1109/ACCESS.2024.3514455

    2024

  6. [14]

    Gianna Reggio, Maurizio Leotta, Filippo Ricca, and Diego Clerissi. 2014. What are the used activity diagram constructs? a survey. In2014 2nd International Conference on Model-Driven Engineering and Software Development. 87–98. doi:10 .5220/0004712000870098

    2014

  7. [15]

    Mersedeh Sadeghi, Simon Scholz, and Adrian Psoch-Bajraktari. 2026. A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation. InProceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS). doi:10.48550/arX...

    2026 doi arXiv

  8. [16]

    Bernhard Schäfer, Han van der Aa, et al. 2021. Sketch2BPMN: Automatic recog- nition of hand-drawn BPMN models. InInternational Conference on Advanced Information Systems Engineering. Springer, 344–360. doi:10.1007/978-3-030- 79382-1_21

    2021 doi

  9. [17]

    Bernhard Schäfer, Han van der Aa, Henrik Leopold, and Heiner Stuckenschmidt

  10. [18]

    Liping Zhao, Waad Alhoshan, et al . 2021. Natural Language Processing for Requirements Engineering: A Systematic Mapping Study.Comput. Surveys54, 3 (2021), 1–41. doi:10.1145/3444689 Received 2026-05-12; accepted 2026-06-19

    2021 doi

  11. [2023]

    doi:10.1109/TSE.2022.3228308

    Sketch2Process: End-to-End BPMN Sketch Recognition Based on Neural Networks.IEEE Transactions on Software Engineering49, 4 (2023), 2621–2641. doi:10.1109/TSE.2022.3228308

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.