{"id":"c8080bc5-6bf1-4717-aea2-05bf6eb1d8fc","arxiv_id":"2608.07036","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CAS2UML provides 557 hand-drawn class and activity UML diagrams with validated PlantUML ground truth, plus a validation tool and scripts for reproducible sketch-to-UML benchmarking.","lead":"This paper introduces CAS2UML, a public dataset of 557 handwritten UML class and activity diagrams, each paired with PlantUML code that was checked for syntactic validity and renderability. It also ships a Gradio validation tool and scripts, giving sketch-to-UML researchers a common benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-annotator visual check cannot establish the 'validated' ground-truth claim: because sketches were redrawn from the PlantUML reference (Sec. 3.2–3.3) and checked only by the first author in the Gradio tool (Sec.","rationale":"Reading the paper in good faith, the dataset appears genuinely useful and the disclosed pipeline is concrete: exact counts, media distribution, source attribution, and released tooling are all specified. The central claim, however, is that every pair is 'manually validated PlantUML code,' and that validation is the load-bearing step for using the resource as executable ground truth. The paper's own narrative (Sec. 3.2–3.4) shows the validator and the sketch author are the same person, that the sketch was produced from the code, and that automated checks only test syntax. This is not a fraud accusation; it is a normal single-annotator limitation, but it is unquantified. Because no independent annotation or equivalence metric is provided, a user cannot distinguish a faithful pair from one where the redraw omitted a class or mislabeled a multiplicity. The naturalness concern (code-first sketches are cleaner than real hand-drawn UML) is real but secondary; the fidelity concern is more immediate because it affects every downstream evaluation. I therefore agree with the reader's weakest-assumption analysis and recommend keeping the verdict conditional: the resource is valuable, but the 'validated' label and benchmark generalizability are not established until fidelity is independently tested.","tokens_in":6347,"tokens_out":4344,"duration_ms":39806,"concrete_test":"Draw a stratified random sample of 50 released pairs (covering class and activity diagrams and each medium in Table 3), hide the released PlantUML, and have two independent UML experts, blind to the reference, annotate each sketch with PlantUML using the same PlantText workflow. Compare expert output to the released code structurally: for class diagrams, match class, attribute, operation, and association endpoints with multiplicities via parsed PlantUML or XMI; for activity diagrams, compare extracted control-flow nodes, edges, decisions, and joins. Compute pairwise F1 between experts and released code and report inter-annotator agreement. If mean F1 is below a pre-specified threshold (e.g., 0.90), the 'validated ground truth' claim is not supported and the dataset needs a revised validation protocol or a per-sample confidence score.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CAS2UML's advertised value rests on each PlantUML annotation being executable and semantically faithful to the paired sketch. The pipeline in Sec. 3.2–3.3 creates the reference PlantUML first, then hand-redraws it on paper or tablet, so the sketch is a copy of the code rather than an independent annotation target. Validation in Sec. 3.4 is a side-by-side visual inspection performed by the same first author who created and redrew the samples; Sec. 3 states 'The first author performed all manual steps.' The automated checks are only PlantUML -syntax parseability and, for class diagrams, XMI serialization; both verify parseability or renderability, not that class names, attributes, associations, multiplicities, or activity control flow in the code match the actual drawing. Because the validator already knows the intended code, a redrawing that drops an attribute or reverses an arrow is likely to be missed, and there is no inter-annotator agreement or semantic equivalence metric to catch it. Thus the abstract's 'manually validated' label is stronger than the evidence supports, and benchmark conclusions drawn from this ground truth inherit the unmeasured fidelity risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CAS2UML, a public dataset of 557 handwritten UML diagrams (271 class diagrams and 286 activity diagrams), each paired with PlantUML code that the authors describe as manually validated and executable. The dataset creation pipeline is described in Section 3: reference PlantUML specifications are created or collected, rendered, then manually redrawn on various media; a Gradio-based validation tool is used for side-by-side visual checks; automated checks include PlantUML syntax parsing and, for class diagrams, XMI serialization. The paper also releases validation scripts, coverage guidelines, and a comparison with prior UML sketch datasets, and it cites the authors' follow-up work as preliminary evidence of utility.","tokens_in":6620,"tokens_out":3399,"duration_ms":31382,"significance":"If the dataset delivers what the abstract promises, it would be a useful resource: it is the first public resource combining handwritten class and activity diagrams with executable, machine-readable ground truth at a scale beyond a handful of examples. The public release on Hugging Face, the companion validation tool, the XMI export for class diagrams, the coverage guideline covering GoF-inspired patterns, and the documented media diversity are concrete strengths that support reproducible benchmarking. However, the central value of the dataset rests on the fidelity of the sketch-to-PlantUML ground truth, and the paper does not currently provide evidence strong enough for the 'manually validated' label. The single-annotator workflow and the absence of any semantic-equivalence metric mean the benchmark's quality is unmeasured. This is a correctness-risk concern for the resource's main advertised property, not a mere presentation issue.","major_comments":[{"comment":"The abstract's claim that each pair is 'manually validated' is stronger than the evidence. The validation workflow is: the first author creates or corrects the reference PlantUML, the same author redraws it by hand, and the same author visually compares sketch and rendered diagram in the Gradio tool; Section 3 states 'The first author performed all manual steps.' The automated checks (-syntax flag and XMI serialization for class diagrams) verify parseability or renderability, but they do not verify that class names, attributes, association types, multiplicities, or activity control flow in the code match the drawing. Because the validator already knows the intended code, a redrawing that drops an attribute or reverses an arrow can be missed. There is no inter-annotator agreement and no semantic-equivalence metric. Please either add an independent second annotator or a documented equivalence-checking protocol with reported agreement, or explicitly downgrade the claim to 'annotated' and 'syntax-checked' in the abstract and throughout the paper.","section":"Section 3.4 and Section 3.2/3.3"},{"comment":"The dataset is constructed in reverse relative to real sketch-to-UML use: reference PlantUML is created first and then redrawn by hand. Every sketch is therefore a copy of a rendered diagram, not an independently drawn UML sketch. This likely biases the collection toward cleaner, more schematic drawings and away from the messiness, ambiguity, and layout noise of naturally occurring hand-drawn UML. Since the stated purpose is to enable benchmarking of sketch-to-UML approaches on realistic handwritten input, this representativeness concern is load-bearing. Please provide evidence about visual and structural diversity relative to naturally collected sketches, add a subset of independently drawn diagrams, or clearly state the reverse-pipeline limitation and refrain from generalizing to real-world handwritten UML.","section":"Section 3.2 and Section 3.3"},{"comment":"The only preliminary evidence of utility is the authors' own follow-up paper, which is not independent evidence that CAS2UML works as a benchmark. The sentence 'As preliminary evidence of utility, we use CAS2UML in follow-up work...' should be labeled as a self-citation, and it does not quantify how much of the reported model improvement is attributable to the dataset itself. To support the benchmark claim, the paper should either include a small baseline evaluation on CAS2UML (for example, simple image-to-PlantUML models evaluated with the released validation script) or remove the utility claim and present the resource on its own terms.","section":"Section 4 and reference [15]"}],"minor_comments":[{"comment":"The link formatting in the abstract is malformed: 'available at:Dataset:https...' and 'Tool and Scripts:https...' lack spaces and punctuation, making the URLs hard to parse.","section":"Abstract"},{"comment":"The LLM-assisted authoring workflow names specific model versions (ChatGPT 5.3, Claude Sonnet 4.6, Gemini 3 Flash), but no prompts, temperature settings, or version dates are provided; please include these in the repository for reproducibility.","section":"Section 3.2"},{"comment":"The Lindholmen filtering step is described only as URL-path keyword occurrences; please specify the exact keywords and the selection procedure for the approximately 80 reference templates so that the process is reproducible.","section":"Section 3.3"},{"comment":"The exclusion of seven class diagrams whose PlantUML code could not be serialized to XMI due to unsupported association-class constructs should be documented as a coverage limitation, since it means the released class-diagram subset under-represents association classes.","section":"Section 3.2"},{"comment":"The coverage guideline lists instance counts for GoF-inspired patterns and constructs, but the counts are not tied to sample identifiers; a coverage-to-sample mapping in the repository would make construct-aware benchmarking easier.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the conference scope and the resource, if properly validated, would likely be of interest to the ASE community. The main issue is that the 'validated' ground-truth claim is not supported by the described methodology. I would not reject outright because the dataset itself is public and the validation gap can in principle be addressed by additional annotation effort or by reframing the claims. I would ask the authors to address the fidelity concern before acceptance, and to avoid relying on their own follow-up paper as the sole evidence of utility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CAS2UML is worth a serious look. It is the first public dataset to pair handwritten class and activity diagrams with executable PlantUML ground truth at any scale larger than a handful of examples: the paper’s own comparison table shows prior work at 4, 70, and 346 samples, none with machine-readable, validated ground truth. That alone makes it useful for sketch-to-UML and image-to-model benchmarking. The authors also release the validation tool, processing scripts, and a clear description of the pipeline, including the LLM-assisted code authoring, the manual redrawing step, and the media diversity (tablet, paper, checkered, photographed). For a dataset paper, that level of disclosure is good and the resource is genuinely new.\n\nThe soft spot is the validation, and the stress-test note has it right. All manual steps were done by the first author, and the sketches were redrawn from the PlantUML reference, not annotated independently. So the \"manually validated\" claim in the abstract means the author checked that a hand copy matches code they already knew. PlantUML’s -syntax flag guarantees parseability, not semantic fidelity; the XMI check does the same for a specific UML serialization. There is no inter-annotator agreement and no metric for sketch-code equivalence. This is a real limitation, but it is also disclosed in the workflow description, and it does not sink the resource: for training and for rough benchmarking, the pairing is still far better than what exists, especially since the dataset is public and users can spot-check or filter it themselves. I’d call the fidelity risk moderate, not fatal.\n\nOne minor thing: the Section 4 teaser citing their own follow-up model paper is fine as a preliminary hint, but it is not independent evidence of utility. No problem there, just don’t over-read it.\n\nThe paper is for anyone working on handwritten UML recognition, LLM-based diagram generation, or multimodal software engineering. It deserves a serious referee: the contribution is concrete, the artifact is public, and the limitations are fixable (e.g., a random-sample inter-annotator audit or a clearer abstract that says \"visually checked by the authors\" instead of \"validated\"). I would engage with it, and I’d expect a conditional accept with a request to strengthen the validation evidence.","headline":"CAS2UML fills a real gap with a public 557-sample dataset of handwritten UML sketches paired with executable PlantUML, but the 'validated' label is stronger than the single-annotator, code-first pipeline actually supports.","tokens_in":7097,"tokens_out":1837,"would_cite":true,"duration_ms":18278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAS2UML pairs 557 hand-drawn UML diagrams with validated PlantUML code, aiming to make sketch-to-UML evaluation reproducible.","keywords":["dataset","UML","hand-drawn diagrams","PlantUML","sketch recognition","benchmark","class diagrams","activity diagrams"],"falsifier":"Render the released PlantUML for every pair and compare the rendered elements with the paired sketch: any class, relation, decision node, fork or join, or swimlane present in one but absent from the other disproves the one-to-one correspondence for that sample. A statistical version is to have independent annotators judge a random subsample of pairs as matching or not; a material non-match rate would refute the claim that the ground truth is validated.","tokens_in":6132,"feed_emoji":"✏️","tokens_out":8934,"duration_ms":74194,"temperature":0.7,"pith_summary":"CAS2UML is a proposed public dataset of 557 hand-drawn UML diagrams—271 class diagrams and 286 activity diagrams—each paired with PlantUML code that the authors manually validated. The paper's aim is to give sketch-to-UML and image-to-model research a shared, executable ground truth, where previous hand-drawn UML resources were small, covered one diagram type, or offered only visual annotations. If the dataset holds up, evaluation stops being a matter of judging image similarity and becomes a check of whether generated UML is syntactically valid, renderable, and reusable in modeling tools. Alongside the data, the paper releases a validation tool and scripts that perform syntax checks and, for class diagrams, XMI export so other researchers can reproduce the same checks.","feed_headline":"557 handwritten UML diagrams, each paired with validated PlantUML","feed_subtitle":"First public benchmark pairing class and activity sketches with executable, machine-readable ground truth.","key_machinery":"The central object is the sketch–PlantUML pair: one handwritten image and one executable text supposed to render to the same diagram structure. The load-bearing mechanism is the validation tool, an interface that shows the sketch next to editable PlantUML code with a live rendered preview, so the annotator can confirm correspondence before release. Machine-checkable support comes from the PlantUML syntax check and, for class diagrams, XMI export, which turn a visual judgment into artifacts other tools can consume.","core_discovery":"The central claim is that CAS2UML is the first public resource combining handwritten class and activity diagrams with manually validated, executable PlantUML ground truth at a scale usable for systematic benchmarking. The pairs were produced by taking reference PlantUML specifications—some derived from existing handwritten images, some newly authored—redrawing them by hand on tablets, plain paper, and checkered paper, and then checking each sketch against its code in a custom side-by-side annotation tool. Every released annotation passes the PlantUML parser's syntax check; class diagrams are also serialized to XMI so they can be opened in UML tools. On the authors' terms, the one-to-one sketch–code correspondence is what makes the dataset a benchmark rather than just a collection of pictures.","pith_inferences":["Editorial inference: because one annotator created or selected every reference and visually checked every pair, an independent re-annotation of a random subsample could quantify how much of the ground truth is individual judgment.","Editorial inference: activity diagrams have no XMI ground truth, so semantic evaluation of behavior diagrams is limited to what PlantUML syntax and rendering can express; a formal control-flow model would allow deeper equivalence checks.","Editorial inference: the redraw-from-reference workflow likely produces cleaner, more schematic sketches than spontaneous whiteboard drawings, so a field-collected test split would reveal how well models transfer to messier real-world input."],"forward_implications":["A sketch-to-UML model can be trained or evaluated on 557 paired examples spanning two diagram types, replacing the handful of samples used in earlier studies.","Automated syntax and renderability checks make results reproducible: another lab can run the same validation script on any generated PlantUML output.","Class-diagram XMI files let generated models be imported into UML modeling tools, so success can be measured by downstream usability rather than image similarity alone.","Activity diagrams covering decisions, forks, joins, loops, and swimlanes extend benchmarking from static structure to behavioral control flow.","Because images come from tablets, plain paper, checkered paper, and photographed paper, a system that performs well on CAS2UML has been tested across varied acquisition conditions."],"supporting_citations":[{"why":"Supplies the 70 existing handwritten class-diagram images that CAS2UML reuses and augments with PlantUML ground truth.","marker":"[12]"},{"why":"Provides the mined UML-image corpus from which roughly eighty activity-diagram reference templates were selected.","marker":"[6]"},{"why":"The closest prior hand-drawn class-diagram resource with PlantUML annotations, based on only four examples, establishes the scale gap CAS2UML fills.","marker":"[3]"},{"why":"A public UML use-case sketch dataset with element annotations but no executable ground truth, defining the comparison point for the dataset's contribution.","marker":"[13]"},{"why":"An example of recent LLM-based UML generation evaluated on a small task-specific set, illustrating why a shared benchmark is needed.","marker":"[2]"}],"fun_headline_variants":["557 handwritten UML diagrams matched to validated PlantUML","First dataset pairing sketch UML with executable PlantUML code","New benchmark: 271 class and 286 activity diagrams with PlantUML","Sketch-to-PlantUML dataset with 557 verified diagram-code pairs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's quality rests on a single annotator's visual judgment that each hand-drawn sketch and its PlantUML code represent the same diagram; there is no second annotator, no inter-annotator agreement, and no metric for semantic equivalence between image and code.","fun_headline_variants_meta":{"raw":{"variants":["557 handwritten UML diagrams matched to validated PlantUML","First dataset pairing sketch UML with executable PlantUML code","New benchmark: 271 class and 286 activity diagrams with PlantUML","Sketch-to-PlantUML dataset with 557 verified diagram-code pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000487,"raw_usage":{"total_tokens":2368,"prompt_tokens":879,"completion_tokens":1489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1427}},"tokens_in":495,"tokens_out":1489,"duration_ms":9987,"temperature":1.0,"reasoning_tokens":1427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:55:38.653772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render the released PlantUML for every pair and compare the rendered elements with the paired sketch: any class, relation, decision node, fork or join, or swimlane present in one but absent from the other disproves the one-to-one correspondence for that sample. A statistical version is to have independent annotators judge a random subsample of pairs as matching or not; a material non-match rate would refute the claim that the ground truth is validated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 70 existing handwritten class-diagram images that CAS2UML reuses and augments with PlantUML ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A public UML use-case sketch dataset with element annotations but no executable ground truth, defining the comparison point for the dataset's contribution."}],"review_version":1}