Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Parsing generated UML into XMI and control-flow graphs gives a small open model trained with formalism-aware rewards a competitive edge on handwritten UML-to-PlantUML conversion.

desk verdict Honest, well-scoped worked example: small open VLM becomes competitive on handwritten UML-to-PlantUML, but the GRPO stage's contribution remains indistinguishable from SFT, and the headline metric is the training reward. read the letter →

arxiv 2607.28987 v1 pith:5EEQRTGU submitted 2026-07-31 cs.SE

Mersedeh Sadeghi , Simon Scholz , Adrian Psoch-Bajraktari This is my paper · ORCID
classification cs.SE
keywords UMLPlantgenerationvision-languagemodelsreinforcementlearningformalism-awarerewardsXMIcomparisoncontrol-flowgraphssketch-to-modelconversion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the right way to grade a model's UML generation is to look at the structure of the model it produces, not the text of the code. It adapts a small open vision-language model with supervised fine-tuning plus reinforcement learning, where the reward is computed by parsing the generated PlantUML into XMI (for class diagrams) or a control-flow graph (for activity diagrams) and comparing against the reference in those formalisms. The adapted model becomes competitive: it compiles every class diagram, raises activity-diagram compilation from 20% to 86.7%, and moves from last to second among four models in both automatic and human evaluations. The paper is careful to say that the added benefit of the reward-guided stage over fine-tuning alone is not yet statistically detectable, and that the automatic metric only partly captures what humans mean by modelling quality. The value of the work is a pattern—derive the learning signal from analysable model representations—that could generalize to other notations.

What carries the argument

The central mechanism is the formalism-aware reward loop. Generated PlantUML is not scored as text; it is compiled and parsed into the formalism's canonical representation—XMI for class diagrams, a control-flow graph for activity diagrams—and compared with the reference in that representation. Class-diagram reward components cover compilation, classes, relationships, methods, and attributes; activity-diagram reward components cover compilation, CFG structure (via weighted Jaccard similarity and Hungarian-algorithm graph assignment), and label similarity (via TF-IDF). This same representation-based scoring is used both as the reinforcement-learning reward and as the automatic evaluation metri

What would settle it

Take the switch-branch example the paper reports (an if-branch rewritten as a switch-branch that renders identically). If a panel of UML practitioners judges the two outputs as equally good interpretations of the same handwritten sketch, yet the formalism-aware metric scores them substantially apart, that falsifies the metric's validity as a proxy for conversion quality—and since the same metric drives both training and evaluation, it would undermine the comparative claims. A quantitative version: on a set of meaning-preserving alternatives, compute the correlation between the metric and human

Watch

Extended reading notes

Core claim

Handwritten-UML-to-PlantUML should be treated as structured model generation, not image-to-text generation. The paper adapts a small open vision-language model via supervised fine-tuning plus group-relative policy optimisation, with rewards derived from parsing the generated PlantUML into XMI (class diagrams) or a control-flow graph (activity diagrams) and comparing those structures to the reference. On a held-out set of 30 sketches the adapted model compiles 100% of class and 86.7% of activity diagrams and moves from last to second in automatic and human rankings, ahead of one proprietary baseline and behind another. The reward stage's gain over fine-tuning alone is not statistically signif

Load-bearing premise

The load-bearing premise is that the formalism-aware scores—XMI structural comparison and CFG comparison with hand-chosen weights and edit costs—are a valid measure of conversion quality; the paper itself finds only moderate agreement with human rankings (rho=0.565) and shows the metric penalises meaning-preserving deviations, so if this metric is arbitrary, the comparative results are an artefact of the metric.

Editorial extensions

If this is right

  • If the full pipeline is as competitive as reported, a small open model can serve as a practical sketch-to-model converter, reducing reliance on larger proprietary systems for this task.
  • Compilability of generated class diagrams can reach 100% and activity diagrams 86.7% on this data, making the outputs usable in downstream model-analysis workflows without manual repair.
  • Formalism-aware metrics capture compilability and recoverable structure but only moderately align with human judgement, so any evaluation of such models that relies solely on them will misrank acceptable alternative formulations.
  • The reward-loop pattern provides a template for turning model analysis into training feedback, though each new formalism requires designing its own representation and comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reward stage shows no significant gain over supervised fine-tuning on this held-out set, the strong results may be driven mostly by the supervised phase; the real test of formalism-aware rewards is whether they add value with larger datasets or with decoupled per-component normalisation.
  • If a reward were designed to recognise semantic equivalence—e.g., treating a switch-branch and an if-branch as the same control flow—the metric-human correlation would likely rise, and the reward loop could become more effective as a training signal.
  • The error analysis points to visual grounding, not PlantUML generation, as the bottleneck for remaining meaning-altering errors; improved fine-grained perception (reading multiplicities, arrowheads) may yield gains that compound with better reward design.
  • A direct extension would apply the same loop to state machines or sequence diagrams; if the pattern transfers, it strengthens the claim that formalism-aware rewards are a general method rather than a two-diagram special case.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a 'formalism-aware reward loop' for handwritten UML-to-PlantUML generation: rather than scoring generated PlantUML as text, the system compiles it into an analysable representation (XMI for class diagrams, a control-flow graph for activity diagrams) and computes rewards by structural comparison with a reference. A small open vision-language model (Qwen3.5-4B) is adapted with supervised fine-tuning followed by GRPO. On a held-out set of 30 diagrams, the paper reports that the full pipeline compiles 100% of class diagrams and 86.7% of activity diagrams, achieves content scores of 17.60 and 13.71, and is ranked second of four by human raters, behind Gemini 3 Flash and ahead of GPT-4.1 Mini. The SFT-only ablation is statistically indistinguishable from the full pipeline (p=0.94 class, p=0.64 activity), and the automatic content score correlates moderately with human rankings (rho=0.565). The paper carefully hedges most claims and frames the work as emerging evidence.

Significance. If the competitiveness result holds, the paper demonstrates that a small open VLM adapted with formalism-aware rewards can approach proprietary baselines on sketch-to-model conversion, a practically relevant task. The reusable pattern of parsing into a formalism-specific representation and comparing against a reference is a useful contribution, and the paper is unusually honest: it provides an explicit SFT ablation, a human ranking study with high inter-rater agreement, an error taxonomy including meaning-preserving deviations, and a public artifact repository. The main value is in the negative/qualified finding that the reward-guided stage does not yet show added value beyond SFT on this small set, which is itself an informative result for the NIER audience.

major comments (3)
  1. [§4.1–4.2, Tables 1–2] The central 'second of four' claim rests on point estimates from n=15 diagrams per type. Class content is 17.60 vs. 17.36 for Gemini; activity is 13.71 vs. 14.01; human Borda is 0.613 vs. 0.731 overall and 0.564 vs. 0.787 for activity. No confidence intervals, bootstrap intervals, or paired significance tests are reported for these comparisons; the only inferential statistics are the SFT-vs-full Wilcoxon tests (p=0.94, p=0.64). With 15 diagrams per type, gaps of 0.24 content points or 0.05–0.17 Borda units are plausibly sampling noise. Please add paired tests or bootstrap CIs for Ours-vs-Gemini and Ours-vs-GPT, or soften the ordering claim to 'not distinguishable from Gemini on class diagrams' and 'ahead of GPT-4.1 Mini' only where statistically supported.
  2. [§4.4 vs. Table 2] The paper credits the 'full adaptation pipeline' with moving the model from last to second, but Table 2 shows the SFT-only checkpoint is equal to the full pipeline on all class-diagram components (compilation 100 vs. 100; class-level 98.3 vs. 98.3; attributes 98.0 vs. 98.0; relationships 83.6 vs. 84.2; methods 91.2 vs. 90.9) and pointwise higher on every activity-diagram component (compilation 100 vs. 86.7; structural 86.0 vs. 73.4; label 78.6 vs. 69.2). Therefore the second-place result could be produced entirely by supervised fine-tuning, and the novel formalism-aware reward loop has no demonstrated positive effect. This is load-bearing for RO1. Either include the SFT-only checkpoint in the overall ranking/Borda analysis, or explicitly restate RO1 as 'SFT-based adaptation is competitive; the reward-stage contribution remains open' so that the headline does not attribute the ranking to
  3. [§4.1 and §3] The automatic evaluation reuses the formalism-aware reward functions of Section 3 as model-level metrics. Because GRPO is trained to maximize exactly those functions, the automatic score gains are partly a consequence of optimizing the metric itself, not independent evidence of conversion quality. The human ranking study is the genuinely independent check, but it is based on 30 sketches and its overall agreement with the automatic score is only moderate (rho=0.565). This circularity is acknowledged indirectly but should be stated more prominently; Table 1's 'automatic and human evaluation' side-by-side suggests two independent confirmations, whereas only the human Borda is independent of the training objective.
minor comments (5)
  1. [References [1] and [17]] The dataset is first cited as [1] ('Dataset Paper Accepted but Not Yet Publicly Available', anonymised), while [17] gives the actual CAS2UML reference. If the dataset paper is under double-anonymous review, unify the citations and state that the dataset will be made public; the current placeholder makes it hard to verify the data provenance.
  2. [§4.1, final paragraph] The sentence ending 'rather than imitation of reference text.Wh' contains a stray 'Wh' fragment. Please fix the typo.
  3. [§4.2, human study] The UML-knowledge test used for screening is described only as a 'competence threshold'. Provide the threshold or a supplementary description, and clarify how the 26 participants were distributed across the three blocks (a random assignment does not guarantee that every block received at least one rater, even though the design intends full coverage).
  4. [Figure 2] The caption says 'The two render to identical diagrams', but the two PlantUML snippets in (b) and (c) are syntactically different. I assume the rendered diagram images are visually identical; please phrase it as 'render to visually identical diagrams' to avoid ambiguity.
  5. [Tables 1–2] The tables would benefit from repeating the sample size (n=15 per diagram type) and explicitly noting that values are point estimates without error bars. This is especially important given the small n and the absence of confidence intervals.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training-reward metric is transparently reused for automatic evaluation, but an independent human ranking study carries the headline claim.

full rationale

The paper explicitly reuses its formalism-aware reward functions as the automatic evaluation metric ('The automatic evaluation reuses the formalism-aware reward functions of Section 3 as model-level metrics'). This is self-referential in the sense that the training objective and one headline measure coincide, but it is not a circular derivation: the reward is not fitted to the held-out outputs, the automatic results are explicitly framed as emerging evidence on a small set, and the metric is validated against an independent human ranking study (rho=0.565, p<0.001; Kendall's W=0.80-0.91). The human Borda results independently reproduce the second-place ordering (Ours 0.613 vs Gemini 0.759, GPT-4.1 Mini 0.442, base 0.186), so the central competitiveness claim does not reduce to the reward by construction. The ablation honestly reports that GRPO is statistically indistinguishable from SFT alone (p=0.94 class, p=0.64 activity) and that 'the specific contribution of the reward-guided stage remains open'; this is a limitation statement, not a disguised prediction. Self-citations to the authors' dataset papers are data provenance, not load-bearing argument, and no uniqueness theorem or ansatz is imported from prior author work. Therefore no load-bearing circularity was found.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central numbers are produced by a hand-parameterized reward/metric (subscore maxima, edit costs) that is reused as both training reward and evaluation metric; the only external anchor is the human ranking, which shows moderate correlation. No entity is postulated beyond the reward functions and custom parser, which are tools rather than new physical or mathematical objects.

free parameters (4)
  • Class-diagram reward subscore weights = compilation 1, classes 3, relationships 5, methods 5, attributes 5
    Hand-chosen maxima in Section 3; every class-diagram content score in Table 2 is a weighted sum using these values, and the same values are used in the GRPO training reward. Different weights would change the ranking.
  • Activity-diagram reward subscore weights = compilation 1, CFG structural 9, label 9
    Hand-chosen maxima in Section 3; the 9/9 split between structure and label is arbitrary and determines the activity content scores.
  • CFG edit costs (Hungarian substitution costs, swimlane penalty, outgoing-edge costs) = not specified in paper; deferred to artifact
    Section 3 says substitution costs distinguish same-kind/different-kind, including swimlane-aware penalty and outgoing-edge-kind costs; exact values are only in the repository, and they drive the structural reward.
  • Format reward threshold (0-1) = not specified beyond range
    A shared prefix/format reward applies to both diagram types but its exact form is not given; excluded from the content score, so it does not affect headline numbers.
assumptions (5)
  • domain assumption PlantUML-to-XMI conversion and the custom ANTLR4 CFG builder faithfully capture the intended UML model semantics.
    Section 3 Infrastructure; if export/parsing is lossy or wrong, every reward component and every metric in Tables 1-2 is corrupted.
  • domain assumption The dataset's human-written PlantUML references correctly encode the intended model for each sketch.
    Section 3 and ref [17]; the XMI/CFG ground truths are derived from these references, so label errors propagate to rewards, test scores, and human-study design.
  • ad hoc to paper The hand-chosen reward functions are a valid proxy for modelling acceptability.
    Section 4.2 shows only moderate correlation with human rankings (rho=0.565) and Figure 2 shows meaning-preserving deviations are penalized; the headline rankings depend on this assumption.
  • domain assumption The human ranking study is a trustworthy ground truth for modelling quality.
    Section 4.2: 26 raters pass a UML-knowledge screen, inter-rater agreement Kendall W=0.80-0.91, but raters are mostly early-career CS/BIS students recruited via SurveyCircle; reference model hidden so they judge against the sketch.
  • domain assumption Qwen3.5-4B base model is a representative small open vision-language model.
    Section 1/4; the generalization claim 'small open model can be competitive' is made from one model family.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation." pith.science (2026). https://pith.science/paper/5EEQRTGU

@misc{pith2026260728987,
  author       = {Pith},
  title        = {Pith review of: A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EEQRTGU}},
  note         = {Machine review of arXiv:2607.28987}
}
read the original abstract

Handwritten UML sketches are common in early software design, but turning them into structured, analysable modelling artefacts still requires manual reconstruction. Vision-language models can generate PlantUML from diagram images, but prompt-based use treats this as image-to-text generation rather than structured model generation. We investigate formalism-aware rewards: feedback signals derived from analysable model representations rather than surface text. In a worked example, we adapt a vision-language model for handwritten UML-to-PlantUML generation using super-vised fine-tuning followed by Group Relative Policy Optimisation. Generated PlantUML is compared against target representations, using XMI for class diagrams and control-flow graphs for activity diagrams. Emerging results show that the adapted model improves compilability and conversion quality over the untuned open model and one proprietary baseline, while remaining competitive with a stronger proprietary baseline on class diagrams. The added benefit of the reward-guided stage remains open on the current held-out set. Error analysis and metric-validity results show that modelling acceptability is only partially captured, motivating rewards and evaluations that combine model analysis with human judgement.

Figures

Figures reproduced from arXiv: 2607.28987 by the authors.

Figure 1
Figure 1. The two-stage, formalism-aware reward loop. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A meaning-preserving deviation: from the handwritten sketch (a), the reference uses an if-branch (b) and the model [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams

    cs.SE 2026-08 conditional novelty 7.0 of 10

    CAS2UML provides 557 hand-drawn class and activity UML diagrams with validated PlantUML ground truth, plus a validation tool and scripts for reproducible sketch-to-UML benchmarking.

Reference graph

Works this paper leans on

21 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anonymous. 2025. Dataset Paper Accepted but Not Yet Publicly Available. Anonymised reference. Full citation withheld for double-anonymous review and will be added in the camera-ready version

    2025

  2. [2]

    Monique Axt. 2023. Transformation of sketchy UML Class Diagrams into for- malPlantUML models

    2023

  3. [3]

    Sebastian Baltes and Stephan Diehl. 2014. Sketches and diagrams in practice. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT/FSE14). ACM, 530–541. doi:10.1145/2635868. 2635891

    doi:10.1145/2635868 2014

  4. [4]

    Averi Bates, Ryan Vavricka, Shane Carleton, Ruosi Shao, and Chongle Pan. 2025. Unified modeling language code generation from diagram images using mul- timodal large language models.Machine Learning with Applications20 (2025), 100660. doi:10.1016/j.mlwa.2025.100660

    arXiv 2025

  5. [5]

    Aaron Conrardy and Jordi Cabot. 2024. From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs. arXiv:2404.11376 [cs.SE] https://arxiv.org/abs/2404.11376

    arXiv 2024

  6. [6]

    Daniele De Bari, Giacomo Garaccione, Riccardo Coppola, Marco Torchiano, and Luca Ardito. 2024. Evaluating Large Language Models in Exercises of UML Class Diagram Modeling. InProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement(Barcelona, Spain)(ESEM ’24). Association for Computing Machinery, New York, ...

    arXiv 2024

  7. [7]

    2025.Harnessing Large Language Models for Automated Soft- ware Diagram Generation

    Grant Guernsey. 2025.Harnessing Large Language Models for Automated Soft- ware Diagram Generation. Master’s thesis. University of Cincinnati. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view? acc_num=ucin1746701542674719

    2025

  8. [8]

    Chaudron

    Bilal Karasneh and Michel R.V. Chaudron. 2013. Extracting UML models from images. In2013 5th International Conference on Computer Science and Information Technology. 169–178. doi:10.1109/CSIT.2013.6588776

    arXiv 2013

Show all 21 references
  1. [9]

    Hatice Koç, Ali Mert Erdoğan, Yousef Barjakly, and Serhat Peker. 2021. UML Diagrams in Software Engineering Research: A Systematic Literature Review. Proceedings74, 1 (2021). doi:10.3390/proceedings2021074013

    2021 doi

  2. [10]

    Jun Ling, Yao Qi, Tao Huang, Shibo Zhou, Yanqin Huang, Jiang Yang, Ziqi Song, Ying Zhou, Yang Yang, Heng Tao Shen, and Peng Wang. 2025. Table2LaTeX- RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models. arXiv:2509.17589 [cs.AI] ht...

    2025

  3. [11]

    Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, et al

  4. [12]

    Yuanqing Liu, Ziming Yang, Yulong Li, and Yue Yang. 2026. FloorplanVLM: A Vision-Language Model for Floorplan Vectorization. arXiv:2602.06507 [cs.CV] https://arxiv.org/abs/2602.06507

    2026

  5. [13]

    Artem Naboichenko and René Peinl. 2026. Unlocking UML Class Diagram Understanding in Vision Language Models. arXiv:2605.11634 [cs.CV] https: //arxiv.org/abs/2605.11634

    2026 arXiv

  6. [14]

    HG Ranjani and Rutuja Prabhudesai. 2025. Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs. arXiv:2509.11667 [cs.LG] https://arxiv.org/abs/2509.11667

    2025

  7. [15]

    Mantas Ražinskas, Benas Mili¯unas, Mantas Jurgelaitis, Lina Čeponien˙e, and Lina Bisikirskien˙e. 2024. Transforming Sketches of UML Use Case Diagrams to Models. IEEE Access12 (2024), 185826–185837. doi:10.1109/ACCESS.2024.3514455

    2024

  8. [16]

    Gianna Reggio, Maurizio Leotta, Filippo Ricca, and Diego Clerissi. 2014. What are the used activity diagram constructs? a survey. In2014 2nd International Con- ference on Model-Driven Engineering and Software Development (MODELSW ARD). 87–98

    2014

  9. [17]

    Simon Scholz and Mersedeh Sadeghi. 2026. CAS2UML: A Handwritten Sketch- to-PlantUML Dataset for Class and Activity Diagrams. InProceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026). Accepted for publication

    2026

  10. [18]

    Harald Störrle. 2017. How are Conceptual Models used in Industrial Software Development? A Descriptive Survey. InProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering(Karlskrona, Sweden)(EASE ’17). Association for Computing Machi...

    2017

  11. [19]

    Jagoda Walny, Jonathan Haber, Marian Dörk, Jonathan Sillito, and Sheelagh Carpendale. 2011. Follow that sketch: Lifecycles of diagrams and sketches in software development. In2011 6th International Workshop on Visualizing Software for Understanding and Analysis (VISSOFT). 1–8....

    2011

  12. [20]

    Zhenchang Xing and Eleni Stroulia. 2005. UMLDiff: an algorithm for object- oriented design differencing. InProceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 54–65

    2005

  13. [2026]

    Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization.arXiv preprint arXiv:2601.05242(2026)

    2026 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.