{"id":"7e8b1b7f-dd16-4fb9-8d6e-acd33feabf04","arxiv_id":"2607.28987","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using XMI and control-flow-graph comparison as reinforcement-learning rewards makes a 4B open vision-language model competitive with larger proprietary models on handwritten UML-to-PlantUML, but the reward stage's benefit over SFT is not statistically detectable.","lead":"This paper tests a training loop that scores a model's generated UML code not by surface text similarity but by whether it can be parsed into a real model — XMI for class diagrams, control-flow graphs for activity diagrams — and compared with the target. A small open vision-language model fine-tuned this way moves from last to second on handwritten UML-to-PlantUML conversion, though the reward-guided step's gain over plain fine-tuning is not yet measurable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Second-place ranking rests on uncorrected point estimates from 30 sketches; reward-stage benefit is statistically indistinguishable from SFT alone.","rationale":"The reader's weakest-assumption choice, metric validity, is a real concern but is substantially mitigated by the independent human ranking study, which reproduces the same overall ordering with strong inter-rater agreement. The more load-bearing weakness is that the paper's headline comparisons are not protected against sampling noise, and the novel component of the method—the formalism-aware reward stage—shows no statistically detectable benefit over SFT alone. The paper is honest about the latter, but the central claim as stated ('full adaptation pipeline yields a competitive model') still leans on the full pipeline even though the evidence is compatible with an SFT-only explanation. These concerns do not require rejection: the paper is framed as emerging results, the human evaluation is a genuine independent signal, and the authors disclose the relevant limitations. They do, however, justify the reader's CONDITIONAL verdict rather than acceptance as a hardened claim.","tokens_in":10726,"tokens_out":10657,"duration_ms":122567,"concrete_test":"Bootstrap the per-sketch paired differences on the 30 held-out examples: compute 95% CIs and paired Wilcoxon p-values for Ours−Gemini and Ours−GPT on content score and per-sketch Borda, and for SFT-only−full on both metrics. If any Ours−Gemini or Ours−GPT CI straddles zero, the 'second of four' ranking is not statistically supported. If the SFT-only−full CI excludes zero in favor of SFT-only, the competitive result is attributable to SFT, not the reward loop.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the full SFT+GRPO pipeline yields a competitive model, second of four—requires two things: (i) the ranking is not sampling noise, and (ii) the full pipeline, not just supervised fine-tuning, is responsible for the rank. Neither is currently established. On (i), all headline comparisons are point estimates from 15 diagrams per type, with no confidence intervals or paired tests for Ours-vs-Gemini or Ours-vs-GPT; the only reported tests are SFT-vs-full (p=0.94 class, p=0.64 activity). On (ii), Table 2 shows the SFT-only checkpoint matches the full pipeline on class components and is pointwise higher on every activity component (compilation 100 vs 86.7; structural 86.0 vs 73.4; labels 78.6 vs 69.2). Thus the full pipeline's second place could be entirely due to supervised fine-tuning, leaving the paper's novel formalism-aware reward loop without a demonstrated positive effect. The metric-validity issue the reader flags is real but partly mitigated by the human ranking (inter-rater W 0.80-0.91); the unaddressed statistical and attribution issues are more load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'formalism-aware reward loop' for handwritten UML-to-PlantUML generation: rather than scoring generated PlantUML as text, the system compiles it into an analysable representation (XMI for class diagrams, a control-flow graph for activity diagrams) and computes rewards by structural comparison with a reference. A small open vision-language model (Qwen3.5-4B) is adapted with supervised fine-tuning followed by GRPO. On a held-out set of 30 diagrams, the paper reports that the full pipeline compiles 100% of class diagrams and 86.7% of activity diagrams, achieves content scores of 17.60 and 13.71, and is ranked second of four by human raters, behind Gemini 3 Flash and ahead of GPT-4.1 Mini. The SFT-only ablation is statistically indistinguishable from the full pipeline (p=0.94 class, p=0.64 activity), and the automatic content score correlates moderately with human rankings (rho=0.565). The paper carefully hedges most claims and frames the work as emerging evidence.","tokens_in":11021,"tokens_out":4906,"duration_ms":56941,"significance":"If the competitiveness result holds, the paper demonstrates that a small open VLM adapted with formalism-aware rewards can approach proprietary baselines on sketch-to-model conversion, a practically relevant task. The reusable pattern of parsing into a formalism-specific representation and comparing against a reference is a useful contribution, and the paper is unusually honest: it provides an explicit SFT ablation, a human ranking study with high inter-rater agreement, an error taxonomy including meaning-preserving deviations, and a public artifact repository. The main value is in the negative/qualified finding that the reward-guided stage does not yet show added value beyond SFT on this small set, which is itself an informative result for the NIER audience.","major_comments":[{"comment":"The central 'second of four' claim rests on point estimates from n=15 diagrams per type. Class content is 17.60 vs. 17.36 for Gemini; activity is 13.71 vs. 14.01; human Borda is 0.613 vs. 0.731 overall and 0.564 vs. 0.787 for activity. No confidence intervals, bootstrap intervals, or paired significance tests are reported for these comparisons; the only inferential statistics are the SFT-vs-full Wilcoxon tests (p=0.94, p=0.64). With 15 diagrams per type, gaps of 0.24 content points or 0.05–0.17 Borda units are plausibly sampling noise. Please add paired tests or bootstrap CIs for Ours-vs-Gemini and Ours-vs-GPT, or soften the ordering claim to 'not distinguishable from Gemini on class diagrams' and 'ahead of GPT-4.1 Mini' only where statistically supported.","section":"§4.1–4.2, Tables 1–2"},{"comment":"The paper credits the 'full adaptation pipeline' with moving the model from last to second, but Table 2 shows the SFT-only checkpoint is equal to the full pipeline on all class-diagram components (compilation 100 vs. 100; class-level 98.3 vs. 98.3; attributes 98.0 vs. 98.0; relationships 83.6 vs. 84.2; methods 91.2 vs. 90.9) and pointwise higher on every activity-diagram component (compilation 100 vs. 86.7; structural 86.0 vs. 73.4; label 78.6 vs. 69.2). Therefore the second-place result could be produced entirely by supervised fine-tuning, and the novel formalism-aware reward loop has no demonstrated positive effect. This is load-bearing for RO1. Either include the SFT-only checkpoint in the overall ranking/Borda analysis, or explicitly restate RO1 as 'SFT-based adaptation is competitive; the reward-stage contribution remains open' so that the headline does not attribute the ranking to","section":"§4.4 vs. Table 2"},{"comment":"The automatic evaluation reuses the formalism-aware reward functions of Section 3 as model-level metrics. Because GRPO is trained to maximize exactly those functions, the automatic score gains are partly a consequence of optimizing the metric itself, not independent evidence of conversion quality. The human ranking study is the genuinely independent check, but it is based on 30 sketches and its overall agreement with the automatic score is only moderate (rho=0.565). This circularity is acknowledged indirectly but should be stated more prominently; Table 1's 'automatic and human evaluation' side-by-side suggests two independent confirmations, whereas only the human Borda is independent of the training objective.","section":"§4.1 and §3"}],"minor_comments":[{"comment":"The dataset is first cited as [1] ('Dataset Paper Accepted but Not Yet Publicly Available', anonymised), while [17] gives the actual CAS2UML reference. If the dataset paper is under double-anonymous review, unify the citations and state that the dataset will be made public; the current placeholder makes it hard to verify the data provenance.","section":"References [1] and [17]"},{"comment":"The sentence ending 'rather than imitation of reference text.Wh' contains a stray 'Wh' fragment. Please fix the typo.","section":"§4.1, final paragraph"},{"comment":"The UML-knowledge test used for screening is described only as a 'competence threshold'. Provide the threshold or a supplementary description, and clarify how the 26 participants were distributed across the three blocks (a random assignment does not guarantee that every block received at least one rater, even though the design intends full coverage).","section":"§4.2, human study"},{"comment":"The caption says 'The two render to identical diagrams', but the two PlantUML snippets in (b) and (c) are syntactically different. I assume the rendered diagram images are visually identical; please phrase it as 'render to visually identical diagrams' to avoid ambiguity.","section":"Figure 2"},{"comment":"The tables would benefit from repeating the sample size (n=15 per diagram type) and explicitly noting that values are point estimates without error bars. This is especially important given the small n and the absence of confidence intervals.","section":"Tables 1–2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and well-scoped for a NIER track, and the SFT-vs-full ablation is a strength. My main concern is that the RO1 claim ('full adaptation pipeline yields a competitive model') is stronger than what the evidence supports: the second-place ranking has no uncertainty quantification, and Table 2 suggests SFT alone reproduces or exceeds the full pipeline's scores. These issues are fixable by adding CI/paired tests and by including the SFT checkpoint in the overall ranking analysis or in the human study. If the authors align the claims with the evidence, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a useful worked example of closing the loop between a modelling formalism and a VLM training signal: generated PlantUML is compiled to XMI (class diagrams) or parsed to a control-flow graph (activity diagrams), and those structural comparisons serve as both GRPO rewards and evaluation metrics. Second, the paper is unusually honest about its own limits—it explicitly says the reward stage's benefit over SFT is not statistically detectable (p=0.94 and p=0.64), and it discloses that the automatic evaluation reuses the training rewards.\n\nWhat's good: the reward design is concrete and the infrastructure is real (persistent conversion server, malformed-generation handling, custom ANTLR4 grammar). The human ranking study is well-run—26 screened raters, strong inter-rater agreement (Kendall's W 0.80–0.91)—and gives the headline ordering independent support. The error analysis is thoughtful, particularly the meaning-preserving vs. meaning-altering distinction, with the switch-vs-if example as a clear illustration. The paper does not overclaim; it frames the result as feasibility, not a definitive win.\n\nThe soft spots are mostly the ones the paper itself flags, but they are load-bearing for the central contribution. The evaluation circularity is real: since the same reward functions were optimized by GRPO, the 'second place' result in Tables 1–2 partly measures how well each model fits that specific metric. The human study mitigates this, but the correlation is only moderate (rho=0.565), and the metric demonstrably penalizes a faithful rendering. The held-out set is 15 diagrams per type, with no confidence intervals on the model-vs-model comparisons. The stress-test note is right: SFT alone matches the full pipeline on class components and is pointwise higher on every activity component, so the pipeline's rank could be entirely due to supervised fine-tuning. The novel reward loop has no demonstrated positive effect. That is a genuine gap, even though the paper is transparent about it.\n\nBottom line: this is a solid, well-scoped empirical paper that deserves serious peer review. It will be of most value to people working on VLM adaptation for structured or model-like outputs and to anyone thinking about reward overfitting and evaluation circularity in RL fine-tuning. I would ask for CIs or paired tests on the headline rankings and a more explicit statement of what the GRPO stage adds if it is not moving the needle.","headline":"Honest, well-scoped worked example: small open VLM becomes competitive on handwritten UML-to-PlantUML, but the GRPO stage's contribution remains indistinguishable from SFT, and the headline metric is the training reward.","tokens_in":11538,"tokens_out":3757,"would_cite":false,"duration_ms":37930,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Parsing generated UML into XMI and control-flow graphs gives a small open model trained with formalism-aware rewards a competitive edge on handwritten UML-to-PlantUML conversion.","keywords":["UML","PlantUML generation","vision-language models","reinforcement learning","formalism-aware rewards","XMI comparison","control-flow graphs","sketch-to-model conversion"],"falsifier":"Take the switch-branch example the paper reports (an if-branch rewritten as a switch-branch that renders identically). If a panel of UML practitioners judges the two outputs as equally good interpretations of the same handwritten sketch, yet the formalism-aware metric scores them substantially apart, that falsifies the metric's validity as a proxy for conversion quality—and since the same metric drives both training and evaluation, it would undermine the comparative claims. A quantitative version: on a set of meaning-preserving alternatives, compute the correlation between the metric and human","tokens_in":10592,"feed_emoji":"📐","tokens_out":9374,"duration_ms":81095,"temperature":0.7,"pith_summary":"This paper tries to establish that the right way to grade a model's UML generation is to look at the structure of the model it produces, not the text of the code. It adapts a small open vision-language model with supervised fine-tuning plus reinforcement learning, where the reward is computed by parsing the generated PlantUML into XMI (for class diagrams) or a control-flow graph (for activity diagrams) and comparing against the reference in those formalisms. The adapted model becomes competitive: it compiles every class diagram, raises activity-diagram compilation from 20% to 86.7%, and moves from last to second among four models in both automatic and human evaluations. The paper is careful to say that the added benefit of the reward-guided stage over fine-tuning alone is not yet statistically detectable, and that the automatic metric only partly captures what humans mean by modelling quality. The value of the work is a pattern—derive the learning signal from analysable model representations—that could generalize to other notations.","feed_headline":"Parsed-structure rewards put a small open UML model second of four","feed_subtitle":"Scoring conversion by parsed XMI and control-flow graphs, not text, makes a small open model competitive on handwritten UML.","key_machinery":"The central mechanism is the formalism-aware reward loop. Generated PlantUML is not scored as text; it is compiled and parsed into the formalism's canonical representation—XMI for class diagrams, a control-flow graph for activity diagrams—and compared with the reference in that representation. Class-diagram reward components cover compilation, classes, relationships, methods, and attributes; activity-diagram reward components cover compilation, CFG structure (via weighted Jaccard similarity and Hungarian-algorithm graph assignment), and label similarity (via TF-IDF). This same representation-based scoring is used both as the reinforcement-learning reward and as the automatic evaluation metri","core_discovery":"Handwritten-UML-to-PlantUML should be treated as structured model generation, not image-to-text generation. The paper adapts a small open vision-language model via supervised fine-tuning plus group-relative policy optimisation, with rewards derived from parsing the generated PlantUML into XMI (class diagrams) or a control-flow graph (activity diagrams) and comparing those structures to the reference. On a held-out set of 30 sketches the adapted model compiles 100% of class and 86.7% of activity diagrams and moves from last to second in automatic and human rankings, ahead of one proprietary baseline and behind another. The reward stage's gain over fine-tuning alone is not statistically signif","pith_inferences":["Because the reward stage shows no significant gain over supervised fine-tuning on this held-out set, the strong results may be driven mostly by the supervised phase; the real test of formalism-aware rewards is whether they add value with larger datasets or with decoupled per-component normalisation.","If a reward were designed to recognise semantic equivalence—e.g., treating a switch-branch and an if-branch as the same control flow—the metric-human correlation would likely rise, and the reward loop could become more effective as a training signal.","The error analysis points to visual grounding, not PlantUML generation, as the bottleneck for remaining meaning-altering errors; improved fine-grained perception (reading multiplicities, arrowheads) may yield gains that compound with better reward design.","A direct extension would apply the same loop to state machines or sequence diagrams; if the pattern transfers, it strengthens the claim that formalism-aware rewards are a general method rather than a two-diagram special case."],"forward_implications":["If the full pipeline is as competitive as reported, a small open model can serve as a practical sketch-to-model converter, reducing reliance on larger proprietary systems for this task.","Compilability of generated class diagrams can reach 100% and activity diagrams 86.7% on this data, making the outputs usable in downstream model-analysis workflows without manual repair.","Formalism-aware metrics capture compilability and recoverable structure but only moderately align with human judgement, so any evaluation of such models that relies solely on them will misrank acceptable alternative formulations.","The reward-loop pattern provides a template for turning model analysis into training feedback, though each new formalism requires designing its own representation and comparison."],"fun_headline_variants":["Parsed UML rewards lift small open model to second place","Structured rewards, not text, make open UML model competitive","Counting parsed XMI and control-flow graphs boosts UML model","Formalism-aware rewards put small model second in UML test","Reward loop parses UML structures to rival proprietary models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the formalism-aware scores—XMI structural comparison and CFG comparison with hand-chosen weights and edit costs—are a valid measure of conversion quality; the paper itself finds only moderate agreement with human rankings (rho=0.565) and shows the metric penalises meaning-preserving deviations, so if this metric is arbitrary, the comparative results are an artefact of the metric.","fun_headline_variants_meta":{"raw":{"variants":["Parsed UML rewards lift small open model to second place","Structured rewards, not text, make open UML model competitive","Counting parsed XMI and control-flow graphs boosts UML model","Formalism-aware rewards put small model second in UML test","Reward loop parses UML structures to rival proprietary models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1309,"prompt_tokens":723,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":504}},"tokens_in":467,"tokens_out":586,"duration_ms":5664,"temperature":1.0,"reasoning_tokens":504,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T15:51:58.325547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the switch-branch example the paper reports (an if-branch rewritten as a switch-branch that renders identically). If a panel of UML practitioners judges the two outputs as equally good interpretations of the same handwritten sketch, yet the formalism-aware metric scores them substantially apart, that falsifies the metric's validity as a proxy for conversion quality—and since the same metric drives both training and evaluation, it would undermine the comparative claims. A quantitative version: on a set of meaning-preserving alternatives, compute the correlation between the metric and human","supporting_citations":[],"review_version":1}