Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that nine LLMs, given a methodless 21-class UML diagram and 21 structured use cases, can generate syntactically valid, structurally faithful method enrichments with strong cross-model naming consensus.

desk verdict Solid multi-model benchmark with honest limits, but the headline naming-consensus result is largely manufactured by the shared prompt and use-case wording. read the letter →

arxiv 2506.00788 v1 pith:ODBXQHTB submitted 2025-06-01 cs.SE cs.AI

Djaber Rouabhia , Ismail Hadjadj This is my paper
classification cs.SEcs.AI
keywords AI-drivenSoftwareEngineeringAutomatedUMLModelingNaturalLanguagetoBehavioralEnrichmentClassDiagramMethodInferenceLLMEvaluationMetricsPlant
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large language models can automate the behavioral enrichment of UML (Unified Modeling Language) class diagrams. Given a methodless 21-class diagram and 21 structured waste-management use cases, nine LLMs produced 90 augmented diagrams containing 3,373 methods, all syntactically valid PlantUML, with 84 of 90 compiling without syntax errors. The more striking result is convergence: method names such as assignRole, verifyEmail, updatePassword, and cancelRequest were generated by all nine models, and core methods were placed in the same dominant class in 92.7% of cases. The authors take this as evidence that LLMs internalize enough domain semantics to scaffold behavioral models, while explicitly stating that semantic and functional correctness were not evaluated and that human oversight remains necessary. If right, this would convert a largely manual bottleneck between static class structure and executable design into a fast, first-draft automation.

What carries the argument

The load-bearing mechanism is a generate–measure–compare pipeline. Each model received the same three-part prompt—instructions, the methodless PlantUML baseline diagram, and the 21 structured use cases—and was allowed a single-shot generation with no clarification or retries. The resulting 90 diagrams were parsed into JSON and scored on six metric families: method quantity, signature richness, annotation completeness, structural fidelity, syntactic correctness, and naming convergence. The instruments that carry the paper's novelty are the self-derived consensus metrics: Top-Method Consensus, which pools normalized method names and measures pairwise Jaccard overlap between models; Core-Method Consensus, which counts how many models independently generate each of the top-37 methods; and Structural Placement Consistency, which measures whether a core method is assigned to its dominant class. These metrics let the paper claim cross-model agreement without relying on any predefined ground-truth benchmark.

What would settle it

Take the 21 use cases, have a human expert produce a reference set of correct methods and class assignments, then compare the nine models' consensus output against it: the central claim would collapse if the methods generated by all nine models (assignRole, verifyEmail, updatePassword, cancelRequest) match the expert reference no better than random or low-consensus methods do.

Watch

Extended reading notes

Core claim

The paper's central discovery is that nine LLMs, prompted once with a methodless 21-class diagram and 21 structured use cases, produced 90 augmented diagrams containing 3,373 methods, every one a syntactically valid PlantUML diagram and 84 of 90 compiling without syntax errors. The more striking empirical finding is consensus: method names like assignRole, verifyEmail, updatePassword, and cancelRequest were generated by all nine models, and core methods were assigned to the same dominant class 92.7% of the time. The authors take this cross-model lexical and structural convergence as evidence that LLMs capture core domain behaviors from use-case text, not just formatting conventions. They are explicit that semantic and functional correctness were not evaluated, so the central claim is scoped to syntactic validity, structural fidelity, annotation completeness, and naming consistency.

Load-bearing premise

The load-bearing premise is that syntactic validity, structural fidelity, and cross-model naming agreement are adequate stand-ins for semantic correctness; the paper itself states that semantic and functional correctness were not evaluated.

Editorial extensions

If this is right

  • Method-level enrichment of class diagrams can be automated to a first-draft standard, reducing the manual effort between use cases and behavioral UML.
  • Consensus across independently prompted models can be used to flag methods and class placements that are safe to accept without expert review and those that need human attention.
  • No single model dominates every dimension, so model selection should be driven by project priorities: coverage and annotation versus parameter detail versus structural preservation.
  • Rapid, single-shot generation supports Agile iteration, allowing design drafts to be refreshed within minutes.
  • The public release of the 90 diagrams and 3,373 methods gives the community a reproducibility baseline for future LLM UML enrichment studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but if cross-model naming consensus tracks semantic validity, the consensus sets could serve as a reference-free ground truth for evaluating single-model output, and the top-37 method list could become a reusable behavioral benchmark for similar domains.
  • The same generate–measure–compare setup could be extended to state-machine and sequence-diagram enrichment, where consensus metrics would reveal which behavioral transitions are strongly entailed by use-case text versus genuinely ambiguous.
  • Because the prompt explicitly forbids UI and infrastructure methods and insists on atomic actions, the reported convergence may partly reflect prompt-induced constraints; varying those constraints would show how much of the consensus is intrinsic to the models rather than shaped by instructions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents an empirical evaluation of nine LLMs on the task of enriching a methodless UML class diagram (21 classes, 17 relationships) with behavioral methods derived from 21 structured waste-management use cases. Each model was run ten times (90 diagrams total, 3,373 methods) and the outputs were assessed on six metrics: method quantity, signature richness, annotation completeness, structural fidelity, syntactic correctness, and naming convergence (TMC/CMC/SPC). The authors report high syntactic validity, strong structural preservation for most models, and notable cross-model agreement on core method names such as assignRole, verifyEmail, updatePassword, and cancelRequest. They interpret this consensus as evidence of shared domain understanding and discuss practical implications for AI-assisted UML modeling, Agile workflows, and the necessity of human oversight. All artifacts are publicly available. The paper is honest in its limitations section that semantic and functional correctness are not evaluated.

Significance. The study addresses a genuinely underexplored problem—automated generation of behavioral methods for UML class diagrams—and provides a reproducible, multi-model benchmark with publicly available artifacts (prompts, .puml files, parsed JSON, metric CSVs). The statistical toolset (Kruskal-Wallis, Dunn with Holm correction, Cliff's delta, bootstrap) is appropriate for non-normal, multi-model comparisons. Explicitly scoping semantic correctness out of the study is a strength, not a weakness, as it sets a measurable baseline. However, the paper's central interpretive claim—that naming consensus reflects 'shared domain understanding'—is not supported by the experimental design, because the consensus is measured against self-derived reference sets and the input use-case wording is the direct source of the most frequent method names. If the claims are reframed as descriptive evidence of prompt-following reliability and cross-model lexical consistency, the empirical contribution is meaningful and suitable for a software-engineering venue after revision.

major comments (4)
  1. [Abstract, Table 6] The abstract states 'All LLMs produced valid PlantUML diagrams adhering to UML conventions,' but Table 6 lists 6 errored diagrams out of 90 (Gemini 2.5 Pro: 2; DeepSeek R1, Grok 3, ChatGPT-4o, Llama 4: 1 each). This is a direct, factual contradiction in the headline claim. Section 4.1 reports 'only six total (6.67%)' syntax errors and Section 5.2 states 'greater than 90% success rates,' which are consistent with Table 6 but not with the abstract. The abstract must be corrected, e.g., to '84 of 90 diagrams compiled without syntax errors.'
  2. [§3.4, §5.3] The interpretation of cross-model naming consensus as evidence of 'shared domain understanding' and that models 'internalize essential domain semantics' is not supported by the experimental design. All models received the identical three-part prompt (§3.2), which explicitly instructs using domain verbs and a 1:1 action-method mapping, and the use-case corpus itself contains the action phrases that later appear as top-k methods (e.g., 'verify email', 'update password'). The TMC and CMC reference sets are derived from the pooled model outputs themselves (§3.4), so coverage measures self-consistency rather than agreement with an independent behavioral ground truth. Moreover, the sentence in §5.3 claiming this convergence was observed 'without explicit shared prompts or references' is factually incorrect: the prompts were standardized and shared. Please reframe the conclusions to state that models reliably extract and reproduce input-specified actions, and ideally add a control (e.g., paraphrased use cases) before claiming deeper semantic internalization.
  3. [§3.6, §3.5] The experimental protocol for the reported 'three independent seeds (17, 42, 123)' is unclear. The paper also states there are ten independent runs per model, yielding 90 diagrams. If three seeds were applied to generation, the expected total would be 30 runs per model (270 diagrams), not 10. Please clarify how the seeds were used (e.g., were the 10 runs split by seed, or do the seeds refer to analysis/bootstrap resampling?). Additionally, the claim that 'sensitivity analyses confirmed stability of main findings across seeds' is asserted in §3.5 but no supporting results are shown anywhere; please provide the sensitivity analysis or remove the claim.
  4. [Abstract, §3.3, Table 3, Appendix B Table 7] The identity of one evaluated model is inconsistent across the manuscript: the abstract and introduction refer to 'Mixtral 8×22B', Table 3 and Figure 2 use 'Mistral 8×7B', and Appendix B Table 7 lists 'Le Chat (Mistral Medium 3)' with undisclosed parameters. Since the model is a central object of the evaluation, this contradiction must be resolved so the reader knows exactly which model was deployed and at what scale.
minor comments (5)
  1. [§3.4] The formulas for Top-Method Consensus and the definition of k are garbled in the rendering (e.g., 'TMC = | TM m=1 Sm|' and 'k = l total methods total diagrams m'). Please repair the LaTeX/typesetting.
  2. [Table 2] TMC and CMC report identical Kruskal–Wallis statistics (H(8)=26.42, p=0.0009). Since the metrics are computed on different data (normalized versus raw method names), this suggests a copy-paste error; please verify the correct values.
  3. [§5.2, §5.10, Table 5] ChatGPT-4o achieves only 49.40% global structural fidelity in Table 5, yet Section 5.10 concludes that 'structural fidelity is generally assured.' This overgeneralization should be qualified to note that some models exhibit substantial structural degradation (ChatGPT-4o, Grok 3).
  4. [§3.5, Appendix B] There are multiple unresolved citation placeholders: 'Romano et al. [?]', 'benchmarking best practices [?]', and several '[?]' entries in the model specification table. The reference list also contains an entry ([17]) that appears incomplete. The manuscript must be finalized before publication.
  5. [§5.3] The phrase 'without explicit shared prompts or references' is misleading because the methodology section describes a standardized, identical prompt for all models. This wording undermines the novelty claim and should be corrected.

Circularity Check

2 steps flagged · score 5.0 of 10

The consensus metrics (TMC/CMC/SPC) build their reference sets from the same pooled outputs they then score, so the paper's central 'consistent naming / shared behavioral understanding' claim is partly self-referential.

  1. self definitional [Section 3.4 (Core-Method Consensus and Top-Method Consensus); interpreted in Section 4.6.2]
    "Method names across all augmented diagrams are pooled and normalized, then the top-k most frequent method names are extracted, where k equals the overall mean number of methods per diagram (k ≈ 38), reflecting typical model output size. For each LLM, we compute: Core-method coverage: Proportion of the top-k core methods present in the model’s outputs."

    The 'core method' set is not an external benchmark: it is the frequency-ranked set of names drawn from the same 90 outputs it is used to score. Each model's own frequent names contribute to the pooled frequencies that select the top-k set, so a model's 'coverage' partly measures self-membership in the corpus. The later claim that assignRole, verifyEmail, updatePassword, and cancelRequest 'were generated by all nine models ... reflecting behaviors that are semantically directly tied to corresponding actions within main scenario use cases' is a restatement of the selection criterion, not independent evidence.

  2. self definitional [Section 3.4 (Structural Placement Consistency); interpreted in Section 4.6.3]
    "dominant class, defined as the class most frequently associated with each method in all diagrams. The Class Match Rate for an LLM is computed as: ClassMatchLLM = |{m ∈ Corek | m assigned to dominant class}| / |{m ∈ Corek | m ∈ LLM output}|"

    The 'dominant class' is the modal class computed from all 90 diagrams, including the LLM whose placement consistency is being computed. A model that consistently places a method in class X helps make X the dominant class and then receives credit for matching that dominant class. The reported 92.7% average placement consistency is therefore agreement with a self-inclusive aggregate rather than clean cross-model agreement. A leave-one-out definition of the dominant class, computed without the scored model, would remove this self-referential component; no such correction is reported.

full rationale

The study has substantial independent content: MQ, SR, AC, SF, and SC are computed directly from the 90 generated diagrams and are reproducible from the public artifacts. The circularity is concentrated in the consensus metrics that anchor the paper's key novel insight (Section 5.3). TMC/CMC define the 'core' method set as the top-k names in the pooled outputs and then measure each model's coverage against that self-derived set; SPC defines the 'dominant class' from all diagrams and then scores each model against it. These are not independent benchmarks but restatements of corpus frequencies and modal placements, so the claim that consensus reflects 'shared behavioral understanding' is partly forced by the metric construction. The paper honestly discloses in Sections 3.7 and 5.5 that semantic and functional correctness were not evaluated and that proxy metrics are used, which limits the overreach but does not remove the self-referential character of the consensus metrics. The standardized shared prompt and use-case action vocabulary (Section 3.2, Appendix A) make the convergence of names like verifyEmail and updatePassword unsurprising; this is a validity confound rather than a definitional circularity and is noted separately. Section 5.3's claim that the convergence was observed 'without explicit shared prompts' is contradicted by the uniform three-part prompt described in Section 3.2. No load-bearing self-citation chain or imported uniqueness theorem was found; the cited works are external related work and standard statistics references. The abstract's 'All LLMs produced valid PlantUML diagrams' is inconsistent with Table 6's six errored diagrams, an internal-consistency issue rather than a circularity step. Overall, because the central consensus claim partially reduces to its own constructed reference sets, the circularity score is 5.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The study is an empirical benchmark, not a derivation. The free parameters are the data-dependent cutoff k for consensus sets and the data-derived 'dominant class' assignment used in SPC. The axioms are domain-level assumptions about the adequacy of proxy metrics and the representativeness of the waste-management benchmark. No new theoretical entities are introduced; the consensus metrics are analytical constructs computed from the data.

free parameters (2)
  • k (top-method/core-method consensus cutoff) = 37 (CMC prose says ~38)
    k = ceil(total methods / total diagrams) = ceil(3373/90) = 38 in the prose, but the paper consistently analyzes 37 core methods in Tables and Figures. All TMC, CMC, and SPC coverage scores depend on this threshold.
  • Dominant class per core method (SPC) = Mode of class assignment across all 90 diagrams
    SPC measures placement consistency against a 'dominant class' that is itself computed from the same 90 diagrams being scored, so the placement benchmark is not independent of the models.
assumptions (4)
  • domain assumption Proxy metrics (syntactic, structural, lexical) suffice to characterize behavioral enrichment quality
    Section 3.7 states semantic and functional correctness are not evaluated; the conclusions about 'well-structured methods' rest entirely on these proxies.
  • domain assumption The waste-management diagram and 21 use cases are a representative benchmark for UML behavioral modeling
    Sections 3.6 and 5.5 acknowledge domain specificity; generalizability to other domains is asserted as future work.
  • domain assumption Runs are independent and exchangeable across LLM API sessions
    Section 5.5 notes API version and environment variability can affect outputs; the statistical tests treat the 10 runs per model as i.i.d. samples.
  • standard math Nonparametric statistical tests (Kruskal-Wallis, Dunn, chi-square, Wilcoxon) are valid for the reported data
    Appendix D defines the tests; no violation of assumptions is discussed, and Holm correction is applied within metric families.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generation." pith.science (2026). https://pith.science/paper/ODBXQHTB

@misc{pith2026250600788,
  author       = {Pith},
  title        = {Pith review of: Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODBXQHTB}},
  note         = {Machine review of arXiv:2506.00788}
}
read the original abstract

Automating the enrichment of UML class diagrams with behavioral methods from natural language use cases is a significant challenge. This study evaluates nine large language models (LLMs) in augmenting a methodless UML diagram (21 classes, 17 relationships) using 21 structured waste-management use cases. A total of 90 diagrams (3,373 methods) were assessed across six metrics: method quantity, signature richness (visibility, names, parameters, return types), annotation completeness (linking to use cases/actions), structural fidelity, syntactic correctness (PlantUML compilation), and naming convergence (across models). All LLMs produced valid PlantUML diagrams adhering to UML conventions. Some models excelled in method coverage and annotation accuracy, while others showed richer parameterization but weaker traceability. These results demonstrate that LLMs can generate well-structured methods with consistent naming, advancing automated behavioral modeling. However, inconsistencies in annotations and signatures highlight the need for improved prompt engineering and model selection. The rapid generation of these methods supports Agile practices by enabling faster design iterations. Despite their capabilities, human oversight is essential to ensure accuracy, appropriateness, and semantic alignment. This positions LLMs as collaborative partners in software design. All experimental artifacts (\texttt{.puml}, \texttt{.png}, \texttt{.csv}) are publicly available for reproducibility.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 36 canonical work pages

  1. [1]

    Soft- ware and Systems Modeling 21(3), 1015–1045 (2022) https://doi.org/10.1007/ s10270-021-00942-6

    Saini, R., Mussbacher, G., Guo, J.L.C., Kienzle, J.: Automated, interactive, and traceable domain modeling empowered by artificial intelligence. Soft- ware and Systems Modeling 21(3), 1015–1045 (2022) https://doi.org/10.1007/ s10270-021-00942-6

    work page 2022

  2. [2]

    Soft- ware and Systems Modeling 21(3), 1071–1089 (2022) https://doi.org/10.1007/ s10270-022-00975-5 35

    Weyssow, M., Sahraoui, H.A., Syriani, E.: Recommending metamodel con- cepts during modeling activities with pre-trained language models. Soft- ware and Systems Modeling 21(3), 1071–1089 (2022) https://doi.org/10.1007/ s10270-022-00975-5 35

    work page 2022

  3. [3]

    In: Innovations in Smart Cities Applications, Volume 8

    Zakaria, B., Bouziane, E.M., Jakimi, A., Oualla, M., Saadane, R.: Towards an approach for extracting uml class diagrams using advanced language models. In: Innovations in Smart Cities Applications, Volume 8. Lecture Notes in Networks and Systems, vol. 1310, pp. 83–92. Springer, ??? (2025). https://doi.org/10.1007/ 978-3-031-88653-9 9

    work page 2025

  4. [4]

    Software and Systems Modeling 22(3), 781–793 (2023) https://doi.org/10.1007/ s10270-023-01105-5

    C´ amara, J., Troya, J., Burgue˜ no, L., Vallecillo, A.: On the assessment of gen- erative ai in modeling tasks: an experience report with chatgpt and uml. Software and Systems Modeling 22(3), 781–793 (2023) https://doi.org/10.1007/ s10270-023-01105-5

    work page 2023

  5. [5]

    arXiv preprint arXiv:2411.09224 (2024) https://doi.org/10.48550/arXiv.2411.09224

    Siam, M.K., Gu, H., Cheng, J.Q.: Programming with AI: Evaluating chat- gpt, gemini, alphacode, and github copilot for programmers. arXiv preprint arXiv:2411.09224 (2024) https://doi.org/10.48550/arXiv.2411.09224

    arXiv doi:10.48550/arxiv.2411.09224 2024

  6. [6]

    arXiv preprint arXiv:2107.03374 (2021) https://doi.org/10.48550/arXiv.2107.03374

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Oliveira Pinto, H.P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., McCandlish, S., Brown, T., Chess, B., Child, R., Gray, S., Radford, A., Sutskever, I., Amodei, D.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021) https://doi.org/10.48550/arXiv.2107.03374

    arXiv doi:10.48550/arxiv.2107.03374 2021

  7. [7]

    arXiv preprint arXiv:2108.07732 (2021) https://doi.org/10.48550/arXiv

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C.J., Terry, M., Le, Q.V., Sutton, C.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021) https://doi.org/10.48550/arXiv. 2108.07732

    arXiv doi:10.48550/arxiv.2108.07732 2021

  8. [8]

    arXiv preprint arXiv:2105.09938 (2021) https://doi.org/ 10.48550/arXiv.2105.09938

    Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., Steinhardt, J.: Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021) https://doi.org/ 10.48550/arXiv.2105.09938

    arXiv doi:10.48550/arxiv.2105.09938 2021

Show all 43 references
  1. [9]

    Science 378(6624), 1092–1097 (2022) https://doi.org/10.1126/science.abq1158

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A.D., Hubert, T., Choy, P., Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D., Robson, E...

    2022 doi

  2. [10]

    In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE)

    Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., Lou, Y.: Evaluating large language models in class-level code generation. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE). ACM, ??? (2024). https:/...

    2024

  3. [11]

    Information and Software Technology 64, 18–33 (2014) https://doi.org/10.1016/j.infsof.2015.03.006 36

    Wohlin, C.: Guidelines for snowballing in systematic literature studies and a repli- cation in software engineering. Information and Software Technology 64, 18–33 (2014) https://doi.org/10.1016/j.infsof.2015.03.006 36

    2014 doi

  4. [12]

    Addison-Wesley Professional, Boston, MA, USA (2000)

    Cockburn, A.: Writing Effective Use Cases. Addison-Wesley Professional, Boston, MA, USA (2000)

    2000

  5. [13]

    Prentice Hall PTR, Upper Saddle River, NJ, USA (2004)

    Larman, C.: Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development, 3rd edn. Prentice Hall PTR, Upper Saddle River, NJ, USA (2004)

    2004

  6. [14]

    Addison-Wesley, Reading, MA, USA (1992)

    Jacobson, I.: Object-Oriented Software Engineering: A Use Case Driven Approach. Addison-Wesley, Reading, MA, USA (1992)

    1992

  7. [15]

    Routledge, Hillsdale, NJ, USA (1988)

    Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, 2nd edn. Routledge, Hillsdale, NJ, USA (1988)

    1988

  8. [16]

    arXiv preprint arXiv:2002.06305 (2020) https://doi.org/10.48550/ arXiv.2002.06305

    Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., Smith, N.A.: Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 (2020) https://doi.org/10.48550/ arXiv.2002.06305

    2020 doi arXiv

  9. [17]

    In: Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp

    Li, X., Wang, D.G., Wang, S., Wang, S., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Z., Wang, Z., et al.: Evaluating large language models trained on code. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 12345–12356 (...

    2022

  10. [18]

    In: IEEE Transactions on Software Engineering, vol

    Arcuri, A., Briand, L.: Random testing: Theoretical results and practical impli- cations. In: IEEE Transactions on Software Engineering, vol. 38, pp. 258–277. IEEE, ??? (2014). https://doi.org/10.1109/TSE.2011.121 37 A Prompt Design The following is the full instruction prompt...

    2014 doi

  11. [19]

    Method Design • Create only methods addressing uncovered actions • Assign methods exclusively to classes owning the corresponding responsibility (SRP enforcement) • Modify existing methods only if responsibilities exactly match

  12. [20]

    Action Mapping • 1 method = 1 atomic business action (split at “And” clauses) • Allow technical sub-actions only when serving a single business purpose

  13. [21]

    State Management • Implement explicit state transitions (e.g., confirmOrder()) • Embed state rules directly in methods; prohibit generic setters

  14. [22]

    Naming & Structure • Apply domain verbs reflecting class responsibilities (e.g., calculateTotal(), never presentation terms) • Respect UML naming conventions and visibility modifiers (‘-‘, ‘+‘, ‘#‘, ‘ ‘) • Maintain original UML notation/style/structure Constraints • Await clas...

    2025

  15. [24]

    39 These metrics correspond directly to the quantitative dimensions analyzed in the Results section, providing a formal foundation for measuring UML behavioral enrichment quality

    the specific context of its use. 39 These metrics correspond directly to the quantitative dimensions analyzed in the Results section, providing a formal foundation for measuring UML behavioral enrichment quality. C.1 Method Generation and Structural Fidelity Method Quantity (M...

  16. [25]

    the corresponding formula, and

  17. [26]

    D.1 Kruskal–Wallis Test

    the specific context of its use in this study. D.1 Kruskal–Wallis Test

  18. [27]

    It is an extension of the Mann–Whitney U test when comparing more than two groups

    Definition: The Kruskal–Wallis test is a rank-based non-parametric test used to determine whether there are statistically significant differences between the medi- ans of three or more independent groups. It is an extension of the Mann–Whitney U test when comparing more than t...

  19. [28]

    The test statistic H is: H = 12 N (N + 1) kX i=1 R2 i ni − 3(N + 1) Under the null hypothesis, H approximates a chi-squared distribution with k − 1 degrees of freedom

    Formula: Let N be the total number of observations, k the number of groups, and Ri the sum of ranks for group i with size ni. The test statistic H is: H = 12 N (N + 1) kX i=1 R2 i ni − 3(N + 1) Under the null hypothesis, H approximates a chi-squared distribution with k − 1 deg...

  20. [29]

    D.2 Dunn’s Post-Hoc Test

    Context: Used to test for global differences across LLM models in metrics such as Method Quantity, Signature Richness, Top-Method Consensus, and Core-Method Consensus where normality assumptions do not hold. D.2 Dunn’s Post-Hoc Test

  21. [30]

    It identifies which specific groups differ

    Definition: Dunn’s test is a non-parametric pairwise multiple comparison pro- cedure performed following a significant Kruskal–Wallis test. It identifies which specific groups differ. 42

  22. [31]

    Formula: For groups i and j, the test statistic Zij is: Zij = Ri/ni − Rj/njr N (N +1) 12 1 ni + 1 nj where Ri, ni are the sum of ranks and size of group i, respectively

  23. [32]

    D.3 Chi-Squared Test of Independence

    Context: Applied post hoc to identify pairwise differences between LLMs in met- rics where Kruskal–Wallis tests indicated significance, with Holm correction for multiple testing. D.3 Chi-Squared Test of Independence

  24. [33]

    Definition: The chi-squared test assesses whether two categorical variables are independent by comparing observed and expected frequencies

  25. [34]

    Formula: χ2 = X i X j (Oij − Eij)2 Eij where Oij and Eij are observed and expected frequencies for cell ( i, j)

  26. [35]

    D.4 Wilcoxon Signed-Rank Test

    Context: Used to examine associations between categorical variables such as annotation completeness and model identity. D.4 Wilcoxon Signed-Rank Test

  27. [36]

    Definition: The Wilcoxon signed-rank test is a non-parametric test for comparing two related samples to assess whether their population mean ranks differ

  28. [37]

    Let W be the sum of ranks for positive differences

    Formula: For paired differences di, order the absolute values and assign ranks. Let W be the sum of ranks for positive differences. The test statistic is: W = min(W +, W−) where W + and W − are sums of ranks for positive and negative differences, respectively

  29. [38]

    D.5 Cliff ’s Delta ( δ)

    Context: Employed to assess Structural Placement Consistency across core methods between pairs of LLM models. D.5 Cliff ’s Delta ( δ)

  30. [39]

    It ranges from −1 to +1, where 0 indicates no difference

    Definition: Cliff’s delta measures the effect size by quantifying the degree of overlap between two ordinal distributions. It ranges from −1 to +1, where 0 indicates no difference

  31. [40]

    Formula: Given two groups X and Y , δ = number of (xi > yj) − number of (xi < yj) nX nY where nX and nY are sample sizes of groups X and Y . 43

  32. [41]

    D.6 Cram´ er’s V

    Context: Reported alongside Kruskal–Wallis and Dunn’s tests to indicate the magnitude of differences between LLMs in metrics such as method counts and parameter richness. D.6 Cram´ er’s V

  33. [42]

    Definition: Cram´ er’s V measures the strength of association between two nominal variables, scaled between 0 (no association) and 1 (perfect association)

  34. [43]

    Formula: V = s χ2 N × (k − 1) where χ2 is the chi-squared statistic, N is total observations, and k is the smaller number of categories in the variables

  35. [44]

    Context: Used to quantify effect size in chi-squared tests, such as the association between model identity and annotation completeness. 44 E Placement consistency T able 8: Placement consistency of core methods across LLMs: method name, dominant class, number of LLMs agreeing ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.