Pith. sign in

REVIEW 2 major objections 1 minor 23 references

A Pilot Study on Detecting Software Design Patterns with Large Language Models: An Empirical Evaluation

T0 review · 2 major / 1 minor · reviewed 2026-05-10 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Large language models show promise for automatically detecting common software design patterns.

desk verdict This is a narrow pilot applying a few LLMs to five design patterns across three input formats, but the accuracy claims rest on unspecified prompts, unlabeled data, and no reported controls for stochasticity. read the letter →

arxiv 2604.17329 v1 submitted 2026-04-19 cs.SE

Oishik Chowdhury , Bastin Tony Roy Savarimuthu , Sherlock A. Licorish This is my paper · ORCID
classification cs.SE
keywords designpatternslargelanguagemodelssoftwareengineeringpatterndetectionempiricalevaluationLLMpromptingsingletonensemblemethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether large language models can recognize five standard design patterns in software. It compares four models plus two ensembles across three input formats: raw source code, PlantUML diagrams, and plain text descriptions. A sympathetic reader would care because automatic detection could help new developers grasp unfamiliar system architectures and let experienced developers spot quality issues faster. The pilot finds that NextCoder and Gemma 3 achieve comparatively higher accuracy while ensembles improve overall efficiency. This approach is presented as an alternative to earlier graph-based or traditional machine-learning detection methods.

What carries the argument

Comparative prompting of LLMs with source code, PlantUML diagrams, or text descriptions to classify the presence of singleton, adapter, bridge, composite, and decorator patterns.

What would settle it

Applying the top models and ensembles to a larger set of real-world codebases with independently verified instances of the five patterns and observing substantially lower accuracy or frequent misclassifications would disprove the reported promise.

Watch

Extended reading notes

Core claim

We evaluate the design pattern recognition capabilities of four Large Language Models and two ensemble approaches consisting three out of the four models. We also compare their performance when prompted with a) Source code, b) PlantUML representation of source code, and c) Text-based descriptions of the source code. We investigate the detection of five design patterns: singleton, adapter, bridge, composite and decorator. Our preliminary results indicate that LLMs show promise for automatically detecting design patterns, with NextCoder and Gemma 3 demonstrating comparatively higher accuracy than other models evaluated, and the ensemble approaches enhancing the overall efficiency of design p

Load-bearing premise

The selected prompts, limited code samples, and three input formats are sufficient for the LLMs to detect the patterns accurately without significant false positives or negatives.

Editorial extensions

If this is right

  • Automatic detection can bootstrap new developers' understanding of unfamiliar software system architectures.
  • Experienced developers can use it to quickly identify and rectify potential quality issues.
  • LLMs provide an alternative to prior graph-based and machine-learning based detection techniques.
  • NextCoder and Gemma 3 deliver higher accuracy than the other models tested.
  • Ensemble approaches improve the overall efficiency of design pattern detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Integrating the better-performing models into IDEs could deliver real-time pattern feedback while writing code.
  • Scaling the evaluation to more patterns and larger open-source projects would reveal where accuracy drops.
  • Hybrid systems that combine LLM detection with static analysis tools might lower error rates further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript presents a pilot study evaluating the ability of four large language models (LLMs) and two ensemble approaches to detect five software design patterns (singleton, adapter, bridge, composite, decorator) using three different input formats: source code, PlantUML representations, and text-based descriptions. The authors report preliminary positive results, highlighting higher accuracy for NextCoder and Gemma 3, and improved performance with ensembles.

Significance. If the empirical findings are robust, this work could demonstrate the potential of LLMs as a new tool for design pattern detection in software engineering, complementing traditional graph-based and ML methods. It identifies future directions that could lead to practical tools for developers.

major comments (2)
  1. The manuscript does not provide details on the dataset size, sampling method for the code samples of the five patterns, ground-truth labeling protocol, exact prompt texts used, evaluation metrics, or how LLM stochasticity (e.g., temperature, multiple runs) was accounted for. These omissions make it impossible to evaluate the reliability of the accuracy claims or to reproduce the study.
  2. The claim that NextCoder and Gemma 3 demonstrate comparatively higher accuracy, and that ensembles enhance efficiency, cannot be assessed without the specific quantitative results, statistical tests, or comparison baselines. The abstract reports 'preliminary results' but supplies no numbers or evidence.
minor comments (1)
  1. The abstract could benefit from a brief mention of the specific models evaluated besides NextCoder and Gemma 3.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our pilot study. We agree that the manuscript requires greater methodological transparency and explicit quantitative support for the claims to allow proper evaluation and reproduction. We address each major comment below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: The manuscript does not provide details on the dataset size, sampling method for the code samples of the five patterns, ground-truth labeling protocol, exact prompt texts used, evaluation metrics, or how LLM stochasticity (e.g., temperature, multiple runs) was accounted for. These omissions make it impossible to evaluate the reliability of the accuracy claims or to reproduce the study.

    Authors: We acknowledge that the current manuscript omits these essential details, which limits reproducibility and assessment of reliability. As this is explicitly a pilot study, the initial draft prioritized high-level exploration over exhaustive documentation. In the revised version, we will add a dedicated Experimental Setup section that specifies the dataset size and sampling method, the ground-truth labeling protocol, the exact prompt texts for all three input formats, the evaluation metrics, and our handling of LLM stochasticity (including temperature settings and multiple runs). We will also make the prompts and a subset of the code samples available via a public repository to support reproduction. revision: yes

  2. Referee: The claim that NextCoder and Gemma 3 demonstrate comparatively higher accuracy, and that ensembles enhance efficiency, cannot be assessed without the specific quantitative results, statistical tests, or comparison baselines. The abstract reports 'preliminary results' but supplies no numbers or evidence.

    Authors: We agree that the abstract and current presentation do not supply the numerical results or supporting analyses needed to evaluate the claims. In the revision, we will update the abstract to report key quantitative findings from our experiments and will add an early results summary (including tables or figures) with statistical tests comparing model performance. We will also expand the related-work discussion to address comparison baselines from prior non-LLM detection methods, while noting the challenges of direct comparison given our LLM-focused scope. These additions will make the evidence for the relative performance of NextCoder, Gemma 3, and the ensembles explicit. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical pilot study with no derivations or self-referential definitions

full rationale

The manuscript is an empirical evaluation of LLM performance on design pattern detection across five patterns and three input formats. It reports observed accuracies for specific models and ensembles without any equations, first-principles derivations, fitted parameters renamed as predictions, or load-bearing self-citations. The central claims rest on direct comparison of model outputs against ground-truth patterns in the tested samples; no step reduces to its own inputs by construction. Methodological limitations (unspecified corpus size, prompt text, labeling protocol) affect validity and generalizability but do not constitute circularity under the defined criteria.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

This is an empirical pilot study with no mathematical derivations, fitted parameters, or new theoretical entities; it rests on standard assumptions about design patterns being identifiable from code representations.

assumptions (1)
  • domain assumption Design patterns can be reliably identified from source code, PlantUML representations, or textual descriptions by LLMs.
    The evaluation assumes the three input formats convey enough structural information for accurate pattern detection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Pilot Study on Detecting Software Design Patterns with Large Language Models: An Empirical Evaluation." pith.science (2026). https://pith.science/paper/2604.17329

@misc{pith2026260417329,
  author       = {Pith},
  title        = {Pith review of: A Pilot Study on Detecting Software Design Patterns with Large Language Models: An Empirical Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.17329}},
  note         = {Machine review of arXiv:2604.17329}
}
read the original abstract

Design patterns provide reusable solutions to recurring software design problems. Automatically detecting these patterns in source code can help bootstrap new developers' understanding of unfamiliar software system architectures, and can help experienced developers to quickly identify and rectify potential quality issues. While many prior research works have explored graph based and machine-learning based detection techniques, this work evaluates the design pattern recognition capabilities of four Large Language Models and two ensemble approaches consisting three out of the four models. We also compare their performance when prompted with a) Source code, b) PlantUML representation of source code, and c) Text-based descriptions of the source code. We investigate the detection of five design patterns: singleton, adapter, bridge, composite and decorator. Our preliminary results indicate that LLMs show promise for automatically detecting design patterns, with NextCoder and Gemma 3 demonstrating comparatively higher accuracy than other models evaluated, and the ensemble approaches enhancing the overall efficiency of design pattern detection. We identify several directions for future work.

Figures

Figures reproduced from arXiv: 2604.17329 by the authors.

Figure 1
Figure 1. Prompt snippets showing a) Code for singleton pattern (top), b) PlantUML code corresponding to the singleton pattern (middle) and c) Description corresponding to the same instance of singleton pattern (bottom) 4.1 Comparison of LLM models (RQ1) [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the aggregated performance metrics of all models [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the different types of representations of the source code. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

  1. [1]

    D., and Mraidha, C

    Abdeljalil, Y., Radermacher, A., Del Fabro, M. D., and Mraidha, C. (2025). On the use of llms for design pattern detection in software models. In LLM4SE 2025-First Large Language Models for Software Engineering Workshop

    work page 2025

  2. [2]

    Natarajan, N. (2025). Robust learning of diverse code edits. arXiv preprint arXiv:2503.03656

    work page arXiv 2025

  3. [3]

    Alghamdi, F. M. and Qureshi, M. R. J. (2014). Im- pact of design patterns on software maintainabil- ity. International Journal of Intelligent Systems and Applications, 6(10):41

    work page 2014

  4. [4]

    Alhusain, S., Coupland, S., John, R., and Kavanagh, M. (2013). Towards machine learning based de- sign pattern recognition. In 2013 13th UK Work- shop on Computational Intelligence (UKCI), pages 244–251. IEEE

    work page 2013

  5. [5]

    H., Malveau, R

    Brown, W. H., Malveau, R. C., McCormick, H. W. S., and Mowbray, T. J. (1998). AntiPatterns: refac- toring software, architectures, and projects in crisis. John Wiley & Sons, Inc

    work page 1998

  6. [6]

    Cardoso, B. d. S. A. and Figueiredo, E. (2015). Co- occurrence of design patterns and bad smells in software systems: An exploratory study

    work page 2015

  7. [7]

    Chaturvedi, S., Chaturvedi, A., Tiwari, A., and Agar- wal, S. (2018). Design pattern detection using machine learning techniques. In 2018 7th In- ternational Conference on Reliability, Infocom Technologies and Optimization (Trends and Fu- ture Directions)(ICRITO), pages 1–6. IEEE

    work page 2018

  8. [8]

    Fowler, M. (2018). Refactoring: improving the design of existing code. Addison-Wesley Professional

    work page 2018

Show all 23 references
  1. [9]

    Gamma, E., Helm, R., Johnson, R., and Vlissides, J. (1995). Design patterns: elements of reusable object-oriented software. Pearson Deutschland GmbH. Guéhéneuc, Y.-G. (2007). P-mart: Pattern-like micro architecture repository. Proceedings of the 1st EuroPLoP Focus Group on pat...

    1995

  2. [10]

    Jin, R., Du, J., Huang, W., Liu, W., Luan, J., Wang, B., and Xiong, D. (2024). A comprehensive eval- uation of quantization strategies for large lan- guage models. In Findings of the Association for Computational Linguistics ACL 2024, pages 12186–12215

    2024

  3. [11]

    and Rasoolzadegan, A

    Kouli, M. and Rasoolzadegan, A. (2022). A feature- based method for detecting design patterns in source code. Symmetry, 14(7):1491

    2022

  4. [12]

    Mayvan, B. B. and Rasoolzadegan, A. (2017). De- sign pattern detection based on the graph the- ory. Knowledge-Based Systems, 120:211–225

    2017

  5. [13]

    Moreira, R., Fernandes, E., and Figueiredo, E. (2022). Based comparison of design pattern detection tools. In 29th International Confer- ence on Pattern Languages of Programs (PLoP), pages 1–16. Association for Computing Machin- ery

    2022

  6. [14]

    Nazar, N., Aleti, A., and Zheng, Y. (2022). Feature- based software design pattern detection. Journal of Systems and Software, 185:111179

    2022

  7. [15]

    Niculescu, V. (2021). Mixin based adaptation of de- sign patterns. In ENASE, pages 261–268

    2021

  8. [16]

    Pan, Z., Song, X., Wang, Y., Cao, R., Li, B., Li, Y., and Liu, H. (2025). Do code llms understand design patterns? In 2025 IEEE/ACM Interna- tional Workshop on Large Language Models for Code (LLM4Code), pages 209–212. IEEE

    2025

  9. [17]

    Ochodek, M., and Durisic, D. (2025). De- sign pattern recognition: a study of large lan- guage models. Empirical Software Engineering, 30(3):69

    2025

  10. [18]

    and Rausch, A

    Schindler, C. and Rausch, A. (2025). Llm- based design pattern detection. arXiv preprint arXiv:2502.18458

    2025

  11. [19]

    and Olsson, R

    Shi, N. and Olsson, R. A. (2005). Reverse engineer- ing of design patterns for high performance com- puting. In Proceedings of the 2005 Workshop on Patterns in High Performance Computing

    2005

  12. [20]

    Thaller, H., Linsbauer, L., and Egyed, A. (2019). Feature maps: A comprehensible software rep- resentation for design pattern detection. In 2019 IEEE 26th international conference on software analysis, evolution and reengineering (SANER), pages 207–217. IEEE

    2019

  13. [21]

    Tsantalis, N., Chatzigeorgiou, A., Stephanides, G., and Halkidis, S. T. (2006). Design pattern detec- tion using similarity scoring. IEEE transactions on software engineering, 32(11):896–909

    2006

  14. [22]

    Wang, L., Song, T., Song, H.-N., and Zhang, S. (2022). Research on design pattern detection method based on uml model with extended im- age information and deep learning. Applied Sci- ences, 12(17):8718

    2022

  15. [23]

    A., and Stella, F

    Zanoni, M., Fontana, F. A., and Stella, F. (2015). On applying machine learning techniques for design pattern detection. Journal of Systems and Soft- ware, 103:102–117

    2015

Pith tools

Reviewed May 10, 2026 · model on record in the stance chip above.