{"id":"01b5419a-7a87-4cc2-bdfe-b288af890305","arxiv_id":"2501.04835","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An empirical study of 11 code LLMs shows they frequently misclassify design patterns and generate code that diverges from pattern-consistent references.","lead":"This paper tests whether eleven popular code-generating AI models can recognize and follow twelve classic software design patterns in Java and Python. Across three tasks, the best models identified the correct pattern only about 39 percent of the time, and generated code often differed sharply from a reference implementation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generation metrics measure text overlap with one reference implementation, not design-pattern conformance; the downstream-reliability claim needs direct validation.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: CS/ES are measured against a single reference and may not reflect whether the generated code correctly applies a design pattern. My stress-test concurs and sharpens the point by tying it directly to the abstract's 'significantly affect reliability' claim, which depends on the generation experiments being interpreted as evidence of pattern violation. The classification results, while useful, only show that LLMs are poor at recognizing patterns in these small, manually labeled samples; they do not by themselves quantify downstream reliability impact. The proposed human-conformance study is the decisive check. If it fails, the paper's headline conclusion should be reduced to 'model outputs differ textually from reference implementations,' and the practical-reliability wording removed. If it passes, the conditional acceptance can move toward acceptance once artifacts are also released. Because the reader already recommended CONDITIONAL, my independent review does not change that verdict; it reinforces the condition. No ad hominem or theatrical language is intended; the issue is an evidentiary gap in the metrics, not a question of author intent.","tokens_in":8722,"tokens_out":4849,"duration_ms":52364,"concrete_test":"Sample, say, 60 function-generation outputs per language (5 outputs per pattern x 12 patterns) that fall in the bottom quartile of CS/ES, plus 60 high-scoring controls. Have two independent software engineers with design-pattern expertise classify each output as conforming, non-conforming, or ambiguous relative to the requested pattern, adjudicating disagreements. Report conformance distributions against CS/ES quartiles. If a substantial fraction (e.g., >50%) of low-similarity outputs are judged conforming, the similarity metrics are not a valid proxy for pattern adherence and the reliability claim must be withdrawn or reworded. If nearly all low-similarity outputs are non-conforming, the proxy survives and the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM biases 'significantly affect the reliability of downstream tasks' rests primarily on the generation experiments, where Section III-A-4 defines Code Similarity (CS) via difflib's LCS overlap and Edit Similarity (ES) via edit distance, both computed against a single hand-picked reference implementation. These are string-similarity measures, not pattern-conformance checks. A valid implementation of the same pattern may use different identifiers, helper-method decomposition, statement ordering, or language idioms and still score near zero. The paper interprets low CS/ES as evidence that generated code 'conflicts with the required design patterns' and imposes post-processing burden, but that inference is valid only if similarity-to-reference is a proxy for pattern conformance. No validation of this proxy is provided: there is no human judgment of conformance, no static design-pattern analysis, and no test with multiple acceptable references. Without such validation, the generation results support only 'outputs differ from the reference text,' not 'outputs violate the design pattern.' The classification result (38.81% best accuracy) is direct evidence of recognition difficulty, but it does not by itself establish a bias affecting downstream code reliability; the generation experiments carry that load, and their metric is unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of code LLMs' understanding of design patterns. It uses a manually collected dataset of 48 GitHub repositories (24 Python, 24 Java) covering 12 design patterns and evaluates 11 models on three tasks: design-pattern classification, line completion, and function generation. In classification, the best models (GPT-4o and Llama-31-70B) reach 38.81% overall accuracy. In the generation tasks, the paper reports Code Similarity and Edit Similarity between model outputs and a single reference implementation, with and without providing the design-pattern label. The authors conclude that LLMs exhibit design-pattern biases that significantly affect the reliability of downstream tasks and impose a post-processing burden on developers.","tokens_in":8785,"tokens_out":6450,"duration_ms":61160,"significance":"The paper addresses a relevant and under-studied question: whether code LLMs can recognize and follow design patterns. The classification experiment provides direct evidence that current models are poor at recognizing design patterns, which is a useful data point for the community. The evaluation framework covering recognition, completion, and generation is a reasonable structure, and the inclusion of both Java and Python with 12 patterns is a breadth that many earlier studies lack. If the generation-metric validity concern were addressed, the downstream-reliability claim would be significant for practitioners and model developers. However, the current evidence is conditional on accepting that text-similarity to a single reference measures design-pattern conformance, which is not established; without that, the paper's central conclusion is unsupported.","major_comments":[{"comment":"Code Similarity (CS) and Edit Similarity (ES) are computed against one hand-picked reference implementation per task. These metrics measure token/string overlap and edit distance, not whether the generated code correctly implements the design pattern. A valid implementation that uses different identifiers, method decomposition, or statement ordering can score near zero, while an incorrect implementation that happens to resemble the reference text can score high. The paper interprets low CS/ES as evidence that generated code 'conflicts with the required design patterns' and imposes a post-processing burden (Abstract, Section III-C Insights). This inference is load-bearing for the central claim about downstream reliability, but no validation is provided: no human conformance judgment, no static design-pattern analysis, and no multiple acceptable references. Without such validation, the generation results support only the weaker conclusion that model outputs differ from the reference text.","section":"Section III-A-4 and Section III-C"},{"comment":"The classification experiment has extremely small per-cell sample sizes (e.g., Java Easy n=14, Java Medium n=7, Python Easy n=9, Python Hard n=14, total 67 files for the 'All' row). No significance tests, confidence intervals, or effect sizes are reported. As a result, claims that GPT-4o and Llama-31-70B outperform other models, and that performance declines from Java to Python and from Easy to Hard, are unsupported: the difference between 71.43% and 64.29% on Java Easy is a single file. The paper should report paired or per-pattern statistical tests, or at minimum confidence intervals, and discuss the statistical power available with these sample sizes.","section":"Section III-B, Table I"},{"comment":"The comparisons between providing and withholding design-pattern knowledge show inconsistent effects across models and patterns, yet no statistical analysis is provided to determine whether any of the differences are meaningful. For example, in Table III, GPT-4o's Singleton CS with pattern is 36.15 vs 32.77 without, a 3.38-point difference on an unknown number of test items, whereas for Observer the pattern-provided condition is lower. The text acknowledges such exceptions but still concludes that providing design-pattern knowledge improves performance. A proper statistical treatment (e.g., paired tests across items, mixed-effects models with pattern and condition as factors) is needed to support this claim.","section":"Section III-C, Tables II and III"},{"comment":"The experimental setup is under-specified to the point of hampering reproducibility. No prompts are shown, no decoding parameters (temperature, top-p, max tokens) are reported, no number of runs or random seeds are given, and the 'None' entries in Table I are not explained (did the model fail to produce a valid output, or was that condition not run?). For the generation tasks, the process of randomly removing 'three separate lines of effective code' and using GPT-4 to write function descriptions is not described with enough detail to allow replication. These omissions should be corrected.","section":"Section III-A-2/3 and Table I"}],"minor_comments":[{"comment":"The sentence 'difflib's SequenceMatcher, which employs the Ratcliff/Obershelp algorithm to finds the Longest Matching Subsequence' contains a grammatical error and is technically imprecise: Ratcliff/Obershelp is not equivalent to exact longest common subsequence, so the description should be corrected.","section":"Section III-A-4"},{"comment":"The 'None' entries in Table I should be defined in the caption. If they indicate that a model failed to produce a classification, that is itself an important result and should be discussed; if they indicate that the condition was not run, that should be stated explicitly.","section":"Table I"},{"comment":"The caption notation is confusing. In Table II the slash separates Line Completion from Function Generation, while in Table III it separates Code Similarity from Edit Similarity. Please make the column labels unambiguous (e.g., 'CS (LC/FG)' and 'ES (LC/FG)') or add a note in the table header.","section":"Tables II and III"},{"comment":"The text states that existing studies 'often use benchmarks such as CodeXGLUE [10]-[12]', but references [10] (Codex), [11] (CodeT), and [12] (code smells) are not CodeXGLUE papers. Either replace the references with the actual CodeXGLUE citation or revise the claim.","section":"Section II-B"},{"comment":"Reference [4] is cited to support claims about the LLaMA 3.1 series, but [4] is a paper on Llama Scope (sparse autoencoders), not the LLaMA 3.1 model. The citation should be replaced with the appropriate technical report or model card.","section":"Section II-A"},{"comment":"The paper would benefit from an explicit threats-to-validity section. The small dataset, single-reference metrics, and lack of statistical testing are important limitations that should be acknowledged in the main text, not merely in the plan for future work.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely question, and the classification experiment is a reasonable direct measurement, but the generation-based downstream-reliability claim relies on an unvalidated metric. The authors should either add a validation of CS/ES as pattern-conformance proxies (e.g., human ratings or static pattern detection), or substantially soften the abstract and conclusion claims. Given the small sample sizes, the paper also needs appropriate statistical caution. The topic fits the journal's scope; the issues are fixable within a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper measures something real and previously unmeasured—how 11 code LLMs classify, complete, and generate code for 12 design patterns in Java and Python—and the classification headline is worth remembering: the best models, GPT-4o and Llama-31-70B, hit only 38.81% accuracy overall. That is a concrete, useful data point for anyone building or benchmarking code assistants.\n\nWhat earns credit: the scope is a genuine extension of existing code-LLM benchmarks, which mostly ignore design-pattern conformance. The misclassification heatmaps give a plausible story about over-prediction of Singleton and Factory and trouble with Facade. The three-task design (classification, line completion, function generation with/without pattern hints) is sensible, and the classification experiment is direct evidence for recognition difficulty. No fitted parameters, no circular derivation.\n\nThe soft spots are real, and the main one is exactly what your stress-test note flags. Section III-A-4 defines Code Similarity via difflib's LCS overlap and Edit Similarity via edit distance, both computed against a single reference implementation. Those are text-overlap measures, not pattern-conformance checks. A valid alternative implementation using different names, method decomposition, or idiom can score near zero. So Tables II and III support only \"outputs differ from the reference text,\" not \"outputs violate the design pattern\" or \"developers must post-process.\" The abstract's claim that biases \"significantly affect the reliability of downstream tasks\" overreaches the evidence.\n\nOther issues, in proportion: the classification sample is thin—per-cell numbers run as low as 14 files for Java Easy, and several cells are below 30—and no significance tests or chance baselines are reported, so the Java/Python and easy/hard trends are suggestive, not established. The dataset, prompts, and evaluation code are not released, and the complexity tiers and manual ground-truth labels are not defined operationally. Self-citations appear as background only, so I don't see a citation-pattern problem.\n\nWho gets value from this: practitioners who want a quick sense of what today's models do on design-pattern recognition, and benchmark designers who can use the classification result as a baseline—and as a cautionary example of why string-overlap metrics need validation. I would send it to peer review, but with the expectation of major revision focused on releasing artifacts, adding significance tests, and either validating the similarity metrics against human or static pattern-conformance judgments or softening the downstream-reliability claims.","headline":"A useful first cut at how code LLMs handle design patterns, with a real classification finding (best accuracy 38.81%) and a generation evaluation whose string-similarity metrics don't support the paper's downstream-reliability claim.","tokens_in":9433,"tokens_out":2688,"would_cite":false,"duration_ms":26776,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper attempts to establish that code LLMs systematically misrecognize and misapply object-oriented design patterns, degrading the reliability of generated code.","keywords":["code large language models","design patterns","empirical evaluation","code classification","code completion","function generation","Java","Python"],"falsifier":"Take a sample of low-similarity outputs from the function-generation task and have expert programmers judge whether each implements the intended pattern's structural and behavioral invariants; if most low-similarity outputs are judged pattern-compliant, the paper's conclusion that models conflict with design patterns would be undermined.","tokens_in":8399,"feed_emoji":"🧩","tokens_out":8727,"duration_ms":76589,"temperature":0.7,"pith_summary":"Code LLMs, despite their coding fluency, carry systematic biases when it comes to object-oriented design patterns, and this paper sets out to show that those biases degrade downstream code reliability. The authors evaluate eleven models on twelve design patterns in Java and Python using three tasks: classifying a code file's pattern, completing missing lines, and generating functions with and without a stated pattern. The two best classifiers reach only 38.81% accuracy overall, with a consistent drop from Java to Python and from easy to hard examples. Supplying the pattern name in the prompt helps some models and hurts others, and similarity to a reference implementation does not reliably predict editing effort. If the findings hold, developers should expect generated code to deviate from project architecture and plan for manual repair.","feed_headline":"Best code LLMs score just 38.81% at spotting design patterns","feed_subtitle":"Across 12 patterns and two languages, generated code often misses project design norms and needs manual edits.","key_machinery":"The evaluation harness is the central mechanism. It consists of 48 manually selected code repositories, 24 in each language, spanning 12 design patterns, with files graded into easy, moderate, and difficult complexity. Three probes are run: design-pattern classification, line completion with missing lines removed, and function generation from natural-language descriptions with or without the pattern name supplied. Generation quality is scored by code similarity and edit similarity: the first measures longest-matching-subsequence overlap between generated and reference code, and the second measures the normalized edit distance needed to turn generated code into the reference. These two proxies convert whether the model follows the design pattern into a number, and the whole argument about reliability rests on them.","core_discovery":"The paper's central claim is that code LLMs exhibit measurable biases in handling design patterns, and that these biases significantly affect the reliability of downstream tasks. On the classification task, the best models top out at 38.81% accuracy across twelve design patterns and two languages; all models degrade from Java to Python and as code complexity rises. Misclassification heatmaps show that Singleton and Factory are systematically over-predicted because their structural cues, such as static methods, single-instance management, and object creation, overlap with patterns such as Facade, Proxy, Command, Abstract Factory, Builder, and Strategy, while Facade is the hardest to recognize because it lacks explicit structural markers. In line completion and function generation, providing the design pattern sometimes improves output similarity and sometimes worsens it, and code similarity and edit similarity diverge often enough that high textual overlap does not guarantee low repair effort. The paper concludes that better training data, clearer pattern distinctions, and improved pattern integration are needed before these models can be trusted in design-pattern-sensitive software workflows.","pith_inferences":["These results measure textual similarity to one reference solution rather than structural conformance to the pattern, so the generation experiments may understate a model's true ability when a valid implementation is written differently.","The observed over-prediction of Singleton and Factory suggests a testable intervention: augmenting training data with negative examples of near-miss patterns should raise classification accuracy on Facade and Strategy categories.","A human or automated behavioral check of generated functions, such as verifying that a Singleton yields one instance or a Factory returns the correct product type, would separate textually different from pattern-violating and sharpen the reliability claim."],"forward_implications":["Model-generated code in pattern-heavy projects will frequently deviate from the intended architecture, so developers should budget for post-editing even when the code is syntactically valid.","Reporting edit similarity alongside code similarity gives a truer picture of developer effort, since high textual overlap does not always mean little repair work.","Prompting a model with the design pattern name is not a reliable universal remedy: it helps on some patterns and models and hurts on others.","Improvements should target the confusion clusters, such as Singleton versus Facade, Proxy, or Command, and the hardest patterns, including Builder, Bridge, and Facade, where current accuracy is lowest."],"supporting_citations":[],"fun_headline_variants":["Code LLMs spot design patterns only 38.81% of the time","Bias in code LLMs skews design pattern recognition, best 38.81%","Design pattern bias in code LLMs hurts downstream reliability","Code LLMs: 38.81% best accuracy on design pattern recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that how close generated code is to one reference implementation tells whether the model correctly applied the design pattern.","fun_headline_variants_meta":{"raw":{"variants":["Code LLMs spot design patterns only 38.81% of the time","Bias in code LLMs skews design pattern recognition, best 38.81%","Design pattern bias in code LLMs hurts downstream reliability","Code LLMs: 38.81% best accuracy on design pattern recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3416,"prompt_tokens":856,"completion_tokens":2560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2479}},"tokens_in":472,"tokens_out":2560,"duration_ms":17791,"temperature":1.0,"reasoning_tokens":2479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:24:00.962780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of low-similarity outputs from the function-generation task and have expert programmers judge whether each implements the intended pattern's structural and behavioral invariants; if most low-similarity outputs are judged pattern-compliant, the paper's conclusion that models conflict with design patterns would be undermined.","supporting_citations":[],"review_version":1}