{"id":"8dbf9789-ea4c-4686-98ec-642b4513b509","arxiv_id":"2509.03093","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMs vary sharply in detecting SOLID violations, GPT-4o Mini leads, and no single prompt strategy wins, with accuracy falling as code complexity rises.","lead":"This paper measures how well four large language models detect violations of the five SOLID design principles in Python, Java, C#, and Kotlin snippets, using a new 240-example dataset and four prompt strategies. It finds that model choice and prompt strategy both matter a lot, with no single best configuration, which matters for anyone relying on AI code assistants to keep software maintainable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manual relabeling of 37% of outputs without blinding or inter-annotator agreement, plus gpt-4o-generated dataset, leaves the 'decisive' model ranking vulnerable to label bias.","rationale":"The paper's central contribution is a ranking and prompt comparisons, all derived from a single manually curated dataset and a labeling pipeline in which 37% of outputs were hand-classified. If the manual labels are biased, every downstream result—model hierarchy, prompt-by-principle interactions, language effects—is suspect. The reader's verdict (CONDITIONAL) already captures this; I do not see a separate more load-bearing concern. The dataset provenance (gpt-4o generation) reinforces the same concern, though the human validation step reduces but does not eliminate it. I would keep the verdict CONDITIONAL, not REJECT, because the presence of a replication package makes the proposed check feasible and because the observed F1 gaps (e.g., SRP 99.7 vs 55.6) may survive blinded re-labeling. The paper deserves the benefit of the doubt, but the 'decisive' language is not supported without inter-annotator agreement or a provenance control.","tokens_in":7811,"tokens_out":6424,"duration_ms":71256,"concrete_test":"Using the replication package's raw outputs, draw a stratified random sample of 200–300 responses from the 1,431 manually labeled cases, covering all four models and four prompts. Have two independent annotators, blinded to model identity, prompt strategy, and original labels, re-label these responses using the exact criteria in the repository. Report Cohen's kappa between the new annotators and between each and the original labels. Then recompute per-model F1 using the new labels (and optionally a consensus label). If GPT-4o Mini remains decisively ahead and kappa > 0.8, the ranking is supported; if agreement is poor or the re-labeled F1 gap narrows to within the range of other models, the 'decisive' claim is an artifact of the labeling process.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C states that because LLMs 'frequently fail to adhere to the requested output structure,' the authors manually reviewed and labeled 1,431 of 3,840 responses (37%). No blinding, randomization, or inter-annotator agreement is reported. The F1/accuracy numbers that support the abstract's 'decisively' superior GPT-4o Mini ranking are computed from these labels, so any systematic bias in that manual labeling directly creates an artifact. The risk is compounded by dataset construction (Section III-A): snippets were generated by prompting OpenAI's gpt-4o, so GPT-4o Mini—same provider family—may be at an advantage if the generated code carries family-specific stylistic cues. The dataset labels are also author-validated rather than independently adjudicated. The paper asserts the manual review 'ensures quality' but provides no evidence of reliability. This is not an accusation of deliberate bias; it is a missing control. If labels favor outputs that look plausible or that match expected class distributions, the reported hierarchy—especially the large SRP gap (GPT-4o Mini 99.7 vs. CodeLlama 55.6)—could be inflated. Without a blinded re-labeling check, the central claim is not yet empirically secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical benchmark for detecting SOLID design-principle violations in code using four LLMs (CodeLlama-70B, DeepSeekCoder-33B, Qwen2.5-Coder-32B, GPT-4o Mini), four programming languages (Java, Python, Kotlin, C#), and four prompt strategies (DEFAULT, EXAMPLE, SMELL, ENSEMBLE). The authors construct a new dataset of 240 manually validated code snippets spanning all five SOLID principles at three difficulty levels, report accuracy/F1 per model, prompt, language, and complexity, and conclude that GPT-4o Mini decisively outperforms the others, that no single prompt strategy is universally best, and that detection accuracy degrades sharply with code complexity. A replication package is provided.","tokens_in":8150,"tokens_out":4717,"duration_ms":58273,"significance":"The study addresses a relevant and under-explored question: whether current LLMs can reliably detect semantic design flaws such as SOLID violations. Its strengths include a multi-language, multi-model, multi-prompt design, a publicly available replication package with raw outputs, and an explicit acknowledgment of limitations. If the reported rankings and prompt-by-principle interactions were statistically and methodologically secure, the benchmark would be a useful reference point for developers and researchers. However, the central quantitative claims—especially the 'decisive' superiority of GPT-4o Mini—rest on a labeling process that lacks blinding and inter-annotator reliability evidence, and on point estimates without confidence intervals or significance tests. These issues are fixable but currently prevent the strongest conclusions from being empirically secure.","major_comments":[{"comment":"The manual relabeling of 1,431 of 3,840 responses (37%) is load-bearing for every accuracy/F1 figure in Section IV and for the abstract's claim that GPT-4o Mini 'decisively outperforms' others. No blinding, no independent second labeler, no inter-annotator agreement (e.g., Cohen's kappa), and no sensitivity analysis are reported. Because the labels are used to compute Figure 3a and all subsequent conclusions, systematic bias in this step—even unconscious—could directly produce the observed model hierarchy. Please report a blind or at least independent re-labeling of a random sample, quantify agreement, and show that the main ordering is unchanged when only regex-derived labels are used.","section":"III-C"},{"comment":"All experiments are single temperature-0 runs, and the paper reports F1/accuracy point estimates without any confidence intervals, bootstrap intervals, or significance tests. For example, Section IV-A states that Qwen is a 'clear second' with DIP F1=10.8 and that CodeLlama is 'weak' with ISP F1=13.6, but the 240-sample size makes such differences potentially within sampling noise. Similarly, Section IV-B's 'dramatic impact' and 'no single strategy is universally best' are based on differences like ENSEMBLE's OCP F1=75.7 versus DEFAULT's OCP value, with no statistical support. Please add uncertainty quantification (e.g., bootstrap over snippets) and tests appropriate for paired classifications (e.g., McNemar) for at least the headline comparisons.","section":"III-E and IV-A"},{"comment":"The dataset is generated by prompting OpenAI's gpt-4o, and the top performer is GPT-4o Mini from the same organization. The paper does not test for contamination or family-specific stylistic cues that could inflate OpenAI-family performance. This is a design confound rather than misconduct, but it directly affects the 'decisive' ranking claim. Please add a contamination check (e.g., see whether GPT-4o Mini's predictions are unusually confident on gpt-4o-generated examples), generate a small external validation set using another source, or at least show examples and analyze whether any surface patterns correlate with the generating model.","section":"III-A"},{"comment":"The task is a six-way classification (five principles plus 'No Violation'), but Eq. (1) defines binary F1. It is unclear whether F1 is macro-averaged over the six classes, micro-averaged, or computed one-vs-rest for each principle; the text reports per-principle F1 but never states the averaging procedure. In addition, the dataset description is ambiguous: the contributions say the dataset 'provides both violating and refactored code versions,' but the total of 240 samples and per-principle class support are not specified. Without per-class confusion matrices and class counts, the numeric F1 values in Figures 3 and 4 cannot be interpreted. Please report the exact class distribution and the averaging method, and include confusion matrices for at least the headline comparisons.","section":"III-D and Eq. (1)"}],"minor_comments":[{"comment":"The description of dataset validation is grammatically tangled ('One author refined outputs... A second author independently verified...'). Please clarify how many authors were involved and whether they were blind to the intended violation label.","section":"III-A"},{"comment":"The text says difficulty labels 'correlate strongly' with character count and cyclomatic complexity, but no correlation coefficients are reported. Please provide the actual values or a brief quantitative summary.","section":"II, Figure 2"},{"comment":"The EXAMPLE prompt embeds one illustrative line per principle. Please confirm whether these examples were taken from the test scenarios; if any overlap exists, few-shot memorization could confound the prompt comparison.","section":"III-B"},{"comment":"Figure 4b is described by reporting SRP scores for C# and Java, but the figure uses average accuracy across all principles. Label the axes and clarify what is plotted to avoid confusion.","section":"IV-C"},{"comment":"The abstract says 'yet even struggles' without an explicit subject; also 'decisively outperforming' is stronger than the 'emerging results' language in the body. Softening the abstract to match the statistical evidence would improve accuracy.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a straightforward empirical benchmark with a useful replication package. The main issue is not novelty or scope but the gap between the strength of the claims ('decisively', 'dramatic impact') and the methodological evidence. The three load-bearing problems—manual relabeling without reliability evidence, absence of uncertainty quantification, and possible same-family dataset contamination—are all addressable within the scope of the paper. I would not reject, but the current version is not yet empirically secure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a genuinely useful first empirical benchmark of LLMs for SOLID violation detection, but the headline result—GPT-4o Mini decisively outperforming the others—rests on shakier ground than the abstract lets on. The dataset is a real contribution: 240 snippets across five principles, four languages, three difficulty levels, with both violating and clean versions, plus a replication package with prompts, outputs, and scripts. That alone makes the paper worth a look for anyone building AI-assisted code review. The systematic comparison of four models and four prompt strategies is also well executed: no single prompt wins everywhere, ENSEMBLE helps OCP, EXAMPLE helps DIP, and complexity sharply degrades accuracy. These findings are plausible and useful.\n\nThe soft spots are exactly where the reader's report lands. The biggest one is the manual relabeling: 1,431 of 3,840 responses (37%) were reviewed and relabeled by the authors, with no blinding, no randomization, no inter-annotator agreement. The paper calls this 'rigorous' but gives no evidence. If the authors unconsciously favored outputs that matched their expectations—say, GPT-4o Mini's responses because they look more coherent—the reported F1 gaps (SRP 99.7 vs 55.6) could shrink or grow. Second, the dataset was generated by prompting gpt-4o, so GPT-4o Mini, from the same family, may have an unfair advantage due to shared style or tokenization. This is a confound, not a fatal flaw, but it undercuts the 'decisive' hierarchy. Third, all results come from a single temperature-0 run, with no confidence intervals or significance tests. Some of the between-model or between-prompt differences might be noise.\n\nNone of this sinks the paper, but it should temper the claims. The authors are honest about limitations in the conclusion, and the replication package is a plus. I'd send this to peer review—it's a solid dataset and a clear research question, and the weaknesses are fixable with blinding, inter-annotator agreement, contamination checks, and some basic statistics. For readers, I'd treat the model ranking as preliminary and focus on the dataset and the qualitative findings about prompt-language interactions.","headline":"Useful first benchmark of LLMs on SOLID detection, but the 'decisive' model ranking is under-supported by unblinded manual labeling and a same-family dataset confound.","tokens_in":8600,"tokens_out":2266,"would_cite":true,"duration_ms":24831,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 240 validated code snippets and four prompt strategies, LLM detection of SOLID violations is model-, prompt-, principle-, and language-dependent, with GPT-4o Mini leading on most principles but no strategy universally best.","keywords":["SOLID principles","design principle violations","large language models","prompt engineering","benchmark dataset","code quality analysis","multi-language code analysis","empirical software engineering"],"falsifier":"Replace the dataset's generated examples with equivalent snippets drawn from real open-source projects, keep the same prompts and models, label them blinded, and recompute the model-prompt F1 table; if GPT-4o Mini's lead and the EXAMPLE-over-SMELL ordering do not reproduce, the reported hierarchy is an artifact of the benchmark's construction.","tokens_in":7792,"feed_emoji":"🤖","tokens_out":5248,"duration_ms":55311,"temperature":0.7,"pith_summary":"This paper tries to establish the first systematic, reproducible picture of how well large language models detect violations of the five SOLID design principles. It builds a 240-sample benchmark spanning Java, Python, C#, and Kotlin, and runs four LLMs under four prompt strategies. The headline result is a clear model hierarchy—GPT-4o Mini on top, then Qwen2.5-Coder, with CodeLlama and DeepSeek far behind—and an equally clear finding that prompt design changes outcomes as much as model choice does. The point matters because if LLMs are to serve as design reviewers, their accuracy on principled code analysis needs to be known per principle, per language, and per complexity level rather than averaged into a single score.","feed_headline":"GPT-4o Mini tops SOLID checks; no prompt is universal","feed_subtitle":"Four LLMs, 240 validated snippets, and four prompts show accuracy depends on model, principle, language, and complexity.","key_machinery":"The load-bearing artifact is a purpose-built dataset of 240 manually validated code snippets, 20 scenarios times three difficulty levels times four languages, each with violating and refactored versions. Around it the paper wraps four prompt strategies—DEFAULT zero-shot, EXAMPLE with one-line hints, SMELL two-step smell-to-principle reasoning, ENSEMBLE five-principle scoring with justification—and a JSON-output classification pipeline that required manual relabeling of 37% of the 3,840 responses. The design lets accuracy and F1 be decomposed by model, prompt, principle, language, and complexity.","core_discovery":"On its own terms, the paper's central discovery is that SOLID-violation detection is not one capability but a bundle of model-prompt-principle interactions. GPT-4o Mini achieves F1 99.7 on SRP but falls to low scores on DIP; the ENSEMBLE prompt raises OCP detection to 75.7 while the two-step SMELL prompt collapses on OCP and LSP; C# and Java are detected far better than Python; and every model's accuracy drops sharply as code complexity rises. The authors offer this as evidence that effective AI-driven design analysis needs to pair the right model with the right prompt for the specific design context, not to look for a single best system.","pith_inferences":["A fair cross-family test would regenerate the benchmark from independently written code, since the examples come from the same model family as the reported winner; if the lead shrinks, part of the ranking reflects dataset provenance rather than design understanding.","The strong drop on 'hard' samples implies that on real industrial code, where incidental complexity is higher, even the best configuration will miss many violations; deployment should pair LLM flags with static-analysis or human review rather than treat the model as a verdict.","A direct extension is to score the refactored versions the dataset already contains: if detection accuracy varies by prompt and language, repair quality probably does too, giving a concrete test of the paper's refactoring follow-up.","Reported per-principle F1 scores should be read with the 37% manual relabeling in mind; a blinded relabeling study could show how much of the ranking depends on how output deviations were resolved."],"forward_implications":["No single model-prompt combination is best across all five SOLID principles; tooling should route different detection tasks to different configurations.","Direct, context-rich prompts (EXAMPLE) help on nuanced principles like DIP and LSP, while an indirect two-step SMELL prompt consistently hurts, so prompt design should avoid abstract reasoning chains for this task.","Detection is reliably better in statically typed languages (C# and Java) than in Python, suggesting that type structure provides signals LLMs can read.","Complexity is the largest cross-cutting factor: accuracy on OCP falls from 64.8 on easy samples to 18.0 on hard ones, which sets a realistic ceiling for detection in real code.","The dataset and replication package establish a baseline for future benchmarks of design-aware code generation and refactoring, not just detection."],"supporting_citations":[{"why":"supplies the CodeLlama-70B model that is benchmarked as one of the four detectors.","marker":"[8]"},{"why":"supplies the DeepSeekCoder-33B model used as a benchmark participant.","marker":"[9]"},{"why":"supplies the Qwen2.5-Coder-32B model benchmarked here.","marker":"[10]"},{"why":"documents GPT-4o Mini, the top-performing model in the study.","marker":"[11]"},{"why":"provides the canonical design-pattern and principle scenarios that define the 20 violation types.","marker":"[25]"},{"why":"grounds the DEFAULT zero-shot prompt strategy.","marker":"[26]"},{"why":"grounds the EXAMPLE few-shot strategy.","marker":"[27]"},{"why":"grounds the SMELL two-step chain-of-thought strategy.","marker":"[28]"}],"fun_headline_variants":["SOLID detection: no single prompt works best","GPT-4o Mini leads SOLID, but prompts shift results","Model, prompt, language: the SOLID detection triple","For SOLID checks, match prompt to principle"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Everything rests on the correctness and neutrality of the reference labels: the 240 snippets were produced from a single model family's output and refined by two authors, and a third of all model responses were hand-labeled, so if those labels or relabels favor one model or prompt, the comparative rankings shift.","fun_headline_variants_meta":{"raw":{"variants":["SOLID detection: no single prompt works best","GPT-4o Mini leads SOLID, but prompts shift results","Model, prompt, language: the SOLID detection triple","For SOLID checks, match prompt to principle"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1758,"prompt_tokens":861,"completion_tokens":897,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":832}},"tokens_in":605,"tokens_out":897,"duration_ms":7986,"temperature":1.0,"reasoning_tokens":832,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:06:06.025537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the dataset's generated examples with equivalent snippets drawn from real open-source projects, keep the same prompts and models, label them blinded, and recompute the model-prompt F1 table; if GPT-4o Mini's lead and the EXAMPLE-over-SMELL ordering do not reproduce, the reported hierarchy is an artifact of the benchmark's construction.","supporting_citations":[{"cited_title":"Gpt-4o: Openai’s new omni model,","cited_arxiv_id":null,"evidence_quote":"documents GPT-4o Mini, the top-performing model in the study."},{"cited_title":"Design principles and design patterns,","cited_arxiv_id":null,"evidence_quote":"provides the canonical design-pattern and principle scenarios that define the 20 violation types."}],"review_version":1}