{"id":"1b8b10d4-2aa8-4628-8dc0-5dba2519cba5","arxiv_id":"2507.12488","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An analysis of 83 student OOP projects shows frequent GUI-to-Model architecture violations, unclosed resources, duplicated string literals, and naming issues as the most common quality defects.","lead":"This paper measured software quality in 83 student object-oriented programming projects using SonarQube and ArchUnit. It found recurring problems: students connected parts of their programs that should stay separate and left files or database connections open. Educators can use this concrete list to decide which quality topics to teach first.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ1's headline percentages rest on an unvalidated, unspecified ArchUnit rule suite; without precision/recall against manual annotation, the architectural violation frequencies cannot be distinguished from rule artifacts.","rationale":"The reader's weakest assumption identifies precisely the same load-bearing dependency: the RQ1 frequencies are generated by a custom, unvalidated ArchUnit rule suite. My reading of the full text reinforces this: Section 3.3 describes the rule suite as 'created by two of the authors' and Section 6 asserts validation without giving any procedure, threshold, sample size, or inter-rater agreement. The paper's other potential weaknesses are real but less load-bearing. The RQ2 frequencies come from SonarQube's standard, widely used default Java rule set, which gives at least a reproducible operationalization even if one might debate whether every rule reflects true 'quality.' The post-hoc filter to issues appearing in more than 50% of projects is disclosed, and the complete tables are promised in the replication package, so the selective emphasis is transparent. The causal interpretation that issues stem from student inexperience or course gaps lacks a professional baseline, but the descriptive frequencies do not depend on that interpretation. The sample spans two teachers and two academic years, and the reporting by group and year actually allows readers to see variation. Thus, the dominant reason the paper cannot be accepted unconditionally is the missing validation of the ArchUnit suite. A concrete precision/recall test against manual annotation would settle whether the architectural findings are real student behavior or an artifact of the rule definitions. Since the reader already conditions acceptance on release and validation of the rule suite, my independent stress test does not change the verdict.","tokens_in":12270,"tokens_out":3597,"duration_ms":49030,"concrete_test":"Publish the exact ArchUnit rule suite; on a stratified random sample of 20 projects (5 from each group-year), have two independent annotators manually classify every inter-package dependency against the taught pattern. Run the ArchUnit suite on the same projects and compute per-rule precision and recall, plus inter-annotator agreement. If precision or recall falls below roughly 90% for any reported coupling, recompute Table 2; if the corrected G-to-M percentage shifts by more than 10 percentage points or leaves the majority band, the RQ1 headline is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim for RQ1 is that 75% of projects contain disallowed GUI-to-Model dependencies, with other coupling frequencies reported similarly. These numbers are produced by a custom ArchUnit suite created by two of the authors, but the paper neither describes the rules nor validates them: Section 3.3 only says the tests 'systematically check for disallowed dependencies,' and Section 6 claims the suite was 'tuned and validated' without reporting any validation procedure. This matters because ArchUnit matches are meaningful only if the suite correctly maps student code to the four taught layers (GUI/Controller/Model/DAO), handles non-canonical package naming and nested or transitive references, and distinguishes genuine direct dependencies from incidental ones (e.g., Model objects passed through generic containers, listeners, or serialization helpers). If a rule is too broad, every project that hands a Model instance to a GUI component is counted as a G-to-M violation; if too narrow, reflection-based or fully-qualified-name accesses are missed. Because the reported metric is presence/absence per project, even a small systematic misclassification in either direction can move a headline value across the 'majority of projects' threshold. The replication package is promised but not evaluated here, so the RQ1 numbers are currently not independently checkable. This is the weakest link in the paper's strongest empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an empirical study of software quality in 83 Java projects developed by 172 undergraduate students across four editions of an OOP course. Using a static-analysis pipeline that combines SonarQube and a custom ArchUnit rule suite, the authors answer two research questions: RQ1 measures the prevalence of architectural violations (e.g., GUI-to-Model dependencies in 75% of projects) and RQ2 measures the prevalence of code smells and bugs (e.g., unclosed resources in 76% and duplicated string literals in 96% of projects). The authors argue that these frequencies identify where educators should focus quality-related teaching, and they provide course details plus a discussion of threats to validity.","tokens_in":12534,"tokens_out":8805,"duration_ms":95284,"significance":"If the reported frequencies are accurate, the paper fills a genuine gap: prior work mostly targets novice programmers and small, self-contained tasks, whereas this corpus consists of multi-component GUI/database projects developed by intermediate-level students. The use of two complementary tools, the relatively large corpus, and the explicit description of the course context are strengths, as is the promise of a replication package. However, the central RQ1 claim rests on a custom ArchUnit rule suite that is neither specified nor independently validated, and the RQ2 headline percentages in Table 4 contain arithmetic inconsistencies with Table 1. These issues must be resolved before the empirical frequencies can be regarded as reliable evidence for the paper's educational recommendations.","major_comments":[{"comment":"The ArchUnit rule suite is the measurement instrument for RQ1, but the paper does not specify the rules, how student packages were mapped to the four taught layers, or how the suite was validated. Section 3.3 only says the suite was 'created by two of the authors' and Section 6 states that it was 'tuned and validated' without reporting any procedure. Because the RQ1 metric is presence/absence of a violation per project, a systematic false-positive or false-negative rate can move the reported percentages (e.g., the 75% G-to-M figure) across the majority threshold. Please provide the full rule definitions, the package-mapping strategy (including handling of non-canonical names and nested or transitive references), and a validation against manually annotated projects, such as precision/recall or inter-rater agreement.","section":"Section 3.3, Table 2 (RQ1)"},{"comment":"The counts in Table 4 are internally inconsistent with Table 1. Table 1 lists 20 projects for Group 1 in 2022/23, but Table 4 reports 17 (77%) for 'Resources should be closed' in that cohort, which would require 17/22, not 17/20; the row 'String literals' (21 (95%)) also implies a denominator of 22. If the true denominator is 22, the total number of projects becomes 85, not 83. Moreover, the 'Overall' percentages do not match the sums of the per-group counts divided by 83: 'Resources' sums to 65 (78%), not 76%, and 'Unnecessary imports' sums to 64 (77%), not 75%. Since these percentages are the headline results of RQ2, the table must be reconciled with the stated corpus size.","section":"Table 4 versus Table 1 (RQ2)"},{"comment":"The selection of the 17 highlighted issue types is not fully reproducible. The paper states that issues 'explicitly related to topics outside the scope of the course' were filtered out, but it gives only one example (lambda expressions) and does not list the excluded rules or the complete set of rules with prevalence above 50%. Because the central RQ2 list is defined by this post-hoc filter, please provide the full per-rule prevalence data, the exclusion list with reasons, and the per-project raw data in the replication package so that readers can verify which rules were considered and why.","section":"Section 4, RQ2 filtering"}],"minor_comments":[{"comment":"The sentence 'A possible cause for these slight reductions... might be due to the increased effectiveness of teaching in person' is speculative; Table 1 shows that mean NCLOC also decreased in the 22/23 editions (e.g., Group 1 from 5797 to 3887), so project size is a confound. Please soften the claim or control for size.","section":"Section 4, RQ2, Table 3"},{"comment":"Please specify the SonarQube version and the exact rule-set version, since the number of default Java rules varies across releases and would affect the reproducibility of the RQ2 counts.","section":"Section 3.3"},{"comment":"The statement that disallowed dependencies are 'highlighted in red' is not accessible in monochrome print; use an explicit marker such as an asterisk or a 'Disallowed?' column.","section":"Table 2"},{"comment":"The term 'social debt' is introduced without definition; if it refers to the social dimension of software quality, please define it explicitly or rephrase.","section":"Section 2.1"},{"comment":"There is a typo: 'betweeen' should be 'between'.","section":"Section 4, RQ1"},{"comment":"The replication package is described as anonymized, but the URL (https://github.com/SergioDME/SEAA2025) appears to contain an author name; please verify that the link is correct and that the package includes the ArchUnit rules and raw per-project data.","section":"Reference [1]"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript is within the scope of an empirical software-engineering education venue, and the corpus is more representative of intermediate-level student projects than much prior work. The main concerns are the unvalidated custom ArchUnit suite for RQ1 and the inconsistent arithmetic in Table 4 for RQ2; both are fixable but require the authors to supply the missing specifications and correct the data. I would not reject on novelty grounds. Please also have the editorial office check that the GitHub link in reference [1] is accessible and that the repository contents match the claims in Section 4."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is best read as a descriptive baseline: a solid, if modest, empirical contribution mapping where intermediate OOP students go wrong in real team projects. The RQ2 results alone—unclosed resources in 76% of projects, duplicated strings in 96%, naming violations in 94%, hard-coded DB credentials in 39%—are worth having on record. Those come from an established tool (SonarQube) and a defined corpus of 83 projects across four course editions, with clear teaching context. That part I trust.\n\nThe architectural half (RQ1) is the problem. The headline 75% GUI-to-Model coupling figure is generated by a custom ArchUnit suite written by two of the authors. The paper says the suite is 'tuned and validated' but gives no details on the rules, no precision/recall against manual inspection, and no description of how student packages are mapped to the four layers. The stress-test note is right: a too-broad rule could count every project that passes a Model object into a GUI listener as a violation, and a too-narrow rule misses reflection or fully qualified names. Since the metric is presence/absence per project, a small systematic bias moves the percentages across the majority threshold. This is not a fatal flaw—the numbers are probably in the right ballpark—but it is load-bearing for RQ1, and currently the code is not checkable because the promised repository is not part of this submission.\n\nThe other soft spots are minor. The causal language ('due to student inexperience') is interpretive without a baseline; the 'filtering' that reduced the issue set to 17 is described loosely, though the full tables are in the replication package; and the usual single-university external validity caveat applies, which they acknowledge. None of this undermines the core contribution.\n\nWho is this for? Educators and researchers in CS education and software quality mining who want concrete frequencies to plan curricula. The RQ2 data is immediately usable; RQ1 needs verification.\n\nMy recommendation: send it to peer review, with a request that the authors provide the ArchUnit rule suite, raw data, and a manual validation sample for RQ1. Conditional accept is the right bar. I'd cite it for RQ2 and as a motivating dataset, with a caveat on the architectural numbers.","headline":"Useful descriptive data on where intermediate OOP students struggle, but the architectural frequencies need validation before being cited as facts.","tokens_in":13067,"tokens_out":2625,"would_cite":true,"duration_ms":30919,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Student Java projects repeat the same quality flaws, study finds","keywords":["software quality","student projects","code smells","architectural anti-patterns","object-oriented programming education","static analysis","SonarQube","ArchUnit"],"falsifier":"Manually inspect a random sample of, say, 20 of the 83 projects, labelling actual GUI-to-Model dependencies and unclosed resources by hand, then compare with the ArchUnit and SonarQube output; if the automated rates deviate substantially from the manual rates, the reported frequencies are test artifacts rather than student behaviour.","tokens_in":12091,"feed_emoji":"🐞","tokens_out":4809,"duration_ms":49808,"temperature":0.7,"pith_summary":"This study measures how often intermediate computer-science students violate basic software-quality rules when they build their first substantial object-oriented projects. Working from 83 team projects and 172 students across four course editions, the authors find the same defects reappearing at high rates: three-quarters of projects contain forbidden direct dependencies from the GUI layer to the data model, and nearly all projects duplicate string literals or break naming conventions. The intended payoff is educational: if these defect frequencies hold, teachers can stop guessing and target the mistakes students actually make.","feed_headline":"Student Java projects repeat the same quality flaws, study finds","feed_subtitle":"Audit of 83 OOP projects maps which mistakes are common enough to teach first.","key_machinery":"The mechanism is a two-part static analysis pipeline: ArchUnit, a Java library that encodes architectural rules as JUnit tests, checks that no dependency crosses the prescribed package boundaries (for example, GUI must not touch Model); SonarQube, configured with its 677 default Java rules, classifies findings into Bugs, Vulnerabilities, and Code Smells. The ArchUnit rule suite was written by two of the authors specifically for the taught architecture, while SonarQube supplies the code-quality categories and severities that drive the second research question. What carries the argument is the coupling of these two detections with per-project violation counts, which converts individual student code into frequencies over 83 projects.","core_discovery":"On the paper's own terms, the discovery is that intermediate OOP students, given a prescribed four-package architecture (GUI, Controller, Model, DAO) and a substantial database-backed GUI assignment, systematically produce a recognizable set of quality defects. The headline numbers: GUI-to-Model dependencies in 75% of projects, unclosed resources in 76%, duplicated string literals in 96%, naming-convention violations in up to 94%, and hard-coded database credentials in 39%. The authors interpret these as symptoms of two underlying causes: students take shortcuts to avoid serialization between layers, and they misunderstand responsibility assignment between packages. They also note that many resource-handling mistakes may come from students carrying over C habits to Java, wrongly trusting garbage collection to manage streams and connections.","pith_inferences":["The high prevalence of GUI-to-Model links suggests a testable hypothesis: students who receive worked examples of serialization refactorings will show lower violation rates in subsequent projects; the paper does not test this.","Because the data comes from a single university with a specific taught architecture, the exact percentages likely shift in other settings, but the ordering (naming > resource handling > architectural shortcuts) may be a stable pattern worth treating as a prior for course design elsewhere.","The finding that 39% of projects hard-code database credentials, though not a majority issue, implies that security basics could be taught at the same stage; the paper flags it as Blocker severity but does not fold it into the main teaching recommendations."],"forward_implications":["If the frequencies hold, OOP courses should explicitly teach why GUI-to-Model shortcuts are harmful, not just how to structure packages; exercises that force serialization through controllers would address the most common architectural violation.","The 76% unclosed-resources rate suggests teaching try-with-resources and contrasting Java's garbage collection with C's manual management early, before students build database-backed apps.","Naming conventions and duplicated-string problems dominate (84–96%), so a first pass with an automated formatter or linter in CS1 could remove most of the maintainability debt before advanced software-engineering courses.","Educators can use the same static-analysis pipeline as an assessment artifact: running SonarQube and ArchUnit on submissions gives objective, reproducible quality baselines for grading and feedback."],"supporting_citations":[{"why":"Supplies the static-analysis pipeline (SonarQube plus ArchUnit) that the study applies to student projects; it is the direct methodological predecessor.","marker":"[9]"},{"why":"Comparative evaluation of six static analysis tools; justifies choosing SonarQube as the most effective for broad issue detection.","marker":"[21]"},{"why":"Large-scale analysis of 2.6 million beginner code snapshots; the baseline the paper contrasts with its own intermediate-level, multi-component projects.","marker":"[20]"},{"why":"Taxonomy of code defects across 114,000 small Python solutions; another baseline for the claim that prior work studies small tasks.","marker":"[15]"},{"why":"Evaluated 54 student projects with SonarQube and CheckStyle; the most directly comparable prior study of course-project quality.","marker":"[6]"},{"why":"Prior study of OOP students' program and test quality, but on smaller assignments; frames the gap this paper addresses.","marker":"[31]"},{"why":"ACM computing curricula recommendations; motivates why quality occupies limited time in bachelor programmes.","marker":"[7]"}],"fun_headline_variants":["83 student projects reveal top code quality mistakes","Student OOP projects repeat GUI-model tangles and leaks","Teach these first: audit of 83 student projects","Student shortcuts cause resource leaks in 76% of projects","Intermediate students still hard-code database credentials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported frequencies rest on the accuracy of the custom ArchUnit rules written by two of the authors and on SonarQube's default 677-rule set as a fair proxy for code quality; neither is independently validated against a manually labelled sample, so the percentages could be artifacts of the rule definitions.","fun_headline_variants_meta":{"raw":{"variants":["83 student projects reveal top code quality mistakes","Student OOP projects repeat GUI-model tangles and leaks","Teach these first: audit of 83 student projects","Student shortcuts cause resource leaks in 76% of projects","Intermediate students still hard-code database credentials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000372,"raw_usage":{"total_tokens":1966,"prompt_tokens":896,"completion_tokens":1070,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":997}},"tokens_in":512,"tokens_out":1070,"duration_ms":13519,"temperature":1.0,"reasoning_tokens":997,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:04:20.572789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually inspect a random sample of, say, 20 of the 83 projects, labelling actual GUI-to-Model dependencies and unclosed resources by hand, then compare with the ArchUnit and SonarQube output; if the automated rates deviate substantially from the manual rates, the reported frequencies are test artifacts rather than student behaviour.","supporting_citations":[{"cited_title":"In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering","cited_arxiv_id":null,"evidence_quote":"Supplies the static-analysis pipeline (SonarQube plus ArchUnit) that the study applies to student projects; it is the direct methodological predecessor."},{"cited_title":"In: Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education","cited_arxiv_id":null,"evidence_quote":"Large-scale analysis of 2.6 million beginner code snapshots; the baseline the paper contrasts with its own intermediate-level, multi-component projects."},{"cited_title":"In: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering","cited_arxiv_id":null,"evidence_quote":"Evaluated 54 student projects with SonarQube and CheckStyle; the most directly comparable prior study of course-project quality."},{"cited_title":"IEEE Access 8, 37505–37517 (2020)","cited_arxiv_id":null,"evidence_quote":"Prior study of OOP students' program and test quality, but on smaller assignments; frames the gap this paper addresses."},{"cited_title":"In: Proceedings of the 50th ACM Tech- nical Symposium on Computer Science Education","cited_arxiv_id":null,"evidence_quote":"ACM computing curricula recommendations; motivates why quality occupies limited time in bachelor programmes."}],"review_version":1}