{"id":"be2358f7-b81c-47c3-b5e0-c37e6b511ea8","arxiv_id":"2502.18466","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MLScent reports 87.5% agreement, recall 0.875, and F1 0.933 for detecting ML anti-patterns on 72 expert-annotated samples from 7 projects, plus prevalence counts from 43 repositories.","lead":"MLScent is a new static analysis tool that uses Python abstract syntax trees to detect 76 machine-learning-specific code smells across TensorFlow, PyTorch, Scikit-learn, Hugging Face, Pandas, and NumPy. If its accuracy claims hold, it gives ML teams a free, framework-aware code quality check that targets reproducibility and maintainability problems other linters miss.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth annotations are not independent of MLScent's detector taxonomy, and precision is never reported; the 87.5% agreement and F1 0.933 therefore measure shared taxonomy rather than verified detection quality.","rationale":"The paper's contribution is a static-analysis tool and the claim that it accurately detects ML anti-patterns. The detectors are built from a catalog of ML smells documented in prior work, and the evaluation asks developers to annotate code after being briefed on exactly those smell types. This creates a circular benchmark: it verifies that MLScent encodes its own specification, not that the taxonomy corresponds to genuine, independently recognized defects. The unreported precision makes the circularity harder to detect, because the reported F1 is only consistent with near-perfect precision, which is suspicious for a static analyzer and should have been reported directly. An independent annotation protocol is standard in code-smell detection research; without it, the headline accuracy claim is unsupported. The reader's weakest_assumption identifies this same circularity, and the CONDITIONAL verdict is appropriate: the tool is plausible and the replication package is promised, but the evaluation must be redone with blind annotation and full metric reporting before 'high accuracy' can be accepted.","tokens_in":15148,"tokens_out":2793,"duration_ms":29064,"concrete_test":"Have the same 7 developers annotate a held-out sample of their code without seeing any list of target smell types, and have a separate expert adjudicate disagreements between MLScent's output and the annotations; then recompute recall, precision, and F1 from the full confusion matrix. If precision is substantially below 1.0, or agreement drops below the reported 87.5%, the original metrics are inflated by annotation priming. Report per-detector confusion matrices so false-positive rates can be inspected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 'high accuracy' rests entirely on Section IV-A's evaluation. In Section IV-A.1, the 7 developers who annotate the 72 ground-truth entries are first 'provided with detailed information about the types of ML-specific code smells we were targeting'—i.e., exactly the taxonomy that MLScent's 76 detectors implement. Agreement between MLScent and annotations primed by that taxonomy demonstrates only that developers can recognize the same patterns MLScent was built to find, not that MLScent detects true smells in unguided practice. The same developers then rate the tool's usefulness (Section IV-A.2, Table V), compounding the dependency. Compounding this, Table IV reports F1 = 0.933 and recall = 0.875 but never reports precision; algebra forces precision ≈ 1.0, which is implausible for a static analyzer and would need to be shown explicitly. The 43-repository prevalence counts (Table VII) are unvalidated detector hits, not verified anti-patterns. Because the only evidence for the headline claim is this non-independent and non-transparent metric set, 'high accuracy' is not supported by the reported evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MLScent, a static-analysis tool for detecting ML-specific anti-patterns and code smells in Python projects. The tool implements 76 detectors across TensorFlow, PyTorch, Scikit-learn, Hugging Face, Pandas, NumPy, and a general ML category. The authors report an evaluation based on 72 ground-truth entries annotated by seven developers from seven ML projects, achieving an agreement rate of 87.50%, recall 0.875, F1-score 0.933, and F2-score 0.897, plus a small user study (N=7) on perceived usefulness. They also report a prevalence analysis over 43 GitHub repositories, finding tens of thousands of detector hits across the supported frameworks.","tokens_in":15302,"tokens_out":3925,"duration_ms":38164,"significance":"A validated ML-specific code smell detector would be a useful contribution, and the paper's replication package (documentation, source code, Zenodo archive) is a concrete strength. The tool addresses a real gap: general-purpose linters and data-validation tools do not cover framework-specific ML anti-patterns. However, the significance of the contribution currently rests on an evaluation whose ground truth is not independent of the detector taxonomy, whose headline metric is incomplete (no precision), and whose prevalence counts are unvalidated detector hits. The paper's central claim of 'high accuracy' is therefore not yet established by the reported evidence.","major_comments":[{"comment":"The ground-truth annotation process is not independent of MLScent's detector taxonomy. The seven developers were 'provided with detailed information about the types of ML-specific code smells we were targeting' before annotating, and those target types are the same smell catalog from which the detectors were built. Agreement between MLScent and annotations primed by that catalog demonstrates that developers can recognize the predefined patterns, not that MLScent detects true smells in unguided practice. A valid evaluation needs an independent ground truth: for example, annotations elicited without exposing annotators to the detector list, or a separately curated benchmark from a different source. This issue directly affects the headline claim of high accuracy.","section":"IV.A.1, III.A"},{"comment":"Precision is never reported, yet the reported F1=0.933 and recall=0.875 algebraically imply precision of approximately 0.999 (since F1 = 2PR/(P+R)). For a static analyzer on real-world code such a value is surprising and must be shown explicitly, together with the confusion matrix and per-framework precision values. Without this, the text's assertion that 'F1-score (0.933) indicates that the tool provides reliable detection capabilities while minimizing false alarms' is unsupported.","section":"Table IV, IV.A.2"},{"comment":"The user study uses the same seven developers who created the ground-truth annotations after being briefed on the target smell types. Their later ratings of tool usefulness and smell detection therefore compound the dependency described above and cannot be treated as an independent qualitative evaluation. The paper should recruit separate practitioners for the usability study, or at minimum explicitly discuss and control for this shared-participant bias.","section":"IV.A.2, Table V"},{"comment":"The prevalence figures (for example, 31,033 General ML issues, 10,795 Array Creation Efficiency instances) are raw detector hits, not verified anti-patterns. Without a validated false-positive rate on these repositories or a sample-based verification of a subset of hits, these numbers cannot support the RQ4 conclusion about the 'prevalence and distribution of different types of ML anti-patterns.' The paper should either verify a sample of these hits or clearly label Table VII as unvalidated detector counts.","section":"IV.D, Tables VII-IX"},{"comment":"No existing tool is run as a baseline. The paper argues that tools like mllint lack framework-specific analysis and positions MLScent as a more complete solution, but the evaluation never compares MLScent against mllint or another accessible baseline on the same projects. Adding even a small comparative experiment would substantially strengthen the relative-advantage claims.","section":"II.A, IV.A"}],"minor_comments":[{"comment":"The annotation instructions are not described beyond 'detailed information about the types of ML-specific code smells we were targeting.' The paper should include the exact checklist or taxonomy given to annotators, and should report inter-annotator agreement (e.g., Cohen's kappa) to assess the reliability of the ground truth.","section":"IV.A.1"},{"comment":"The text states that the study comprised 72 total ground truth entries and 39 unique anti-pattern types, while Table VI lists counts that sum to 72 across five framework categories. The relationship between 'entries,' 'unique types,' and the per-framework counts in Table VI should be clarified.","section":"IV.A, IV.A.2"},{"comment":"The manuscript says 'For each detected smell, we categorized it as true positive, false positive, or false negative based on manual verification by ML experts' but does not state who these experts were or whether they were the same developers. This should be specified, along with the exact annotation procedure.","section":"IV.A.1"},{"comment":"With N=7, the survey results are descriptive at best; the paper should avoid interpreting small differences in means (e.g., 3.29 vs 3.43) as meaningful findings and should provide the full response distribution rather than only summary statistics.","section":"Table V, IV.A.2"},{"comment":"The discussion acknowledges false positives as a limitation, but the evaluation does not quantify them. Reporting precision in Table VI would directly address this acknowledged concern.","section":"V.B, IV.A.2"},{"comment":"The illustrative code snippet for chain-indexing detection only checks nested Subscript nodes at the top level of a parsed module; this is a simplification that does not reflect how the real detector handles nested expressions. The example should be labeled as a simplified illustration or aligned with the actual implementation.","section":"III.G"},{"comment":"There is a typo in the Index Terms: 'Anit-patterns' should be 'Anti-patterns.'","section":"Index Terms"}],"recommendation":"major_revision","confidential_remarks":"This paper has a useful tool and a working replication package, but the evaluation as reported does not support the central claim of high detection accuracy. The ground-truth design is circular with respect to the detector taxonomy, precision is omitted despite being implied to be near-perfect, and the prevalence analysis counts unvalidated detector hits. These are fixable in a revision, but they require substantial additional data collection and analysis, so I recommend major revision rather than rejection. The reader's assessment aligns with my own reading of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MLScent is a real tool with real breadth: 76 AST-based detectors covering TensorFlow, PyTorch, scikit-learn, Hugging Face, Pandas, and NumPy, plus a general ML category. That is the paper's actual contribution, along with prevalence counts from 43 popular ML repositories. The architecture is clean and the replication package addresses reproducibility. If you work on ML code quality tooling, this is worth knowing about.\n\nThe evaluation, however, does not back the abstract's 'high accuracy' claim. The ground truth is 72 annotations from 7 developers who were first told exactly which smell types the tool targets. Unsurprisingly, their labels agree with the tool's detectors: the 87.5% agreement and F1 of 0.933 largely measure shared taxonomy, not independent detection quality. The same developers later rated the tool's usefulness, which compounds the dependency. Precision is never reported, and the reported recall and F1 force precision to be almost exactly 1.0—implausible for a static analyzer and worth reporting explicitly. No baseline tool (mllint, SonarQube, or even a simple linter) is run for comparison. The 54,000 hits across 43 repos in Table VII are raw detector firings, not verified anti-patterns, so the prevalence story is suggestive, not measured. There are also minor internal inconsistencies: PyTorch shows 9 issues in Table VI but 10 in the text, Hugging Face 11 vs 12, and a couple of citations ([14], [37]) do not point where the text seems to intend.\n\nNone of this sinks the tool itself. The detectors are built from documented smell catalogs, the framework-specific coverage is a genuine extension over existing tools, and the paper is transparent about many limitations in its threats-to-validity section. The soft spots are in the evaluation design and the overclaiming in the abstract, not in the artifact.\n\nWho is this for? Researchers and practitioners building or using ML code quality checkers. With a revised evaluation—independent annotators, precision reported, a baseline comparison, and a sampled validation of the prevalence data—this could be a solid contribution. As it stands, the tool is worth citing; the accuracy numbers are not.\n\nMy recommendation: send it to peer review with the expectation of major revision. The artifact and dataset justify referee time, and the evaluation weaknesses are fixable rather than fatal.","headline":"A genuinely useful ML smell detector with an evaluation that does not yet support its accuracy claims; worth serious revision.","tokens_in":15857,"tokens_out":1904,"would_cite":false,"duration_ms":17944,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MLScent finds ML code smells at 87.5% agreement","keywords":["ML code smells","static analysis","abstract syntax tree","anti-pattern detection","technical debt","machine learning projects","Python","code quality"],"falsifier":"Run a blind study in which new ML developers, who have not seen MLScent's detector list, independently annotate the same 72 code snippets; if their agreement with MLScent falls substantially below the reported 87.5% (for example, F1 below 0.7), the central accuracy claim is not reproducible. A simpler check is to compute confidence intervals for the 72-entry metrics, which are currently reported as point values with no uncertainty.","tokens_in":14872,"feed_emoji":"🐍","tokens_out":4526,"duration_ms":36899,"temperature":0.7,"pith_summary":"MLScent is a static-analysis tool built to find code smells that are specific to machine learning projects rather than generic Python issues. The paper claims that its 76 detectors, working on Python abstract syntax trees, identify these anti-patterns accurately: 87.5% agreement with developer annotations and an F1 score of 0.933 across 72 ground-truth entries in seven real projects. The same tool, run over 43 popular GitHub ML repositories, reports that framework-agnostic smells such as array creation inefficiency and missing axis specification dominate. If accurate, the tool gives ML practitioners a way to catch data leakage, poor data splitting, and reproducibility problems that general-purpose linters miss.","feed_headline":"MLScent finds ML code smells at 87.5% agreement","feed_subtitle":"New AST-based checker with 76 detectors targets TensorFlow, PyTorch, Pandas and more; recall 0.875 on real-world code.","key_machinery":"The central mechanism is Abstract Syntax Tree (AST) analysis using the Astroid library: MLScent parses Python code into an AST, identifies which ML framework is in use, and applies the matching set of pattern-matching detectors. There are 76 detectors total, split across TensorFlow (13), General ML (16), PyTorch (12), Hugging Face (10), Scikit-learn (9), Pandas (8), and NumPy (8). A representative detector, chain indexing in Pandas, fires when the AST contains nested Subscript nodes representing accesses like df['column']['row']. The AST-based design is what lets the tool recognise framework-specific constructions that generic linters ignore.","core_discovery":"The paper introduces MLScent and argues that it accurately detects ML-specific anti-patterns across real-world Python projects. It reports an overall agreement rate of 87.50% with manually annotated ground truth, a recall of 0.875, an F1-score of 0.933 and an F2-score of 0.897 across 72 ground-truth entries from 7 ML projects. Framework-level results range from perfect agreement for Hugging Face (100%) to lower agreement for NumPy (75%), and a scan of 43 high-star GitHub ML repositories found 31,033 general ML issues, 17,634 NumPy issues, and thousands of Pandas, TensorFlow, and PyTorch issues. The paper interprets these numbers as evidence that MLScent is a practical, comprehensive checker for ML code quality that existing generic tools cannot provide.","pith_inferences":["The ground-truth annotators were briefed on the tool's target smells before labelling, so the 87.5% agreement may partly measure shared terminology; an independent blind labelling by experts who have not seen the detector list would test this.","Because the tool is static and AST-based, it cannot detect smells that only appear at runtime or through data flow, such as actual data leakage by shared scalers; the reported numbers should be read as coverage of syntactic patterns, not of all correctness risks.","The prevalence results mix performance hints (array creation efficiency) with correctness smells (randomness control); treating all 76 detector outputs as equally severe could overstate technical debt in real projects.","A direct head-to-head against SonarQube, PMD, or mllint on the same 43 repositories would clarify how much of the detection is genuinely ML-specific rather than a rediscovery of general code issues."],"forward_implications":["ML teams can run MLScent in CI to catch data leakage, improper train-test splits, and hardcoded hyperparameters before they invalidate model evaluation.","The prevalence scan gives a first large-scale baseline: array creation efficiency (10,795) and missing axis specification (5,996) are the most frequent smells in popular ML projects.","The modular detector architecture means new frameworks and smells can be added as best practices evolve, keeping the tool current with the ML ecosystem.","The reported F1 of 0.933 suggests that, for the seven annotated projects, the tool's detections mostly align with developer judgment, making its output actionable rather than noisy."],"supporting_citations":[{"why":"Sculley et al.'s hidden-technical-debt analysis supplies the core notion that ML systems accumulate debt through data dependencies, a foundation for MLScent's smell definitions.","marker":"[7]"},{"why":"Zhang, Cruz and van Deursen's catalog of ML code smells is the direct source for many of the 76 detector rules.","marker":"[12]"},{"why":"van Oort et al.'s prevalence study motivates the need for ML-specific detection and shapes the framework categories.","marker":"[13]"},{"why":"O'Brien et al.'s 23 patterns of self-admitted technical debt in ML software informs the general ML and documentation detectors.","marker":"[9]"}],"fun_headline_variants":["MLScent sniffs out ML anti-patterns with 76 detectors","AST-based MLScent catches ML code smells across 7 frameworks","MLScent: 76 detectors for ML code smells, 87.5% agreement","MLScent detects ML anti-patterns in TensorFlow, PyTorch, and more","MLScent achieved 87.5% agreement on real ML code reviews"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy numbers rest on 72 ground-truth annotations made by developers who were told in advance which ML-specific smells the tool targets; if that briefing shaped their labels, the agreement rate measures a shared vocabulary rather than independent detection quality.","fun_headline_variants_meta":{"raw":{"variants":["MLScent sniffs out ML anti-patterns with 76 detectors","AST-based MLScent catches ML code smells across 7 frameworks","MLScent: 76 detectors for ML code smells, 87.5% agreement","MLScent detects ML anti-patterns in TensorFlow, PyTorch, and more","MLScent achieved 87.5% agreement on real ML code reviews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1594,"prompt_tokens":916,"completion_tokens":678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":573}},"tokens_in":532,"tokens_out":678,"duration_ms":6550,"temperature":1.0,"reasoning_tokens":573,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:07:23.988033+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a blind study in which new ML developers, who have not seen MLScent's detector list, independently annotate the same 72 code snippets; if their agreement with MLScent falls substantially below the reported 87.5% (for example, F1 below 0.7), the central accuracy claim is not reproducible. A simpler check is to compute confidence intervals for the 72-entry metrics, which are currently reported as point values with no uncertainty.","supporting_citations":[{"cited_title":"Hidden technical debt in machine learning systems,","cited_arxiv_id":null,"evidence_quote":"Sculley et al.'s hidden-technical-debt analysis supplies the core notion that ML systems accumulate debt through data dependencies, a foundation for MLScent's smell definitions."},{"cited_title":"Code smells for machine learning applications,","cited_arxiv_id":null,"evidence_quote":"Zhang, Cruz and van Deursen's catalog of ML code smells is the direct source for many of the 76 detector rules."},{"cited_title":"The prevalence of code smells in machine learning projects,","cited_arxiv_id":null,"evidence_quote":"van Oort et al.'s prevalence study motivates the need for ML-specific detection and shapes the framework categories."}],"review_version":1}