{"id":"bbb9490a-a3aa-43fe-adf8-ba64f6c35191","arxiv_id":"2412.11869","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"This paper defines seven inter-language design smells for Python/C++ deep learning frameworks and reports a detector (CPSMELL) with 98.17% precision on five frameworks.","lead":"Researchers built a tool that finds seven kinds of design problems that appear when Python and C++ code connect inside deep learning frameworks. They tested it on five major frameworks and report that it flags these problems with high precision.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98.17% headline accuracy is precision against a ground truth defined as 'conformance to the detection rules of Section 3' — exactly what CPSMELL implements — so it largely measures self-consistency, plus the 18 special-case corrections, not whether the smells are genuine maintainability defects.","rationale":"The reader and I independently converge on the same load-bearing point: the 98.17% figure is computed against a ground truth that is definitionally tied to the tool under test. Section 4.2.2's first validation criterion — 'a true ILDS instance should fully conform to the detection rule defined in Section 3' — makes each TP/FP decision an internal consistency check of implementation against specification, because CPSMELL is the implementation of exactly those rules. Only the second criterion (contextual special-case judgment) admits independent evidence, and that channel produced precisely the 18 FPs separating 100% from the 98.17% headline. The smells with perfect per-smell scores (LLF, LREC, LSD, LILBC, EILC) carry no independent evidence beyond rule conformance. Calling the metric 'accuracy' is also imprecise: TP/(TP+FP) is precision with no TN/FN terms, and Section 4.4 acknowledges recall is unmeasured, so the abstract's 'achieved an accuracy of 98.17%' overstates what the validation establishes. Credit is due where the paper has independent support: the three-ILCM parsing design, explicit rules and thresholds for six of seven smells, a published tool repository and validation data, and a limitations section that honestly flags missing recall and small per-smell counts. A few detected instances demonstrably correspond to real maintainer actions (PyTorch removing test_deploy_lib.cpp; the THPSize_init refactoring in Section 3.3), giving the catalog partial external grounding, but these are anecdotes, not systematic validation. Two secondary issues are fixable without changing the assessment: Section 5.3.1's text attributes the 67.71% share to UNE while Table 6 shows 28.04% for UNE and 67.71% for LLF (a smell-name mix-up), and Rule 7's EILC threshold is never stated, so the EILC rows in Tables 6 and 9 are not independently reproducible as reported. The proposed concrete test — external expert re-validation of a stratified sample judged from the ILDS descriptions alone, with a pre-specified decision bound — directly measures the external-validity gap the concern identifies rather than re-measuring rule conformance. Since this concern coincides with the reader's weakest assumption, the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":24009,"tokens_out":14648,"duration_ms":118685,"concrete_test":"Select a stratified random sample of ~120 of the 982 detected instances (~12%, proportional across the 7 ILDS and 5 DLFs). Have judges with no involvement in the paper — core maintainers of the five projects, or failing that senior contributors from their ecosystems — independently classify each instance as a genuine maintainability problem worth refactoring or as acceptable/intentional code. Provide only the code location and the plain-language ILDS description from Section 3, not the formal detection rules and not the tool's label. Compute the confirmed proportion with a 95% Wilson interval. If the lower bound falls below ~90%, the 98.17% figure is inflated by rule-conformance labeling and should be reframed; if it stays above ~95%, the self-referentiality objection is empirically answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 98.17% accuracy figure (abstract, §4.3). The load-bearing premise is that the §4.2.2 manual validation is an independent correctness check; it is not. Criterion 1 for labeling a 'true ILDS instance' is that it 'fully conforms to the detection rule defined in Section 3', and CPSMELL was built to implement exactly those rules, so the TP/FP labels certify that the tool's output agrees with its own specification. The only independent channel is criterion 2, the contextual special-case judgment, and all 18 false positives (17 UNE, 1 NURP in Table 2) came from that channel. The metric is also precision, not accuracy: Acc = TP/(TP+FP) with no TN/FN terms, and §4.4 concedes recall is unmeasured. The headline number therefore cannot support the inference that the smells are genuine defects: the catalog is validated only against its own extraction process (keyword mining plus team discussion, §3), not against external ground truth such as maintainer acknowledgment. Fragility is concentrated: four smells (LREC, LSD, LILBC, EILC) have zero FPs, and the two with FPs (UNE 236/253, NURP 15/16) have modest counts, so per-smell accuracy has wide error bars. Rule 7's EILC threshold is never disclosed, limiting reproducibility of the EILC rows. The core issue, however, is the self-referential validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes seven inter-language design smells (ILDS) for multi-language deep learning frameworks written in Python and C/C++, defines a detection rule for each smell, implements these rules in a tool called CPSMELL that parses three inter-language communication mechanisms (Python/C API, pybind11, ctypes), and manually validates the tool on five popular DLFs. The authors report an accuracy of 98.17% in this validation and then conduct an empirical study across ten versions of each DLF, analyzing the distribution of ILDS, the proportion of smells fixed between versions, and the evolution of smell counts. The abstract and the conclusion present the accuracy figure as the main evidence of the approach's effectiveness.","tokens_in":24395,"tokens_out":9701,"duration_ms":82451,"significance":"If the validation were methodologically sound, this would be a valuable contribution to maintainability analysis of multi-language systems: it provides a concrete catalog of seven inter-language smells with refactoring advice, an open implementation (CPSMELL) covering three ILCMs, and an empirical snapshot of smell prevalence and evolution in five widely used DLFs. The authors also publish their validation data and report inter-rater agreement (Kappa = 0.82), which supports the transparency of the manual labeling process. However, the central accuracy claim currently overstates what the validation establishes: the metric is precision, not accuracy, and the ground truth is largely defined by the tool's own detection rules, so the empirical study inherits this limitation.","major_comments":[{"comment":"The metric labeled 'accuracy' in the abstract and in Section 4.3 is defined in Section 4.2.2 as TP/(TP+FP), which is precision. No true negatives or false negatives are measured, and Section 4.4 explicitly states that recall is not computed. The headline figure '98.17% accuracy' should be renamed to 'precision' throughout, or supplemented with a recall estimate based on a gold standard that includes instances not detected by the tool. As written, the central claim is misleading.","section":"§4.2.2, §4.3, §4.4"},{"comment":"The validation ground truth is self-referential: criterion 1 in Section 4.2.2 states that a true ILDS instance 'should fully conform to the detection rule defined in Section 3,' and CPSMELL implements exactly those rules. Consequently, the TP/FP labels largely certify that the tool agrees with its own specification; the only independent human judgment is the special-case criterion, which produced all 18 false positives in Table 2. This protocol does not establish that the detected instances are genuine maintainability defects. I recommend an external validation step, such as confirmation of a sample of instances by the DLFs' maintainers or by independent developers who are given the smell descriptions but not the formal rules.","section":"§4.2.2"},{"comment":"The threshold MaxNBCallsFilesThreshold used by Rule 7 (EILC) is never disclosed. Section 5.3.1 even states that 'the exact threshold should be set according to practical needs.' Since EILC counts appear in Tables 2, 6, 7, and 9 and in the RQ1/RQ2 results, the EILC results are not reproducible. The paper should report the threshold value used in the experiments and, ideally, a sensitivity analysis over a range of thresholds.","section":"§3.7 (Rule 7), §5.3.1"},{"comment":"The per-ILDS validation sets for LREC (9 instances), NURP (16), and EILC (19) are very small, so the 100% precision reported for these rows is not statistically robust. The paper acknowledges this imbalance in Section 6, but the overall 98.17% figure is dominated by LLF and UNE (741 of 982 instances). Please qualify the per-smell accuracy claims and avoid implying that all seven smells are equally well validated.","section":"Table 2, §6"}],"minor_comments":[{"comment":"The formal rules contain garbled function identifiers (e.g., 'GetPyModule_AddObjectField', 'GetcdllField', 'GetCDLLField'). These should be replaced with the actual API names (PyModule_AddObject, CDLL, WinDLL, LoadLibrary) to make the rules readable and implementable.","section":"§3.3, Rule 3; §3.5, Rule 5"},{"comment":"The sentence 'UNE instances in TensorFlow account for 67.71%' is inconsistent with Table 6, where the 67.71% belongs to LLF; the subsequent 30.90% for PyTorch also corresponds to LLF. Please correct the smell name.","section":"§5.3.1"},{"comment":"The statement that LLF and UNE 'each ... accounting for over 25% of all the instances in each DLF' does not hold for PyTorch (UNE 15.94%) and MindSpore (UNE 21.74%, LLF 0%). Please rephrase to 'in most DLFs' or report the exact exceptions.","section":"§5.2.1 (Answer to RQ1)"},{"comment":"Rule 2 uses an 80-character threshold for the lambda length, but the example in Listing 2 describes a lambda of 50 lines; please clarify whether the measure is characters, lines, or something else and ensure the threshold unit matches the implementation.","section":"§3.2"},{"comment":"The sentence 'the accuracy of detection for other six ILDS reaches 100%' should read 'other five ILDS' (LLF, LREC, LSD, LILBC, EILC), since UNE and NURP are the two exceptions.","section":"§4.3"},{"comment":"The exact percentages for lambda usage in PyTorch (78.71%) and TensorFlow (70.72%) are not shown in any table; please include the underlying data or remove the precise figures.","section":"§5.3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid engineering contribution (tool, dataset, empirical data), but the validation design is the main weakness. If the authors can reframe the claims as precision of an implementation against its own rules and add a small external validation, the paper could be acceptable. The swapped UNE/LLF percentages in §5.3.1 and the garbled API names in the rules suggest a need for careful proofreading of the results discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: this is a useful, clearly described contribution for anyone working on multi-language software maintainability. The seven ILDS are defined with explicit detection rules grounded in three real ILCMs (Python/C API, pybind11, ctypes), and the CPSMELL tool is published with validation data. The empirical part—distribution across five DLFs, fix rates, and evolution over 50 versions—is a reasonable first map of where inter-language smells actually live in frameworks like TensorFlow and PyTorch.\n\nThe soft spot is exactly where the stress-test note lands. The 98.17% figure is precision, not accuracy (no TN/FN terms), and the validation criterion says a true instance 'should fully conform to the detection rule defined in Section 3.' Since CPSMELL implements those rules, the number largely certifies that the tool agrees with its own specification. The only independent check is the special-case review, which found the 18 false positives. So the headline should be reframed as 'precision of the tool against its own detection rules plus human correction of edge cases.' That is still useful, but it does not establish that the smells are real maintainability defects. The paper acknowledges recall is unmeasured and that threshold values are experience-based. Small per-smell samples (LREC=9, NURP=16, EILC=19) make the 100% rows fragile. The EILC threshold is never disclosed, which harms reproducibility for that rule.\n\nThat said, the paper is honest about these limitations in Section 6, and the tool and data are available. The detection rules are specified precisely enough to reimplement or tune. For a first catalogue in a largely unstudied PL combination, this is a legitimate step forward, not a claim that should be taken at face value.\n\nVerdict from me: send it to peer review, but the reviewers should push for (a) renaming the metric to precision, (b) an external validation sample (e.g., maintainer confirmation or issue reports), and (c) disclosure of all thresholds. The empirical study can stand with the corrected framing.\n\nWho gets value: SE researchers on multi-language systems, and maintainers of Python/C++ frameworks looking for a linting starting point. Not a breakthrough, but a solid, citable building block.","headline":"Useful catalog and tool for Python/C++ inter-language smells, but the 98.17% headline is precision against the tool's own rules, not evidence the smells are genuine defects.","tokens_in":25006,"tokens_out":1692,"would_cite":false,"duration_ms":16098,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that seven inter-language design smells in Python/C++ deep learning frameworks can be detected automatically, and that the CPSMELL tool implementing the detection rules achieves 98.17% manually validated accuracy across…","keywords":["inter-language design smells","multi-language software","deep learning frameworks","Python/C API","pybind11","ctypes","design smell detection","software maintainability"],"falsifier":"Have independent experts who have never seen the detection rules label all inter-language design smells in a held-out set of Python/C++ deep learning framework files, then run CPSMELL on the same files and compare both precision and recall; if precision on this blind set falls well below 98.17% or recall turns out to be low, the accuracy claim and the empirical counts built on it would not survive.","tokens_in":23779,"feed_emoji":"🛠️","tokens_out":7736,"duration_ms":62966,"temperature":0.7,"pith_summary":"This paper sets out to show that maintainability problems in multi-language deep learning frameworks can be captured as inter-language design smells: bad practices that arise only at the boundary where Python code meets C/C++ code. The authors define seven such smells, each with an explicit detection rule, and build a tool, CPSMELL, that implements the rules. They report a manually validated accuracy of 98.17% across five popular frameworks, and an empirical study of ten versions per framework showing that long lambda bindings and unused native code are the most common smells while the overall smell count rises over time. Success here would give maintainers a concrete checklist and an automated way to find code that hurts maintainability and that single-language smell detectors miss.","feed_headline":"New tool flags seven inter-language design smells in deep learning","feed_subtitle":"Long lambda bindings and unused native code are the most common smells, and counts rise across versions.","key_machinery":"The load-bearing mechanism is a catalog of seven ILDS, each paired with a deterministic detection rule written in terms of inter-language binding, calling, loading, and file-level call patterns. The CPSMELL pipeline turns C/C++ files into XML representations and Python files into abstract syntax trees, then a language-interface detector recognizes Python/C API, pybind11, and ctypes constructs, and a design-smell detector applies the rules with fixed thresholds, notably an 80-character limit for lambda functions in bindings and a limit of seven bound functions per class. The same machinery produces the empirical counts used in the distribution and evolution analysis.","core_discovery":"The central claim is that seven inter-language design smells—Unused Native Entity, Long Lambda Function For Inter-language Binding, Lack of Rigorous Error Check, Lack of Static Declaration, Not Using Relative Path, Large Inter-language Binding Class, and Excessive Inter-Language Communication—can be identified from the three communication mechanisms commonly used in Python/C++ frameworks (Python/C API, pybind11, and ctypes), and that a tool implementing the identification rules detects them with 98.17% accuracy in TensorFlow, PyTorch, Chainer, PaddlePaddle, and MindSpore. In the authors' empirical study, LLF and UNE together dominate the 982 detected instances, TensorFlow, PyTorch, and PaddlePaddle are the most smell-dense frameworks, and across three years of versions most frameworks show a fluctuating but generally increasing number of ILDS instances even though some smells, notably Excessive Inter-Language Communication and Large Inter-language Binding Class, are fixed at higher rates.","pith_inferences":["The manual validation used the paper's own detection rules as the definition of a true instance, so the 98.17% figure is largely a measure of the tool's consistency with its rules; an independent expert panel working only from the informal smell descriptions could produce a materially different precision.","The paper reports precision but not recall, so the true number of ILDS instances in the five frameworks is unknown; the empirical distribution and upward-trend counts are likely lower bounds.","The thresholds, such as 80 characters, seven bound functions, and the number of native files called per Python file, are judgment calls that control which instances are counted; changing them would shift the observed rankings and fix ratios.","If the seven smells generalize, the same detection pipeline could be extended to other Python/C++ polyglot systems beyond deep learning, making inter-language maintainability measurable in a way that single-language tools cannot."],"forward_implications":["Maintainers of TensorFlow, PyTorch, and PaddlePaddle can use CPSMELL to locate the two most common smells, long lambda bindings and unused native entities, which together account for about three quarters of the 982 detected instances.","The near-zero fix rates for Lack of Rigorous Error Check and Not Using Relative Path imply that these smells persist across releases and need targeted cleanup rather than incidental improvement.","The overall upward trend in ILDS counts implies that adding inter-language APIs is currently introducing smells faster than refactoring removes them.","The high fix rates for Excessive Inter-Language Communication and Large Inter-language Binding Class suggest that architecture-level smells tend to disappear during major restructuring, not through small maintenance edits.","Because the tool recognizes only three communication mechanisms, the reported totals will undercount smells in code that uses other binding technologies such as SWIG or Cython."],"supporting_citations":[{"why":"Supplies the prior definition of inter-language design smells for Java/C++ systems and the detection approach that this paper adapts to Python/C++ deep learning frameworks.","marker":"Abidi et al., 2021"},{"why":"Provides the 80-character long-lambda threshold used by the LLF detection rule.","marker":"Chen et al., 2016"},{"why":"Gives the cognitive-load rationale for the threshold of seven bound functions used by the LILBC rule.","marker":"Lippert and Roock, 2006"},{"why":"Documents Python/C API bug patterns and evolution, grounding the choice of communication mechanisms and the motivation for inter-language quality analysis.","marker":"Hu and Zhang, 2020"},{"why":"Supplies the Kappa statistic used to assess inter-rater agreement in the manual validation of CPSMELL.","marker":"Viera et al., 2005"},{"why":"Provides the prior empirical baseline on code smells in deep learning Python code that this work extends to the inter-language setting.","marker":"Jebnoun et al., 2020"}],"fun_headline_variants":["Tool sniffs out 7 design smells in Python-C++ DL frameworks","98% accurate detection of inter-language design smells in deep learning","CPSMELL: automated detector for seven multi-language design smells","Inter-language design smells plague TensorFlow, PyTorch, and PaddlePaddle","Study: five deep learning frameworks hide seven inter-language code smells"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy claim rests on treating the tool's own detection rules as the ground truth for what a true smell is, so the 98.17% figure mostly shows that the tool follows its own definitions; if that standard does not match what practitioners regard as harmful inter-language design, the accuracy number does not validate the smells.","fun_headline_variants_meta":{"raw":{"variants":["Tool sniffs out 7 design smells in Python-C++ DL frameworks","98% accurate detection of inter-language design smells in deep learning","CPSMELL: automated detector for seven multi-language design smells","Inter-language design smells plague TensorFlow, PyTorch, and PaddlePaddle","Study: five deep learning frameworks hide seven inter-language code smells"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1769,"prompt_tokens":1105,"completion_tokens":664,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":721,"completion_tokens_details":{"reasoning_tokens":570}},"tokens_in":721,"tokens_out":664,"duration_ms":6049,"temperature":1.0,"reasoning_tokens":570,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:29:52.520734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent experts who have never seen the detection rules label all inter-language design smells in a held-out set of Python/C++ deep learning framework files, then run CPSMELL on the same files and compare both precision and recall; if precision on this blind set falls well below 98.17% or recall turns out to be low, the accuracy claim and the empirical counts built on it would not survive.","supporting_citations":[{"cited_title":", author Rahman, M.S","cited_arxiv_id":null,"evidence_quote":"Supplies the prior definition of inter-language design smells for Java/C++ systems and the detection approach that this paper adapts to Python/C++ deep learning frameworks."},{"cited_title":", author Chen, L","cited_arxiv_id":null,"evidence_quote":"Provides the 80-character long-lambda threshold used by the LLF detection rule."},{"cited_title":", author Roock, S","cited_arxiv_id":null,"evidence_quote":"Gives the cognitive-load rationale for the threshold of seven bound functions used by the LILBC rule."},{"cited_title":", author Garrett, J.M","cited_arxiv_id":null,"evidence_quote":"Supplies the Kappa statistic used to assess inter-rater agreement in the manual validation of CPSMELL."},{"cited_title":", author Ben Braiek, H","cited_arxiv_id":null,"evidence_quote":"Provides the prior empirical baseline on code smells in deep learning Python code that this work extends to the inter-language setting."}],"review_version":1}