{"id":"ddc3c142-9422-4b83-a16f-351621e3818f","arxiv_id":"2502.05012","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EnseSmells fuses code embeddings from pre-trained models with hand-crafted metrics and reports higher F1 and MCC than prior baselines on MLCQ for four code smells.","lead":"A team built a code-smell detector that combines two kinds of signals: statistical meaning learned by large code models and classic software metrics like complexity and coupling. In tests on a public Java benchmark, the hybrid detector beat previous state-of-the-art tools on all four smell types, though the reported margins contain inconsistencies and may be optimistic.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-smell 'optimal configurations' appear to be selected on the test set, so the reported margins conflate model quality with selection bias; the loss equation as printed is also not a valid BCE loss.","rationale":"The reader's weakest assumption is the same one I find most load-bearing: the evaluation may be test-set-aware. I agree with that assessment rather than manufacturing a new objection. In a comparison where several reported advantages are single-digit F1/MCC margins, the way 'optimal configuration' was chosen determines whether the central claim is true. The paper's own text never documents a validation-based selection procedure; it only says an 80/20 split and 5-fold cross-validation were used. Tables 6-7 list four embedding choices per smell, and Table 8 selects different winners for different smells. If selection is based on test-fold numbers, then even a null model with random noise would show a best-of-four gain, and statements such as 'incorporating structural modules improves performance by up to 40%' (RQ1) cannot be interpreted causally. The loss-function typo is a separate, concrete reproducibility defect: Eq. (1) has the wrong sign, so as written it would not train the reported classifier. It may be only a typesetting error, but it reinforces the need to inspect the code. I would keep the reader's CONDITIONAL verdict: the architecture is plausible and the paper provides a dataset and a repository, but the current evidence does not yet establish the SOTA claim. A nested cross-validation re-analysis and a code check of the loss function would settle the question.","tokens_in":22568,"tokens_out":4824,"duration_ms":55908,"concrete_test":"Re-run the pipeline with nested cross-validation: hold out the outer test folds, select the per-smell embedding choice and hyperparameters (beta, k, classifier width) on an inner validation split, then evaluate the outer test folds once and report mean and standard deviation for EnseSmells, DeepSmells, and all baselines. If the selected embedding changes per smell, or if the F1/MCC advantage over ML_CuBERT and the autoencoder variants shrinks below the reported 5.98%-28.26% margins, the claim that structural features significantly improve detection is not supported. In parallel, inspect the released code to confirm the implemented loss is the signed binary cross-entropy with a positive-class weight, not Eq. (1).","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that EnseSmells outperforms state-of-the-art baselines rests on Tables 6-9, which compare, for each smell, the best configuration from Table 6 with the best configuration from Table 7. Different smells select different embeddings as 'optimal' (e.g., LM uses token-indexing, FE uses CuBERT, DC uses code2vec), and the paper never states that this per-smell selection, or the tuned hyperparameters beta, kernel size k, and classifier width described in Sections 3.1.1 and 3.2, was performed on a validation split before any test-fold results were observed. Section 4 reports only an 80/20 split followed by stratified 5-fold cross-validation, with no explicit train/validation/test separation, and Tables 6-9 give single point estimates with no variance or significance tests. Selecting the best of four embedding variants after seeing test metrics is a textbook winner's-curse problem: the maximum of four configurations is expected to beat fixed baselines even when no model is truly better. The GC margins over AE-CNN (0.76% in F1, 1.18% in MCC) are precisely the size that such selection bias could manufacture. Additionally, Eq. (1) as printed is not a valid loss: it lacks the leading minus sign, so minimizing it drives predictions toward the wrong labels. If that equation was actually used, the reported results would be impossible; if it was corrected in code, the manuscript misstates the training objective. Both issues must be resolved before the superiority claim is evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EnseSmells, a two-tier deep learning architecture for automated code smell detection that concatenates statistical semantic features extracted from pre-trained code embedding models (code2vec, CodeBERT, CuBERT, and a token-indexing baseline processed by the DeepSmells CNN/LSTM adapter) with structural features learned from object-oriented software metrics via a single adaptive-layer DNN, and classifies the fused representation with an imbalanced deep neural network. The evaluation on the MLCQ dataset covers Feature Envy, Long Method, Data Class, and God Class, and the paper claims state-of-the-art performance with improvements over baselines ranging from 5.98% to 28.26%, depending on the smell. The manuscript also reports a per-smell comparison of embedding choices and a comparison against seven classical machine learning classifiers.","tokens_in":22902,"tokens_out":3810,"duration_ms":40568,"significance":"If the reported evaluation is valid, the paper makes a useful empirical contribution to code smell detection by showing that combining structural metrics with learned code embeddings can improve detection, and by providing a systematic comparison of embedding models across four smell types. The authors re-ran all baselines on the same dataset and provide a public repository, which supports reproducibility. However, the paper's central claim of state-of-the-art performance is currently not established because the evaluation protocol as described does not separate model selection from test evaluation, the printed loss function is not a valid cross-entropy loss, and several textual claims are inconsistent with the reported tables.","major_comments":[{"comment":"Equation (1) is not a valid binary cross-entropy loss. As printed, L(x_i) = β ŷ_i log(y_i) + (1 − ŷ_i) log(1 − y_i) has no leading minus sign, so minimizing it drives predictions toward the incorrect label. If this objective was actually used in training, the reported results would be impossible; if the code contains the standard negative log-likelihood, the manuscript misstates the training objective. Please correct the equation and state explicitly which loss was optimized, including the role of β.","section":"§3.2, Eq. (1)"},{"comment":"The paper does not describe any train/validation/test separation. Section 4 reports an 80/20 split followed by stratified 5-fold cross-validation, but Tables 6 and 7 present per-smell 'optimal configurations' (with tuned β, kernel size k, classifier width, and distinct winning embeddings per smell) without stating that these choices were made on validation folds before test metrics were observed. If the configuration with the best test performance was selected after evaluation, the reported margins, especially the small GC gains over AE-CNN (F1 +0.76 pp, MCC +1.18 pp), are subject to winner's-curse bias and do not support the claimed superiority. Please specify the exact model-selection procedure, or better, use nested cross-validation or a separate validation split, and report the selection policy for hyperparameters and embedding variants.","section":"§4 and Tables 6–9"},{"comment":"The claim of 'approximately 40%' improvement for Data Class is not supported by the tables. Comparing DeepSmells and EnseSmells under their optimal configurations in Table 8, the F1 gain for DC is from 0.5839 to 0.6393 (about 5.5 percentage points, 9.5% relative), and the largest F1 gain visible in Table 6 vs. Table 7 is for token-indexing (0.2531 to 0.6203, an absolute 36.7 pp change), not 40% in both F1 and MCC. Please re-state the improvement using a consistent and clearly defined comparison base and reconcile the abstract's '5.98% to 28.26%' claim with the specific metrics in Tables 8 and 9.","section":"§5.1, RQ1 and Table 8"},{"comment":"The comparison with baselines lacks any measure of variance or statistical significance. Tables 6–9 report single point estimates even though the evaluation uses 5-fold cross-validation, and the FE and GC margins are small (e.g., 0.76 pp F1 and 1.18 pp MCC for GC over AE-CNN). Without standard deviations, confidence intervals, or significance tests (e.g., paired bootstrap or a signed test across folds), the statement that EnseSmells 'demonstrates superior prediction performance for all four types' is stronger than the evidence warrants. Please report per-fold results or dispersion measures and perform a significance test for the differences that drive the conclusions.","section":"§5.4, Table 9"}],"minor_comments":[{"comment":"The text says EnseSmells 'outperforms' all baselines on all metrics, but for Long Method its recall (0.7873) is lower than every baseline (AE-LSTM 0.8480, AE-CNN 0.8315, AE-Dense 0.8274, ML_CuBERT 0.8142). The claim should be restricted to F1 and MCC or the trade-off should be discussed.","section":"§5.4, Table 9"},{"comment":"The prose states that for Data Class EnseSmells improves F1 by 7.22% and MCC by 5.5%, but Table 8 shows F1 0.5839→0.6393 (+5.54 pp) and MCC 0.5185→0.5907 (+7.22 pp). The two percentages appear to be swapped.","section":"§5.4, Table 8"},{"comment":"The notation in Eq. (1) is confusing: ŷ_i is used for the actual label and y_i for the model prediction, the opposite of the conventional convention. Please define all symbols and use standard notation for observed labels and predictions.","section":"§3.2, Eq. (1)"},{"comment":"There are several typographical and consistency issues, including 'with with' in the abstract and Section 7 stating 'five research questions' although only four RQs (RQ1–RQ4) are defined in Section 4.1.","section":"Abstract and throughout"},{"comment":"The metric 'CisibleFieldsQty' appears to be a typo for 'VisibleFieldsQty', and the table caption states there are 44 class-level metrics but only 40 entries are listed; please verify the count and the Appendix.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extension of the authors' prior EASE 2023 work (Ho et al.), and the novelty is mainly the structural module and broader embedding comparison, which is acceptable for a journal if the evaluation is made sound. The main concern is methodological: the lack of a clear validation-based model-selection protocol could invalidate the quantitative claims, and the invalid printed loss must be resolved. I do not see evidence of deliberate misconduct; these are standard but serious empirical-reporting issues that should be fixable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. EnseSmells is a plausible extension of DeepSmells: it adds a software-metrics branch to the CNN+LSTM semantic branch and evaluates four code embeddings (token-indexing, code2vec, CodeBERT, CuBERT) on four MLCQ smells. That fusion design is new, and the systematic comparison of embeddings across smell types is a genuinely useful empirical contribution. They also shipped the code and processed data, which is more than most papers in this space do. If the results hold, this is a solid incremental advance in automated smell detection.\n\nNow the soft spots.\n\nThe most serious one is the model-selection protocol. The paper reports an 80/20 split plus stratified 5-fold CV, but never states that hyperparameters (beta, kernel size k, classifier width) and the per-smell best embedding were chosen on a validation split before touching the test set. Tables 6 and 7 report the 'optimal configuration' per embedding, and different smells pick different embeddings. If the best-of-four selection was done after observing test metrics, then the reported margins over fixed baselines are exactly what winner's curse predicts, and the GC margin over AE-CNN (0.76% F1) is right in that noise band. The authors need to explain the exact validation protocol and ideally report fold-level variance.\n\nSecond, every key number is a single point estimate. No error bars, no significance tests. Given the small positive-class counts (FE has 64 positives), the FE results in particular could move a lot across folds. They need to show the variance.\n\nThird, the numbers in the prose are inconsistent with the tables. The 'approximately 40%' improvement for Data Class does not match Table 8 (the largest F1 gap is about 9.5% relative, or 5.5 percentage points). The '5% to 10%' over DeepSmells in the RQ4 answer also does not match. These overstatements need correcting.\n\nFourth, Eq. (1) as printed is not a valid BCE loss: the leading minus sign is missing, so minimizing it would push predictions toward the wrong labels. If the code used the correct loss, the paper misstates the training objective. Either way it must be fixed.\n\nThe MLCQ dataset is the only evaluation domain, so external validity is limited, and the paper acknowledges this in the threats section. That is a minor issue.\n\nBottom line: the architecture is reasonable, the study is carefully scoped, and the reproducibility artifacts are real, but the central superiority claim is not yet established because of the selection-bias ambiguity and missing variance. I would not cite this in its current form, but I would send it to a serious referee; the issues are fixable and the contribution is worth a round of revision. For a reading group, it would spark a good discussion about evaluation hygiene in ML-based SE.","headline":"Useful fusion architecture and solid empirical scope, but the reported superiority is undermined by unclear model-selection protocol, missing variance, and a loss-function typo.","tokens_in":23434,"tokens_out":2922,"would_cite":false,"duration_ms":28007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EnseSmells claims that combining pre-trained code embeddings with software metrics improves code-smell detection beyond prior state of the art, with MCC gains of 5.98% to 28.26% on four common smells.","keywords":["code smell detection","software metrics","pre-trained code models","deep ensemble","imbalanced classification","MLCQ dataset","Long Method","Feature Envy"],"falsifier":"A decisive check would rerun the same configuration search with a strict three-way split, choosing the best settings on validation data only and reporting test performance once, on held-out projects from the MLCQ dataset; if the fusion model no longer beats ML_CuBERT by the reported margins, the central claim is refuted.","tokens_in":22372,"feed_emoji":"📈","tokens_out":6560,"duration_ms":63785,"temperature":0.7,"pith_summary":"Code smells are design flaws that do not crash a program but make it harder to understand and change. This paper tries to establish that detection improves when two very different views of a code snippet are combined: statistical semantics learned by pre-trained models for programming languages, and structural features from classic object-oriented software metrics. Its architecture, EnseSmells, runs each view through its own neural module and merges them before an imbalanced classifier. On the MLCQ benchmark, the paper reports that this fusion outperforms state-of-the-art baselines for Long Method, Feature Envy, Data Class, and God Class, with Matthews Correlation Coefficient improvements ranging from 5.98% to 28.26%. A sympathetic reading cares because prior deep-learning detectors treated code mostly as text, while metric-based detectors ignored learned context; this paper argues the two signal types are complementary.","feed_headline":"Fusing metrics with code embeddings beats smell detectors by up to 28%","feed_subtitle":"On four common smells, the hybrid model raises MCC by 6 to 28 points over prior state of the art.","key_machinery":"The central machinery is the ensemble itself: a statistical-semantic module that turns learned code embeddings into smell-specific feature vectors through a convolutional and bidirectional long short-term memory adapter, and a structural module that learns non-linear combinations of source-code metrics through a single adaptive-layer deep network. The two output vectors are concatenated and passed to an imbalanced deep classifier whose loss weights the minority (smelly) class by a sensitivity factor $\\beta$. This design is what lets the paper attribute gains to the fusion: the semantic module alone is the earlier DeepSmells baseline, so adding the structural module isolates the contribution of design-related metrics.","core_discovery":"On the paper's own terms, the discovery is that a two-tier ensemble of code-representation features and metric features is a more powerful detector for four widely studied code smells than either feature family alone. The semantic tier encodes a snippet with one of several pre-trained models (token indexing, CodeBERT, CuBERT, or code2vec) and adapts the embedding with convolutional and bidirectional long short-term memory layers drawn from the earlier DeepSmells model; the structural tier feeds 44 class-level or 26 method-level metrics into a deep network with a single adaptive layer. Their concatenated output is classified by a deep network trained with a sensitivity-weighted binary cross-entropy loss to handle the severe class imbalance of smelly versus clean code. The paper reports that the full model beats the ML_CuBERT state of the art by 5.98% to 28.26% in MCC depending on the smell, and also surpasses the best autoencoder baselines and classical machine-learning classifiers trained on metrics alone.","pith_inferences":["Extending beyond the paper, the per-smell dependence on embedding choice suggests that a single global encoder may be the wrong unit of design; a future model could learn to route or weight multiple embeddings per smell rather than selecting one.","The paper reports random 80/20 splits; a natural stronger test is a project-level split of the MLCQ data, keeping all snippets from the same repository on one side. If the fusion gains shrink when the model must generalize to unseen projects, the practical claim would need qualification.","The authors do not test whether the reported benefits extend to other smells or languages; a direct extension would apply the same two-module design to additional design smells such as refused parent request or to Python code, where pre-trained embeddings and metric tools differ.","The hyperparameter search (loss weight, kernel size, classifier width) is reported per smell; a reader who wants to use the model should verify that those settings were chosen on validation folds before the test evaluation, because the size of the claimed improvement depends on it."],"forward_implications":["If the reported gains hold, code-smell detectors should treat learned code representations and software metrics as complementary inputs rather than competing approaches.","The per-smell best embedding pattern (token indexing for Long Method and God Class, CuBERT for Feature Envy, code2vec for Data Class) gives practitioners a concrete way to choose an encoder for a given smell.","The structural module alone outperformed seven classical machine-learning classifiers on metrics, so a metrics-only deep network may be a strong fallback when code-embedding models are too expensive.","The sensitivity-weighted loss offers a direct lever for trading precision against recall on highly imbalanced smell data, which matters for real codebases where smelly snippets are rare."],"supporting_citations":[{"why":"Supplies the DeepSmells semantic adapter and token-indexing baseline that EnseSmells extends with a structural module.","marker":"Ho et al., 2023"},{"why":"Provides the ML_CuBERT state-of-the-art baseline whose MCC results EnseSmells claims to surpass by 5.98% to 28.26%.","marker":"Kovačević et al., 2022"},{"why":"Provides the three autoencoder baselines (AE-Dense, AE-CNN, AE-LSTM) used as comparison systems.","marker":"Sharma et al., 2021"},{"why":"Provides the MLCQ dataset of Java code snippets with human labels for the four studied smells.","marker":"Madeyski and Lewowski, 2020"},{"why":"Supplies the code2vec embedding method used in the statistical-semantic module.","marker":"Alon et al., 2019"},{"why":"Supplies the CodeBERT embedding method for encoding code snippets in the semantic tier.","marker":"Feng et al., 2020"},{"why":"Supplies the CuBERT embedding method and its Java tokenizer used for method- and class-level representations.","marker":"Kanade et al., 2020"},{"why":"Provides the metric taxonomy and classical machine-learning baselines that the structural module is compared against.","marker":"Arcelli Fontana et al., 2016"}],"fun_headline_variants":["Hybrid features beat smell detectors by up to 28%","Metrics plus code embeddings boost smell detection up to 28%","Ensemble of code embeddings and metrics beats smell detectors by 28%","Two-tier ensemble sniffs out code smells up to 28% better","Code smell detection: hybrid features gain up to 28%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-smell hyperparameters (the sensitivity weight, kernel size, and classifier width) were selected on training and validation folds before the test set was used, so the reported margins over the baselines are honest estimates of generalization rather than optimistic in-sample tuning.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid features beat smell detectors by up to 28%","Metrics plus code embeddings boost smell detection up to 28%","Ensemble of code embeddings and metrics beats smell detectors by 28%","Two-tier ensemble sniffs out code smells up to 28% better","Code smell detection: hybrid features gain up to 28%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001364,"raw_usage":{"total_tokens":5537,"prompt_tokens":952,"completion_tokens":4585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":4495}},"tokens_in":568,"tokens_out":4585,"duration_ms":33344,"temperature":1.0,"reasoning_tokens":4495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T20:35:16.253107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would rerun the same configuration search with a strict three-way split, choosing the best settings on validation data only and reporting test performance once, on held-out projects from the MLCQ dataset; if the fusion model no longer beats ML_CuBERT by the reported margins, the central claim is refuted.","supporting_citations":[],"review_version":1}