{"id":"14c4669b-1621-4d42-8849-c91db9588045","arxiv_id":"2506.07594","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"In a head-to-head test, Gemini detected test smells most consistently across Python and Java, but LLM-based refactoring frequently reduced test coverage and could introduce new smells.","lead":"This study compared how well three large language models (GPT-4, LLaMA 3 70B, and Gemini 1.5 Pro) can detect and fix test smells in Python and Java code. Gemini matched the reference tools most often, but all models sometimes introduced new problems and often reduced test coverage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Detection 'accuracy' is agreement with PyNose/TsDetect, not ground truth; using the same tools for sampling, prompt selection, and evaluation invalidates the headline ranking.","rationale":"The reader's weakest assumption is exactly the load-bearing one: the study's 'accuracy' is tool agreement, not ground truth. I agree. The paper explicitly calls PyNose and TsDetect 'oracles' for sampling, prompt selection, and final evaluation (Sections III-A and III-C), and one of the tools is co-authored by a study author, so there is no independent check on the labels. Because all positive/negative instances are defined by these tools, the reported detection rates and the refactoring removal metrics in Table V inherit any tool error. The Threats to Validity section acknowledges reliance on PyNose but frames it as a definitional threat rather than a threat to the central accuracy claim, so it does not defuse the concern. A second, independent problem is that the abstract's coverage claim is contradicted by the paper's own results: Observation 4 states refactoring led to 'inconsistent and often negative impacts on code coverage' with Gemini and LLaMA frequently increasing uncovered statements more than GPT-4, and Figures 5-7 show all models reducing Python coverage. Even if the oracle concern were resolved, this internal inconsistency would require at least a revised abstract. The study does have strengths: a large dataset, a documented pipeline, and an artifact link. But the central contributions as stated are not supported. The reader's REJECT verdict is appropriate, so no change is needed.","tokens_in":18244,"tokens_out":5735,"duration_ms":62084,"concrete_test":"Draw a stratified random sample of 100 positive and 100 negative test methods per language from the released dataset, balanced across the 15 smell types. Have two annotators, blind to tool output and model identity, independently label each snippet using the smell definitions from Section III-C, and measure inter-rater agreement. Recompute the three LLMs' detection accuracy and F1 against these human labels; if Gemini's ranking versus GPT-4/LLaMA changes, or if tool labels are contradicted in more than about 10% of cases, the abstract's detection accuracy claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central detection claim ('Gemini achieved the highest detection accuracy, LLaMA lowest') is an agreement rate with the outputs of PyNose and TsDetect, not with any independent ground truth. The same two tools (a) label the dataset from which positive/negative samples are drawn (Section III-A states 'the test smell detection tools used to validate the LLM results, specifically PyNose and tsDetect, served as oracles'), (b) are the comparison target during the five-iteration prompt selection phase (Section III-C: 'we compared the LLMs' outputs against the results produced by PyNose and TsDetect, which served as oracle tools'), and (c) are the reference for the final reported 'accuracy' numbers. Any systematic bias or error in these rule-based/ML detectors therefore propagates unchanged into every headline number. The paper's own Threats to Validity (Section VI) acknowledges reliance on PyNose but frames it as a definitional issue, not as a threat to the 'accuracy' label; and PyNose is co-authored by a study author, so independence of the oracle is not established. Additionally, the abstract's coverage claim is internally contradicted by Observation 4 and Figures 4-7, where Gemini and LLaMA frequently increase uncovered statements more than GPT-4 and all models reduce Python coverage. The core empirical ranking is therefore unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an empirical study comparing GPT-4-Turbo, LLaMA 3 70B, and Gemini 1.5 Pro on detecting and refactoring test smells in Python and Java. The authors use PyNose and TsDetect to label candidate smells, engineer five prompt templates per language, and report per-smell detection percentages as well as refactoring outcomes such as smell removal, collateral introduction, and coverage changes. The abstract claims Gemini achieved the highest detection accuracy (74.35% Python, 80.32% Java) and that Gemini improved test coverage, while GPT-4 and LLaMA often reduced it. The body of the paper reports Observation 4, which states that coverage impacts were inconsistent and often negative across both languages, with Gemini and LLaMA frequently introducing more uncovered statements than GPT-4.","tokens_in":18619,"tokens_out":9157,"duration_ms":107548,"significance":"The problem is relevant and underexplored, and the study has strengths: it is large in scope (more than 1,400 detection tasks and 450 refactoring tasks over 15 smell types and two languages), it uses a fixed temperature, and it makes artifacts and scripts publicly available. As a comparative analysis of how LLMs agree with established detectors, the study could be a useful empirical data point. However, the headline 'accuracy' numbers are agreement rates with the same tools used for labeling and prompt selection, and the abstract's coverage claim is contradicted by the paper's own Observation 4. The contribution is therefore not yet established as stated; reframing and additional validation are needed.","major_comments":[{"comment":"The reported detection 'accuracy' (abstract: 74.35% Python, 80.32% Java for Gemini) is an agreement rate with PyNose and TsDetect outputs, not accuracy against independent ground truth. The same tools label the sampled data (Section III-A), are the comparison target during prompt selection (Section III-C), and are the reference for the final reported numbers. One of these tools, PyNose [23], is co-authored by a study author, and no human-annotated subset or independent detector is used. Any systematic error in these oracles therefore propagates into every headline number, so the current evidence supports only an agreement ranking. Please either add a manually validated sample to estimate true detection accuracy, or consistently reframe all 'accuracy' claims as 'agreement with PyNose/TsDetect' in the abstract, results, and conclusions.","section":"Sections III-A and III-C, Table IV"},{"comment":"The prompt-selection and final-evaluation phases are not separated by a held-out split. The paper states that the LLM outputs were compared against the oracle tools during each iteration of prompt selection, and that after five iterations the best prompt was applied to the 'entire sampled test smell dataset' and compared again. If the final evaluation set overlaps with the set used to select prompts, the reported percentages are optimistically biased and do not measure generalization. Please specify the exact split between prompt-selection and evaluation data, or re-run the evaluation on a disjoint held-out sample.","section":"Section III-C"},{"comment":"The abstract's claim that 'Gemini also improved test coverage, unlike GPT-4 and LLaMA, which often reduced it' is contradicted by the paper's own results. Observation 4 states that refactoring led to 'inconsistent and often negative impacts on code coverage across both Java and Python projects, with Gemini and LLaMA frequently introducing more uncovered statements than GPT-4,' and Figures 6 and 7 show all models increasing uncovered statements, with Gemini the worst in Python. This internal contradiction changes the paper's headline message; the abstract, Section IV, and Section V must be aligned with Observation 4.","section":"Abstract vs. Section IV-B, Observation 4, Figures 4-7"},{"comment":"The Threats to Validity section acknowledges reliance on PyNose but frames it mainly as a definitional issue rather than as a threat to the 'accuracy' construct. The manuscript does not quantify oracle misclassification risk, and the same oracle is used to measure refactoring success (Section III-D, Table V). Since PyNose's published precision and recall are for its own detection task, not for the LLM comparison, the construct validity of both RQ1 and RQ2 depends on an unvalidated assumption. Please add an explicit oracle-error analysis (e.g., manual inspection of a random sample) or weaken the conclusions to state that the results reflect agreement with existing tools.","section":"Section VI"}],"minor_comments":[{"comment":"The index terms are placeholder template text ('Article submission, IEEE, IEEEtran, LATEX, paper, template, typesetting'); replace them with actual keywords describing test smells, LLMs, and software testing.","section":"Index Terms"},{"comment":"The footnote about AI chatbots and history questions is irrelevant to the model-selection rationale; replace it with a software-engineering benchmark or remove it.","section":"Section I, footnote 4"},{"comment":"The per-smell percentages are labeled 'accuracy,' but the class balance of the samples is not reported; please clarify whether these are raw agreement rates, balanced accuracy, or another metric, and report sample sizes and confidence intervals.","section":"Section IV-A, Table IV"},{"comment":"The text says Gemini-1.5 Pro is a '130B model,' but the parameter count of Gemini 1.5 Pro is not publicly documented; either cite a reliable source or remove the specific number.","section":"Section III-B"},{"comment":"The table is said to contain only statistically significant Wilcoxon results, but no p-values, effect sizes, or multiple-comparison corrections are reported; please add them or explicitly describe the table as descriptive.","section":"Table V"},{"comment":"Figure 3, the correlation matrix of test smells, is not explained in the text; please add a paragraph describing what the correlations mean and how they support the refactoring analysis.","section":"Section IV-B, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the circular use of the same oracle tools for labeling, prompt selection, and evaluation; a revision should prioritize a manually annotated validation sample or a full reframing to agreement analysis. I also note that the artifact repository URL contains 'ICSE26,' so the editor may wish to verify that submission to this journal does not conflict with the policies of any other venue, and that the co-authorship of PyNose by a study author is disclosed or addressed in the revised manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper is a serious empirical effort—the first side-by-side test of GPT-4-Turbo, LLaMA 3 70B, and Gemini-1.5 Pro for both detecting and refactoring 15 test smell types across Python and Java. The per-smell breakdowns are useful, and the analysis of collateral smell introduction and coverage impact is worth having. I'd cite the dataset.\n\nThe soft spots are real but fixable. Most importantly, 'detection accuracy' is agreement with PyNose and TsDetect, not true accuracy. Those same tools label the sample, guide prompt selection, and serve as the final oracle. So the ranking tells you which LLM best mimics these tools, not which finds smells in the wild. The tools have published precision/recall, so it's not worthless, but the abstract has to say 'agreement' not 'accuracy.' Having a study author co-author PyNose doesn't invalidate the work, but it underlines the need for an independent oracle.\n\nThe bigger problem is the abstract's claim that 'Gemini also improved test coverage, unlike GPT-4 and LLaMA.' The paper's own Observation 4 and Figures 4-7 show Gemini and LLaMA frequently increase uncovered statements more than GPT-4 in Java, and all models degrade Python coverage. The abstract cherry-picks the Java statement-coverage gains. That's an internal contradiction that has to be corrected.\n\nOne more thing: prompt templates were selected by iterating on a validation sample against the same oracle, with no held-out split reported. That's a moderate overfitting risk. It would be easy to address by reporting the selection procedure transparently or validating on a separate sample.\n\nWho's this for? Researchers building LLM-based test maintenance tools and anyone evaluating LLMs against static analyzers. The empirical core is reusable, and the flaws are not fatal—they're about framing and internal consistency. I'd send it to peer review with major revisions rather than desk reject. The reviewer's job would be to force the authors to reframe the detection results as tool-agreement rates, fix the coverage summary, and disclose the oracle dependency more prominently.","headline":"Useful empirical comparison, but the 'accuracy' numbers measure agreement with PyNose/TsDetect rather than ground truth, and the abstract's coverage claim contradicts the paper's own results.","tokens_in":19049,"tokens_out":3176,"would_cite":true,"duration_ms":31661,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims Gemini-1.5 Pro is the most accurate LLM for test smell detection and the only one that improves coverage when refactoring.","keywords":["test smells","large language models","LLM refactoring","GPT-4","Gemini","LLaMA","PyNose","TsDetect"],"falsifier":"Re-label a sample of the same test cases with human experts (or a second, independent detection tool) and recompute each LLM's accuracy against that gold standard; if agreement with human labels is much lower than the reported 74.35% and 80.32%, the claimed Gemini advantage is an artifact of oracle agreement rather than true detection skill.","tokens_in":18068,"feed_emoji":"🤖","tokens_out":5235,"duration_ms":56294,"temperature":0.7,"pith_summary":"This paper asks whether large language models can do more than flag test smells—whether they can also refactor the offending test code—and tests three models (GPT-4-Turbo, LLaMA 3 70B, Gemini-1.5 Pro) on Python and Java test suites. The headline finding is that Gemini detects test smells most accurately, with 74.35% accuracy on Python and 80.32% on Java, while LLaMA trails; Gemini also tends to improve statement coverage during refactoring, whereas GPT-4 and LLaMA often reduce it. The study is notable for pairing traditional static detectors with LLM-driven transformation, and for measuring side effects such as newly introduced smells and coverage changes. If the results hold, they make a concrete case for where LLM-based test maintenance tools are ready to help and where they still need guardrails.","feed_headline":"Gemini leads LLMs in detecting and refactoring test smells","feed_subtitle":"Across 1,400+ detection and 450 refactoring tasks on Python and Java tests, Gemini led on accuracy and coverage.","key_machinery":"The study's central machinery is a two-stage hybrid workflow: rule-based detectors (PyNose for Python, TsDetect for Java) supply both the labeled identification tasks and the ground-truth oracle, and prompt-engineered LLMs perform detection and refactoring. The argument is carried by the prompt templates—built from persona, few-shot, chain-of-thought, and zero-shot chain-of-thought techniques and iterated to saturation—and by the before/after comparison of test smells and coverage (using pytest-cov and JaCoCo), with statistical significance tested by the Wilcoxon signed-rank test. The 'collateral smell' concept, tracking smells unintentionally added or removed during refactoring, is what turns the evaluation from simple detection accuracy into an assessment of refactoring side effects.","core_discovery":"The paper claims that state-of-the-art LLMs can detect test smells with practically useful accuracy and can perform automated refactoring, but that effectiveness varies sharply by model, language, and smell type. Gemini-1.5 Pro is the strongest overall performer, with the highest detection accuracy (74.35% Python, 80.32% Java) and positive statement-coverage effects in Java, while GPT-4-Turbo is the most consistent for Python and LLaMA 3 70B is the weakest on both tasks. The authors further claim that refactoring frequently removes non-targeted smells as a side benefit, but also introduces new smells and, in many cases, reduces code coverage, so automated refactoring is not yet safe to run unattended.","pith_inferences":["Because the reported detection accuracy is agreement with PyNose and TsDetect rather than with a human gold standard, the true detection quality on real projects could be lower; a human-annotated benchmark would settle whether Gemini's lead is real or an artifact of tool bias.","The large between-model gaps and per-smell variation imply that 'LLMs for test refactoring' is not a single capability; a model-selection policy may need per-smell routing, since the best model for one smell can be the worst for another.","The observed coverage losses might reflect tests being deleted or assertions weakened, not just reordering; running a mutation-testing pass on refactored suites would test whether coverage loss corresponds to real fault-detection loss."],"forward_implications":["Integrating an LLM like Gemini into an IDE or CI pipeline could catch and fix test smells automatically, but only if the model version is pinned and quality-assured, since the three models differ sharply.","Refactoring one smell often removes related smells (e.g., Ignored Test almost always disappears), so teams may get cleanup beyond the explicit request.","Coverage can drop after refactoring, especially for Python and for LLaMA; a refactoring step should therefore be followed by a coverage check before acceptance.","The prompt engineering results provide a recipe: chain-of-thought with few-shot examples performed best for identification, and the winning refactoring prompts were selected by balancing direct removals against collateral smells.","For smells like Magic Number Test and Sleepy Test, models only refactor when explicitly instructed, so generic cleanup prompts will not touch them."],"supporting_citations":[{"why":"Provides PyNose, the Python test smell detector that serves as both the source of labeled detection data and the oracle for measuring LLM agreement.","marker":"[23]"},{"why":"Provides TsDetect, the Java test smell detector that serves as the analogous oracle for the Java portion of the study.","marker":"[30]"},{"why":"Supplies the machine-learning-based test smell detection methodology and the Java dataset (drawn from IDoFT) used to build the evaluation corpus.","marker":"[34]"},{"why":"Defines GPT-4-Turbo, one of the three LLMs under evaluation in both detection and refactoring tasks.","marker":"[6]"},{"why":"Defines LLaMA 3 70B, the open-source model that the study compares against the proprietary models and finds weakest overall.","marker":"[7]"},{"why":"Defines Gemini-1.5 Pro, the model that the study reports as the strongest detector and the only one that improves coverage during refactoring.","marker":"[8]"},{"why":"Provides the chain-of-thought prompting technique that appears in the best-performing identification and refactoring prompt versions.","marker":"[40]"},{"why":"Supplies the Wilcoxon signed-rank test used to determine which refactoring effects on smell counts are statistically significant.","marker":"[42]"}],"fun_headline_variants":["Gemini tops LLMs in test-smell detection and refactoring","LLMs spot test smells, but refactoring can backfire","Gemini leads LLM test-smell fix, but coverage drops","Test-smell battle: Gemini wins detection, refactoring risky","LLM refactoring: fixes smells, often hurts coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study treats PyNose and TsDetect as ground-truth oracles: LLM detection accuracy is the match rate with these tools, and refactoring is deemed successful when the tools stop flagging the smell, so if either tool mislabels code every headline number shifts.","fun_headline_variants_meta":{"raw":{"variants":["Gemini tops LLMs in test-smell detection and refactoring","LLMs spot test smells, but refactoring can backfire","Gemini leads LLM test-smell fix, but coverage drops","Test-smell battle: Gemini wins detection, refactoring risky","LLM refactoring: fixes smells, often hurts coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1294,"prompt_tokens":903,"completion_tokens":391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":300}},"tokens_in":519,"tokens_out":391,"duration_ms":4837,"temperature":1.0,"reasoning_tokens":300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:29:45.196229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label a sample of the same test cases with human experts (or a second, independent detection tool) and recompute each LLM's accuracy against that gold standard; if agreement with human labels is much lower than the reported 74.35% and 80.32%, the claimed Gemini advantage is an artifact of oracle agreement rather than true detection skill.","supporting_citations":[{"cited_title":"Machine learning-based test smell detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the machine-learning-based test smell detection methodology and the Java dataset (drawn from IDoFT) used to build the evaluation corpus."}],"review_version":1}