{"id":"1f012674-8e1f-4131-9269-c3cacc63de45","arxiv_id":"2506.17627","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CodeMorph perturbs source code with 26 semantic-preserving transformations selected by an evolutionary algorithm, reducing a code LLM's completion accuracy on contaminated benchmarks by an average of 24.67%.","lead":"The paper presents CodeMorph, a method that uses LLMs to apply 26 code transformations and a selection algorithm called PESO to create modified versions of benchmark code, reducing data leakage in code LLM evaluation. CodeMorph lowered code-completion accuracy of the tested model by an average of 24.67% across five languages, indicating the model was less able to rely on memorized training examples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy-drop evidence does not isolate decontamination from perturbation-induced difficulty; without a non-contaminated control model, the 24.67% drop does not by itself establish leakage mitigation.","rationale":"The reader's weakest assumption concerned whether an accuracy drop on semantically-equivalent perturbed code is a valid indicator of leakage mitigation, and whether the similarity score is circular. My analysis agrees that the similarity-score circularity is real, but I judge the more load-bearing issue to be the missing non-contaminated control model. The paper's own evidence is consistent with two explanations: (1) StarCoder memorized the original The Stack code and loses that memorization advantage after perturbation, or (2) perturbation makes code completion harder in general, independent of contamination. The paper only tests StarCoder, which is contaminated by construction, so there is no baseline for how much accuracy drop a clean model would exhibit on the same perturbed tasks. Without that baseline, the headline number 24.67% cannot be attributed specifically to decontamination. The human semantic-equivalence audit, if taken at face value, rules out the alternative that perturbed code is simply semantically different; it does not rule out difficulty effects from syntactic restructuring. The PESO comparison is also circular because ss is both the fitness function and the reported outcome metric, so it cannot validate ss as a proxy for decontamination; however, this is a secondary claim about PESO's effectiveness, whereas the central contribution claims leakage mitigation. The proposed control-model experiment is feasible, directly tests the contested inference, and would either strengthen or refute the central claim. Since the paper is already CONDITIONAL and this concern reinforces that condition without necessarily requiring rejection, I recommend no change to the verdict.","tokens_in":19074,"tokens_out":2962,"duration_ms":33562,"concrete_test":"Run the identical CodeMorph perturbation and completion pipeline on a control model whose training data provably excludes The Stack (e.g., a smaller code LLM trained on a different, disclosed corpus), after matching the control model's original-code completion accuracy to StarCoder's on the same 100 samples. If the control model shows a comparable accuracy drop (within noise) on perturbed versus original code, then the 24.67% drop is attributable to perturbation-induced difficulty rather than leakage mitigation. Conversely, if the control model shows a substantially smaller drop, the decontamination claim gains direct support. Additionally, report a paired test (e.g., permutation or Wilcoxon) across the 20 samples per language for the StarCoder drop to confirm it is not sampling noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CodeMorph 'mitigates data leakage' rests on one empirical observation: after perturbation, StarCoder's code-completion accuracy drops by an average of 24.67% (§V-A, Figure 6). This is interpreted as evidence that the model can no longer rely on memorized patterns from The Stack. But the same observation would occur if semantically-preserving perturbations simply make completion harder for reasons unrelated to contamination: renamed identifiers, extracted functions, inserted control flow, and changed statement structure can alter the completion surface even for a model with zero exposure to the original code. The paper provides no control condition that separates these two explanations. StarCoder was chosen because The Stack is its training data (§II), and all main experiments use only StarCoder; the Limitation section explicitly states that no other LLM with a known corresponding training dataset was evaluated. Consequently, we cannot tell whether the drop reflects reduced memorization of contaminated code or merely increased task difficulty on more complex code variants. The PESO results (§V-B, Table IV) are also not direct evidence of decontamination: the similarity score ss is the objective that PESO optimizes (Algorithm 1, Equation 1), so showing that PESO lowers ss relative to random selection demonstrates optimization success, not that lower ss corresponds to less leakage. The human verification in §III-A2 and §IV-B mitigates the semantic-equivalence confound, but it does not address the missing-control confound. The small sample size (20 code samples per language, no error bars or significance tests) further weakens the inference, but the decisive gap is the absence of a model not trained on The Stack.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CodeMorph is a pipeline for semantically-preserving code perturbation built on 26 transformation methods executed by an LLM, with compile checks and three LLM voters for semantic verification, plus PESO, a genetic/Boltzmann-selection algorithm that iteratively chooses perturbation categories to reduce a weighted surface/semantic similarity score. The evaluation uses StarCoder on samples from The Stack across five languages, reporting that code-completion accuracy drops by an average of 24.67% after perturbation (45% for Python single-line completion), and that PESO achieves a 7.01% lower similarity score than random perturbation. The manuscript concludes that CodeMorph effectively mitigates data leakage and that PESO optimizes perturbation combinations.","tokens_in":19364,"tokens_out":4618,"duration_ms":47426,"significance":"The manuscript has real strengths: a public artifact, a deliberately contaminated evaluation setup using StarCoder trained on The Stack, 26 documented transformations, multi-language compilation checks, human audit of perturbed code, and the use of three voter LLMs. If the central claims were substantiated, CodeMorph would be a practical tool for reducing memorization effects in code LLM evaluation. However, the two headline results are indirect: the accuracy drop does not isolate decontamination from perturbation-induced difficulty, and the similarity metric is the very objective PESO optimizes. The empirical base is small (20 samples per language, no significance tests), and the paper's own limitation section acknowledges that only one model/training-data pair was evaluated. These issues are load-bearing but addressable with additional controls and careful re-framing; the tool contribution itself is plausible.","major_comments":[{"comment":"GetSimilarity(c_ori, c_pert) should be GetSimilarity(c_ori, c_inter). As written, the algorithm computes the similarity of the previously accepted code rather than the candidate produced in the current iteration, so the acceptance test and the optimization gain do not evaluate the perturbation that was just applied. This contradicts the prose in §III-B2, which states that the similarity score is computed for the intermediate perturbed code after verification. Unless the algorithm is corrected and the experiments rerun, the PESO results in §V-B are not supported.","section":"Algorithm 1, line 6"},{"comment":"The overall similarity score ss is exactly the objective that PESO optimizes in Algorithm 1 through Equation (1). Reporting that PESO lowers ss relative to random perturbation is an evaluation by construction: it shows that selection optimizes its own fitness function. The conclusion in §V-B that \"code similarity can effectively indicate decontamination\" requires an external validation of ss against a direct contamination measurement, which the paper does not provide. The non-circular evidence for PESO is therefore limited to the accuracy comparison in Table III.","section":"§III-B and Table IV"},{"comment":"The 24.67% average accuracy drop is interpreted as evidence of reduced data leakage, but no control condition separates decontamination from perturbation-induced difficulty. Semantically-preserving perturbations can still change the completion surface (renamed identifiers, extracted functions, inserted control flow, changed statement structure) and lower accuracy for a model with zero exposure to the original code. The paper itself notes in the discussion of Figure 8 that perturbation can sometimes increase accuracy by introducing hints, which shows that perturbations change task difficulty independently of contamination. The Limitation section states that no other LLM with a known corresponding training dataset was evaluated, so the observed drop is consistent with both increased difficulty and reduced leakage. A clean-model control, a direct membership test, or an ablation that holds task difficulty fixed is needed to support the decontamination conclusion.","section":"§V-A, Figure 6"},{"comment":"With 20 samples per language and no error bars, confidence intervals, or significance tests, the aggregate claims (24.67%, 7.01%, 15%) are not shown to be statistically robust. Table IV contains several code IDs where random perturbation attains a lower similarity score than PESO (e.g., Python IDs 3, 6, 13, 19), so the comparison is noisy and the reported averages may be dominated by a few samples. A paired statistical test and standard dispersion measures should be reported for both the accuracy and similarity comparisons.","section":"§IV-A and Tables III–IV"}],"minor_comments":[{"comment":"The \"15% average / 25% maximum accuracy reduction\" in the abstract refers to PESO relative to random perturbation, while the abstract's phrasing makes it appear to be relative to the original code; please clarify the baseline explicitly in both places.","section":"Abstract and §V-B"},{"comment":"Table I contains typos and inconsistent capitalization, including \"Add conditon\", \"Divde the long expression\", and inconsistent renderings of \"Div if else\" versus \"Div composed if\".","section":"Table I"},{"comment":"The loop header \"for iter←1 to maxIter and ss<ss_threshold do\" conflates a for-loop with a while-condition; a while-loop with a guard on ss would be clearer and would avoid the separate iter←iter+1 statement.","section":"Algorithm 1"},{"comment":"The arithmetic leading to 600 tasks should be stated more explicitly: 100 original samples plus 100 perturbed samples, each evaluated at 1-, 3-, and 5-line completion, gives 600 tasks; the current text says only \"300 tasks for original and 300 for perturbed\" without clearly explaining the factor of three.","section":"§IV-A2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds something real. CodeMorph is an LLM-driven, multi-language code perturbation pipeline with 26 semantics-preserving transforms, compile checks, three LLM voters, and a human audit pass. That last piece matters: the authors manually verified all perturbed code, and they preserve cross-file dependencies, which existing tools like CodART and rope don't handle. The artifact is linked on GitHub, which is a plus, though I haven't run it. The PESO genetic algorithm is a reasonable way to select perturbation methods per iteration, and the seven newly proposed transforms are catalogued clearly. For benchmark builders, this is a genuinely useful engineering contribution.\n\nThe soft spot is load-bearing: the paper concludes that a 24.67% average drop in StarCoder's code-completion accuracy means leakage was mitigated. But every experiment uses StarCoder, which was trained on The Stack—the same data the samples come from. A semantically-equivalent but structurally altered program can easily be harder to complete even for a model that never saw the original code: renamed identifiers, extracted functions, and inserted control flow all change the completion surface. Without a control model not trained on The Stack, or a direct contamination measure (e.g., membership tests before and after perturbation), the accuracy drop alone doesn't separate decontamination from perturbation-induced difficulty. The paper actually shows in Figure 8 that perturbation sometimes makes tasks easier, which reinforces that the completion surface changes non-monotonically. The authors admit in the Limitations that only StarCoder was tested, so the gap is known but not addressed.\n\nThe PESO evaluation is also partly circular: the similarity score ss is the objective PESO optimizes, so Table IV showing lower ss is optimization success, not decontamination. The accuracy comparison in Table III is less circular but inherits the same control problem. Minor issue: only 20 samples per language, no error bars or significance tests, though the API cost explanation is plausible.\n\nRecommendation: the paper is honest, well-written, and addresses a real problem, but the central empirical claim is not supported. I'd send it to review with a request for a non-contaminated control model and a re-framing of PESO's contribution as \"producing less similar code\" rather than \"mitigating leakage.\" The pipeline itself is worth the referee time.","headline":"A well-engineered multi-language code perturbation pipeline, but the missing control model keeps the leakage-mitigation claim from sticking.","tokens_in":19914,"tokens_out":5625,"would_cite":false,"duration_ms":51162,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeMorph claims that rewriting code into semantically equivalent but surface-different variants cuts a contaminated model's code-completion accuracy by an average of 24.67% across five programming languages.","keywords":["data leakage","data contamination","code LLM evaluation","code perturbation","semantic-preserving transformation","code completion","genetic algorithm","benchmark decontamination"],"falsifier":"Take the perturbed samples and run both original and rewritten code against a common input suite, keeping only pairs with identical outputs, then re-measure StarCoder's accuracy drop on those execution-verified pairs; separately, run CodeMorph on code written after StarCoder's training cutoff. If the drop largely vanishes on verified pairs, or if never-seen code drops by roughly the same 24.67%, the reported drop tracks rewriting difficulty rather than the loss of memorized contamination.","tokens_in":18889,"feed_emoji":"🔄","tokens_out":15726,"duration_ms":140855,"temperature":0.7,"pith_summary":"CodeMorph targets a specific failure in how code models are evaluated: if a model trained on public code is tested on code it already saw, its score measures memorization, not ability. The paper proposes to fix this by rewriting test code into new, semantically equivalent forms, so the model faces the same functionality in a surface it has not memorized. Its central claim is that a rewrite pipeline built on 26 semantic-preserving transformations, executed by an LLM and verified by compilation plus three voter LLMs, dropped StarCoder's code-completion accuracy by an average of 24.67% across five programming languages, with Python single-line completion down 45%. A companion selector, PESO, chooses rewrite methods by tracking how much each lowers a combined similarity score; the paper reports that PESO-selected rewrites score 7.01% lower in similarity on average than randomly chosen rewrites. If the claim holds, contaminated benchmarks can be decontaminated by rewriting rather than by waiting for new data.","feed_headline":"Rewritten code cuts LLM memorization scores by 24.7%","feed_subtitle":"CodeMorph rewrites leaked benchmark code into new equivalents, showing how much model accuracy was surface memorization.","key_machinery":"The machinery that carries the argument is a feedback loop between two components. The perturbation half treats the rewrite as an LLM task: each of the 26 semantic-preserving transformations is rendered as a prompt template covering role, task, format, and code, and the generated output must pass a verification gate of successful compilation plus a semantic-equivalence vote in which at least two of three voter LLMs must answer True. The selection half, PESO, closes the loop: it measures every accepted rewrite with the overall similarity $ss=\\mu s_1+\\nu s_2$ (Levenshtein surface similarity $s_1$ and JPlag semantic similarity $s_2$, with default weights $\\mu=\\nu=0.5$), records the gain over the running minimum $mss$ as an optimization gain $og_i$ for the method's category, and chooses the next category with Boltzmann selection $P_i=\\exp(og_i/T)/\\sum_j \\exp(og_j/T)$ at temperature $T=2$. Because every category keeps a nonzero selection probability, the loop explores broadly while favoring the categories that recently produced the largest similarity reductions, yielding an ordered, code-specific sequence of rewrites whose cumulative effect is the reported accuracy drop.","core_discovery":"On its own terms, the paper claims that data leakage in code-LLM assessment can be measured and mitigated through semantic-preserving perturbation. CodeMorph applies 26 transformations—renaming, junk-code insertion, statement reordering, condition and loop reshaping, boolean-logic and arithmetic rewriting, and code extraction—by prompting a general-purpose LLM, and keeps a rewrite only if it compiles in a multi-language execution engine and at least two of three voter LLMs judge it semantically equivalent to the original, with human audit as a final gate. On 100 code samples taken from The Stack, StarCoder's accuracy on code-completion tasks fell below its original-code accuracy by an average of 24.67%, reaching 45% for single-line Python tasks, which the paper reads as the model losing its memorized advantage on the rewritten surface. PESO gives each of the six method categories an optimization gain equal to how much a rewrite lowered the overall similarity score $ss = \\mu s_1 + \\nu s_2$ below the running minimum, then selects the next category by Boltzmann selection over those gains; the paper reports that PESO-perturbed code reaches similarity scores 7.01% lower on average than randomly perturbed code (peak 42.86%) and completion accuracy 15% lower on average (peak 25%). It concludes that CodeMorph effectively mitigates data contamination while PESO optimizes perturbation combinations for code.","pith_inferences":["The same pipeline is effectively a contamination probe: if the rewrites truly preserve semantics, the accuracy gap between original and perturbed code quantifies how much any model leaned on surface form, so the method could estimate leakage for model–dataset pairs whose training data is not public.","The cleanest control the paper does not run is code the model has never seen, such as samples written after StarCoder's training cutoff; if the accuracy drop on such code matches 24.67%, the metric reflects added task difficulty, not neutralized memorization.","Because PESO's objective (the similarity score) doubles as its success metric, an independent outcome measure—for instance unit-test pass rates on samples where execution-based equivalence can be checked mechanically—would separate 'more heavily rewritten' from 'genuinely less contaminated.'","The verification gate's cost and small benchmark (100 samples, one model, one task type) are the practical bottleneck; replacing LLM voters with execution-based equivalence where tests exist would let the approach scale and make the equivalence guarantee machine-checkable rather than probabilistic."],"forward_implications":["Contaminated corpora can be recycled into evaluation sets: the paper claims that rewriting The Stack's own code makes StarCoder's memorized advantage measurable and largely removable, so decontamination no longer requires waiting for brand-new data.","On the paper's account, the 24.67% average accuracy drop (45% for single-line Python) is a lower bound on how much of StarCoder's completion score depended on surface memorization rather than generalized ability.","Because the transformations are prompt-driven and gated by a multilingual compile-and-vote check, the method claims to extend to additional languages and to repository-level code whose cross-file dependencies are left intact.","PESO's 7.01% average similarity improvement over random selection indicates that choosing perturbation methods by similarity feedback beats blind combinations, while the Boltzmann temperature keeps every category in play and deliberately sacrifices some optimality for exploration."],"supporting_citations":[{"why":"StarCoder, the model whose contamination is being measured; its public training data grounds the premise that evaluation code appears in training.","marker":"[5]"},{"why":"The Stack, the contaminated training corpus that supplies the test samples and establishes why the model may have memorized them.","marker":"[6]"},{"why":"Prior work establishing that data perturbation can mitigate contamination, the starting point CodeMorph extends.","marker":"[12]"},{"why":"The contamination-quantification study whose surface-plus-semantic similarity approach inspires PESO's similarity score.","marker":"[28]"},{"why":"Demonstrates that widely used benchmarks such as HumanEval and MBPP leak into training data, motivating the need for perturbation-based decontamination.","marker":"[41]"},{"why":"Supplies the Boltzmann selection rule that PESO uses to choose perturbation method categories.","marker":"[22]"},{"why":"Defines the Levenshtein edit-distance metric that supplies the surface-level half of the similarity score.","marker":"[23]"},{"why":"JPlag, the multi-language plagiarism detector that supplies the semantic-similarity half of the score.","marker":"[9]"}],"fun_headline_variants":["Rewriting code exposes LLM memorization, drops accuracy 24.7%","CodeMorph cuts benchmark leakage: accuracy drops 24.67% on rewritten code","Genetic algorithm optimizes code rewrites to beat LLM memorization","Semantic-preserving rewrites cut LLM code completion by up to 45%","CodeMorph: 26 transformations reduce LLM evaluation scores by 24.7%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that lower completion accuracy on code that compiles and is judged semantically equivalent by LLM voters means the model's memorized training data was neutralized rather than the rewritten task simply becoming harder, and that the paper's similarity score is a valid proxy for decontamination even though PESO optimizes for that same score.","fun_headline_variants_meta":{"raw":{"variants":["Rewriting code exposes LLM memorization, drops accuracy 24.7%","CodeMorph cuts benchmark leakage: accuracy drops 24.67% on rewritten code","Genetic algorithm optimizes code rewrites to beat LLM memorization","Semantic-preserving rewrites cut LLM code completion by up to 45%","CodeMorph: 26 transformations reduce LLM evaluation scores by 24.7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000937,"raw_usage":{"total_tokens":4096,"prompt_tokens":1126,"completion_tokens":2970,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":742,"completion_tokens_details":{"reasoning_tokens":2863}},"tokens_in":742,"tokens_out":2970,"duration_ms":21396,"temperature":1.0,"reasoning_tokens":2863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:04:45.306904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the perturbed samples and run both original and rewritten code against a common input suite, keeping only pairs with identical outputs, then re-measure StarCoder's accuracy drop on those execution-verified pairs; separately, run CodeMorph on code written after StarCoder's training cutoff. If the drop largely vanishes on verified pairs, or if never-seen code drops by roughly the same 24.67%, the reported drop tracks rewriting difficulty rather than the loss of memorized contamination.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"StarCoder, the model whose contamination is being measured; its public training data grounds the premise that evaluation code appears in training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Stack, the contaminated training corpus that supplies the test samples and establishes why the model may have memorized them."},{"cited_title":"Binary codes capable of correcting deletions, inser- tions, and reversals,","cited_arxiv_id":null,"evidence_quote":"Defines the Levenshtein edit-distance metric that supplies the surface-level half of the similarity score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"JPlag, the multi-language plagiarism detector that supplies the semantic-similarity half of the score."}],"review_version":2}