CodeMorph: Mitigating Data Leakage in Large Language Model Assessment

Haoyu Wang; Hongzhou Rao; Ling Xiao; Meizhen Wang; Wenjie Zhu; Yanjie Zhao

REVIEW 4 major objections 4 minor 51 references

CodeMorph: Mitigating Data Leakage in Large Language Model Assessment

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CodeMorph claims that rewriting code into semantically equivalent but surface-different variants cuts a contaminated model's code-completion accuracy by an average of 24.67% across five programming languages.

desk verdict A well-engineered multi-language code perturbation pipeline, but the missing control model keeps the leakage-mitigation claim from sticking. read the letter →

arxiv 2506.17627 v1 pith:SY4S7XMZ submitted 2025-06-21 cs.SE

Hongzhou Rao , Yanjie Zhao , Wenjie Zhu , Ling Xiao , Meizhen Wang , Haoyu Wang This is my paper · ORCID

classification cs.SE

keywords dataleakagecontaminationcodeLLMevaluationperturbationsemantic-preservingtransformationcompletiongeneticalgorithmbenchmarkdecontamination

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CodeMorph targets a specific failure in how code models are evaluated: if a model trained on public code is tested on code it already saw, its score measures memorization, not ability. The paper proposes to fix this by rewriting test code into new, semantically equivalent forms, so the model faces the same functionality in a surface it has not memorized. Its central claim is that a rewrite pipeline built on 26 semantic-preserving transformations, executed by an LLM and verified by compilation plus three voter LLMs, dropped StarCoder's code-completion accuracy by an average of 24.67% across five programming languages, with Python single-line completion down 45%. A companion selector, PESO, chooses rewrite methods by tracking how much each lowers a combined similarity score; the paper reports that PESO-selected rewrites score 7.01% lower in similarity on average than randomly chosen rewrites. If the claim holds, contaminated benchmarks can be decontaminated by rewriting rather than by waiting for new data.

What carries the argument

The machinery that carries the argument is a feedback loop between two components. The perturbation half treats the rewrite as an LLM task: each of the 26 semantic-preserving transformations is rendered as a prompt template covering role, task, format, and code, and the generated output must pass a verification gate of successful compilation plus a semantic-equivalence vote in which at least two of three voter LLMs must answer True. The selection half, PESO, closes the loop: it measures every accepted rewrite with the overall similarity $ss=\mu s_1+\nu s_2$ (Levenshtein surface similarity $s_1$ and JPlag semantic similarity $s_2$, with default weights $\mu=\nu=0.5$), records the gain over the running minimum $mss$ as an optimization gain $og_i$ for the method's category, and chooses the next category with Boltzmann selection $P_i=\exp(og_i/T)/\sum_j \exp(og_j/T)$ at temperature $T=2$. Because every category keeps a nonzero selection probability, the loop explores broadly while favoring the categories that recently produced the largest similarity reductions, yielding an ordered, code-specific sequence of rewrites whose cumulative effect is the reported accuracy drop.

What would settle it

Take the perturbed samples and run both original and rewritten code against a common input suite, keeping only pairs with identical outputs, then re-measure StarCoder's accuracy drop on those execution-verified pairs; separately, run CodeMorph on code written after StarCoder's training cutoff. If the drop largely vanishes on verified pairs, or if never-seen code drops by roughly the same 24.67%, the reported drop tracks rewriting difficulty rather than the loss of memorized contamination.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that data leakage in code-LLM assessment can be measured and mitigated through semantic-preserving perturbation. CodeMorph applies 26 transformations—renaming, junk-code insertion, statement reordering, condition and loop reshaping, boolean-logic and arithmetic rewriting, and code extraction—by prompting a general-purpose LLM, and keeps a rewrite only if it compiles in a multi-language execution engine and at least two of three voter LLMs judge it semantically equivalent to the original, with human audit as a final gate. On 100 code samples taken from The Stack, StarCoder's accuracy on code-completion tasks fell below its original-code accuracy by an average of 24.67%, reaching 45% for single-line Python tasks, which the paper reads as the model losing its memorized advantage on the rewritten surface. PESO gives each of the six method categories an optimization gain equal to how much a rewrite lowered the overall similarity score $ss = \mu s_1 + \nu s_2$ below the running minimum, then selects the next category by Boltzmann selection over those gains; the paper reports that PESO-perturbed code reaches similarity scores 7.01% lower on average than randomly perturbed code (peak 42.86%) and completion accuracy 15% lower on average (peak 25%). It concludes that CodeMorph effectively mitigates data contamination while PESO optimizes perturbation combinations for code.

Load-bearing premise

The load-bearing premise is that lower completion accuracy on code that compiles and is judged semantically equivalent by LLM voters means the model's memorized training data was neutralized rather than the rewritten task simply becoming harder, and that the paper's similarity score is a valid proxy for decontamination even though PESO optimizes for that same score.

Editorial extensions

If this is right

Contaminated corpora can be recycled into evaluation sets: the paper claims that rewriting The Stack's own code makes StarCoder's memorized advantage measurable and largely removable, so decontamination no longer requires waiting for brand-new data.
On the paper's account, the 24.67% average accuracy drop (45% for single-line Python) is a lower bound on how much of StarCoder's completion score depended on surface memorization rather than generalized ability.
Because the transformations are prompt-driven and gated by a multilingual compile-and-vote check, the method claims to extend to additional languages and to repository-level code whose cross-file dependencies are left intact.
PESO's 7.01% average similarity improvement over random selection indicates that choosing perturbation methods by similarity feedback beats blind combinations, while the Boltzmann temperature keeps every category in play and deliberately sacrifices some optimality for exploration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same pipeline is effectively a contamination probe: if the rewrites truly preserve semantics, the accuracy gap between original and perturbed code quantifies how much any model leaned on surface form, so the method could estimate leakage for model–dataset pairs whose training data is not public.
The cleanest control the paper does not run is code the model has never seen, such as samples written after StarCoder's training cutoff; if the accuracy drop on such code matches 24.67%, the metric reflects added task difficulty, not neutralized memorization.
Because PESO's objective (the similarity score) doubles as its success metric, an independent outcome measure—for instance unit-test pass rates on samples where execution-based equivalence can be checked mechanically—would separate 'more heavily rewritten' from 'genuinely less contaminated.'
The verification gate's cost and small benchmark (100 samples, one model, one task type) are the practical bottleneck; replacing LLM voters with execution-based equivalence where tests exist would let the approach scale and make the equivalence guarantee machine-checkable rather than probabilistic.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper builds something real. CodeMorph is an LLM-driven, multi-language code perturbation pipeline with 26 semantics-preserving transforms, compile checks, three LLM voters, and a human audit pass. That last piece matters: the authors manually verified all perturbed code, and they preserve cross-file dependencies, which existing tools like CodART and rope don't handle. The artifact is linked on GitHub, which is a plus, though I haven't run it. The PESO genetic algorithm is a reasonable way to select perturbation methods per iteration, and the seven newly proposed transforms are catalogued clearly. For benchmark builders, this is a genuinely useful engineering contribution.

The soft spot is load-bearing: the paper concludes that a 24.67% average drop in StarCoder's code-completion accuracy means leakage was mitigated. But every experiment uses StarCoder, which was trained on The Stack—the same data the samples come from. A semantically-equivalent but structurally altered program can easily be harder to complete even for a model that never saw the original code: renamed identifiers, extracted functions, and inserted control flow all change the completion surface. Without a control model not trained on The Stack, or a direct contamination measure (e.g., membership tests before and after perturbation), the accuracy drop alone doesn't separate decontamination from perturbation-induced difficulty. The paper actually shows in Figure 8 that perturbation sometimes makes tasks easier, which reinforces that the completion surface changes non-monotonically. The authors admit in the Limitations that only StarCoder was tested, so the gap is known but not addressed.

The PESO evaluation is also partly circular: the similarity score ss is the objective PESO optimizes, so Table IV showing lower ss is optimization success, not decontamination. The accuracy comparison in Table III is less circular but inherits the same control problem. Minor issue: only 20 samples per language, no error bars or significance tests, though the API cost explanation is plausible.

Recommendation: the paper is honest, well-written, and addresses a real problem, but the central empirical claim is not supported. I'd send it to review with a request for a non-contaminated control model and a re-framing of PESO's contribution as "producing less similar code" rather than "mitigating leakage." The pipeline itself is worth the referee time.

Referee Report

4 major / 4 minor

Summary. CodeMorph is a pipeline for semantically-preserving code perturbation built on 26 transformation methods executed by an LLM, with compile checks and three LLM voters for semantic verification, plus PESO, a genetic/Boltzmann-selection algorithm that iteratively chooses perturbation categories to reduce a weighted surface/semantic similarity score. The evaluation uses StarCoder on samples from The Stack across five languages, reporting that code-completion accuracy drops by an average of 24.67% after perturbation (45% for Python single-line completion), and that PESO achieves a 7.01% lower similarity score than random perturbation. The manuscript concludes that CodeMorph effectively mitigates data leakage and that PESO optimizes perturbation combinations.

Significance. The manuscript has real strengths: a public artifact, a deliberately contaminated evaluation setup using StarCoder trained on The Stack, 26 documented transformations, multi-language compilation checks, human audit of perturbed code, and the use of three voter LLMs. If the central claims were substantiated, CodeMorph would be a practical tool for reducing memorization effects in code LLM evaluation. However, the two headline results are indirect: the accuracy drop does not isolate decontamination from perturbation-induced difficulty, and the similarity metric is the very objective PESO optimizes. The empirical base is small (20 samples per language, no significance tests), and the paper's own limitation section acknowledges that only one model/training-data pair was evaluated. These issues are load-bearing but addressable with additional controls and careful re-framing; the tool contribution itself is plausible.

major comments (4)

[Algorithm 1, line 6] GetSimilarity(c_ori, c_pert) should be GetSimilarity(c_ori, c_inter). As written, the algorithm computes the similarity of the previously accepted code rather than the candidate produced in the current iteration, so the acceptance test and the optimization gain do not evaluate the perturbation that was just applied. This contradicts the prose in §III-B2, which states that the similarity score is computed for the intermediate perturbed code after verification. Unless the algorithm is corrected and the experiments rerun, the PESO results in §V-B are not supported.
[§III-B and Table IV] The overall similarity score ss is exactly the objective that PESO optimizes in Algorithm 1 through Equation (1). Reporting that PESO lowers ss relative to random perturbation is an evaluation by construction: it shows that selection optimizes its own fitness function. The conclusion in §V-B that "code similarity can effectively indicate decontamination" requires an external validation of ss against a direct contamination measurement, which the paper does not provide. The non-circular evidence for PESO is therefore limited to the accuracy comparison in Table III.
[§V-A, Figure 6] The 24.67% average accuracy drop is interpreted as evidence of reduced data leakage, but no control condition separates decontamination from perturbation-induced difficulty. Semantically-preserving perturbations can still change the completion surface (renamed identifiers, extracted functions, inserted control flow, changed statement structure) and lower accuracy for a model with zero exposure to the original code. The paper itself notes in the discussion of Figure 8 that perturbation can sometimes increase accuracy by introducing hints, which shows that perturbations change task difficulty independently of contamination. The Limitation section states that no other LLM with a known corresponding training dataset was evaluated, so the observed drop is consistent with both increased difficulty and reduced leakage. A clean-model control, a direct membership test, or an ablation that holds task difficulty fixed is needed to support the decontamination conclusion.
[§IV-A and Tables III–IV] With 20 samples per language and no error bars, confidence intervals, or significance tests, the aggregate claims (24.67%, 7.01%, 15%) are not shown to be statistically robust. Table IV contains several code IDs where random perturbation attains a lower similarity score than PESO (e.g., Python IDs 3, 6, 13, 19), so the comparison is noisy and the reported averages may be dominated by a few samples. A paired statistical test and standard dispersion measures should be reported for both the accuracy and similarity comparisons.

minor comments (4)

[Abstract and §V-B] The "15% average / 25% maximum accuracy reduction" in the abstract refers to PESO relative to random perturbation, while the abstract's phrasing makes it appear to be relative to the original code; please clarify the baseline explicitly in both places.
[Table I] Table I contains typos and inconsistent capitalization, including "Add conditon", "Divde the long expression", and inconsistent renderings of "Div if else" versus "Div composed if".
[Algorithm 1] The loop header "for iter←1 to maxIter and ss<ss_threshold do" conflates a for-loop with a while-condition; a while-loop with a guard on ss would be clearer and would avoid the separate iter←iter+1 statement.
[§IV-A2] The arithmetic leading to 600 tasks should be stated more explicitly: 100 original samples plus 100 perturbed samples, each evaluated at 1-, 3-, and 5-line completion, gives 600 tasks; the current text says only "300 tasks for original and 300 for perturbed" without clearly explaining the factor of three.

Circularity Check

1 steps flagged · score 6.0 of 10

PESO's headline similarity improvement is the optimizer's own objective, so Table IV reports optimization success rather than independent decontamination evidence; the accuracy-drop claim is only partly independent support for the central leakage-mitigation claim.

self definitional [Section V-B (RQ2), Table IV; Section III-B, Algorithm 1 lines 6-13 and Equation 1]
"6 s1,s2←GetSimilarity(cori,cpert); 7 ss←µs1+νs2; ... 9 og←mss−ss; ... In terms of code similarity, as illustrated in Table IV, we present surface-level similarity, semantic similarity, and the overall similarity score ss for Python and Go. Across these metrics, PESO consistently achieves lower similarity scores than the random algorithm, showing an average reduction in ss of 7.01% and a maximum reduction of 42.86%."

Algorithm 1 defines the overall similarity score ss (Equation 1) as the objective that PESO optimizes: the optimization gain is og = mss − ss, and Boltzmann selection in lines 16-19 chooses future perturbation categories using these og values. Thus the final perturbed code produced by PESO is, by construction, the result of minimizing ss relative to previous iterations. Table IV then uses that same ss as the headline metric to show that PESO beats random perturbation, and the paper concludes that 'code similarity can effectively indicate decontamination.' This is a self-definitional evaluation: the metric used to validate the method is exactly the metric the method was built to reduce.

full rationale

The strongest circularity in the paper is confined to the PESO similarity comparison. Algorithm 1 and Equation 1 define ss as the objective; PESO's selection probabilities are computed from og = mss − ss. Table IV reports that PESO produces lower ss than random perturbation and uses this as evidence that similarity is a valid decontamination indicator. That claim is circular because the same quantity is both the optimized objective and the validation metric. The central 'mitigates data leakage' claim also relies on the accuracy drop in Figure 6/Table III (24.67% average reduction), which is not equivalent to the PESO objective and is therefore partly independent evidence. However, the paper itself explicitly uses the similarity result as part of its central conclusion that 'CodeMorph effectively reduces data contamination while PESO optimizes perturbation combinations for code,' so this is more than a minor issue. The lack of a non-contaminated control model (acknowledged in the Limitation section) weakens the accuracy-drop inference, but that is a validity concern rather than a by-construction circularity, so it is not scored as a separate circular step. There are no load-bearing self-citations or imported uniqueness theorems. Overall, one main result reduces by construction to its own objective while the other main result retains independent content, yielding a partial-circularity score of 6.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claims rely on hand-set hyperparameters and on three domain assumptions: contaminated training data, sufficient semantic-equivalence verification, and similarity as a decontamination proxy. No formal proof or machine-checked artifact is provided.

free parameters (5)

mu (surface similarity weight) = 0.5
Hand-set in Equation 1 to balance Levenshtein surface similarity and JPlag semantic similarity; directly shapes the PESO objective and the evaluation metric.
nu (semantic similarity weight) = 0.5
Hand-set with mu + nu = 1 in Equation 1; same effect on the PESO objective and evaluation.
Temperature T (Boltzmann selection) = 2
Controls exploration versus exploitation in Equation 2; hand-set in Section IV-B without sensitivity analysis.
maxIter (iteration limit) = 15
Termination condition in Algorithm 1; hand-set and stated as sufficient without evidence of convergence behavior.
ss_threshold (early stopping) = 0.2
Termination condition; the authors state that further perturbation is unlikely to reduce similarity below this level, but no supporting data are shown.

assumptions (3)

domain assumption The Stack dataset is contaminated with StarCoder training data, so lower accuracy on perturbed code measures reduced leakage.
Sections II and V-A treat StarCoder as guaranteed contaminated because The Stack is its training set, but no direct contamination test or membership inference is performed.
domain assumption Compilation plus a 2/3 majority of three voter LLMs reliably establishes semantic equivalence after perturbation.
Section III-A2 relies on this to ensure accuracy drops are not caused by semantic changes. The paper also manually audits perturbed code, but that audit is subjective and not formal.
domain assumption Levenshtein edit distance and JPlag similarity adequately capture perturbation effectiveness and decontamination.
Section III-B2 defines the overall similarity score ss as both the optimization objective for PESO and the primary evaluation metric in Table IV, making the assumption load-bearing.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CodeMorph: Mitigating Data Leakage in Large Language Model Assessment." pith.science (2026). https://pith.science/paper/SY4S7XMZ

@misc{pith2026250617627,
  author       = {Pith},
  title        = {Pith review of: CodeMorph: Mitigating Data Leakage in Large Language Model Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SY4S7XMZ}},
  note         = {Machine review of arXiv:2506.17627}
}

read the original abstract

Concerns about benchmark leakage in large language models for code (Code LLMs) have raised issues of data contamination and inflated evaluation metrics. The diversity and inaccessibility of many training datasets make it difficult to prevent data leakage entirely, even with time lag strategies. Consequently, generating new datasets through code perturbation has become essential. However, existing methods often fail to produce complex and diverse variations, struggle with complex cross-file dependencies, and lack support for multiple programming languages, which limits their effectiveness in enhancing LLM evaluations for coding tasks. To fill this gap, we propose CodeMorph, an approach designed to support multiple programming languages while preserving cross-file dependencies to mitigate data leakage. CodeMorph consists of two main components that work together to enhance the perturbation process. The first component employs 26 semantic-preserving transformation methods to iteratively perturb code, generating diverse variations while ensuring that the modified code remains compilable. The second component introduces a genetic algorithm-based selection algorithm, PESO, to identify the more effective perturbation method for each iteration by targeting lower similarity scores between the perturbed and original code, thereby enhancing overall perturbation effectiveness. Experimental results demonstrate that after applying CodeMorph, the accuracy of the LLM on code completion tasks across five programming languages decreased by an average of 24.67%, with Python showing the most significant reduction at 45%. The similarity score of code optimized by PESO is, on average, 7.01% lower than that of randomly perturbed code, peaking at a reduction of 42.86%.

Figures

Figures reproduced from arXiv: 2506.17627 by the authors.

**Figure 3.** The left example shows the original code. The code [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** The workflow of CODEMORPH Algorithm 1: Optimization Algorithm for Perturbation Method Selection Input: Original code cori, Perturbed code cpert, Selected perturbation methods pm, Optimization gain list OG[og1 . . . ogn], Method categories list MC[mc1 . . . mcn] Output: Perturbed code cpert 1 ss, mss ← 1, 1; 2 OG, cpert ← Initialization(MC, cpert); 3 for iter ← 1 to maxIter and ss < ssthreshold do 4 cinter ← Perturb… view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Benchmark construction process 1) Data Filtering: The first step in constructing a benchmark from The Stack dataset involves data filtering, which includes both rule-based filtering and manual selection. We now detail these two parts. First, we filter out the most uns…

**Figure 6.** Figure 6: Accuracy distribution across different code completion tasks for five languages [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 8.** Figure 8: Experimental results for Java code completion tasks [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 26 canonical work pages

[1]

https://github.com/m-zakeri/CodART

work page
[2]

https://github.com/openrewrite/rewrite

work page
[3]

https://github.com/python-rope/rope

work page
[5]

https://huggingface.co/bigcode/starcoder

work page
[6]

https://huggingface.co/datasets/bigcode/the-stack

work page
[7]

https://github.com/pylint-dev/pylint

work page
[8]

https://tree-sitter.github.io/tree-sitter/

work page
[9]

https://github.com/jplag/JPlag

work page

Show all 51 references

[10]

Palm 2 technical report,

R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chenet al., “Palm 2 technical report,”arXiv preprint arXiv:2305.10403, 2023

2023 arXiv
[11]

Program synthesis with large language models,

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Leet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

2021 arXiv
[12]

Concerned with data contam- ination? assessing countermeasures in code language model,

J. Cao, W. Zhang, and S.-C. Cheung, “Concerned with data contam- ination? assessing countermeasures in code language model,”arXiv preprint arXiv:2403.16898, 2024

2024 arXiv
[13]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

2021 arXiv
[14]

Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,

Y . Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Rothet al., “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,” Advances in Neural Information Processing Systems, vol. 36, 2024

2024
[15]

On the effectiveness of graph data augmentation for source code learning,

Z. Dong, Q. Hu, Z. Zhang, and J. Zhao, “On the effectiveness of graph data augmentation for source code learning,”Knowledge-Based Systems, vol. 285, p. 111328, 2024

2024
[16]

Complexcodeeval: A benchmark for evaluating large code models on more complex code,

J. Feng, J. Liu, C. Gao, C. Y . Chong, C. Wang, S. Gao, and X. Xia, “Complexcodeeval: A benchmark for evaluating large code models on more complex code,”arXiv preprint arXiv:2409.10280, 2024

2024 arXiv
[17]

Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Liet al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,”arXiv preprint arXiv:2401.14196, 2024

2024 arXiv
[18]

Large language models for software engi- neering: A systematic literature review,

X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, 2023

2023
[19]

Livecodebench: Holistic and contamination free evaluation of large language models for code,

N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,”arXiv preprint arXiv:2403.07974, 2024

2024 arXiv
[20]

Detecting logic bugs in database engines via equivalent expression transformation,

Z.-M. Jiang and Z. Su, “Detecting logic bugs in database engines via equivalent expression transformation,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 821–835

2024
[21]

Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval,

M. A. M. Khan, M. S. Bari, D. Long, W. Wang, M. R. Parvez, and S. Joty, “Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval,” inProceedings of the 62nd Annual Meeting of the Association for C...

2024
[22]

Optimization by simulated annealing,

S. Kirkpatrick, C. D. Gelatt Jr, and M. P. Vecchi, “Optimization by simulated annealing,”science, vol. 220, no. 4598, pp. 671–680, 1983

1983
[23]

Binary codes capable of correcting deletions, inser- tions, and reversals,

V . Levenshtein, “Binary codes capable of correcting deletions, inser- tions, and reversals,”Proceedings of the Soviet physics doklady, 1966

1966
[24]

Codemind: A framework to challenge large language models for code reasoning,

C. Liu, S. D. Zhang, and R. Jabbarvand, “Codemind: A framework to challenge large language models for code reasoning,”arXiv preprint arXiv:2402.09664, 2024

2024 arXiv
[25]

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,”Advances in Neural Information Processing Systems, vol. 36, 2024

2024
[26]

{EaTVul}:{ChatGPT-based}evasion attack against software vulnerability detection,

S. Liu, D. Cao, J. Kim, T. Abraham, P. Montague, S. Camtepe, J. Zhang, and Y . Xiang, “{EaTVul}:{ChatGPT-based}evasion attack against software vulnerability detection,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 7357–7374

2024
[27]

Starcoder 2 and the stack v2: The next generation,

A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Weiet al., “Starcoder 2 and the stack v2: The next generation,”arXiv preprint arXiv:2402.19173, 2024

2024 arXiv
[28]

Quantifying contamination in evaluating code generation capabilities of language models,

M. Riddell, A. Ni, and A. Cohan, “Quantifying contamination in evaluating code generation capabilities of language models,”arXiv preprint arXiv:2403.04811, 2024

2024 arXiv
[29]

Code llama: Open foundation models for code,

B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

2023 arXiv
[30]

Detecting pretraining data from large language models,

W. Shi, A. Ajith, M. Xia, Y . Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, “Detecting pretraining data from large language models,”arXiv preprint arXiv:2310.16789, 2023

2023 arXiv
[31]

Repairbench: Leaderboard of frontier models for program repair,

A. Silva and M. Monperrus, “Repairbench: Leaderboard of frontier models for program repair,”arXiv preprint arXiv:2409.18952, 2024

2024
[32]

Codemark: Imperceptible water- marking for code datasets against neural code completion models,

Z. Sun, X. Du, F. Song, and L. Li, “Codemark: Imperceptible water- marking for code datasets against neural code completion models,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing, 2023, ...

2023
[33]

Recode: Robustness evaluation of code generation models,

S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V . Kumar, S. Tan, B. Ray, P. Bhatia, R. Nallapati, M. K. Ramanathan, D. Roth, and B. Xiang, “Recode: Robustness evaluation of code generation models,” 2022

2022
[34]

Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,

M. Weyssow, A. Kamanda, and H. Sahraoui, “Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,”arXiv preprint arXiv:2403.09032, 2024

2024 arXiv
[35]

Livebench: A challenging, contamination-free llm benchmark,

C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz- Ziv, N. Jain, K. Saifullah, S. Naiduet al., “Livebench: A challenging, contamination-free llm benchmark,”arXiv preprint arXiv:2406.19314, 2024

2024 arXiv
[36]

Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,

C. S. Xia, Y . Deng, and L. Zhang, “Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,” arXiv preprint arXiv:2403.19114, 2024

2024 arXiv
[37]

Transformcode: A contrastive learning framework for code embedding via subtree transformation,

Z. Xian, R. Huang, D. Towey, C. Fang, and Z. Chen, “Transformcode: A contrastive learning framework for code embedding via subtree transformation,”IEEE Transactions on Software Engineering, 2024

2024
[38]

Benchmark data contamination of large language models: A survey,

C. Xu, S. Guan, D. Greene, M. Kechadiet al., “Benchmark data contamination of large language models: A survey,”arXiv preprint arXiv:2406.04244, 2024

2024 arXiv
[39]

Codescope: An execution-based multi- lingual multitask multidimensional benchmark for evaluating llms on code understanding and generation,

W. Yan, H. Liu, Y . Wang, Y . Li, Q. Chen, W. Wang, T. Lin, W. Zhao, L. Zhu, H. Sundaramet al., “Codescope: An execution-based multi- lingual multitask multidimensional benchmark for evaluating llms on code understanding and generation,”arXiv preprint arXiv:2311.08588, 2023

2023 arXiv
[40]

Srcmarker: Dual-channel source code watermarking via scalable code transformations,

B. Yang, W. Li, L. Xiang, and B. Li, “Srcmarker: Dual-channel source code watermarking via scalable code transformations,” in2024 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2024, pp. 97–97

2024
[41]

Rethinking benchmark and contamination for language models with rephrased samples,

S. Yang, W.-L. Chiang, L. Zheng, J. E. Gonzalez, and I. Stoica, “Rethinking benchmark and contamination for language models with rephrased samples,”arXiv preprint arXiv:2311.04850, 2023

2023 arXiv
[42]

Exploiting the adversarial example vulnerability of transfer learning of source code,

Y . Yang, H. Fan, C. Lin, Q. Li, Z. Zhao, and C. Shen, “Exploiting the adversarial example vulnerability of transfer learning of source code,” IEEE Transactions on Information Forensics and Security, 2024

2024
[43]

Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12

2024
[44]

Data augmentation by program transfor- mation,

S. Yu, T. Wang, and J. Wang, “Data augmentation by program transfor- mation,”Journal of Systems and Software, vol. 190, p. 111304, 2022

2022
[45]

Coderujb: An executable and unified java benchmark for practical programming scenarios,

Z. Zeng, Y . Wang, R. Xie, W. Ye, and S. Zhang, “Coderujb: An executable and unified java benchmark for practical programming scenarios,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 124–136

2024
[46]

Statfier: Automated testing of static analyzers via semantic-preserving program transformations,

H. Zhang, Y . Pei, J. Chen, and S. H. Tan, “Statfier: Automated testing of static analyzers via semantic-preserving program transformations,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing...

2023
[47]

Codebert- attack: Adversarial attack against source code deep learning models via pre-trained model,

H. Zhang, S. Lu, Z. Li, Z. Jin, L. Ma, Y . Liu, and G. Li, “Codebert- attack: Adversarial attack against source code deep learning models via pre-trained model,”Journal of Software: Evolution and Process, vol. 36, no. 3, p. e2571, 2024

2024
[48]

Challenging machine learning-based clone detectors via semantic-preserving code transformations,

W. Zhang, S. Guo, H. Zhang, Y . Sui, Y . Xue, and Y . Xu, “Challenging machine learning-based clone detectors via semantic-preserving code transformations,”IEEE Transactions on Software Engineering, vol. 49, no. 5, pp. 3052–3070, 2023

2023
[49]

Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,

Q. Zheng, X. Xia, X. Zou, Y . Dong, S. Wang, Y . Xue, L. Shen, Z. Wang, A. Wang, Y . Liet al., “Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x,” inProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min...

2023
[50]

Don’t make your llm an evaluation benchmark cheater,

K. Zhou, Y . Zhu, Z. Chen, W. Chen, W. X. Zhao, X. Chen, Y . Lin, J.-R. Wen, and J. Han, “Don’t make your llm an evaluation benchmark cheater,”arXiv preprint arXiv:2311.01964, 2023

2023 arXiv
[51]

Dynamic evaluation of large language models by meta probing agents,

K. Zhu, J. Wang, Q. Zhao, R. Xu, and X. Xie, “Dynamic evaluation of large language models by meta probing agents,” inForty-first International Conference on Machine Learning, 2024

2024
[52]

Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paulet al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” arXiv preprint arXiv:2406.15877, 2024

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

https://github.com/m-zakeri/CodART

work page

[2] [2]

https://github.com/openrewrite/rewrite

work page

[3] [3]

https://github.com/python-rope/rope

work page

[4] [5]

https://huggingface.co/bigcode/starcoder

work page

[5] [6]

https://huggingface.co/datasets/bigcode/the-stack

work page

[6] [7]

https://github.com/pylint-dev/pylint

work page

[7] [8]

https://tree-sitter.github.io/tree-sitter/

work page

[8] [9]

https://github.com/jplag/JPlag

work page