{"id":"dd68d148-debf-4a3a-ae0f-3c5dd2824c14","arxiv_id":"2412.08109","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A code-obfuscation benchmark shows LLM code generation pass rates fall sharply when descriptions, code, and dependencies are rewritten to remove training-data familiarity.","lead":"The paper builds a new benchmark, OBFUSEVAL, that rewrites real C functions from five open-source projects using three levels of obfuscation so LLMs cannot recognize code they may have seen during training. On this benchmark, four LLMs' test pass rates dropped by up to 62.5% relative to the un-obfuscated tasks, suggesting current benchmarks may overstate code generation ability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Obfuscation TPR drops persist on code already outside the training cutoff, so the decrease is confounded by obfuscation difficulty, not just familiarity removal.","rationale":"The reader's weakest assumption correctly identifies that obfuscation may increase task difficulty independent of training exposure. My concern sharpens this into a concrete internal inconsistency: the paper already includes models whose original code is post-training-cutoff, so obfuscation on those cases cannot be removing familiarity. The persistent TPR drops on those models (e.g., Table IV redis rows for GPT3.5 and GPT4-1106) are direct evidence that the obfuscation pipeline itself degrades performance. Section V's admission that structure obfuscation increases code length and token count further supports this reading. The benchmark and the measurement of TPR declines remain valuable empirical contributions, and the paper's descriptive claims about obfuscation reducing pass rates are supported. However, the causal claim about 'inflated capabilities' from training-data exposure is not established by the current analysis. The reader's CONDITIONAL verdict is appropriate: the paper should be accepted only if the authors add the exposure-stratified analysis (or an equivalent control) and temper the causal language accordingly. I therefore leave the verdict unchanged rather than moving to reject, because the benchmark itself is reusable and the requested control is feasible with existing data.","tokens_in":20681,"tokens_out":5167,"duration_ms":51334,"concrete_test":"Stratify Table IV's TPR decrease ratios by training exposure: compute the drop separately for GPT-3.5-turbo-1106 and GPT-4-1106 (original code is post-cutoff, hence unseen) and for GPT-4-0125 and DeepSeek-Coder-V2 (which may have seen the code). If the unseen group shows a substantial TPR drop comparable to or larger than the seen group, the drop is an obfuscation difficulty effect, not familiarity removal. Report per-strategy and overall drops with confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that obfuscation reveals 'real capability' by removing training-data familiarity requires that the TPR decrease measure familiarity loss, not task difficulty. The paper's own setup provides a natural control that contradicts this. In Section IV-A2, the authors state that gpt-3.5-turbo-1106 and gpt-4-turbo-1106 have training data finalized before April 2023, while the raw functions were collected from PRs merged May–Dec 2023. Thus, for these two models, the 'Original' code is already unseen. Yet Table IV shows large TPR decreases after obfuscation even for these models (e.g., on redis: GPT3.5 Original TPR 11.9 -> Symbol 9.7, Structure 6.5, Symbol+Structure 2.9; GPT4-1106 17.6 -> 17.1, 7.0, 4.7). If the drop were caused by removing familiarity, it should be near zero on already-unseen code; it is not. Section V 'Code complexity' also concedes that structure obfuscation inlines called functions, increasing code length and token count, which 'potentially affect[s] the generation capabilities of large models.' The paper asserts it 'did not intentionally increase the complexity,' but intent is not a control. Therefore, the measured 15.3%–62.5% TPR decrease conflates familiarity removal with obfuscation-induced difficulty, and the conclusion that existing benchmarks overestimate LLM capability does not follow from the data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that existing code-generation benchmarks overestimate LLM capability because target code may have been seen during pretraining, and proposes OBFUSEVAL, a benchmark of 1,354 C functions from five mature projects, with functions drawn from post-training-cutoff PRs and obfuscated at symbol, structure, and semantic levels. Four LLMs are evaluated on code generation and code completion tasks using official test suites in a Docker-based execution platform. The reported average test pass rate (TPR) decreases by 15.3% to 62.5% after obfuscation, and the paper concludes that obfuscation removes training-data familiarity and thereby reveals the LLMs' real capability.","tokens_in":20965,"tokens_out":4435,"duration_ms":43375,"significance":"If the causal interpretation were supported, the paper would make a valuable methodological contribution: obfuscation is a reusable way to refresh benchmarks, and the project-level execution platform plus official test suites is a solid evaluation harness. The empirical finding that LLM TPR on these real C projects is low even on the original code (average 21.1%) is itself useful. However, the central causal claim and the headline decrease ratios depend on the assumption that obfuscation does not materially increase task difficulty, which the manuscript does not establish; the internal evidence actually suggests otherwise.","major_comments":[{"comment":"For gpt-3.5-turbo-1106 and gpt-4-turbo-1106, the manuscript states that training data were finalized before April 2023, while the raw functions were collected from PRs merged from May to December 2023, making the original code already unseen by these models. Table IV nevertheless shows substantial TPR drops under obfuscation for these models, e.g., redis GPT4-1106 goes from 17.6 on Original to 4.7 on Symbol+Structure, and libgit2 GPT4-1106 goes from 23.1 to 0.0. If the drop were specifically caused by removing training familiarity, it should be near zero on already-unseen code. The persistence of large drops on unseen code implies that obfuscation-induced difficulty contributes substantially, so the conclusion that existing benchmarks overestimate LLM capability does not directly follow. This is the load-bearing confound and requires a control analysis, such as comparing obfuscation drops on pre-cutoff versus post-cutoff code or using a difficulty-matched baseline.","section":"Section IV-A2 and Table IV"},{"comment":"The threats-to-validity section concedes that structure obfuscation inlines called functions and can increase code length and token count, which 'potentially affect[s] the generation capabilities of large models.' The statement that the authors 'did not intentionally increase the complexity' is not a control. Because symbol obfuscation also replaces all identifiers, making the prompt lexically novel, the observed TPR decrease could reflect increased prompt difficulty (longer context, unfamiliar names, inlined code) rather than removal of memorized target code. The manuscript needs a quantitative analysis of prompt/code complexity before and after obfuscation and/or a control condition to separate difficulty from familiarity.","section":"Section V, Code complexity"},{"comment":"Semantic obfuscation is manual, applied to only 159 functions (106 from redis, 17 from libvips, 15 from lvgl, 10 from libgit2, and 11 from fluent), and Section IV-B2 admits that the semantic strategy is 'relatively simple.' Because the per-strategy and combination results are computed on different, small subsets rather than on a common benchmark, the aggregate comparison of strategy effectiveness and the headline 'up to 62.5%' is not a representative estimate across the dataset. The authors should report results on matched subsets and include confidence intervals, especially for the semantic and combination conditions.","section":"Section III-B3 and Table III"}],"minor_comments":[{"comment":"The text contains formatting artifacts such as 'O BFUS EVAL' and '/user-edit' before the RQ conclusions; these should be cleaned before publication.","section":"Throughout"},{"comment":"Table IV uses 'libgits' while the repository name and Table II use 'libgit2'; please make the labels consistent.","section":"Table IV"},{"comment":"Figure 7 contains code typos ('Void' for 'void') and prose typos ('Differenet'); the figure and caption should be proofread.","section":"Fig. 7"},{"comment":"The paper defines pass@5 but does not state how many samples were generated per function or the sampling temperature; please specify the sampling setup for reproducibility.","section":"Section IV-A3"},{"comment":"The paper says 'we did not deliberately increase the difficulty' in Section IV-B2 and Section V, but Section III-B2 describes structure obfuscation as unfolding functions, which the authors concede may increase token count; please reconcile these statements with a quantitative measure of prompt length before and after obfuscation.","section":"Sections IV-B2 and V"}],"recommendation":"major_revision","confidential_remarks":"The paper's internal control is actually present but not analyzed: the two 1106 models have training data finalized before April 2023, and the raw functions are post-cutoff. If the authors can show that obfuscation drops are much smaller on this already-unseen subset than on models with later cutoffs, the familiarity interpretation would gain support; as it stands, Table IV shows substantial drops even on the already-unseen code. I would recommend asking for that analysis and for a difficulty measurement, not rejecting the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper: it ships a genuinely reusable obfuscation-based benchmark for code generation, and its headline interpretation does not survive its own control. The TPR drops are real; the attribution to familiarity removal is not established.\n\nWhat is new and good: OBFUSEVAL applies symbol, structure, and semantic obfuscation to C functions drawn from five mature projects, evaluates against official test suites, and provides a Docker-based execution platform. The paper reports systematic CPR/TPR measurements and shows consistent drops after obfuscation. The manual code review of non-functional issues (resource leaks, missing null checks, inefficient loops) is a useful secondary contribution. The benchmark itself, with the repo link, is a credible artifact for future contamination-resistant evaluation work.\n\nNow the soft spots, and they are load-bearing. Section IV-A2 states that gpt-3.5-turbo-1106 and gpt-4-turbo-1106 have training data finalized before April 2023, while the raw functions were collected from PRs merged May–Dec 2023. For those two models, the original code is already unseen. Yet Table IV shows large TPR drops after obfuscation for exactly those models (e.g., redis GPT3.5 goes from 11.9 to 2.9 under symbol+structure; GPT4-1106 from 17.6 to 4.7). If the drop measured removal of training familiarity, it should be near zero on already-unseen code. It is not. Section V concedes that structure obfuscation inlines called functions and increases code length and token count, which \"potentially affect[s] the generation capabilities of large models.\" Saying the authors did not intentionally increase complexity is not a control. The experiment therefore conflates familiarity removal with obfuscation-induced difficulty, and the paper's conclusion that existing benchmarks overestimate LLM capability does not follow from the data.\n\nAdditional, smaller issues: the headline 15.3%–62.5% decrease is computed on non-uniform subsets (structure applies to 450 functions, semantic to only 159), and many per-software numbers have tiny denominators. There are no confidence intervals, and the pass@5 measure does not remove variance across runs. Semantic obfuscation is manual and uneven across projects, so strategy comparisons should be read cautiously.\n\nWho this is for: anyone working on code generation evaluation or contamination in benchmarks. The method and dataset are worth engaging with, and the confound is a useful cautionary case. But the causal claim needs to be reanalyzed or substantially softened. I would send this to peer review with a request for major revision: measure complexity changes, compare seen vs. unseen subsets directly, and either provide a control for obfuscation difficulty or drop the \"inflated capabilities\" language.","headline":"A reusable obfuscation-based benchmark with real TPR drops, but the paper's central claim that the drops measure removed training familiarity is undercut by its own cutoff control.","tokens_in":21549,"tokens_out":2257,"would_cite":true,"duration_ms":25097,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Existing code benchmarks overestimate LLM ability: obfuscating target code drops test pass rates by up to 62.5%.","keywords":["large language models","code generation","benchmark contamination","code obfuscation","training data leakage","test pass rate","code familiarity","real-world C projects"],"falsifier":"Give expert human C programmers the original and obfuscated versions of the same functions and compare their success rates; if humans show a comparable drop on obfuscated tasks, then the obfuscation itself raises difficulty, which would undermine the paper's attribution of the LLM drop to lost familiarity.","tokens_in":20480,"feed_emoji":"📉","tokens_out":11935,"duration_ms":98527,"temperature":0.7,"pith_summary":"This paper argues that current code-generation benchmarks give inflated estimates of what large language models can actually do, because the target code is often already present in the models' training data. To remove that familiarity, the authors build OBFUSEVAL, a benchmark that collects 1,354 test-covered functions from five mature C projects and applies three levels of obfuscation - symbol renaming, structural restructuring, and semantic rewriting - to prompts, code, and dependencies. Across four large language models, the average test pass rate falls by 15.3% to 62.5% after obfuscation, with the largest drop for combined symbol-plus-structure obfuscation. The authors conclude that models' apparent expertise on familiar tasks is partly a specialist-in-familiarity effect, and that obfuscated benchmarks reveal a truer picture of production readiness.","feed_headline":"Obfuscating benchmark code drops LLM pass rates by up to 62.5%","feed_subtitle":"Real-world C functions, stripped of training-data familiarity, show models cannot yet handle production code","key_machinery":"The load-bearing mechanism is three-level code obfuscation applied to otherwise real, test-covered functions. Symbol obfuscation rewrites every meaningful identifier in the target function and its context, so the prompt no longer matches any training snippet. Structure obfuscation uses compiler intermediate representation to inline called functions, changing the code's organization and length. Semantic obfuscation rewrites code to a functionally equivalent but implementationally different form, which currently requires human effort. The evaluation machinery is a containerized execution platform that replaces the original function with the model's output, recompiles the project, and runs the official test suite; comparing test pass rates before and after obfuscation is the instrument that quantifies loss of familiarity.","core_discovery":"The authors' central claim is that benchmark scores are inflated by code leakage: functions drawn from open-source projects are likely to appear in LLM training corpora, so models can solve them by recall rather than by genuine code synthesis. To expose this, OBFUSEVAL takes functions that entered real projects after known training cutoffs, rewrites their descriptions, and applies three levels of obfuscation to the code and its context: symbol obfuscation replaces all meaningful identifiers, structure obfuscation inlines called functions to change execution organization, and semantic obfuscation re-implements logic in a different but equivalent way. Generated code is then compiled inside a containerized sandbox and exercised against the projects' official test suites. The measured test pass rate drops consistently across all four evaluated models after obfuscation, by an average of 15.3% for semantic obfuscation up to 62.5% for combined symbol-plus-structure obfuscation. The authors read this as evidence that the models rely on familiarity with original code, so obfuscation measures the models' real, transferable capability.","pith_inferences":["If the familiarity explanation holds, contamination is not binary: code written after a training cutoff can still resemble training code through reused idioms, so routine obfuscation would harden any code benchmark against gradual saturation.","The same de-familiarization treatment could transfer to code repair, code review, or test-generation benchmarks, where target-code exposure also risks inflating measured ability.","One way to disentangle difficulty from familiarity would be to measure how quickly models adapt to obfuscated versions through fine-tuning; fast recovery would suggest the drop is about retrieval rather than reasoning."],"forward_implications":["Scores on existing code benchmarks should not be equated with production capability; a model can perform well on familiar functions yet fail on the same task once the code is obfuscated.","Benchmark builders can reuse existing code corpora by obfuscating them, rather than waiting for new code, which addresses the timeliness problem.","Future LLM generations should be evaluated on obfuscated versions of the same tasks to distinguish genuine learning from memorization of training data.","Even code that passes official tests can carry non-functional defects such as missing null checks and inefficient loops, so functional correctness alone is an incomplete measure of readiness."],"supporting_citations":[{"why":"Supplies the classic code-generation benchmark paradigm that the paper argues is inflated by training-data exposure.","marker":"[9]"},{"why":"Provides a real-project benchmark that rewrites descriptions but not code, motivating Gap 1; also the source of the 70% contextual-dependency statistic.","marker":"[13]"},{"why":"Provides a benchmark built from historical code modifications that may still be exposed to training through code clones, motivating Gap 2.","marker":"[14]"},{"why":"Provides a real-project benchmark that supplies complete dependencies, contrasting with the paper's partial, obfuscated dependency provision (Gap 3).","marker":"[15]"},{"why":"Shows a periodic-update approach to code benchmarks that still risks code-clone similarity, supporting the timeliness argument.","marker":"[40]"},{"why":"Supplies the code-obfuscation concept that the benchmark's symbol, structure, and semantic strategies are built on.","marker":"[53]"},{"why":"Provides the compiler infrastructure used to implement structure obfuscation by inlining called functions.","marker":"[61]"}],"fun_headline_variants":["Obfuscation strips familiarity, LLM pass rates drop 62.5%","Code obfuscation reveals LLMs rely on memorization","Beyond training data: LLMs' true code synthesis exposed","Benchmark scores inflated? Obfuscated code says yes","LLMs pass less when code is obfuscated: up to 62.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that obfuscation changes the code without increasing its intrinsic difficulty, so that the measured drop in test pass rate can be attributed entirely to the loss of training-data familiarity rather than to harder prompts.","fun_headline_variants_meta":{"raw":{"variants":["Obfuscation strips familiarity, LLM pass rates drop 62.5%","Code obfuscation reveals LLMs rely on memorization","Beyond training data: LLMs' true code synthesis exposed","Benchmark scores inflated? Obfuscated code says yes","LLMs pass less when code is obfuscated: up to 62.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1989,"prompt_tokens":1096,"completion_tokens":893,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":796}},"tokens_in":712,"tokens_out":893,"duration_ms":8378,"temperature":1.0,"reasoning_tokens":796,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:12:16.897213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give expert human C programmers the original and obfuscated versions of the same functions and compare their success rates; if humans show a comparable drop on obfuscated tasks, then the obfuscation itself raises difficulty, which would undermine the paper's attribution of the LLM drop to lost familiarity.","supporting_citations":[{"cited_title":"Protecting software through obfuscation: Can it keep pace with progress in code analysis?","cited_arxiv_id":null,"evidence_quote":"Supplies the code-obfuscation concept that the benchmark's symbol, structure, and semantic strategies are built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the compiler infrastructure used to implement structure obfuscation by inlining called functions."}],"review_version":1}