{"id":"58ccafb3-9c49-4686-a62e-c46ac1b77fd9","arxiv_id":"2607.07619","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"Re-evaluating four LLM code-efficiency benchmarks with 30-run statistical testing shows 93.89% of 'performant' implementations are indistinguishable from baselines; a multi-agent test-generation framework reveals hidden significant improvements in ~24% of previously non-significant tasks.","lead":"This paper shows that four popular LLM code-efficiency benchmarks rarely detect real performance differences, because their test inputs are too small and execution runs too few. The authors propose a multi-agent LLM framework that generates larger, performance-stressing tests, revealing previously hidden speedups in ~24% of tasks.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Generated tests validated only against canonical solution create a self-referential oracle: if both share a bug, performance gains are measured on incorrect workloads, inflating RQ3/RQ4 significance rates.","rationale":"The reader's verdict of CONDITIONAL with HIGH confidence is appropriate. The central finding (RQ1: 93.89% of benchmark-provided performant implementations are not statistically distinguishable from canonical solutions) is robust — it uses the benchmark's own tests under rigorous 30-run execution with Mann-Whitney U testing, and the manual analysis (Kappa 0.93) provides strong corroborating evidence. This finding stands independently of the framework's test-generation quality. The load-bearing concern is specifically about RQ3/RQ4, where the framework's generated tests are validated only against the canonical solution (§4.3). This creates a self-referential oracle: the canonical solution is both the performance baseline and the correctness oracle. If a generated test exploits a bug shared by both the canonical solution and the test-generation LLM's understanding, or if it triggers a degenerate path unique to the canonical solution, the measured 'performance improvement' is an artifact rather than a genuine algorithmic advantage. The paper acknowledges this risk partially in Example 5 (function misidentification) but does not systematically verify test correctness against independent oracles. The 12x gap between the framework (24-25%) and COFFE (0.45-2%) is suspiciously large and could partly reflect overfitting of generated tests to implementation-specific quirks rather than genuine performance-relevant behavior. However, I note several mitigating factors: (1) the Stage 1 evaluation on 209 manually verified tasks with known performance impact shows 44-48% significance, suggesting the framework does find genuine improvements on at least some tasks; (2) the replication package is publicly available; (3) the paper is transparent about limitations including function misidentification. The LLM-as-a-Judge agreement (Kappa 0.72-0.75) introduces moderate label noise into RQ2's extended analysis but does not affect the manually verified subset. The COFFE comparison lacks confirmation of identical prompting conditions, which is a secondary concern. Overall, the CONDITIONAL verdict is correct: the benchmark deficiency finding is solid, but the remediation claims (RQ3/RQ4) depend on generated test validity that has not been independently verified. The concrete test I propose — manually checking expected outputs against problem specifications for a sample of significant tasks — would determine whether the 24-25% rate is genuine or inflated. My verdict matches the reader's: CONDITIONAL, unchanged.","tokens_in":27520,"tokens_out":1094,"duration_ms":539149,"concrete_test":"For a random sample of 50 tasks where the framework reported statistically significant improvements (p<0.05), independently verify the correctness of each generated test's expected output against the problem description (not the canonical solution). Specifically: (1) manually derive the correct output for each generated test input from the problem specification, (2) compare this independent oracle against the expected output encoded in the generated test, (3) flag any test where the expected output matches the canonical solution but disagrees with the specification-derived oracle. If more than 5% of sampled tests have incorrect expected outputs, the 24-25% significance rate is substantially inflated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the core vulnerability. The framework's validation loop (§4.3) executes generated tests only against the canonical implementation: 'candidate tests... are then executed against the canonical implementation.' A test passes if it produces the same output as the canonical solution. This means the canonical solution serves as both (1) the baseline for performance comparison and (2) the functional oracle for test validation. This creates a circular dependency: any test that is functionally consistent with the canonical solution is accepted, regardless of whether it exercises the intended semantics of the problem description. The RQ3 claim that 24-25% of tasks show 'statistically significant improvements' is load-bearing on these generated tests measuring genuine performance differences. But consider: if the LLM generates a test with a very large input (e.g., 'ab'*500000 in Example 4) that triggers a code path where the canonical and performant implementations differ in complexity, but the expected output was derived from the canonical solution itself (not from an independent specification), the test may be measuring a degenerate or unintended execution path rather than the task's actual performance-relevant behavior. For instance, if the canonical solution has a bug on certain edge-case inputs (e.g., integer overflow, hash collision patterns, or pathological branch behavior), the generated test could exploit that bug to create an artificial performance gap. The performant implementation, which may not share the same bug, would then appear 'faster' simply because it avoids the buggy path — not because it is algorithmically superior. This concern is amplified by the fact that the paper reports 24-25% significance rates for its own framework vs. 0.45-2% for COFFE. Such a large gap could partly reflect that the framework's tests are more aggressive at finding inputs where the two implementations diverge — including divergences caused by bugs,浮","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper revisits four function-level Python performance benchmarks (EffiBench, Enamel, EvalPerf, Mercury) and evaluates whether benchmark-provided performant implementations are statistically distinguishable from canonical solutions under rigorous measurement (30 runs, Mann-Whitney U test, Cliff's delta). The key finding is that only 6.11% of 1,538 tasks show significant improvement, primarily due to insufficient test inputs. A manual analysis of 308 non-significant tasks (Cohen's Kappa 0.93) identifies two root causes: no meaningful performance change (99 tasks) and potential improvements not exposed by tests (209 tasks). The authors then propose a multi-agent LLM framework for generating performance-oriented tests, which reveals significant improvements in 24.01% (DeepSeek-v3.1) and 25.43% (GPT-4o) of previously non-significant tasks, substantially outperforming COFFE. The paper concludes with recommendations for benchmark design.","tokens_in":28405,"tokens_out":1617,"duration_ms":384796,"significance":"The paper makes a timely and valuable contribution to the software engineering community by systematically demonstrating that widely-used performance benchmarks for LLM-generated code often cannot reliably distinguish efficient implementations from canonical solutions. The experimental methodology is generally sound: 30-run repeated execution, non-parametric statistical testing, effect-size analysis, and a well-executed manual analysis with high inter-rater agreement (Kappa 0.93). The multi-agent test-generation framework is a practical contribution, and the replication package with generated code and measured data is publicly available. The finding that only 6.11% of benchmark-provided performant implementations are significantly faster under rigorous testing is important and actionable for benchmark designers. The comparison against COFFE provides a useful baseline. The paper also provides falsifiable predictions about which types of code changes (algorithmic vs. refactoring) are likely to be exposed by stronger test suites.","major_comments":[{"comment":"§4.3, validation loop: The framework validates generated tests by executing them against the canonical implementation only. This creates a self-referential oracle: if the canonical solution has a bug on certain edge-case inputs (e.g., integer overflow, hash collision patterns), the generated test may exploit that bug to create an artificial performance difference rather than measuring genuine algorithmic improvement. The paper acknowledges this risk partially in Example 5 (function misidentification) but does not systematically verify test correctness against independent oracles (e.g., the problem specification, the performant implementation, or differential testing). Since the RQ3/RQ4 claims (24-25% improvement rates) are load-bearing on these generated tests measuring genuine performance differences, the authors should either (a) validate a sample of generated tests against an oracle独立","section":null},{"comment":"§4.3, missing test quality analysis: The paper reports that the framework generates 10 tests per task and filters out failing ones, but does not report what fraction of generated tests pass validation, how many tasks end up with zero valid tests, or the distribution of final test-suite sizes. Without these numbers, it is unclear whether the 24-25% significance rate reflects strong test quality or simply the fact that any sufficiently large input will create a runtime difference. A table reporting pass rates, final test counts, and input-size distributions would strengthen the evaluation.","section":null},{"comment":"§4.4, RQ4 setup: The paper states that LLM-generated implementations from GPT-4o-mini, Claude-Sonnet-4.5, and Gemini-2.5-Flash are compared against canonical solutions under both original and generated test suites. However, it is unclear whether the LLM-generated implementations are first validated for functional correctness against the generated tests before performance measurement. If an LLM-generated implementation fails some generated tests, is it excluded from performance comparison? The paper should clarify the correctness filtering protocol used for RQ4.","section":null}],"minor_comments":[{"comment":"§4.2, Table 3: The 'Changes with performance impact' row lists 209 (67.86%) for the manual analysis, but the subcategory counts sum to 183 (algorithm strategy) + 26 (data structure) = 209, which is consistent. However, the LLM-as-a-Judge columns list 864 (76.06%) for DeepSeek and 840 (73.94%) for GPT-4o, but the subcategory percentages for DeepSeek sum to approximately 76.06% only if 'No changes' (6%) and 'Changes without performance impact' (17.96%) are added. The table should clarify whether the subcategory percentages are column-conditional or row-conditional.","section":null},{"comment":"§4.2, p. 15: The text states '72.1% and 68.31% of the tasks are identified as involving algorithm strategy changes by DeepSeek-v3.1 and GPT-4o-2024-08-06, respectively.' However, Table 3 shows 76.06% and 73.94% for 'Changes with performance impact,' which includes both algorithm strategy changes and data structure replacements. The 72.1% and 68.31% figures appear to refer to algorithm strategy changes only, but this is not clearly stated.","section":null},{"comment":"§3.2: The one-sided Mann-Whitney U test is used with the alternative hypothesis that the performant implementation is faster. The paper should clarify how tasks where the performant implementation is significantly slower are handled — are they counted in the p≥0.05 category, or separately?","section":null},{"comment":"§4.1, Table 2: The total row shows 1,444 (93.89%) non-significant tasks, but 1,538 - 94 = 1,444 is correct. However, the effect-size column header uses 'Neg.' as an abbreviation, while the text uses 'negligible.' Consistent terminology would improve readability.","section":null},{"comment":"Example 4 (p. 20): The generated test 'assert lengthOfLongestSubstring('ab'*500000) == 2' uses a very large input. While this effectively exposes the performance difference, the paper should discuss whether such inputs reflect realistic usage patterns or represent pathological cases that would not occur in practice.","section":null},{"comment":"§4.3, Fig. 2: The prompt template for Test Generation contains a placeholder 'ystandalone' which appears to be a formatting artifact ('exactly y standalone'). This should be corrected.","section":null},{"comment":"§5, p. 24: The text states 'an average of 32.66% more performance differences' are revealed by enhanced test suites, but this figure is not directly derivable from the results in Table 4 or Table 5. The paper should clarify how this number is computed or correct it.","section":null},{"comment":"References: Several entries have formatting issues (e.g., 'art, 2026' for the replication package, 'Demˇ sar' for Demšar). These should be cleaned up to match the journal's formatting requirements.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important gap in the LLM code performance benchmarking literature. The core finding (RQ1) is well-supported and valuable. However, the RQ3/RQ4 claims depend on the validity of LLM-generated tests, and the self-referential oracle concern is a genuine methodological gap that needs to be addressed before publication. The authors should consider adding a validation step where generated tests are checked against the performant implementation (not just the canonical solution) or against the problem specification. A small-scale manual validation of generated test correctness would also help. The paper is within scope for a software engineering journal and the topic is timely. The multi-agent framework, while not deeply novel architecturally, is well-motivated and the empirical evaluation is substantial."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee identifies three major concerns: (1) the self-referential oracle problem in test validation, (2) missing test-quality metrics for the generated test suites, and (3) unclear correctness filtering for LLM-generated implementations in RQ4. All three comments are valid and actionable. We will address each through manuscript revisions and, where feasible, additional analysis.","responses":[{"response":"The referee raises a legitimate concern. Our current validation loop executes generated tests against the canonical implementation only, which means that if the canonical solution has a latent bug on certain edge-case inputs, a generated test could exploit that bug to produce an artificial performance difference. We agree that this is a threat to the validity of the RQ3/RQ4 results and that the manuscript does not currently address it systematically enough. In revision, we will take two steps. First, we will conduct a differential-testing validation on a random sample of at least 100 tasks from the 1,345 previously non-significant tasks: for each sampled task, we will execute the generated tests against both the canonical solution and the benchmark-provided performant implementation, and flag any test where the two implementations produce different outputs. Such cases would indicate either a genuine algorithmic difference (expected) or a test exploiting a canonical-solution bug (problematic). We will report the fraction of tests that produce divergent outputs and manually inspect those cases to determine whether they reflect genuine performance differences or oracle artifacts. Second, we will add a paragraph to Section 7 (Threats to Validity) explicitly discussing the self-referential oracle risk and the mitigation we performed. We note that the risk is partially bounded by the fact that our Stage 1 evaluation (Table 4) focuses on 209 tasks where the code changes were manually verified to have genuine performance impact, and the significance rates on this subset (44.98% and 48.80%) are consistent with the overall rates, suggesting that the results are not primarily driven by oracle artifacts. However, we agree that a systematic differential-testing check is needed and承诺","revision_made":"yes","referee_comment":"§4.3, validation loop: The framework validates generated tests by executing them against the canonical implementation only. This creates a self-referential oracle: if the canonical solution has a bug on certain edge-case inputs, the generated test may exploit that bug to create an artificial performance difference rather than measuring genuine algorithmic improvement. The paper acknowledges this risk partially in Example 5 but does not systematically verify test correctness against independent oracles. The authors should either (a) validate a sample of generated tests against an oracle or (b) use differential testing."},{"response":"This is a fair and important point. The manuscript currently does not report the test-generation pass rates, the distribution of final test-suite sizes, or the input-size characteristics of the generated tests. These metrics are necessary for the reader to assess whether the 24-25% significance rate reflects meaningful test quality or is an artifact of trivially large inputs. We will add a new table to Section 4.3 reporting: (1) the fraction of generated tests that pass validation on the first attempt versus after repair iterations, (2) the number of tasks that end up with zero valid tests after the iteration limit is reached, (3) the distribution of final test-suite sizes (mean, median, min, max), and (4) the distribution of input sizes in the generated tests (e.g., string lengths, list sizes) compared to the original benchmark tests. Regarding the concern that any sufficiently large input will create a runtime difference: we note that all statistically significant cases in Table 4 have large effect sizes (Cliff's delta >= 0.474), and the comparison against COFFE (which also generates large inputs but achieves only 2.00% and 0.45% significance) suggests that input size alone is not sufficient — the test must also exercise the correct performance-critical code path. We will make this argument explicit in the revised text and support it with the input-size distribution data.","revision_made":"yes","referee_comment":"§4.3, missing test quality analysis: The paper reports that the framework generates 10 tests per task and filters out failing ones, but does not report what fraction of generated tests pass validation, how many tasks end up with zero valid tests, or the distribution of final test-suite sizes. Without these numbers, it is unclear whether the 24-25% significance rate reflects strong test quality or simply the fact that any sufficiently large input will create a runtime difference. A table reporting pass rates, final test counts, and input-size distributions would strengthen the evaluation."},{"response":"The referee is correct that the manuscript does not clearly state the correctness filtering protocol for RQ4. To clarify: in RQ4, LLM-generated implementations are first validated for functional correctness against the benchmark-provided test suites (the same correctness check used in the original benchmarks). Only implementations that pass all benchmark-provided tests are included in the performance comparison. The generated performance-oriented tests from RQ3 are then used solely for runtime measurement, not for correctness filtering. This design choice was made to ensure consistency with the original benchmark evaluation protocols, where correctness is determined by the benchmark-provided tests and performance is measured separately. However, we agree that this should be stated explicitly. We will add a paragraph to Section 4.4 (Approach) describing the correctness filtering protocol: which tests are used for correctness validation, which are used for performance measurement, and how implementations that fail correctness checks are handled. We will also discuss in Threats to Validity the potential concern that an LLM-generated implementation could be functionally correct on the benchmark tests but incorrect on the generated performance-oriented tests, which could affect the runtime comparison.","revision_made":"yes","referee_comment":"§4.4, RQ4 setup: It is unclear whether the LLM-generated implementations are first validated for functional correctness against the generated tests before performance measurement. If an LLM-generated implementation fails some generated tests, is it excluded from performance comparison? The paper should clarify the correctness filtering protocol used for RQ4."}],"tokens_in":27346,"tokens_out":1284,"duration_ms":275353,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper shows that four popular LLM code-efficiency benchmarks (EffiBench, Enamel, EvalPerf, Mercury) almost never distinguish performant implementations from canonical solutions under proper statistical testing. Only 6.11% of 1,538 tasks show significant differences with 30-run Mann-Whitney U testing. That finding is robust and important for anyone working in LLM code evaluation. The paper also builds a multi-agent framework to generate better performance tests, which is useful but has a real validation gap you should know about before recommending acceptance as-is.","headline":"Solid benchmark audit with a useful but imperfect test-generation framework; the core finding holds, the remediation has a validation gap.","tokens_in":28309,"tokens_out":671,"would_cite":true,"duration_ms":57786,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Only 6% of LLM Code Benchmarks Show Real Speed Gains","keywords":["LLM code generation","performance benchmarks","EffiBench","Enamel","EvalPerf","Mercury","Mann-Whitney U test","multi-agent framework"],"falsifier":"If an independent oracle or human-verified test suite found that a substantial fraction of the LLM-generated performance tests produce incorrect expected outputs or test the wrong function, the 24-25% of newly significant results could be artifacts of invalid tests rather than genuine performance differences being exposed.","tokens_in":27489,"feed_emoji":"⚡","tokens_out":1267,"duration_ms":124733,"temperature":0.7,"pith_summary":"This paper argues that four popular benchmarks for evaluating whether LLMs generate efficient code are fundamentally unreliable under their default settings. The authors re-evaluated 1,538 tasks from EffiBench, Enamel, EvalPerf, and Mercury using 30 repeated runs per task and Mann-Whitney U statistical testing (p<0.05). They found that only 6.11% of benchmark-provided performant implementations are actually statistically faster than canonical baseline solutions. Through manual analysis of 308 non-significant tasks, they identified two root causes: (1) about 21% of performant implementations contain no meaningful performance change (e.g., refactoring, variable renaming), and (2) about 68% contain genuine algorithmic or data-structure improvements (e.g., O(n^3) to O(n^2)) whose benefits are invisible because the benchmark test inputs are too small to stress the performance-critical code paths. To address the second cause, the authors built a multi-agent LLM framework with three agents (generation, diagnosis, repair) that produces deterministic, performance-oriented test cases with larger and more demanding inputs. These generated tests revealed statistically significant performance improvements in 24.01% (DeepSeek-v3.1) and 25.43% (GPT-4o) of previously non-significant tasks, compared to 0.45-2% for the prior method COFFE. When applied to LLM-generated code from three models, the enhanced test suites uncovered an additional 22.19% of statistically significant performance improvements that original tests missed.","feed_headline":"93% of LLM Code Benchmarks Fail to Show Real Speed Gains","feed_subtitle":"Only 6% of benchmark performant implementations are statistically faster under rigorous testing. Weak test inputs, not just weak models, are","key_machinery":"Multi-agent LLM test generation framework with three agents: Test Generation Agent (creates deterministic performance-stressing test inputs), Test Diagnosis Agent (analyzes failures and produces structured root-cause diagnostics), and Test Repair Agent (fixes failed tests using diagnostics). Combined with 30-run repeated execution and Mann-Whitney U statistical testing protocol.","core_discovery":"The central discovery is that the failure of existing LLM code-performance benchmarks to show efficiency gains is primarily a measurement and test-design problem, not solely a model-capability problem. The benchmark-provided performant implementations---which are supposed to demonstrate that a task has optimization opportunities---are statistically indistinguishable from baseline solutions 93.89% of the time under rigorous 30-run statistical testing. This is because test suites designed for functional correctness use inputs too small to expose asymptotic differences between algorithms (e.g., an input of length 6 cannot distinguish O(n^3) from O(n^2)). A multi-agent framework that generates,診","pith_inferences":["If the generated performance tests are validated only against the canonical solution (as described in Section 4.3), any bug shared between the canonical solution and the generated test would pass validation while producing incorrect performance measurements. This is a systematic blind spot: the test oracle inherits the baseline's errors.","The 24-25% success rate of the multi-agent framework means that 75-76% of previously non-significant tasks remain non-significant even with enhanced tests. This could indicate either that those tasks genuinely lack optimization opportunities, or that the LLM-generated tests still fail to find the right input sizes or structures to expose the performance differences---the ceiling of this approach i","If stronger test suites reveal that LLM-generated code is more efficient than previously thought, prior studies concluding that LLMs cannot match human-written code efficiency may need to be revisited with improved test infrastructure.","The framework's limitation in handling programs with multiple nested functions (Example 5) suggests that performance test generation for real-world code with complex function interfaces remains an open problem beyond the controlled benchmark setting."],"forward_implications":["Published rankings of LLMs on code efficiency benchmarks may be unreliable if they use single-run or small-sample execution without statistical significance testing, as observed differences could be measurement noise rather than genuine efficiency gains.","Benchmark designers should select tasks with meaningful asymptotic optimization opportunities and construct test inputs large enough to expose algorithmic differences, rather than reusing correctness-oriented test suites from HumanEval/MBPP.","The multi-agent test generation approach could be applied to other domains where benchmark test inputs are insufficient to stress performance-critical behavior, such as database query optimization or compiler benchmarking.","Future performance benchmarks should extend beyond isolated function-level tasks to class-level or repository-level settings where performance bottlenecks arise from component interactions, I/O, concurrency, or long-running workloads.","The finding that 21% of benchmark performant implementations contain no meaningful performance change suggests existing benchmarks need data cleaning and quality auditing before they are used for evaluation."],"fun_headline_variants":["Most LLM code benchmark tests are too weak to expose performance gains","Benchmark tests miss 94% of real LLM code speed improvements","LLM code benchmarks fail because test inputs are too small to show efficiency gaps","94% of benchmark performant solutions show no statistical speedup under rigorous testing","Weak test suites, not weak models, hide LLM code performance gains"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that the LLM-generated performance tests are functionally correct and deterministic. The framework validates tests only against the canonical solution, so if both the canonical solution and the generated test share the same bug, the test passes validation but produces incorrect performance measurements. This assumption is load-bearing because the entire evaluation of whether stronger tests reveal hidden performance gains depends on those tests being valid.","fun_headline_variants_meta":{"raw":{"variants":["Most LLM code benchmark tests are too weak to expose performance gains","Benchmark tests miss 94% of real LLM code speed improvements","LLM code benchmarks fail because test inputs are too small to show efficiency gaps","94% of benchmark performant solutions show no statistical speedup under rigorous testing","Weak test suites, not weak models, hide LLM code performance gains"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":741,"prompt_tokens":662,"completion_tokens":79,"prompt_tokens_details":null},"tokens_in":662,"tokens_out":79,"duration_ms":29767,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T04:51:20.168886+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If an independent oracle or human-verified test suite found that a substantial fraction of the LLM-generated performance tests produce incorrect expected outputs or test the wrong function, the 24-25% of newly significant results could be artifacts of invalid tests rather than genuine performance differences being exposed.","supporting_citations":[],"review_version":1}