Pith. sign in

REVIEW 3 major objections 7 minor 51 references

Optimo speeds up Python by routing profiled bottlenecks to specialized prompts across algorithm, statement, expression, and API levels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 16:22 UTC pith:HTR4GMJA

load-bearing objection Solid multi-level LLM optimizer with real benchmark lifts; the MoP + differential-profiling combo is the real contribution, with a fair but fixable correctness-asymmetry caveat. the 3 major comments →

arxiv 2607.23665 v1 pith:HTR4GMJA submitted 2026-07-26 cs.SE

Multi-level Code Optimization via Mixture of Prompts

Yun Peng , Jun Wan , Jiakun Liu , Shuzheng Gao , David Lo , Xiaoxue Ren This is my paper
classification cs.SE
keywords Code OptimizationLarge Language ModelCode EfficiencyMixture of PromptsDifferential ProfilingMulti-level OptimizationCode Generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Runtime efficiency matters for software quality, but compilers cannot optimize dynamic languages the way they optimize static ones, and recent LLM optimizers often pick the wrong targets and only edit a few lines. This paper argues that a Mixture-of-Prompts design can fix both problems: differential profiling finds code whose cost grows with input size, then routes those structures to prompts that encode how developers actually rewrite slow code, plus a general shared prompt when no specialist fits. Optimization runs in four successive layers—from whole-algorithm redesign down to API swaps—so coarse gains are not erased by later fine edits. On two efficiency benchmarks the method produces correct programs that are at least 10% faster far more often than prior LLM baselines, and it improves both human-written and LLM-generated code, sometimes past the human reference. A sympathetic reader cares because the same recipe turns ordinary LLM coding ability into systematically faster programs without training new expert models.

Core claim

The authors claim that multi-level LLM optimization under a Mixture-of-Prompts architecture—differential profiling to select time-critical structures, routing them to mined special strategies plus shared strategies, then fusing validated edits from algorithm through API level—yields substantially higher rates of correct, meaningfully faster Python programs than single-level or untargeted LLM baselines on COFFE and EffiBench.

What carries the argument

Mixture-of-Prompts (MoP): optimization-strategy prompts play the role of experts; differential profiling and adaptive AST routing send each bottleneck to the matching strategy; correctness reflection and performance validation gate each level before the next.

Load-bearing premise

That lines whose time share is above average and rises on larger inputs are the right places to rewrite, and that rewrite recipes mined from cleaned contest submissions transfer to the evaluation programs.

What would settle it

Re-run the same human-written and GPT-4o programs on COFFE and EffiBench with identical timing protocol: if Optimo’s opt% and overall speedup no longer exceed the strongest baseline by the reported margins, or if ablating differential routing collapses the gains, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLM code optimizers should treat bottleneck selection and strategy routing as first-class design, not a single generic rewrite prompt.
  • Four ordered levels (algorithm → statement → expression → API) can compound speedups that statement-only methods miss.
  • Strategies mined from a few thousand slow–fast contest pairs, plus adaptive AST abstraction, already cover most time-critical structures in both human and LLM code.
  • A non-trivial share of fully LLM-written-and-optimized programs can outrun the human reference solutions on the same problems.
  • Higher token and call cost is the explicit trade for the measured opt% and speedup gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same MoP pattern could be ported to other dynamic languages once a modest slow–fast corpus and a profiler exist, because nothing in the design is Python-syntax-bound beyond the AST and API map.
  • If correctness reflection stays incomplete, pairing MoP with stronger test generators or differential oracles would be the natural next lever on correct%.
  • Contest-mined strategies may under-represent long-running systems code (I/O-bound services, numerical libraries); a domain-specific mining pass would test how far the coverage claim generalizes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The manuscript presents Optimo, an LLM-based source-code optimization system for Python built around a 'Mixture-of-Prompts' (MoP) architecture. Differential profiling on small/large test inputs identifies 'time-critical' statements and expressions; these are routed to optimization strategies mined from slow-fast Codeforces submission pairs (with AST-depth abstraction for coverage); optimization proceeds at four levels (algorithm, statement, expression, API) with correctness reflection against self-generated tests and performance validation at each stage. On COFFE and Effibench, with GPT-4o as the base model for all approaches, Optimo reports up to 57.48% opt% (correct and >=10% faster) and up to 3.97x overall speedup on human-written code, and up to 42.42% opt% / 13.51x speedup on GPT-4o-generated code, outperforming six baselines. Ablations remove each level and MoP component; a coverage study, per-instance speedup distributions with Wilcoxon tests, Venn uniqueness analysis, and a token/API-call cost analysis are provided. Code and data are publicly archived.

Significance. If the results hold, this is a solid contribution to LLM-based performance engineering: a training-free MoE-analog architecture, a scalable strategy-mining pipeline, and clear state-of-the-art results on two public benchmarks. The empirical package is unusually complete for this subfield: component-wise ablations (Table 4), a coverage study isolating the routing contribution (Table 1), Wilcoxon tests on per-instance speedups with effect sizes, Venn-based uniqueness analysis, a cost accounting in model-independent units (Table 5), explicit deduplication against the mining corpus (§4.3), and a public artifact at a Zenodo DOI. The honest statement that correctness reflection is 'neither complete nor sound' (§3.4.1) is also to the authors' credit. These strengths make the manuscript worth getting right; the concerns below are about what exactly the headline numbers measure, not about the plausibility of the approach.

major comments (3)
  1. [§3.4.1, §4.5, Tables 2-3] Every headline number (opt%, speedup, FH rates) is conditioned on 'correct,' and the manuscript never states which test suite adjudicates correctness in Tables 2-3. Section 3.4.1 says correctness reflection uses the small self-generated tests from Section 3.2 (STGen/LLM-generated) and repairs candidates until they pass those tests; Section 4.5 says the evaluation 'adopts the evaluation framework from COFFE,' which itself relies on LLM-generated stress tests. If the tests used for final correct% overlap with the tests the reflection loop repairs against, then Optimo's 98.74%/98.30% correct% (and hence the 57.48%/42.42% opt% headline figures) partly measures test-suite fit, while baselines such as EffiLearner and PIE receive no equivalent repair-to-pass stage. This is the gate on the central comparative claim. The paper should (a) state explicitly that final correctness is measured on a su
  2. [§5.2, §5.5 (Table 5)] Optimo uses 8.46-16.69 LLM calls and up to ~20K input tokens per instance, versus 1-3 calls for all baselines (Table 5). The comparative claims in §5.2 ('outperforms the best baseline by up to 96.51% in opt%') therefore conflate the contribution of the MoP/multi-level design with best-of-N candidate generation, execution-time-based selection, and iterative repair that baselines do not receive at any budget. The ablations in Table 4 remove one component at a time but never give a baseline the same budget: there is no control such as 'CoT with N candidates + correctness reflection + performance validation' or 'EffiLearner with matched API calls.' The w/o-Special ablation (opt% drops to 38.69%/47.28%) suggests much of the gap over CoT (35.43%/21.09%) persists, which is encouraging, but a budget-matched control is needed to attribute the
  3. [§5.4 (Table 4), §3.2] Challenge 1 and the differential-profiling mechanism (§3.2) are presented as a core contribution, but RQ4 ablates only the four levels, the MoP strategies, and correctness reflection - never the target-identification method itself. The natural control is to keep the full Optimo pipeline but replace time-critical structure selection with EffiLearner-style hottest-line selection, and show opt%/speedup degrade. Without this, the claim in §2/§5.2 that 'correctly identifying optimization targets' drives the improvement over EffiLearner is asserted via one anecdotal example (Fig. 1) rather than measured. Additionally, the operational thresholds ('exceeds the program average,' 'ratios increase on large inputs') have no sensitivity analysis, and the motivating logic has an internal tension: Challenge 1 argues I/O-dominated lines (e.g., print/input) are poor targets because they 'cannot be fur
minor comments (7)
  1. [§5.2] §5.2: the Wilcoxon signed-rank results are reported only as 'p < 0.05' with 'positive effect sizes in every comparison.' Please report the actual statistics, effect-size magnitudes (e.g., Vargha-Delaney A12 or Cliff's delta), and whether p-values were corrected across the 18 comparisons.
  2. [§4.3] §4.3: the duplicate-removal step (64 problems) applies to COFFE, but the manuscript should also state whether any Effibench problems overlap with the 2,005 mining problems, and whether the COFFE dedup is exact-match or also catches same-problem/different-submission near-duplicates, since both the mining corpus and COFFE derive from Codeforces.
  3. [§3.3.3] §3.3.3: the efficient-API map covers 685 groups, with functional equivalence 'measured by test cases.' Please report how many test cases per group, how they were generated, and the false-equivalence risk; the API-level edits feed directly into final outputs with only the same self-generated tests as a check.
  4. [Figures 4-5] Figures 4-5: boxplots and Venn diagrams are hard to read at print scale; Fig. 5's four-category legend (NC/NO/NH/FH) should define the comparison baseline (which human-written solution time is used per problem) in the caption.
  5. [Abstract] Abstract/§5: the 'up to' phrasing mixes benchmarks and settings (57.48% opt% is COFFE-File; 3.97x is COFFE-Function; 13.51x is COFFE-Function GPT-4o code). Consider reporting the paired numbers per benchmark to avoid an inflated impression.
  6. [§3.4.1] §3.4.1: the reflection loop's iteration budget (max repair rounds, temperature, stopping criterion) is not specified; these details matter for reproducibility even though the artifact is available.
  7. [Table 2] Table 2: Optimo's correct% on Effibench (85.15%) is notably lower than on COFFE (98%+); a brief explanation (e.g., Effibench test strictness, the 707/1000 filtering) would help readers interpret the Effibench column.

Circularity Check

1 steps flagged

Empirical systems paper with no derivation-by-construction; only mild non-load-bearing self-use of author-overlapping COFFE/STGen tooling.

specific steps
  1. self citation load bearing [§3.2 Differential Profiling; §4.5 Implementation; COFFE citation [34]]
    "Optimo adopts the stress test case generation approach STGen [34] to generate large test cases. ... We adopt the evaluation framework from Coffe [34]. Specifically, each code solution is executed 12 times in independent Docker containers"

    STGen and the COFFE harness are from overlapping-author work and are reused both to drive profiling/reflection and to report metrics. This is a mild self-tooling dependency, not a derivation that forces opt% or speedup by construction; EffiBench and external baselines still provide independent content. Not load-bearing for the central empirical claim.

full rationale

Optimo’s headline claims are measured opt% and speedup on COFFE and EffiBench, not first-principles predictions. Optimization strategies are mined from an external cleaned Codeforces corpus (§3.3), duplicates with the eval sets are removed (§4.3), and results are compared to independent baselines under a shared base model. There is no equation in which a fitted parameter is renamed as a prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via self-citation. The only self-reference is operational: STGen and the COFFE evaluation framework come from overlapping-author work [34] and are used for differential profiling, correctness reflection, and reporting (§3.2, §3.4.1, §4.5). That is ordinary benchmark reuse, not a circular reduction of the claimed speedups. The paper itself labels correctness reflection “neither complete nor sound,” so it does not treat those tests as a closed oracle. Score 1 reflects that minor self-tooling overlap without elevating it to load-bearing circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Load-bearing content is mostly engineering design choices and domain assumptions about what is optimizable in Python source, not physical constants. The central claim rests on the operational definition of time-critical structure, the 10% significance/opt threshold shared with prior work, transfer of Codeforces-mined strategies, and the four AST-inspired levels—not on new particles or fitted laws of nature.

free parameters (5)
  • opt% / significance threshold (10%) = 10%
    A program counts as optimized only if ≥10% faster; same cutoff used when building slow–fast pairs and API groups. Inherited from prior work but still a discrete choice that moves headline opt%.
  • max special strategies per target after merge (20) = 20
    Phase II caps groups per optimization target at 20 to avoid overloading MoP (§3.3.2).
  • AST depth-reduction steps for adaptive routing / abstraction = reduce depth by one repeatedly until match
    How far subtrees are simplified when exact match fails is a design knob affecting coverage (Table 1) and which expert prompt fires.
  • four optimization levels and fixed order (algorithm→statement→expression→API) = 4 levels, coarse-to-fine
    Level set and ordering are chosen from AST node granularity, not learned; ablation shows each level contributes but the cardinality is hand-set.
  • execution protocol (12 Docker runs, mean of middle 10, wall-clock) = 12 runs / middle 10
    Defines measured t_s and t_o; wall-clock includes I/O and system noise.
axioms (5)
  • domain assumption Code structures whose runtime share exceeds average and grows with input scale admit more efficient source-level variants more often than hot but scale-invariant operations.
    Core justification for differential profiling as the router (§3.1–3.2); motivates rejecting pure “most expensive line” selection used by baselines.
  • domain assumption Slow–fast pairs from competitive programming encode transferable optimization knowledge for general Python efficiency benchmarks after cleaning.
    Strategy mining corpus is Codeforces Python submissions (§3.3.1); evaluation is COFFE/EffiBench.
  • ad hoc to paper Ordering optimizations from coarse AST roots to fine leaves reduces destructive overwrite of earlier gains.
    Stated design rationale for sequential four-level pipeline (§3.1, §3.4).
  • domain assumption Small LLM- or public-generated tests plus reflection suffice as a practical filter for keeping candidate rewrites (explicitly not complete or sound).
    Correctness reflection (§3.4.1); authors acknowledge limitation.
  • domain assumption Functionally equivalent APIs can be ranked by microbenchmarks into a static efficient-API map.
    API level bypasses MoP and uses a 685-group map (§3.3.3, §3.4.3).
invented entities (3)
  • Mixture-of-Prompts (MoP) architecture no independent evidence
    purpose: Replace trained MoE experts with routed optimization-strategy prompts (special + shared) for code optimization without large supervised expert training.
    Central architectural claim of the paper; analogy to MoE is conceptual, implementation is prompt routing + validation.
  • Time-critical code structure (operational definition) independent evidence
    purpose: Define optimizable bottlenecks as statements/expressions with above-average time share that increases from small to large inputs, represented as AST subtrees.
    Distinct from “hottest line” profiling; drives routing targets (§3.2).
  • Special vs shared optimization strategies (mined + manual) no independent evidence
    purpose: Encode developer optimization patterns as structured prompts (routing pattern, method, result) plus a general CoT fallback per level.
    191/851/805 special strategies reported at algorithm/statement/expression levels after mining and summarization.

pith-pipeline@v1.2.0-grok45-kimik3 · 25542 in / 3854 out tokens · 93940 ms · 2026-07-30T16:22:51.142063+00:00 · methodology

0 comments
read the original abstract

Runtime efficiency is a critical factor that impacts both software quality and user satisfaction. There are many approaches proposed for code optimization to improve runtime efficiency. Traditional code optimization methods operate on intermediate representations (IRs) during compilation for static languages. They are effective but struggle to handle dynamic languages that do not require compilation. Recently, large language models (LLMs) have been leveraged to directly optimize source code in dynamic languages. However, these methods fail to identify suitable optimization targets and usually conduct incomprehensive single-level optimization. To address these challenges, we propose Optimo, a multi-level LLM-based code optimization approach built on a novel Mixture-of-Prompts (MoP) architecture. In the MoP architecture, Optimo identifies time-critical code structures as performance bottlenecks via differential profiling. These structures are then routed to some optimization strategies, akin to expert models in MoE, each tailored to optimize specific code patterns. Unlike traditional approaches that focus only on statement-level optimizations, Optimo operates at four levels of abstraction, ranging from coarse-grained algorithmic improvements to fine-grained optimizations in API usage. We evaluate Optimo on two code efficiency benchmarks, COFFE and Effibench. Our results demonstrate that Optimo achieves an up to 57.48% opt%, i.e., the percentage of optimized programs that are correct and at least 10% faster than the original programs, and an up to 3.97x speedup when optimizing human-written code, and it consistently outperforms the best baseline by up to 96.51% in terms of opt%. Furthermore, Optimo achieves an up to 42.42% opt% and an up to 13.51x speedup when optimizing LLM-generated code.

Figures

Figures reproduced from arXiv: 2607.23665 by David Lo, Jiakun Liu, Jun Wan, Shuzheng Gao, Xiaoxue Ren, Yun Peng.

Figure 1
Figure 1. Figure 1: A motivating example from Coffe. The code in the dashed box is the optimization target for the next level, while the highlighted code is the optimized code from the identified optimization target at the last level. Recent advances in large language models (LLMs) have shown strong capabilities in source code understanding [2, 5, 27, 48] and generation [6, 21, 41, 47], leading to a growing line of work on LL… view at source ↗
Figure 2
Figure 2. Figure 2: The overview of Optimo. It optimizes a program at four levels sequentially: algorithm, statement, expression, and API. At each level except API, the MoP architecture highlighted in the dashed box performs targeted optimization. the program average in both small and large test cases, indicating that they are critical to time efficiency under any circumstance. It then compares the time ratios of these lines … view at source ↗
Figure 3
Figure 3. Figure 3: An initial optimization strategy extracted from a [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The per-instance speedup distribution and Venn [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The distribution of optimized LLM-generated solutions on two benchmarks. “NC”, “NO”, “NH”, “FH” indicate the final [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 6 canonical work pages · 3 internal anchors

  1. [1]

    Thomas Ball and James R Larus. 1996. Efficient path profiling. InProceedings of the 29th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO

    1996

  2. [2]

    Claas Beger and Saikat Dutta. 2025. CoCoNUT: Structural Code Understanding does not fall out of a tree. InProceedings of the 2nd International Workshop on Large ASE ’26, October 12–16, 2026, Munich, Germany Yun Peng, Jun Wan, Jiakun Liu, Shuzheng Gao, David Lo, and Xiaoxue Ren Language Models for Code (LLM4Code). 128–136. doi:10.1109/LLM4CODE66737. 2025.00021

    arXiv 2025

  3. [3]

    François Belias, Leuson Da Silva, Foutse Khomh, and Cyrine Zid. 2025. Perfor- mance Smells in ML and Non-ML Python Projects: A Comparative Study.arXiv preprint arXiv:2504.20224(2025). doi:10.48550/ARXIV.2504.20224

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.20224 2025

  4. [4]

    Binghong Chen, Daniel Tarlow, Kevin Swersky, Martin Maas, Pablo Ariel Heiber, Ashish Naik, Milad Hashemi, and Parthasarathy Ranganathan. 2022. Learning to Improve Code Efficiency.CoRRabs/2208.05297 (2022). arXiv:2208.05297 doi:10.48550/ARXIV.2208.05297

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2208.05297 2022

  5. [5]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning Runtime Behavior of a Program with LLM: How Far Are We?. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE/ACM, 1869–1881. doi:10.1109/ICSE55347.2025.00012

    arXiv 2025

  6. [6]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net. https://openreview.net/forum?id=KuPixIqPiq

    2024

  7. [7]

    Zimin Chen, Sen Fang, and Martin Monperrus. 2024. Supersonic: Learning to Generate Source Code Optimizations in C/C++.IEEE Trans. Software Eng.50, 11 (2024), 2849–2864. doi:10.1109/TSE.2024.3423769

    arXiv 2024

  8. [8]

    The MITRE Corporation. 2024. Performance efficiency CWEs. https://cwe.mitre. org/data/definitions/1132.html https://cwe.mitre.org/data/definitions/1132.html

    2024

  9. [9]

    Ron Cytron, Jeanne Ferrante, Barry K Rosen, Mark N Wegman, and F Kenneth Zadeck. 1991. Efficiently computing static single assignment form and the control dependence graph.ACM Transactions on Programming Languages and Systems (TOPLAS)13, 4 (1991), 451–490. doi:10.1145/115372.115320

    arXiv 1991

  10. [10]

    DeepSeek-AI et al. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] doi:10.48550/ARXIV.2412.19437

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.19437 2025

  11. [11]

    Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A Code Efficiency Benchmark for Code Large Language Models. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Dani...

    work page doi:10.52202/079017-0529 2024

  12. [12]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39

    2022

  13. [13]

    Free Software Foundation

    Inc. Free Software Foundation. 2026. Optimize Options in GCC Compiler. https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

    2026

  14. [14]

    Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. 2024. Search-Based LLMs for Code Optimization. arXiv:2408.12159 [cs.SE] doi:10.48550/ARXIV.2408. 12159

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.12159 2024

  15. [15]

    Clement, Neel Sun- daresan, and Chen Wu

    Spandan Garg, Roshanak Zilouchian Moghaddam, Colin B. Clement, Neel Sun- daresan, and Chen Wu. 2022. DeepDev-PERF: a deep learning-based approach for improving software performance. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Singapore, Singapore)(ESEC/FSE 2022). Ass...

    arXiv 2022

  16. [16]

    Spandan Garg, Roshanak Zilouchian Moghaddam, and Neel Sundaresan

  17. [17]

    Rafail Giavrimis, Alexis Butler, Constantin Cezar Petrescu, Michail Basios, and Santanu Kumar Dash. 2021. Genetic Optimisation of C++ Applications. In36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021. IEEE, 1180–1182. doi:10.1109/ ASE51524.2021.9678650

    arXiv 2021

  18. [18]

    Inc. GitHub. 2022. GitHub Octoverse report on programming lan- guages. https://octoverse.github.com/2022/top-programming-languages https://octoverse.github.com/2022/top-programming-languages

    2022

  19. [19]

    Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie M. Zhang. 2024. EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization. arXiv:2405.15189 [cs.SE] doi:10.52202/ 079017-2684

    Pith/arXiv arXiv 2024

  20. [20]

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. 2024. EffiBench: Benchmarking the Efficiency of Automatically Generated Code.CoRR abs/2402.02037 (2024). arXiv:2402.02037 doi:10.48550/ARXIV.2402.02037

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.02037 2024

  21. [21]

    Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. SelfEvolve: A Code Evo- lution Framework via Large Language Models.CoRRabs/2306.02907 (2023). arXiv:2306.02907 doi:10.48550/ARXIV.2306.02907

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.02907 2023

  22. [22]

    Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. 2012. Understanding and detecting real-world performance bugs.ACM SIGPLAN Notices47, 6 (2012), 77–88. doi:10.1145/2345156.2254075

    arXiv 2012

  23. [23]

    Milan Jovic, Andrea Adamoli, and Matthias Hauswirth. 2011. Catch me if you can: performance bug detection in the wild. InProceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications. 155–170. doi:10.1145/2048066.2048081

    arXiv 2011

  24. [24]

    Rahul Krishna, Md Shahriar Iqbal, Mohammad Ali Javidian, Baishakhi Ray, and Pooyan Jamshidi. 2020. CADET: A Systematic Method For Debugging Mis- configurations using Counterfactual Reasoning.CoRRabs/2010.06061 (2020). arXiv:2010.06061 doi:10.48550/ARXIV.2010.06061

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2010.06061 2020

  25. [25]

    Gall, and Philipp Leitner

    Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner. 2020. Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(Virtual Event, USA)(ESE...

    doi:10.1145/3368089 2020

  26. [26]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668(2020). doi:10.48550/ARXIV.2006.16668

    Pith/arXiv arXiv doi:10.48550/arxiv.2006.16668 2020

  27. [27]

    Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, et al. [n. d.]. Chain of Code: Reasoning with a Language Model-Augmented Code Emulator. InForty-first International Conference on Machine Learning

  28. [28]

    Adrian Nistor, Tian Jiang, and Lin Tan. 2013. Discovering, reporting, and fixing performance bugs. InProceedings of the 10th Working Conference on Mining Software Repositories, MSR ’13, San Francisco, CA, USA, May 18-19, 2013, Thomas Zimmermann, Massimiliano Di Penta, and Sunghun Kim (Eds.). IEEE Computer Society, 237–246. doi:10.1109/MSR.2013.6624035

    arXiv 2013

  29. [29]

    doi:10.1109/MICRO.1996.566449

    IEEE, 46–57. doi:10.1109/MICRO.1996.566449

    arXiv 1996

  30. [30]

    The opensource community. 2025. Line Profiler. https://github.com/pyutils/line_ profiler https://github.com/pyutils/line_profiler

    2025

  31. [31]

    OpenAI. 2024. GPT-4o. https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/

    2024

  32. [32]

    Yun Peng, Shuzheng Gao, Cuiyun Gao, Yintong Huo, and Michael R. Lyu. 2024. Domain Knowledge Matters: Improving Prompts with Fix Templates for Re- pairing Python Type Errors. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 4:1–4:13. doi:10.1145/3597503.3608132

    arXiv 2024

  33. [33]

    International Standard Organization. 2026. ISO/IEC 25010:2023 - Product quality model. https://www.iso.org/standard/78176.html https://www.iso.org/standard/78176.html

    2026

  34. [34]

    Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. COFFE: A Code Efficiency Benchmark for Code Generation.Proc. ACM Softw. Eng.2, FSE, Article FSE012 (June 2025), 24 pages. doi:10.1145/3715727

    doi:10.1145/3715727 2025

  35. [35]

    Yun Peng, Akhilesh Deepak Gotmare, Michael Lyu, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback. arXiv:2412.03578 [cs.SE] doi:10.48550/ARXIV.2412.03578

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.03578 2024

  36. [36]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538(2017). doi:10.48550/ARXIV.1701.06538

    Pith/arXiv arXiv doi:10.48550/arxiv.1701.06538 2017

  37. [37]

    Xiaoxue Ren, Jun Wan, Yun Peng, Zhongxin Liu, Ming Liang, Dajun Chen, Wei Jiang, and Yong Li. 2025. PEACE: Towards Efficient Project-Level Efficiency Optimization via Hybrid Code Editing. In40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 1831–1843. doi:10.1109/ASE63...

    arXiv 2025

  38. [38]

    Matrix Studio. 2024. The Python Codeforces submission dataset. https://huggingface.co/datasets/MatrixStudio/Codeforces-Python-Submissions https://huggingface.co/datasets/MatrixStudio/Codeforces-Python-Submissions

    2024

  39. [39]

    Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh

    Alexander G Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob R. Gardner, Yiming Yang, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. 2024. Learning Performance-Improving Code Edits. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=ix7rLVHXyY

    2024

  40. [40]

    Luca Traini, Vittorio Cortellessa, Daniele Di Pompeo, and Michele Tucci. 2023. Towards effective assessment of steady state performance in Java software: are we there yet?Empirical Softw. Engg.28, 1 (Jan. 2023), 57 pages. doi:10.1007/s10664- 022-10247-x

    doi:10.1007/s10664- 2023

  41. [41]

    Luca Della Toffola, Michael Pradel, and Thomas R. Gross. 2015. Performance problems you can fix: a dynamic analysis of memoization opportunities. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015, J...

    arXiv 2015

  42. [42]

    Mark N Wegman and F Kenneth Zadeck. 1991. Constant propagation with conditional branches.ACM Transactions on Programming Languages and Systems (TOPLAS)13, 2 (1991), 181–210. doi:10.1145/103135.103136

    arXiv 1991

  43. [43]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R. Lyu. 2022. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, Novem...

    arXiv 2022

  44. [44]

    Michael E Wolf and Monica S Lam. 1991. A data locality optimizing algorithm. InProceedings of the ACM SIGPLAN 1991 conference on Programming language design and implementation. 30–44. doi:10.1145/113445.113449

    arXiv 1991

  45. [45]

    Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution. Springer, 196–202. doi:10.1007/978-1-4612-4380-9_16

    doi:10.1007/978-1-4612-4380-9_16 1992

  46. [46]

    Dmitrijs Zaparanuks, Milan Jovic, and Matthias Hauswirth. 2009. Accuracy of performance counter measurements. In2009 IEEE International Symposium on Performance Analysis of Systems and Software. 23–32. doi:10.1109/ISPASS.2009. 4919635

    work page doi:10.1109/ispass.2009 2009

  47. [47]

    Michael E Wolf and Monica S Lam. 1991. A loop transformation theory and an algorithm to maximize parallelism.IEEE transactions on parallel and distributed systems2, 4 (1991), 452–471. doi:10.1109/71.97902

    work page doi:10.1109/71.97902 1991

  48. [48]

    Yuze Zhao, Tianyun Ji, Wenjun Feng, Zhenya Huang, Qi Liu, Zhiding Liu, Yixiao Ma, Kai Zhang, and Enhong Chen. 2025. Unveiling the Magic of Code Rea- soning through Hypothesis Decomposition and Amendment. InThe Thirteenth International Conference on Learning Representations

    2025

  49. [49]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Ju...

    doi:10.18653/v1/ 2023

  50. [51]

    Yutong Zhao, Lu Xiao, Andre B Bondi, Bihuan Chen, and Yang Liu. 2022. A large- scale empirical study of real-life performance issues in open source projects.IEEE Transactions on Software Engineering49, 2 (2022), 924–946. doi:10.1109/TSE.2022. 3167628 Received 2026-03-26; accepted 2026-06-18

    doi:10.1109/tse.2022 2022

  51. [2025]

    arXiv:2306.17077 [cs.SE] doi:10.1109/ICSE-SEIP66354.2025.00017

    RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shot. arXiv:2306.17077 [cs.SE] doi:10.1109/ICSE-SEIP66354.2025.00017

    Pith/arXiv arXiv 2025