Pith. sign in

REVIEW 11 cited by

Learning Performance-Improving Code Edits

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.07867 v5 pith:WA3S3WIY submitted 2023-02-15 cs.SE cs.AIcs.LGcs.PF

Alexander Shypula , Aman Madaan , Yimeng Zeng , Uri Alon , Jacob Gardner , Milad Hashemi , Graham Neubig , Parthasarathy Ranganathan
show 2 more authors
Osbert Bastani Amir Yazdanbakhsh
This is my paper · ORCID
classification cs.SEcs.AIcs.LGcs.PF
keywords codefastestoptimizationsprogramdataseteditsgenerationshigh-level
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the decline of Moore's law, optimizing program performance has become a major focus of software research. However, high-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code. Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks. To that end, we introduce a framework for adapting LLMs to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs, accompanied by extensive unit tests. A major challenge is the significant variability of measuring performance on commodity hardware, which can lead to spurious "improvements." To isolate and reliably evaluate the impact of program optimizations, we design an environment based on the gem5 full system simulator, the de facto simulator used in academia and industry. Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play. A combination of these techniques achieves a mean speedup of 6.86 with eight generations, higher than average optimizations from individual programmers (3.66). Using our model's fastest generations, we set a new upper limit on the fastest speedup possible for our dataset at 9.64 compared to using the fastest human submissions available (9.56).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Source and Cross-Scenario Strategy-Guided Code Optimization

    cs.SE 2026-07 conditional novelty 7.0 of 10

    MoST improves LLM-guided code optimization by clustering optimization strategies from heterogeneous knowledge sources and transferring them across programming languages.

  2. JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java

    cs.SE 2026-06 conditional novelty 7.0 of 10

    JETO-Mine is a reusable three-phase pipeline that mines 1.8 million Java commits to produce JETO-Bench containing 91 verified executable ETIPs, on which OpenHands succeeds at 14.3%.

  3. RLPF: Reinforcement Learning from Performance Feedback for Code Generation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RLPF's staged performance reward lifts Qwen3-32B on PerfCodeBench from 11.1% to 54.6% correct-and-runnable and from 8.1% to 38.6% relative efficiency.

  4. PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.

  5. SemOpt: LLM-Driven Code Optimization via Rule-Based Analysis

    cs.SE 2025-10 conditional novelty 6.0 of 10

    SemOpt generates Semgrep static-analysis rules from LLM-summarized optimization commits and uses them to locate and apply optimization strategies, outperforming retrieval-based baselines on C/C++ code.

  6. TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewards

    cs.CL 2025-07 conditional novelty 6.0 of 10

    TRPrompt trains an 8B prompt model directly on natural-language textual rewards and reports the highest accuracies on GSMHard and MATH among the compared methods.

  7. Rethinking LLM-Based RTL Code Optimization Via Timing Logic Metamorphosis

    cs.SE 2025-07 reject novelty 6.0 of 10

    LLM-based RTL optimizers degrade on timing-heavy mutants, but the study's own data and methods do not fully support the headline claim.

  8. SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation

    cs.SE 2025-07 conditional novelty 6.0 of 10

    All 18 evaluated LLMs pass fewer SIMD-intrinsic code-generation tests than scalar-code tests on the new SimdBench benchmark, with the largest drops on SVE and RVV.

  9. Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

    cs.SE 2025-05 conditional novelty 6.0 of 10

    Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.

  10. AuPair: Golden Example Pairs for Code Repair

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Providing a different submodularly selected (guess, fix) example on each repair call improves LLM code repair over best-of-N and self-repair.

  11. Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.

Pith tools