REVIEW 12 cited by
Evaluating Language Models for Efficient Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce Differential Performance Evaluation (DPE), a framework designed to reliably evaluate Large Language Models (LLMs) for efficient code generation. Traditional coding benchmarks often fail to provide reliable insights into code efficiency, due to their reliance on simplistic test inputs and the absence of effective compound metrics. DPE addresses these issues by focusing on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. DPE operates in two phases: To curate efficiency datasets, it selects efficiency-demanding tasks from existing coding benchmarks and generates computationally expensive inputs to stress the efficiency of LLM solutions. To assess the code efficiency, DPE profiles the new solution and compares it globally against a set of reference solutions that exhibit distinct efficiency levels, where the matched level defines its efficiency score. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks. Our comprehensive evaluation draws interesting findings on the efficiency impact of model sizes, instruction tuning, and prompting. For example, while the scaling law fails to account for code efficiency, general instruction tuning benefits both code correctness and efficiency. We also evaluate the evaluation by examining the effectiveness of DPE, showing that EvalPerf is reliable and convenient to use even across platforms.
Forward citations
Cited by 12 Pith papers
-
SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
SWE-NFI measures coding agents' non-functional code improvements with 92 deterministic rules and 188 real GitHub tasks, finding agents lag humans most on structural logic-pattern improvements.
-
JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java
JETO-Mine is a reusable three-phase pipeline that mines 1.8 million Java commits to produce JETO-Bench containing 91 verified executable ETIPs, on which OpenHands succeeds at 14.3%.
-
PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization
A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.
-
Rethinking Code Performance Benchmarks for LLMs
Re-evaluating four LLM code-efficiency benchmarks with 30-run statistical testing shows 93.89% of 'performant' implementations are indistinguishable from baselines; a multi-agent test-generation framework reveals hidd...
-
Locus: Agentic Predicate Synthesis for Directed Fuzzing
Locus uses an LLM agent to synthesize and validate intermediate predicates that make directed fuzzing reach target bug states faster, reporting an average 41.6x speedup across eight fuzzers.
-
SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?
A repository-level benchmark shows LLMs and agents currently achieve far smaller, statistically verified speedups than expert human patches on 140 real-world Python performance tasks.
-
Evaluating Efficiency and Novelty of LLM-Generated Code for Graph Analysis
In a benchmark of LLM-generated C code for graph analysis, Claude Sonnet 4 Extended produced the most correct, fastest, and most memory-efficient implementations, beating human baselines on triangle counting.
-
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
Reinforcement learning with execution feedback enables a code model to iteratively improve the efficiency of its own generated code, surpassing supervised and preference-based training methods.
-
SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents
SWE-MiniSandbox replaces container-based isolation with kernel-backed workspaces and pre-caching for RL-based SWE agents, cutting disk usage to 5% and setup time to 25% of baselines while matching evaluation performance.
-
dots.llm1 Technical Report
A 14B-active MoE model roughly matches Qwen2.5-72B on a broad benchmark suite while reporting about a 4x reduction in training GPU-hours.
-
Think Only When You Need with Large Hybrid-Reasoning Models
The authors introduce a two-stage training pipeline (supervised fine-tuning plus reinforcement learning) that lets a language model adaptively choose between a reasoning mode and a direct-answer mode, and a new metric...
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Continue with ORCID to comment.