REVIEW 7 cited by
Planning In Natural Language Improves LLM Search For Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While scaling training compute has led to remarkable improvements in large language models (LLMs), scaling inference compute has not yet yielded analogous gains. We hypothesize that a core missing component is a lack of diverse LLM outputs, leading to inefficient search due to models repeatedly sampling highly similar, yet incorrect generations. We empirically demonstrate that this lack of diversity can be mitigated by searching over candidate plans for solving a problem in natural language. Based on this insight, we propose PlanSearch, a novel search algorithm which shows strong results across HumanEval+, MBPP+, and LiveCodeBench (a contamination-free benchmark for competitive coding). PlanSearch generates a diverse set of observations about the problem and then uses these observations to construct plans for solving the problem. By searching over plans in natural language rather than directly over code solutions, PlanSearch explores a significantly more diverse range of potential solutions compared to baseline search methods. Using PlanSearch on top of Claude 3.5 Sonnet achieves a state-of-the-art pass@200 of 77.0% on LiveCodeBench, outperforming both the best score achieved without search (pass@1 = 41.4%) and using standard repeated sampling (pass@200 = 60.6%). Finally, we show that, across all models, search algorithms, and benchmarks analyzed, we can accurately predict performance gains due to search as a direct function of the diversity over generated ideas. Code can be found at https://github.com/scaleapi/plansearch.
Forward citations
Cited by 7 Pith papers
-
Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
Fine-tuning LLMs on an unseen language teaches syntax but fails to transfer semantic competence, leaving Python with up to a 19% performance advantage and no tested intervention closing the gap.
-
StaAgent: An Agentic Framework for Testing Static Analyzers
An LLM-powered four-agent framework performs metamorphic testing on static analyzers and reports 64 faulty rule implementations across SpotBugs, SonarQube, ErrorProne, Infer, and PMD.
-
Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards
A guidance-augmented RLVR framework improves software engineering agent pass@1 on SWE-Bench Verified from 9.4% to 22.4%, with guidance being the critical component.
-
Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs
The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.
-
Optimizing Sequential Multi-Step Tasks with Parallel LLM Agents
Running multiple multi-agent teams in parallel, stopping at the first successful answer, cuts end-to-end latency by up to 2.2x with roughly unchanged completion rates on GAIA.
-
EvoVerilog: Large Langugage Model Assisted Evolution of Verilog Code
EvoVerilog uses multiobjective evolutionary search with LLMs to generate Verilog code, reporting higher pass@10 than prior methods on VerilogEval-Machine and VerilogEval-Human.
-
A Survey on Large Language Models for Mathematical Reasoning
Recent advances in LLM mathematical reasoning are organized into comprehension and generation phases, covering methods from prompting to test-time scaling.
Discussion (0). Sign in to comment.