REVIEW 10 cited by
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models have demonstrated the ability to generate both natural language and programming language text. Such models open up the possibility of multi-language code generation: could code generation models generalize knowledge from one language to another? Although contemporary code generation models can generate semantically correct Python code, little is known about their abilities with other languages. We propose MultiPL-E, a system for translating unit test-driven code generation benchmarks to new languages. We create the first massively multilingual code generation benchmark by using MultiPL-E to translate two popular Python code generation benchmarks to 18 additional programming languages. We use MultiPL-E to extend the HumanEval benchmark and MBPP benchmark to 18 languages that encompass a range of programming paradigms and popularity. Using these new parallel benchmarks, we evaluate the multi-language performance of three state-of-the-art code generation models: Codex, CodeGen, and InCoder. We find that Codex matches or even exceeds its performance on Python for several other languages. The range of programming languages represented in MultiPL-E allow us to explore the impact of language frequency and language features on model performance. Finally, the MultiPL-E approach of compiling code generation benchmarks to new programming languages is both scalable and extensible, making it straightforward to evaluate new models, benchmarks, and languages.
Forward citations
Cited by 10 Pith papers
-
FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
An execution-graded benchmark built from FindStat shows that current LLMs plateau near 50% on exact combinatorial code synthesis, with extra examples sometimes hurting and open-weight models matching closed ones.
-
SCOPE: Synthetic Conditional Objectives for Policy Evolution in Black-Box Combinatorial Optimization
SCOPE evolves LLM-generated auxiliary objective functions and selects a validated portfolio of them to guide fixed combinatorial search engines under strict black-box query budgets.
-
Think Before You Code: Dual Reasoning for the NLSafety-Utility Trade-Off in LLM Code Generation
Dual Reasoning with explicit safety audits improves the new SUDS metric by 1.32x to 3.42x over baselines on code generation benchmarks containing injected harmful keywords.
-
AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.
-
Coding Triangle: How Does Large Language Model Understand Code?
The Coding Triangle framework evaluates LLMs on editorials, code, and test cases, revealing that models are self-consistent yet lack diversity and that model mixtures improve robustness.
-
AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length
AdaptiveLLM routes coding problems to cost-effective LLMs by clustering chain-of-thought lengths into three difficulty levels, improving pass@1 by 7.86% over ComplexityNet while cutting inference cost by 88.9%.
-
SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
SwiftEval, a 28-problem hand-crafted Swift benchmark, evaluates 44 code LLMs and shows large performance drops on Swift tasks, especially for smaller models.
-
Context-Aware CodeLLM Eviction for AI-assisted Coding
CACE, a context-aware eviction policy, cuts code-model reloads and response latency in self-hosted AI-assistant serving compared with LRU, though its future-demand factor reads the actual test workload.
-
Towards a Science of Causal Interpretability in Deep Learning for Software Engineering
The dissertation presents docode, a causal interpretability method for neural code models, and uses a case study to show that some correlations between code properties and model performance are confounded rather than causal.
-
Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead
A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.
Discussion (0). Continue with ORCID to comment.