Pith. sign in

REVIEW 5 cited by

Multi-lingual Evaluation of Code Generation Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.14868 v3 pith:XZHAFFC6 submitted 2022-10-26 cs.LG cs.CL

Ben Athiwaratkun , Sanjay Krishna Gouda , Zijian Wang , Xiaopeng Li , Yuchen Tian , Ming Tan , Wasi Uddin Ahmad , Shiqi Wang
show 17 more authors
Qing Sun Mingyue Shang Sujan Kumar Gonugondla Hantian Ding Varun Kumar Nathan Fulton Arash Farahani Siddhartha Jain Robert Giaquinto Haifeng Qian Murali Krishna Ramanathan Ramesh Nallapati Baishakhi Ray Parminder Bhatia Sudipta Sengupta Dan Roth Bing Xiang
This is my paper · ORCID
classification cs.LGcs.CL
keywords codemodelsgenerationlanguagesbenchmarksdatasetslanguagemulti-lingual
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 28 citations worldwide. Full citation record

  1. The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages

    cs.SE 2026-07 conditional novelty 6.0 of 10

    For five coding agents on 100 competitive-programming tasks, output-token cost varies sharply by language — OCaml costs roughly 1.3–1.7x Python — even after controlling for problem difficulty.

  2. Learning from Execution: Self-Evolving Memory for Private-Library Code Generation

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    MEMCoder boosts LLM code generation for private libraries by 16.31% pass@1 via a multi-dimensional evolving memory that distills usage guidelines from execution feedback and combines them with static docs.

  3. SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation

    cs.SE 2025-07 conditional novelty 6.0 of 10

    All 18 evaluated LLMs pass fewer SIMD-intrinsic code-generation tests than scalar-code tests on the new SimdBench benchmark, with the largest drops on SVE and RVV.

  4. Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A new dataset and a token-masked preference optimization loss reduce security vulnerabilities in LLM-generated Python code while preserving code quality.

  5. In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code

    cs.SE 2025-07 conditional novelty 4.0 of 10

    Few-shot in-context examples improve LLM-based functional correctness estimation for generated code relative to zero-shot judgment, but the gains are modest and uneven.

Pith tools