REVIEW 9 cited by
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Recently, pre-trained large language models (LLMs) have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. However, the evaluation of these models has often been performed in a scattered way on only one or two specific tasks, in a few languages, at a partial granularity (e.g., function) level, and in many cases without proper training data. Even more concerning is that in most cases the evaluation of generated codes has been done in terms of mere lexical overlap with a reference code rather than actual execution. We introduce xCodeEval, the largest executable multilingual multitask benchmark to date consisting of $25$M document-level coding examples ($16.5$B tokens) from about $7.5$K unique problems covering up to $11$ programming languages with execution-level parallelism. It features a total of $7$ tasks involving code understanding, generation, translation and retrieval. xCodeEval adopts an execution-based evaluation and offers a multilingual code execution engine, ExecEval that supports unit test based execution in all the $11$ languages. To address the challenge of balancing the distributions of text-code samples over multiple attributes in validation/test sets, we propose a novel data splitting and a data selection schema based on the geometric mean and graph-theoretic principle. Our experiments with OpenAI's LLMs (zero-shot) and open-LLMs (zero-shot and fine-tuned) on the tasks and languages demonstrate **xCodeEval** to be quite challenging as per the current advancements in language models.
Forward citations
Cited by 9 Pith papers
-
Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study
LLM-generated programs, especially when paired across languages or with human code, produce measurable 1-out-of-2 reliability gains under classical software-diversity analysis on three programming-contest specifications.
-
CityRAG: Stepping Into a City via Spatially-Grounded Video Generation
PlayCoder combines a repository-aware coding agent with a vision-based GUI testing agent and an automated program repair loop to detect and fix silent logic errors in LLM-generated interactive application code.
-
Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking
A data synthesis framework generates four syntax/semantics code pair types, and fine-tuning embedding models on the resulting datasets improves code clone detection, functional consistency, and retrieval.
-
MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
MRG-Bench is a multi-language, executable repository-level code generation benchmark on which current LLMs and RAG methods perform poorly, with the dominant failure mode being poor understanding of user requirements.
-
CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors
MRG-Bench is a multi-language, runnable-test benchmark showing that LLMs' main failure in repository-level code generation is understanding user requirements.
-
IFEvalCode: Controlled Code Generation
A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.
-
Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure
TestCase-Eval introduces fault coverage and fault exposure tasks over 500 Codeforces problems and finds that 19 LLMs, led by Qwen3-32B, remain far below human expert performance.
-
HPCTransCompile: An AI Compiler Generated Dataset for High-Performance CUDA Transpilation and LLM Preliminary Exploration
A TVM-based pipeline generates 20,000 CUDA-CPU code pairs, and fine-tuning code LLMs on this data improves transpilation success and CPU performance, with an average speedup improvement of 43.8%.
-
Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead
A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.
Discussion (0). Sign in to comment.