REVIEW 9 cited by
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large pre-trained code generation models, such as OpenAI Codex, can generate syntax- and function-correct code, making the coding of programmers more productive and our pursuit of artificial general intelligence closer. In this paper, we introduce CodeGeeX, a multilingual model with 13 billion parameters for code generation. CodeGeeX is pre-trained on 850 billion tokens of 23 programming languages as of June 2022. Our extensive experiments suggest that CodeGeeX outperforms multilingual code models of similar scale for both the tasks of code generation and translation on HumanEval-X. Building upon HumanEval (Python only), we develop the HumanEval-X benchmark for evaluating multilingual models by hand-writing the solutions in C++, Java, JavaScript, and Go. In addition, we build CodeGeeX-based extensions on Visual Studio Code, JetBrains, and Cloud Studio, generating 4.7 billion tokens for tens of thousands of active users per week. Our user study demonstrates that CodeGeeX can help to increase coding efficiency for 83.4% of its users. Finally, CodeGeeX is publicly accessible and in Sep. 2022, we open-sourced its code, model weights (the version of 850B tokens), API, extensions, and HumanEval-X at https://github.com/THUDM/CodeGeeX.
Forward citations
Cited by 9 Pith papers
-
IFEvalCode: Controlled Code Generation
A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.
-
Enhancing Project-Specific Code Completion by Inferring Internal API Information
A retrieval-augmented code completion method that infers project-internal APIs from a rough draft and a static knowledge base, beating existing repo-level baselines.
-
On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization
Even the best tested LLM judge, GPT-4-turbo, frequently misjudges code correctness (e.g., 50% of wrong Java functions judged correct) but agrees moderately with humans when judging code summary quality.
-
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
All 18 evaluated LLMs pass fewer SIMD-intrinsic code-generation tests than scalar-code tests on the new SimdBench benchmark, with the largest drops on SVE and RVV.
-
Turning the Tide: Repository-based Code Reflection
A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
MPL: Multiple Programming Languages with Large Language Models for Information Extraction
Using multiple programming languages as code-style prompts during fine-tuning improves LLM information extraction accuracy over single-language prompting.
-
One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs
ArrAttack fine-tunes a judge on the SmoothLLM defense, uses it to filter rewriting-attack data, and trains a generator that produces jailbreak prompts transferring across defenses.
-
Salamandra Technical Report
Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.
Discussion (0). Continue with ORCID to comment.