REVIEW 8 cited by
CodeJudge: Evaluating Code Generation with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have shown promising performance in code generation. However, how to reliably evaluate code generated by LLMs remains an unresolved problem. This paper presents CodeJudge, a code evaluation framework that leverages LLMs to evaluate the semantic correctness of generated code without the need for test cases. We investigate different ways to guide the LLM in performing "slow thinking" to arrive at an in-depth and reliable evaluation. We experimented with four LLMs as evaluators on four code generation datasets and five programming languages. The results show that CodeJudge significantly outperformed existing methods in most settings. Furthermore, compared with a SOTA GPT-3.5-based code evaluation method, CodeJudge achieved better results even when using a much smaller model, Llama-3-8B-Instruct. Our code and datasets are available on GitHub https://github.com/VichyTong/CodeJudge.
Forward citations
Cited by 8 Pith papers
-
SEDCoT: Enhancing LLM-Based COBOL Code Translation via Symbolic Execution and Delta Debugging
SEDCoT combines LLM translation, symbolic-execution test generation, and delta-debugging repair to raise COBOL-to-C correctness by ≥12% over SOTA LLM baselines while preserving human-readable output.
-
Evaluating Efficiency and Novelty of LLM-Generated Code for Graph Analysis
In a benchmark of LLM-generated C code for graph analysis, Claude Sonnet 4 Extended produced the most correct, fastest, and most memory-efficient implementations, beating human baselines on triangle counting.
-
FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation
A new benchmark adds 148 interactive front-end development tasks with automated sandbox tests, reporting a 90.54% agreement rate with human evaluation across four LLMs.
-
Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
ROSI bakes the refusal direction into a model's weight matrices via a rank-one update, raising refusal and jailbreak robustness with minimal measured utility cost.
-
Learning Bug Context for PyTorch-to-JAX Translation with LLMs
T2J uses 163 human-annotated bug–fix pairs from 20 PyTorch-to-JAX translations to prompt gpt-4o-mini, improving its JAX output on several LLM-judged metrics while showing mixed results on CodeBLEU.
-
Towards High Supervised Learning Utility Training Data Generation: Data Pruning and Column Reordering
PRRO combines signal-based data pruning and column reordering to improve the supervised learning utility of synthetic tabular data, but its evaluation is undermined by data manipulation and an ill-defined correlation measure.
-
Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
Prompt rewrites of LeetCode problems cause large accuracy swings in nine LLMs, but invalid negation test cases and inconsistent tables make the headline numbers unreliable.
-
Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy
A five-task case study shows DSPy prompt optimization can improve LLM accuracy on some tasks, notably contradiction detection (46.2% to 64.0%), but results vary and no code or data are released.
Discussion (0). Sign in to comment.