Pith. sign in

REVIEW 3 cited by

AixBench: A Code Generation Benchmark Dataset

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.13179 v2 pith:RFXSIEZ2 submitted 2022-06-27 cs.SE

Yiyang Hao (1) , Ge Li (2) , Yongqiang Liu (1) , Xiaowei Miao (1) , He Zong (1) , Siyuan Jiang (1) , Yang Liu (1) , He Wei (1) ((1) aiXcoder
show 1 more author
(2) Peking University)
This is my paper · ORCID
classification cs.SE
keywords codedatasetbenchmarkevaluatingevaluationgeneratedgenerationsamples
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present a benchmark dataset for evaluating method-level code generation task. The benchmark contains a dataset of 175 samples for automated evaluation and a dataset of 161 samples for manual evaluation. We also present a new metric for automatically evaluating the correctness of the generated code, and a set of criteria to manually evaluating the overall quality of the generated code.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Across 83 SE benchmarks, average leakage into StarCoder's pretraining data is 4.8% (Python), 2.8% (Java), and 0.7% (C/C++), but QuixBugs and BigCloneBench are 100% and 55.7% leaked.

  2. Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar

    cs.SE 2024-12 conditional novelty 6.0 of 10

    A code-obfuscation benchmark shows LLM code generation pass rates fall sharply when descriptions, code, and dependencies are rewritten to remove training-data familiarity.

  3. The Current Challenges of Software Engineering in the Era of Large Language Models

    cs.SE 2024-12 conditional novelty 4.0 of 10

    The paper reports 26 challenges in LLM-based software engineering, grouped into seven aspects, derived from a structured discussion among 24 academics and practitioners.

Pith tools