Pith. sign in

REVIEW 2 cited by

CodeQA: A Question Answering Dataset for Source Code Comprehension

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2109.08365 v1 pith:BT2BO6RJ submitted 2021-09-17 cs.CL cs.SE

Chenxiao Liu , Xiaojun Wan This is my paper · ORCID
classification cs.CLcs.SE
keywords datasetcodecomprehensionquestionansweringcodeqapairsquestion-answer
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose CodeQA, a free-form question answering dataset for the purpose of source code comprehension: given a code snippet and a question, a textual answer is required to be generated. CodeQA contains a Java dataset with 119,778 question-answer pairs and a Python dataset with 70,085 question-answer pairs. To obtain natural and faithful questions and answers, we implement syntactic rules and semantic analysis to transform code comments into question-answer pairs. We present the construction process and conduct systematic analysis of our dataset. Experiment results achieved by several neural baselines on our dataset are shown and discussed. While research on question-answering and machine reading comprehension develops rapidly, few prior work has drawn attention to code question answering. This new dataset can serve as a useful research benchmark for source code comprehension.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering

    cs.SE 2024-12 conditional novelty 6.0 of 10

    The paper presents CodeRepoQA, a large-scale multi-turn repository-level QA benchmark, and evaluates ten LLMs on it.

  2. A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models

    cs.SE 2024-11 conditional novelty 6.0 of 10

    FAUN-Eval is a curated benchmark of 300 real GitHub issue-PR pairs that scores LLMs separately on question answering, fault localization, and code editing.

Pith tools