Pith. sign in

REVIEW 4 cited by

Natural Language to Code Generation in Interactive Data Science Notebooks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2212.09248 v1 pith:JEVYTYU2 submitted 2022-12-19 cs.CL cs.SE

Pengcheng Yin , Wen-Ding Li , Kefan Xiao , Abhishek Rao , Yeming Wen , Kensen Shi , Joshua Howland , Paige Bailey
show 4 more authors
Michele Catasta Henryk Michalewski Alex Polozov Charles Sutton
This is my paper · ORCID
classification cs.CLcs.SE
keywords codedatanotebookslanguagemodelarcadecomputationalgeneration
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowledge-Enhanced Program Repair for Data Science Code

    cs.SE 2025-02 conditional novelty 6.0 of 10

    DSrepair combines a knowledge graph of data science APIs with AST-level bug localization to repair LLM-generated code, fixing more DS-1000 tasks than five baseline repair methods.

  2. CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories

    cs.SE 2025-02 conditional novelty 6.0 of 10

    CSR-Bench and CSR-Agents show that LLM agents can complete under half of setup and data-download steps, and between 15 and 29 percent of training, inference, and evaluation steps, on 100 research repositories.

  3. In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code

    cs.SE 2025-07 conditional novelty 4.0 of 10

    Few-shot in-context examples improve LLM-based functional correctness estimation for generated code relative to zero-shot judgment, but the gains are modest and uneven.

  4. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

Pith tools