REVIEW 14 cited by
DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
In this work, we investigate the potential of large language models (LLMs) based agents to automate data science tasks, with the goal of comprehending task requirements, then building and training the best-fit machine learning models. Despite their widespread success, existing LLM agents are hindered by generating unreasonable experiment plans within this scenario. To this end, we present DS-Agent, a novel automatic framework that harnesses LLM agent and case-based reasoning (CBR). In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle, and facilitate consistent performance improvement through the feedback mechanism. Moreover, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm to adapt past successful solutions from the development stage for direct code generation, significantly reducing the demand on foundational capabilities of LLMs. Empirically, DS-Agent with GPT-4 achieves 100\% success rate in the development stage, while attaining 36\% improvement on average one pass rate across alternative LLMs in the deployment stage. In both stages, DS-Agent achieves the best rank in performance, costing \$1.60 and \$0.13 per run with GPT-4, respectively. Our data and code are open-sourced at https://github.com/guosyjlu/DS-Agent.
Forward citations
Cited by 14 Pith papers
-
Autodata: An agentic data scientist to create high quality synthetic data
Autodata trains meta-optimized AI agents to generate superior synthetic datasets, yielding performance gains over classical methods on CS research, legal, and math reasoning tasks.
-
iML: Executable, Problem-Grounded, and Broadly Exploratory Code-Driven AutoML
iML's contract-based modular code generation with runtime verification reports 85% valid submissions and a 45% medal rate on MLE-BENCH, ahead of MLZero, MLE-STAR, and AutoML-Agent.
-
VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report
A retrieval-augmented LLM approach that identifies vulnerability-related issue reports and CWE types from screenshots and code snippets, improving F1 by 11 points and AUPRC by 20 points over baselines.
-
Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data
Non-programmer business users often fail to spot critical mistakes in AI-generated data analyses, even when explicitly warned and incentivized.
-
EXP-Bench: Can AI Conduct AI Research Experiments?
EXP-Bench is a new benchmark of 461 end-to-end AI research experiments, and leading AI agents complete fewer than 1 percent of them successfully.
-
Aviary: training language agents on challenging scientific tasks
A small open-source LLM trained in the new Aviary environments with expert iteration and majority voting matches or exceeds a frontier LLM agent on SeqQA and LitQA2 at far lower inference cost.
-
Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering
Matryoshka Agent’s orchestrator–sub-agent hierarchy plus tree-ranked RL raises MLE-Dojo HumanRank, letting a 4B orchestrator approach o4-mini and giving a 30B coder up to 36.7% relative gain.
-
PINNsAgent: Automated PDE Surrogation with Large Language Models
An LLM-based multi-agent system that automates PINNs hyperparameter optimization, beating random and Bayesian search on 12 of 14 benchmark PDEs but only matching or beating the PINNacle benchmark on 6 of 14.
-
AI for Auto-Research: Roadmap & User Guide
The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.
-
Large Language Models in the Data Science Lifecycle: A Systematic Mapping Study
A systematic mapping study classifying 66 papers on LLM use across five data science lifecycle stages, finding data analysis most studied and deployment almost ignored.
-
AI4Research: A Survey of Artificial Intelligence for Scientific Research
A survey that organizes AI-for-research work into five tasks, comprehension, survey, discovery, writing, and peer review, and compiles associated tools and benchmarks.
-
A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications
A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.
-
MDSF: Context-Aware Multi-Dimensional Data Storytelling Framework based on Large language Model
MDSF is an LLM-based framework for automated data insight ranking and storytelling that, by its own reported results, does not outperform GPT-4 on ranking and most narrative metrics.
-
Agentic Web: Weaving the Next Web with AI Agents
A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.
Discussion (0). Continue with ORCID to comment.