REVIEW 5 cited by
InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this paper, we introduce InfiAgent-DABench, the first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks. These tasks require agents to end-to-end solving complex tasks by interacting with an execution environment. This benchmark contains DAEval, a dataset consisting of 257 data analysis questions derived from 52 CSV files, and an agent framework which incorporates LLMs to serve as data analysis agents for both serving and evaluation. Since data analysis questions are often open-ended and hard to evaluate without human supervision, we adopt a format-prompting technique to convert each question into a closed-form format so that they can be automatically evaluated. Our extensive benchmarking of 34 LLMs uncovers the current challenges encountered in data analysis tasks. In addition, building on top of our agent framework, we develop a specialized agent, DAAgent, which surpasses GPT-3.5 by 3.9% on DABench. Evaluation datasets and toolkits for InfiAgent-DABench are released at https://github.com/InfiAgent/InfiAgent .
Forward citations
Cited by 5 Pith papers
-
GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks
GABench provides the first agentic graph-analysis benchmark with 10,400 executable tasks, and finds existing LLM agents succeed on under 40% of complex graph tasks.
-
EXP-Bench: Can AI Conduct AI Research Experiments?
EXP-Bench is a new benchmark of 461 end-to-end AI research experiments, and leading AI agents complete fewer than 1 percent of them successfully.
-
Interactive Data Harmonization with LLM Agents: Opportunities and Challenges
An LLM agent can interactively synthesize data harmonization pipelines by orchestrating data integration primitives, as shown by the Harmonia prototype on a clinical oncology dataset.
-
CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents
A decoupled exploration-selection framework for test-time scaling lifts data-science agent accuracy by up to roughly 12 percentage points over matched baselines.
-
AI for Auto-Research: Roadmap & User Guide
The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.
Discussion (0). Continue with ORCID to comment.