REVIEW 15 cited by
PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) generalize well across language tasks, but suffer from hallucinations and uninterpretability, making it difficult to assess their accuracy without ground-truth. Retrieval-Augmented Generation (RAG) models have been proposed to reduce hallucinations and provide provenance for how an answer was generated. Applying such models to the scientific literature may enable large-scale, systematic processing of scientific knowledge. We present PaperQA, a RAG agent for answering questions over the scientific literature. PaperQA is an agent that performs information retrieval across full-text scientific articles, assesses the relevance of sources and passages, and uses RAG to provide answers. Viewing this agent as a question answering model, we find it exceeds performance of existing LLMs and LLM agents on current science QA benchmarks. To push the field closer to how humans perform research on scientific literature, we also introduce LitQA, a more complex benchmark that requires retrieval and synthesis of information from full-text scientific papers across the literature. Finally, we demonstrate PaperQA's matches expert human researchers on LitQA.
Forward citations
Cited by 15 Pith papers
-
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
NatureBench evaluates ten frontier AI coding agents on 90 tasks from Nature papers under web-search-disabled conditions and finds the strongest agent surpasses published SOTA on only 17.8% of tasks, succeeding mainly ...
-
AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs
AISE-Bench is a real-user-query benchmark with annotated API trajectories and grounded answers that exposes LLM agents' weak performance in multi-step academic information seeking.
-
EXP-Bench: Can AI Conduct AI Research Experiments?
EXP-Bench is a new benchmark of 461 end-to-end AI research experiments, and leading AI agents complete fewer than 1 percent of them successfully.
-
Single-agent or Multi-agent Systems? Why Not Both?
On 15 agentic benchmarks, the accuracy advantage of multi-agent LLM systems over single-agent systems mostly disappears with stronger base models, and a hybrid single/multi-agent cascade improves accuracy and cuts cost.
-
Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models
A new benchmark (TruthHypo) and a knowledge-grounded hallucination detector (KnowHD) show that grounding scores can partially select truthful LLM-generated biomedical hypotheses, but the result is at risk from knowled...
-
VecTree-RAG: An Agentic Retrieval-Augmented Generation Framework Combining Vector and Tree Retrieval for Efficiency and Accuracy
An agentic RAG framework that splits corpus-level discovery (vector retrieval) from within-paper evidence localization (tree navigation) reports top scores on QASPER, LitQA2, and a new MOSAIC benchmark.
-
VASP Agent: An Agentic Framework for Autonomous First-principles Calculations
An LLM-driven agent with predefined VASP workflows and parameter-checking tools completes DFT simulation tasks more reliably and accurately than standalone LLMs, with a new 80-task benchmark.
-
GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models
An LLM can memorize a knowledge graph into LoRA weights and answer relation/reasoning queries about it without graph context, but the evaluation partly trains on the test task.
-
DeepResearch$^{\text{Eco}}$: A Recursive Agentic Workflow for Complex Scientific Question Answering in Ecology
A recursive agentic pipeline for literature synthesis showing a 21-fold source increase and 14.9-fold density gain when depth and breadth are raised.
-
AI for Auto-Research: Roadmap & User Guide
The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.
-
How Far Are AI Scientists from Changing the World?
This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.
-
Open-Source Agentic Hybrid RAG Framework for Scientific Literature Review
A DPO-tuned agentic hybrid RAG system that routes queries between a knowledge graph and a vector store beat a static baseline on a self-generated benchmark.
-
Reading Between the Timelines: RAG for Answering Diachronic Questions
TA-RAG uses LLM-extracted time intervals, time-filtered retrieval with averaged temporal query embeddings, and chronologically ordered context to beat standard RAG by 13-27 points on the new ADQAB benchmark of 525 mul...
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
-
On the Comprehensibility of Multi-structured Financial Documents using LLMs and Pre-processing Tools
Preprocessing financial PDFs into text, tables, and chart data with existing tools improves LLM question-answering accuracy over direct GPT-4o image input in a small private evaluation.
Discussion (0). Continue with ORCID to comment.