REVIEW 7 cited by
Fact or Fiction: Verifying Scientific Claims
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that simple domain adaptation techniques substantially improve performance compared to models trained on Wikipedia or political news. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus. Our experiments indicate that SciFact will provide a challenging testbed for the development of new systems designed to retrieve and reason over corpora containing specialized domain knowledge. Data and code for this new task are publicly available at https://github.com/allenai/scifact. A leaderboard and COVID-19 fact-checking demo are available at https://scifact.apps.allenai.org.
Forward citations
Cited by 7 Pith papers
-
Mutual Linearity in and out of Stationarity for Markov Jump Processes: A Trajectory-Based Approach
Trajectory-level linear response yields mutual linearity of observables under single-edge rate perturbation for Markov jump processes, including non-stationary state and counting observables.
-
ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts
A large-scale benchmark shows that leading multimodal language models still underperform expert humans at verifying climate claims from scientific charts.
-
LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services
LocalSearchBench—1.3M merchant records and 900 multi-hop local-life QA tasks across 9 Chinese cities—shows the best reasoning agent reaches only 35.6% correctness.
-
ScienceMeter: Tracking Scientific Knowledge Updates in Language Models
ScienceMeter evaluates language model knowledge updates across three axes, preservation of old scientific claims, acquisition of new claims, and projection to future findings, and finds all current methods fall short.
-
Hypercube-Based Retrieval-Augmented Generation for Scientific Question-Answering
A document hypercube with LLM-extracted labels and hybrid sparse/dense matching improves retrieval and QA accuracy over RAG baselines on three datasets.
-
CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models
A new 3,008-question assertion-reason benchmark shows language models often use semantic similarity as a substitute for causal explanation, with best models reaching only 0.55 MCC.
-
A New Query Expansion Approach via Agent-Mediated Dialogic Inquiry
AMD uses three LLM agents (Socratic questioning, dialogic answering, reflective feedback) to generate and refine pseudo-answers for query expansion, reporting gains over prior methods on BEIR and TREC benchmarks.
Discussion (0). Continue with ORCID to comment.