REVIEW 5 cited by
SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent breakthroughs in Large Language Models (LLMs) have revolutionized scientific literature analysis. However, existing benchmarks fail to adequately evaluate the proficiency of LLMs in this domain, particularly in scenarios requiring higher-level abilities beyond mere memorization and the handling of multimodal data. In response to this gap, we introduce SciAssess, a benchmark specifically designed for the comprehensive evaluation of LLMs in scientific literature analysis. It aims to thoroughly assess the efficacy of LLMs by evaluating their capabilities in Memorization (L1), Comprehension (L2), and Analysis \& Reasoning (L3). It encompasses a variety of tasks drawn from diverse scientific fields, including biology, chemistry, material, and medicine. To ensure the reliability of SciAssess, rigorous quality control measures have been implemented, ensuring accuracy, anonymization, and compliance with copyright standards. SciAssess evaluates 11 LLMs, highlighting their strengths and areas for improvement. We hope this evaluation supports the ongoing development of LLM applications in scientific literature analysis. SciAssess and its resources are available at \url{https://github.com/sci-assess/SciAssess}.
Forward citations
Cited by 5 Pith papers
-
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
StatEval is a new 16,000-question statistics benchmark showing that even strong LLMs score below 60% on research-level statistical proof tasks.
-
Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learning
Reinforcement learning on questions extracted from CRISPR expert forums improves LLM accuracy on a new benchmark (Genome-Bench) by over 15 percentage points.
-
EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs
EarthSE provides a two-level QA benchmark and an open-ended dialogue benchmark for Earth science and shows current LLMs perform poorly on both.
-
Benchmarking Large Language Models on Homework Assessment in Circuit Analysis
A benchmark of GPT-3.5 Turbo, GPT-4o, and Llama 3 70B on five homework assessment metrics for circuit analysis shows the two newer models substantially outperform the older one.
-
From Text to Discovery: How Large Language Models Are Reshaping Research Across Scientific and Humanistic Disciplines
LLMs accelerate research workflows from idea generation to writing but introduce challenges like hallucination, bias, opacity, and ten systemic risks requiring new governance frameworks.
Discussion (0). Sign in to comment.