REVIEW 6 cited by
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Learned representations of scientific documents can serve as valuable input features for downstream tasks without further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks. In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. It includes 24 challenging and realistic tasks, 8 of which are new, across four formats: classification, regression, ranking and search. We then use this benchmark to study and improve the generalization ability of scientific document representation models. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that simple multi-task training fails to improve them. However, a new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance. We experiment with task-format-specific control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over 2 points absolute. We release the resulting family of multi-format models, called SPECTER2, for the community to use and build on.
Forward citations
Cited by 6 Pith papers
-
ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes
Conference accept/reject outcomes yield 15 operational ideation patterns that, as an LLM skill suite, improve automated-judged research-proposal quality over no-skill and generic-skill baselines.
-
From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models
A weak-supervision pipeline using LLM-extracted concepts and rank fusion creates silver-standard patent-to-SDG labels that recover known citation-derived associations and show high network modularity.
-
Translating AI into scientific impact: Field context, career position, and institutional capability in AI-enabled research
Citing AI literature is tied to higher citation impact overall, yet field, career stage, and institutional AI capability redistribute that benefit, with intermediate-capability institutions gaining the most per unit o...
-
SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts
Using multiple LLM-generated summaries of the same abstract as positive pairs trains scientific text embeddings that beat citation-trained baselines on retrieval and clustering, while the new benchmark shares its trai...
-
Extracting Information About Publication Venues Using Citation-Informed Transformers
SPECTER paper embeddings show that some CS venues are nearly indistinguishable and that several venue pairs converged between 2015 and 2023.
-
A Multi-Task Evaluation of LLMs' Processing of Academic Text Input
The abstract reports Gemini underperforms on four academic text tasks, but the attached full text is an unrelated biomedical retrieval paper, leaving the claims unverifiable.
Discussion (0). Sign in to comment.