REVIEW 7 cited by
LinkBERT: Pretraining Language Models with Document Links
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not capture dependencies or knowledge that span across documents. In this work, we propose LinkBERT, an LM pretraining method that leverages links between documents, e.g., hyperlinks. Given a text corpus, we view it as a graph of documents and create LM inputs by placing linked documents in the same context. We then pretrain the LM with two joint self-supervised objectives: masked language modeling and our new proposal, document relation prediction. We show that LinkBERT outperforms BERT on various downstream tasks across two domains: the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links). LinkBERT is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT sets new states of the art on various BioNLP tasks (+7% on BioASQ and USMLE). We release our pretrained models, LinkBERT and BioLinkBERT, as well as code and data at https://github.com/michiyasunaga/LinkBERT.
Forward citations
Cited by 7 Pith papers
-
ETT-CKGE: Efficient Task-driven Tokens for Continual Knowledge Graph Embedding
ETT-CKGE replaces manual importance scoring in continual knowledge graph embedding with learned token masks, achieving competitive accuracy with much lower training time and memory.
-
BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum
BehaviorSFT uses `reactive` and `proactive` control tokens to fine-tune clinical LLM agents, improving their scores on the authors' new BehaviorBench dataset, though the benchmark is AI-generated and minimally clinici...
-
Error-Aware Curriculum Learning for Biomedical Relation Classification
A teacher-student pipeline in which GPT-4o diagnoses a student's errors, assigns difficulty scores, and generates remediations, then trains a smaller model by curriculum learning, reports new state-of-the-art F1 on fo...
-
Advancing Medical Image Segmentation via Self-supervised Instance-adaptive Prototype Learning
SIPL fuses common and instance-specific prototype proposals and reports the best mean Dice scores on BTCV, Lungs, and BraTS benchmarks.
-
Platform for Representation and Integration of multimodal Molecular Embeddings
An autoencoder-based platform merges nine gene embedding types into one 512-dimensional representation, with a permutation-adjusted SVCCA analysis showing the sources are largely complementary.
-
MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction
MedVQA-TREE fuses three levels of ultrasound image features with UMLS-guided PubMed retrieval to predict sarcopenia, reporting 99% accuracy on a 24-patient proprietary dataset.
-
Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion
A BERT model with LoRA adapters and adaptive layer fusion, fed templated text versions of ICU tabular data, edges out tabular classifiers on AUPRC for 24-hour mortality prediction.
Discussion (0). Continue with ORCID to comment.