REVIEW 7 cited by
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State-of-the-art natural language understanding classification models follow two-stages: pre-training a large language model on an auxiliary task, and then fine-tuning the model on a task-specific labeled dataset using cross-entropy loss. However, the cross-entropy loss has several shortcomings that can lead to sub-optimal generalization and instability. Driven by the intuition that good generalization requires capturing the similarity between examples in one class and contrasting them with examples in other classes, we propose a supervised contrastive learning (SCL) objective for the fine-tuning stage. Combined with cross-entropy, our proposed SCL loss obtains significant improvements over a strong RoBERTa-Large baseline on multiple datasets of the GLUE benchmark in few-shot learning settings, without requiring specialized architecture, data augmentations, memory banks, or additional unsupervised data. Our proposed fine-tuning objective leads to models that are more robust to different levels of noise in the fine-tuning training data, and can generalize better to related tasks with limited labeled data.
Forward citations
Cited by 7 Pith papers
-
Climber-Pilot: A Non-Myopic Generative Recommendation Model Towards Better Instruction-Following
A generative recommender combining time-masked multi-item training with condition-guided sparse attention improves long-horizon retrieval and instruction following, with a 4.24% like-rate lift online.
-
Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Directly predicting whether annotators will disagree on a value label outperforms inferring disagreement from per-annotator value predictions on the Touché23-ValueEval dataset.
-
Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning
A two-stage fine-tuning and reinforcement-learning method makes LLMs generate token-efficient natural-language search plans, reporting strong accuracy gains on financial and news search benchmarks.
-
SEFRQO: A Self-Evolving Fine-Tuned RAG-Based Query Optimizer
SEFRQO claims a self-evolving fine-tuned LLM with retrieval and execution feedback reduces query latency versus PostgreSQL, but the provided body is a different paper, blocking verification.
-
MetaExplainer: A Framework to Generate Multi-Type User-Centered Explanations for AI Systems
A neuro-symbolic pipeline decomposes user questions, delegates to model explainers, and synthesizes natural-language explanations, achieving moderate stage-wise scores on a diabetes dataset.
-
Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small Language Models
HomeLLaMA fine-tunes a local 8B language model on cloud-generated smart home data to provide personalized, privacy-preserving device control, with optional obfuscated cloud queries when the local model falls short.
-
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.
Discussion (0). Continue with ORCID to comment.