Supervised Contrastive Learning
read the original abstract
Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models. Modern batch contrastive approaches subsume or significantly outperform traditional contrastive losses such as triplet, max-margin and the N-pairs loss. In this work, we extend the self-supervised batch contrastive approach to the fully-supervised setting, allowing us to effectively leverage label information. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above the best number reported for this architecture. We show consistent outperformance over cross-entropy on other datasets and two ResNet variants. The loss shows benefits for robustness to natural corruptions and is more stable to hyperparameter settings such as optimizers and data augmentations. Our loss function is simple to implement, and reference TensorFlow code is released at https://t.ly/supcon.
This paper has not been read by Pith yet.
Forward citations
Cited by 16 Pith papers
-
Patient-Aware Contrastive Learning Preserves Per-Patient Structure in RR-Interval Representations
Patient-aware contrastive objective preserves per-patient SR structure in RR-interval embeddings, reaching 0.989 AUROC on patient-independent PAF detection with lower variance than SupCon or BCE baselines.
-
VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models
VISA improves closed-set 3D occupancy mIoU on nuScenes by using VLM instance audits as reliability-weighted semantic supervisors during training of existing world models.
-
Querying Counterfactuals on Tissue Graphs with Supervised Disentanglement
Cellina uses supervised disentanglement to separate cell intrinsic states from spatial contexts for counterfactual predictions on tissue graphs, outperforming baselines on 2.5M+ cells from cancer and brain data.
-
$A^2$: Smaller Self-Supervised ViTs Localize Better than Larger Ones
Smaller self-supervised ViTs localize objects better via attention than larger ViTs, enabling A² to decouple localization from feature extraction for competitive performance on distribution-shifted benchmarks.
-
Template Collapse and Information-Theoretic Limits in Camera rPPG Pulse Morphology Restoration
Empirical tests of 16 architectures on 153 subjects show camera rPPG signals contain no recoverable subject-specific pulse morphology, with all models exhibiting template collapse.
-
Decoding Alignment without Encoding Alignment: A critique of similarity analysis in neuroscience
Decoding alignment metrics can remain high and unchanged even when encoding manifold topology is causally altered, so they do not imply similar function or computation across neural populations.
-
Explainable Disentangled Representation Learning for Generalizable Authorship Attribution in the Era of Generative AI
EAVAE disentangles style from content via contrastive pretraining and an explainable discriminator in a VAE setup, claiming SOTA authorship attribution on multiple datasets and strong few-shot AI text detection.
-
Revisiting Feature Prediction for Learning Visual Representations from Video
V-JEPA models trained only on feature prediction from 2 million public videos achieve 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet-1K using frozen ViT-H/16 backbones.
-
Text and Code Embeddings by Contrastive Pre-Training
Contrastive pre-training on unsupervised data at scale creates text and code embeddings that set new state-of-the-art results on classification and semantic search benchmarks.
-
MARVEL: Margin-Aware Robust von Mises-Fischer Expert Learning for Long-Tailed Out-of-Distribution Detection
MARVEL introduces a multi-expert NvMF-based system with an outlier expert that reduces FPR95 in OOD detection on medical datasets by 8-37%.
-
CLIP-guided Diffusion Model for Backdoor Generation in Sensor-based Human Activity Recognition
A CLIP-guided diffusion model generates backdoor triggers for IMU-based HAR models, achieving successful attacks at a 10% backdoor injection rate.
-
CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmark
CrossView Suite supplies a 1.6M-sample dataset, scene-disjoint benchmark, and explicit-alignment framework to advance MLLMs from single-view perception to cross-view spatial intelligence.
-
Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models
JAM aligns frozen vision and language models via joint autoencoders and multimodal Spread Loss, reliably inducing cross-modal alignment across layer depths, objectives, and model scales.
-
SCENIC: Semantic-Conditioned Edge-Aware Neural Framework for Structured IoT Command Generation
SCENIC framework reports up to 99% exact match on structured IoT command generation using sub-0.2B models, with pruned INT8 versions retaining 91% EM@1 after 25% size reduction.
-
Backdoor Attacks on Fault Detection and Localization in Cyber-Physical Systems
Backdoor attacks succeed against ML fault detection and localization in CPS even when only 10% of the training data is poisoned.
-
LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning
LLMSniffer improves detection of LLM-generated code on GPTSniffer and Whodunit benchmarks by fine-tuning GraphCodeBERT via two-stage supervised contrastive learning plus preprocessing and MLP classification.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.