Pith. sign in

REVIEW 3 cited by

Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2101.06983 v2 pith:CFWLZOAV submitted 2021-01-18 cs.LG cs.CLcs.IR

Luyu Gao , Yunyi Zhang , Jiawei Han , Jamie Callan This is my paper · ORCID
classification cs.LGcs.CLcs.IR
keywords batchcontrastivelearninglossmemorynegativesencoderexample
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example's loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders

    cs.IR 2026-07 conditional novelty 7.0 of 10

    Bekko a8m, with 7.7M active parameters, scores 56.2 on MMTEB Multilingual v2 Retrieval, beating mE5 models and BGE-M3, while a25m reaches 57.5, on par with gte-multilingual-base.

  2. Illuminating Visual Identity in Universal Multimodal Embeddings

    cs.CV 2026-08 conditional novelty 6.0 of 10

    By adding identity-aware sampling and a contrastive loss on a new 28-dataset benchmark, the authors build multimodal embeddings that are far better at visual identity matching without losing general retrieval accuracy.

  3. CLaDMoP: Learning Transferrable Models from Successful Clinical Trials via LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    CLaDMoP pre-trains a two-branch model on successful clinical trials with a pair-matching objective, then fine-tunes it to predict trial success, outperforming prior models on the TOP benchmark.

Pith tools