On the Out-of-Distribution Generalization of Self-Supervised Learning

Changwen Zheng; Jiangmeng Li; Jingyao Wang; Wenwen Qiang; Zeen Song

REVIEW 4 major objections 4 minor 35 references

On the Out-of-Distribution Generalization of Self-Supervised Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read If each self-supervised training batch satisfies PID, the SSL model is minimax optimal for worst-case OOD performance; the paper proposes a balancing-score sampler to build such batches.

desk verdict New batch-sampling method with broad empirical gains, but the minimax theorem is assumed, not proven; worth a serious referee to separate the useful heuristics from the unearned theory. read the letter →

arxiv 2505.16675 v1 pith:NHMZJXEK submitted 2025-05-22 cs.LG

Wenwen Qiang , Jingyao Wang , Zeen Song , Jiangmeng Li , Changwen Zheng This is my paper · ORCID

classification cs.LG

keywords self-supervisedlearningout-of-distributiongeneralizationspuriouscorrelationpost-interventiondistributionbalancingscorecausalinferencemini-batchsamplinglatentvariablemodel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Self-supervised learning (SSL) transfers well, but it also absorbs spurious correlations—background, texture, or style cues that line up with the anchor in a training batch—and these hurt out-of-distribution (OOD) performance. This paper argues that the cure is to train on batches drawn from a post-intervention distribution (PID), in which the spurious factor $s$ is independent of the anchor $x_{\mathrm{label}}$. The central result, Theorem 3.4, says that if a model minimizes the SSL objective on every PID batch, it is minimax optimal over all environments: no model trained on other batch distributions can achieve a better worst-case OOD loss. The paper then constructs such batches with a latent variable model that estimates $s$ and a balancing-score matcher that pairs samples with the same spurious profile, and Theorem 4.7 shows that perfect matching yields a PID batch. Experiments across unsupervised, semi-supervised, transfer, and few-shot benchmarks report consistent gains when this sampler replaces random mini-batch construction.

What carries the argument

The load-bearing object is the post-intervention distribution (PID), a joint distribution $p(x_+, x_{\mathrm{label}}, s) = p(x_+ | x_{\mathrm{label}}, s)\,p(x_{\mathrm{label}})\,p(s)$ obtained by severing the spurious link between $s$ and the anchor while keeping the causal mechanism that generates the positive sample. The sampling side rests on the balancing score $\mathrm{ba}(s)$, a function of the latent spurious factor that makes $s$ and $x_{\mathrm{label}}$ conditionally independent given $\mathrm{ba}(s)$, computed in Equation (5) from propensity scores $p_e(x_{\mathrm{label},j} | s)$ of an exponential-family latent variable model. The proof that PID batches give minimax OOD optimality (Theorem 3.4) and the proof that exact balancing-score matching yields a PID batch (Theorem 4.7) connect the causal definition to the practical Algorithm 1.

What would settle it

Take a dataset with a known, controllable spurious factor (for example ColoredMNIST with a tunable digit–color correlation), train the proposed sampler, and check whether the true $s$ remains correlated with the anchor within matched batches; then compare worst-group accuracy of the PID-trained model against a model trained on batches matched using the oracle spurious factor. If perfect matching on the learned $\mathrm{ba}(s)$ leaves a measurable $s$–$x_{\mathrm{label}}$ correlation, or if the oracle-matched model does not dominate standard SSL, the central claims fail.

Watch

Extended reading notes

Core claim

Treating each SSL mini-batch as a multi-class task with the anchor as the label, the paper claims that SSL's OOD generalization is governed by the correlation between the anchor $x_{\mathrm{label}}$ and an unobserved spurious variable $s$ in the generative model $x_+ = F(s, x_{\mathrm{label}}) + \epsilon$. It defines PID by $p(x_+, x_{\mathrm{label}}, s) = p(x_+ | x_{\mathrm{label}}, s)\,p(x_{\mathrm{label}})\,p(s)$, so that $s$ and $x_{\mathrm{label}}$ are independent while the causal mechanism $s \to x_+ \leftarrow x_{\mathrm{label}}$ is preserved. The paper proves (Theorem 3.4) that under a conditional-independence assumption, the alignment component of SSL is maximum-likelihood estimation of $p_f(x_{\mathrm{label}} | x_+)$, and any $f^*$ that minimizes the risk on every PID batch is the minimax solution over all environments. The method learns an exponential-family latent variable model over $s$, computes a balancing score $\mathrm{ba}(s)$ from the induced propensity scores, and samples mini-batches by matching pairs with identical $\mathrm{ba}(s)$; Theorem 4.7 states that perfect matching produces a PID batch. The claimed effect is that models trained this way achieve optimal worst-case OOD performance rather than merely better average accuracy.

Load-bearing premise

The whole chain rests on Assumption 4.1: the true spurious factor $s$ is faithfully captured by an exponential-family latent variable model, so the balancing score computed from the learned VAE is a function of the real spurious variable. The paper explicitly states in Appendix G that it does not prove the latent model identifies $s$; if $s$ is misidentified, PID batches are independent only with respect to a proxy, and Theorem 4.7 does not apply to the true spurious factor.

Editorial extensions

If this is right

Train every mini-batch under PID and the SSL model is minimax optimal for worst-case OOD loss across the environment family $D$ (Theorem 3.4).
Exact balancing-score matching in Algorithm 1 produces batches drawn from PID (Theorem 4.7), so the sampler is a drop-in replacement for random batch construction in any discriminative or generative SSL method.
The latent variable model that supplies $s$ is identifiable up to an equivalence relation under Assumption 4.1, so the learned components can be reused across tasks.
Empirically, replacing random batches with PID batches improves linear probing, semi-supervised classification, transfer detection and segmentation, and few-shot accuracy, including worst-group accuracy on Waterbirds and a near-10-point gain on ColoredMNIST.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: a direct test of the weakest link would use a dataset with a known spurious factor and compare Algorithm 1 batches against oracle-matched batches built from the true $s$; if the oracle does not beat the learned sampler, the minimax claim is not what drives the gains.
Beyond the paper: Theorem 3.4 quantifies over environments in $D$, so the guarantee does not cover test shifts that introduce spurious mechanisms absent from training; PID sampling should be seen as robust over the range of environments already seen.
Beyond the paper: the framework reframes batch construction as causal adjustment—changing which samples co-occur instead of reweighting losses—so combining PID sampling with invariant-risk or reweighting objectives is a natural next step, and evaluation should emphasize worst-group rather than average accuracy.
Beyond the paper: if the latent model misidentifies the spurious factor, matching on $\mathrm{ba}(s)$ enforces independence with respect to a proxy rather than the true $s$; the paper flags exactly this gap in Appendix G, and a proxy failure would show up as spurious correlations surviving inside matched batches.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this paper has a genuinely new batch-sampling method for SSL, and the empirical coverage is unusually broad. Second thing: the central theorem claiming optimal worst-case OOD performance is not actually proven. If you read Theorem 3.4 expecting a derivation, Appendix A.2 supplies an extra "environment diversity" assumption that essentially assumes the minimax conclusion.

What's new and good. The post-intervention distribution (PID) view—make the spurious variable s independent of the anchor within each mini-batch—is a reasonable causal target, and the two-stage implementation (VAE to learn the distribution, balancing-score matching to build batches) is new. The paper shows consistent gains across SimCLR, MoCo, SimSiam, Barlow Twins, SwAV, DINO, VICRegL, and even MAE, on ImageNet, transfer, few-shot, and OOD benchmarks like Waterbirds and ColoredMNIST. Code is released. The identifiability theorem for the latent variable model is a fair application of iVAE, and Appendix G is candid that the spurious variable s is not actually identified.

Where it falls short. The minimax theorem is the load-bearing claim, and the proof doesn't get there. The environment-diversity sentence in Appendix A.2 ("since we assume the environment diversity...") is doing all the work: it asserts that every alternative classifier is worse than random in some environment, which is stronger than any stated assumption. Also, the theorem's antecedent—that a single f* minimizes L_e for every PID environment—is never connected to the proof, and is itself suspect when different PID marginals have different Bayes classifiers. Then Theorem 4.7 only holds under perfect matching on the true s, and the paper admits s is not identifiable. So the optimality claim is conditional on something that isn't delivered. Empirically, the reporting is selective: Table 2 shows BYOL+Ours at 200 epochs dropping from 54.8 to 46.5 (1% labels), a serious regression that goes unmentioned. The stated O(D^2) complexity for the matching step is also hard to square with ImageNet-scale training unless some approximation is used, and the paper doesn't say what it is.

Bottom line. If you work on debiasing SSL or causally-motivated sampling, the method and the evaluation are worth your attention; the theory is mostly scaffolding. I'd send this to a serious referee, because the empirical contribution deserves examination and the authors should be pushed to either prove or drop the minimax claim. As is, I would not accept it, but I also wouldn't desk-reject it.

Referee Report

4 major / 4 minor

Summary. The paper argues that SSL training can be understood as learning a distribution over mini-batch classification tasks, that such tasks are contaminated by a spurious latent variable s correlated with the anchor/label, and that a post-intervention distribution (PID) in which s and x_label are independent yields optimal worst-case OOD generalization. The authors introduce Theorem 3.4 to support this minimax claim, then propose a two-stage method: a regularized VAE that learns the conditional distribution p(s|x_label) (with an identifiability guarantee in Theorem 4.3), and a balancing-score matching batch sampler (Algorithm 1) that, under perfect matching, is claimed in Theorem 4.7 to produce mini-batches drawn from a PID. Experiments across ImageNet, semi-supervised, transfer, few-shot, and explicitly OOD benchmarks report consistent gains when the sampler is added to existing SSL methods.

Significance. If the central minimax theorem and the PID guarantee were rigorously established, the paper would provide a useful causal framework for batch construction in SSL, with a practical and cheap intervention (replacing random batch sampling). The manuscript has real strengths: the code is released; the empirical evaluation is broad and includes explicit OOD benchmarks (PACS, OfficeHome, Waterbirds, ColoredMNIST); and Theorem 4.3 is a faithful, careful application of nonlinear iVAE identifiability. However, the load-bearing theoretical claims are not established: the proof of Theorem 3.4 inserts an unstated environment-diversity assumption that essentially contains the minimax conclusion, and Theorem 4.7 depends on identifying the true spurious variable s, which Appendix G explicitly says is not theoretically proven. These gaps undermine the abstract's main claim that PID mini-batches achieve optimal worst-case OOD performance.

major comments (4)

[Appendix A.2, Theorem 3.4] The proof of Theorem 3.4 is incomplete and circular. The derivation establishes only that L_e(p_PI(x_label|x_+)) <= L_e(random) for each environment e. The strict minimax comparison is then introduced by the sentence 'Since we assume the environment diversity, that is for any p_e with x_label not independent of s, there exists an environment e′ such that p_e(x_label|x_+) performs worse than a random guess.' No such assumption is stated in Theorem 3.4, in Assumption 3.3, or in the definition of D in Eq. (2); it is a strong minimax-dominance property that is essentially the theorem's conclusion. If that strict inequality fails, a non-PID classifier can have worst-case risk no larger than the PID classifier, and the claimed optimality does not follow. This is a load-bearing gap because the abstract's central claim is exactly that PID mini-batches guarantee optimal worst-case OOD performance.
[Section 4.2 / Theorem 4.7 / Appendix G] The PID guarantee in Theorem 4.7 is conditional on the balancing score ba(s) being computed from the true spurious variable s. But the paper itself states in Appendix G, Step 4: 'We do not theoretically prove that the latent variable model can directly identify the spurious variable s. In this paper, the identification of s is based on a strong assumption, Assumption 4.1.' If the s sampled from q_phi(s|x_+, x_label) is not the true spurious variable, then matching on ba(s) does not imply s ⊥ x_label in the real data-generating process, and the conclusion p_hat(x_+, x_label, s) = p_PI(x_+, x_label, s) collapses. Thus the method's theoretical guarantee in the main text is stronger than what the manuscript admits in the appendix, and Theorem 4.7 does not establish that Algorithm 1 produces PID mini-batches for the actual spurious factor.
[Theorem 3.4, first sentence and assumption] The theorem's antecedent, 'if f* = arg min L_e(f) for all e in PID', is both unconnected to the proof and hard to satisfy. The proof in Appendix A.2 never uses this assumption; it directly analyzes the conditional p_PI(x_label|x_+). Moreover, for a nontrivial spurious variable s, different PID environments have different marginals p(x_label) or different conditional mechanisms p(x_+|x_label, s), so the Bayes-optimal classifier p(x_label|x_+) generally differs across environments. A single f* cannot be exactly optimal for all of them. The paper does not address this tension, so the theorem's scope and applicability to actual SSL training are unclear.
[Section 2 / Section 3.2 / Theorem 3.4] The reduction of SSL objectives to maximizing p_f(x_label|x_+) is asserted rather than derived. Theorem 3.4 begins 'From a Bayesian perspective' and the text states that SimCLR uses a contrastive loss and MAE an L2 loss 'to achieve p_f(x_label|x_+)', but no formal equivalence is given between these losses and conditional-likelihood maximization. Since the minimax result is stated for p_f(x_label|x_+), the relevance of Theorem 3.4 to actual D-SSL and G-SSL training losses is not established. At minimum, the paper should state precisely which SSL losses are exactly or approximately equivalent to maximizing p_f(x_label|x_+) and under which conditions.

minor comments (4)

[Appendix A.4] The proof of Theorem 4.7 ends by interpreting PID as 'Permuted and Injected Distribution', which contradicts the earlier definition 'post-intervention distribution' in Definition 3.2; this inconsistency should be corrected.
[Appendix A.4] The derivation appears to show only that the balanced distribution agrees with the original distribution on p(x_label|s); it does not show the key independence x_label ⊥ s or p(x_+, x_label, s) factorizing as p(x_+|x_label, s)p(x_label)p(s). The proof should be rewritten to state explicitly how perfect matching on ba(s) yields this factorization.
[Equation (4)] The regularizer notation 'A·,i·A·,j' is ambiguous; it should be written as a dot product or inner product, e.g., <A_{·,i}, A_{·,j}>.
[Appendix D] The sentence 'Another notable contribution comes from Chen et al.(Zhu et al., 2023)' attaches an incorrect author name to the Zhu et al. reference; the citation should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 3.4's minimax optimality is imported through an unstated 'environment diversity' premise in Appendix A.2 that is essentially the theorem's conclusion; the paper's independent iVAE identifiability and empirical results keep the circularity partial rather than total.

self definitional [Appendix A.2, proof of Theorem 3.4 (main text Section 3.2)]
"Since we assume the environment diversity, that is for any pe with xlabel⊥ ⊥es, there exists an environment e′ such that pe(xlabel|x+) performs worse than a random guess."

The theorem claims that a model trained on PID is minimax optimal over D. The derivation immediately before this sentence only establishes L_e(p_PI) <= L_e(random) in every environment, which bounds the PID classifier's worst-case loss by the random-guess loss. The strict inequality that makes PID uniquely minimax is supplied by the quoted 'environment diversity' premise, which asserts exactly that every non-PID alternative p_e is worse than random in some environment. That strict dominance property is the theorem's minimax conclusion, not a consequence of Assumption 3.3 or of the definition of D in Eq. (2). If it fails, a non-PID classifier could have worst-case loss no larger than the PID classifier, and Theorem 3.4 would be false.

full rationale

The load-bearing theoretical claim of the paper is Theorem 3.4, which motivates the entire PID sampling strategy. Its appendix proof depends on an unstated 'environment diversity' assumption that is exactly the strict minimax dominance the theorem purports to prove; this is a genuine circular step. The iVAE-based identifiability result (Theorem 4.3) is a faithful application of Khemakhem et al. (2020) and is not circular. Appendix G candidly admits that identification of the spurious variable s is assumed rather than proven, which is a limitation of the method but not a circularity. The empirical comparisons against external baselines are independent evidence and prevent the score from being higher. Theorem 4.7 restates the balancing-score property as a PID guarantee, but that is standard propensity-score reasoning and is not treated here as a separate circular step. Overall, the headline worst-case OOD guarantee reduces by construction to an assumed environment-diversity property, so the paper is partially circular in its central theoretical claim.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central claims rest on a network of assumptions about the data generation process and the identifiability of the spurious variable. The most consequential are Assumption 3.3 (invertibility), Assumption 4.1 (exponential family conditional), and the unproven environment-diversity condition used in the proof of Theorem 3.4. The paper itself flags Assumption 4.1 as strong and acknowledges that s identification is not proven.

free parameters (2)

alpha (regularization weight) = 1
Controls the orthogonality regularizer in Equation (4); set to 1 after ablation on ImageNet-100 (Figure 5), where performance peaks.
a (matching count, batch size a+1) = tuned
Number of additional pairs matched in Algorithm 1; Figure 4 shows the method's sensitivity to a and the authors select it experimentally.

assumptions (5)

ad hoc to paper Assumption 3.3: there exist F_xlabel and F_s such that (x_label, s) = F^{-1}(x+ - epsilon), implying x_label independent of s given x+ under PID.
Invertibility assumption used to prove Theorem 3.4 and to connect the observed positive sample to both the label and spurious factor.
ad hoc to paper Assumption 4.1: the per-task conditional p^e_{T,lambda_e}(s|x_label) is an exponential family shared across tasks up to lambda_e(x_label).
Underlies identifiability (Theorem 4.3) and the definition of the propensity score in Equation (5); the paper concedes in Appendix G that s is identified only under this strong assumption.
ad hoc to paper Environment diversity: for any environment e with x_label independent of s, there exists an environment e' where p_e(x_label|x+) is worse than random guessing.
Introduced without proof in Appendix A.2; it supplies the minimax conclusion of Theorem 3.4, so the central claim is assumed rather than derived.
domain assumption The spurious variable s contains no semantic information relevant to the task.
Needed so that removing s via PID does not discard label-relevant content; stated in Section 3.1.
domain assumption SSL alignment objectives (contrastive or L2) can be unified as maximizing p_f(x_label|x+).
Bridges D-SSL and G-SSL; Theorem 3.4 is formulated in terms of this conditional likelihood.

invented entities (1)

spurious latent variable s
purpose: Encodes background, texture, and style confounders; PID is defined by s being independent of x_label.
s is unobserved and the paper provides no independent falsifiable handle; identifiability relies on Assumption 4.1, which the authors state is not proven (Appendix G).

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the Out-of-Distribution Generalization of Self-Supervised Learning." pith.science (2026). https://pith.science/paper/NHMZJXEK

@misc{pith2026250516675,
  author       = {Pith},
  title        = {Pith review of: On the Out-of-Distribution Generalization of Self-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHMZJXEK}},
  note         = {Machine review of arXiv:2505.16675}
}

read the original abstract

In this paper, we focus on the out-of-distribution (OOD) generalization of self-supervised learning (SSL). By analyzing the mini-batch construction during the SSL training phase, we first give one plausible explanation for SSL having OOD generalization. Then, from the perspective of data generation and causal inference, we analyze and conclude that SSL learns spurious correlations during the training process, which leads to a reduction in OOD generalization. To address this issue, we propose a post-intervention distribution (PID) grounded in the Structural Causal Model. PID offers a scenario where the spurious variable and label variable is mutually independent. Besides, we demonstrate that if each mini-batch during SSL training satisfies PID, the resulting SSL model can achieve optimal worst-case OOD performance. This motivates us to develop a batch sampling strategy that enforces PID constraints through the learning of a latent variable model. Through theoretical analysis, we demonstrate the identifiability of the latent variable model and validate the effectiveness of the proposed sampling strategy. Experiments conducted on various downstream OOD tasks demonstrate the effectiveness of the proposed sampling strategy.

Figures

Figures reproduced from arXiv: 2505.16675 by the authors.

**Figure 2.** The SCM for p PI(x +, xlabel, s). nal sample is then referred to as x 2 i . Thus, the augmented dataset in SSL (whether D-SSL or G-SSL) is represented as X aug tr = {x 1 i , x2 i } N i=1. The pair {x 1 i , x2 i } forms the i-th pair, and SSL aims to learn a feature extractor f from these pairs. The objective of D-SSL methods typically consists of two components: alignment and regularization (Wang & Isola, 2020; Chen… view at source ↗

**Figure 3.** Two specific instances illustrate the variability in the causal relationship between [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 5.** Influence of the hyperparameter α. 5.3. Ablation Study Influence of the batch size hyperparameter a. As shown in Theorem 4.7, a suitable a is important. To explore whether the SSL model is more sensitive to original batch size or a, we conduct experiments using ImageNet and BYOL [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Figures from the paper (1 more)

**Figure 6.** Figure 6: Evaluation for the VAE-based components. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 20 canonical work pages

[3]

Reformulation of OOD Generalization as Generalization on Task Distributions

From Equation (2) in the main text, we have: p(x+, xlabel, s) = p(x+|xlabel, s)p(xlabel)p(s|xlabel). If we select sample pairs for a mini-batch such that all pairs share the same ba(s), the resulting mini-batch can be considered as constructed under the same ba(s). In other words, the samples in the mini-batch are conditioned on ba(s). Combined with the a...

work page 2017
[5]

An empirical study of training self-supervised vision transformers

Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. Cornell University - arXiv,Cornell University - arXiv, Apr 2021b. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 2...

work page 2009
[6]

Even though we train a V AE, it does not affect other components of SimCLR’s training pipeline, including the training objective, network architecture, and optimization algorithm

First, our proposed method only modifies the mini-batch construction process during the training phase of SimCLR. Even though we train a V AE, it does not affect other components of SimCLR’s training pipeline, including the training objective, network architecture, and optimization algorithm. Second, training a V AE independently on ImageNet and using its...

work page 2020
[8]

Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A

URL https:// arxiv.org/abs/1705.06950. Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. Variational autoencoders and nonlinear ica: A unify- ing framework. In International conference on artificial intelligence and statistics, pp. 2207–2217. PMLR,

arXiv
[10]

Li, D., Yang, Y ., Song, Y .-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In 2017 IEEE International Conference on Computer Vision (ICCV) , Oct

work page 2017
[11]

URL http: //dx.doi.org/10.1109/iccv.2017.591

doi: 10.1109/iccv.2017.591. URL http: //dx.doi.org/10.1109/iccv.2017.591. Li, J., Qiang, W., Zheng, C., Su, B., and Xiong, H. Metaug: Contrastive learning via meta feature augmentation. In International conference on machine learning, pp. 12964– 12978. PMLR,

doi:10.1109/iccv.2017.591 2017
[12]

A., and Doretto, G

Motiian, S., Piccirilli, M., Adjeroh, D. A., and Doretto, G. Unified deep supervised domain adaptation and gener- alization. In 2017 IEEE International Conference on Computer Vision (ICCV), Oct

work page 2017
[13]

2017.609

doi: 10.1109/iccv. 2017.609. URL http://dx.doi.org/10.1109/ iccv.2017.609. Murray, N., Marchesotti, L., and Perronnin, F. Ava: A large- scale database for aesthetic visual analysis. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2408–2415. IEEE,

doi:10.1109/iccv 2017

Show all 35 references

[15]

Towards the sparseness of projection head in self- supervised learning

Song, Z., Su, X., Wang, J., Qiang, W., Zheng, C., and Sun, F. Towards the sparseness of projection head in self- supervised learning. arXiv preprint arXiv:2307.08913,

arXiv
[17]

Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119,

Tomasev, N., Bica, I., McWilliams, B., Buesing, L., Pascanu, R., Blundell, C., and Mitrovic, J. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119,

arXiv
[18]

Hacking task confounder in meta-learning

Wang, J., Qiang, W., Ren, Y ., Song, Z., Su, X., and Zheng, C. Hacking task confounder in meta-learning. arXiv preprint arXiv:2312.05771, 2023a. Wang, J., Song, Z., Qiang, W., and Zheng, C. Unleash model potential: Bootstrapped meta self-supervised learning. arXiv preprint arX...

arXiv
[19]

Wang, X., Saxon, M., Li, J., Zhang, H., Zhang, K., and Wang, W. Y . Causal balancing for domain generalization. arXiv preprint arXiv:2206.05263,

arXiv
[20]

S., and Finn, C

Xie, J., Lee, Y ., Chen, A. S., and Finn, C. Self- guided masked autoencoders for domain-agnostic self- supervised learning. arXiv preprint arXiv:2402.14789,

arXiv
[21]

Barlow twins: Self-supervised learning via redundancy reduction

Zbontar, J., Jing, L., Misra, I., LeCun, Y ., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230,

arXiv
[22]

ibot: Image bert pre-training with online tokenizer

Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with online tokenizer. Zhu, W., Liu, S., Fernandez-Granda, C., and Razavian, N. Making self-supervised learning robust to spurious corre- lation via learning-speed aware samp...

arXiv
[23]

Improving event causality identification via self-supervised representation learning on external causal statement

Zuo, X., Cao, P., Chen, Y ., Liu, K., Zhao, J., Peng, W., and Chen, Y . Improving event causality identification via self-supervised representation learning on external causal statement. arXiv preprint arXiv:2106.01654,

arXiv
[24]

Since this holds for all x ∈ R and h ̸= 0, we conclude that the distribution is not strongly exponential

By integrating, we find that ⟨Ti(x), h⟩ = const. Since this holds for all x ∈ R and h ̸= 0, we conclude that the distribution is not strongly exponential. Thus, by contradiction, there must exist k points ¯x1 i , ¯x2 i , . . . ,¯xk i such that dTi(¯x1 i ) d¯x1 i , . . . ,dTi(¯...

2020
[27]

with the standard 1 × schedule and C4-backbone (Wu et al., 2019), reporting results on the validation split. Few-shot Learning The protocol outlined in (Wang et al., 2024b; 2023b) is followed for few-shot learning, where we evaluate the proposed method on three standard few-sh...

2019
[29]

The ViT backbone is adapted for use with FPN (Lin et al., 2017)

end-to-end on COCO (Lin et al., 2014). The ViT backbone is adapted for use with FPN (Lin et al., 2017). We apply this approach to all entries in Table

2014
[30]

Following the standard setting, we evaluate on top 60 common classes with mean Average Precision (mAP) as the metric

to the downstream action detection dataset A V A (Murray et al., 2012). Following the standard setting, we evaluate on top 60 common classes with mean Average Precision (mAP) as the metric. The results are Table 7: COCO object detection and segmentation using a ViT Mask R-CNN ...

2012
[32]

During training, waterbirds (landbirds) are predominantly paired with water (land) backgrounds

For the Waterbirds dataset, we adopt the implementation from (Zare & Van Nguyen, 2023). During training, waterbirds (landbirds) are predominantly paired with water (land) backgrounds. However, at test time, the distribution of backgrounds is altered, creating a domain shift. I...

2023
[33]

Finally, for the Colored-MNIST dataset, we follow the experimental settings mentioned in (Huang et al., 2024)

From the results, we can observe that our method consistently yields improvements, with particularly significant gains observed for the worst-performing group, indicating the effectiveness of our approach in addressing domain shifts. Finally, for the Colored-MNIST dataset, we ...

2024
[128]

For this experiment, we utilize several benchmark datasets to evaluate the model’s performance

The learning rate starts at 5 × 10−2 and decays to 5 × 10−6. For this experiment, we utilize several benchmark datasets to evaluate the model’s performance. CIFAR-10 and CIFAR-100 are small-scale image classification datasets consisting of 60,000 32×32 color images in 10 and 1...

2021
[1981]

URL http://dx.doi.org/10.21236/ada114514

doi: 10.21236/ada114514. URL http://dx.doi.org/10.21236/ada114514. Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30,

doi
[2009]

An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

2010 arXiv
[2015]

We also conduct experiments on a smaller version of Pascal VOC, the VOC 07 set (5K images), with a reduced number of iterations

model on the combined VOC 2007 and 2012 datasets (VOC 07+12), which contains around 16,000 images, and adjust the learning rate at 18K and 22K iterations. We also conduct experiments on a smaller version of Pascal VOC, the VOC 07 set (5K images), with a reduced number of itera...

2007
[2016]

The following section provides a comparison of the models

and has a tendency to overfit. The following section provides a comparison of the models. Table 5: Comparison between models. Method scratch, original scratch, our impl. baseline MAE MAE + Ours Top 1 76.5 82.5 84.9 86.4 Table 6: Comparisons with previous results on ImageNet-1K...

2021
[2017]

URL http://dx

doi: 10.1080/01621459.2017.1285773. URL http://dx. doi.org/10.1080/01621459.2017.1285773. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. CoRR, abs/2006.09882,

2017
[2018]

Bootstrap your own latent-a new approach to self-supervised learning

Grill, J.-B., Strub, F., Altch´e, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:2127...

2006 arXiv
[2019]

and Welling, M

Kingma, D. and Welling, M. Auto-encoding variational bayes. arXiv: Machine Learning,arXiv: Machine Learn- ing, Dec 2013a. Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013b. Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. ...

arXiv
[2020]

Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A

URL https://arxiv.org/ abs/2006.09882. Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging proper- ties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vi- sion (ICCV), Oct

2006 arXiv
[2021]

doi: 10.1109/iccv48922.2021. 00951. URL http://dx.doi.org/10.1109/ iccv48922.2021.00951. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597–1607. PMLR,

2021
[2022]

F., Torr, P

Bertinetto, L., Henriques, J. F., Torr, P. H., and Vedaldi, A. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.08136,

arXiv
[2023]

On the discriminability of self-supervised representation learning

Song, Z., Qiang, W., Zheng, C., Sun, F., and Xiong, H. On the discriminability of self-supervised representation learning. arXiv preprint arXiv:2407.13541,

arXiv
[2024]

proposed that spuriously correlated attributes make neural networks inductively biased towards encoding lower effective rank representations and used rank regularization to eliminate biased samples. Another notable contribution comes from Chen et al.(Zhu et al., 2023), who exp...

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

Reformulation of OOD Generalization as Generalization on Task Distributions

From Equation (2) in the main text, we have: p(x+, xlabel, s) = p(x+|xlabel, s)p(xlabel)p(s|xlabel). If we select sample pairs for a mini-batch such that all pairs share the same ba(s), the resulting mini-batch can be considered as constructed under the same ba(s). In other words, the samples in the mini-batch are conditioned on ba(s). Combined with the a...

work page 2017

[2] [5]

An empirical study of training self-supervised vision transformers

Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. Cornell University - arXiv,Cornell University - arXiv, Apr 2021b. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 2...

work page 2009

[3] [6]

Even though we train a V AE, it does not affect other components of SimCLR’s training pipeline, including the training objective, network architecture, and optimization algorithm

First, our proposed method only modifies the mini-batch construction process during the training phase of SimCLR. Even though we train a V AE, it does not affect other components of SimCLR’s training pipeline, including the training objective, network architecture, and optimization algorithm. Second, training a V AE independently on ImageNet and using its...

work page 2020

[4] [8]

Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A

URL https:// arxiv.org/abs/1705.06950. Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. Variational autoencoders and nonlinear ica: A unify- ing framework. In International conference on artificial intelligence and statistics, pp. 2207–2217. PMLR,

arXiv

[5] [10]

Li, D., Yang, Y ., Song, Y .-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In 2017 IEEE International Conference on Computer Vision (ICCV) , Oct

work page 2017

[6] [11]

URL http: //dx.doi.org/10.1109/iccv.2017.591

doi: 10.1109/iccv.2017.591. URL http: //dx.doi.org/10.1109/iccv.2017.591. Li, J., Qiang, W., Zheng, C., Su, B., and Xiong, H. Metaug: Contrastive learning via meta feature augmentation. In International conference on machine learning, pp. 12964– 12978. PMLR,

doi:10.1109/iccv.2017.591 2017

[7] [12]

A., and Doretto, G

Motiian, S., Piccirilli, M., Adjeroh, D. A., and Doretto, G. Unified deep supervised domain adaptation and gener- alization. In 2017 IEEE International Conference on Computer Vision (ICCV), Oct

work page 2017

[8] [13]

2017.609

doi: 10.1109/iccv. 2017.609. URL http://dx.doi.org/10.1109/ iccv.2017.609. Murray, N., Marchesotti, L., and Perronnin, F. Ava: A large- scale database for aesthetic visual analysis. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2408–2415. IEEE,

doi:10.1109/iccv 2017