Pith. sign in

REVIEW 4 major objections 6 minor 49 references

CLIP-like Model as a Foundational Density Ratio Estimator

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLIP-style models can be reused off the shelf as density ratio estimators, turning similarity scores into importance weights and KL divergences.

desk verdict A useful, honest paper that correctly derives the density-ratio interpretation of CLIP scores but oversells the off-the-shelf transfer to arbitrary target distributions. read the letter →

arxiv 2506.22881 v3 pith:G2LZ6VBD submitted 2025-06-28 cs.CV

Fumiya Uchiyama , Rintaro Yanagi , Shohei Taniguchi , Shota Takashiro , Masahiro Suzuki , Hirokatsu Kataoka , Yusuke Iwasawa , Yutaka Matsuo This is my paper · ORCID
classification cs.CV
keywords densityratioestimationCLIPcontrastivelearningimportanceweightingKLdivergencedatacurationInfoNCESigLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that contrastive vision–language models such as CLIP and SigLIP are, by construction, density ratio estimators: their image–text similarity scores encode the log ratio of the conditional distribution of one modality given the other to its marginal distribution. If that claim holds, a frozen, off-the-shelf CLIP can replace purpose-built density ratio estimators in statistical workflows. The paper demonstrates two such workflows: importance weighting that re-targets a pretraining corpus toward a domain described by a single prompt, and KL divergence estimation that measures how much conditioning on an image or caption shifts the other modality's distribution. In the reported experiments, importance weighting raises downstream F1 scores by up to 7 points, and filtering pretraining data by text-side KL divergence performs on par with LAION2B filtering while using a quarter of the data.

What carries the argument

The load-bearing object is the identity $\frac{p_T(t|i)}{p_T(t)} \propto \exp(a\langle v_t, v_i\rangle)$, together with its symmetric image-side version. It converts the inner product of a CLIP-style embedding into a log density ratio, with the logit scale $a$ and the normalizer $Z(i)$ carrying the proportionality. For KL estimation, the argument passes through an exponential-family reading of the conditional distribution $p_T(t|v_i)=p_T(t)\exp(a\langle v_t,v_i\rangle-\psi(v_i))$, which yields quadratic approximations $D_W(i)=a^2(v_i-\hat v_I)^\top \hat G_T (v_i-\hat v_I)$ and $D_C(i)=a^2\|v_i-\hat v_I\|^2$; these centered-norm metrics connect the density ratio view to semantic frequency and diversity. The machinery also includes the NCE/InfoNCE derivations for SigLIP and CLIP that justify the identity from their respective losses.

What would settle it

A controlled test on data with known density ratios outside the training distribution: take a CLIP trained on one image corpus, evaluate its similarity-based ratio on images from a disjoint distribution against an exact ratio computed from the generative model, and check whether the correlation collapses while in-distribution correlation stays high. A second concrete test is a covariate shift that cannot be expressed by any single prompt; if the importance weights no longer recover the true ratio between training and test inputs, the single-prompt assumption fails.

Watch

Extended reading notes

Core claim

The central discovery is that the score function learned by CLIP-style contrastive training is not merely an embedding similarity but an estimate of a probabilistic density ratio. For an image $i$ and text $t$, the optimum of InfoNCE or NCE satisfies $\frac{p_T(t|i)}{p_T(t)} = \frac{\exp(a\langle v_t, v_i\rangle)}{Z(i)}$, so the cosine similarity is proportional to a log density ratio, with a per-image normalizer $Z(i)$; the symmetric identity holds with images conditioned on text. The paper leverages this identity in two applications. First, Importance Weight Learning reweights pretraining samples by $\exp(a\langle v_x, v_t\rangle)$ for a single domain prompt $t$, treating the test distribution as approximately equal to the conditional image distribution given that prompt; this improves zero-shot accuracy and F1 on Food101, Oxford-IIIT Pet, and Flowers102 by up to 7 points. Second, four KL-based quantities ($D_{KL}$, $D_{KL}^R$, $D_W$, $D_C$) are derived from the density ratio and from an exponential-family approximation of the embedding geometry; high KL scores mark images and captions with diverse semantics, and keeping the top 25% of captions by text-side KL yields ImageNet zero-shot accuracy close to LAION2B filtering. A toy Gaussian-mixture experiment directly verifies that a small CLIP trained with InfoNCE reproduces ground-truth density ratios with near-perfect correlation.

Load-bearing premise

The pretrained model's similarity scores faithfully estimate density ratios for the target data, including out-of-distribution samples; the paper states that the estimated density ratio of conditional and marginal distributions is equal to that of the training data.

Editorial extensions

If this is right

  • A pretrained CLIP can act as a drop-in density ratio estimator for covariate-shift correction, with no per-task training and only one prompt as supervision.
  • KL divergences computed from frozen CLIP embeddings give a principled, interpretable ranking of samples by semantic informativeness, not just alignment.
  • Filtering a large image-text pool by text-side KL preserves downstream accuracy at a quarter of the data, so data curation can be done with simple density ratios.
  • The density ratio identity applies to SigLIP as well, so the result transfers across contrastive objectives and model families.
  • Because the same identity holds symmetrically for images given text, similar reasoning can measure how conditioning on either modality shifts the other.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct corollary the paper does not develop is mutual information estimation: since these KL terms are density-ratio expectations, the same scores could yield estimates of I(image; text) over a sample, connecting the result to likelihood-free inference and representation evaluation.
  • The paper's observation that text-side KL outperforms image-side KL for curation suggests captions carry more distributional signal than images in web-scale data; if true, filtering pipelines should weight text informativeness first.
  • Combining a minimum alignment threshold with KL informativeness selection is a testable extension the paper mentions but leaves open; it could plausibly beat either filter alone.
  • With a calibrated logit scale, the exponential form of the ratio could give likelihood-like scores for individual samples, enabling principled outlier detection rather than rank-based filtering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reinterprets CLIP-like vision-language models (CLIP and SigLIP) as pretrained density ratio estimators, arguing that InfoNCE/NCE objectives make similarity scores proportional to log density ratios between conditional and marginal distributions. It derives this relationship in Sec. 3, proposes Importance-Weighted Learning (IWL) using a single prompt, and introduces KL-divergence-based metrics for semantic diversity and data curation. Experiments include zero-shot classification after IWL pretraining on CC12M, qualitative and N-gram analyses on MSCOCO captions, DataComp filtering at small scale, and a synthetic Gaussian-mixture validation with known ground-truth ratios. The paper claims that this density-ratio perspective enables off-the-shelf reuse of CLIP-like models as foundational density ratio estimators.

Significance. If the central claim holds, the paper offers a valuable unification: it connects contrastive pretraining objectives to density ratio estimation and demonstrates two practical applications, IWL and KL-guided data curation, that go beyond the usual embedding-based use of CLIP. The derivation is grounded in standard NCE/InfoNCE theory, the synthetic validation against ground-truth ratios is a good direct check, and the code is promised to be released. The bootstrap-based finite-sample error analysis in the supplementary is a useful addition. However, the paper's strongest claim, that CLIP-like models can be reused off-the-shelf as density ratio estimators for arbitrary target distributions, is not supported by the current evidence; the derivation characterizes the pretraining distribution only, and the Limitations section concedes this point explicitly.

major comments (4)
  1. [Sec. 3, Eqs. (1)-(2); Sec. 4, Eq. (6); Sec. 6, Limitations] The derivation in Sec. 3 shows that CLIP/SigLIP scores estimate the density ratio of the pretraining distribution (LAION/WebLI), yet all real-data applications evaluate on different distributions (MSCOCO, DataComp, CC12M). The Limitations section explicitly concedes that "the density ratio of the conditional and marginal distribution is equal to that of the training data," so the off-the-shelf transfer claim is not supported by the current evidence. The synthetic validation in Sec. 14 trains and evaluates on the same Gaussian mixture, so it cannot detect transfer failure. For IWL, Eq. (6) requires p_test_I/p_train_I, but the model yields p_LAION(i|t)/p_LAION(i); the missing factor p_LAION(x)/p_CC12M(x) is never addressed. I recommend either providing a real-data transfer experiment with known target ratios or reframing the central claim as density-ratio estimation for the pretraining distribution.
  2. [Sec. 5.1, Eq. (10); Supplementary Sec. 10, Eq. (34)] Eq. (10) contains an incorrect second term: it subtracts log sum_{t in D_T} a<v_t,v_i> exp(a<v_t,v_i>) + log|D_T|, but the derivation in Eq. (33) requires subtracting log Z(i) = log E_{t~p_T}[exp(a<v_t,v_i>)], which is approximated by log sum_{t in D_T} exp(a<v_t,v_i>) - log|D_T|. The same mistake appears in the supplementary derivation, Eq. (34), and in Eq. (12) for the text modality. As written, the estimator is not a consistent estimator of the KL divergence and may produce negative values for typical positive scores. Please correct the formula and check whether the reported KL values were computed with the corrected estimator.
  3. [Sec. 5.4, Table 2] The text states that filtering by D_KL, D_KLR, or D_C of text achieves "5-8 percentage points higher accuracy on the ImageNet1k zero-shot classification task" compared with no filtering. Table 2 shows differences of 0.005, 0.0075, and 0.0062 (0.5-0.75 percentage points) for these methods, not 5-8 percentage points. The reported gains are also below or comparable to the Basic baseline (0.03/0.142) and well below CLIPScore (0.051/0.173). This claim should be corrected, and the discussion of competitiveness with LAION2B filtering should be revisited in light of the actual numbers.
  4. [Sec. 4, Eq. (7)] The claim that optimizing Eq. (7) is "proportionally equal to minimizing the loss on the desired domain defined by the prompt" is not derived. The weight exp(a<u_i,u_t_dagger>) is proportional to p_LAION(i|t)/p_LAION(i) under the reference model's pretraining distribution, whereas the importance weight needed for covariate shift is p_test_I(i)/p_train_I(i) with p_train_I being the CC12M training distribution. An additional assumption linking these two ratios is required; otherwise the weighting is an ad hoc heuristic rather than a principled density-ratio correction. Please provide the missing derivation or state the assumption explicitly.
minor comments (6)
  1. [Sec. 5.1, Eq. (8)] The quantity softmax_i(t) is used in Eq. (10) but defined only in the supplementary material; please define it in the main text.
  2. [Sec. 2.2.1] The subsection "Data Curation for Large Multi-modal Pre-training" is numbered 2.2.1, but it appears to be a separate topic from "Density Ratio Relationship in Contrastive Learning"; consider renumbering it as Sec. 2.3.
  3. [Fig. 1] The top images in Fig. 1 all receive identical scores (10.127), suggesting numerical saturation or rounding; a brief note would help readers interpret the ranking.
  4. [Sec. 4 and Supplementary Sec. 11] The hyperparameter section says "we follow the default OpenCLIP configuration" but does not specify the batch size, learning rate, or other settings used for the baseline run; please list them explicitly.
  5. [Sec. 6, Limitations] The discussion of finite-sample approximation is qualitative; referencing the bootstrap analysis in Supplementary Sec. 15 would strengthen this passage.
  6. [Sec. 14, Table 3] SigLIP achieves R2 as low as 0.1094 for d=16, K=8, and its accuracy varies widely across settings; this negative result should be discussed in the main text rather than only mentioned in the supplementary.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Eqs. (1)-(2) are derived from external InfoNCE/NCE theory and validated on synthetic ground truth, so the downstream claims are not reducible to their inputs.

full rationale

I walked the derivation chain and found no step where a claimed prediction is equivalent by construction to a fitted input or a self-citation chain. Equations (1) and (2) state that optimal InfoNCE/NCE similarity scores are proportional to log density ratios of the pretraining joint distribution; the supplementary derivation (Sec. 9) reduces this to the standard posterior form of noise-contrastive classification [13, 25], and Section 14 validates it on a synthetic Gaussian mixture with known ground-truth ratios, so the density-ratio claim has independent content. The KL estimators in Eqs. (10)-(13) are Monte Carlo substitutions of Eq. (1); they do not fit any parameter to the KL values they report, and the N-gram diversity analysis (Fig. 4) is an external lexical check rather than an output of the model. The IWL loss in Eq. (7) uses a frozen pretrained model's prompt score as a fixed weight; no parameter is fitted to the downstream accuracy or F1, and the gain is an empirical consequence of upweighting domain-relevant samples, not a forced identity. Data curation in Sec. 5.4 filters by model-computed KL and then trains a fresh model, so evaluation is not leakage from the filter. The paper explicitly acknowledges the main assumptions: the Limitations state 'the density ratio of the conditional and marginal distribution is equal to that of the training data,' which is a transferability limitation rather than a circular definition, and Sec. 5.2 openly notes that D_C is equivalent to minus Conformity, so no known result is hidden. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. The off-the-shelf claim is therefore assumption-dependent but not circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central derivation (Eq. 1) relies on the known NCE/InfoNCE optimality result and assumptions about the model being at optimum and the embedding mapping being injective. The applications further assume that the pretrained ratios transfer to new data and that a single prompt can specify the target distribution. No free parameters are fitted to the downstream labels; the IWL logit scale is a hand-set value.

free parameters (3)
  • IWL logit scale a = 10
    In Sec 4, the authors scale the pretrained CLIP's logit scale from ~100 to 10 to prevent overflow in mixed-precision training; this choice affects the weight distribution and is a hand-tuned hyperparameter.
  • Data curation threshold = top 25%
    In Sec 5.4, pairs are kept if they have the top 25% D_KL values; no ablation on this threshold is provided.
  • Reference set size D_T, D_I = 10k
    In Sec 5.4, 10k images and texts are sampled from the DataComp pool to compute reference statistics; the sensitivity to this size is analyzed only for MSCOCO (Sec 15.2), not for the curation experiments.
assumptions (5)
  • standard math NCE/InfoNCE optimality
    The paper relies on the known result that the optimal score for NCE/InfoNCE equals log(p_cond/p_marg) up to a constant, derived in the supplementary (Eqs. 27-32).
  • domain assumption One-to-one mapping from text to embedding
    In the derivation of Eq. (1), the authors assume a one-to-one mapping from text to embedding, which does not hold in practice; see supplementary around Eq. (27).
  • domain assumption Transfer of density ratios to target distribution
    Stated in Limitations: the density ratio of the conditional and marginal distribution equals that of the training data, i.e., the pretrained model's ratios transfer to new data.
  • domain assumption Prompt represents test distribution
    In Eq. (6), the test image distribution is assumed to be approximately the conditional distribution given a prompt t, p_test(I) ≈ p_I(·|t).
  • standard math Exponential family local approximation
    For D_W and D_C, the paper assumes an exponential family with the mean embedding as the natural parameter for the marginal, and uses a second-order KL approximation (Eq. 16), following Oyama et al. [26].

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLIP-like Model as a Foundational Density Ratio Estimator." pith.science (2026). https://pith.science/paper/G2LZ6VBD

@misc{pith2026250622881,
  author       = {Pith},
  title        = {Pith review of: CLIP-like Model as a Foundational Density Ratio Estimator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G2LZ6VBD}},
  note         = {Machine review of arXiv:2506.22881}
}
read the original abstract

Density ratio estimation is a core concept in statistical machine learning because it provides a unified mechanism for tasks such as importance weighting, divergence estimation, and likelihood-free inference, but its potential in vision and language models has not been fully explored. Modern vision-language encoders such as CLIP and SigLIP are trained with contrastive objectives that implicitly optimize log density ratios between joint and marginal image-text distributions, which implicitly learn similarity scores proportional to log density ratios. However, prior work has largely focused on their embedding utility, and the density-ratio structure induced by contrastive learning has not been systematically examined or exploited in multimodal applications. To address this gap, we reinterpret CLIP-style models as pretrained and general-purpose density ratio estimators and show that this perspective enables new algorithmic capabilities. We present a unified explanation of how contrastive objectives estimate density ratios and propose two practical applications: Importance Weight Learning and KL divergence estimation. Our Importance Weight Learning method requires only a single additional prompt and improves F1 scores by up to 7 points. We further show that CLIP-based density ratios support estimation of KL divergences that quantify how conditioning on an image or text alters the distribution of the other modality. Through qualitative examples and an N-gram analysis of captions, we find that these divergences capture semantic diversity and mode structure in multimodal data. Leveraging this property, we introduce a simple KL-guided data curation method that achieves performance competitive with LAION2B filtering.

Figures

Figures reproduced from arXiv: 2506.22881 by the authors.

Figure 1
Figure 1. Top and bottom image examples in MSCOCO captions ranked by KL divergence Eq. ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Zero-shot classification performance comparison between baseline and our IWL method across three downstream datasets. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Top and bottom captions ranked by DKL. is also classified into another ten groups by DKL of itself [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: N-gram probability coverage across DKL deciles. Each decile in Fig. 4a is a group of captions corresponding to images i which has the same level of DKL(i). Each decile in Fig. 4b is a group of captions t which has the same level of DKL(t). aspects of the data: CLIPScor…
Figure 5
Figure 5. Figure 5: Top and bottom image examples in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Top and bottom image examples in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Top and bottom image examples in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Top and bottom captions ranked by DKLR. Side by side view of two oval plates, one with fork, with chicken salad sandwiches and rosy new potatoes, by an open and an unopened bottle of lager, a pepper mill, paper towel roll, basket behind. Score: 10591 A blue and black r…
Figure 9
Figure 9. Figure 9: Top and bottom captions ranked by DC [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Top and bottom captions ranked by DW. 0 500 1000 1500 2000 2500 Rank 0.2 0.4 0.6 0.8 1.0 Cumulative probability Uni-gram 0 2000 4000 6000 8000 10000 Rank Bi-gram 0 2500 5000 7500 10000 12500 15000 17500 Rank Tri-gram DKLR decile 0 (Low) 1 2 3 4 5 6 7 8 9 (High) (a) Cu…
Figure 11
Figure 11. Figure 11: N-gram probability coverage across KL deciles. Each decile in Figs. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Zero-shot classification performance comparison between baseline and our IWL method using SigLIP similarity scores across [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Top and bottom image examples in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Top and bottom image examples in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Top and bottom image examples in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Top and bottom image examples in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Top and bottom captions in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Top and bottom captions in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Top and bottom captions in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Top and bottom captions in MSCOCO captions ranked by [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: N-gram probability coverage across KL deciles calculated by SigLIP. Each decile in Figs. [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Visualized ground truth and predicted density ratio on image space. Each cross mark is the mean vector corresponding to the [PITH_FULL_IMAGE:figures/full_fig_p027_22.png]
Figure 23
Figure 23. Figure 23: Scatter plots of Dˆ∗(i) and normalized Bias(i) (left column), Variance(i) (center column), RMSE(i) (right column). Each row represents the errors of each estimation method. KL divergences are computed using CLIP [PITH_FULL_IMAGE:figures/full_fig_p028_23.png]
Figure 24
Figure 24. Figure 24: Scatter plots of Dˆ∗(i) and normalized Bias(i) (left column), Variance(i) (center column), RMSE(i) (right column). Each row represents the errors of each estimation method. KL divergences are computed using SigLIP [PITH_FULL_IMAGE:figures/full_fig_p029_24.png]
Figure 25
Figure 25. Figure 25: Violin plot of RMSE(i) on each estimation method across different sample size n [PITH_FULL_IMAGE:figures/full_fig_p030_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 44 canonical work pages

  1. [1]

    Differential Geometry of Curved Expo- nential Families-Curvatures and Information Loss.The An- nals of Statistics, 10(2):357 – 385, 1982

    Shun-Ichi Amari. Differential Geometry of Curved Expo- nential Families-Curvatures and Information Loss.The An- nals of Statistics, 10(2):357 – 385, 1982. 5

    work page 1982

  2. [2]

    Whitened CLIP as a likelihood surrogate of images and captions

    Roy Betser, Meir Yossef Levi, and Guy Gilboa. Whitened CLIP as a likelihood surrogate of images and captions. In Forty-second International Conference on Machine Learn- ing, 2025. 6

    work page 2025

  3. [3]

    Dis- criminative learning for differing training and test distribu- tions

    Steffen Bickel, Michael Br ¨uckner, and Tobias Scheffer. Dis- criminative learning for differing training and test distribu- tions. InProceedings of the International Conference on Machine Learning, page 81–88, New York, NY , USA, 2007. Association for Computing Machinery. 1, 3

    work page 2007

  4. [4]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. InProceedings of the European Conference on Com- puter Vision, 2014. 5

    work page 2014

  5. [5]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 5, 3

    work page 2021

  6. [6]

    K. F. Cheng and C. K. Chu. Semiparametric density estima- tion under a two-sample density ratio model.Bernoulli, 10 (4):583–604, 2004. 1

    work page 2004

  7. [7]

    Clip benchmark,

    Mehdi Cherti and Romain Beaumont. Clip benchmark,

    work page

  8. [8]

    Word associa- tion norms, mutual information, and lexicography.Compu- tational Linguistics, 16(1):22–29, 1990

    Kenneth Ward Church and Patrick Hanks. Word associa- tion norms, mutual information, and lexicography.Compu- tational Linguistics, 16(1):22–29, 1990. 3

    work page 1990

Show all 49 references
  1. [9]

    On the importance of embedding norms in self- supervised learning

    Andrew Draganov, Sharvaree Vadgama, Sebastian Damrich, Jan Niklas B ¨ohm, Lucas Maes, Dmitry Kobak, and Erik J Bekkers. On the importance of embedding norms in self- supervised learning. InForty-second International Confer- ence on Machine Learning, 2025. 6

    2025

  2. [10]

    Cambridge University Press, 2022

    Bradley Efron.Exponential Families in Theory and Practice. Cambridge University Press, 2022. 5

    2022

  3. [11]

    Datacomp: In search of the next generation of multimodal datasets.arXiv preprint arXiv:2304.14108, 2023

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Or- gad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussm...

    2023 arXiv

  4. [12]

    Generative adversarial nets.Advances in neural information processing systems, 27, 2014

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets.Advances in neural information processing systems, 27, 2014. 1

    2014

  5. [13]

    Gutmann and Aapo Hyv ¨arinen

    Michael U. Gutmann and Aapo Hyv ¨arinen. Noise- contrastive estimation of unnormalized statistical models, with applications to natural image statistics.Journal of Ma- chine Learning Research, 13(11):307–361, 2012. 1, 3, 4

    2012

  6. [14]

    CLIPScore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. InProceedings of the Confer- ence on Empirical Methods in Natural Language Process- ing, pages 7514–7528, Online and Punta Cana, Domini...

    2021

  7. [15]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. If you use this software, please cite it as be...

    2021

  8. [16]

    A least-squares approach to direct importance estimation

    Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance estimation. Journal of Machine Learning Research, 10(48):1391–1445,

  9. [17]

    Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes.arXiv preprint arXiv:1312.6114, 2013. 1

    2013 arXiv

  10. [18]

    The double-ellipsoid ge- ometry of CLIP

    Meir Yossef Levi and Guy Gilboa. The double-ellipsoid ge- ometry of CLIP. InForty-second International Conference on Machine Learning, 2025. 6

    2025

  11. [19]

    Neural word embedding as implicit matrix factorization

    Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. InAdvances in Neural Infor- mation Processing Systems. Curran Associates, Inc., 2014. 3, 4

    2014

  12. [20]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InPro- ceedings of the International Conference on Machine Learn- ing, pages 19730–19742. PMLR, 2023. 3

    2023

  13. [21]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vi- sion, pages 740–755. Springer, 2014. 6

    2014

  14. [22]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 3

    2023

  15. [23]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. InAdvances in Neu- ral Information Processing Systems. Curran Associates, Inc.,

  16. [24]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008. 5

    2008

  17. [25]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 4

    2018 arXiv

  18. [26]

    Norm of word embedding encodes information gain

    Momose Oyama, Sho Yokoi, and Hidetoshi Shimodaira. Norm of word embedding encodes information gain. InPro- ceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2108–2130, Singapore, 2023. Association for Computational Linguistics. 2, 3, 5, 6

    2023

  19. [27]

    Parkhi, Andrea Vedaldi, Andrew Zisserman, and C

    Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V . Jawahar. Cats and dogs. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

  20. [28]

    Inferences for case-control and semiparametric two-sample density ratio models.Biometrika, 85(3):619– 630, 1998

    Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models.Biometrika, 85(3):619– 630, 1998. 1

    1998

  21. [29]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the Int...

    2021

  22. [30]

    Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Advances in neural information processing systems, 29, 2016. 1, 5

    2016

  23. [31]

    Improving predictive inference un- der covariate shift by weighting the log-likelihood function

    Hidetoshi Shimodaira. Improving predictive inference un- der covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227– 244, 2000. 4

    2000

  24. [32]

    Covariate shift adaptation by importance weighted cross validation.Journal of Machine Learning Research, 8: 985–1005, 2007

    Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert M¨uller. Covariate shift adaptation by importance weighted cross validation.Journal of Machine Learning Research, 8: 985–1005, 2007. 3

    2007

  25. [33]

    Direct importance es- timation with model selection and its application to covariate shift adaptation

    Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance es- timation with model selection and its application to covariate shift adaptation. InAdvances in Neural Information Process- ing Systems. Curran Associates, Inc., 2007. 1, 3

    2007

  26. [34]

    Importance Sampling, page 119–139

    Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Importance Sampling, page 119–139. Cambridge University Press, 2012. 2, 4

    2012

  27. [35]

    Estimating squared- loss mutual information for independent component analy- sis

    Taiji Suzuki and Masashi Sugiyama. Estimating squared- loss mutual information for independent component analy- sis. InIndependent Component Analysis and Signal Sepa- ration, pages 130–137, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. 3

    2009

  28. [36]

    Approximating mutual information by maximum likelihood density ratio estimation

    Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual information by maximum likelihood density ratio estimation. InProceedings of the Workshop on New Challenges for Feature Selection in Data Mining and Knowledge Discovery, pages 5–20, Antwerp, ...

    2008

  29. [37]

    Mutual information estimation reveals global as- sociations between stimuli and biological processes.BMC bioinformatics, 10 Suppl 1:S52, 2009

    Taiji Suzuki, Masashi Sugiyama, Takafumi Kanamori, and Jun Sese. Mutual information estimation reveals global as- sociations between stimuli and biological processes.BMC bioinformatics, 10 Suppl 1:S52, 2009. 3

    2009

  30. [38]

    Kashima, S

    Yuta Tsuboi, H. Kashima, S. Hido, Steffen Bickel, and M. Sugiyama. Direct density ratio estimation for large-scale co- variate shift adaptation.Journal of Information Processing, 17:138–155, 2009. 3

    2009

  31. [39]

    Per- ceived age estimation under lighting condition change by co- variate shift adaptation

    Kazuya Ueki, Masashi Sugiyama, and Yasuyuki Ihara. Per- ceived age estimation under lighting condition change by co- variate shift adaptation. InProceedings of the International Conference on Pattern Recognition, pages 3400–3403, 2010. 2010 20th International Conference on Pat...

    2010

  32. [40]

    Makoto Yamada and Masashi Sugiyama. Dependence mini- mizing regression with model selection for non-linear causal inference under non-gaussian noise.Proceedings of the AAAI Conference on Artificial Intelligence, 24(1):643–648, 2010. 3

    2010

  33. [41]

    Sigmoid loss for language image pre-training,

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training,

  34. [43]

    log h σ(s(tpos(ij ), ij)) i (24) + X t′∈neg(ij ) log[σ(−s(t′, ij))] # =− 1 N NX j=1

    Derivations of Eqs. (1) and (2) In SigLIP, the learning objective is to classify whether a given pair of image and text is a positive pair or a nega- tive pair. Given an imagei, the text paired withicould be considered as a sample from the conditional text distribu- tionp T (·...

  35. [44]

    (10) and (11) To treat images and text in the same manner, we abusep T likep T (t) =p T (vt)andp T (t|i) =p T (vt|vi)

    Derivations of Eqs. (10) and (11) To treat images and text in the same manner, we abusep T likep T (t) =p T (vt)andp T (t|i) =p T (vt|vi). By substituting Equation (1) into Equation (8), DKL(i) = Z pT (vt′|vi) log pT (vt′|vi) pT (vt′) dvt′ = Z pT (vt′) exp(a⟨vt′, vi⟩) Z(i) log...

  36. [45]

    4 We trained ViT-B/32 models on the CC12M [5] dataset us- ing the official OpenCLIP training pipeline

    Hyper-parameters for Sec. 4 We trained ViT-B/32 models on the CC12M [5] dataset us- ing the official OpenCLIP training pipeline. Unless other- wise noted, we follow the default OpenCLIP configuration. We use AdamW with a learning rate of5×10−4, weight de- cay of 0.2, and mixed...

  37. [46]

    5.3 Figures 5 to 7 show top and bottom 18 images ranked by DKLR,D C,D W

    Other results of Sec. 5.3 Figures 5 to 7 show top and bottom 18 images ranked by DKLR,D C,D W. Some bottom samples ofD C have shared concepts such as patterned animals, and the top samples of DW resemble the bottom samples ofD KL. However, un- likeD KL, qualitative differences...

  38. [47]

    4 and 5, we empirically examined the applications of CLIP-like models as density ratio estimators using CLIP, which is commonly used in various tasks

    SigLIP Results In Secs. 4 and 5, we empirically examined the applications of CLIP-like models as density ratio estimators using CLIP, which is commonly used in various tasks. In this section, we report the result of Importance-Weighted Learning and KL divergence estimation usi...

  39. [48]

    On the other hand, we also conducted small-scale experiments to directly val- idate the density ratio modeling of the CLIP-like mod- els

    Ground-truth Experiments in a Toy Setting In the main part of the paper, we have verified the den- sity ratio modeling of CLIP-like models by implementing applications based on density ratio. On the other hand, we also conducted small-scale experiments to directly val- idate t...

  40. [49]

    Let us consider estimates of the image KL divergence

    Estimation Error Analysis All approximations of the KL divergences need a sample set. Let us consider estimates of the image KL divergence. Right sides of Eqs. (10) and (11) employ sample text set DT , and Eqs. (18) and (19) use both text set and image set for calculating ˆGT ...

    1929

  41. [2023]

    1, 3 CLIP-like Model as a Foundational Density Ratio Estimator Supplementary Material

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.