Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Contrast-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned CLIP models regain calibrated confidence when their logits are rescaled by the divergence from the original CLIP, with consistent gains on both train and unseen classes across 11 datasets.

desk verdict Genuinely novel post-hoc calibration idea for fine-tuned CLIP, but the headline empirical claim is compromised by hyperparameter selection on the same unseen-class test sets used for evaluation, plus an unvalidated per-image surrogate and an unmeasured inference-speed claim. read the letter →

arxiv 2501.19060 v3 pith:OIOFSLRI submitted 2025-01-31 cs.CV cs.LG

Song-Lin Lv , Yu-Yang Chen , Zhi Zhou , Yu-Feng Li , Lan-Zhe Guo This is my paper · ORCID
classification cs.CVcs.LG
keywords confidencecalibrationvision-languagemodelsCLIPopen-vocabularyclassificationpromptfine-tuningimage-textalignmentexpectederrorpost-hoc
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuned CLIP models often give overconfident, mis-calibrated predictions on classes they were not trained on, which undermines reliability in open-vocabulary deployment. This paper proposes Contrast-Aware Calibration (CAC), a post-hoc method that rescales each image's logits using the difference between the fine-tuned model and the original, well-calibrated CLIP. The paper argues that the per-image L1 distance between the two logit vectors measures how badly fine-tuned adaptation has degraded image-text alignment, and that a simple exponential-piecewise transformation of that distance yields a per-image calibration weight. Across 11 datasets and 5 prompt-fine-tuning methods, CAC is reported to achieve the best calibration on both training and unseen classes without changing accuracy or inference speed.

What carries the argument

The carrier of the argument is the Contrast-Aware Weight (CAW): a per-image scalar $\gamma=\alpha e^{-kz}$ computed from the L1 distance $z=\frac{1}{N}\sum_{i=1}^{N}|P_i-\hat P_i|$ between the original CLIP logits $P_i$ and the fine-tuned CLIP logits $\hat P_i$, combined with the piecewise map of Eq. (7): $\hat\gamma=\gamma^2$ below $\lambda_1$, $\hat\gamma=\gamma$ in the middle interval, and $\hat\gamma=\gamma^2$ above $\lambda_2$. The exponential enforces the negative correlation the paper observes between contrast and calibration error, the factor $k$ amplifies the small normalized distances, and $\alpha>1$ allows correction of underconfidence. The final rectified logits are $L^{\text{CAC}}_i=\hat\gamma_i\,\tau\,\text{logits}_i$, so the entire mechanism is a per-image temperature rescaling that needs no training and no access to training features.

What would settle it

Compute $z$ and ECE per dataset on an open-vocabulary benchmark where the fine-tuned model is underconfident on unseen classes while its logits closely match the original CLIP's; CAC would leave confidence nearly unchanged, so an ECE that remains large would show that $z$ does not track miscalibration there.

Watch

Extended reading notes

Core claim

The paper's central claim is that miscalibration in fine-tuned CLIP is caused by degraded intra-class and inter-class discriminability on unseen classes, and that this degradation can be read off directly from the logits. Because the original CLIP is better calibrated, the paper treats the difference between its logits and the fine-tuned model's logits as a confidence-bias signal $z=\frac{1}{N}\sum_{i=1}^N|P_i-\hat P_i|$, then maps $z$ through $\gamma=\alpha e^{-kz}$ and a piecewise squaring operation to obtain a per-image scaling weight. Multiplying the fine-tuned logits by this weight realigns their confidence with the original model's image-text alignment, restoring calibration on both train and unseen classes. The paper states that this mechanism makes CAC training-free, parameter-efficient, and applicable to any CLIP fine-tuning method.

Load-bearing premise

The method assumes that the per-image distance between the fine-tuned model's class scores and the original model's class scores reliably signals how far off the fine-tuned model's confidence is, across every dataset and tuning method.

Editorial extensions

If this is right

  • CAC can be added on top of any CLIP fine-tuning method at inference time, since it only compares original and fine-tuned logit vectors.
  • It calibrates train classes and unseen classes with the same mechanism, closing a gap left by text-feature-only calibration methods such as DAC.
  • Because it only rescales logits, it preserves the fine-tuned model's accuracy and adds no inference overhead.
  • The reported gains hold across 11 datasets and 5 fine-tuning methods, including PromptSRC, which the paper notes is already well calibrated before CAC is applied.
  • The mechanism is designed to transfer to any CLIP-based contrastive model, because it relies only on the original model's image-text alignment rather than on task-specific training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: CAC could be applied to class sets outside the 11-classification suite, such as zero-shot retrieval or open-vocabulary detection, as long as a reference CLIP logit exists; the per-image weighting does not depend on the task head.
  • The paper's own conclusion flags the need for manual parameter tuning of $k$, $\alpha$, $\lambda_1$, and $\lambda_2$; since the default constants are selected to minimize ECE on the evaluation suite, an automated per-dataset choice of constants would be a cleaner test of the mechanism itself.
  • A stronger version of the paper's claim would imply that CAC also works when the reference is a differently seeded or partially fine-tuned CLIP checkpoint rather than the frozen pretrained model; the paper does not run this comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Contrast-Aware Calibration (CAC), a post-hoc calibration method for fine-tuned CLIP models in open-vocabulary settings. CAC computes an L1 distance z between the logits of the original CLIP and a fine-tuned CLIP (Eq. 5), transforms z with an exponential function (Eq. 6) and a piecewise thresholding function (Eq. 7), and uses the resulting weight to rescale the fine-tuned logits (Eq. 8). The authors claim that this method consistently achieves the best calibration performance on both train and unseen classes across 11 datasets and 5 fine-tuning methods, while preserving accuracy and inference speed. The paper also presents an empirical analysis linking the contrast metric (Eq. 3) to ECE and argues that poor intra-class and inter-class discriminative ability on unseen classes is the root cause of miscalibration.

Significance. If the central empirical claim were established, CAC would be a simple, training-free calibration method that addresses both train and unseen classes in fine-tuned CLIP, an improvement over prior methods such as DAC that handle only unseen classes. The connection between the contrast metric and calibration is a useful observation, and the proposed method is conceptually clean and parameter-efficient in the sense of using only four global constants. However, the current experimental design does not support the headline claim: the hyperparameters are selected on the same unseen-class test sets used for evaluation, no error bars are reported, and the proposed mechanism linking the L1 logit distance to calibration is not validated at the per-sample level. These issues are fixable with additional experiments, but as presented the evidence is not sufficient to conclude that CAC is consistently superior.

major comments (4)
  1. [§6.1 and §6.3, Tables 3, 5, 6] The default hyperparameters k=15, alpha=1.10, lambda1=0.9, lambda2=1.0 are selected by ablations over the same 11 unseen-class test sets (Tables 3, 5, 6) whose calibration errors are reported in Tables 1, 14-17, so the reported 'consistently best' results are optimistically biased by test-set selection. Please provide a held-out protocol, for example selecting constants on train classes or on a subset of datasets and evaluating on the remaining datasets, or using a separate calibration set.
  2. [§4.3 and §5.1, Eq. (5)] The claim that the per-image L1 logit distance z is a surrogate for the label-dependent contrast metric in Eq. (3) and monotonically tracks miscalibration is supported only by dataset-level scatter in Figure 1(c-d). No per-sample validation of z against confidence residuals is provided, so the mechanism by which CAC improves calibration is not established. Please add per-sample analyses (e.g., binned z versus accuracy/confidence) across fine-tuning methods and datasets.
  3. [§5.3 and Abstract] The claim that CAC does not sacrifice inference speed is contradicted by Eq. (5), which requires a second forward pass through the original CLIP for every test image to obtain P_i. No latency measurement is reported. Please either measure and report the actual inference overhead or revise the claim to state the additional computational cost.
  4. [Tables 1, 2, 14-17] Results are reported as averages over 3 runs without error bars or variance. Given that some improvements are small (e.g., KgCoOp unseen ECE from 3.98 to 3.85 in Table 1), the statistical significance of the claimed gains cannot be assessed. Please report standard deviations or confidence intervals for all main results.
minor comments (5)
  1. [Author footnote] The manuscript is labeled 'Preliminary work' in the author footnote; please clarify the submission status if this is intended for review.
  2. [§4.2, Observation 1] The phrase 'untruth classes' should be 'untrue classes' or 'incorrect classes'.
  3. [Appendix A, Table 7] Appendix A lists hyperparameters for ProDA, but ProDA is not among the compared fine-tuning methods in the main text; either add it to the experiments or remove the entry.
  4. [Tables 9-17] The column labeled 'A VG' likely denotes the average across datasets; please rename it to 'Avg' for clarity.
  5. [Figure 1] Figure 1(a-b) is referenced as showing reliability diagrams, but the figures are not visible in the provided text; please ensure the figures are included in the submission.

Circularity Check

1 steps flagged · score 6.0 of 10

CAC's headline success is selected on the reported test sets: k, alpha, lambda1, lambda2 are chosen by ablations over the same 11 unseen-class evaluations that appear in the main tables.

  1. fitted input called prediction [Section 6.1 (Implementation details) and Section 6.3 (Ablation Studies), with Tables 3-6 vs Tables 1/14-17]
    "In the main experiments of this paper, we set k = 15 and α = 1.10 as the default parameters for our Contrast-Aware Weight. We selected λ1 = 0.9 and λ2 = 1.0 as the threshold values for the piecewise function. The rationality of these parameter choices will be validated through ablation experiments."

    The ablations that 'validate' these choices (Tables 3-6) measure ECE on the same 11 datasets and 5 tuning methods whose unseen-class ECE is then reported as CAC's advantage in Tables 1 and 14-17. The default tuple (k=15, alpha=1.10, lambda1=0.9, lambda2=1.0) is the one that minimizes those very averages, e.g. CoCoOp ECE 4.24 at k=15 versus 4.82 at k=10 and 6.41 at k=20. The claim that CAC 'consistently achieved the best calibration effect' is therefore a fitted optimum over the evaluation collection, not a parameter-free derivation or a held-out prediction. The paper's own Conclusion admits 'the need for manual parameter tuning for each module.'

full rationale

The derivation chain is otherwise not circular: z in Eq. 5 is an explicitly defined feature (L1 distance between original and fine-tuned CLIP logits), the transformations in Eqs. 6-7 are stated monotone maps, and the contrast-ECE link in Section 4.2 is an empirical hypothesis supported by dataset-level scatter rather than an identity built into the equations. No load-bearing self-citation or imported uniqueness theorem appears. The circularity is confined to the empirical headline: the four global hyperparameters are selected by minimizing ECE on the same unseen-class test sets that are then used to demonstrate superiority, so the reported 'consistent' gains are partly self-confirming. The inference-speed claim (Section 5.3) is also questionable because computing z requires an additional original-CLIP forward pass, but that is a factual/correctness concern, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four hand-tuned hyperparameters selected by ablation over the evaluation datasets, plus an asserted bridge between a label-dependent contrast metric and a per-image L1 logit distance. No code or data artifacts are provided, so behavior outside the reported datasets is unverified.

free parameters (4)
  • k = 15
    Exponential scaling factor in Eq. (6); selected by sweeping k=10,15,20,25 over the 11 test datasets (Table 3) and picking the best ECE.
  • alpha = 1.10
    Upper bound of CAW in Eq. (6); selected by sweeping alpha=1.00,1.05,1.10,1.20 over the 11 test datasets (Table 5).
  • lambda1 = 0.9
    Lower threshold in piecewise function Eq. (7); chosen by testing 0.85 and 0.95 (Table 6).
  • lambda2 = 1.0
    Upper threshold in piecewise function Eq. (7); chosen by testing 0.95 and 1.05 (Table 6).
assumptions (4)
  • domain assumption Original CLIP logits are well-calibrated for both train and unseen classes and can serve as a reference for fine-tuned CLIP.
    Invoked throughout Section 5.1; supported only by citations to Minderer et al. (2021) and Tu et al. (2023), not re-verified here.
  • ad hoc to paper Higher contrast (Eq. 3) is negatively correlated with ECE across datasets, and this dataset-level relationship transfers to a per-image L1 logit distance.
    Section 4.2 shows only coarse scatter plots across 11 datasets; the transfer to Eq. (5) is asserted without derivation.
  • ad hoc to paper The exponential and piecewise transformations in Eqs. (6)-(7) preserve the desired monotonic relationship between z and calibration error.
    The function is chosen because its range and monotonicity are convenient; no derivation from first principles is provided.
  • domain assumption The logits output by CLIP are equivalent to the similarity matrix used in the contrast metric (S = logits_CLIP).
    Equation (4) states this as a definitional identity for contrastive models; it is true by construction for CLIP.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contrast-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment." pith.science (2026). https://pith.science/paper/OIOFSLRI

@misc{pith2026250119060,
  author       = {Pith},
  title        = {Pith review of: Contrast-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OIOFSLRI}},
  note         = {Machine review of arXiv:2501.19060}
}
read the original abstract

Vision-language models (VLMs), such as CLIP, have demonstrated exceptional generalization capabilities and can quickly adapt to downstream tasks through prompt fine-tuning. Unfortunately, in classification tasks involving non-training classes, known as open-vocabulary setting, fine-tuned VLMs often overfit to train classes, resulting in a misalignment between confidence scores and actual accuracy on unseen classes, which significantly undermines their reliability in real-world deployments. Existing confidence calibration methods typically require training parameters or analyzing features from the training dataset, restricting their ability to generalize unseen classes without corresponding train data. Moreover, VLM-specific calibration methods rely solely on text features from train classes as calibration indicators, which inherently limits their ability to calibrate train classes. To address these challenges, we propose an effective multimodal calibration method Contrast-Aware Calibration (CAC). Building on the original CLIP's zero-shot adaptability and the conclusion from empirical analysis that poor intra-class and inter-class discriminative ability on unseen classes is the root cause, we calculate calibration weights based on the contrastive difference between the original and fine-tuned CLIP. This method not only adapts to calibrating unseen classes but also overcomes the limitations of previous VLM calibration methods that could not calibrate train classes. In experiments involving 11 datasets with 5 fine-tuning methods, CAC consistently achieved the best calibration effect on both train and unseen classes without sacrificing accuracy and inference speed.

Figures

Figures reproduced from arXiv: 2501.19060 by the authors.

Figure 1
Figure 1. (a) & (b) The reliability of KgCoOp evaluated on the FGVCAircraft and Food101 datasets. (c) & (d) The relationship between contrast and ECE for the logits output by KgCoOp and MaPLe across 11 datasets. cross-modal fine-tuning methods, the results of KgCoOp and MaPLe are shown in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning

    cs.CV 2026-01 conditional novelty 5.0 of 10

    SoC replaces the strong orthogonality constraint in O-TPT with a Huber-based penalty that caps the repulsion of semantically similar class prototypes, improving expected calibration error across 11 datasets and 4 dist...

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [2]

    ImageNet: a large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: a large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 248–255,

    work page 2009

  2. [9]

    M., Vedaldi, A., Zisserman, A., and Jawahar, C

    Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498–3505,

    work page 2012

  3. [11]

    Visual-language prompt tuning with knowledge-guided context optimization

    Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pp. 6757–6767,

    work page 2023

  4. [12]

    Task residual for tuning vision-language models

    Yu, T., Lu, Z., Jin, X., Chen, Z., and Wang, X. Task residual for tuning vision-language models. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10899–10909,

    work page 2023

  5. [13]

    C., and Liu, Z

    Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In Proceed- ings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816–16825, 2022a. Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Jour- nal of Computer Vision, pp. 2...

    work page 2022

  6. [15]

    The calibration performance is averaged across three variants

    Expected Calibration Error (ECE) on diverse downstream datasets using various tuning methods for CLIP-ViT-B/16. The calibration performance is averaged across three variants. (a) Train Method Caltech101 DTD EuroSAT FGVC Food101 ImageNet OF OP CARS UCF SUN A VG ZeroshotCLIP 6.48 4.71 8.29 2.71 1.57 1.53 3.09 2.25 3.80 1.54 3.43 3.58 CoCoOp 1.45 2.20 7.19 3...

    work page

  7. [16]

    Accuracy comparison of existing prompt tuning in the open-vocabulary setting. (a) Train Method Caltech101 DTD EuroSAT FGVC Food101 ImageNet OF OP CARS UCF SUN A VG ZeroshotCLIP 97.16 53.24 57.00 27.61 90.06 72.42 71.70 91.33 63.77 69.32 70.89 69.50 CoCoOp 97.91 77.16 85.61 35.07 90.53 75.89 94.24 95.04 71.67 79.29 81.94 80.40 KgCoOp 97.89 80.71 89.51 39.3...

    work page 2017

  8. [2004]

    Additionally, we performed a remote sensing recognition task using the EuroSAT (Helber et al.,

    datasets, fine-grained object recognition involving datasets such as Oxford Pets (Parkhi et al., 2012), Food-101 (Bossard et al., 2014), Stanford Cars (Krause et al., 2013), Oxford Flowers102 (Nilsback & Zisserman, 2008), and FGVCAircraft (Maji et al., 2023). Additionally, we performed a remote sensing recognition task using the EuroSAT (Helber et al.,

    work page 2012

Show all 16 references
  1. [2008]

    W., Zhang, L., Jerfel, G., and Tran, D

    Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., and Tran, D. Measuring calibration in deep learning. In Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 38–41,

    2019

  2. [2009]

    Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories

    Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In the 2024 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 178,

    2024

  3. [2014]

    Describing textures in the wild

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606–3613,

    2014

  4. [2019]

    Towards calibrated robust fine-tuning of vision-language models

    Oh, C., Kim, M., Lim, H., Park, J., Jeong, E., Cheng, Z., and Song, K. Towards calibrated robust fine-tuning of vision-language models. CoRR, abs/2311.01723,

    arXiv

  5. [2020]

    Lee, D., Song, S., Suh, J., Choi, J., Lee, S., and Kim, H. J. Read-only prompt optimization for vision-language few- shot learning. In Proceedings of the 2023 IEEE Interna- tional Conference on Computer Vision, pp. 1401–1411,

    2023

  6. [2022]

    3D object representations for fine-grained categorization

    Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3D object representations for fine-grained categorization. In the 2013 IEEE International Conference on Computer Vision Workshops, pp. 554–561,

    2013

  7. [2023]

    U., Rasheed, H., Maaz, M., Khan, S., and Khan, F

    Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. MaPLe: Multi-modal prompt learning. In Proceedings of the 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 19113–19122, 2023a. Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M...

    2023

  8. [2024]

    A., Oliva, A., and Torralba, A

    Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485– 3492,

    2010

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.