{"id":"ca125207-c758-48f7-a5bf-f0832ded6042","arxiv_id":"2412.14640","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"APT uses cross-attention between image patches and text embeddings to dynamically refine CLIP prompts for fine-grained few-shot classification.","lead":"The paper introduces Adaptive Prompt Tuning (APT), a cross-attention layer that dynamically adjusts CLIP's text prompts using the image being classified, and tests it on fine-grained few-shot datasets. It reports accuracy gains over static prompt tuning methods and adds Monte-Carlo dropout for uncertainty estimates.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is unverified because the main few-shot curves omit CoCoOp, the closest adaptive-prompt baseline, and the only CoCoOp comparison in Table 1 reports an implausibly low CUBirds score, so APT's advantage over existing conditional prompting is not established.","rationale":"The paper's core novelty is the use of cross-attention to make text prompts image-specific, but the evidence for this being an advance over existing adaptive prompting is incomplete. CoCoOp is the direct prior work that also produces image-conditional prompts, yet it is absent from Figure 4, the main few-shot accuracy comparison against CoOp and VPT. The only CoCoOp numbers are in Table 1, and the CUBirds entry is implausibly low, which undermines confidence in that comparison. This is not a flaw in the method's internal logic but a gap in the empirical support for the central claim: the reported gains over static prompts could plausibly arise from any image-conditional mechanism, not specifically from the cross-attention design. The concern is load-bearing because the paper's stated contribution and 'state-of-the-art' language depend on showing superiority over the closest competing adaptive method. The proposed test, adding a correct CoCoOp baseline to the main protocol, would settle whether the central claim holds. This aligns with the reader's identified weakest assumption, so the conditional verdict should remain.","tokens_in":14014,"tokens_out":6071,"duration_ms":46950,"concrete_test":"Run the official CoCoOp implementation on the same datasets, shots, seeds, and backbone as Figure 4, and verify whether the Table 1 CUBirds CoCoOp score (New=8.04) is reproducible; if APT does not beat the corrected CoCoOp on the main few-shot curves, the state-of-the-art claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim (Section 4.1) is supported only by comparisons to CoOp and VPT in Figure 4; both methods use static prompts and do not test the paper's core idea of image-conditional prompt adaptation. CoCoOp (Zhou et al., 2022a) already conditions prompts on image features, making it the necessary baseline. The sole CoCoOp results appear in Table 1, where the CUBirds New-class accuracy is reported as 8.04 (F1 14.40), far below typical published CoCoOp values on the same base-to-new split; this suggests a non-standard implementation. Without a correct CoCoOp comparison on the main few-shot curves, the claim of state-of-the-art gains over adaptive prompting is unsupported, and the observed improvements over static baselines cannot be attributed to the cross-attention mechanism rather than to the mere fact of image conditioning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Adaptive Prompt Tuning (APT), a cross-attention module that refines CLIP text-prompt embeddings using image features at inference time, with frozen CLIP encoders and a lightweight trainable layer. The method is evaluated on fine-grained few-shot classification (CUBirds, Oxford Flowers, FGVC Aircraft, and Caltech101) against CoOp and VPT, and is augmented with Monte-Carlo Dropout for uncertainty quantification, reporting ECE, reliability plots, confidence-uncertainty plots, and OOD analysis. The central claims are that APT improves accuracy over static prompt tuning and that MCD yields calibrated uncertainty without sacrificing accuracy.","tokens_in":14152,"tokens_out":4774,"duration_ms":93851,"significance":"If the claims are substantiated, the paper would offer a simple, parameter-efficient way to make prompt tuning image-conditional, which is a plausible extension of CoCoOp-style conditional prompting. The explicit integration of MCD with the prompt-tuning pipeline is a practical direction, and the release of code is a strength. However, the empirical evidence as presented is incomplete: the main few-shot curves omit the closest adaptive-prompt baseline (CoCoOp), the single CoCoOp comparison reports an implausibly low number, the ECE formula is mis-specified, and accuracy results are reported without variance or significance tests. These gaps prevent the paper from establishing its claimed state-of-the-art performance, though they appear fixable within the manuscript's scope.","major_comments":[{"comment":"The central claim that APT outperforms existing adaptive prompting is not tested against the closest baseline. Figure 4 compares only with CoOp and VPT, which are static-prompt methods, while APT's premise is image-conditioned adaptation; CoCoOp (Zhou et al., 2022a) is the required baseline. The only CoCoOp results, in Table 1, report CUBirds New-class accuracy 8.04 and F1 14.40, far below published CoCoOp values on the same base-to-new protocol (e.g., roughly 60-70 on new classes in the original CoCoOp paper). This discrepancy indicates a non-standard implementation, so the table does not support the claim that APT improves over conditional prompting. Please add CoCoOp to the main few-shot curves and correct the CoCoOp results.","section":"Section 4.1, Table 1"},{"comment":"The Expected Calibration Error formula is mis-specified. The weight |B_p|/N_p is always 1 because N_p is defined as the number of data points in bin B_p, reducing ECE to an unweighted average of per-bin gaps rather than the standard sample-weighted ECE. The denominator should be the total number of data points N, or the estimator should be explicitly stated as equal-weight ECE. Additionally, the summation index M in the formula conflicts with the earlier use of M as the MC sample size. Please recompute all ECE values in Section 4.2 with a corrected formula.","section":"Section 3.4, ECE definition"},{"comment":"The method description is ambiguous about what serves as keys and values in the cross-attention module. The text states that 'keys k and values v are created from z' where z is the CLS token, a single d-dimensional vector, which would provide only one key/value and cannot implement meaningful cross-attention over image regions. Figure 1, however, shows the 'Sequence of Patches' entering the cross-attention block. Please clarify whether the keys/values are the patch tokens or the CLS token, and make the text and figure consistent; this is essential for reproducing the method.","section":"Section 3.2, Figure 1"},{"comment":"Accuracy results are reported as averages of 3 runs without standard deviations or significance testing. In Figure 4, several curves are close (e.g., Flowers102 at 8 and 16 shots), and a 3-run average without spread does not demonstrate a reliable difference. Please report per-seed results or add standard deviations/confidence intervals for all datasets and shot values, and for the base/new results in Table 1.","section":"Section 4.1 and Table 1"},{"comment":"The paper claims that Monte-Carlo Dropout improves calibration 'without drops in accuracy,' but the accuracy results in Figure 4 do not state whether inference used MC sampling or a deterministic forward pass, and no direct comparison of accuracy with versus without MCD is reported. Please specify the inference protocol for all accuracy results and provide a head-to-head comparison (deterministic APT vs APT+MCD) to substantiate the 'no drop in accuracy' claim.","section":"Sections 1, 3.4, 4.2"}],"minor_comments":[{"comment":"The paragraph beginning 'As shown in the preliminary experiments ...' through '... general architecture of the proposed model.' is duplicated verbatim; please remove the repetition.","section":"Section 3.2"},{"comment":"The caption contains the typo 'netter fit the images' which should be 'better fit the images'.","section":"Figure 1 caption"},{"comment":"The text repeatedly misspells 'FGVC Aircraft' as 'FVGC Aircraft' in the calibration discussion; please correct it.","section":"Section 4.2"},{"comment":"Figure 5 appears only as a caption with no corresponding plot in the provided text; please add the figure or remove the reference.","section":"Figure 5"},{"comment":"The VPT description states that learnable tokens are added 'to the Transformer layer' without specifying whether this applies to one or all layers; the notation X = [P1,...,PK,x1,...,xN] would benefit from a precise definition of layer indices and token dimensions.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The CoCoOp numbers in Table 1 are implausibly low for CUBirds (New=8.04, F1=14.40); I would ask the authors to verify their CoCoOp implementation and report published or reproducible values. The paper would also benefit from a clearer statement of experimental protocol, especially regarding MC dropout during inference for the accuracy numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is clean: add a cross-attention layer that lets CLIP text prompts attend to image patches at inference time, train only that layer, and use the existing dropout for MC-dropout uncertainty. That is a reasonable and testable variation on CoOp and VPT, and the preliminary variance analysis gives a sensible motivation for why static prompts struggle on FGVC Aircraft. The reported gains on Aircraft (27% to 47% vs 17% zero-shot) are substantial, the code is public, and the UQ analysis, while not deep, is a useful addition to a literature that mostly ignores calibration in prompt tuning.\n\nThe problem is that the paper's central claim is not actually tested. The main few-shot curves compare APT only to CoOp and VPT, both static prompt methods. CoCoOp already conditions prompts on image features, so it is the necessary baseline for showing that cross-attention adds something beyond mere image conditioning. The only CoCoOp numbers, in Table 1, include a CUBirds New-class accuracy of 8.04, far below any published CoCoOp result I know of on that split. That is not a small gap; it is a red flag that the CoCoOp implementation is broken. With correct CoCoOp numbers, the claimed state-of-the-art results may shrink considerably.\n\nOther soft spots are real but more minor. The average of 3 runs has no error bars or significance tests. Section 3.2 has a duplicated paragraph. The ECE formula in Section 3.4 is mis-specified: weighting by |B_p|/N_p collapses to a mean over bins rather than the standard sample-weighted ECE. The UQ experiments would also be stronger with a deterministic APT baseline to show what MC dropout adds.\n\nNone of this is fatal to the method. The fixes are straightforward, and the paper deserves a serious referee. I would send it to review, but with a clear instruction: put CoCoOp on the main curves with a verifiable implementation, add error bars or significance tests, and correct the ECE formula. If the authors do that and the CoCoOp results come out right, the paper could be a decent contribution. As is, I would not cite it for the claimed advantage over adaptive prompting.","headline":"APT is a plausible cross-attention prompt-tuning variant with promising fine-grained few-shot results, but the paper never validates against CoCoOp properly, so the central claim of beating adaptive prompting is unproven.","tokens_in":14761,"tokens_out":1376,"would_cite":false,"duration_ms":10948,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptive Prompt Tuning makes CLIP's text prompts react to the image at hand, and this per-image refinement is what lifts fine-grained few-shot accuracy over static prompt tuning.","keywords":["CLIP","few-shot learning","fine-grained image classification","prompt tuning","cross-attention","adaptive inference","uncertainty quantification","Monte-Carlo dropout"],"falsifier":"Run APT against CoCoOp on the same 1, 2, 4, 8, 16-shot protocol on FGVC Aircraft and CUBirds, and ablate the cross-attention layer by replacing it with a static learned prompt of equal parameter count; if APT does not match or beat CoCoOp or removing the layer does not lower Aircraft accuracy, the central claim fails.","tokens_in":13745,"feed_emoji":"🎯","tokens_out":8555,"duration_ms":63008,"temperature":0.7,"pith_summary":"The paper proposes Adaptive Prompt Tuning (APT), a method that makes CLIP's text prompts specific to each test image before classifying it. Instead of learning one fixed prompt (CoOp) or fixed visual tokens (VPT), APT uses a cross-attention layer in which text-encoded prompts act as queries and the image's ViT patch features act as keys and values; only that layer is trained on the few labeled examples while CLIP stays frozen. The claim is that this dynamic, image-guided refinement helps most when classes differ only subtly and vary a lot within a class, as in FGVC Aircraft and CUBirds, and that adding Monte-Carlo Dropout yields calibrated confidence estimates without sacrificing accuracy. If correct, this offers a parameter-lean way to adapt foundation models to specialist classification tasks with very few labels and trustworthy confidence.","feed_headline":"One attention layer tailors CLIP prompts per image for few-shot accuracy","feed_subtitle":"Image-guided prompt refinement beats static prompts on fine-grained few-shot classification, with calibrated confidence via MC dropout.","key_machinery":"The load-bearing component is the cross-attention module inserted between CLIP's frozen text encoder and the cosine-similarity classifier. Queries come from the text-feature matrix W (one column per class prompt), keys and values from the sequence of image patch features z, so the refined prompt is conditioned on the specific image; layer normalization, a feed-forward block, skip connections, and 20% dropout complete the module, and the dropout doubles as the stochastic sampling mechanism for MC Dropout. This is the only trained part of the model, so the method adapts per image without unfreezing CLIP or storing the training set at inference.","core_discovery":"The central discovery claimed is that a single cross-attention refinement of CLIP text embeddings by image patches is enough to outperform static prompt tuning on fine-grained few-shot classification. APT feeds the text features as queries and the image patch tokens as keys and values, producing tuned text features $W'$ that are then compared with the image CLS token by cosine similarity. In experiments on FGVC Aircraft, Oxford Flowers, and CUBirds, APT reports higher accuracy than CoOp and VPT at most shot counts; on Aircraft it rises from 27% one-shot to 47% at 16 shots versus a 17% zero-shot baseline, and on the base-to-new generalization split its harmonic-mean F1 beats CoOp and CoCoOp on Aircraft and Flowers while matching CLIP on CUBirds. The paper also claims that keeping dropout active at inference (MC Dropout, 100 samples) gives uncertainty estimates that separate correct from incorrect predictions and whose ECE falls as shots increase.","pith_inferences":["A natural port would be to other frozen dual-encoder models: the cross-attention layer is small and trained independently of CLIP, so the same recipe could be tried on any vision-language model that exposes patch tokens and text embeddings; the paper only demonstrates it with CLIP ViT-B/16 on four datasets.","The appendix's variance analysis predicts where dynamic prompting will help: datasets with high intra-class and low inter-class image-feature variance should show the largest APT advantage, so ranking additional fine-grained datasets by that gap is a direct testable extension.","The Caltech101 results suggest that cluttered backgrounds can pull the cross-attention toward irrelevant patches; masking salient regions or adding a background-suppression term would test whether that is the cause.","MCD at a fixed 20% dropout is one calibration recipe; attaching Deep Ensembles or variational inference to the same cross-attention layer would test whether the residual overconfidence seen in high-confidence bins is a property of dropout or of the layer itself."],"forward_implications":["On FGVC Aircraft, APT's accuracy rises from 27% at one shot to 47% at 16 shots, a larger gain over the 17% zero-shot CLIP baseline than CoOp or VPT achieve.","On Oxford Flowers, APT reaches 84% with a single example per class and 97% at 16 shots, with CoOp close behind.","On the base-to-new generalization split, APT's harmonic-mean F1 exceeds CoOp and CoCoOp on FGVC Aircraft (36.46 vs. 28.75 and 27.74) and Oxford Flowers (83.23 vs. 74.06 and 81.71), and matches the CLIP baseline on CUBirds.","Keeping dropout active at inference turns APT into a stochastic model whose expected calibration error falls as the number of training shots grows, giving confidence estimates that track accuracy."],"supporting_citations":[{"why":"Defines CLIP and the frozen image/text encoder setup, the cosine-similarity classifier, and the zero-shot baseline used throughout.","marker":"Radford et al., 2021"},{"why":"Supplies the multi-head cross-attention formulation, layer normalization, feed-forward block, and residual connections that make up the APT module.","marker":"Vaswani et al., 2017"},{"why":"CoOp is the static-prompt baseline APT is compared against; its learned context vector is the main thing APT replaces.","marker":"Zhou et al., 2022b"},{"why":"CoCoOp is the image-conditioned prompt baseline used in the base-to-new generalization comparison that APT claims to beat.","marker":"Zhou et al., 2022a"},{"why":"VPT is the visual-prompt-token baseline whose accuracy curves APT is measured against.","marker":"Jia et al., 2022"},{"why":"Justifies using dropout at inference as the Monte-Carlo sampling procedure that generates APT's uncertainty estimates.","marker":"Gal and Ghahramani, 2016"},{"why":"Provides the CUBirds dataset used for fine-grained few-shot evaluation.","marker":"Wah et al., 2011"},{"why":"Provides the Oxford Flowers dataset used for fine-grained few-shot evaluation.","marker":"Nilsback and Zisserman, 2008"},{"why":"Provides the FGVC Aircraft dataset, the high-intra-class-variance case where APT reports its largest gains.","marker":"Maji et al., 2013"},{"why":"The only prior CLIP few-shot UQ approach; APT's MCD-based estimates are positioned as a simpler, dataset-independent alternative.","marker":"Miao et al., 2024"}],"fun_headline_variants":["Cross-attention refines CLIP prompts per image for few-shot fine-grained","Vision-guided prompt tuning with cross-attention improves few-shot CLIP","Adaptive prompt tuning: cross-attention tailors CLIP per image in few-shot","Dynamic CLIP prompts via cross-attention beat static tuning on fine-grained few-shot","Image-guided cross-attention sharpens CLIP prompts for fine-grained few-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The premise that one thin cross-attention layer, trained on only a handful of examples per class, can generalize to unseen images and re-weight text prompts better than static or image-conditioned prompts; this is assumed in the method definition rather than established by an ablation.","fun_headline_variants_meta":{"raw":{"variants":["Cross-attention refines CLIP prompts per image for few-shot fine-grained","Vision-guided prompt tuning with cross-attention improves few-shot CLIP","Adaptive prompt tuning: cross-attention tailors CLIP per image in few-shot","Dynamic CLIP prompts via cross-attention beat static tuning on fine-grained few-shot","Image-guided cross-attention sharpens CLIP prompts for fine-grained few-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3305,"prompt_tokens":983,"completion_tokens":2322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":2220}},"tokens_in":599,"tokens_out":2322,"duration_ms":47561,"temperature":1.0,"reasoning_tokens":2220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:02:32.453469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run APT against CoCoOp on the same 1, 2, 4, 8, 16-shot protocol on FGVC Aircraft and CUBirds, and ablate the cross-attention layer by replacing it with a static learned prompt of equal parameter count; if APT does not match or beat CoCoOp or removing the layer does not lower Aircraft accuracy, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VPT is the visual-prompt-token baseline whose accuracy curves APT is measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CUBirds dataset used for fine-grained few-shot evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FGVC Aircraft dataset, the high-intra-class-variance case where APT reports its largest gains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The only prior CLIP few-shot UQ approach; APT's MCD-based estimates are positioned as a simpler, dataset-independent alternative."}],"review_version":1}