{"id":"02b8a631-d0f5-4510-bba8-d5d27e5495ca","arxiv_id":"2505.02406","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Token Coordinated Prompt Attention gives CLS and image tokens separately matched prompt pools and masks attention, improving visual prompt tuning accuracy on HTA and VTAB.","lead":"This paper proposes a plug-in module, TCPA, that assigns different prompts to different image tokens when fine-tuning Vision Transformers. Adding it to existing visual prompting methods yields consistent, small accuracy gains on standard image classification benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equal-prompt-capacity baselines are missing, so TCPA's reported gains may reflect a larger prompt budget rather than token-coordinated assignment.","rationale":"The reader's verdict is already CONDITIONAL, and my read keeps it there. The reader's weakest assumption points to matching stability and the functional effect of unnormalized masking; I agree those are underexamined. However, the more decisive gap is the absence of any control that holds the prompt budget fixed while removing coordination. The paper's own ablation and hyperparameter study indicate that simply enlarging the prompt pool changes accuracy substantially, so the positive deltas in Tables 1 and 2 do not yet isolate the token-coordinated mechanism. This is a standard attribution confound, not evidence of misconduct: the method may well work as claimed, and the code is promised, but the causal claim in the abstract is under-supported. The concrete test above would settle whether the coordination or the added capacity drives the improvements. Since this concern is a reason to require additional evidence rather than to reject the paper, the conditional verdict stands unchanged.","tokens_in":15648,"tokens_out":6119,"duration_ms":88941,"concrete_test":"Run a controlled ablation on CUB, Stanford Dogs, GTSRB, and the three VTAB groups: (a) shared-prompt control with the same total prompt budget as TCPA, i.e., 10 CLS prompts plus 20 image prompts per layer, all attended to by every token without token-specific masking; (b) full TCPA with the same pool sizes and masking. Use 3 seeds per condition and report paired differences with standard deviations. If the shared-prompt control recovers most of the VPT-to-VPT+TCPA gap, the coordinated-assignment mechanism is not the source of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim is that TCPA 'consistently enhances' existing visual prompting methods because it assigns coordinated prompts to different tokens. The evidence for this mechanistic attribution is confounded by prompt capacity. Section 5.3 sets the CLS prompt pool to Nc=10 and the image prompt pool to Ni=20 per layer, so TCPA uses up to 30 prompt tokens per layer. In contrast, VPT's prompts are replaced rather than augmented, and VP/DAMVP/AutoVP are input-level methods with no token-level prompts at all. No experiment controls for the total number of prompt tokens: there is no baseline that adds the same 30 shared prompt tokens without token-specific masking. Table 3 shows that R-TCPA, which separates CLS and image prompts but does not perform per-token coordination, already accounts for roughly half of the total gain, and Figure 4 shows that prompt-pool size alone moves accuracy by about one point. Consequently, the observed improvements could come from the extra prompt capacity and not from the Eq. 9-13 matching and masking mechanism. This is a correctness risk in attributing the empirical results to the proposed mechanism, not an internal inconsistency. A secondary gap is that Eq. 13 zeroes post-softmax attention entries without renormalization, and the non-differentiable top-K selection is not analyzed, but the missing equal-capacity control is the more load-bearing issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Token Coordinated Prompt Attention (TCPA), a plug-and-play module for visual prompting with Vision Transformers. TCPA disentangles prompts into a CLS prompt pool and an image prompt pool, and assigns token-specific prompts via cosine-distance matching (Eq. 9), top-K binarization (Eq. 10), and binary masking of the post-softmax attention map (Eq. 13). The authors claim that this token-coordinated assignment makes extracted features more diverse and discriminative, and they report consistent accuracy improvements when TCPA is added to VP, VPT, DAMVP, AutoVP, and VFPT on the HTA and VTAB benchmarks, with negligible training-time overhead. The paper includes ablations of the two main components, hyperparameter sensitivity, attention-map visualizations, t-SNE feature visualizations, and a theoretical motivation based on the low-rankness of self-attention.","tokens_in":16005,"tokens_out":3840,"duration_ms":45890,"significance":"If the central claim is validated, TCPA would be a practically useful, low-overhead plug-in for existing visual prompting methods, with broad applicability. The paper has several strengths: evaluations across two benchmarks and multiple base methods, an ablation showing both components contribute, attention and feature visualizations, an efficiency analysis, and released code. However, the mechanistic attribution of the gains to token-coordinated assignment is not yet established because the experiments do not control for the larger prompt capacity introduced by TCPA, and the reported gains are small and lack uncertainty quantification. The theoretical discussion borrows external results and does not directly prove that prompt coordination is necessary. The central idea is plausible and the empirical evidence is suggestive, but additional controls and analysis are needed before the claim can be accepted.","major_comments":[{"comment":"The experiments do not include an equal-prompt-capacity baseline. TCPA uses a CLS prompt pool of Nc=10 and an image prompt pool of Ni=20 per layer (Section 5.3), i.e., up to 30 prompt tokens per layer, whereas VPT's prompts are replaced rather than augmented and VP/DAMVP/AutoVP have no token-level prompts. Table 3 shows that R-TCPA, which separates CLS and image prompts but does not perform per-token coordination, already accounts for roughly half of the total gain, and Figure 4 shows that prompt-pool size alone moves accuracy by about one point. To support the claim that the improvements come from the token-coordinated matching mechanism (Eqs. 9–13) and not merely from the added prompt capacity, the authors should add a control that uses the same total number of shared prompts without per-token masking, e.g., VPT with 30 prompt tokens per layer or TCPA with all mask entries set to 1. Without such a control, the central mechanistic attribution is confounded.","section":"§5.3, Tables 1–3, Figure 4"},{"comment":"Eq. 13 zeroes entries of the post-softmax attention map without renormalizing the rows, so the resulting attention matrix is not row-stochastic. The paper does not analyze how this affects the frozen ViT's attention behavior or training dynamics. The non-differentiable top-K selection in Eq. 10 also lacks any gradient or stability analysis; gradients for the prompt indicators κ appear only through the regularization terms in Eq. 14 and not through the masking path, so the claim that prompts are \"automatically assigned\" in an end-to-end trained system needs either a straight-through estimator discussion or an empirical stability check. An ablation that renormalizes masked attention rows, or an analysis of matching stability across training, would strengthen the paper considerably.","section":"§3.3, Eq. 13"},{"comment":"The reported gains are small (0.2–3.1 percentage points) and the paper provides no standard deviations, number of seeds, or significance tests. For VTAB, only group averages are reported, not per-task results, making it impossible to judge whether the \"consistent\" improvements hold across the 19 individual tasks. Given the headline claim of consistency, the authors should report per-task VTAB numbers and either multi-seed variance or a paired significance test across datasets. Hyperparameters such as top-K, prompt length, and the regularization weights λ_i and λ_c are fixed without sensitivity analysis beyond the pool-size study in Figure 4.","section":"§5.4, Tables 1–2"}],"minor_comments":[{"comment":"Figure 2's caption contains the LaTeX placeholder \"在此处键入公式。\" (\"type the formula here\"); this production artifact must be removed before publication.","section":"Figure 2 caption"},{"comment":"Table 4 contains garbled unicode tokens in its header and Figure 4 has similar encoding artifacts in the axis labels; these need to be regenerated cleanly.","section":"Table 4 and Figure 4"},{"comment":"The text says \"employing all three modules\" but only two components (R-TCPA and T-TCPA) are defined and ablated in Table 3; this appears to be a typo and should be corrected.","section":"§5.5.1"},{"comment":"In the definition of the binarized matrix, the text says \"elements of the i-th row\" but should refer to the m-th row (matching the index of the image token); also the notation for the top-K value should be consistent (K_i with subscript i, not Ki).","section":"Eq. 10"},{"comment":"The theorems are cited from prior work, but the paper's extrapolation that low-rank self-attention implies existing prompts \"tend to focus on the same image regions\" is not rigorously established; the authors should either provide a direct analysis or soften the claim, since this is the stated motivation for the token-coordination mechanism.","section":"§4, Theorems 4.1–4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a computer vision or machine learning journal, and the plug-and-play design makes it potentially useful to practitioners. The main risk is the prompt-capacity confound: the missing equal-capacity control could, in the worst case, reduce the central claim to \"more prompt tokens help,\" which would undermine the paper's novelty. This is fixable in a revision with a straightforward control experiment and, ideally, a renormalization or stability analysis of the masking. I therefore recommend major revision rather than rejection, provided the authors can supply the missing controls and uncertainty quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible, cleanly written module paper with consistent small gains, but it does not yet prove that the token-coordinated mechanism is what produces them. The missing equal-prompt-capacity baseline is a real confound, not a nitpick.\n\nWhat is actually new: TCPA splits prompts into a CLS pool and an image-token pool, then assigns prompts to individual tokens via cosine-distance matching and a binary attention mask. That specific mechanism is not in VPT, DAMVP, AutoVP, or VFPT. The paper does the right empirical work in the main: it plugs into five base methods, tests on HTA and VTAB, reports per-dataset numbers, and shows both components matter in the Table 3 ablation. The time overhead is measured and small. The code link is there.\n\nThe soft spot: Section 5.3 sets Nc=10 and Ni=20 per layer, so TCPA uses up to 30 prompt tokens per layer, while the base methods are not given the same budget. No baseline adds 30 shared prompts without the token-specific mask. The stress-test note is right that R-TCPA already accounts for roughly half of the gain, and Figure 4 shows pool size alone moves accuracy by around a point. That means the headline claim, that coordinated assignment to different tokens is what helps, is not supported by the current experiments. The gains could simply come from the extra prompt capacity. That is fixable: add an equal-capacity shared-prompt VPT baseline, and ideally a variable-K_i ablation. Also missing: standard deviations or significance tests, and the values of K_i and L_p are not reported, which makes reproduction harder. The zeroing of post-softmax attention without renormalization in Eq. 13 is worth a sentence of justification. The low-rank discussion in Section 4 is mostly imported and over-interpreted; I would cut or soften it.\n\nI would not call this fatal. The empirical claim that TCPA adds about 1-2 points on average looks robust across many point estimates, and the module is cheap. But as written, the mechanism attribution is unproven. This is exactly the kind of paper that should go to peer review: it is useful, well-scoped, and the fix is straightforward. If I were an editor, I would send it out and ask for the capacity-matched control, error bars, and full hyperparameters before acceptance.","headline":"A genuinely new token-coordinated prompt mechanism with consistent small gains, but the missing prompt-capacity control leaves the central mechanism attribution under-supported.","tokens_in":16437,"tokens_out":2264,"would_cite":false,"duration_ms":26874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-token prompt matching lifts visual prompting accuracy.","keywords":["visual prompting","visual prompt tuning","vision transformer","attention masking","prompt pool","token-coordinated prompts","parameter-efficient fine-tuning","feature diversity"],"falsifier":"Run TCPA with the same prompt pools and mask density but with random or fixed-permutation token–prompt assignments instead of cosine matching; if accuracy stays roughly the same, the coordination itself is not the source of the gains. As a second check, examine whether post-mask attention row sums collapse or destabilize compared with the unmasked baseline.","tokens_in":1745,"feed_emoji":"🧩","tokens_out":1901,"duration_ms":85366,"temperature":0.7,"pith_summary":"Visual prompting adapts a frozen pretrained Vision Transformer (ViT) by adding a small number of learnable prompt tokens, but current methods give every token the same prompts, so different tokens attend to similar image regions and the extracted features become redundant. This paper argues that prompts should be token-coordinated: CLS and image tokens should draw from separate prompt pools, and each image token should be matched to a subset of image prompts through a cosine-distance matching rule. The proposed Token Coordinated Prompt Attention (TCPA) module realizes this by building a binary attention mask that lets each token interact only with its assigned prompts, applied to the softmax attention map inside each transformer block. The paper reports that plugging TCPA into several established visual prompting methods improves accuracy on the HTA and VTAB benchmarks by roughly 0.5 to 3 percentage points while adding negligible training time. If this is right, TCPA is a cheap, general upgrade for visual prompting rather than a replacement for it.","feed_headline":"Per-token prompt matching lifts visual prompting accuracy","feed_subtitle":"Pairs each token with its own prompt, adding 0.5–3 points on visual benchmarks at tiny extra cost.","key_machinery":"The carrier of the argument is a binary attention mask built from token–prompt matching. A learnable indicator vector is attached to each prompt in each pool; for every image token $h^j_m$ the cosine distance $S(h^j_m,\\kappa^k_i)=1-\\cos(h^j_m,\\kappa^k_i)$ gives an affinity row, and the top-$K_i$ entries are set to one with all others zero (Eqs. 9–11). The CLS token gets the same treatment against the CLS prompt pool. The two masks are concatenated, expanded to the attention-map shape, and multiplied element-wise into the post-softmax attention map $\\mathrm{Attn}' = \\mathrm{Attn}\\odot M$ (Eq. 13), so each query attends only to its matched prompts while all attention weights are still computed in a single pass. This single-pass masking is what keeps the overhead low, and the disentangling of CLS versus image prompts is what the paper credits for preserving the two tokens' distinct roles.","core_discovery":"On the paper's own terms, the central claim is that the failure mode of existing visual prompting is representational collapse: because all tokens use the same prompt tokens in attention, the self-attention matrix stays low-rank after prompting and different tokens extract near-identical, biased features. TCPA breaks this by disentangling prompts into a CLS prompt pool and an image prompt pool, then assigning coordinated prompts per token. For each image token, the cosine distance to learnable prompt indicators produces an affinity matrix, whose top-K entries are binarized into a mask; the image mask and the CLS mask are concatenated and multiplied element-wise into the post-softmax attention map, so queries attend only to their own matched prompts. Training keeps the ViT frozen and optimizes only the prompt pools, the mask indicators, and the classification head, with an auxiliary term that pulls each token's selected indicator close to it. The claimed result is that extracted features become more diverse and discriminative, and that the module reliably improves every method it is attached to.","pith_inferences":["A natural controlled test the paper does not run is to match the number of extra parameters (prompt pool size and mask sparsity) while ablating the matching rule; without that control, part of the gain could be attributable to capacity rather than coordination.","Because the mask thresholding is non-differentiable, the training signal for which prompt a token receives flows only through the auxiliary cosine-distance loss; a differentiable soft assignment might train more smoothly and could be tested as a drop-in variant.","The same token-role argument should transfer to other transformer input types, such as video patch tokens or language word tokens, where tokens carry heterogeneous local information, but the paper only demonstrates vision-image tokens."],"forward_implications":["Any of the studied visual prompting methods (VP, VPT, DAMVP, AutoVP, VFPT) can be upgraded by inserting TCPA while keeping the ViT frozen, with reported per-dataset gains of roughly 0.5 to 3 percentage points on HTA.","VTAB gains appear across all three task groups (Natural, Specialized, Structured), with the largest reported gains in Specialized and Structured.","Training-time overhead is small: on CUB the reported per-epoch time rises by about 0.05 to 0.3 seconds depending on the base method.","Because attention weights are computed once and masks are applied only before the multiplication with values, enlarging the prompt pool does not multiply feed-forward network cost.","The intended payoff is more diverse and discriminative features, which the paper supports with attention-map and t-SNE visualizations."],"supporting_citations":[{"why":"Defines VPT, the token-level visual prompting baseline whose shared prompts TCPA replaces and against which TCPA is most directly compared.","marker":"(Jia et al., 2022)"},{"why":"Defines VP, the input-level prompting method that TCPA augments at the token level in the HTA experiments.","marker":"(Bahng et al., 2022)"},{"why":"Provides DAMVP and the HTA benchmark setup, the main comparison baseline and training protocol used by the paper.","marker":"(Huang et al., 2023)"},{"why":"Supplies the frozen ViT-B/16 backbone used in all experiments.","marker":"(Dosovitskiy et al., 2020)"},{"why":"Gives Theorem 4.1 on low-rank self-attention, used to argue that shared prompts cause redundant attention.","marker":"(Wang et al., 2020)"},{"why":"Gives Theorem 4.2 on low-rank attention after prompting, used to extend the redundancy argument to prompted attention.","marker":"(Kim et al., 2024)"},{"why":"Defines AutoVP, another baseline that TCPA is added to and compared against.","marker":"(Tsao et al., 2024)"},{"why":"Defines VFPT, the strongest baseline on which TCPA is tested.","marker":"(Zeng et al., 2024)"},{"why":"Defines the VTAB benchmark used for the second set of experiments.","marker":"(Zhai et al., 2019)"}],"fun_headline_variants":["Token-matched prompts unlock ViT diversity","One prompt per token: visual prompting upgrade","Coordinated prompts boost visual prompting","Per-token prompts improve ViT feature diversity"],"cache_read_input_tokens":18560,"weakest_assumption_plain":"The load-bearing premises are that the binarized cosine-distance matching assigns stable, meaningful token–prompt pairs and that zeroing entries of the post-softmax attention map without renormalizing leaves the frozen transformer's attention mechanism functionally intact; neither is analyzed in the paper, and if either fails the reported gains could come from the extra parameters rather than from coordination.","fun_headline_variants_meta":{"raw":{"variants":["Token-matched prompts unlock ViT diversity","One prompt per token: visual prompting upgrade","Coordinated prompts boost visual prompting","Per-token prompts improve ViT feature diversity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1391,"prompt_tokens":992,"completion_tokens":399,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":608,"tokens_out":399,"duration_ms":4810,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:52:32.823511+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TCPA with the same prompt pools and mask density but with random or fixed-permutation token–prompt assignments instead of cosine matching; if accuracy stays roughly the same, the coordination itself is not the source of the gains. As a second check, examine whether post-mask attention row sums collapse or destabilize compared with the unmasked baseline.","supporting_citations":[{"cited_title":"Visual prompt tuning","cited_arxiv_id":null,"evidence_quote":"Defines VPT, the token-level visual prompting baseline whose shared prompts TCPA replaces and against which TCPA is most directly compared."},{"cited_title":"Diversity-aware meta visual prompting","cited_arxiv_id":null,"evidence_quote":"Provides DAMVP and the HTA benchmark setup, the main comparison baseline and training protocol used by the paper."},{"cited_title":"Do we really need a large number of visual prompts? Neural Networks, 177: 0 106390, 2024","cited_arxiv_id":null,"evidence_quote":"Gives Theorem 4.2 on low-rank attention after prompting, used to extend the redundancy argument to prompted attention."},{"cited_title":"AutoVP: An Automated Visual Prompting Framework and Benchmark","cited_arxiv_id":null,"evidence_quote":"Defines AutoVP, another baseline that TCPA is added to and compared against."},{"cited_title":"Visual Fourier Prompt Tuning","cited_arxiv_id":"2411.01327","evidence_quote":"Defines VFPT, the strongest baseline on which TCPA is tested."}],"review_version":1}