{"id":"c83bb3be-f65e-4bc8-a4e0-e31a9196677d","arxiv_id":"2507.05668","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A CLIP adapter variant that varies per-token adaptation capacity by token importance improves new-class generalization by about one point over prior adapters.","lead":"This paper introduces Dynamic Rank Adaptation (DRA), a fine-tuning method for CLIP-style vision-language models that gives more important image and text tokens more adaptation capacity during training. The authors report small but consistent gains on unseen-class accuracy across 11 benchmarks, with code not yet released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dynamic rank mechanism is not actually present at inference and, as written, its sorting/masking path is non-differentiable, so the reported new-class gains may come from W_p/L1 regularization rather than token-importance grouping.","rationale":"The paper's value proposition is not merely the two-point HM gain over MMA but the explanation that adaptively preserving informative tokens improves novel-class generalization. For that explanation to be true, the importance scores in Eq. 10 must be meaningfully learned and the masking must be the operative component. The text does not specify any differentiable relaxation for the sort-and-mask path, and the inference equation removes the mechanism entirely. This is an internal inconsistency, not a disagreement with community consensus, so it is the right kind of concern. The empirical tables are internally consistent and the component ablations trend in the expected direction, which is credit to the authors, but Table 4 cannot distinguish 'importance-driven masking' from 'extra W_p + L1 + stochastic mask,' and no code or standard deviations are available. A random-grouping control would settle causation cleanly. This concern strengthens the conditions under which the paper should be accepted, but it does not justify rejection because the empirical claim may survive; the reader's CONDITIONAL verdict already captures that state, so I leave it unchanged.","tokens_in":19786,"tokens_out":6715,"duration_ms":88241,"concrete_test":"On the eleven base-to-new benchmarks, retrain DRA with W_t frozen at initialization and token groups assigned by a fixed per-sample random permutation instead of Eq. 10, holding group sizes, rank ratios, W_p, L1 loss, epochs, and hyperparameters identical. If the average New/HM remains close to 77.75/80.32 (within the reported seed variation), token-importance ranking is not causally load-bearing and the mechanism claim is unsupported; if performance drops substantially, the concern is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical claim in §4.2.1 (New 77.75, HM 80.32) is the central result, and the paper attributes it to 'dynamically allocating adaptation ranks based on token importance' (§3.2). Three load-bearing links in that causal chain are not established. First, Eq. 11 adds Gumbel noise to the scores and states this 'ensures the sorting process is differentiable,' but hard arg-sort followed by the binary mask M in Eq. 15 has no defined gradient with respect to W_t; without an explicitly stated straight-through or soft-sort estimator, the token-importance projection W_t cannot be trained. Second, the deployed model at inference is Eq. 18, which contains no sorting, masking, or importance grouping; it is a fixed-rank adapter with an extra linear layer W_p. The dynamic masking is therefore only a training-time auxiliary path, and Eq. 19 actively regularizes that path toward the unmasked inference path, so the final classifier never uses per-token ranks. Third, Table 4's ablations do not separate the contribution of the learned importance ranking from the contribution of the added W_p, the L1 loss, or stochastic masking as a generic regularizer. Consequently, the reported gains do not yet confirm the paper's mechanistic claim; they are equally consistent with the hypothesis that token-importance grouping is inert and the improvements come from the extra projection and regularization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynamic Rank Adaptation (DRA), an adapter-based method for few-shot fine-tuning of CLIP on base-to-new class generalization. DRA computes per-token importance scores in a low-rank space (Eq. 10), sorts tokens into K groups (Eq. 11), and assigns each group a different effective rank by masking channels in a group-dependent manner (Eqs. 14-16). A channel-variance criterion (Eq. 12) is used to retain the most informative channels, and an L1 loss (Eq. 19) aligns the training-time dynamic output with a fixed-rank inference-time path. The authors report average base/new/HM accuracies of 83.06/77.75/80.32 across 11 base-to-new datasets, claiming SOTA new-class and harmonic-mean performance, together with cross-dataset and domain-generalization results.","tokens_in":20096,"tokens_out":8339,"duration_ms":100321,"significance":"If the mechanistic claim were established, DRA would be a valuable contribution: token-conditional rank allocation is a plausible way to avoid overfitting to background or template tokens during CLIP adaptation, and the new-class evaluation is a genuine held-out test rather than a fit to the evaluation classes. The paper provides broad benchmark coverage (11 datasets for base-to-new, 10 for cross-dataset, 4 for domain generalization) and reports efficiency comparisons. However, the significance is currently limited because the central mechanism described in Sections 3.2.1-3.2.3 is not supported as written: the importance projections are not trainable under the stated equations, the dynamic path is absent at inference, and the ablations do not isolate the proposed components from the added linear layer and the regularization loss.","major_comments":[{"comment":"The sorting and masking operations are non-differentiable with respect to the 'learnable' importance projections W_t and W_c. Equation (11) applies arg sort to a_t + g, and Eq. (13) does the same for a_c + g; hard arg-sort has zero derivative almost everywhere, and the binary mask M in Eq. (15) has no gradient channel back to the scores. Adding Gumbel noise does not make the sorting process differentiable; a straight-through estimator or a soft-sort relaxation would be required but is never stated. Consequently, W_t and W_c cannot be trained by the given loss, and the reported token grouping is determined by the random initialization of W_t rather than by learned importance. The authors should either specify a concrete gradient estimator or compare against a variant with frozen/random W_t and W_c to demonstrate that learned grouping contributes to the results.","section":"§3.2.1 (Eqs. 11 and 13) and §3.2.3 (Eq. 15)"},{"comment":"The deployed model at inference is a fixed-rank adapter with an extra linear layer W_p; it contains no sorting, no group masks, and no importance grouping. The paper states that DRA is enabled only during training, and Eq. (19) explicitly penalizes the training-time dynamic output for deviating from this static inference output (with λ_T = λ_V = 1 in the experiments). Therefore, the headline gains in Table 1 cannot be attributed to dynamic rank allocation at inference time; they may instead come from the added projection W_p or from stochastic masking acting as a training regularizer. The authors should ablate W_p alone and also compare against a variant that uses static or random masks during training to separate these effects.","section":"§3.2.3 (Eq. 18) and §3.3 (Eq. 19)"},{"comment":"The component ablation does not isolate the proposed components. 'Baseline' is not defined in the experimental settings, and the DRA-T&V rows all include W_p as part of the DRA module. Removing CR and L1 leaves a variant that differs from Baseline by both token-group masking and the extra linear layer W_p, so the observed +0.33 HM over Baseline (and the full +0.91 HM) could be entirely due to W_p. The authors should ablate W_p alone and compare DRA against random token/channel masks to establish that the learned importance scores, rather than the extra capacity or the regularization, drive the reported improvement.","section":"§4.3.1, Table 4"},{"comment":"The hyperparameters K, the rank ratios e_i, the intermediate dimension r, and the starting layer h were chosen by maximizing harmonic mean on the same 11 datasets used for the headline comparison, and no separate validation split is reported. This selection-on-test-set procedure biases the reported averages in DRA's favor and makes the 0.95 New / 0.45 HM margin over MMA difficult to interpret. The authors should either fix hyperparameters a priori, use a validation split, or report a leave-one-dataset-out selection protocol.","section":"§4.1.2, §4.3.2-4.3.4, Table 5, Figure 3"}],"minor_comments":[{"comment":"The softmax is written over an N×N matrix and then averaged over the row index i, but the normalization dimension and the exact indexing should be stated explicitly; the current notation is ambiguous about whether Softmax is applied row-wise or column-wise.","section":"§3.2.1, Eq. (10)"},{"comment":"Equation (16) contains a typo: 'RestoringSort(Sort(X′,t)⊙M′),t)' has an extra comma and parenthesis; the argument order of RestoringSort should be clarified.","section":"§3.2.3, Eq. (16)"},{"comment":"Equation (18) uses X on both sides of the assignment, which is confusing because the left-hand side is the adapter output while the right-hand side is the input to the adapter; the output should be renamed (e.g., X_out).","section":"§3.2.3, Eq. (18)"},{"comment":"The row labels 'DRA-T&V - -', 'DRA-T&V ✓ -', and 'DRA-T&V - ✓' are not self-explanatory; the table should explicitly define which components each checkmark denotes.","section":"§4.3.1, Table 4"},{"comment":"The histogram axes and the color scale for the token importance and rank allocation are not labeled, and the cropping of the <SOS> score is mentioned but not shown, which makes the visualization difficult to interpret.","section":"§4.4, Figure 4"},{"comment":"The text claims that DRA improves over SOTA methods on new classes and harmonic mean, but the average base-class accuracy (83.06) is below LFA (83.62), MetaPrompt (83.38), and TCP (84.13); the conclusion should qualify that the improvement is specifically on new-class and harmonic-mean metrics, not on base-class accuracy.","section":"§4.2.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are promising and the new-class evaluation is genuinely held out, but the mechanism as described is not supported by the equations: the sorting/masking path is non-differentiable for W_t and W_c, the deployed inference model lacks the dynamic grouping, and the ablations do not separate the proposed components from W_p and the L1 regularization. These are load-bearing issues for the paper's central claim, but they are fixable in principle by adding a proper gradient estimator, reframing the method as a training-time masking regularizer, and providing the missing ablations and validation protocol. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DRA is a serious adapter variant: it assigns per-token rank budgets by grouping tokens according to an attention-based importance score, keeps the most responsive channels for lower-ranked tokens, and adds an L1 regularizer. The evaluation is broad — 11 datasets, base-to-new, cross-dataset, domain generalization, plus ablations and efficiency numbers — and the headline result is consistent: +0.45 HM over the next-best adapter (MMA), with wins on 7/11 new-class splits and 8/11 harmonic means. The method is also cheap, converging in 5 epochs.\n\nThat said, the paper's central mechanistic claim is not supported by the equations. First, Eq. 11 and Eq. 13 use hard arg-sort with Gumbel noise and claim differentiability, but that only works with an explicit straight-through or soft-sort estimator, which is never stated. As written, W_t and W_c get no gradient through the sorting/masking path, so the \"learnable\" importance grouping is not actually learned. Second, at inference the model reduces to Eq. 18 — a fixed-rank adapter with an extra linear layer W_p. The dynamic masking is training-only, and Eq. 19 actively regularizes the training output toward this unmasked inference path. The deployed model never uses per-token ranks. That is not automatically fatal, but it reframes the contribution as a regularizer rather than dynamic rank adaptation. Third, the ablations in Table 4 do not isolate the importance grouping from the extra W_p and the L1 loss; the improvement from baseline to DRA-T&V (79.74 vs 79.41) already includes W_p, so the marginal value of the learned token ranking is untested.\n\nThere are also standard-but-important issues: no error bars (only \"averaged over three seeds\"), no code, and hyperparameters (K, rank ratios, r, h) selected by optimizing the same benchmarks that produce the headline numbers. None of these, individually, sink the paper; they are all addressable.\n\nWho is this for? Researchers working on parameter-efficient VLM adaptation, especially those interested in why adapter fine-tuning hurts new-class generalization. The empirical result is positive and the mechanism is novel enough that a serious referee should see it. My recommendation: send it to peer review, but with a clear request for major revision — fix or rename the mechanism, add proper gradient handling or correct the description, report variance, and run ablations that separate the importance grouping from the extra capacity and regularization.","headline":"A plausible adapter variant with a genuinely new training-time mechanism, but the dynamic-rank story does not survive contact with the inference-time equations, and the sorting is non-differentiable as written.","tokens_in":20648,"tokens_out":3444,"would_cite":false,"duration_ms":40421,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dynamic Rank Adaptation claims that fine-tuning CLIP-like vision-language models with adapter capacity allocated by token importance improves accuracy on unseen new classes while keeping base-class accuracy nearly intact.","keywords":["Vision-language models","CLIP fine-tuning","Adapters","Dynamic rank allocation","Token importance","Base-to-new generalization","Few-shot learning","Prompt learning"],"falsifier":"Measure new-class accuracy after replacing the learned token importance scores with random scores drawn from the same Gumbel distribution while keeping the identical rank budget; if random grouping matches DRA's new-class gain over the fixed-rank baseline, the importance mechanism is not the source of the improvement.","tokens_in":19523,"feed_emoji":"🎯","tokens_out":5003,"duration_ms":53829,"temperature":0.7,"pith_summary":"The paper tries to solve a known failure: when CLIP is fine-tuned on base classes, prompt and adapter methods often overfit and lose accuracy on unseen classes. It proposes an adapter variant, Dynamic Rank Adaptation (DRA), that gives more adaptation rank to tokens judged important and compresses unimportant ones. On 11 datasets with 16 shots per class, DRA reports average base accuracy 83.06, new-class accuracy 77.75, and harmonic mean 80.32, beating the prior adapter MMA by 0.95 on new classes and 0.45 on harmonic mean. The paper argues this shows allocating capacity by token importance preserves general representations.","feed_headline":"Rank-by-importance adapter lifts unseen-class accuracy 0.95","feed_subtitle":"Dynamic Rank Adaptation beats prior adapter methods on new classes across 11 benchmarks while keeping base accuracy.","key_machinery":"The load-bearing object is the token-level rank mask $\\mathbf{M}'$. Token importance scores $a_t$ are computed as averaged self-attention over the low-rank adapter projection; tokens are sorted and split into $K=4$ groups with rank ratios $[1.0, 0.8, 0.6, 0.4]$. Each group keeps the top $\\lfloor e_i \\times r \\rfloor$ channels according to a channel response score $a_c$, the per-channel variance of the projected low-rank features. The mask is applied after sorting tokens and then restored, so each sample receives its own rank allocation while the final projection path remains a standard adapter at inference.","core_discovery":"The central claim is that the fixed-rank bottleneck shared by all tokens limits new-class generalization, and replacing it with adaptive per-token rank allocation removes that limitation. DRA computes token importance scores from a low-rank self-attention map, sorts tokens into four groups, and assigns rank ratios [1.0, 0.8, 0.6, 0.4] so important tokens keep full capacity while unimportant tokens are compressed. A channel response mechanism keeps the most active channels of each sample when compressing, and an L1 loss pulls training output toward inference output to stabilize training. The paper reports that DRA outperforms prior methods on new-class accuracy in 7/11 datasets, on harmonic mean in 8/11 datasets, and also tops cross-dataset and domain generalization averages.","pith_inferences":["If the importance scores are genuinely semantic, the same dynamic-rank principle should transfer beyond classification: any transformer adapter could allocate capacity by token importance for detection, segmentation, or retrieval tasks that the paper does not test.","The paper's own ablations leave open the possibility that part of the gain is a regularization effect of masking; comparing DRA against random token masks with the same rank budget would separate those explanations.","The token importance scores could be reused as free saliency maps; evaluating them against foreground annotations would both test the core mechanism and produce a side product.","Because DRA converges in five epochs, the same capacity-allocation idea may reduce fine-tuning compute for larger vision-language models, though the paper only reports results on CLIP ViT-B/16."],"forward_implications":["On base-to-new generalization over 11 datasets, DRA improves average new-class accuracy to 77.75 and harmonic mean to 80.32, with base accuracy 83.06 nearly unchanged from the MMA baseline.","DRA improves new-class accuracy on 7/11 datasets and harmonic mean on 8/11 datasets compared with the best prior method.","Channel response and L1 regularization each add measurable gains: removing either lowers the harmonic mean relative to the full DRA configuration.","Cross-dataset average accuracy (66.71) and domain generalization average accuracy (60.64) are the highest reported among the compared methods.","At inference, dynamic allocation is unwound: features pass through the same linear projections as a normal adapter, so DRA adds no inference-time token selection step."],"supporting_citations":[{"why":"Supplies the frozen CLIP backbone and the zero-shot baseline that all fine-tuning methods build on.","marker":"[28]"},{"why":"The main adapter-based baseline to beat; its average base-new performance and design motivate DRA's improvements.","marker":"[35]"},{"why":"CoOp is the prompt-learning baseline that defines the base-new generalization benchmark and shows the overfitting problem.","marker":"[48]"},{"why":"CoCoOp provides the conditional prompt baseline and a key comparison point for new-class accuracy.","marker":"[47]"},{"why":"MaPLe is the multimodal prompt baseline whose base-new results DRA directly compares against.","marker":"[18]"},{"why":"LoRA supplies the low-rank adaptation rationale and the re-parameterization idea that adapters in this paper build on.","marker":"[13]"},{"why":"DyT is the prior dynamic-tuning work that skips unimportant tokens and is discussed as a contrast to DRA's rank-preserving masking.","marker":"[44]"},{"why":"AdaptFormer provides the standard adapter structure in transformer layers that DRA modifies.","marker":"[3]"},{"why":"KgCoOp is a prompt-tuning baseline that regularizes text features toward handcrafted prompts and is used for comparison.","marker":"[36]"}],"fun_headline_variants":["Dynamic rank adaptation sharpens new-class VLM accuracy","Importance-aware ranks preserve VLM general knowledge","Token-importance ranks lift unseen-class performance","Adaptive per-token ranks beat fixed adapters on new classes","VLM fine-tuning with dynamic ranks keeps old, learns new"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the self-attention scores computed inside the adapter rank tokens by true semantic importance, so the most informative tokens are the ones kept at full rank; the paper shows only a qualitative visualization, not a quantitative check.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic rank adaptation sharpens new-class VLM accuracy","Importance-aware ranks preserve VLM general knowledge","Token-importance ranks lift unseen-class performance","Adaptive per-token ranks beat fixed adapters on new classes","VLM fine-tuning with dynamic ranks keeps old, learns new"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1363,"prompt_tokens":958,"completion_tokens":405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":574,"tokens_out":405,"duration_ms":4797,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:20:53.853454+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure new-class accuracy after replacing the learned token importance scores with random scores drawn from the same Gumbel distribution while keeping the identical rank budget; if random grouping matches DRA's new-class gain over the fixed-rank baseline, the importance mechanism is not the source of the improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP backbone and the zero-shot baseline that all fine-tuning methods build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The main adapter-based baseline to beat; its average base-new performance and design motivate DRA's improvements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CoOp is the prompt-learning baseline that defines the base-new generalization benchmark and shows the overfitting problem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MaPLe is the multimodal prompt baseline whose base-new results DRA directly compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DyT is the prior dynamic-tuning work that skips unimportant tokens and is discussed as a contrast to DRA's rank-preserving masking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AdaptFormer provides the standard adapter structure in transformer layers that DRA modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KgCoOp is a prompt-tuning baseline that regularizes text features toward handcrafted prompts and is used for comparison."}],"review_version":1}