{"id":"fc0f238b-d728-4c48-9383-a81c5c108b47","arxiv_id":"2505.06002","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Task-Adapter++ adapts frozen CLIP encoders with task-specific visual adapters and order-aware semantic adapters, and reports state-of-the-art results on five few-shot action recognition benchmarks.","lead":"The paper proposes Task-Adapter++, a parameter-efficient method that adapts frozen CLIP image and text encoders for few-shot action recognition using task-specific adapters and LLM-generated sub-action descriptions. It reports state-of-the-art accuracy on five action recognition benchmarks while tuning only a small fraction of the model parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The semantic branch's contribution hinges on an unverified assumption that GPT-4's 'beginning/process/end' sub-action descriptions line up with the three fixed temporal segments in Eq. 16; absent evidence for that correspondence, the cross-modal term in Eq.","rationale":"The reader's weakest-assumption identification is correct and is the most load-bearing point because the paper's own ablation isolates it: the alignment module without sub-action descriptions reduces accuracy (Table 4 row 3), so the claimed improvement from the semantic branch is exactly the stage-wise matching between LLM text and fixed video segments. This is not a disagreement with prior work; it is an internal coherence check. The paper otherwise has strong empirical support: open-sourced code, consistent ablations, and gains across five benchmarks, so I do not want to overstate the concern. The issue is addressable by the proposed diagnostic; if the correspondence is confirmed, the central claim stands. I keep the reader's CONDITIONAL verdict because the concern is about a load-bearing mechanism rather than a demonstrated error.","tokens_in":20759,"tokens_out":9375,"duration_ms":92621,"concrete_test":"Run a stage-correspondence diagnostic on SSv2-Small and HMDB51: for sampled query videos, compute the cosine similarity between each GPT-4 sub-action description feature and the visual features from each of the three Eq. 16 segments, and check whether the intended 'beginning/process/end' description is most similar to segment 1/2/3 significantly above chance. Additionally, compare the current overlapping split against non-overlapping thirds and against a shuffled stage assignment (same training, only the matching in Eq. 17 permuted). If the intended correspondence is not reliably best, or if shuffled assignment gives comparable accuracy, the fine-grained stage-wise alignment is not the mechanism driving the reported gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Task-Adapter++'s SOTA claim rests on the product score in Eq. 18, and Table 4 attributes the gain from 60.2 to 63.6 (SSv2-Small 5-way 1-shot) to the semantic branch. Row 3 of Table 4 is the key stress point: adding fine-grained cross-modal alignment without the three sub-action descriptions drops accuracy from 60.2 to 58.5. So the entire benefit is contingent on the three GPT-4-generated 'beginning, process, end' descriptions corresponding, in order, to the three video segments in Eq. 16. That correspondence is assumed but never established. Eq. 15 produces aligned features of length T-1 via cross-attention between adjacent frames, and Eq. 16 then averages indices 1-3, 3-5, and 5-7 as the three stages. There is no argument that these intervals match the temporal stages in the videos, the overlap at indices 3 and 5 is unexplained, and the indexing of aligned features versus original frames is ambiguous. If the LLM descriptions do not match the actual stage intervals (e.g., action phases have variable duration), the cosine matching in Eq. 17 is misaligned, so the cross-modal factor in Eq. 18 can degrade rather than improve classification. The paper reports no diagnostic tying description content to frames.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Task-Adapter++, a parameter-efficient framework for few-shot action recognition built on frozen CLIP. It inserts task adapters with a reused task-specific MSA into the last L layers of the visual encoder, uses GPT-4 to generate three ordered sub-action descriptions per class and models them with order adapters in the text encoder, and computes a fine-grained cross-modal alignment score by splitting aligned query features into three temporal segments and matching them to the three semantic features. The final classification score is the product of the video-prototype matching score and the cross-modal matching score. Experiments on SSv2-Small, SSv2-Full, HMDB51, UCF101, and Kinetics report state-of-the-art results with 3.5M or 7.5M trainable parameters.","tokens_in":21083,"tokens_out":8904,"duration_ms":87669,"significance":"Assuming the semantic-stage alignment is validated, the paper makes a solid contribution: it demonstrates that a frozen CLIP backbone with lightweight task adapters and LLM-generated sub-action descriptions can outperform full fine-tuning baselines on five few-shot action recognition benchmarks using only 3.5M or 7.5M trainable parameters. The component-wise ablations in Tables 3 and 4 are informative, the comparison across metric modules in Fig. 10 is useful, and the open-sourced code supports reproducibility. The main risk is that the cross-modal gain rests on the unverified correspondence between LLM descriptions and fixed temporal segments, and the current text does not fully explain or validate that correspondence.","major_comments":[{"comment":"The stage-wise cross-modal score rests on an unverified correspondence between the three GPT-4 sub-action descriptions and the three temporal segments defined in Eq. (16). First, the indexing of the aligned features is ambiguous: Eq. (15) defines aligned features from frames i-1 and i, so with T=8 the aligned features are naturally indexed as i=2,...,8, whereas Eq. (16) sums i=1..3, 3..5, and 5..7; if instead the aligned feature index is meant to denote the pair (i,i+1), the segments overlap at pairs 3 and 5, so the three segments cover frames 1-4, 3-6, and 5-8. Neither the overlap nor the boundary choice is justified. More importantly, the paper provides no evidence that the LLM-generated 'beginning, process, end' descriptions correspond to these fixed intervals across classes with highly variable action durations; Table 4 shows that adding cross-modal alignment without the sub-action descriptions decreases accuracy from 60.2 to 58.5, so the gain of the full model is contingent on the semantic-stage correspondence being correct. Please provide a corrected and unambiguous definition of the segments, an ablation with non-overlapping or learned/adaptive segmentation, and a diagnostic or control such as shuffling the three sub-action descriptions or annotating temporal boundaries for a sample of classes.","section":"3.6, Eqs. (15)-(17), Table 4"},{"comment":"The third row of Table 4 is difficult to interpret because the text does not specify what the text branch contains when the 'Three Sub-actions' component is off. Equation (17) requires three semantic features to match the three visual segments, but Section 3.5 only defines those features from the three sub-action prompts; if cross-modal alignment is applied without sub-action descriptions, it is unclear whether the static label is duplicated three times, whether the alignment module is trained with a different loss, or whether the semantic branch is omitted entirely. The accompanying sentence that 'the original Task-Adapter already accomplishes semantic alignment during feature extraction' does not resolve this, since Task-Adapter is a visual-only module. Please specify the exact configuration of row 3 and, if a static-text duplicate is used, justify it; as written, this row cannot support the claim that fine-grained alignment is redundant without enriched semantics.","section":"Table 4"},{"comment":"All results are reported as averages over 10,000 episodes without variance or confidence intervals. Several improvements over the closest baselines are small: in Table 2, Task-Adapter++ ties TSAM at 96.2 on Kinetics 1-shot, and several UCF101 margins are around 0.2-0.5%. Without error bars or repeated-run statistics, the headline claim of consistent state-of-the-art performance is not fully supported. Please report mean and standard deviation over at least three runs, or confidence intervals for the 10,000-task estimates, and discuss significance for the small-margin cases.","section":"4.1, Tables 1-2"},{"comment":"Equation (18) multiplies the video matching 'probability' P from Eq. (11) by the video-text matching 'probability' from Eq. (17). Since both quantities are described as probabilities or normalized cosine scores, their product is not a logit; if the product is used directly as the cross-entropy logit, the model is effectively optimizing a different objective than the one described. Please clarify whether P and the video-text score are raw similarity scores, whether the product is re-normalized before loss computation, or whether the intended combination is log P plus the log of the video-text score.","section":"3.7, Eq. (18)"}],"minor_comments":[{"comment":"Equation (5) has malformed parentheses; it should be written as z_l = Adapter(LN(z_s_l)) + MLP(LN(z_s_l)) + z_s_l if the adapter is parallel to the MLP as stated in the text.","section":"3.3, Eq. (5)"},{"comment":"The sentence 'we first map all the videos ... into their patch embeddings by Eq. (1) and Eq. (2)' is imprecise, since Eqs. (1)-(2) describe ViT block updates rather than patch embedding construction.","section":"3.4"},{"comment":"Please specify the exact output shape of the semantic features used in Eq. (17): if the end-of-text tokens of the three sub-action prompts are concatenated, the result should be stated as a C x 3 x D tensor so that the indexing F_T^c[s] is unambiguous.","section":"3.5"},{"comment":"The statement that 'the first three datasets focus on scene understanding' is inaccurate for Kinetics, which is primarily an action dataset; consider saying 'scene-related datasets' or 'datasets with less stringent temporal requirements.'","section":"4.1"},{"comment":"Figure 8 has no axis labels or legend; as printed, the reader cannot tell which curve corresponds to which dataset or what the x-axis represents. Please add these details.","section":"Figure 8"},{"comment":"The term 'Partial Adapting' is used before it is defined; please define it in the text, for example as inserting adapters only into the last L layers, before the table is discussed.","section":"Table 3"},{"comment":"On Kinetics 1-shot, Task-Adapter++ ties TSAM at 96.2; the claim of consistent state-of-the-art performance should be qualified for tied or statistically indistinguishable cases.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this paper is an extended version of the authors' own MM'24 Task-Adapter paper. The contribution bullet claiming to be the first to perform task-specific adaptation during spatio-temporal feature extraction should be checked against that prior work, since the visual Task-Adapter already appeared there. The main technical risk is the semantic-stage alignment assumption described in major comment 1; if the authors can supply the requested control experiments and clarify the ablation configuration, the paper would be acceptable. The manuscript's style, including highlights and a graphical abstract, suggests a journal such as Pattern Recognition, and the experimental scope is appropriate for that venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical extension of the authors' MM'24 Task-Adapter. The genuinely new piece is putting task-specific adapters inside the last L layers of a frozen CLIP ViT so that feature extraction itself reacts to the support set in each episode, and pairing that with a semantic branch that uses LLM-generated three-stage sub-action descriptions and order adapters. As an engineering contribution for few-shot action recognition it is above the current bar, and the ablations are unusually thorough: they test each component, insertion depths, parameter counts, alignment module choices, and several metric heads.\n\nWhat it does well: the gains on SSv2 are real and sizeable (63.6 vs 60.2 for their own Task-Adapter, vs 60.5 for TSAM on 5-way 1-shot). The visual task-adaptation story is convincing: the Task-MSA provides cross-video attention in the frozen blocks, and Table 3 shows partial adaptation plus task-specific attention is what makes the difference. The claim to be first to do task-specific adaptation during spatio-temporal feature extraction is plausible; FEAT and Ta-Adapter operate on extracted features, not within the encoder.\n\nSoft spots: the semantic stage alignment is the least supported part. Eq. 16 splits the seven aligned features into three overlapping segments, 1-3, 3-5, 5-7, and matches them to GPT-4's beginning/process/end descriptions. There is no explanation for the overlap or any evidence that GPT-4's stages correspond to those frame intervals. Table 4 makes this load-bearing: cross-modal alignment alone lowers accuracy (58.5 vs 60.2), so the entire semantic gain rides on the correspondence being at least approximately right. That concern does not sink the paper, but it needs a diagnostic, such as permuting the sub-action order or visualizing which frames each description attends to. Also, all reported numbers are means over 10,000 tasks with no variance; some HMDB/Kinetics gaps are under a point, so I would want error bars before trusting the SOTA claim on those datasets.\n\nOverall, this is a paper for the FSAR audience, and a referee will get value from it. I would send it to review, asking for the stage-alignment diagnostic and confidence intervals. With those, it is a clear accept; without, the semantic contribution stays somewhat under-evidenced.","headline":"Solid FSAR paper where the visual task-adaptation idea is the real contribution; the semantic stage-alignment story needs a diagnostic before the SOTA claim fully lands.","tokens_in":21593,"tokens_out":4439,"would_cite":true,"duration_ms":40259,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Task-Adapter++ shows few-shot action recognition improves by freezing CLIP and tuning tiny order-aware adapters.","keywords":["few-shot action recognition","parameter-efficient fine-tuning","CLIP adaptation","task-specific adaptation","semantic order adaptation","cross-modal alignment","LLM-generated descriptions"],"falsifier":"Take a few-shot episode and permute the order of the three LLM sub-action descriptions--for example, swap the end description with the beginning description--before computing the cross-modal term in Eq. (18); if accuracy stays nearly the same, the order-aware stage correspondence is not doing the claimed work, while a large drop would confirm the alignment matters. A second check is to replace the fixed equal thirds of Eq. (16) with manually labeled temporal phase boundaries and see whether accuracy rises.","tokens_in":1732,"feed_emoji":"🎬","tokens_out":1730,"duration_ms":55172,"temperature":0.7,"pith_summary":"The paper claims that few-shot action recognition can be improved by keeping a pre-trained CLIP model frozen and inserting small trainable adapters into both its visual and text branches, instead of full fine-tuning. It introduces a Task-Adapter inside the visual encoder that lets support and query videos attend to each other at shared spatio-temporal locations, highlighting the most task-discriminative cues. On the text side, it uses LLM-generated ordered sub-action descriptions (beginning, process, end) and a semantic order adapter to model the sequence. A fine-grained cross-modal alignment then segments aligned query features into temporal stages and matches them stage-wise with the sub-action semantics. The combination is reported to reach new state-of-the-art results on five benchmarks while training only 3.5M or 7.5M parameters.","feed_headline":"Tiny adapters set new few-shot action recognition records","feed_subtitle":"A frozen CLIP with just 3.5-7.5M trained parameters outperforms full fine-tuning on five benchmarks.","key_machinery":"The central objects are the task adapter and the semantic order adapter. The task adapter reuses the frozen MSA layer as a Task-MSA that performs self-attention across the same spatio-temporal positions across all videos in an episode, so the most discriminative task-specific information is emphasized during feature extraction. The semantic order adapter reuses the frozen MSA as an O-MSA to model the sequential relationships among the three ordered sub-action description prompts, injecting order information into the text features. The fine-grained cross-modal alignment applies cross-attention between adjacent frames, producing aligned features of length $T-1$, then splits them into three overlapping segments--frames 1-3, 3-5, and 5-7--and matches each segment by cosine similarity with the corresponding LLM-generated sub-action feature, combining the two matching scores by multiplication in Eq. (18).","core_discovery":"On its own terms, the paper establishes that parameter-efficient dual adaptation--task-specific adapters in the visual branch and order-aware adapters in the text branch of a frozen CLIP--consistently outperforms both full fine-tuning and prior PEFT methods on SSv2-Small, SSv2-Full, HMDB51, UCF101, and Kinetics. The principal novelty claimed is performing task-specific adaptation during spatio-temporal feature extraction: a reused frozen MSA layer acts as a Task-MSA that runs cross-video attention over tokens at the same spatio-temporal location, with learnable adapters around it. The semantic branch uses a semantic order adapter with a reused O-MSA to model the relationships among three LLM-generated sub-action descriptions. Finally, a fine-grained cross-modal alignment module applies one cross-attention step between adjacent frames, divides the aligned features into three temporal segments, and matches each segment to the corresponding sub-action semantic feature; the final classification score is the element-wise product of the video matching score and the video-text matching score.","pith_inferences":["The fixed equal-thirds segmentation of the 8-frame video is a crude stand-in for actual action phases; if the LLM's beginning/process/end descriptions were aligned via learned temporal boundaries or annotated phase labels, the cross-modal term might become more reliable and accurate.","The dual-adapter recipe likely transfers to other video-language problems such as temporal action localization, video captioning, or step-level instruction understanding, wherever ordered text descriptions exist alongside video.","Because only adapters are trained, the approach could be paired with other frozen backbones or extended toward zero-shot regimes, though the paper does not test those settings.","The product rule for combining video and video-text scores may suppress correct predictions when one branch is noisy; a learned fusion or sum rule could be more robust, and the paper does not compare these alternatives."],"forward_implications":["If the central claim holds, frozen-CLIP few-shot action recognition can reach higher accuracy than full fine-tuning while reducing trainable parameters from about 149.6M to 3.5M or 7.5M.","The method is metric-agnostic: it is reported to improve performance when combined with TRX, OTAM, Bi-MHM, or simple prototype averaging, meaning the visual adapter carries most of the gain.","Decomposing an action label into ordered sub-action descriptions and matching stage-wise outperforms averaging the descriptions or concatenating them into one prompt, supporting the paper's claim that semantic order matters.","On temporal-heavy datasets like SSv2, inserting the task adapter into the top six layers works best; on scene-heavy datasets like Kinetics, UCF101, and HMDB51, the top two layers suffice, yielding different parameter counts.","The video-text matching branch alone improves from 50.4% to 54.3% after adding order adaptation and fine-grained alignment, indicating the semantic branch becomes informative rather than redundant."],"supporting_citations":[{"why":"Supplies the frozen CLIP visual and text encoders that the two adaptation branches build on.","marker":"[1]"},{"why":"Provides the frozen T-MSA and S-MSA adapter recipe that the task-specific visual adaptation extends.","marker":"[5]"},{"why":"TRX is one of the interchangeable video metric modules used to compute the visual matching score.","marker":"[15]"},{"why":"Bi-MHM is another interchangeable metric module demonstrating that the adapter works across matching methods.","marker":"[16]"},{"why":"MA-FSAR is the PEFT-based CLIP few-shot action recognition baseline that the method is compared against and outperforms.","marker":"[19]"},{"why":"TSAM is the recent sequence-aware PEFT baseline used as a state-of-the-art comparison.","marker":"[21]"},{"why":"The conference version Task-Adapter supplies the task-specific visual adaptation foundation that this paper extends with semantic order adaptation.","marker":"[28]"},{"why":"CLIP-FSAR is the full fine-tuning baseline, establishing the performance gap that parameter-efficient adaptation closes.","marker":"[60]"},{"why":"GPT-4 generates the three ordered sub-action descriptions per action class that feed the semantic order adapter and cross-modal alignment.","marker":"[65]"}],"fun_headline_variants":["Order-aware adapters beat full fine-tuning on 5 benchmarks","Few-shot action recognition: Task-Adapter++ sets new SOTA","Task-specific adapters align temporal semantics for SOTA few-shot action","Dual adapters on frozen CLIP outperform full fine-tuning in FSAR","Task-Adapter++: order-aware alignment boosts few-shot action recognition"],"cache_read_input_tokens":23680,"weakest_assumption_plain":"Cross-modal matching assumes that the LLM's three ordered sub-action descriptions correspond, in order, to thirds of the query video, yet the video is divided into three equal overlapping segments without evidence that the LLM's beginning, process, and end stages line up with those frame intervals.","fun_headline_variants_meta":{"raw":{"variants":["Order-aware adapters beat full fine-tuning on 5 benchmarks","Few-shot action recognition: Task-Adapter++ sets new SOTA","Task-specific adapters align temporal semantics for SOTA few-shot action","Dual adapters on frozen CLIP outperform full fine-tuning in FSAR","Task-Adapter++: order-aware alignment boosts few-shot action recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2861,"prompt_tokens":1043,"completion_tokens":1818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":1726}},"tokens_in":659,"tokens_out":1818,"duration_ms":13021,"temperature":1.0,"reasoning_tokens":1726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:50:17.894889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a few-shot episode and permute the order of the three LLM sub-action descriptions--for example, swap the end description with the beginning description--before computing the cross-modal term in Eq. (18); if accuracy stays nearly the same, the order-aware stage correspondence is not doing the claimed work, while a large drop would confirm the alignment matters. A second check is to replace the fixed equal thirds of Eq. (16) with manually labeled temporal phase boundaries and see whether accuracy rises.","supporting_citations":[{"cited_title":"Perrett, A","cited_arxiv_id":null,"evidence_quote":"TRX is one of the interchangeable video metric modules used to compute the visual matching score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Bi-MHM is another interchangeable metric module demonstrating that the adapter works across matching methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TSAM is the recent sequence-aware PEFT baseline used as a state-of-the-art comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The conference version Task-Adapter supplies the task-specific visual adaptation foundation that this paper extends with semantic order adaptation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLIP-FSAR is the full fine-tuning baseline, establishing the performance gap that parameter-efficient adaptation closes."}],"review_version":1}