{"id":"b38b6063-3ffe-4c4d-8926-029f3b077952","arxiv_id":"2501.04293","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A prompt-and-dynamic-filter PEFT design for multi-task dense prediction beats MTLoRA on PASCAL-Context with fewer trainable parameters.","lead":"TADFormer is a parameter-efficient fine-tuning method for multi-task dense prediction that adds task prompts and input-dependent dynamic filters to a frozen vision transformer. On PASCAL-Context it reports higher average accuracy than MTLoRA and full fine-tuning while training roughly 3.6 to 7.4 million parameters instead of 30.06 million.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TPC-operator assumes a dense T×N task attention map, but Swin's windowed MHSA is local; the paper never specifies how prompts enter window attention, so the core mechanism is underspecified.","rationale":"Read in good faith. The strongest claim is clear, Table 1 is internally consistent, and the ablation study in Table 2 supports the qualitative contribution of DTF and TPC. I did not find an arithmetic or parameter-count contradiction. The weakest point is the construction of the task attention map: the formulas in Sec. 3.3 assume a global prompt-to-patch attention matrix, while the chosen Swin backbone computes local window attention. This is more fundamental than the reader's concern about prompt-information dilution across depth, though it is the same component. It matters because the TPC operator is not a peripheral trick; the ablation credits it with most of the final gain, and the paper cannot be reproduced from the text without knowing how prompts interact with windows. The missing DTF kernel size, absent error bars, and odd supplementary adapter parallel-configuration numbers are real but secondary; the central LoRA-based claim is not affected by the adapter table. Since no code or checkpoint is provided for the main experiments, the correct gate is to require either the implementation or a precise description of the prompt-window handling. This leaves the reader's CONDITIONAL verdict unchanged.","tokens_in":18183,"tokens_out":13406,"duration_ms":142942,"concrete_test":"Request the main training code (the supplementary mentions code only for adapter experiments) and trace the construction of ATAM in the last Swin block of a stage: print the attention-weight mask for one prompt row and compute the fraction of the N patch tokens with nonzero attention, and inspect whether prompts are duplicated into every window before MHSA. If the nonzero support is confined to a single 7×7 window (or if a dense H×T×N map cannot be produced without a custom attention path), Eq. 2 is not implementable on the stated backbone and the +2.33 Δm attributed to TPC+DTF in Table 2 needs to be re-derived. If the code duplicates prompts per window or uses a custom attention block, state that modification explicitly in the method section.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The TPC operator is the paper's central mechanism for task-adaptive PEFT, and the ablations credit it with +2.33 Δm (Baseline +1.30 vs full TADFormer +3.63, Table 2). Its construction requires a task attention map ATAM ∈ R^{H×T×N} between each prompt and every image patch token (Sec. 3.3.1, Eq. 2). However, the encoder is Swin-T (Sec. 4.1), whose MHSA runs inside shifted local windows rather than over the full sequence. The paper never specifies how the T prompts are inserted into this windowed attention: prepended to the whole sequence (they would fall inside one window), duplicated per window, or handled by a modified global-attention variant. Each choice changes what the task attention map contains and whether Eq. 2 is even evaluable. Under the literal 'prepended' reading, most patch tokens have zero or negligible prompt attention, so the TPC enhances at most a local window, contradicting the paper's depiction of task-adapted features for all N tokens. If instead prompts were duplicated per window or a custom attention variant was used, that is a significant unstated architectural modification absent from Figure 4 and the equations. This is not a parameter-count or noise issue: it is an underspecification of the module whose ablation is the main evidence for the method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes TADFormer, a parameter-efficient fine-tuning method for multi-task dense prediction. It prepends task prompts to image patch tokens, uses LoRA-style task-shared modules in most Swin Transformer blocks, and introduces a Task-Prompt Conditional (TPC) operator plus a Dynamic Task Filter (DTF) in the last block of each stage. The TPC operator uses attention between task prompts and patch tokens to produce task-adapted features, while the DTF generates channel-wise convolutional filters conditioned on input features. Experiments on PASCAL-Context with four dense prediction tasks report higher Δm than MTLoRA and full fine-tuning with fewer trainable parameters, and the paper also provides ablations, decoder/backbone variants, and an adapter-based extension.","tokens_in":18486,"tokens_out":10697,"duration_ms":102182,"significance":"If the reported results are reproducible, TADFormer is a useful contribution: it combines prompt-based task adaptation with input-conditioned filters in a parameter-efficient multi-task framework, and the ablation study gives per-component evidence for the TPC operator and DTF. The paper provides full per-task tables, multiple rank settings, decoder/backbone variants, computational cost analysis, and attention-map visualizations. However, the central mechanism is underspecified for the Swin windowed-attention backbone, the main code and seed-level statistics are not provided, and the adapter extension contains severely degraded parallel configurations that are presented too positively. These issues currently prevent verification of the headline claim that TADFormer consistently outperforms existing PEFT methods.","major_comments":[{"comment":"The TPC operator is underspecified for the Swin-T backbone. The text defines a global attention map A in R^{H x (T+N) x (T+N)} and extracts ATAM in R^{H x T x N} between task prompts and every patch token, but Swin's windowed MHSA only produces per-window attention matrices. If task prompts are simply prepended to the patch-token sequence, the prompts lie in one window and most patch tokens have no direct prompt attention, making Eq. (2) not evaluable as written. If a modified global attention or per-window prompt insertion is used, this is a significant architectural change that must be stated and reflected in Figure 4 and the equations. Relative position bias for prompt tokens also needs to be defined. This issue is load-bearing because Table 2 credits the task attention map with +0.39 Δm and the full TPC+DTF combination with +2.33 Δm over the baseline.","section":"3.3.1 / Eq. (2) / 4.1"},{"comment":"The adapter extension is presented as evidence of generality, but the parallel configurations show a severe performance collapse. AdaptFormer parallel rows A2 and A4 have semantic segmentation mIoU 55.28 and 55.18 with Δm -10.54 and -11.06; TADFormer versions AO2 and AO4 improve to 61.41 and 60.37 with Δm -5.02 and -6.25, still far below the MTL full-fine-tuning baseline (+2.23, semseg 67.56). The text's conclusion that TADFormer yields an overall enhancement in parallel configurations is true only in the narrow sense of making a badly degraded configuration less degraded. Please diagnose this failure or remove the parallel-configuration claim; as it stands, the claim that TADFormer is compatible with adapter-based frameworks is not supported.","section":"Supplementary Section 8, Table 4"},{"comment":"All main results are single-run numbers without error bars, seed counts, or released main-experiment code. The headline differences (e.g., TADFormer r=16 Δm +2.44 vs MTLoRA r=16 +1.35, and the reported 1.2-1.7 Δm gains at r=32/64) are small enough that seed variance in multi-task dense prediction could affect the ranking. Please report mean and standard deviation over at least three seeds for the main comparisons and make the main training/evaluation code available. Without this, the central claim of consistent superiority over MTLoRA cannot be verified.","section":"Table 1 and Section 4.1"},{"comment":"The DTF is motivated by the need to capture input context, but its parameter generation network uses global average pooling of the down-projected feature, which discards spatial structure before generating the channel-wise filters. The paper does not explain how spatially global statistics preserve the fine-grained, input-dependent features that the method claims to extract, and no analysis is provided beyond the single ablation row and the Grad-CAM figure. A mechanism analysis or an ablation comparing GAP with spatial pooling would be needed to support the design claim.","section":"3.4.1 / Eqs. (3)-(4)"}],"minor_comments":[{"comment":"With g initialized to zero, sigmoid(g)=0.5, so Eq. (5) begins by averaging fi and the frozen-path output rather than starting from the frozen path; please clarify whether this is intended and, if so, motivate the equal-weight initialization.","section":"3.5 / Eq. (5)"},{"comment":"The Adapter baseline is cited as [17], but [17] is the 'Towards a unified view' paper; the original adapter citation should be checked and corrected.","section":"4.2 / References"},{"comment":"The label 'TCP-operator' in Figure 2(b) is inconsistent with the 'TPC-operator' used throughout the text; please fix the acronym.","section":"Figure 2"},{"comment":"The notation fqkv is introduced as the output of the 'QKV module' without specifying whether it refers to the value features, the projected output, or the entire QKV concatenation; please define it precisely to make Eq. (2) unambiguous.","section":"3.3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's related work cites two unpublished preprints from the same group ([8], [9]) that are not needed for the argument; the authors may want to remove or replace them. The VMT-Adapter baseline in Table 4 is the authors' own reproduction because no official code is available; given the severe degradation in the parallel rows, the reproduction should be carefully documented. The paper would also benefit from a public main-code release before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things before you read this one. The headline numbers are genuinely good: TADFormer beats MTLoRA at every rank with fewer parameters, and the ablations show each added piece earns its keep. But the core mechanism—the TPC operator—is underspecified for the Swin backbone it actually runs on. That is the difference between a paper you can build on and a paper that needs another revision first.\n\nWhat is new: the specific combination of shallow task prompts, a task-attention-map-based reweighting of encoder features, and a dynamic convolution filter (DTF) inserted inside LoRA-style projections. Every ingredient is known, but the combination isn't in the cited baselines. The ablations in Table 2 are the strongest part: TPC + task prompts adds +0.49 Δm, DTF adds +1.65, and the full model adds +2.33 over the TADFormer baseline. Table 1 backs the central claim: at r=64, TADFormer gets Δm +4.24 with 7.38M parameters, versus MTLoRA's Δm +2.55 with 8.34M. If those numbers hold, this is a meaningful improvement within PEFT-for-MTL.\n\nNow the soft spots, in order of severity. First, the TPC operator as written computes a task attention map ATAM ∈ R^{H×T×N} between every task prompt and every patch token, which assumes full-sequence MHSA. But the encoder is Swin-T, and Swin uses windowed MHSA. The paper never states how the prompts are inserted into window attention—prepended to the whole sequence (they'd sit in one window and mostly not see the other patches), duplicated per window, or something else. Each choice changes what the attention map contains and whether Eq. 2 is even defined. This is not a notation nitpick; the TPC operator is the module whose ablation is the main evidence for the method. The authors need to specify the attention layout and give actual tensor shapes.\n\nSecond, reproducibility: no code, no error bars, no seeds, and the DTF kernel size k is never reported. Third, the adapter extension in Table 4 has parallel configurations collapsing to ~55 semseg mIoU for both baseline and TADFormer; the text compares only within configuration and doesn't address why parallel adapters fail so badly. A reviewer should ask for an explanation.\n\nOverall, the paper is worth a serious referee. The central claim is plausible and the improvements are consistent across ranks, backbones, and decoders. Send it to review, but the reviewer's first job is to pin down how the TPC operator actually works inside Swin's windowed attention.","headline":"The headline numbers are strong and the ablations are clean, but the TPC operator is underspecified for Swin's windowed attention, so the paper needs a clarifying revision before it can be fully trusted.","tokens_in":19040,"tokens_out":4977,"would_cite":false,"duration_ms":45125,"reading_group":"yes","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TADFormer claims that input-conditioned task prompts and a dynamic task filter let a frozen Swin encoder outperform full fine-tuning on multi-task dense prediction while training up to about 8 times fewer parameters.","keywords":["multi-task learning","parameter-efficient fine-tuning","task prompting","dynamic convolution","dense prediction","transformer","low-rank adaptation","visual prompt tuning"],"falsifier":"Run the rank-32 TADFormer on PASCAL-Context with the task attention map $A_{TAM}$ replaced by a fixed random map of the same shape, keeping all trained weights fixed; if the reported $\\Delta m$ of $+3.63$ does not fall by roughly the $0.39$ points the ablation attributes to the TAM, then the TPC operator is not contributing through the attention it claims to compute.","tokens_in":17982,"feed_emoji":"🎯","tokens_out":7729,"duration_ms":63940,"temperature":0.7,"pith_summary":"This paper tries to establish that a frozen pre-trained Transformer encoder can be adapted to several dense-prediction tasks at once with fewer trainable parameters than full fine-tuning and with higher accuracy, by making the adaptation depend on each input image's context rather than on static per-task parameters. The proposed TADFormer prepends a small set of trainable task prompts to the image patch tokens, reads the resulting task-attention map from the last self-attention block of each stage, and uses those maps to build task-adapted features. Those features pass through a low-rank module augmented by a dynamic task filter, a set of convolution kernels generated on the fly from global-average-pooled features, so the same module serves different tasks differently depending on the input. On the PASCAL-Context benchmark, the method reports higher overall multi-task accuracy than full fine-tuning and than recent PEFT baselines such as MTLoRA, while training roughly 4 to 8 times fewer parameters. If correct, this means input-conditioned task prompts can carry much of the task-specific adaptation that would otherwise require tuning most of the model.","feed_headline":"7.4M trainable parameters beat full fine-tuning on four tasks","feed_subtitle":"On PASCAL-Context, rank-64 TADFormer gains +4.24% while full fine-tuning gains +2.23%.","key_machinery":"The load-bearing objects are the Task-Prompt Conditional (TPC) operator and the Dynamic Task Filter (DTF). The TPC operator takes the attention map $A_{TAM} \\in \\mathbb{R}^{H \\times T \\times N}$ between $T$ task prompts and $N$ patch tokens from the last MHSA of each stage and produces a task-adapted feature $f_i = f_{qkv} + S_{inv}(a_i \\otimes \\hat{f}_{qkv})$ for each task $i$; this is what injects task identity into the frozen encoder without training the encoder. The DTF sits between the low-rank down- and up-projections of the task-aware module, generates kernels $\\theta_i = \\phi(f_i W_{down})$ from global-average-pooled down-projected features, and applies channel-wise convolution so the output is $\\tilde{F}_i = \\Phi(f_i) + (\\theta_i \\odot (f_i W_{down})) W_{up}$, with FilterNorm stabilizing training. The two act in sequence: the TPC operator decouples per-task features from task-agnostic ones, and the DTF adapts those features to the specific input while a cross-task projection lets tasks exchange information; a gated skip connection then mixes the task-adapted feature with the block output before the task-specific decoder.","core_discovery":"At rank 16, TADFormer reports $\\Delta m = +2.44\\%$ with 3.56M trainable parameters, against $+2.23\\%$ with 30.06M for full fine-tuning, and at rank 64 it reports $+4.24\\%$ with 7.38M parameters, against $+2.55\\%$ with 8.34M for MTLoRA at the same rank. The claim is that this gain comes from two interacting mechanisms: the Task-Prompt Conditional (TPC) operator, which computes per-task attention maps between task prompts and image patch tokens inside the last multi-head self-attention of each stage and rescales patch features with them, and the Dynamic Task Filter (DTF), which generates channel-wise convolution kernels from global-average-pooled, down-projected task-adapted features, so that task-specific filtering adapts to the input context. Ablations attribute roughly $+1.65\\,\\Delta m$ to the DTF, $+0.49$ to the TPC operator plus task prompts, and $+0.39$ specifically to the task attention map, with the two components together adding $+2.33$ over a LoRA-only baseline. The paper also claims the design transfers to larger backbones and pretraining datasets and can be grafted onto adapter-based PEFT methods, improving their multi-task accuracy.","pith_inferences":["The paper does not test whether the task attention map would survive insertion of prompts at every stage rather than only the first; a natural follow-up is to measure whether prompt information is diluted by layer depth, which would directly probe the weakest assumption.","Because the DTF generates kernels from global-average-pooled features, a spatially sensitive variant that pools per region or uses multi-scale pooling might recover spatial cues the current GAP discards; the paper's own framing suggests this as a testable extension, but it does not investigate it.","The reported 0.39-point gap induced by the attention map is measured at equal trainable parameter counts, so it isolates the information content of the TAM; an adversarial test that corrupts or shuffles the TAM at inference would tell whether the model has learned to rely on it or has found a shortcut.","If the same recipe transfers to other frozen backbones that expose per-head attention maps, the mechanism could become a generic plug-in for multi-task PEFT beyond the Swin and HRNet combination used here; the paper only demonstrates Swin backbones."],"forward_implications":["At rank 64, TADFormer reaches $\\Delta m = +4.24\\%$ on PASCAL-Context with 7.38M trainable parameters, exceeding both full fine-tuning's $+2.23\\%$ with 30.06M parameters and MTLoRA's $+2.55\\%$ with 8.34M parameters.","The DTF alone contributes about $+1.65\\,\\Delta m$ over the LoRA-only baseline, and the TPC operator with task prompts contributes about $+0.49\\,\\Delta m$; the task attention map itself is worth about $0.39\\,\\Delta m$ at equal parameter counts.","Switching the backbone from Swin-T on ImageNet-1k to Swin-B or to ImageNet-22k pretraining increases the relative gain, indicating the method scales with model and data size rather than saturating.","Grafting the TPC-plus-DTF design onto adapter-based PEFT frameworks improves their multi-task accuracy, e.g., AdaptFormer with the design reaches $\\Delta m = +2.69\\%$ versus $-0.63\\%$ for AdaptFormer alone at comparable parameter counts."],"supporting_citations":[{"why":"Provides the MTLoRA baseline whose parameter efficiency and accuracy TADFormer is compared against, and whose dual-module low-rank design it extends.","marker":"[1]"},{"why":"Supplies the low-rank down and up projection structure used in both the task-shared and task-aware modules.","marker":"[21]"},{"why":"Source of the shallow prompt insertion strategy and the prompt-upsampling design for hierarchical Transformer stages.","marker":"[24]"},{"why":"Conceptual origin of generating convolution filters conditioned on inputs, which the DTF adapts for multi-task use.","marker":"[25]"},{"why":"Provides FilterNorm, used to stabilize the DTF's dynamic kernel generation during training.","marker":"[59]"},{"why":"The frozen Swin Transformer backbone on which all main experiments are run.","marker":"[36]"},{"why":"The HRNet decoder used for fair comparison with MTLoRA and other baselines.","marker":"[42]"},{"why":"Supplies the fixed per-task loss weights used in the multi-task training objective.","marker":"[44]"},{"why":"Cited alongside [10] as the basis for the task attention map between task prompts and patch tokens.","marker":"[51]"}],"fun_headline_variants":["TADFormer: 8x fewer params, better multi-task accuracy","Task-adaptive transformer slashes params 8.4x, boosts accuracy","8.4x fewer params than full fine-tuning, higher accuracy","TADFormer: dynamic task filters cut params, boost MTL accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that the attention weights between the task prompts and image patch tokens, read from the last self-attention layer of each stage, still carry task-relevant context after passing through earlier frozen layers, and that global average pooling of down-projected features does not discard the spatial cues the dynamic filters need.","fun_headline_variants_meta":{"raw":{"variants":["TADFormer: 8x fewer params, better multi-task accuracy","Task-adaptive transformer slashes params 8.4x, boosts accuracy","8.4x fewer params than full fine-tuning, higher accuracy","TADFormer: dynamic task filters cut params, boost MTL accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000936,"raw_usage":{"total_tokens":4058,"prompt_tokens":1052,"completion_tokens":3006,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":2927}},"tokens_in":668,"tokens_out":3006,"duration_ms":20077,"temperature":1.0,"reasoning_tokens":2927,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:37:54.197724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the rank-32 TADFormer on PASCAL-Context with the task attention map $A_{TAM}$ replaced by a fixed random map of the same shape, keeping all trained weights fixed; if the reported $\\Delta m$ of $+3.63$ does not fall by roughly the $0.39$ points the ablation attributes to the TAM, then the TPC operator is not contributing through the attention it claims to compute.","supporting_citations":[{"cited_title":"Mti-net: Multi-scale task interaction networks for multi-task learning","cited_arxiv_id":null,"evidence_quote":"Supplies the fixed per-task loss weights used in the multi-task training objective."},{"cited_title":"Mtlora: Low-rank adaptation approach for efficient multi-task learn- ing","cited_arxiv_id":null,"evidence_quote":"Provides the MTLoRA baseline whose parameter efficiency and accuracy TADFormer is compared against, and whose dual-module low-rank design it extends."},{"cited_title":"Vi- sual prompt tuning","cited_arxiv_id":null,"evidence_quote":"Source of the shallow prompt insertion strategy and the prompt-upsampling design for hierarchical Transformer stages."},{"cited_title":"Decoupled dynamic filter networks","cited_arxiv_id":null,"evidence_quote":"Provides FilterNorm, used to stabilize the DTF's dynamic kernel generation during training."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"The frozen Swin Transformer backbone on which all main experiments are run."},{"cited_title":"Taskprompter: Spatial-channel multi-task prompting for dense scene understanding","cited_arxiv_id":null,"evidence_quote":"Cited alongside [10] as the basis for the task attention map between task prompts and patch tokens."}],"review_version":1}