{"id":"3601a059-b021-4446-9f39-5d4a972f1bfa","arxiv_id":"2411.18823","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Hierarchical task tokens discover feature- and prediction-level pseudo supervision, improving partially annotated multi-task dense prediction.","lead":"This paper trains compact per-task tokens that generate pseudo labels to supervise unlabeled dense prediction tasks when only some tasks have ground truth. The method beats prior partially-supervised multi-task approaches on three benchmarks while using fewer parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline ΔMTL gains are not reproducible: Tables 2/3 and supplement Tables 6/8 report different ΔMTL values for identical raw metrics, so the +6.45/+7.41/+8.76% SOTA claims depend on an unstated metric-choice and averaging convention.","rationale":"The reader's formal weakest_assumption targets the cross-task affinity A×F mixing noise for weakly related tasks. That is a plausible limitation, but the paper's three benchmark suites contain mostly correlated tasks, the ablations show both inter- and intra-task attention contribute, and a low-correlation failure would narrow scope without overturning the reported results. The metric inconsistency is more load-bearing because it bears directly on every headline number. Two tables with identical raw metrics yield ΔMTL values differing by several points (e.g., Ours 16.91 vs 13.23 on NYUD-v2 one-label; XTC 10.46 vs 3.60). Since Eq. 16 does not specify the metric set and the main text cites specific gains over XTC, the central quantitative claim is not uniquely reproducible. I agree with the reader's CONDITIONAL verdict: the raw metric tables and ablations are supportive, code is released, and the method is plausible, but the ΔMTL reporting must be reconciled before the specific SOTA margins can be accepted. Hence the verdict should remain CONDITIONAL/UNCHANGED.","tokens_in":28959,"tokens_out":9335,"duration_ms":79924,"concrete_test":"Recompute ΔMTL for all rows of Tables 2, 3, 6, and 8 using one fixed formula: ΔMTL = (100/T) Σ_i (-1)^{l_i}(M_{m,i}-M_{s,i})/M_{s,i}, with T equal to the number of tasks and exactly one primary metric per task (mIoU for Semseg, AbS for Depth, mErr for Normal). Then recompute the same tables with the 9-metric variant used in the supplement. If the main-text numbers (+6.45, +7.41, +8.76) do not match either convention, correct the tables and restate the SOTA claims; if Ours ceases to be SOTA under the intended convention, the central claim is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 16 defines ΔMTL as a sum over N, but N is never fixed. Main Table 2 (NYUD-v2 one-label) reports MTLbaseline ΔMTL=5.61, XTC=10.46, Ours=16.91; Supplement Table 6, with identical Semseg/Depth/Normal numbers, reports 0.11, 3.60, 13.23. Similarly, Cityscapes Ours is 5.31 in Table 3 but 3.45 in Table 8, changing the claimed gain over XTC from +8.76 to +6.98. The raw metrics are the same, so the differences come from an unstated change in which metrics enter Eq. 16 (likely the 9-row full metric set in Tables 6/8 instead of the 3-task primary metrics used in Tables 2/3). Because every SOTA comparison and ablation conclusion is expressed in ΔMTL, the exact magnitude of the central claim is not uniquely determined by the paper. The direction of the improvement is consistent and the raw metrics do show Ours ahead, so this is a reproducibility/verifiability defect rather than a demonstrated inversion of results; nonetheless, the precise headline numbers cannot be checked until the convention is fixed and all tables are recomputed under it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hierarchical Task Tokens (HiTTs), consisting of global task tokens θ_i and fine-grained task tokens φ_i, for partially annotated multi-task dense prediction. Global tokens are learned via inter- and intra-task attention and are used to discover feature-level pseudo supervision (L_f), while fine-grained tokens are projected from the global tokens with orthogonal embeddings and are used to discover prediction-level pseudo labels (L_p). The method follows a two-stage self-training pipeline: first train on labeled data, then generate offline pseudo supervision, then retrain from scratch. Experiments are reported on PASCAL-Context, NYUD-v2, and Cityscapes under one-label and random-labels settings, with claims of state-of-the-art ΔMTL gains over XTC and DiffusionMTL. Code is released.","tokens_in":29342,"tokens_out":4930,"duration_ms":43528,"significance":"If the quantitative claims are reproducible, the paper makes a significant contribution: it replaces heavy cross-task mapping networks with compact learnable tokens, reports lower parameter counts (62M vs. 133M) and GFLOPs than DiffusionMTL, and demonstrates consistent improvements across three benchmarks and multiple dense prediction tasks. The component ablations broadly support the design choices, and the release of code is a strength. However, the central ΔMTL claims are currently not uniquely determined because the main text and the supplementary material use different metric sets in Eq. (16) without stating the convention; until the tables are recomputed under a single convention, the exact magnitude of the claimed improvements cannot be verified.","major_comments":[{"comment":"The ΔMTL values for identical raw metrics differ between the main text and the supplementary material. For NYUD-v2 one-label, Table 2 reports Ours=16.91 and XTC=10.46, while Table 6 reports Ours=13.23 and XTC=3.60 with the same Semseg/Depth/Normal primary metrics. For Cityscapes one-label, Table 3 reports Ours=5.31 and XTC=-3.45, while Table 8 reports Ours=3.45 and XTC=-3.53. The main text claims +6.45/+7.41/+8.76 ΔMTL over XTC, but Supp B.5.1 claims +9.63/+8.62/+6.98. Eq. (16) sums over N without ever fixing N or listing which metrics enter the sum. This makes the headline SOTA gains unverifiable. The authors must fix a single metric convention, state it explicitly, and recompute all ΔMTL values in both the main text and the supplement.","section":"Section 4.2, Supp B.5.1, Eq. (16)"},{"comment":"The ablation conclusions also shift under the two ΔMTL conventions. The additive gain of HiTTs over the MTL baseline is +4.37 in Table 4 but +6.40 in Table 9, while the text in Supp B.5.2 says +9.64, which matches neither. The degradation from removing orthogonal embeddings is -7.32 in Table 4 but -4.38 in Table 9. Since every component comparison and the interpretation of which design choice matters is expressed in ΔMTL, the relative importance of the components is currently not reproducible. The same fix as above is required.","section":"Section 4.3, Supp B.5.2, Table 4 vs. Table 9"},{"comment":"The cross-task affinity mechanism Θ' = A × Θ and F' = A × F assumes that affine combination with other task features improves the features of an unlabeled task. This is a load-bearing assumption for the proposed feature-supervision transfer, but the paper does not analyze the learned affinity matrix A or test settings with weakly correlated task pairs. If a labeled task is weakly related to an unlabeled task, the affine combination could inject noise rather than supervision. Add an analysis of the learned A and/or a sensitivity experiment with deliberately mismatched task subsets to demonstrate that the mechanism is robust.","section":"Section 3.1, Eqs. (5)-(6)"},{"comment":"All reported results are from single runs without error bars or repeated seeds. Given the ΔMTL inconsistencies above, the absence of variance information makes it impossible to assess whether smaller margins (e.g., the PASCAL-Context gains of +2.44 and +1.46 over DiffusionMTL(F) in Table 1) are significant. Reporting mean and standard deviation over at least three seeds would materially strengthen the empirical claims.","section":"All experimental tables"}],"minor_comments":[{"comment":"There is a duplicated word in the sentence introducing Eq. (2): “where where L_i(·) is the loss function.”","section":"Section 3.3, Eq. (2)"},{"comment":"The text refers to “MTAN [28]” twice, but the MTAN reference is [33] in the bibliography; the citation should be corrected.","section":"Section 4.2, NYUD-v2 paragraph"},{"comment":"The table uses “%” symbols in place of the checkmark/cross markers for model components (e.g., “MTDNet” and “Mapping Network”), which is confusing and should be replaced with readable symbols or –/✓ entries.","section":"Table 1"},{"comment":"The header lists both Depth and Normal columns as “mErr↓”; the Depth column should be an absolute error metric (AbS or absErr) consistent with the other tables.","section":"Table 13"},{"comment":"The caption reads “Quantitative analysis of the quality of pseudo labels,” but the figure shows qualitative visualizations; the caption should be corrected to “qualitative comparison.”","section":"Figure 14"},{"comment":"The sentence “each item is combined with the weight 1 to form the total loss” is ambiguous; specify whether all three loss terms are simply added with equal weight and whether any loss balancing is applied.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-positioned for the venue and the code release is a plus. Please ask the authors to reconcile the main-text and supplementary ΔMTL conventions before the paper is reconsidered; the current inconsistency affects the headline claims and every ablation conclusion. The underlying direction of improvement appears consistent, so the defect is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I checked the tables and the stress-test concern holds up. Eq. 16 defines ΔMTL as a sum over N without ever fixing N, and the paper never states which metrics enter the sum. Main Table 2 reports MTLbaseline 5.61, XTC 10.46, Ours 16.91 for NYUD-v2 one-label; Supplement Table 6, with identical Semseg/Depth/Normal numbers, reports 0.11, 3.60, 13.23. Same story on Cityscapes: Table 3 gives Ours 5.31, Table 8 gives 3.45, so the claimed gain over XTC changes from +8.76 to +6.98. The ablation tables disagree too: HiTTs is 9.98 in Table 4 and 6.51 in Table 9. The likely cause is that the main tables use the three primary task metrics while the supplement uses the fuller nine-metric set, but that convention is unstated and Eq. 16 is unusable as written. The direction of the improvement is consistent and the raw metrics do show Ours ahead, so the core result probably survives a rerun, but the exact magnitudes cannot be checked from the paper.\n\nWhat is genuinely new: the two-level token design, with global tokens generating feature-space pseudo supervision and fine-grained tokens generating prediction-space pseudo labels, plus the orthogonal embedding trick to stop fine-grained tokens from collapsing. The component ablations in Table 4 mostly make sense: removing OE hurts a lot, and the two loss terms are complementary. Comparing against naive pseudo-labeling (L*p) is the right baseline to include. The method is also compact (62M params vs 133M+ for DiffusionMTL) and code is released, both of which help.\n\nOther soft spots are minor relative to the ΔMTL problem: single-run results with no error bars, the untested assumption that labeled task features help unlabeled tasks even when tasks are weakly related, and per-dataset thresholds τ_i with sensitivity analysis done only under a different setup than the main SegNet experiments. None of these would sink the method.\n\nThis paper is for people working on partially annotated multi-task dense prediction. It deserves a serious referee: a careful reviewer could ask for one consistent ΔMTL definition, all tables recomputed under it, and error bars. I would not cite the headline numbers in their current form, but I would keep an eye on the corrected version.","headline":"Genuinely new token-based pseudo-labeling method with strong raw results, but the headline ΔMTL numbers are not reproducible as reported because the metric convention is never fixed.","tokens_in":729,"tokens_out":951,"would_cite":false,"duration_ms":33435,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that learning compact hierarchical task tokens alongside a multi-task dense prediction network can discover pixel-level pseudo-supervision for tasks without annotations, and that this sets state-of-the-art results on…","keywords":["multi-task learning","partially annotated dense prediction","label discovery","hierarchical task tokens","pseudo-labeling","feature-space supervision","prediction-space supervision","cross-task attention"],"falsifier":"A direct test would be to add a deliberately unrelated supervised task, such as predicting a fixed random class id per pixel, and compare the unlabeled tasks' accuracy with inter-task mixing enabled versus disabled; if the unrelated labels do not help or actively hurt, the cross-task affinity assumption is not transferring useful supervision.","tokens_in":28796,"feed_emoji":"🏷️","tokens_out":13173,"duration_ms":103453,"temperature":0.7,"pith_summary":"The paper is trying to establish that in partially annotated multi-task dense prediction — where each image is labeled for only some of the tasks it should predict — a set of compact learnable task tokens can recover the missing pixel-level supervision without heavy auxiliary mapping networks or diffusion decoders. The method learns a two-level token hierarchy: global tokens capture cross-task relations and provide feature-level pseudo-supervision, and fine-grained tokens inherited from them generate pixelwise pseudo-labels in prediction space. On PASCAL-Context, NYUD-v2, and Cityscapes, this yields state-of-the-art results under both one-label and random-labels annotation settings, including a 6.45-point gain in ΔMTL over XTC on NYUD-v2 one-label and an 8.76-point gain on Cityscapes, while using substantially fewer parameters than the best prior method.","feed_headline":"Hierarchical tokens beat heavy networks on partly labeled multi-task vision","feed_subtitle":"Global and fine-grained task tokens generate pixel-level pseudo-labels for unlabeled tasks, gaining up to 8.76 points of ΔMTL.","key_machinery":"The load-bearing object is the two-level token hierarchy trained inside the decoder. In the global stage, tokens $\\theta_i$ are updated by inter-task attention, which computes an affinity matrix $A = \\mathrm{Softmax}(QK^\\top) \\in \\mathbb{R}^{T\\times T}$ and applies affine combinations $\\Theta' = A\\Theta$, $F' = AF$, so supervised tasks' representations are mixed into unsupervised task features; intra-task self-attention then refines each token-feature pair. In the fine-grained stage, updated $\\theta_i$ are projected by an MLP into spatial task tokens $\\varphi_i$, with orthogonal embeddings added to the MLP input to keep the token rows distinguishable, and $\\varphi_i$ modulates the projected feature map through $G'_i = \\mathrm{Conv}_{3\\times3}(G_i \\odot \\mathrm{Softplus}(\\varphi_i \\times G_i))$. This hierarchy produces two losses: feature-level $L_f$ masked by confidence $M^f_i = \\mathrm{Sigmoid}(\\theta_i^\\top F_i)$, and prediction-level $L_p$ using pseudo-labels $\\tilde{Y}_i = \\arg\\max(\\mathrm{Softmax}(G'_i))$ masked by $M^p_i = \\max(\\mathrm{Softmax}(G'_i)) > \\tau_i$, summed as $L = L_s + L_p + L_f$.","core_discovery":"The central claim is that two hierarchies of tokens can discover consistent pixel-wise supervision for tasks that lack labels: global task tokens θ_i learn through inter- and intra-task attention over the shared feature map and yield feature-level pseudo-supervision, while fine-grained task tokens φ_i, projected from θ_i and regularized by orthogonal embeddings, refine task predictions and supply high-confidence pseudo-labels. The paper argues that because encoder features are shared across tasks, the learned tokens transfer discriminative information from labeled tasks to unlabeled ones, and that this transfer happens more cheaply and effectively than cross-task mapping networks or diffusion decoding. Its evidence is the reported performance on three benchmarks: on NYUD-v2 one-label, the method reaches 16.91% ΔMTL versus 10.46% for XTC, and on Cityscapes one-label it reaches 5.31% ΔMTL versus −3.45% for XTC; on PASCAL-Context it exceeds DiffusionMTL (F) by 2.44% ΔMTL with roughly 45% of its parameters and 70% of its FLOPs.","pith_inferences":["Inference: if the inter-task affinity is the cause of the gains, then performance should scale with the number of labeled tasks available to mix and degrade when only one task carries ground truth; a controlled sweep over the number of labeled tasks would test this directly.","Inference: the same token hierarchy should transfer to other missing-supervision settings, such as sparse point labels, incomplete object masks, or absent input modalities, because the tokens learn a task's expected feature distribution rather than an explicit mapping from another task's predictions.","Inference: the orthogonality regularization suggests a boundary condition — when a task's categories are not separable by orthogonal directions, the fine-grained tokens may collapse and the pseudo-labels would need a different prior; testing on an ordinal regression task without binning would reveal this."],"forward_implications":["On NYUD-v2 one-label, the method's 16.91% ΔMTL (versus 10.46% for XTC) means partial annotations can support strong semantic segmentation, depth, and normal estimation simultaneously, so the limit is not the missing labels but how well the model can mine them.","On Cityscapes one-label, the method reaches 5.31% ΔMTL while XTC sits at −3.45%, and it is the only compared method with balanced gains on segmentation and depth, indicating that cross-task token transfer can turn an annotation imbalance into a symmetric improvement.","The ablations show feature-space and prediction-space supervision are complementary: $L_f$ alone adds 7.27 ΔMTL over baseline, $L_p$ alone adds 9.88, and together they add 11.30, implying both hierarchies should be trained jointly rather than choosing one.","On PASCAL-Context, the method improves ΔMTL by 2.44 points over DiffusionMTL (F) while using about 45% of its parameters and 70% of its GFLOPs, suggesting label discovery via tokens is a lighter alternative to diffusion-based pseudo-labeling."],"supporting_citations":[{"why":"Defines the partially annotated multi-task setting and supplies the XTC baseline, the paper's main point of comparison on NYUD-v2 and Cityscapes.","marker":"[28]"},{"why":"Supplies the JTR baseline, an alternative cross-task regularization method that the paper must outperform.","marker":"[41]"},{"why":"Supplies DiffusionMTL, the strongest prior on PASCAL-Context, with prediction and feature diffusion modes and the parameter/FLOP counts the paper compares against.","marker":"[74]"},{"why":"Provides the MTAN attention baseline as well as the SegNet/ResNet-18 task decoding setup and data-resolution conventions used in the experiments.","marker":"[33]"},{"why":"Provides the depth-bin and normal-cluster quantization that converts continuous regression tasks into pixel classification, a prerequisite for token-based label discovery.","marker":"[4]"},{"why":"Supplies the offline pseudo-labeling scheme (label clean images, train on augmented images) that the paper adapts from classification to dense prediction.","marker":"[65]"},{"why":"Defines the ΔMTL aggregate metric used for all multi-task performance comparisons.","marker":"[56]"}],"fun_headline_variants":["Task tokens generate pixel-level pseudolabels for partial labels","Hierarchical tokens produce cheap pseudo-labels for dense tasks","Task tokens outperform mapping networks in partial MTL","Small hierarchical tokens, big gains for partly labeled vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that carries the method is that features from labeled tasks are informative enough to improve features of tasks without labels when mixed through the affinity matrix; if a supervised task is weakly related to an unsupervised one, the affine combination can inject noise instead of useful supervision.","fun_headline_variants_meta":{"raw":{"variants":["Task tokens generate pixel-level pseudolabels for partial labels","Hierarchical tokens produce cheap pseudo-labels for dense tasks","Task tokens outperform mapping networks in partial MTL","Small hierarchical tokens, big gains for partly labeled vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2384,"prompt_tokens":1004,"completion_tokens":1380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1316}},"tokens_in":620,"tokens_out":1380,"duration_ms":9666,"temperature":1.0,"reasoning_tokens":1316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:50:14.292856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to add a deliberately unrelated supervised task, such as predicting a fixed random class id per pixel, and compare the unlabeled tasks' accuracy with inter-task mixing enabled versus disabled; if the unrelated labels do not help or actively hurt, the cross-task affinity assumption is not transferring useful supervision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the partially annotated multi-task setting and supplies the XTC baseline, the paper's main point of comparison on NYUD-v2 and Cityscapes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the JTR baseline, an alternative cross-task regularization method that the paper must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DiffusionMTL, the strongest prior on PASCAL-Context, with prediction and feature diffusion modes and the parameter/FLOP counts the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the offline pseudo-labeling scheme (label clean images, train on augmented images) that the paper adapts from classification to dense prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ΔMTL aggregate metric used for all multi-task performance comparisons."}],"review_version":1}