{"id":"7a122921-03b5-4cac-b7be-9567d5a0360e","arxiv_id":"2507.19077","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"FGMoE combines fine-grained intra-task experts, always-active shared experts, and a global expert layer in a frozen-backbone decoder to improve multi-task dense prediction with fewer trainable parameters.","lead":"This paper proposes FGMoE, a mixture-of-experts decoder for multi-task dense prediction that adds intra-task, shared, and global experts while fine-tuning only the decoder. It reports accuracy gains on PASCAL-Context and NYUD-v2 against prior multi-task models, but the results contain internal inconsistencies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is contradicted by the paper's own reported numbers: Table 2 shows FGMoE underperforming MLoRE on NYUD-v2, and §4.2's HRNet18 results do not match Table 1.","rationale":"The reader's verdict of REJECT is well supported, but I identify a different weak point than the reader's listed weakest_assumption. The reader focused on the frozen-backbone design assumption; in contrast, the most load-bearing issue is the internal inconsistency and self-contradiction of the reported experimental evidence. The central claim of significant superiority over MoE-based MTL models is directly contradicted by Table 2 for NYUD-v2, and the text/table mismatch for HRNet18 on PASCAL-Context further undermines the reliability of the results. Even if the frozen-backbone assumption were perfectly sound, the current evidence would not establish the headline claim because the numbers do not add up. A concrete recomputation and a checkpoint re-evaluation would settle whether the reported numbers are transposition errors or genuine results. Since the reader's verdict already rejects the paper, my analysis does not change that verdict. I mark agreement as 'partial' because the reader's rationale does cite the Table 2 contradiction and the text/table mismatch, but their formal 'weakest_assumption' identifies a different (less immediately decisive) assumption.","tokens_in":14685,"tokens_out":3992,"duration_ms":36534,"concrete_test":"Recompute the per-task average performance drop for FGMoE and MLoRE from Table 2 using the formula in §4.1, applying the standard inversion for lower-is-better metrics (rmse, mErr). Check whether Δm(FGMoE) reproduces -0.58 and whether Δm(MLoRE) reproduces +0.11. Separately, reconcile the HRNet18 row in Table 1 with the values quoted in §4.2 (SemSeg 60.52 vs 68.12; PartSeg 59.01 vs 63.03) by rerunning the published training script or, if code is released, by re-evaluating the released checkpoint on PASCAL-Context. If the Table 2 numbers are correct, the central claim fails on NYUD-v2; if the §4.2 text numbers are correct, Table 1 and the reported Δm must be corrected before the claim can be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FGMoE 'uses fewer parameters and significantly outperforms current MoE-based competitive MTL models on two dense prediction datasets.' The load-bearing condition for this claim is that the reported experiments are accurate and internally consistent. That condition fails. Table 2 (NYUD-v2, ViT-L) shows FGMoE with mErr 19.19 vs. MLoRE's 18.33 for surface normal estimation, and Δm of -0.58 vs. MLoRE's +0.11. Since MLoRE is an MoE-based MTL model, the claim of outperforming MoE-based models is false on this dataset if the table is correct. Additionally, §4.2 states that FGMoE-HRNet18 achieves the highest SemSeg mIoU (68.12) and PartSeg mIoU (63.03) on PASCAL-Context, but Table 1 lists 60.52 and 59.01, which are lower than MTI-Net's 61.70 and 60.18. A direct recomputation of Δm from Table 2 using the standard per-task drop formula (with lower-is-better metrics inverted) gives approximately -0.85%, not the reported -0.58%. These internal contradictions mean the reported evidence cannot be trusted to support the headline superiority claim. The architectural ideas may have merit, but the current paper does not provide a coherent experimental basis for its main conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FGMoE, a decoder architecture for multi-task dense prediction that combines three components: intra-task experts obtained by splitting the intermediate hidden dimensions of expert MLPs, always-active shared experts that consolidate common information, and a global MoE layer for cross-task knowledge transfer. The model is evaluated on NYUD-v2 and PASCAL-Context with several backbones, under both full-model training and decoder-only fine-tuning. The abstract claims that FGMoE uses fewer parameters than existing MoE-based MTL models and significantly outperforms them on both datasets. The paper also reports ablations on the proposed components, the number of experts, and the top-k routing value.","tokens_in":14990,"tokens_out":4473,"duration_ms":38541,"significance":"If the reported results were accurate, the proposed fine-grained expert decomposition and global cross-task routing would be a useful addition to parameter-efficient multi-task dense prediction. The ablation studies in Section 4.3 provide component-level evidence that the deformable mixer, task-specific MoEs, and global MoEs contribute positively, and the decoder-only fine-tuning results in Tables 3 and 4 indicate competitive parameter counts. However, the paper's central comparative claim is not supported by its own reported numbers: Table 2 shows FGMoE underperforming MLoRE on NYUD-v2, and Section 4.2's text describing HRNet18 results contradicts Table 1. These internal inconsistencies prevent the reader from assessing the actual significance of the method, despite the plausibility of the architectural ideas.","major_comments":[{"comment":"The abstract claims FGMoE \"significantly outperforms current MoE-based competitive MTL models,\" but Table 2 (NYUD-v2, ViT-L) shows FGMoE with surface normal mErr of 19.19 versus 18.33 for MLoRE and 18.56 for the single-task baseline, and with Δm of -0.58 versus +0.11 for MLoRE. Since MLoRE is an MoE-based MTL model, this table directly contradicts the headline claim on this dataset. The claim as written is false if the table is correct.","section":"Abstract and Table 2"},{"comment":"The text in Section 4.2 states that with HRNet18, FGMoE achieves the highest SemSeg mIoU (68.12) and PartSeg mIoU (63.03) on PASCAL-Context, but Table 1 lists the FGMoE HRNet18 row as SemSeg 60.52 and PartSeg 59.01, which are lower than MTI-Net's 61.70 and 60.18. The text and table are irreconcilably inconsistent; the claim of state-of-the-art performance in this setting is unsupported as printed.","section":"Section 4.2 and Table 1"},{"comment":"Recomputing Δm from Table 2 using the formula in Section 4.1, with lower-is-better metrics inverted so that higher is better, gives approximately -0.85% for FGMoE ((56.16-56.77)/56.77 + (0.5141-0.5071)/0.5141 + (18.56-19.19)/18.56 + (78.70-78.93)/78.93, averaged over four tasks), not the reported -0.58%. The same computation for MLoRE yields approximately +0.11%, matching the table. This is an arithmetic or reporting error in the paper's primary aggregate metric.","section":"Section 4.1 and Table 2"},{"comment":"Tables 4 and 5 both report experiments on NYUD-v2 with a Swin-T backbone, but they list different single-task baselines: Table 4 gives SemSeg 33.18 and Depth rmse 0.667, while Table 5 gives SemSeg 42.92 and Depth rmse 0.6104. Unless a different training protocol or task set is being used, these baselines should be identical; without clarification, the fine-tuning gains in Table 4 cannot be compared with the full-model ablation in Table 5.","section":"Tables 4 and 5"},{"comment":"The text after Eq. (13) says that the intra-task expert design \"increases the number of non-zero gates to N_it, enabling full activation of all task-specific experts,\" but Eq. (12) defines a top-k gate that produces exactly K non-zero values. With the default K=3 and N_it=6 used in the experiments, six experts are not fully activated; the method description is internally inconsistent with the routing equations and with the top-k ablation in Table 7.","section":"Section 3.4, Eqs. (10)-(13)"}],"minor_comments":[{"comment":"There are several typos and grammatical errors: \"Th overall\" in Section 3.1, \"miminzing\" in Section 2.2, \"mxier\" in Section 4.3, and \"cures\" for \"captures\" in Section 4.4. The paper should be carefully proofread.","section":"Throughout"},{"comment":"The method label \"Adapte\" should be \"Adapter\". Also, the single-task baseline in Table 3 (SemSeg 67.21, PartSeg 61.93, Sal 62.35, Normal 17.97) differs substantially from the HRNet18 baseline in Table 1; the backbone used for the Table 3 experiments should be stated explicitly.","section":"Table 3"},{"comment":"The sentence \"we re-implement several leading CNN-based multi-task learning methods on the ViT-L backbone\" is ambiguous: it is unclear whether the ViT-L rows for PAD-Net, MTI-Net, ATRC, and InvPT in Table 1 are official results, re-implementations, or evaluations with the official code. The '†' footnote appears only for the Swin-L InvPT row.","section":"Section 4.2"},{"comment":"The qualitative discussion contains an incomplete sentence: \"it frequently maintain contextual coherence in complex scenes\" lacks a verb form consistent with the subject. The paragraph should be revised.","section":"Section 4.4"},{"comment":"The definition of Δm does not specify how lower-is-better metrics (rmse, mErr) are incorporated. The paper should state explicitly that these metrics are inverted, e.g., using (F_s - F_m)/F_s, so that Δm is consistently higher-is-better.","section":"Section 4.1"}],"recommendation":"reject","confidential_remarks":"The manuscript contains several numerical contradictions that directly affect the paper's central claim: the abstract's assertion of significant outperformance over MoE-based models is contradicted by Table 2, Section 4.2's HRNet18 description does not match Table 1, and the reported Δm for FGMoE in Table 2 does not match the formula. These are not cosmetic issues; they undermine the credibility of all experimental comparisons. The architectural ideas may have merit, but the paper is not in a form where the claims can be trusted. I would encourage the authors to correct the tables, rerun or re-check the experiments, and resubmit with a revised abstract and conclusions that accurately reflect the measured results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe honest summary is this: the architecture is plausible and worth knowing about, but the paper's headline claim is not supported by its own tables. I would not trust the numbers enough to cite them.\n\nWhat is new: the specific combination of fine-grained intra-task experts (hidden-dimension splitting, in the spirit of DeepSeekMoE), always-active shared experts, and a global cross-task MoE layer in a frozen-backbone decoder. That combination is not in TaskExpert or MLoRE. The idea of isolating common knowledge in non-routed experts to reduce redundancy is sound, and the ablations are directionally consistent: each component adds a bit, and the expert-count and top-k sweeps show sensible saturation. If the decoder-only fine-tuning numbers hold up, they would be genuinely useful.\n\nThe soft spots are not minor. Table 2 shows FGMoE with mErr 19.19 on NYUD-v2 normal estimation versus MLoRE's 18.33, and Δm of -0.58 versus MLoRE's +0.11. That directly contradicts the abstract's claim of consistent wins over MoE-based models. Section 4.2 says HRNet18 gets 68.12 SemSeg and 63.03 PartSeg, but Table 1 lists 60.52 and 59.01, and those values trail MTI-Net. I also recomputed Δm from Table 2 using the paper's own formula (inverting lower-is-better tasks) and got about -0.85%, not -0.58%. No single one of these would sink an empirical paper; together they mean the central evidence has not been checked. The default hyperparameters are not the best in the paper's own ablations (Top-4 and 8 experts beat Top-3 and 6), which undercuts the setup story. No code, no error bars, and no seeds reported, so there is no way to tell whether the remaining gaps are real or noise.\n\nThe literature coverage looks fine. The work draws on DeepSeekMoE, TaskExpert, MLoRE, and DeMT; self-citations are to relevant prior work, not padding.\n\nWho is this for? People working on parameter-efficient multi-task dense prediction might borrow the shared-expert idea. But they should not rely on these numbers. I would not send this to peer review in its current form; the authors need to fix the internal inconsistencies and provide code or multiple seeds before it can be evaluated seriously. If they come back with consistent tables and reproducible code, it is worth another look.\n\nRecommendation: desk reject with an invitation to resubmit after the numbers are reconciled.","headline":"Plausible fine-grained MoE decoder idea, but the paper's own tables contradict its headline performance claims, so the evidence is not usable as reported.","tokens_in":15510,"tokens_out":5627,"would_cite":false,"duration_ms":52227,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decoder built from fine-grained, shared, and global mixture-of-experts layers can beat full-model multi-task training while keeping the backbone frozen and training only 2.5–4.7M parameters.","keywords":["Scene Understanding","Multi-task Learning","Mixture-of-Experts","Fine-Tuning","Dense Prediction","Parameter-Efficient Fine-Tuning","Shared Experts","Global Expert"],"falsifier":"Train the identical FGMoE decoder twice on NYUD-v2 with Swin-T, once with the backbone frozen and once with the final backbone stage unfrozen under matched parameter and compute budgets; if unfreezing raises $\\Delta_m$ by more than a small margin, the claim that decoder-only fine-tuning is sufficient collapses, and if a decoder on a randomly initialized frozen backbone recovers comparable $\\Delta_m$, the decoder architecture rather than the pretrained features would be doing the work.","tokens_in":1900,"feed_emoji":"🧩","tokens_out":2444,"duration_ms":87836,"temperature":0.7,"pith_summary":"This paper claims that multi-task dense prediction—predicting segmentation, depth, normals, boundaries, and saliency from one image—can be done better by training only a small decoder on top of a frozen Swin Transformer than by fine-tuning the whole network. The proposed architecture, FGMoE, replaces monolithic task-specific decoders with three kinds of mixture-of-experts blocks: fine-grained intra-task experts, always-on shared experts, and a global expert layer for cross-task knowledge. On NYUD-v2 and PASCAL-Context, the authors report that FGMoE outperforms prior MoE-based multi-task models such as TaskExpert and MLoRE while using substantially fewer trainable parameters. If the claim holds, parameter-efficient fine-tuning is not a compromise but a route to stronger multi-task dense prediction.","feed_headline":"Frozen-backbone MoE decoder beats full multi-task fine-tuning","feed_subtitle":"Fine-grained experts train only 2.5–4.7M decoder parameters yet beat MoE baselines on NYUD-v2 and PASCAL-Context.","key_machinery":"The load-bearing mechanism is the task-specific MoE block, whose output is the sum of always-on shared experts plus a top-k weighted combination of fine-grained intra-task experts. Intra-task experts are created by splitting the intermediate hidden dimension of an MLP, so more experts can be activated without adding parameters; shared experts, fixed at two, consolidate common context and reduce redundancy in the routed experts; and a global MoE layer, using the same block structure, aggregates knowledge across tasks and is added to each task-specific output before the prediction head. Around these blocks, a feature aggregator fuses four Swin stages and a deformable mixer, combining channel convolution, deformable convolution, and multi-head self-attention, aligns spatial features so the decoder can specialize per task while the backbone stays frozen.","core_discovery":"The paper claims that the routing granularity of a decoder, not its parameter count or backbone fine-tuning, is what determines multi-task dense-prediction quality. FGMoE splits each expert's hidden MLP dimension into fine-grained sub-experts so that a fixed parameter budget activates more experts, marks a subset of experts as always-active shared experts to absorb redundant common features, and adds a global MoE layer that routes cross-task information per pixel and per task. With the Swin backbone frozen and only the roughly 2.5–4.7M-parameter decoder trained, FGMoE reports higher mean per-task improvement than the leading decoder-only tuning baselines on PASCAL-Context and NYUD-v2, and with full training it beats TaskExpert and MLoRE on PASCAL-Context with a ViT-L backbone while using fewer decoder parameters. The intended upshot is that task-specific specialization and cross-task sharing can be engineered entirely in a small decoder, making parameter-efficient fine-tuning the preferred mode for multi-task dense prediction.","pith_inferences":["A natural but untested extension is to swap the frozen Swin backbone for other frozen feature extractors such as convnets or self-supervised ViTs; if the decoder is the true source of task specialization, the gains should transfer, and if not, the frozen-feature premise is what carries the result.","The fine-grained expert-splitting trick is parameter-agnostic: splitting MLP hidden dimensions could be applied to any MoE layer outside dense prediction, such as language or speech decoders, to raise expert count without raising parameter count; the paper does not claim this.","The small depth-error gap on NYUD-v2 (FGMoE rmse 0.618 vs DITASK 0.606) hints that the average $\\Delta_m$ metric may hide per-task trade-offs; a metric-by-metric comparison rather than only the mean would clarify where the advantage actually lies.","If the frozen-backbone premise is pushed, one testable corollary is that FGMoE's advantage should shrink on tasks requiring high-frequency detail not present in Swin features, such as boundary detection at low resolution; a controlled backbone-capacity study would settle it."],"forward_implications":["If FGMoE is right, decoder-only fine-tuning becomes the default mode for multi-task dense prediction: with 4.7M trainable parameters on PASCAL-Context with Swin-L it reaches +7.89% mean per-task improvement, slightly above DITASK's +7.79% at 7.13M parameters.","Task-specific decoders no longer need per-task parameter pools; fine-grained expert splitting lets a fixed decoder budget cover multiple tasks, so task count can grow without proportional decoder growth.","The global expert layer that routes per pixel and per task can reduce negative transfer: FGMoE reports a positive $\\Delta_m$ on full PASCAL-Context training with ViT-L (+0.39%) where prior MoE decoders report negative drops, meaning multi-task training is no longer a loss relative to single-task baselines.","The ablation result that 8 routed experts outperform 16, and that top-k near 4–6 is best, implies expert count and sparsity should be tuned as hyperparameters rather than maximized; the paper itself lists adaptive top-k as future work.","A small positive $\\Delta_m$ in decoder-only fine-tuning, despite using only about 2–4% of the parameters of full-network training, suggests parameter-efficient multi-task models can be deployed where full fine-tuning is too costly."],"supporting_citations":[{"why":"Supplies the first MoE-based MTL decoder and the ViT-L PASCAL-Context baseline against which FGMoE reports gains.","marker":"[45]"},{"why":"Supplies the low-rank MoE baseline with the strongest prior $\\Delta_m$ on PASCAL-Context; FGMoE compares against it under the same ViT-L backbone.","marker":"[42]"},{"why":"Provides the Swin-T and Swin-L backbones that FGMoE freezes in decoder-only fine-tuning and uses in full-model training.","marker":"[24]"},{"why":"Provides the NYUD-v2 indoor dataset (segmentation, depth, normals, boundary) used for the decoder fine-tuning experiments.","marker":"[29]"},{"why":"Provides the PASCAL-Context dataset (semseg, parts, saliency, normals, boundary) used for the main comparisons.","marker":"[4]"},{"why":"Gives the routing formulation and gating notation that FGMoE's intra-task and shared expert design modifies.","marker":"[19]"},{"why":"Supplies the deformable mixer structure that FGMoE adapts in its feature aggregator.","marker":"[39]"},{"why":"Parameter-efficient multi-task adaptation baseline compared in both decoder-only fine-tuning tables.","marker":"[1]"},{"why":"The strongest decoder-only fine-tuning baseline; FGMoE reports higher $\\Delta_m$ with similar or lower trainable parameters.","marker":"[27]"}],"fun_headline_variants":["Fine-grained experts let small decoder beat full fine-tuning","Frozen backbone, fine-grained MoE: more accuracy, less training","Split experts for better multi-task dense prediction","Decoder-only fine-grained MoE beats full fine-tuning","Fine-grained routing in decoder outperforms full fine-tuning"],"cache_read_input_tokens":17536,"weakest_assumption_plain":"The load-bearing premise is that a frozen Swin Transformer already contains all the generic visual features these dense tasks need, so no backbone parameter has to change; if the frozen features are insufficient for a task, the small decoder has no way to recover them.","fun_headline_variants_meta":{"raw":{"variants":["Fine-grained experts let small decoder beat full fine-tuning","Frozen backbone, fine-grained MoE: more accuracy, less training","Split experts for better multi-task dense prediction","Decoder-only fine-grained MoE beats full fine-tuning","Fine-grained routing in decoder outperforms full fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2316,"prompt_tokens":971,"completion_tokens":1345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":1267}},"tokens_in":587,"tokens_out":1345,"duration_ms":10299,"temperature":1.0,"reasoning_tokens":1267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:00:57.197128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical FGMoE decoder twice on NYUD-v2 with Swin-T, once with the backbone frozen and once with the final backbone stage unfrozen under matched parameter and compute budgets; if unfreezing raises $\\Delta_m$ by more than a small margin, the claim that decoder-only fine-tuning is sufficient collapses, and if a decoder on a randomly initialized frozen backbone recovers comparable $\\Delta_m$, the decoder architecture rather than the pretrained features would be doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the low-rank MoE baseline with the strongest prior $\\Delta_m$ on PASCAL-Context; FGMoE compares against it under the same ViT-L backbone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Swin-T and Swin-L backbones that FGMoE freezes in decoder-only fine-tuning and uses in full-model training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NYUD-v2 indoor dataset (segmentation, depth, normals, boundary) used for the decoder fine-tuning experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PASCAL-Context dataset (semseg, parts, saliency, normals, boundary) used for the main comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the routing formulation and gating notation that FGMoE's intra-task and shared expert design modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the deformable mixer structure that FGMoE adapts in its feature aggregator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Parameter-efficient multi-task adaptation baseline compared in both decoder-only fine-tuning tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The strongest decoder-only fine-tuning baseline; FGMoE reports higher $\\Delta_m$ with similar or lower trainable parameters."}],"review_version":2}