REVIEW 4 major objections 4 minor 1 cited by
Feature Partitioning for Efficient Multi-Task Architectures
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Searching over channel-sharing fractions, guided by a one-minute distillation proxy, finds multi-task networks that beat fixed sharing strategies while using fewer channels per task.
desk verdict A useful search-space parameterization and a sensible distillation proxy, but the headline resource trade-off is unsupported because Table 2 never reports channel use. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the normalized channel-overlap matrix $P = \frac{1}{C}M^{\top}M$, where $M \in \{0,1\}^{C \times N}$ is a binary matrix of channel masks for the $N$ tasks; diagonal entries of $P$ are the fraction of channels each task uses and off-diagonal entries are the fraction of channels shared between task pairs. Because raw mask matrices are redundant under channel permutation, the paper searches over $P$ and remaps raw values into feasible overlap constraints, then samples a mask that realizes the target $P$. The second mechanism is the feature-distillation proxy: shared layers under a candidate mask are trained for about 3000 iterations to reproduce the features of single-task teachers at a target depth, and the resulting accuracy is used to score the candidate without training the full model to convergence.
What would settle it
Take a held-out set of fifty random partitionings, score each with the one-minute distillation proxy, then train each to convergence; if the proxy ranking and the final ranking reverse substantially, or if the top proxy candidates systematically lose to lower-ranked ones after full training, the search's guidance fails. A concrete place to look is daimlerpedcls, where the paper already reports the lowest distillation-to-final correlation.
Extended reading notes
Core claim
The paper's central claim is that the right search space for multi-task architectures is not whole layers but individual feature channels: every task gets a binary mask selecting a subset of channels, and the search operates on $P = \frac{1}{C}M^{\top}M$, the normalized overlap matrix whose diagonal gives each task's channel fraction and whose off-diagonal entries give pairwise sharing. It claims this parameterization is low-dimensional, interpretable, independent of layer width, and mappable back to feasible masks with median error under one percent. On the Visual Decathlon benchmark, the paper reports that random sampling over $P$ maps the accuracy-versus-resource frontier, and that evolutionary strategies with a diagonal weight-decay term push beyond fixed baselines: the found partitioning both reduces average per-task channel use and raises the overall validation score relative to independent, share-half, and share-all schemes. The search is made practical by a feature-distillation proxy that evaluates a candidate in about one minute, with correlations to final validation accuracy that the paper reports as mostly higher than short-training baselines.
Load-bearing premise
The load-bearing premise is that the one-minute feature-distillation score ranks multi-task architectures in the same order as full training to convergence, even though the reported correlations vary by task and are as low as about 0.48 for two of the nine tasks.
Editorial extensions
If this is right
- If the parameterization works, multi-task architecture search becomes a continuous optimization problem with a single resource knob: diagonal weight decay moves the solution along the accuracy-versus-compute frontier.
- The one-minute distillation evaluation means thousands of candidate partitionings can be ranked in a few hours on a single GPU, making resource-aware search practical where full training is prohibitive.
- Because $P$ does not depend on channel count, a partitioning strategy found for one layer width transfers to other widths and depths without re-searching.
- At inference time, masks prune unused channels directly, so a task that needs only a subset of the shared features runs at a fraction of the compute while still benefiting from joint training.
- If these results hold, fixed sharing recipes such as sharing all layers or splitting channels evenly are dominated by automatic search and should be replaced by task-specific channel budgets.
Reading between the lines
- The experiments freeze and share the first two-thirds of an ImageNet-pretrained network, so the conclusion that partitioning matters only in the last block likely depends on that strong initialization; a testable extension is repeating the search from random initialization to see whether earlier blocks need partitioning too.
- The proxy is validated by overall correlation on random architectures, not by how often the top-ranked candidate actually wins after full training; a practical extension is a top-K selection procedure that trains the top few proxy candidates to convergence and compares their final ordering.
- Because $P$ records only pairwise overlap, the search cannot represent three-task interactions such as a channel block shared exclusively by a triplet; a natural extension is a higher-order sharing tensor or a hierarchical partition of channels.
- The resource metric is average per-task channel use in the last block; an extension would optimize for FLOPs, peak memory, or heterogeneous per-task budgets, which the current diagonal penalty only approximates.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a channel-level feature-partitioning approach to multi-task architecture search. Instead of searching directly over binary channel masks, the authors introduce a continuous matrix P whose diagonal captures per-task channel use and whose off-diagonal entries capture pairwise sharing, together with a constraint-mapping procedure to ensure feasibility. They evaluate architectures with a fast feature-distillation proxy and search by random sampling and by evolutionary strategies with an L2 penalty on the diagonal of P. On Visual Decathlon (nine tasks), they compare full-training validation accuracy of independent, share-half, share-all, and ES-discovered partitions, and claim that the ES result reduces average channel use while improving overall accuracy. The paper also reports per-task correlations between distillation accuracy and final validation accuracy.
Significance. If the resource-efficiency claim can be substantiated, this is a useful contribution: the P-matrix parameterization is a compact and interpretable search space for multi-task sharing at channel granularity, and the distillation proxy is a clever way to make architecture evaluation cheap. The empirical study is careful in its use of multi-run averages and standard errors, and the analysis in Figure 5 (random-sample distributions and fixed-diagonal sharing sensitivity) gives real insight into the role of sharing. The strengths are the continuous parameterization, the fast proxy, and the clear presentation of the search-space design. However, the central trade-off claim is currently not verifiable because resource use is never reported numerically, and the mask-construction procedure is asserted rather than specified. These omissions limit the current significance of the empirical results.
major comments (4)
- [5.2, Table 2] The central resource-accuracy trade-off claim is not verifiable as written. Table 2 reports only validation accuracies for the four configurations, and the text states that the ES-found parameterization 'both reduces average channel use and achieves better performance overall,' but no column or figure reports the mean diagonal of P, actual per-task channel counts, parameter counts, or FLOPs for any row. Since the abstract promises architectures that 'effectively make trade-offs between task resource requirements and final performance,' the paper must report both axes of the trade-off. Please add a resource-use column (measured on the actual masks used for evaluation) to Table 2 and, if possible, show the accuracy-resource Pareto plot for the ES runs at different weight-decay values.
- [3.1, Eqs. (1)-(4)] The conversion from the constraint matrix ~P to a binary mask M is asserted but never specified. The paper says only that a mask 'can quickly be derived' with median error under one percent. This is load-bearing for two reasons: the final trained architectures are masks rather than ~P matrices, so an approximate conversion could change both the deployed channel use and the achieved accuracy; and the resource metric (mean diagonal of P) is defined on P, so the reader cannot tell whether the reported efficiency corresponds to the evaluated mask. Please provide the algorithm (e.g., rounding or a greedy/optimization construction), report the distribution of the approximation error, and confirm that the resource numbers in the final evaluation are measured on the actual deployed masks.
- [5.1, Table 1] The evidence for distillation as a reliable surrogate is mixed. Per-task correlations range from 0.477 (daimlerpedcls) and 0.478 (dtd) to 0.954 (cifar100); for omniglot distillation (0.801) is worse than 10k-iteration training (0.906), and for svhn it is worse than 10k-iteration training (0.676 vs 0.761). Because the ES search selects parameterizations using distillation scores, the validity of the search depends on these correlations for the tasks being optimized. Please report the pooled correlation across all tasks, per-task scatter/confidence intervals, and how the ES objective aggregates task-level distillation scores (e.g., total score or per-task average).
- [5.2, Figure 5 (right)] The ES comparison lacks the resource-use values needed to interpret the 'wd 1e-3' result. The text says weight decay controls resource use and the right panel overlays ES samples on the random-sampling distribution, but no final mean-diagonal value (or range) is reported for the model in Table 2. Without this, the reader cannot tell how much channel use was actually saved relative to share-all or share-half, and 'adjusting the weight decay penalty controls resource use' remains untested quantitatively. Please include the wd values and the resulting resource-use statistics for the ES trajectories.
minor comments (4)
- [3.1] In the sentence 'no off-diagonal element Mij should be greater than the corresponding diagonal elements Mii and Mjj,' the indices refer to the matrix P, not to M; as written M is C×N and has no Mij entry for a task pair. Please fix the notation.
- [5.1] Please define how 'distillation accuracy' is measured; the text describes an MSE loss on intermediate features, but Figure 4 and Table 1 presumably use classification accuracy after replacing the pretrained layers with the distilled ones, and this should be stated explicitly.
- [5] The text says 'all scores reported in the paper are averaged across multiple runs,' but it later specifies different numbers of trials for different experiments (e.g., 3 trials for random sampling and distillation, 5 trials for Table 2). Please make the trial counts explicit for each table and figure.
- [3] The claim that applying masks at every other layer ensures that two half-channel tasks use mutually exclusive weights depends on the network's connectivity; please add a short formal explanation or diagram to make this precise.
Circularity Check
No significant circularity: the search parameterization is not fitted to final results, the distillation proxy is validated on held-out random samples, and final architectures are independently trained to convergence.
full rationale
The paper's derivation chain is self-contained. The search space is a parameterization P of mask statistics (Section 3.1), and the resource metric (mean diagonal of P) is an explicit optimization target: the ES objective adds an L2 penalty on the diagonal (Section 4.1), so any reduction in channel use is a designed objective, not a fitted claim. The distillation proxy is validated against converged full training on randomly sampled parameterizations before being used to guide search (Section 5.1, Table 1), and the final Table 2 accuracies are obtained by independently training selected parameterizations to convergence, not by re-using distillation scores. Self-citations ([11], [16]) appear only in related-work or training-curriculum context and are not load-bearing for the central claim. The paper under-reports the actual channel-use numbers for the baselines and ES result, so the resource trade-off is incompletely evidenced, and the mask-derivation step in Section 3.1 is described without an algorithm; these are completeness/correctness concerns, not circular reasoning.
Assumptions & free parameters
free parameters (3)
- learning rate =
0.05 (full training), 1 (distillation), 0.1 (ES)
- curriculum temperature =
not reported numeric value
- ES weight decay =
1e-3
assumptions (3)
- domain assumption A pretrained ImageNet ResNet provides features that transfer to all nine Decathlon tasks, so only the last third of the network needs partitioning.
- ad hoc to paper The matrix P (diagonal = per-task channel use, off-diagonal = pairwise sharing) is a sufficient statistic for predicting architecture performance.
- ad hoc to paper A binary mask M can always be derived from a feasible ~P with negligible error.
Cite this review
Pith. "Pith review of Feature Partitioning for Efficient Multi-Task Architectures." pith.science (2026). https://pith.science/paper/EHW76FQ4
@misc{pith2026190804339,
author = {Pith},
title = {Pith review of: Feature Partitioning for Efficient Multi-Task Architectures},
year = {2026},
howpublished = {\url{https://pith.science/paper/EHW76FQ4}},
note = {Machine review of arXiv:1908.04339}
}
read the original abstract
Multi-task learning holds the promise of less data, parameters, and time than training of separate models. We propose a method to automatically search over multi-task architectures while taking resource constraints into consideration. We propose a search space that compactly represents different parameter sharing strategies. This provides more effective coverage and sampling of the space of multi-task architectures. We also present a method for quick evaluation of different architectures by using feature distillation. Together these contributions allow us to quickly optimize for efficient multi-task models. We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify multi-task architectures that effectively make trade-offs between task resource requirements while achieving a high level of final performance.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Learning in Deep Networks under Dale's Constraint
An on-off two-channel network with fixed-sign synapses and local Hebbian learning is claimed to recover backpropagation exactly under symmetric weights and to beat comparable vanilla networks on Tiny ImageNet.
Reference graph
Works this paper leans on
- [1]
-
[2]
S. L. Beaulieu, S. Kriegman, and J. C. Bongard. Combat- ing catastrophic forgetting with developmental compression. arXiv preprint arXiv:1804.04286, 2018
work page Pith review arXiv 2018
- [3]
-
[4]
R. Caruana. Multitask learning. Machine learning, 28(1):41– 75, 1997
work page 1997
-
[5]
Unifying and Merging Well-trained Deep Neural Networks for Inference Stage
Y .-M. Chou, Y .-M. Chan, J.-H. Lee, C.-Y . Chiu, and C.-S. Chen. Unifying and merging well-trained deep neural net- works for inference stage. arXiv preprint arXiv:1805.04980, 2018
work page Pith review arXiv 2018
-
[6]
B. Deng, J. Yan, and D. Lin. Peephole: Predicting network performance before training. arXiv preprint arXiv:1712.03351, 2017
arXiv 2017
-
[7]
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunning, et al. Im- pala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018
arXiv 2018
-
[8]
C. Fernando, D. Banarse, C. Blundell, Y . Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017
arXiv 2017
Show all 39 references
-
[9]
R. M. French. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4):128–135, 1999
1999
-
[10]
X. He, Z. Zhou, and L. Thiele. Multi-task zipping via layer- wise neuron sharing. arXiv preprint arXiv:1805.09791, 2018
2018 arXiv
-
[11]
Jiang, Z
L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei. Men- tornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018
2018
-
[12]
Kaiser, A
L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, and J. Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017
2017 arXiv
-
[13]
Kumar, G
M. Kumar, G. E. Dahl, V . Vasudevan, and M. Norouzi. Par- allel architecture and hyperparameter search via successive halving and classification. arXiv preprint arXiv:1805.10255, 2018
2018 arXiv
-
[14]
Li and A
L. Li and A. Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv:1902.07638, 2019
1902 arXiv
-
[15]
Liang, E
J. Liang, E. Meyerson, and R. Miikkulainen. Evolutionary ar- chitecture search for deep multitask networks. arXiv preprint arXiv:1803.03745, 2018
2018 arXiv
-
[16]
C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. arXiv preprint arXiv:1712.00559, 2017
2017 arXiv
-
[17]
H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representations for efficient architecture search. arXiv preprint arXiv:1711.00436, 2017
2017 arXiv
-
[18]
S. Liu, E. Johns, and A. J. Davison. End-to-end multi-task learning with attention. arXiv preprint arXiv:1803.10704 , 2018
2018 arXiv
-
[19]
Mallya and S
A. Mallya and S. Lazebnik. Piggyback: Adding multiple tasks to a single, fixed network by learning to mask. arXiv preprint arXiv:1801.06519, 2018
2018 arXiv
-
[20]
Mania, A
H. Mania, A. Guy, and B. Recht. Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055, 2018
2018 arXiv
-
[21]
Meyerson and R
E. Meyerson and R. Miikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint arXiv:1711.00108, 2017
2017 arXiv
-
[22]
Misra, A
I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross- stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 3994–4003, 2016
2016
-
[23]
H. Pham, M. Y . Guan, B. Zoph, Q. V . Le, and J. Dean. Effi- cient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018
2018 arXiv
-
[24]
E. Real, A. Aggarwal, Y . Huang, and Q. V . Le. Regular- ized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018
2018 arXiv
-
[25]
Rebuffi, H
S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pages 506–516, 2017
2017
-
[26]
Rebuffi, H
S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Efficient parametriza- tion of multi-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8119–8127, 2018
2018
-
[27]
Rosenbaum, T
C. Rosenbaum, T. Klinger, and M. Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learn- ing. arXiv preprint arXiv:1711.01239, 2017
2017 arXiv
-
[28]
Rosenfeld and J
A. Rosenfeld and J. K. Tsotsos. Incremental learning through deep adaptation. arXiv preprint arXiv:1705.04228, 2017
2017 arXiv
-
[29]
S. Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017
2017 arXiv
-
[30]
Ruder, J
S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard. Learning what to share between loosely related tasks. arXiv preprint arXiv:1705.08142, 2017
2017 arXiv
-
[31]
A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Des- jardins, J. Kirkpatrick, R. Pascanu, V . Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015. 9
2015 arXiv
-
[32]
Sharma, A
S. Sharma, A. Jha, P. Hegde, and B. Ravindran. Learn- ing to multi-task by active sampling. arXiv preprint arXiv:1702.06053, 2017
2017 arXiv
-
[33]
Wierstra, T
D. Wierstra, T. Schaul, J. Peters, and J. Schmidhuber. Natu- ral evolution strategies. In Evolutionary Computation, 2008. CEC 2008.(IEEE World Congress on Computational Intelli- gence). IEEE Congress on, pages 3381–3387. IEEE, 2008
2008
-
[34]
J. Yim, D. Joo, J. Bae, and J. Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In The IEEE Conference on Computer Vision and Pattern Recognition, volume 2, 2017
2017
-
[35]
J. Yoon, E. Yang, J. Lee, and S. J. Hwang. Lifelong learn- ing with dynamically expandable networks. In International Conference on Learning Representations, 2018
2018
-
[36]
A. R. Zamir, A. Sax, W. Shen, L. Guibas, J. Malik, and S. Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3712–3722, 2018
2018
-
[37]
A. Zela, A. Klein, S. Falkner, and F. Hutter. Towards auto- mated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.06906 , 2018
2018 arXiv
-
[38]
Zhang and Q
Y . Zhang and Q. Yang. A survey on multi-task learning.arXiv preprint arXiv:1707.08114, 2017
2017 arXiv
-
[39]
Zoph and Q
B. Zoph and Q. V . Le. Neural architecture search with rein- forcement learning. CoRR, abs/1611.01578, 2016. 10
2016 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.