Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Feature Partitioning for Efficient Multi-Task Architectures

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Searching over channel-sharing fractions, guided by a one-minute distillation proxy, finds multi-task networks that beat fixed sharing strategies while using fewer channels per task.

desk verdict A useful search-space parameterization and a sensible distillation proxy, but the headline resource trade-off is unsupported because Table 2 never reports channel use. read the letter →

arxiv 1908.04339 v1 pith:EHW76FQ4 submitted 2019-08-12 cs.LG cs.CVstat.ML

classification cs.LGcs.CVstat.ML
keywords multi-tasklearningneuralarchitecturesearchfeaturepartitioningchannelmaskingknowledgedistillationevolutionarystrategiesVisualDecathlonresource-efficientarchitectures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make multi-task architecture search practical by changing the search space from whole layers to individual feature channels. It claims that a compact matrix of per-task channel usage and pairwise sharing, together with a one-minute feature-distillation evaluation, lets an evolutionary strategy find architectures that make the right trade-offs between accuracy and per-task compute. On nine Visual Decathlon tasks, the search finds a partitioning that beats independent, share-half, and share-all baselines in overall validation performance while reducing the average number of channels each task uses. If this holds, multi-task model design becomes an automatic, resource-aware optimization rather than a hand-designed sharing recipe.

What carries the argument

The load-bearing object is the normalized channel-overlap matrix $P = \frac{1}{C}M^{\top}M$, where $M \in \{0,1\}^{C \times N}$ is a binary matrix of channel masks for the $N$ tasks; diagonal entries of $P$ are the fraction of channels each task uses and off-diagonal entries are the fraction of channels shared between task pairs. Because raw mask matrices are redundant under channel permutation, the paper searches over $P$ and remaps raw values into feasible overlap constraints, then samples a mask that realizes the target $P$. The second mechanism is the feature-distillation proxy: shared layers under a candidate mask are trained for about 3000 iterations to reproduce the features of single-task teachers at a target depth, and the resulting accuracy is used to score the candidate without training the full model to convergence.

What would settle it

Take a held-out set of fifty random partitionings, score each with the one-minute distillation proxy, then train each to convergence; if the proxy ranking and the final ranking reverse substantially, or if the top proxy candidates systematically lose to lower-ranked ones after full training, the search's guidance fails. A concrete place to look is daimlerpedcls, where the paper already reports the lowest distillation-to-final correlation.

Watch

Extended reading notes

Core claim

The paper's central claim is that the right search space for multi-task architectures is not whole layers but individual feature channels: every task gets a binary mask selecting a subset of channels, and the search operates on $P = \frac{1}{C}M^{\top}M$, the normalized overlap matrix whose diagonal gives each task's channel fraction and whose off-diagonal entries give pairwise sharing. It claims this parameterization is low-dimensional, interpretable, independent of layer width, and mappable back to feasible masks with median error under one percent. On the Visual Decathlon benchmark, the paper reports that random sampling over $P$ maps the accuracy-versus-resource frontier, and that evolutionary strategies with a diagonal weight-decay term push beyond fixed baselines: the found partitioning both reduces average per-task channel use and raises the overall validation score relative to independent, share-half, and share-all schemes. The search is made practical by a feature-distillation proxy that evaluates a candidate in about one minute, with correlations to final validation accuracy that the paper reports as mostly higher than short-training baselines.

Load-bearing premise

The load-bearing premise is that the one-minute feature-distillation score ranks multi-task architectures in the same order as full training to convergence, even though the reported correlations vary by task and are as low as about 0.48 for two of the nine tasks.

Editorial extensions

If this is right

  • If the parameterization works, multi-task architecture search becomes a continuous optimization problem with a single resource knob: diagonal weight decay moves the solution along the accuracy-versus-compute frontier.
  • The one-minute distillation evaluation means thousands of candidate partitionings can be ranked in a few hours on a single GPU, making resource-aware search practical where full training is prohibitive.
  • Because $P$ does not depend on channel count, a partitioning strategy found for one layer width transfers to other widths and depths without re-searching.
  • At inference time, masks prune unused channels directly, so a task that needs only a subset of the shared features runs at a fraction of the compute while still benefiting from joint training.
  • If these results hold, fixed sharing recipes such as sharing all layers or splitting channels evenly are dominated by automatic search and should be replaced by task-specific channel budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The experiments freeze and share the first two-thirds of an ImageNet-pretrained network, so the conclusion that partitioning matters only in the last block likely depends on that strong initialization; a testable extension is repeating the search from random initialization to see whether earlier blocks need partitioning too.
  • The proxy is validated by overall correlation on random architectures, not by how often the top-ranked candidate actually wins after full training; a practical extension is a top-K selection procedure that trains the top few proxy candidates to convergence and compares their final ordering.
  • Because $P$ records only pairwise overlap, the search cannot represent three-task interactions such as a channel block shared exclusively by a triplet; a natural extension is a higher-order sharing tensor or a hierarchical partition of channels.
  • The resource metric is average per-task channel use in the last block; an extension would optimize for FLOPs, peak memory, or heterogeneous per-task budgets, which the current diagonal penalty only approximates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a channel-level feature-partitioning approach to multi-task architecture search. Instead of searching directly over binary channel masks, the authors introduce a continuous matrix P whose diagonal captures per-task channel use and whose off-diagonal entries capture pairwise sharing, together with a constraint-mapping procedure to ensure feasibility. They evaluate architectures with a fast feature-distillation proxy and search by random sampling and by evolutionary strategies with an L2 penalty on the diagonal of P. On Visual Decathlon (nine tasks), they compare full-training validation accuracy of independent, share-half, share-all, and ES-discovered partitions, and claim that the ES result reduces average channel use while improving overall accuracy. The paper also reports per-task correlations between distillation accuracy and final validation accuracy.

Significance. If the resource-efficiency claim can be substantiated, this is a useful contribution: the P-matrix parameterization is a compact and interpretable search space for multi-task sharing at channel granularity, and the distillation proxy is a clever way to make architecture evaluation cheap. The empirical study is careful in its use of multi-run averages and standard errors, and the analysis in Figure 5 (random-sample distributions and fixed-diagonal sharing sensitivity) gives real insight into the role of sharing. The strengths are the continuous parameterization, the fast proxy, and the clear presentation of the search-space design. However, the central trade-off claim is currently not verifiable because resource use is never reported numerically, and the mask-construction procedure is asserted rather than specified. These omissions limit the current significance of the empirical results.

major comments (4)
  1. [5.2, Table 2] The central resource-accuracy trade-off claim is not verifiable as written. Table 2 reports only validation accuracies for the four configurations, and the text states that the ES-found parameterization 'both reduces average channel use and achieves better performance overall,' but no column or figure reports the mean diagonal of P, actual per-task channel counts, parameter counts, or FLOPs for any row. Since the abstract promises architectures that 'effectively make trade-offs between task resource requirements and final performance,' the paper must report both axes of the trade-off. Please add a resource-use column (measured on the actual masks used for evaluation) to Table 2 and, if possible, show the accuracy-resource Pareto plot for the ES runs at different weight-decay values.
  2. [3.1, Eqs. (1)-(4)] The conversion from the constraint matrix ~P to a binary mask M is asserted but never specified. The paper says only that a mask 'can quickly be derived' with median error under one percent. This is load-bearing for two reasons: the final trained architectures are masks rather than ~P matrices, so an approximate conversion could change both the deployed channel use and the achieved accuracy; and the resource metric (mean diagonal of P) is defined on P, so the reader cannot tell whether the reported efficiency corresponds to the evaluated mask. Please provide the algorithm (e.g., rounding or a greedy/optimization construction), report the distribution of the approximation error, and confirm that the resource numbers in the final evaluation are measured on the actual deployed masks.
  3. [5.1, Table 1] The evidence for distillation as a reliable surrogate is mixed. Per-task correlations range from 0.477 (daimlerpedcls) and 0.478 (dtd) to 0.954 (cifar100); for omniglot distillation (0.801) is worse than 10k-iteration training (0.906), and for svhn it is worse than 10k-iteration training (0.676 vs 0.761). Because the ES search selects parameterizations using distillation scores, the validity of the search depends on these correlations for the tasks being optimized. Please report the pooled correlation across all tasks, per-task scatter/confidence intervals, and how the ES objective aggregates task-level distillation scores (e.g., total score or per-task average).
  4. [5.2, Figure 5 (right)] The ES comparison lacks the resource-use values needed to interpret the 'wd 1e-3' result. The text says weight decay controls resource use and the right panel overlays ES samples on the random-sampling distribution, but no final mean-diagonal value (or range) is reported for the model in Table 2. Without this, the reader cannot tell how much channel use was actually saved relative to share-all or share-half, and 'adjusting the weight decay penalty controls resource use' remains untested quantitatively. Please include the wd values and the resulting resource-use statistics for the ES trajectories.
minor comments (4)
  1. [3.1] In the sentence 'no off-diagonal element Mij should be greater than the corresponding diagonal elements Mii and Mjj,' the indices refer to the matrix P, not to M; as written M is C×N and has no Mij entry for a task pair. Please fix the notation.
  2. [5.1] Please define how 'distillation accuracy' is measured; the text describes an MSE loss on intermediate features, but Figure 4 and Table 1 presumably use classification accuracy after replacing the pretrained layers with the distilled ones, and this should be stated explicitly.
  3. [5] The text says 'all scores reported in the paper are averaged across multiple runs,' but it later specifies different numbers of trials for different experiments (e.g., 3 trials for random sampling and distillation, 5 trials for Table 2). Please make the trial counts explicit for each table and figure.
  4. [3] The claim that applying masks at every other layer ensures that two half-channel tasks use mutually exclusive weights depends on the network's connectivity; please add a short formal explanation or diagram to make this precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the search parameterization is not fitted to final results, the distillation proxy is validated on held-out random samples, and final architectures are independently trained to convergence.

full rationale

The paper's derivation chain is self-contained. The search space is a parameterization P of mask statistics (Section 3.1), and the resource metric (mean diagonal of P) is an explicit optimization target: the ES objective adds an L2 penalty on the diagonal (Section 4.1), so any reduction in channel use is a designed objective, not a fitted claim. The distillation proxy is validated against converged full training on randomly sampled parameterizations before being used to guide search (Section 5.1, Table 1), and the final Table 2 accuracies are obtained by independently training selected parameterizations to convergence, not by re-using distillation scores. Self-citations ([11], [16]) appear only in related-work or training-curriculum context and are not load-bearing for the central claim. The paper under-reports the actual channel-use numbers for the baselines and ES result, so the resource trade-off is incompletely evidenced, and the mask-derivation step in Section 3.1 is described without an algorithm; these are completeness/correctness concerns, not circular reasoning.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method relies on three main assumptions: transferability of ImageNet features, sufficiency of the P matrix, and the feasibility of mask construction. The first is common in prior work; the latter two are specific to this paper and are not proven formally. The free parameters are standard hyperparameters, though the ES weight decay directly controls the resource-efficiency trade-off.

free parameters (3)
  • learning rate = 0.05 (full training), 1 (distillation), 0.1 (ES)
    Chosen by grid search or fixed; affects convergence and final performance.
  • curriculum temperature = not reported numeric value
    Controls task sampling weights; tuned via grid search.
  • ES weight decay = 1e-3
    Regularizes diagonal of P to reduce per-task channel use; hand-chosen to target a resource trade-off.
assumptions (3)
  • domain assumption A pretrained ImageNet ResNet provides features that transfer to all nine Decathlon tasks, so only the last third of the network needs partitioning.
    All experiments start from a pretrained ImageNet model; the paper freezes the first two-thirds based on a correlation analysis.
  • ad hoc to paper The matrix P (diagonal = per-task channel use, off-diagonal = pairwise sharing) is a sufficient statistic for predicting architecture performance.
    The search space is defined by P, and the paper assumes that these pairwise statistics capture the relevant differences between architectures.
  • ad hoc to paper A binary mask M can always be derived from a feasible ~P with negligible error.
    Section 3.1 states this as an empirical finding without providing the derivation algorithm.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Feature Partitioning for Efficient Multi-Task Architectures." pith.science (2026). https://pith.science/paper/EHW76FQ4

@misc{pith2026190804339,
  author       = {Pith},
  title        = {Pith review of: Feature Partitioning for Efficient Multi-Task Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EHW76FQ4}},
  note         = {Machine review of arXiv:1908.04339}
}
read the original abstract

Multi-task learning holds the promise of less data, parameters, and time than training of separate models. We propose a method to automatically search over multi-task architectures while taking resource constraints into consideration. We propose a search space that compactly represents different parameter sharing strategies. This provides more effective coverage and sampling of the space of multi-task architectures. We also present a method for quick evaluation of different architectures by using feature distillation. Together these contributions allow us to quickly optimize for efficient multi-task models. We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify multi-task architectures that effectively make trade-offs between task resource requirements while achieving a high level of final performance.

Figures

Figures reproduced from arXiv: 1908.04339 by the authors.

Figure 1
Figure 1. Feature partitioning can be used to control how much network capacity is used by tasks, and how much sharing is done across [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Multi-task distillation: At a given step, a teacher network pretrained on a single task is chosen from the available tasks. Features at [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Measuring correlation of feature activations across three blocks of task-specific ResNet models given a shared input image. Even [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Correlation between distillation accuracy and final validation [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: (left) Distribution of performance of random partitioning strategies; (middle) Distribution of performance when fixing channel use [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Distillation performance per task as a function of parameters allocated to that task. Color indicates the average parameters used [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning in Deep Networks under Dale's Constraint

    cs.AI 2026-08 reject novelty 7.0 of 10

    An on-off two-channel network with fixed-sign synapses and local Hebbian learning is claimed to recover backpropagation exactly under symmetric weights and to beat comparable vanilla networks on Tiny ImageNet.

Reference graph

Works this paper leans on

39 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Baker, O

    B. Baker, O. Gupta, R. Raskar, and N. Naik. Accelerating neu- ral architecture search using performance prediction. arXiv preprint arXiv:1705.10823, 2017

  2. [2]

    S. L. Beaulieu, S. Kriegman, and J. C. Bongard. Combat- ing catastrophic forgetting with developmental compression. arXiv preprint arXiv:1804.04286, 2018

  3. [3]

    Brock, T

    A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05344, 2017

  4. [4]

    R. Caruana. Multitask learning. Machine learning, 28(1):41– 75, 1997

  5. [5]

    Unifying and Merging Well-trained Deep Neural Networks for Inference Stage

    Y .-M. Chou, Y .-M. Chan, J.-H. Lee, C.-Y . Chiu, and C.-S. Chen. Unifying and merging well-trained deep neural net- works for inference stage. arXiv preprint arXiv:1805.04980, 2018

  6. [6]

    B. Deng, J. Yan, and D. Lin. Peephole: Predicting network performance before training. arXiv preprint arXiv:1712.03351, 2017

  7. [7]

    Espeholt, H

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunning, et al. Im- pala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018

  8. [8]

    Fernando, D

    C. Fernando, D. Banarse, C. Blundell, Y . Zwols, D. Ha, A. A. Rusu, A. Pritzel, and D. Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734, 2017

Show all 39 references
  1. [9]

    R. M. French. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4):128–135, 1999

  2. [10]

    X. He, Z. Zhou, and L. Thiele. Multi-task zipping via layer- wise neuron sharing. arXiv preprint arXiv:1805.09791, 2018

  3. [11]

    Jiang, Z

    L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei. Men- tornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018

  4. [12]

    Kaiser, A

    L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, and J. Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017

  5. [13]

    Kumar, G

    M. Kumar, G. E. Dahl, V . Vasudevan, and M. Norouzi. Par- allel architecture and hyperparameter search via successive halving and classification. arXiv preprint arXiv:1805.10255, 2018

  6. [14]

    Li and A

    L. Li and A. Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv:1902.07638, 2019

  7. [15]

    Liang, E

    J. Liang, E. Meyerson, and R. Miikkulainen. Evolutionary ar- chitecture search for deep multitask networks. arXiv preprint arXiv:1803.03745, 2018

  8. [16]

    C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. arXiv preprint arXiv:1712.00559, 2017

  9. [17]

    H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representations for efficient architecture search. arXiv preprint arXiv:1711.00436, 2017

  10. [18]

    S. Liu, E. Johns, and A. J. Davison. End-to-end multi-task learning with attention. arXiv preprint arXiv:1803.10704 , 2018

  11. [19]

    Mallya and S

    A. Mallya and S. Lazebnik. Piggyback: Adding multiple tasks to a single, fixed network by learning to mask. arXiv preprint arXiv:1801.06519, 2018

  12. [20]

    Mania, A

    H. Mania, A. Guy, and B. Recht. Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055, 2018

  13. [21]

    Meyerson and R

    E. Meyerson and R. Miikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint arXiv:1711.00108, 2017

  14. [22]

    Misra, A

    I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross- stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 3994–4003, 2016

  15. [23]

    H. Pham, M. Y . Guan, B. Zoph, Q. V . Le, and J. Dean. Effi- cient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018

  16. [24]

    E. Real, A. Aggarwal, Y . Huang, and Q. V . Le. Regular- ized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018

  17. [25]

    Rebuffi, H

    S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pages 506–516, 2017

  18. [26]

    Rebuffi, H

    S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Efficient parametriza- tion of multi-domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8119–8127, 2018

  19. [27]

    Rosenbaum, T

    C. Rosenbaum, T. Klinger, and M. Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learn- ing. arXiv preprint arXiv:1711.01239, 2017

  20. [28]

    Rosenfeld and J

    A. Rosenfeld and J. K. Tsotsos. Incremental learning through deep adaptation. arXiv preprint arXiv:1705.04228, 2017

  21. [29]

    S. Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017

  22. [30]

    Ruder, J

    S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard. Learning what to share between loosely related tasks. arXiv preprint arXiv:1705.08142, 2017

  23. [31]

    A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Des- jardins, J. Kirkpatrick, R. Pascanu, V . Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015. 9

  24. [32]

    Sharma, A

    S. Sharma, A. Jha, P. Hegde, and B. Ravindran. Learn- ing to multi-task by active sampling. arXiv preprint arXiv:1702.06053, 2017

  25. [33]

    Wierstra, T

    D. Wierstra, T. Schaul, J. Peters, and J. Schmidhuber. Natu- ral evolution strategies. In Evolutionary Computation, 2008. CEC 2008.(IEEE World Congress on Computational Intelli- gence). IEEE Congress on, pages 3381–3387. IEEE, 2008

  26. [34]

    J. Yim, D. Joo, J. Bae, and J. Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In The IEEE Conference on Computer Vision and Pattern Recognition, volume 2, 2017

  27. [35]

    J. Yoon, E. Yang, J. Lee, and S. J. Hwang. Lifelong learn- ing with dynamically expandable networks. In International Conference on Learning Representations, 2018

  28. [36]

    A. R. Zamir, A. Sax, W. Shen, L. Guibas, J. Malik, and S. Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3712–3722, 2018

  29. [37]

    A. Zela, A. Klein, S. Falkner, and F. Hutter. Towards auto- mated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv preprint arXiv:1807.06906 , 2018

  30. [38]

    Zhang and Q

    Y . Zhang and Q. Yang. A survey on multi-task learning.arXiv preprint arXiv:1707.08114, 2017

  31. [39]

    Zoph and Q

    B. Zoph and Q. V . Le. Neural architecture search with rein- forcement learning. CoRR, abs/1611.01578, 2016. 10

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.