{"id":"b561e025-63fa-4504-959e-3e3f10fd0940","arxiv_id":"1908.04339","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A channel-level feature-partitioning search space with a distillation proxy lets multi-task architecture search quickly find efficient sharing patterns.","lead":"The authors search for efficient multi-task neural networks by partitioning feature channels among tasks and optimizing the sharing pattern with a compact parameterization. They show that a quick distillation-based evaluator can find architectures that beat simple fixed-sharing baselines on nine Visual Decathlon tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed trade-off lacks its resource axis: Table 2 reports only validation accuracy, never average channel use or FLOPs, so 'reduces average channel use and achieves better performance' is unsupported.","rationale":"The reader's conditional verdict is appropriate, and the reader correctly flags missing details. However, I see the most load-bearing gap differently. The reader's weakest assumption concerns whether the distillation proxy accurately ranks architectures. That is a real concern, and Table 1 shows per-task correlations that are sometimes low (daimlerpedcls 0.477, dtd 0.478). But even if the proxy were perfectly faithful, the paper still would not demonstrate the central trade-off claim because it never reports the resource usage of the final compared architectures. The resource axis is defined as average channel use from the diagonal of P, so it is computable from the final masks; its absence means the reader cannot check whether the ES result actually uses fewer resources than the baselines. Similarly, the mask-conversion algorithm is part of the resource story: without knowing how P becomes M, one cannot be confident that the optimized P describes the deployed mask. Both issues are addressable, so I would keep the verdict conditional rather than accepting or rejecting, and I would ask the authors to supply the resource numbers and the conversion procedure or code.","tokens_in":10964,"tokens_out":5376,"duration_ms":63246,"concrete_test":"For each row of Table 2 (independent, share half, share all, es wd=1e-3), compute and report the mean diagonal of P from the final trained masks, equivalent to the average fraction of channels used per task, plus per-task channel counts and, if feasible, FLOPs on the last partitioned block. Independently recompute 1/C M^T M from the derived masks to check whether the claimed P targets are satisfied. If the ES architecture's average channel use is not strictly below share-all at comparable or better accuracy, the trade-off claim fails; if the mask-conversion error exceeds a few percent, the optimized P is not the architecture actually evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that the search identifies architectures that effectively trade off task resource requirements against final performance. To substantiate that claim, the paper must report both axes of the trade-off. Table 2 gives validation accuracies for independent, share-half, share-all, and ES(wd 1e-3), but it contains no column reporting average channel use, per-task channel counts, or any other resource measurement. The text asserts that the evolutionary search 'reduces average channel use' relative to baselines, but no number is supplied anywhere, so the reader cannot verify that the ES result is actually more efficient. This is not a cosmetic omission: if the ES architecture uses the same channel budget as share-all, then Table 2 merely shows a slightly better full-sharing model, and the resource trade-off claim collapses. The concern is compounded by the unpublished mask-conversion step in Section 3.1. The paper defines the resource metric as the mean diagonal of P, then says only that a mask M can be 'quickly derived' that satisfies ~P with median error under one percent, without describing the algorithm. If the conversion is approximate, the actual channel use of the deployed mask may differ from the optimized P, and the resource numbers implied by P may not describe the architecture that was trained and evaluated. Thus the central claim rests on both an unstated resource measurement and an unpublished construction. This remains load-bearing even if the distillation proxy ranks architectures perfectly: a perfect proxy can identify high-accuracy architectures, but it cannot establish low resource use unless resource use is actually measured and reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a channel-level feature-partitioning approach to multi-task architecture search. Instead of searching directly over binary channel masks, the authors introduce a continuous matrix P whose diagonal captures per-task channel use and whose off-diagonal entries capture pairwise sharing, together with a constraint-mapping procedure to ensure feasibility. They evaluate architectures with a fast feature-distillation proxy and search by random sampling and by evolutionary strategies with an L2 penalty on the diagonal of P. On Visual Decathlon (nine tasks), they compare full-training validation accuracy of independent, share-half, share-all, and ES-discovered partitions, and claim that the ES result reduces average channel use while improving overall accuracy. The paper also reports per-task correlations between distillation accuracy and final validation accuracy.","tokens_in":11217,"tokens_out":5695,"duration_ms":57410,"significance":"If the resource-efficiency claim can be substantiated, this is a useful contribution: the P-matrix parameterization is a compact and interpretable search space for multi-task sharing at channel granularity, and the distillation proxy is a clever way to make architecture evaluation cheap. The empirical study is careful in its use of multi-run averages and standard errors, and the analysis in Figure 5 (random-sample distributions and fixed-diagonal sharing sensitivity) gives real insight into the role of sharing. The strengths are the continuous parameterization, the fast proxy, and the clear presentation of the search-space design. However, the central trade-off claim is currently not verifiable because resource use is never reported numerically, and the mask-construction procedure is asserted rather than specified. These omissions limit the current significance of the empirical results.","major_comments":[{"comment":"The central resource-accuracy trade-off claim is not verifiable as written. Table 2 reports only validation accuracies for the four configurations, and the text states that the ES-found parameterization 'both reduces average channel use and achieves better performance overall,' but no column or figure reports the mean diagonal of P, actual per-task channel counts, parameter counts, or FLOPs for any row. Since the abstract promises architectures that 'effectively make trade-offs between task resource requirements and final performance,' the paper must report both axes of the trade-off. Please add a resource-use column (measured on the actual masks used for evaluation) to Table 2 and, if possible, show the accuracy-resource Pareto plot for the ES runs at different weight-decay values.","section":"5.2, Table 2"},{"comment":"The conversion from the constraint matrix ~P to a binary mask M is asserted but never specified. The paper says only that a mask 'can quickly be derived' with median error under one percent. This is load-bearing for two reasons: the final trained architectures are masks rather than ~P matrices, so an approximate conversion could change both the deployed channel use and the achieved accuracy; and the resource metric (mean diagonal of P) is defined on P, so the reader cannot tell whether the reported efficiency corresponds to the evaluated mask. Please provide the algorithm (e.g., rounding or a greedy/optimization construction), report the distribution of the approximation error, and confirm that the resource numbers in the final evaluation are measured on the actual deployed masks.","section":"3.1, Eqs. (1)-(4)"},{"comment":"The evidence for distillation as a reliable surrogate is mixed. Per-task correlations range from 0.477 (daimlerpedcls) and 0.478 (dtd) to 0.954 (cifar100); for omniglot distillation (0.801) is worse than 10k-iteration training (0.906), and for svhn it is worse than 10k-iteration training (0.676 vs 0.761). Because the ES search selects parameterizations using distillation scores, the validity of the search depends on these correlations for the tasks being optimized. Please report the pooled correlation across all tasks, per-task scatter/confidence intervals, and how the ES objective aggregates task-level distillation scores (e.g., total score or per-task average).","section":"5.1, Table 1"},{"comment":"The ES comparison lacks the resource-use values needed to interpret the 'wd 1e-3' result. The text says weight decay controls resource use and the right panel overlays ES samples on the random-sampling distribution, but no final mean-diagonal value (or range) is reported for the model in Table 2. Without this, the reader cannot tell how much channel use was actually saved relative to share-all or share-half, and 'adjusting the weight decay penalty controls resource use' remains untested quantitatively. Please include the wd values and the resulting resource-use statistics for the ES trajectories.","section":"5.2, Figure 5 (right)"}],"minor_comments":[{"comment":"In the sentence 'no off-diagonal element Mij should be greater than the corresponding diagonal elements Mii and Mjj,' the indices refer to the matrix P, not to M; as written M is C×N and has no Mij entry for a task pair. Please fix the notation.","section":"3.1"},{"comment":"Please define how 'distillation accuracy' is measured; the text describes an MSE loss on intermediate features, but Figure 4 and Table 1 presumably use classification accuracy after replacing the pretrained layers with the distilled ones, and this should be stated explicitly.","section":"5.1"},{"comment":"The text says 'all scores reported in the paper are averaged across multiple runs,' but it later specifies different numbers of trials for different experiments (e.g., 3 trials for random sampling and distillation, 5 trials for Table 2). Please make the trial counts explicit for each table and figure.","section":"5"},{"comment":"The claim that applying masks at every other layer ensures that two half-channel tasks use mutually exclusive weights depends on the network's connectivity; please add a short formal explanation or diagram to make this precise.","section":"3"}],"recommendation":"major_revision","confidential_remarks":"I found no evidence of circularity: the search space and distillation proxy are validated on held-out random samples before being used for optimization, and the final architectures are independently trained to convergence. The main risk is verifiability: the paper's headline claim concerns a resource-accuracy trade-off, but the resource axis is never quantified, and the mask-conversion step is only sketched. If the authors supply the missing measurements and the algorithm, I would expect the manuscript to become acceptable. I do not see a basis for rejection on the grounds of disagreement with current multi-task NAS methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is solid: instead of routing whole layers, parameterize multi-task sharing at the channel level through a pairwise-overlap matrix P, and use feature distillation as a cheap evaluator. The parameterization is clean, interpretable, and makes the search space continuous enough for ES optimization. The distillation proxy is validated against full training on random samples, and it correlates better than early training on most tasks, which is a genuinely useful finding for the NAS community. The result that sharing matters more when total resource use is low is also informative.\n\nThe soft spots are real, and the stress-test note is right. The abstract's central claim is that the search finds architectures that trade off resource use against performance, but nowhere in the paper is actual resource use reported alongside accuracy. Table 2 gives validation accuracies for independent, share-half, share-all, and ES, with no column for average channels, per-task channel counts, or FLOPs. The text says ES \"reduces average channel use\" but never gives the number. If the ES architecture consumes the same channels as share-all, the trade-off claim collapses and you just have a slightly better full-sharing model. This is not cosmetic; the paper's whole selling point is the efficiency frontier.\n\nSecond, the mask conversion from P to M is asserted but never described. Section 3.1 says a mask can be \"quickly derived\" with median error under one percent, but no algorithm is given. That means the resource numbers implied by P may not match what is actually deployed. The reader cannot verify that the optimized P led to the mask that was trained.\n\nThe baselines are also limited: only simple hand-designed partitions. No comparison to routing networks, cross-stitch, or layer-wise learned sharing. And the distillation proxy is weak for some tasks (daimlerpedcls at 0.477 correlation), though the aggregate trend is convincing.\n\nThat said, the paper is not fundamentally broken. The parameterization and proxy evaluation are useful contributions that can be fixed with transparency. I would send it to peer review, but with major revision: report average channel use for every architecture in Table 2, describe the mask conversion algorithm or ship code, and add at least one stronger learned-sharing baseline.\n\nWorth a serious referee, conditional on those additions.","headline":"A useful search-space parameterization and a sensible distillation proxy, but the headline resource trade-off is unsupported because Table 2 never reports channel use.","tokens_in":11734,"tokens_out":1255,"would_cite":false,"duration_ms":14550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Searching over channel-sharing fractions, guided by a one-minute distillation proxy, finds multi-task networks that beat fixed sharing strategies while using fewer channels per task.","keywords":["multi-task learning","neural architecture search","feature partitioning","channel masking","knowledge distillation","evolutionary strategies","Visual Decathlon","resource-efficient architectures"],"falsifier":"Take a held-out set of fifty random partitionings, score each with the one-minute distillation proxy, then train each to convergence; if the proxy ranking and the final ranking reverse substantially, or if the top proxy candidates systematically lose to lower-ranked ones after full training, the search's guidance fails. A concrete place to look is daimlerpedcls, where the paper already reports the lowest distillation-to-final correlation.","tokens_in":10728,"feed_emoji":"🧩","tokens_out":7335,"duration_ms":66505,"temperature":0.7,"pith_summary":"The paper sets out to make multi-task architecture search practical by changing the search space from whole layers to individual feature channels. It claims that a compact matrix of per-task channel usage and pairwise sharing, together with a one-minute feature-distillation evaluation, lets an evolutionary strategy find architectures that make the right trade-offs between accuracy and per-task compute. On nine Visual Decathlon tasks, the search finds a partitioning that beats independent, share-half, and share-all baselines in overall validation performance while reducing the average number of channels each task uses. If this holds, multi-task model design becomes an automatic, resource-aware optimization rather than a hand-designed sharing recipe.","feed_headline":"Feature-level search beats fixed sharing in multi-task nets","feed_subtitle":"A compact overlap matrix plus fast distillation lets evolution find per-task channel budgets that beat share-all baselines.","key_machinery":"The load-bearing object is the normalized channel-overlap matrix $P = \\frac{1}{C}M^{\\top}M$, where $M \\in \\{0,1\\}^{C \\times N}$ is a binary matrix of channel masks for the $N$ tasks; diagonal entries of $P$ are the fraction of channels each task uses and off-diagonal entries are the fraction of channels shared between task pairs. Because raw mask matrices are redundant under channel permutation, the paper searches over $P$ and remaps raw values into feasible overlap constraints, then samples a mask that realizes the target $P$. The second mechanism is the feature-distillation proxy: shared layers under a candidate mask are trained for about 3000 iterations to reproduce the features of single-task teachers at a target depth, and the resulting accuracy is used to score the candidate without training the full model to convergence.","core_discovery":"The paper's central claim is that the right search space for multi-task architectures is not whole layers but individual feature channels: every task gets a binary mask selecting a subset of channels, and the search operates on $P = \\frac{1}{C}M^{\\top}M$, the normalized overlap matrix whose diagonal gives each task's channel fraction and whose off-diagonal entries give pairwise sharing. It claims this parameterization is low-dimensional, interpretable, independent of layer width, and mappable back to feasible masks with median error under one percent. On the Visual Decathlon benchmark, the paper reports that random sampling over $P$ maps the accuracy-versus-resource frontier, and that evolutionary strategies with a diagonal weight-decay term push beyond fixed baselines: the found partitioning both reduces average per-task channel use and raises the overall validation score relative to independent, share-half, and share-all schemes. The search is made practical by a feature-distillation proxy that evaluates a candidate in about one minute, with correlations to final validation accuracy that the paper reports as mostly higher than short-training baselines.","pith_inferences":["The experiments freeze and share the first two-thirds of an ImageNet-pretrained network, so the conclusion that partitioning matters only in the last block likely depends on that strong initialization; a testable extension is repeating the search from random initialization to see whether earlier blocks need partitioning too.","The proxy is validated by overall correlation on random architectures, not by how often the top-ranked candidate actually wins after full training; a practical extension is a top-K selection procedure that trains the top few proxy candidates to convergence and compares their final ordering.","Because $P$ records only pairwise overlap, the search cannot represent three-task interactions such as a channel block shared exclusively by a triplet; a natural extension is a higher-order sharing tensor or a hierarchical partition of channels.","The resource metric is average per-task channel use in the last block; an extension would optimize for FLOPs, peak memory, or heterogeneous per-task budgets, which the current diagonal penalty only approximates."],"forward_implications":["If the parameterization works, multi-task architecture search becomes a continuous optimization problem with a single resource knob: diagonal weight decay moves the solution along the accuracy-versus-compute frontier.","The one-minute distillation evaluation means thousands of candidate partitionings can be ranked in a few hours on a single GPU, making resource-aware search practical where full training is prohibitive.","Because $P$ does not depend on channel count, a partitioning strategy found for one layer width transfers to other widths and depths without re-searching.","At inference time, masks prune unused channels directly, so a task that needs only a subset of the shared features runs at a fraction of the compute while still benefiting from joint training.","If these results hold, fixed sharing recipes such as sharing all layers or splitting channels evenly are dominated by automatic search and should be replaced by task-specific channel budgets."],"supporting_citations":[{"why":"Supplies the Visual Decathlon benchmark, its train/validation/test splits, and the per-task batch-normalization convention used in all multi-task experiments.","marker":"[25]"},{"why":"Supplies the ImageNet-pretrained ResNet that initializes the shared model and the teacher networks for distillation; the paper freezes its first two blocks.","marker":"[26]"},{"why":"Provides the evolutionary-strategies update rule the search uses to optimize the continuous parameterization P.","marker":"[20]"},{"why":"Establishes random sampling as the strong NAS baseline that the paper compares against evolutionary strategies and uses to map the accuracy-resource frontier.","marker":"[14]"},{"why":"Shows how binary masks over a fixed network can add tasks without changing weights, the masking idea that channel partitioning extends and searches over.","marker":"[19]"},{"why":"Provides feature-level knowledge distillation, the basis for the one-minute proxy used to evaluate candidate partitionings.","marker":"[34]"}],"fun_headline_variants":["Overlap-matrix search finds lean channel budgets per task","Channel masks via evolution beat share-all on Visual Decathlon","Feature distillation accelerates multi-task architecture search","Search over channel overlaps yields efficient multi-task models","Partitioning features beats sharing layers in multi-task nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the one-minute feature-distillation score ranks multi-task architectures in the same order as full training to convergence, even though the reported correlations vary by task and are as low as about 0.48 for two of the nine tasks.","fun_headline_variants_meta":{"raw":{"variants":["Overlap-matrix search finds lean channel budgets per task","Channel masks via evolution beat share-all on Visual Decathlon","Feature distillation accelerates multi-task architecture search","Search over channel overlaps yields efficient multi-task models","Partitioning features beats sharing layers in multi-task nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000702,"raw_usage":{"total_tokens":3124,"prompt_tokens":860,"completion_tokens":2264,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":2189}},"tokens_in":476,"tokens_out":2264,"duration_ms":16303,"temperature":1.0,"reasoning_tokens":2189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:44:09.646103+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of fifty random partitionings, score each with the one-minute distillation proxy, then train each to convergence; if the proxy ranking and the final ranking reverse substantially, or if the top proxy candidates systematically lose to lower-ranked ones after full training, the search's guidance fails. A concrete place to look is daimlerpedcls, where the paper already reports the lowest distillation-to-final correlation.","supporting_citations":[{"cited_title":"Rebuffi, H","cited_arxiv_id":null,"evidence_quote":"Supplies the Visual Decathlon benchmark, its train/validation/test splits, and the per-task batch-normalization convention used in all multi-task experiments."},{"cited_title":"Rebuffi, H","cited_arxiv_id":null,"evidence_quote":"Supplies the ImageNet-pretrained ResNet that initializes the shared model and the teacher networks for distillation; the paper freezes its first two blocks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides feature-level knowledge distillation, the basis for the one-minute proxy used to evaluate candidate partitionings."}],"review_version":1}