{"id":"f259a0e9-70fb-4c38-a9fb-526c89d81f98","arxiv_id":"2505.07300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"L-SWAG, a zero-shot proxy combining inverse gradient standard deviation with layer-wise activation-pattern counts, plus the LIBRA ensemble rule, outperforms prior proxies on transformer and convolutional search spaces on average.","lead":"The paper introduces L-SWAG, a training-free metric that ranks Vision Transformer architectures by combining layer-wise gradient statistics with activation-pattern counts, and LIBRA-NAS, an algorithm that merges this metric with existing zero-cost proxies. If the results hold, architecture search for vision models can be done in hours on a single GPU instead of days or weeks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported L-SWAG and LIBRA gains may be in-sample: per-benchmark layer intervals and LIBRA's proxy choice both use the validation accuracies on which Spearman is then reported; a held-out protocol is needed.","rationale":"The reader's weakest assumption and my independent reading converge on the same load-bearing concern: the evaluation protocol does not separate what is tuned from what is measured. In Sec. 3.1, layer intervals are motivated by gradient-statistic spikes, but their selection is validated and apparently tuned against per-percentile correlation maps (Fig. 2a, Tab. 5) on the same benchmark labels used for the reported Spearman values. Since Fig. 1 says the relevant layers are benchmark-specific and no y-free rule is given, the reported average ρ=0.72 is at least partly an in-sample optimum. LIBRA's Algorithm 1 likewise consumes validation accuracies before re-scoring on those same accuracies, so the ensemble gains in Tab. 1 are also in-sample. These issues affect the paper's central claim (best ranking consistency) and the downstream 17.0% ImageNet search result, while the Autoformer benchmark being a pretrained one-shot supernet rather than independently trained subnetworks adds a qualification. The corrected ZiCO bound, extensive ablations, and breadth of benchmarks are genuine contributions, but they do not resolve the selection leakage. The concern is concrete and addressable with a held-out split, so the existing CONDITIONAL verdict remains appropriate rather than unqualified acceptance or rejection.","tokens_in":23739,"tokens_out":8988,"duration_ms":86454,"concrete_test":"For each benchmark, split the architecture set into two disjoint halves, A and B. Use only A to select (lhat,Lhat) for L-SWAG and to run LIBRA's Algorithm 1 (including z1,z2,z3 and the 0.1 correlation tolerance), then freeze those selections; compute Spearman ρ on B. Repeat over random splits (or use leave-one-benchmark-out) and compare held-out ρ_L-SWAG and ρ_LIBRA against NWOT and other baselines under the same protocol. If the held-out averages fall materially below 0.72 or the margin over NWOT shrinks below the reported 0.10, the headline result is in-sample.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Sec. 4.1 (Fig. 3/4: average ρ_L-SWAG=0.72 vs ρ_NWOT=0.62) requires that the layer interval (lhat,Lhat) in Eq. (1) be chosen by a rule that does not depend on the benchmark's validation accuracies. The paper does not provide such a fixed rule. Sec. 3.1 selects intervals from gradient-statistic 'spikes' (Fig. 2b), but the evidence that these spike endpoints are correct is Fig. 2a and Tab. 5, which are correlation-per-percentile maps computed against the same validation accuracies later used to report ρ. Fig. 1 also states that relevant layers are 'specific for each benchmark.' Reported ρ values are therefore close to per-benchmark maxima over an interval grid, and the 0.72 average is an in-sample optimum rather than the performance of a fixed zero-shot metric on a new search space. LIBRA has the same structure: Algorithm 1 uses ρ(Bij) and validation accuracy y to choose z1, z2, z3 (Eq. 9, bias matching), then Tab. 1 reports Spearman on the same y. The 17.0% ImageNet result inherits this leakage and further reports search time while excluding supernet training and label collection. Without a held-out protocol, 'best ranking consistency across several benchmarks' cannot be accepted as a generalizable claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"L-SWAG (Eq. (1)) combines a layer-wise gradient-variance term Λ with a layer-wise activation-pattern cardinality term Ψ to produce a training-free proxy score. The paper evaluates it against 19 benchmark/task combinations, including a newly trained Autoformer ViT search space (2000 architectures, 6 tasks), and reports an average Spearman ρ=0.72 versus ρ=0.62 for NWOT. The paper also introduces LIBRA-NAS (Algorithm 1), which selects three existing proxies per benchmark using information gain and bias matching, and reports improved ranking and an ImageNet-1k searched architecture with 17.0% test error in 0.1 GPU days.","tokens_in":23967,"tokens_out":9424,"duration_ms":88134,"significance":"If the claims hold under a truly held-out protocol, the contribution is significant: it is one of the few zero-cost proxies that transfers across convolutional and transformer search spaces, and the new ViT benchmark plus GeLU adaptation of activation-pattern metrics are useful community assets. The ablation study (Tab. 3) is thorough, and the authors make an honest attempt to correct an earlier proof of the ZiCO bound. However, the headline numbers are currently in-sample, because the layer intervals are chosen by inspecting per-benchmark correlations against validation accuracies, and LIBRA consumes those same accuracies when selecting proxies. These issues bear directly on the paper's central generalization claim and prevent acceptance in the present form.","major_comments":[{"comment":"The central claim that L-SWAG 'achieves the best ranking consistency across several benchmarks' (Sec. 4.1) is not established as a statement about a fixed zero-shot metric. The metric in Eq. (1) depends on the interval (lhat,Lhat). Section 3.1 states that these endpoints are chosen as the beginning and end of 'spikes' in the gradient statistics, but the evidence that this choice is correct is Fig. 2a and Tab. 5, which are Spearman correlations of the resulting intervals against the same validation accuracies that are later used to report L-SWAG's ρ. The caption of Fig. 1 additionally states that the relevant layers are 'specific for each benchmark.' Consequently, the quoted average ρ=0.72 over the second-best ρ=0.62 is an in-sample optimum over a per-benchmark interval grid, not the performance of a rule that can be applied to a new search space. The authors should provide a fixed, label-free rule for choosing (lhat,Lhat), or evaluate the metric with a held-out protocol (e.g., choose intervals on a subset of benchmarks and report correlations on the rest).","section":"Sec. 3.1, Sec. 4.1, Eq. (1), Fig. 2a, Tab. 5"},{"comment":"Algorithm 1 and Eq. (9) make LIBRA's proxy selection depend directly on the validation accuracies y of the benchmark being evaluated. In particular, z1 is the proxy with the highest Spearman correlation against y, z2 minimizes IG(zj)=H(y|zi)-H(y|zi,zj), and z3 is chosen to match the bias of the validation accuracy. Table 1 then reports Spearman correlation between the LIBRA-combined proxy and the same y. This is a selection-on-labels protocol: the 13-out-of-19 improvement over AZ-NAS and other baselines may reflect overfitting to the evaluation set rather than a generalizable ensemble rule. A valid evaluation would select z1,z2,z3 on held-out benchmarks or using only proxy-side metadata, and then evaluate on unseen benchmark/task combinations.","section":"Sec. 3.2, Algorithm 1, Eq. (9), Tab. 1"},{"comment":"The efficiency comparison in Tab. 2 is incomplete. The row for Autoformer-Small reports 0.1 GPU days for LIBRA-NAS, but the 17.0% test error is obtained by evaluating subnetworks of a supernet that, according to supp. B.1, was trained three times for 500 epochs each on ImageNet-1k. Excluding supernet training and label collection from 'search time' makes the comparison against Autoformer's 24 GPU days (which includes evolution on the same supernet) and against PC-DARTS (which trains from scratch) misleading. The claim should be re-scoped to 'search cost after a pre-trained supernet is available,' or the total cost should be reported.","section":"Sec. 4.2, Tab. 2, supp. B.1"}],"minor_comments":[{"comment":"The corrected proof in the supplementary material ends with a bound 1/(2M)Σ_j(σ_j² + (Mη−1)²μ_j²), while Theorem 1 in the main text states (M/2)Σ_j(σ_j² + ((Mη−1)μ_j)²). These differ by a factor of M². The direction of the argument is unaffected, but the theorem and its proof must be consistent.","section":"Supp. A, Eq. (17) vs main Eq. (5)"},{"comment":"Algorithm 1 fixes the correlation tolerance at 0.1, but supp. E reports that for TNB101-micro-room and TNB101-macro-room the filtered set was obtained with a tolerance of 0.2. Please specify the exact tolerance used for every benchmark and justify the exception.","section":"Algorithm 1 vs supp. E"},{"comment":"The paper states that results were obtained from '5 multiple runs' but no error bars or standard deviations are reported; for the headline average ρ=0.72, please report variance or a significance test.","section":"Fig. 3, Tab. 1"},{"comment":"The reference list contains corrupted entries (e.g., the Autoformer [10] and NAS-Bench-201 [14] author names), and Eq. (6) has a formatting issue with the '//' operator; please clean these up.","section":"References and Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The paper is well executed empirically, but the central 'zero-shot' framing is undercut by the use of validation accuracies in hyperparameter selection. If the authors respond with a held-out protocol and re-scope the ImageNet efficiency claim, the contribution would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to the chase. The paper is worth a serious referee, but the headline ranking numbers should not be accepted as zero-shot. The strongest contribution is the new Autoformer ViT benchmark: 2000 trained architectures on six tasks, with existing proxies adapted to GeLU. That is a real community asset, assuming it is released. The L-SWAG metric is a plausible fusion of ZiCO's gradient-variance term and the activation-pattern cardinality from SWAP/NWOT, with a per-benchmark layer interval and a GeLU adaptation. The empirical work is extensive: 19 tasks, ablations of every component, five runs. The correction of the ZiCO bound in the supplementary is a genuine small result.\n\nWhere the paper goes soft is the evaluation protocol. Eq. (1) has the layer interval (lhat, Lhat). The paper says it is selected from gradient-statistic spikes (Fig. 2b), but the evidence that those spikes are the right endpoints is Fig. 2a and Tab. 5, which are correlation-per-percentile maps computed against the same validation accuracies on which Spearman is then reported. That makes the reported rho = 0.72 close to a per-benchmark optimum over an interval grid, not the performance of a fixed metric on a new search space. LIBRA is worse: Algorithm 1 consumes rho and validation accuracy y to pick the three proxies, and Tab. 1 then reports Spearman on the same y. The 17.0% ImageNet result inherits that leakage, and the 0.1 GPU days excludes supernet training and label collection.\n\nNone of this kills the paper's usefulness. The benchmark alone justifies engagement. But the central claim that L-SWAG outperforms others by a large margin as a zero-shot metric is unproven until there is a held-out protocol: benchmark-specific choices made without inspecting y, or calibration on a subset and evaluation on a disjoint set. The authors should also release code, data, and the Autoformer model weights. As written, a strict reviewer should not accept the generalizable claim. A revision with a fixed layer-selection rule and a LIBRA evaluation on held-out benchmarks would make it a solid contribution.\n\nWho is this for? Anyone working on zero-cost NAS or benchmarking ViT search spaces. It deserves peer review, but not acceptance in its current form.","headline":"A genuinely useful ViT zero-shot NAS benchmark and a sensible new proxy, but the reported Spearman correlations are in-sample: both the layer intervals in L-SWAG and the proxy selection in LIBRA are chosen on the same validation accuracies later used for evaluation.","tokens_in":24573,"tokens_out":3480,"would_cite":false,"duration_ms":32028,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces L-SWAG, a zero-cost proxy that ranks Vision Transformers without training, with average Spearman correlation 0.72 across 14 tasks, and LIBRA-NAS, which finds an ImageNet-1k architecture with 17.0% test error in 0.1…","keywords":["zero-shot NAS","training-free architecture search","zero-cost proxies","vision transformers","Spearman rank correlation","gradient statistics","activation patterns","proxy ensembling"],"falsifier":"Hold out a search space and dataset not used in the paper; choose the layer interval for L-SWAG using only the average gradient-statistic spike curves, then compute Spearman correlation against validation accuracies that were never looked at. If the correlation drops to roughly the 0.62 level of the next-best proxy, the advantage of L-SWAG came from benchmark-specific interval selection rather than from the metric itself.","tokens_in":23467,"feed_emoji":"⚡","tokens_out":8921,"duration_ms":81267,"temperature":0.7,"pith_summary":"The paper aims to make training-free neural architecture search work for Vision Transformers, not only convolutional networks. It introduces L-SWAG, a zero-cost proxy that multiplies a layer-wise gradient-variance term with a layer-wise activation-pattern term, and reports that this single metric ranks architectures with an average Spearman correlation of 0.72 across 14 tasks, against 0.62 for the best prior proxy. It also introduces LIBRA-NAS, a rule that merges three complementary proxies by picking the best one, the one with lowest information gain given it, and the one with the closest bias to the validation accuracy. Applied to a transformer search space, the paper claims LIBRA finds an ImageNet-1k architecture with 17.0% test error in 0.1 GPU days, beating evolution and gradient-based search. The contribution matters because proxy-based search removes the expensive training step that normally dominates NAS.","feed_headline":"Zero-shot metric ranks ViTs; 0.1-GPU-day search hits 17% error","feed_subtitle":"Training-free proxy reports average Spearman 0.72 across tasks, beating the prior best 0.62.","key_machinery":"The load-bearing object is the L-SWAG score, Eq. (1): $L\\text{-}SWAG = \\Lambda_{\\hat L} \\times \\Psi^{}_{\\hat L}$, where $\\Lambda_{\\hat L}$ aggregates layer-wise gradient variance (with the mean term removed) and $\\Psi^{}_{\\hat L}$ is the cardinality of layer-wise sample-wise activation patterns. Its two supporting mechanisms are the spike-based selection of the layer interval $(\\hat l, \\hat L)$ from the distribution of gradient statistics, and the multiplication rule that keeps either term from dominating the ranking. For LIBRA-NAS, the machinery is the selection loop that takes a set of precomputed proxy correlations and biases and outputs three proxies: the highest-correlation proxy, the proxy with minimal information gain $IG(z_j) = H(y \\mid z_i) - H(y \\mid z_i, z_j)$ given it, and the proxy whose bias is closest to the validation accuracy's bias; this triple replaces the single proxy during search.","core_discovery":"On its own terms, the paper establishes that a single zero-cost score can rank both convolutional and transformer architectures if it captures trainability and expressivity layer by layer. L-SWAG is the product of two layer-wise statistics: a trainability term $\\Lambda_{\\hat L}$ that sums, over a selected interval of layers, the logarithm of the inverse standard deviation of per-weight gradients, deliberately dropping the gradient mean that the prior ZiCO proxy includes; and an expressivity term $\\Psi^{}_{\\hat L}$ that counts distinct binarised activation patterns across samples, extended from ReLU to GeLU activation. The interval of layers is chosen from benchmark-visible spikes in the gradient statistics, and the paper shows that this choice markedly improves Spearman rank correlation over using all layers. The same experimental campaign reports an average correlation of $\\rho = 0.72$ against $0.62$ for the second-best proxy, and that L-SWAG is the only proxy that consistently beats parameter count in the transformer search space. The companion contribution, LIBRA-NAS, is an algorithm that selects three proxies -- the best-scoring one, the one with lowest conditional information gain given it, and the one whose parameter-count bias most closely matches the validation accuracy -- and this ensemble, used to guide search, yields a 17.0% test error on ImageNet-1k in 0.1 GPU days.","pith_inferences":["If the spike-based layer interval is genuinely chosen from gradient statistics alone and not from validation accuracies, the same rule should transfer to a new ViT search space; a user could test this by fixing the interval-selection procedure and never looking at validation accuracies before computing the correlation.","The 17.0% ImageNet result is specific to one transformer search space and to LIBRA's proxy pool; a reasonable expectation is that the gain over the best single proxy will shrink when the proxy pool is small or when the validation-accuracy bias is not representative of the target task.","Because LIBRA's information-gain minimization is admitted to be heuristic, an alternative worth testing is whether a decorrelation-based or maximum-information-gain selection would generalize better on held-out benchmarks; the paper's ablations only compare against random and maximization on known benchmarks.","If the layer-wise formulation transfers to video transformers, the same metric could be used to rank video backbones at initialization, a stated future direction that is not yet demonstrated."],"forward_implications":["If L-SWAG's reported correlations hold, a single forward pass through an untrained ViT is enough to rank candidate architectures, making zero-shot search practical for transformer spaces that previously resisted zero-cost proxies.","Because the expressivity term is adapted to GeLU, the metric applies to modern transformer blocks without assuming ReLU activations, so the same proxy can compare convnets and ViTs in one ranking.","LIBRA-NAS shows that combining proxies by information gain and bias matching can beat evolution and gradient-based NAS on ImageNet-1k with 0.1 GPU days of search, a cost reduction of orders of magnitude relative to training-based search.","Layer selection speeds up the metric as well as improving correlation, so the method becomes cheaper as it becomes more accurate.","The same proxy can guide search across six tasks and multiple search spaces with one fixed rule, suggesting the approach transfers across datasets."],"supporting_citations":[{"why":"Supplies the gradient-variance proxy whose mean term L-SWAG drops; its theorem is corrected and reused in the derivation.","marker":"[29]"},{"why":"Defines sample-wise activation patterns, the expressivity component that L-SWAG extends to GeLU and to layer-wise computation.","marker":"[39]"},{"why":"Defines the activation-pattern baseline that is the second-best comparator in the benchmark campaign.","marker":"[36]"},{"why":"Provides the correlation, bias, and information-gain conventions that LIBRA-NAS reuses for proxy selection.","marker":"[25]"},{"why":"Supplies the transformer search space and the trained supernet from which the 2000-network benchmark is sampled.","marker":"[10]"},{"why":"Motivates multiplying the trainability and expressivity terms and interprets the variance component through compressibility.","marker":"[52]"},{"why":"Contributes the cross-task benchmark where most existing proxies fail and where L-SWAG reports its largest gains.","marker":"[15]"},{"why":"Provides the pruning-based search procedure used to integrate L-SWAG and LIBRA into the search loop.","marker":"[8]"}],"fun_headline_variants":["L-SWAG: zero-cost ViT ranking, 17% error in 0.1 GPU-day","No training: ViT NAS hits 17% in 0.1 GPU-day with L-SWAG","Layer-wise zero-cost proxy beats prior, 17% in 0.1 GPU-day","Vision Transformer NAS: 17% error, 0.1 GPU-day, no training","L-SWAG: 0.1 GPU-day to 17% error on ImageNet-1k"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline correlation rests on the assumption that the layer interval where gradient statistics spike can be picked for each benchmark without consulting that benchmark's validation accuracies, so the reported 0.72 average is not an in-sample optimum.","fun_headline_variants_meta":{"raw":{"variants":["L-SWAG: zero-cost ViT ranking, 17% error in 0.1 GPU-day","No training: ViT NAS hits 17% in 0.1 GPU-day with L-SWAG","Layer-wise zero-cost proxy beats prior, 17% in 0.1 GPU-day","Vision Transformer NAS: 17% error, 0.1 GPU-day, no training","L-SWAG: 0.1 GPU-day to 17% error on ImageNet-1k"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1884,"prompt_tokens":1091,"completion_tokens":793,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":665}},"tokens_in":707,"tokens_out":793,"duration_ms":6925,"temperature":1.0,"reasoning_tokens":665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:20:00.384910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a search space and dataset not used in the paper; choose the layer interval for L-SWAG using only the average gradient-statistic spike curves, then compute Spearman correlation against validation accuracies that were never looked at. If the correlation drops to roughly the 0.62 level of the next-best proxy, the advantage of L-SWAG came from benchmark-specific interval selection rather than from the metric itself.","supporting_citations":[{"cited_title":"Zico: Zero-shot NAS via inverse coefficient of variation on gradients","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-variance proxy whose mean term L-SWAG drops; its theorem is corrected and reused in the derivation."},{"cited_title":"Fayek, Vic Ciesiel- ski, and Xiaojun Chang","cited_arxiv_id":null,"evidence_quote":"Defines sample-wise activation patterns, the expressivity component that L-SWAG extends to GeLU and to layer-wise computation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the activation-pattern baseline that is the second-best comparator in the benchmark campaign."},{"cited_title":"NAS-bench-suite-zero: Accelerating research on zero cost proxies","cited_arxiv_id":null,"evidence_quote":"Provides the correlation, bias, and information-gain conventions that LIBRA-NAS reuses for proxy selection."},{"cited_title":"Autoformer: Searching transformers for visual recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the transformer search space and the trained supernet from which the 2000-network benchmark is sampled."},{"cited_title":"Exploiting network compress- ibility and topology in zero-cost NAS","cited_arxiv_id":null,"evidence_quote":"Motivates multiplying the trainability and expressivity terms and interprets the variance component through compressibility."},{"cited_title":"Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search","cited_arxiv_id":null,"evidence_quote":"Contributes the cross-task benchmark where most existing proxies fail and where L-SWAG reports its largest gains."},{"cited_title":"Neural archi- tecture search on imagenet in four gpu hours: A theoretically inspired perspective","cited_arxiv_id":null,"evidence_quote":"Provides the pruning-based search procedure used to integrate L-SWAG and LIBRA into the search loop."}],"review_version":1}