{"id":"e98247e5-b34e-4047-ae0c-0201fd86a025","arxiv_id":"2412.01420","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Pretraining an RL-based NAS agent on one of four Trans-NASBench-101 tasks generally improves its final performance and shortens training on another task.","lead":"The paper tests whether a reinforcement-learning agent that designs neural network architectures can be pretrained on one computer vision task and then fine-tuned to work on another, using the Trans-NASBench-101 benchmark. It finds that pretraining usually improves final performance and cuts training time, suggesting transfer learning could lower the cost of using such NAS agents on new tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported confidence intervals are internally inconsistent with N=3 and the reported standard deviations; the significance claims built on CI non-overlap are unsupported unless the intervals are recomputed.","rationale":"The reader's weakest_assumption pinpoints the single most load-bearing issue: the reported bootstrap confidence intervals are numerically inconsistent with N=3 and the reported standard deviations. The central claim that pretraining benefits target-task performance and shortens training rests on comparing means, but the paper's own significance statements rely on non-overlap of these intervals. If the intervals are recomputed correctly, the non-overlap almost certainly disappears, so the statistical support for 'benefits' and 'significantly shortened' collapses. I considered the alternative concern that transferred agents have more total training (source+target) than from-scratch agents (10M+10M vs 10M); however, the fine-tuning regime (10M+1M vs 10M) partially addresses this, and the cross-over-time analysis is a fairer metric, so the CI inconsistency is the more decisive and checkable flaw. The point estimates nevertheless consistently favor transfer, so the paper should not be outright rejected; it needs corrected confidence intervals, a softened significance language, and ideally raw data or code to allow verification. This is exactly the conditional acceptance the reader recommended, so the verdict is unchanged.","tokens_in":15838,"tokens_out":4868,"duration_ms":42006,"concrete_test":"Obtain the raw per-run values for Figure 5 (or the reported mean, std, and N=3) and recompute 95% confidence intervals for each cell using (a) a percentile bootstrap with at least 100,000 resamples and (b) a t-interval with df=2. Then check whether the from-scratch interval overlaps the transferred-agent intervals for each target task. If overlap occurs, re-evaluate the paper's claims of 'statistically significant' transfer benefit and 'significantly shortened' training without the non-overlap argument, and require the authors to release the raw data or code to verify the corrected intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Figures 5–7 and 12–13 report 95% bootstrap confidence intervals around means from N=3 runs, but the intervals are impossibly narrow given the accompanying standard deviations. For example, Figure 5 (autoencoder target, from-scratch) gives mean=0.518, std=0.038, CI=[0.518,0.519], a width of about 0.001. With N=3, the standard error of the mean is 0.038/sqrt(3)≈0.022; a t-interval with 2 degrees of freedom would be ±4.303·SE≈±0.095, and a bootstrap percentile interval over three observations cannot be narrower than the range of the three sample means. The reported width is therefore arithmetically inconsistent with the stated sample size and variance. The same pattern appears throughout: e.g., Figure 6 shows many intervals of width 0.001–0.002 despite standard deviations of 0.015–0.038, and Figures 12–13 show cross-over time CIs far narrower than the reported standard deviations allow. Because the paper explicitly uses CI non-overlap as its statistical significance criterion ('there is no overlap in the confidence intervals ... indicating a statistically significant difference'), a correct computation will almost certainly make the intervals overlap, erasing the significance of the transfer benefit and of the 'significantly shortened' training claims. The point estimates still favor transfer, so the qualitative direction may survive, but the strength of the central claim as stated is not supported by the reported statistics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether a reinforcement-learning-based neural architecture search (NAS) agent, originally introduced by Cassimon et al., can be reused across tasks via simple transfer learning. Four TransNASBench-101 tasks are selected (autoencoder, class-object, room-layout, segmentsemantic), and three regimes are compared: zero-shot transfer, fine-tuning for 1e6 timesteps, and retraining for 1e7 timesteps. The central empirical claims are that pretraining on one task benefits final performance on another task in all but one target task, and that transferred agents can be trained to match a from-scratch agent in significantly fewer timesteps. The paper uses point estimates, standard deviations, and 95% bootstrap confidence intervals from N=3 random seeds, and uses confidence-interval non-overlap as its statistical significance criterion.","tokens_in":16166,"tokens_out":5220,"duration_ms":49050,"significance":"If the qualitative findings hold, the paper addresses a practical bottleneck of RL-based NAS: the high cost of training a new agent for every task. The study is clearly scoped, uses a standardized benchmark, and documents the agent architecture and hyperparameters in enough detail to be reproducible in principle. It also offers two plausible mechanistic hypotheses (local optima and a grokking-like effect) for why transfer helps. The paper's usefulness, however, depends on the reliability of its statistical significance claims, and those claims are currently undermined by internally inconsistent confidence intervals. With corrected statistics and appropriately softened language, the direction of the results would still be informative; the study would then be a useful empirical data point on cross-task transfer for RL-based NAS agents.","major_comments":[{"comment":"The reported 95% bootstrap confidence intervals are arithmetically inconsistent with the stated sample size and standard deviations, and the non-overlap argument built on them is therefore unsupported. For example, in Figure 5 the autoencoder from-scratch cell reports mean=0.518, SD=0.038, and CI=[0.518,0.519], a width of about 0.001. With N=3, the standard error of the mean is 0.038/sqrt(3)≈0.022, and a bootstrap percentile interval over three observations cannot be narrower than the range of the three values. The same pattern appears throughout Figures 5–7 and in Figures 12–13. Because Sections 4.1 and 4.3 explicitly use CI non-overlap as the significance criterion, the claims 'indicating a statistically significant difference' and 'statistically significant improvement' rest on invalid intervals. The authors should recompute intervals with a correct small-sample method, report the method and number of resamples, and revise the significance language accordingly.","section":"§4.1, Figures 5–7"},{"comment":"The cross-over time statistics are not credible as reported and need to be recomputed. In Figure 13, the autoencoder→autoencoder cell gives 5.359×10^6 ± 6.801×10^2 timesteps with CI [4.820×10^6, 6.041×10^6]; the CI width is about 1.2×10^6, which is three orders of magnitude larger than the reported standard deviation, and the SD value appears to be copied from the wall-clock version in Figure 12. Moreover, the 'all-to-all comparison' creates 9 pairs from 3×3 runs, but these pairs are not independent because the same runs are reused; a bootstrap over pairs will understate uncertainty relative to a bootstrap over independent seeds. Without corrected estimates, the claim that pretraining 'significantly' shortens training is not supported by the reported evidence.","section":"§4.3, Figures 10–13"},{"comment":"Even apart from the CI arithmetic, the strength of the language is disproportionate to a sample size of N=3 per condition. The class-object case is acknowledged as non-significant, yet the surrounding text states that transferred agents outperform from-scratch agents in 'all but 1 task' and that the effects occur 'regardless of the source or target task.' Some of the reported differences are also small in absolute terms (e.g., 49.2% vs. 49.5% for class-object in Figure 5). The paper should report effect sizes, present per-seed values or at least the raw data necessary to verify them, and phrase the conclusions in terms of the direction of the point estimates rather than binary statistical significance.","section":"§4.1 and §4.2"},{"comment":"The shaded 95% confidence bands in the training-curve plots are computed from the same N=3 runs, and consecutive timesteps of the same run are highly correlated. The manuscript does not state how the moving-average filter, the multiple timesteps, and the repeated measures are accounted for in the band computation. Without this clarification, the visual separation of the bands should not be interpreted as evidence of statistically significant differences in convergence speed. This is a load-bearing point for the 'shortened training' claim and should either be rigorously justified or de-emphasized.","section":"§4, Figures 8–9"},{"comment":"The gamma reward shaping parameter for the segmentsemantic task was selected empirically by sweeping gamma and choosing the value that maximizes validation reward spread. Because the same task is then used in the transfer comparisons, the paper should clarify whether this selection is a form of model selection on the evaluation task and, if so, how it affects the comparison between from-scratch and transferred agents on that task. At minimum, the sweep range, the exact definition of 'spread,' and the criterion for 'optimal' should be stated explicitly.","section":"§3.2"}],"minor_comments":[{"comment":"There are several typos and inconsistencies, including 'trained froms scratch' in the captions of Figures 10, 12, and 13, 'CIF AR-10' in the related-work section, 'up-front of cost' in the introduction, and inconsistent use of 'class-object' versus 'class_object' between text and figures.","section":"Throughout"},{"comment":"The 'all-to-all comparison scheme' is only described in one sentence. Please define precisely how the cross-over point is computed, how runs that never reach the reference performance within the 1e7-timestep budget are handled, and whether censored runs are excluded or assigned a maximum value.","section":"§4.3"},{"comment":"The data availability statement says data are available 'upon reasonable request.' For a study whose conclusions rest on small-sample statistics, depositing the per-seed results and the code used to compute the confidence intervals would substantially increase confidence in the reported numbers.","section":"§5.5"},{"comment":"Figure 1 reports Kendall's tau correlations between tasks, but the text does not say what is being correlated (architectures? validation scores? a sample size?) or whether the values are computed on the validation or test set. Please add this information.","section":"§3.1, Figure 1"},{"comment":"The reference for the Adam optimizer is incomplete ('Kingma D (2014) Adam: a method for stochastic optimization'), and some reference entries have inconsistent formatting. These should be cleaned up before publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical direction is plausible and potentially useful, and I do not see an inherent circularity problem: the Cassimon et al. agent is used as a tool, not as evidence for the transfer conclusion. The blocking issue is the statistical reporting. The reported confidence intervals are internally inconsistent with N=3 and the reported standard deviations, which invalidates the significance claims as stated. This is fixable within the manuscript's scope: the authors can recompute intervals with a correct method, report effect sizes, and soften the significance language. I would give the authors the opportunity to do so rather than reject, because the point estimates still favor the transfer direction. There is also a minor scope concern: the conclusions are drawn from four computer-vision tasks, one benchmark, and one RL algorithm, but the authors already acknowledge several of these limitations in Section 5."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the first evaluation of fine-tuning transfer for the incremental RL NAS agent from Cassimon et al. (2024). The setup is clean: four tasks from Trans-NASBench-101, three transfer regimes, and they measure both final performance and time-to-match a from-scratch baseline. They also include a sensible limitations section and are honest about the sample-efficiency weakness of Ape-X. On the empirical side, the point estimates consistently favor transferred agents, and the cross-over time results, if the numbers are right, could be practically useful for someone deciding whether to reuse a pretrained agent.\n\nThe soft spot is not small. The reported 95% confidence intervals in Figures 5–7 are arithmetically impossible. For example, the autoencoder diagonal in Figure 5 reports mean 0.518 with standard deviation 0.038 across N=3 runs, but a confidence interval of [0.518, 0.519], width 0.001. With three runs, the standard error alone is about 0.022, and no bootstrap procedure can produce a percentile interval narrower than the spread of the three sample means. The same pattern appears throughout those figures. Since the paper explicitly uses CI non-overlap as its significance criterion, all of the “statistically significant” claims about transfer benefits and shortened training are unsupported as reported. The cross-over time intervals in Figures 12–13 are less obviously impossible, but they also look suspiciously narrow given the accompanying standard deviations.\n\nThe point estimates still support the qualitative direction: transferred agents usually match or beat from-scratch agents at the final evaluation, and they often reach the from-scratch performance level faster. So I would not dismiss the paper’s main hypothesis. But the strength of the central claim, as stated in the abstract, is not backed by the statistics. The gamma reward shaping is a fitted parameter with a single value chosen on the validation set, which is a minor concern compared to the CI problem.\n\nFor a reader in the NAS subfield, this is a worthwhile empirical question, and the paper is a legitimate extension of prior work rather than a breakthrough. It deserves a serious referee, but not a quick acceptance. The authors should recompute all of the intervals using a correct bootstrap or a proper small-sample test, report effect sizes instead of relying on overlap, and ideally release code and raw data so the numbers can be checked. I would not cite the quantitative significance claims until that is fixed, but the paper is worth engaging with in review rather than desk rejecting.","headline":"A useful empirical transfer study whose significance claims rest on impossible confidence intervals; the qualitative direction is plausible, but the statistics need to be redone before the strong claims can be trusted.","tokens_in":16628,"tokens_out":1803,"would_cite":false,"duration_ms":17902,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining a neural architecture search agent on one vision task gives it a better start on another task in almost every case tested, cutting the training needed.","keywords":["neural architecture search","reinforcement learning","transfer learning","fine-tuning","zero-shot transfer","TransNASBench-101","reward shaping","Ape-X"],"falsifier":"Recompute the 95% bootstrap confidence intervals from the raw per-run scores for each cell; if, as the reported numbers suggest, three runs with a standard deviation of about 0.038 yield an interval only about 0.001 wide, the bootstrap procedure is not producing valid intervals, and the non-overlap significance claims fail. A complementary check would be to repeat the 4x4 transfer matrix with ten or more seeds and test whether transferred agents still consistently beat from-scratch agents on three of the four target tasks.","tokens_in":15673,"feed_emoji":"🤖","tokens_out":9120,"duration_ms":71225,"temperature":0.7,"pith_summary":"Neural architecture search (NAS) is computationally expensive because each new task normally requires training the search agent from scratch. This paper tests whether a reinforcement-learning NAS agent that improves architectures incrementally can instead be pretrained on one computer-vision task and then transferred to another. Using four tasks from the TransNASBench-101 benchmark, the authors report that transferred agents match or beat from-scratch agents in final performance for three of the four target tasks, and that pretrained agents reach a given performance level in substantially fewer time steps. If the result holds, the upfront cost of training NAS agents could be amortized across many tasks, which matters because such agents can take dozens to hundreds of GPU-hours to train.","feed_headline":"Pretrained search agents beat from-scratch training on new tasks","feed_subtitle":"Transferred agents match or surpass scratch training in three of four target tasks, using far fewer steps.","key_machinery":"The central object is the incremental-improvement RL NAS agent from previous work: a transformer-encoded policy, trained with Ape-X, double Q-learning, dueling heads, three-step bootstrapping with partial episode bootstrapping, and prioritized experience replay, that repeatedly proposes small edits to the current architecture until the episode ends. The transfer mechanism is parameter initialization: the entire learned policy, including the architecture encoder and transformer layers, is copied from the source task and then trained on the target task. The paper's one new component is a gamma reward transform, $R'(s,a)=R(s,a)^{\\gamma}$, applied to the semantic-segmentation task (with $\\gamma=0.478$ chosen to maximize reward spread), because the raw mIoU reward is concentrated near zero and otherwise gives the agent too little learning signal.","core_discovery":"The paper's central claim is that a single RL-based NAS policy, pretrained on one task, is a better initialization for another task than random initialization, and that this holds across different source-target pairs despite the tasks having low ranking correlation with each other. Under both a fine-tuning regime ($10^6$ target steps) and a full retraining regime ($10^7$ target steps), the transferred agents achieve validation performance that meets or beats the from-scratch baseline on three of the four target tasks, with non-overlapping bootstrapped 95% confidence intervals cited as evidence of statistical significance. The same data show that the crossover point at which a transferred agent surpasses a from-scratch agent's $10^7$-step performance often occurs orders of magnitude earlier in training. The paper attributes part of the benefit to the shared search space: all tasks use the same architecture encoding, so the early embedding layers can transfer largely unchanged, while later layers are retrained.","pith_inferences":["The reported numbers hint that the bootstrapped confidence intervals may be too narrow for three runs (for example, a standard deviation of 0.038 with N=3 should not yield an interval width near 0.001); recomputing them with a correct percentile bootstrap could either strengthen or weaken the significance conclusions.","A natural testable extension would be freezing the architecture-embedding layers after pretraining and retraining only the policy head, which the paper's shared-search-space argument suggests should retain most of the transfer benefit at lower cost.","Because only Ape-X was tested, the authors' conclusions are tied to that algorithm's exploration behavior; trying a sample-efficient method would show whether the transfer advantage is a property of the task or of the optimizer.","The paper's two explanations for the effect—from-scratch runs getting stuck in local optima and a grokking-like late improvement—make distinguishable predictions about when transferred curves overtake scratch curves, so tracking per-run trajectories could separate them."],"forward_implications":["Pretraining an RL-based NAS agent on one task can shorten target-task training by orders of magnitude: in several source-target pairs the transferred agent matches the from-scratch agent's $10^7$-step performance within $10^3$ to $10^5$ steps.","Transfer benefits appear even between tasks with low ranking correlation, so the shared search space alone may explain part of the gain rather than task similarity.","Zero-shot transfer is consistently weaker, while even $10^6$ fine-tuning steps recover most of the benefit, so some target-task interaction is still required.","The same training-cost observation now holds on three standardized NAS benchmarks, supporting the view that search-space size is not the main driver of the agent's convergence time.","If the pretrained initialization is understood as a better starting point rather than a warm start, the effective cost of applying NAS to a new task may drop substantially, making RL-based NAS practical in settings where full training is unaffordable."],"supporting_citations":[{"why":"Supplies the RL-based NAS agent and training setup that the transfer experiments reuse.","marker":"Cassimon et al (2024)"},{"why":"Provides the TransNASBench-101 benchmark, the four selected tasks, and the evaluation metrics.","marker":"Duan et al (2021)"},{"why":"Provides the partial episode bootstrapping used in the agent's three-step target update.","marker":"Pardo et al (2018)"},{"why":"Provides the prioritized replay buffer used during training.","marker":"Schaul et al (2016)"},{"why":"Supplies the grokking phenomenon invoked to explain late from-scratch improvements.","marker":"Power et al (2021)"},{"why":"Provides the transfer-learning taxonomy that situates the fine-tuning regime.","marker":"Zhu et al (2023)"},{"why":"Demonstrates the effectiveness of fine-tuning in a prior RL setting, motivating the transfer protocol.","marker":"Julian et al (2021)"}],"fun_headline_variants":["Transferred RL agents match or beat scratch in 3 of 4 NAS tasks","Pretrained RL agents cut NAS training time and beat scratch","RL NAS agents transfer well across tasks, beating scratch in 3/4","One pretrained RL agent applies to multiple NAS tasks, beating scratch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the bootstrapped 95% confidence intervals, computed from only three random-seed runs, are valid; if the interval calculation is wrong, the non-overlap argument used to claim statistical significance collapses, even though the point estimates may still favor transferred agents.","fun_headline_variants_meta":{"raw":{"variants":["Transferred RL agents match or beat scratch in 3 of 4 NAS tasks","Pretrained RL agents cut NAS training time and beat scratch","RL NAS agents transfer well across tasks, beating scratch in 3/4","One pretrained RL agent applies to multiple NAS tasks, beating scratch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000895,"raw_usage":{"total_tokens":3828,"prompt_tokens":887,"completion_tokens":2941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2863}},"tokens_in":503,"tokens_out":2941,"duration_ms":18840,"temperature":1.0,"reasoning_tokens":2863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:23:27.991978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the 95% bootstrap confidence intervals from the raw per-run scores for each cell; if, as the reported numbers suggest, three runs with a standard deviation of about 0.038 yield an interval only about 0.001 wide, the bootstrap procedure is not producing valid intervals, and the non-overlap significance claims fail. A complementary check would be to repeat the 4x4 transfer matrix with ten or more seeds and test whether transferred agents still consistently beat from-scratch agents on three of the four target tasks.","supporting_citations":[{"cited_title":"Neural Computing and Applications doi:10.1007/s00521-024-10445-2, ://doi.org/10.1007/s00521-024-10445-2","cited_arxiv_id":null,"evidence_quote":"Supplies the RL-based NAS agent and training setup that the transfer experiments reuse."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 5251--5260","cited_arxiv_id":null,"evidence_quote":"Provides the TransNASBench-101 benchmark, the four selected tasks, and the evaluation metrics."},{"cited_title":"In: Dy J, Krause A (eds) Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 80","cited_arxiv_id":null,"evidence_quote":"Provides the partial episode bootstrapping used in the agent's three-step target update."},{"cited_title":"In: Proceedings of the 4th Inter national Conference on Learning Representations, ICLR","cited_arxiv_id":null,"evidence_quote":"Provides the prioritized replay buffer used during training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the grokking phenomenon invoked to explain late from-scratch improvements."},{"cited_title":"In: Kober J, Ramos F, Tomlin C (eds) Proceedings of the 2020 Conference on Robot Learning, Proceedings of Machine Learning Research, vol 155","cited_arxiv_id":null,"evidence_quote":"Demonstrates the effectiveness of fine-tuning in a prior RL setting, motivating the transfer protocol."}],"review_version":1}