{"id":"40ad0e80-8ab4-43b5-95dc-124fe18fb4ad","arxiv_id":"1909.00311","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Reinforcement-learning-based neural architecture search finds smaller and faster neural networks with accuracy comparable to, or better than, manually designed networks on three cancer drug-response benchmarks.","lead":"This paper builds an automated search that uses reinforcement learning to design deep neural networks for three cancer prediction benchmarks, and it scales the search to more than a thousand supercomputer nodes. For a business or research reader, the practical question is whether automated architecture search can replace manual, expert-driven network design for tabular medical data while shrinking model size and training time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low-fidelity reward noise plus selection on the validation set leaves the accuracy half of the headline claim unquantified; parameter and time reductions are solid.","rationale":"The reader's weakest assumption identifies the noisy-reward rank-preservation issue, and I agree it is load-bearing. I would push further: even if rankings were preserved, the final accuracy comparison is made on the same validation set used to select the best of 50 architectures and without repeated-seed error bars, so selection bias and optimizer noise are confounded with search quality. The cleanest single test is a stratified post-training evaluation with seeds and a held-out split. This does not impugn the scalability contribution, the open-source software, or the parameter/time reductions; those are supported by the experiments. The verdict should remain CONDITIONAL pending this validation.","tokens_in":21100,"tokens_out":4586,"duration_ms":44827,"concrete_test":"Use the saved A3C search logs to sample about 100 architectures stratified by one-epoch reward, including low-reward ones. Post-train each with at least 5 random seeds for 20 epochs on full data, then compute the rank correlation (e.g., Kendall tau) between one-epoch reward and mean post-training metric, and compare the best-of-50 distribution to a random or uniform sample under identical post-training. Also reserve a held-out test split before any selection and re-evaluate both NAS and manual baselines on it. If tau is high and the best-of-50 advantage persists on held-out data with seed error bars, the concern is resolved; otherwise the accuracy claim should be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that A3C finds architectures with accuracy similar to or higher than manual designs. Parameter and training-time reductions are deterministic properties of the discovered architectures and are credible. Accuracy is not. The top-50 architectures are selected on a one-epoch reward that the paper itself shows is very noisy: Section 5.1 reports the same NT3 architecture receiving rewards of 1.0 and 0.4 from different random initializations, and Section 5.4 shows that changing the training-data fraction changes what the agent learns, so low-fidelity reward is not a neutral proxy. Section 5.5 quantifies only the randomness of the search trajectory, not of the final post-training metrics. Table 1 reports a single R2/ACC per benchmark with no error bars, and the same validation set used for reward and post-training is used for the manual comparison; choosing the best of the 50 post-trained architectures on that set can inflate the reported accuracy. Unless one-epoch reward is shown to rank architectures like 20-epoch post-training, and unless post-training is repeated across seeds and evaluated on a truly held-out split, the 'accuracy similar to or higher' part of the claim is unsupported, even though the parameter/time parts stand.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a reinforcement-learning-based neural architecture search (NAS) framework for deep learning models on three cancer-related benchmarks (Combo, Uno, NT3) from the CANDLE project. The search space is customized with multiple input layers, variable/constant/mirror nodes, and skip connections to capture cancer-data-specific structure. The authors scale asynchronous advantage actor-critic (A3C) and synchronous A2C with proximal policy optimization on up to 1,024 Intel Knights Landing nodes, compare against random search, and show that A3C achieves higher reward trajectories and better utilization. For the final claim, the top-50 architectures from each search are post-trained and compared with manually designed networks on accuracy, trainable parameter count, and training time; the paper reports that A3C finds networks with substantially fewer parameters and shorter training time, and with accuracy similar to or higher than the manual baselines.","tokens_in":21338,"tokens_out":3173,"duration_ms":30608,"significance":"If the central accuracy claim held as stated, this would be a valuable demonstration of scalable RL-based NAS on nonimage, nontext cancer data, an application area where NAS has been little explored. The paper's strengths include the open-source DeepHyper-based implementation, the careful system-level scalability study up to 1,024 nodes, the comparison of A3C with A2C and random search under controlled settings, and the domain-motivated search space primitives such as MirrorNode and ConstantNode. The parameter and training-time reductions are credible because they are deterministic properties of the discovered architectures. However, the accuracy half of the headline claim is not yet supported with the evidence presented, due to single-run post-training metrics and selection on the same validation set used for reward estimation.","major_comments":[{"comment":"The accuracy comparisons are based on a single post-training run per architecture, with no error bars or standard deviations reported. Table 1 reports one R2 or ACC value for each A3C-best architecture, and Figs. 7 and 8 plot individual ratio points without uncertainty. Given that Section 5.1 itself reports the same NT3 architecture receiving rewards of 1.0 and 0.4 under different random initializations during reward estimation, the final post-training metrics are also expected to be seed-sensitive. The claim 'accuracy similar to or higher than those of manually designed architectures' therefore requires multiple post-training repetitions with mean and variance reported, and evaluation on a held-out test split not used for either reward estimation or top-50 selection.","section":"Section 5.2, Table 1, Figs. 7-8"},{"comment":"The selection of the top-50 architectures is based on a low-fidelity reward: one training epoch, a 10-minute timeout, and only 10% of the training data for Combo. Section 5.4 shows that changing the training-data fraction changes what architectures the agent learns to generate, so the low-fidelity reward is not a neutral proxy for the post-training objective. The paper does not validate the load-bearing assumption that one-epoch reward ranks architectures in the same order as 20-epoch full-data post-training. To support the central claim, the authors should directly measure rank preservation, for example by computing the correlation between reward estimates and post-training R2/ACC on a random sample of architectures, or by using a multi-fidelity schedule that confirms early rankings with higher-fidelity evaluations.","section":"Section 3.3 and Section 5.4"},{"comment":"The randomness analysis in Section 5.5 quantifies only the variation of the A3C search trajectory (reward quantiles over time across 10 replications). It does not quantify the variance of the final post-training metrics for the selected architectures, which is the quantity that matters for Table 1. The conclusion that 'randomness does not have a significant impact on the search trajectory' cannot be extended to the accuracy of the best reported architectures without replicating the full selection-plus-post-training pipeline across multiple seeds and reporting the resulting distribution of R2 and ACC values.","section":"Section 5.5"}],"minor_comments":[{"comment":"There is a typo in the first sentence: 'OOn Uno' should read 'On Uno.'","section":"Section 5.6"},{"comment":"The text says NT3 training time speedup is 'up to 20x', but Table 1 reports the NT3 A3C-best training time as 16.65 s versus 247.63 s, which is about 14.8x; these numbers should be reconciled.","section":"Section 5.2"},{"comment":"In the 'Open source software' paragraph, 'scalabiltiy' is a typo for 'scalability.'","section":"Section 6"},{"comment":"The row labeled 'A3C-best' does not indicate which search-space configuration (small or large) or which node count (256, 512, or 1,024) produced each architecture; providing this provenance would aid reproducibility.","section":"Table 1"},{"comment":"Figure 13 reports 10%, 50%, and 90% quantiles computed from only 10 replications; this is a small sample for quantile estimation, and the paper does not state the random seeds or the initialization scheme used, so the reader cannot assess the stability of the displayed bands.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution for HPC-scale NAS, and the parameter/time reductions appear credible. The main risk is that the accuracy claim is currently supported by single-run, validation-selection-inflated numbers. I would encourage the editor to insist on the rank-preservation analysis and repeated post-training before accepting the central claim; if the authors can provide those, the paper would be a strong fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take before you read: this is a real engineering contribution—RL-based NAS on HPC applied to three cancer tabular benchmarks, with custom search-space primitives and open-source code in DeepHyper. But treat the accuracy half of the headline claim with caution. The parameter and training-time reductions are solid; the accuracy comparison is not.\n\nWhat's actually new: the multi-input graph search spaces with ConstantNode and MirrorNode are a sensible way to encode cancer-domain structure (paired drugs, shared feature encoders), and the paper is the first to apply NAS to these CANDLE benchmarks. Scaling A3C to 1,024 KNL nodes and the comparison against random search and A2C are the right experiments: they show the search strategy does something beyond chance. The paper is also transparent about reward noise and fidelity effects, which I appreciate.\n\nThe soft spot is the accuracy claim. The top-50 architectures are chosen on a one-epoch reward that Section 5.1 shows is very noisy—the same NT3 architecture gets rewards of 1.0 and 0.4 from different initializations. Post-training is a single run per architecture, no error bars, and the validation set used for reward selection is the same set used to compare against the manual baselines. Picking the best of 50 on that set can inflate accuracy. So 'accuracy similar to or higher than manual networks' is not demonstrated. The parameter and time ratios are deterministic and credible, but the accuracy comparison needs repeated seeds and a truly held-out split.\n\nOne more caveat: the search spaces are built around the manual architectures, so the comparison is somewhat favorable. Still, finding networks with 7–800x fewer parameters and 2.5–15x faster training is a real result for practitioners, and the random-search baseline mitigates the circularity somewhat.\n\nWho should read this: anyone doing AutoML for scientific or tabular data, and anyone building HPC infrastructure for NAS. It deserves a serious referee—I'd send it to review with a request to fix the accuracy evaluation and soften that claim. Not a desk reject.","headline":"Solid HPC-scale RL-NAS engineering for cancer tabular data, but the accuracy claim is overreach given a noisy low-fidelity reward and selection on the validation set.","tokens_in":21905,"tokens_out":2789,"would_cite":true,"duration_ms":26118,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning agent that generates neural-network architectures from a search space of custom nodes finds drug-response and gene-expression models that match or beat hand-built baselines while using up to 800x fewer trainable…","keywords":["neural architecture search","reinforcement learning","asynchronous actor-critic","cancer deep learning","drug response prediction","tabular data","parameter efficiency","high-performance computing"],"falsifier":"Sample, say, 200 architectures from the Combo search space, compute both the cheap reward (one epoch, 10% of training data, 10-minute timeout) and the full post-training R2 for each, and measure the rank correlation between the two; if the correlation is weak, or if the top 50 by cheap reward perform no better when fully trained than 50 random architectures, the claimed advantage comes from the fidelity shortcut rather than from the search strategy.","tokens_in":20875,"feed_emoji":"🧬","tokens_out":9943,"duration_ms":86823,"temperature":0.7,"pith_summary":"The paper claims that automated neural architecture search driven by reinforcement learning can replace manual trial-and-error design of deep networks for cancer prediction on tabular data. An asynchronous advantage actor-critic agent builds architectures from a graph search space whose multiple input branches, fixed domain-knowledge nodes, and shared-weight mirror nodes are tailored to drug-response and gene-expression data. On three cancer benchmarks, the best discovered networks match or exceed the accuracy of the manually designed baselines while using far fewer parameters and less training time; the largest reduction is roughly 800x fewer parameters with 14.8x faster training on the NT3 gene-expression task. The authors take this as evidence that a scalable search strategy, not random chance, can automate deep-learning model development for cancer researchers.","feed_headline":"Automated search finds cancer models 800x leaner, same accuracy","feed_subtitle":"Reinforcement learning picked drug-response networks that beat manual designs while training up to 14.8x faster.","key_machinery":"The machinery is the graph-based search space joined to a PPO-clipped actor-critic agent. A search space is a directed acyclic graph of cells, each cell containing blocks of nodes; a variable node is a placeholder whose choices are operations such as Dense(100, relu), Dropout(0.1), or a skip connection. Two node types encode the cancer-specific structure: constant nodes fix an operation (for example, adding the dose value at each block in Uno), and mirror nodes reuse the same submodel for both drug descriptors in Combo, enforcing weight sharing. The agent chooses the sequence of node operations, receives a reward equal to validation R2 or accuracy after one training epoch, and updates the shared policy with the PPO clipped surrogate objective. A3C runs many agents asynchronously against a parameter server, accepting stale gradients in exchange for keeping compute nodes busy.","core_discovery":"The central discovery is that the asynchronous advantage actor-critic variant (A3C), which tolerates stale gradient updates in exchange for higher node utilization, finds better architectures faster than its synchronous counterpart or random search on all three benchmarks. The search is organized around a directed-acyclic-graph search space with multiple input cells, variable nodes whose options are layer operations, constant nodes that fix operations such as adding the dose input at every block, and mirror nodes that force the two drug-descriptor branches in Combo to share weights. After ranking generated architectures by a cheap one-epoch reward (on 10% of the Combo training data, with a 10-minute timeout), the top 50 are retrained fully; the best ones reach Combo R2 0.930 versus 0.926 with 7.3x fewer parameters, Uno R2 0.729 versus 0.649 with 11.5x fewer parameters, and NT3 accuracy 0.989 versus 0.986 with about 800x fewer parameters. The paper presents these numbers as support for replacing manual model design with automated search on high-performance computing systems.","pith_inferences":["A reader could infer that the manual baselines are heavily overparameterized for these small tabular datasets; if so, similarly large parameter reductions may appear when the same search is applied to other genomics or drug-response problems without further search-space engineering.","The noisy reward signal (one NT3 architecture scored 1.0 and 0.4 under different random seeds) suggests the top-50 selection would be more reliable if each candidate's cheap reward were averaged over several seeds before ranking.","The fidelity results imply an implicit accuracy-versus-speed tradeoff: with 10% data the agent maximizes accuracy, while with 40% data the timeout forces it to favor fast-to-train architectures. A testable extension would make that tradeoff explicit in the reward function.","Because the search space is defined as a graph of cells and nodes, the same code and search strategy could be applied to other scientific tabular benchmarks by swapping cell definitions; that experiment would separate the value of the RL search from the value of the cancer-specific prior knowledge encoded in the search space."],"forward_implications":["Cancer researchers can replace weeks of manual network design with a roughly six-hour automated search, since the paper demonstrates the full pipeline on leadership-class supercomputing resources.","The large parameter reductions imply cheaper retraining, hyperparameter tuning, and scaling to larger datasets, because training time scales with parameter count.","Asynchronous A3C is the preferred scalable variant of RL-based NAS: it reaches higher rewards in less wall-clock time than synchronous A2C or random search and keeps node utilization higher.","The custom search-space building blocks (multiple input cells, constant nodes, mirror nodes) provide a reusable template for defining NAS spaces over other multi-modal tabular data.","The fidelity experiments show that the cheap reward is not neutral: changing the fraction of training data used to estimate rewards changes which architectures the agent favors, trading accuracy for training speed as the data fraction grows."],"supporting_citations":[{"why":"Defines the Combo drug-pair response problem and the manually designed network whose accuracy and parameter count serve as the baseline.","marker":"[3]"},{"why":"Defines the NT3 gene-expression classification problem and the manual network baseline.","marker":"[8]"},{"why":"Defines the Uno drug-response problem with its four input modalities and the manual network baseline.","marker":"[9]"},{"why":"Supplies the clipped proximal policy optimization objective used for the actor-critic updates in both A2C and A3C.","marker":"[68]"},{"why":"Establishes the reinforcement-learning-based neural architecture search approach that the paper scales and adapts to cancer data.","marker":"[88]"},{"why":"Provides the prior large-scale RL-based NAS demonstration that this paper's 1,024-node runs extend to a shorter wall-clock setting.","marker":"[89]"},{"why":"The workflow system that launches and monitors the many reward-estimation tasks; it carries the asynchronous scalability result.","marker":"[66]"},{"why":"The distributed search package used for dispatching reward evaluations and caching agent-specific results; it supports the large-scale search runs.","marker":"[15]"}],"fun_headline_variants":["Reinforcement learning searches cancer models with 800x fewer parameters","AI-designed cancer models match or beat manual ones with far fewer weights","RL finds cancer models up to 800x smaller with equal accuracy","Scalable RL neural architecture search finds leaner cancer models","Cancer models via RL are up to 800x leaner and train 14.8x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the cheap one-epoch, often partial-data reward ranks architectures the same way full post-training would, even though the paper records the same NT3 architecture scoring 1.0 from one seed and 0.4 from another.","fun_headline_variants_meta":{"raw":{"variants":["Reinforcement learning searches cancer models with 800x fewer parameters","AI-designed cancer models match or beat manual ones with far fewer weights","RL finds cancer models up to 800x smaller with equal accuracy","Scalable RL neural architecture search finds leaner cancer models","Cancer models via RL are up to 800x leaner and train 14.8x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001174,"raw_usage":{"total_tokens":4868,"prompt_tokens":974,"completion_tokens":3894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":3798}},"tokens_in":590,"tokens_out":3894,"duration_ms":26445,"temperature":1.0,"reasoning_tokens":3798,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:55:42.328543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample, say, 200 architectures from the Combo search space, compute both the cheap reward (one epoch, 10% of training data, 10-minute timeout) and the full post-training R2 for each, and measure the rank correlation between the two; if the correlation is weak, or if the top 50 by cheap reward perform no better when fully trained than 50 random architectures, the claimed advantage comes from the fidelity shortcut rather than from the search strategy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior large-scale RL-based NAS demonstration that this paper's 1,024-node runs extend to a shorter wall-clock setting."},{"cited_title":"Salim, Thomas D","cited_arxiv_id":null,"evidence_quote":"The workflow system that launches and monitors the many reward-estimation tasks; it carries the asynchronous scalability result."}],"review_version":1}