{"id":"6fd091a4-c50d-4d01-bdde-41a9f3b826de","arxiv_id":"2411.17339","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"KEGNAS uses a knowledge base of pre-evaluated GNN architectures to generate and rank transfer candidates that warm-start a multi-objective evolutionary search, improving accuracy on several graph datasets.","lead":"The paper introduces KEGNAS, a method that uses prior knowledge from the NAS-Bench-Graph database to seed a multi-objective evolutionary search for graph neural network architectures on new datasets. It combines a dataset-conditioned architecture generator and a deep multi-output Gaussian process surrogate to produce and rank candidate architectures in seconds, then warm-starts NSGA-II with the top candidates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world experiments train all 500 transfer candidates on the target dataset and build a separate Gaussian surrogate, so the claimed 'few GPU seconds' transfer evaluation is never actually used in the reported real-world accuracy gains.","rationale":"The reader's weakest assumption concerns DMOGP prediction error and the fact that real-world evaluation uses a modified algorithm. My focus is sharper: the real-world experiments do not merely 'sidestep' DMOGP; they replace its role entirely by training all 500 candidates on the target and fitting a separate Gaussian surrogate. This makes the reported real-world results a test of 'surrogate-assisted NSGA-II with 500 initial samples,' not of the knowledge-aware transfer mechanism. The paper does include an ablation for DMOGP on NAS-Bench-Graph (Table 7), which is genuine evidence, but the central real-world claim of swift generation of top-performing architectures is not supported because the expensive training of 500 candidates dominates the runtime and the transfer sub-step is not independently credited. A random-sampling control with the same surrogate pipeline would settle whether the transfer is load-bearing; if the control performs equally, the paper's headline overclaims. This does not invalidate the NAS-Bench-Graph results, so the appropriate verdict is CONDITIONAL rather than REJECT: the authors should add the control, report DMOGP prediction quality on held-out datasets, and clarify that the real-world pipeline trains candidates on the target before fitting the surrogate.","tokens_in":27003,"tokens_out":4928,"duration_ms":46106,"concrete_test":"Run a control on the five real-world datasets: replace the knowledge model and DMOGP with uniform random sampling from the NAS-Bench-Graph search space, while keeping everything else identical—train 500 sampled architectures on the target, fit the same Gaussian surrogate, run NSGA-II with the same population size and generations, and retrain final architectures for 400 epochs. If this random-sampled surrogate pipeline matches KEGNAS's reported accuracies (90.81, 90.98, 82.70, 37.84, 53.25), then the transfer components are not responsible for the claimed gains. Additionally, to test whether DMOGP generalizes, compute Spearman rank correlation between DMOGP-predicted and true #Acc on each real-world dataset using only source-task training; if the correlation is near zero or negative, the warm-start selection from predicted metrics is unjustified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that KEGNAS swiftly generates top-performing architectures using the NAS-Bench-Graph knowledge base—rests on the real-world experiments in Section 5.2. But those experiments do not test the proposed transfer mechanism as described. Section 5.2.1 states: 'we modify KEGNAS-NSGAII... we first train all transfer architectures on real-world graph datasets. Subsequently, Gaussian models are constructed using the transfer architectures and their real performance metrics.' Thus, all 500 candidate architectures generated by the knowledge model are fully trained on the target dataset (1.0K–32.0K GPU seconds in Table 6), and a standard Gaussian surrogate is fit to those real labels. The DMOGP predictions—the component that is supposed to make evaluation cheap—are not used to evaluate the candidates; they are only used (implicitly) to choose an initial population, and even that selection could have been replaced by the real training labels once all 500 candidates were trained. The 'few GPU seconds' claim covers only the knowledge model and DMOGP sub-step, but the reported 4.27%/11.54% improvements come from a pipeline that includes expensive per-target training. Crucially, no baseline is run with the same pipeline (train 500 sampled architectures, fit a Gaussian surrogate, run NSGA-II) but without the knowledge model and DMOGP. Without that control, the real-world gains could be attributed to the surrogate-assisted NSGA-II or the larger evaluation budget, not to knowledge transfer. The NAS-Bench-Graph ablations (Section 5.3) do show DMOGP helps, but they use a different evaluation protocol and do not transfer this evidence to the real-world setting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KEGNAS, a knowledge-aware evolutionary graph neural architecture search method. It trains a knowledge model and a deep multi-output Gaussian process (DMOGP) on the NAS-Bench-Graph knowledge base. For a new dataset, the knowledge model generates candidate transfer architectures, the DMOGP predicts their #Acc and #Params, and non-dominated candidates warm-start an NSGA-II search. Experiments on NAS-Bench-Graph with leave-one-out source tasks and on five real-world datasets report accuracy improvements over evolutionary and differentiable baselines; ablations show the knowledge model and DMOGP each contribute.","tokens_in":99,"tokens_out":10147,"duration_ms":334343,"significance":"The idea of reusing NAS benchmark knowledge to warm-start multi-objective GNAS is timely and practically relevant, and the paper ships its code. The NAS-Bench-Graph leave-one-out experiments (Section 5.1, Tables 2 and 7) are clean: the source tasks exclude the target, and the ablation against NSGA-II and random-transfer warm starting isolates the contributions of the knowledge model and DMOGP. However, the real-world experiments (Section 5.2) evaluate a modified pipeline that trains all transfer architectures on the target and then uses a Gaussian surrogate; this weakens the central claim that KEGNAS swiftly generates top-performing architectures via the knowledge base. The paper is a solid candidate after the real-world experimental gap is addressed.","major_comments":[{"comment":"The real-world experiments do not evaluate the proposed Algorithm 2 as described. The paper states that \"we modify KEGNAS-NSGAII... we first train all transfer architectures on real-world graph datasets. Subsequently, Gaussian models are constructed using the transfer architectures and their real performance metrics.\" Thus all 500 candidate transfer architectures are actually trained on the target dataset (pre-search cost 1.0K–32.0K GPU seconds in Table 6), and the DMOGP predictions are not used to evaluate the candidates. The \"few GPU seconds\" claim therefore covers only the generation and DMOGP inference sub-step, while the accuracy numbers in Table 5 are produced by a pipeline whose dominant cost is training the transfer architectures. This mismatch means the real-world experiments do not support the abstract's claim that KEGNAS \"swiftly\" generates top-performing architectures through the knowledge base.","section":"Section 5.2.1 and Table 6"},{"comment":"The real-world setup lacks a control that isolates the effect of prior knowledge. Since the Gaussian surrogate is built from the real performance of all 500 transfer architectures, the reported gains over CTFGNAS and SANE could in principle be obtained by any surrogate-assisted NSGA-II fed with 500 randomly sampled trained architectures. A control experiment running the same pipeline (train 500 randomly sampled architectures, fit the same Gaussian surrogate, run NSGA-II) is necessary to attribute the 4.27% and 11.54% improvements to the knowledge model and DMOGP rather than to the surrogate-assisted search or the larger number of target trainings. The ablation in Section 5.3 is performed only on NAS-Bench-Graph, so it does not close this gap.","section":"Section 5.2.1 and Table 5"},{"comment":"The real-world evaluation budgets are not matched across baselines. KEGNAS-NSGAII trains 500 transfer architectures plus retrains the selected architecture for 400 epochs, while Random trains 100 architectures and F2GNN/SANE use supernet-style training. Because the accuracy comparison in Table 5 is not conditioned on equal compute or an equal number of target model trainings, the reported percentage gains are not an apples-to-apples comparison. The authors should either match the number of trained architectures across methods or report accuracy as a function of total GPU time for all methods.","section":"Section 5.2.1 and Table 6"}],"minor_comments":[{"comment":"The text says the macro space of NAS-Bench-Graph contains eight DAGs, but it then lists nine labels (A, B, C, D, E, F, G, H, I). Please correct the count or the list.","section":"Section 2.2"},{"comment":"The symbol Σ' is used both for the noise matrix in the mean formula and for the predicted covariance on the left-hand side; please use distinct notation for these two quantities.","section":"Section 2.5, Eq. (8)"},{"comment":"The Cornell row for Genetic-GNN reports \"54.K\", which appears to be a typo for \"54.0K\" or \"54K\"; also the Flickr KEGNAS row uses a lowercase 'k' in \"32.0K\" while other entries use uppercase.","section":"Table 6"},{"comment":"The Gaussian surrogate model used in the modified real-world experiments is not specified (kernel choice, hyperparameter fitting, validation strategy). Please provide these details for reproducibility.","section":"Section 5.2.1"},{"comment":"The sentence \"generating and evaluating candidate transfer architectures requires only a few GPU seconds in total, which is negligible compared with the search cost of NSGAII\" is misleading in context, because the pre-search cost in Table 6 is dominated by training the transfer architectures, not by the few seconds of DMOGP inference.","section":"Section 5.2.2"},{"comment":"The paper does not report any measure of DMOGP predictive accuracy (e.g., rank correlation or RMSE) on held-out or real-world datasets; such a measurement would directly support the claim that transfer architectures are selected by accurate predicted metrics.","section":"Sections 4.2 and 5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of Knowledge-Based Systems and the NAS-Bench-Graph experiments are convincing. The main issue is that the real-world evaluation is not of the proposed algorithm as described, and the comparison lacks a knowledge-free surrogate-assisted control. I would like the editor to require the authors to close this gap before publication. The code availability and the clean leave-one-out setup are strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on KEGNAS. The core idea is sensible: use NAS-Bench-Graph as a knowledge base to train a dataset-conditioned graph generator and a deep multi-output GP, then use the GP's predicted non-dominated architectures to warm-start NSGA-II. The combination is new for graph NAS, and the paper does the obvious benchmark test cleanly: leave-one-out on the eight NAS-Bench-Graph datasets, with the same evaluation budget for all methods, and ablations that separate the knowledge model from the DMOGP. On that evidence, warm-starting helps, and the DMOGP gives a modest but real improvement over randomly selected transfer architectures. That part holds up.\n\nThe soft spot is the real-world section. Section 5.2.1 says explicitly that they train all 500 transfer architectures on the target dataset, fit a standard Gaussian surrogate to those labels, and replace the NSGA-II evaluation with that surrogate. So the reported 4.27% and 11.54% gains come from a pipeline where the DMOGP is not used to evaluate anything; it only picks the initial population, and even that could have been replaced by the real labels once the 500 are trained. The 'few GPU seconds' claim covers only the generation-and-DMOGP substep, not the 1K-32K GPU seconds spent training the transfer candidates. There's no control baseline that runs the same surrogate-assisted NSGA-II without the knowledge model, so the real-world gain could be from the surrogate or the larger evaluation budget rather than from transfer. The paper also never reports DMOGP prediction error on the target datasets, which is the load-bearing assumption of the whole transfer idea.\n\nNone of this breaks the NAS-Bench-Graph results, which use the actual evaluation protocol. But it means the real-world experiments don't test the system as described, and the headline 'swiftly generates top-performance architectures' is not supported by what was actually run. The fix is straightforward: add a no-transfer control with the same surrogate pipeline, and either use the DMOGP for evaluation on the target (and measure its accuracy) or present the real-world numbers as a different variant. The paper also owes a comparison with the closest transfer-NAS baselines from the cited literature (Lee et al., Shala et al.), since it directly borrows from both.\n\nThis is a solid first draft for a graphs/NAS audience. I'd send it to review but require the real-world experiment to be fixed and the missing controls added. Worth a reading group slot if you care about transfer optimization in NAS.","headline":"Real-world experiments don't test the transfer mechanism as described, but the NAS-Bench-Graph evidence supports the core idea; the paper deserves revision, not rejection.","tokens_in":27945,"tokens_out":2768,"would_cite":false,"duration_ms":24958,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained on the NAS-Bench-Graph knowledge base, KEGNAS generates and evaluates transfer architectures in a few GPU seconds and warm-starts a multi-objective evolutionary search, outperforming strong GNAS baselines by 4.27% to 11.54%…","keywords":["graph neural architecture search","evolutionary transfer optimization","multi-objective optimization","deep multi-output Gaussian process","NAS-Bench-Graph","warm-start evolutionary algorithm","graph neural networks","surrogate-assisted search"],"falsifier":"Train the DMOGP and knowledge model on the eight NAS-Bench-Graph datasets, then on a held-out graph dataset compute the rank correlation between DMOGP-predicted and true #Acc and #Params; if the transfer architectures selected by the predicted Pareto front are no better than randomly sampled architectures when used to warm-start the same MOEA, the claimed transfer gain is not real.","tokens_in":26798,"feed_emoji":"🧬","tokens_out":8108,"duration_ms":69763,"temperature":0.7,"pith_summary":"KEGNAS sets out to end the zero-knowledge state of graph neural architecture search (GNAS): instead of starting every new graph dataset from scratch, it reuses a public architecture-and-performance database, NAS-Bench-Graph, to seed the search. The paper's central claim is that this prior knowledge can be transferred cheaply through two components, a knowledge model that maps a dataset to candidate architectures and a deep multi-output Gaussian process that predicts accuracy and parameter count for those candidates. Only the non-dominated candidates are fed into a multi-objective evolutionary algorithm as its initial population. On eight NAS-Bench-Graph datasets and five real-world graph datasets, the paper reports that this warm start produces architectures with 4.27% higher average accuracy than the strongest evolutionary baseline and 11.54% higher than the strongest differentiable baseline, while the transfer-architecture generation and evaluation stage itself takes only a few GPU seconds. The importance, if true, is that expensive per-dataset architecture search can be accelerated by standing on accumulated benchmark knowledge rather than beginning each task in ignorance.","feed_headline":"Prior knowledge lifts graph NAS accuracy by up to 11.54%","feed_subtitle":"A knowledge model and surrogate predictor warm-start evolutionary search, reaching better GNNs in a few GPU seconds.","key_machinery":"The machinery is a two-model warm-start pipeline. A knowledge model, built on a DAG graph decoder, learns the mapping from a dataset's task-feature distribution (computed by a variational graph autoencoder) to Pareto-optimal architectures in NAS-Bench-Graph, so that a new dataset can quickly generate candidate architectures. A deep multi-output Gaussian process, a multi-output GP whose kernel is parameterized by a neural network over architecture and dataset encodings, predicts #Acc and #Params for each candidate on the new dataset without training the candidates. Fast non-dominated sorting on these predicted metrics selects the transfer architectures that initialize a multi-objective evolutionary algorithm, using SBX crossover and PM mutation. This placement matters: the two learned models only shape the initial population, leaving the underlying MOEA's evaluation, selection, and reproduction untouched, so the framework can wrap any MOEA.","core_discovery":"The paper's discovery is that a database of evaluated GNN architectures can be turned into a reusable prior for new graph tasks. After training once on NAS-Bench-Graph, KEGNAS uses a variational graph autoencoder to represent each dataset by a task-feature distribution, a DAG-decoder knowledge model to generate candidate transfer architectures for a new dataset, and a deep multi-output Gaussian process with learned deep kernels to predict each candidate's #Acc and #Params without training it. Fast non-dominated sorting selects the predicted Pareto candidates, and those architectures initialize an NSGA-II population that then continues the search normally. The reported result is that this pipeline outperforms six established GNAS baselines on NAS-Bench-Graph, and on five real-world datasets it exceeds the best surrogate-assisted evolutionary baseline by 4.27% average accuracy and the leading differentiable baseline by 11.54%, while the warm-start stage costs a few GPU seconds. Ablations attribute the gain to both the knowledge model and the DMOGP: removing the surrogate's selection or the knowledge transfer entirely reduces the hypervolume of the final architecture set.","pith_inferences":["The paper reports the transfer-architecture generation and evaluation time separately from the search time, but the real-world experiments in Section 5.2.1 build an additional Gaussian surrogate from trained transfer architectures before running NSGA-II; a careful reader should therefore treat the headline speedup as warm-start plus surrogate-assisted search, and expect the warm-start-only gain to","The same recipe could be applied to other public NAS benchmarks whose search spaces are DAG-encodable: train the knowledge model and DMOGP on the benchmark's evaluated architectures, then warm-start search on a new task, though the paper's own limitation discussion notes that heterogeneous search spaces and non-node-classification tasks would require new search spaces and training data.","A sharper test of the transfer hypothesis than the paper reports would be to measure DMOGP rank correlation on genuinely out-of-distribution datasets; if the surrogate's predictions are poor there, the warm start degrades toward random initialization, which is precisely the failure mode the ablation does not fully exclude."],"forward_implications":["GNAS no longer needs to be a from-scratch process: any future task can inherit knowledge stored once from NAS-Bench-Graph, so a new dataset's search starts from a population already enriched with plausible Pareto architectures.","The warm-start step is cheap by design, since generating and evaluating 500 candidate transfer architectures costs only a few GPU seconds, an order of magnitude below the search cost of the MOEA itself.","Because the knowledge model and DMOGP are decoupled from the evolutionary loop, KEGNAS can be combined with any MOEA, not just NSGA-II, without disturbing existing evaluation strategies.","The ablation results imply that both learned components matter: randomly chosen transfer architectures and randomly initialized populations both produce lower-hypervolume architecture sets on NAS-Bench-Graph.","On real-world datasets, the same warm-started framework stays competitive or better than state-of-the-art differentiable and RL-based search methods while using much less search time than most evolutionary baselines."],"supporting_citations":[{"why":"Supplies NAS-Bench-Graph's architecture library and per-dataset #Acc/#Params metrics, the prior knowledge base that KEGNAS trains on and searches.","marker":"[21]"},{"why":"VGAE is used to extract the task-feature distribution p(s|D_b) that represents each dataset for the knowledge model and DMOGP.","marker":"[66]"},{"why":"D-VAE provides the DAG graph decoder used to generate architectures and the graph encoder used to embed them for the surrogate.","marker":"[67]"},{"why":"NSGA-II is the embedded multi-objective evolutionary solver and its fast non-dominated sorting selects transfer architectures and ranks the final population.","marker":"[65]"},{"why":"Establishes the approach of learning to generate graphs from dataset features, which the knowledge model directly adapts to GNAS.","marker":"[57]"},{"why":"Provides the transfer-NAS-with-meta-learned-surrogates template, including deep-kernel surrogate training, that the DMOGP builds on.","marker":"[14]"},{"why":"Defines the multi-output Gaussian process with multi-output covariance that the DMOGP extends with deep-learned kernels.","marker":"[62]"}],"fun_headline_variants":["Prior knowledge warm-starts GNAS, gains up to 11.54% accuracy","Knowledge-aware GNAS beats differentiable baselines by 11.54%","Reuse GNN architecture databases to warm-start evolutionary search","Warm-start graph NAS with prior knowledge, not scratch"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's load-bearing premise is that knowledge of good architectures on the eight NAS-Bench-Graph datasets transfers to a new dataset: the knowledge model must generate candidates whose predicted metrics are accurate enough that the non-dominated warm-start population beats a random initial population.","fun_headline_variants_meta":{"raw":{"variants":["Prior knowledge warm-starts GNAS, gains up to 11.54% accuracy","Knowledge-aware GNAS beats differentiable baselines by 11.54%","Reuse GNN architecture databases to warm-start evolutionary search","Warm-start graph NAS with prior knowledge, not scratch"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3602,"prompt_tokens":1051,"completion_tokens":2551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":2475}},"tokens_in":667,"tokens_out":2551,"duration_ms":17200,"temperature":1.0,"reasoning_tokens":2475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:13:00.371391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the DMOGP and knowledge model on the eight NAS-Bench-Graph datasets, then on a held-out graph dataset compute the rank correlation between DMOGP-predicted and true #Acc and #Params; if the transfer architectures selected by the predicted Pareto front are no better than randomly sampled architectures when used to warm-start the same MOEA, the claimed transfer gain is not real.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies NAS-Bench-Graph's architecture library and per-dataset #Acc/#Params metrics, the prior knowledge base that KEGNAS trains on and searches."},{"cited_title":"Zhang, S","cited_arxiv_id":null,"evidence_quote":"D-VAE provides the DAG graph decoder used to generate architectures and the graph encoder used to embed them for the surrogate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the approach of learning to generate graphs from dataset features, which the knowledge model directly adapts to GNAS."},{"cited_title":"Shala, T","cited_arxiv_id":null,"evidence_quote":"Provides the transfer-NAS-with-meta-learned-surrogates template, including deep-kernel surrogate training, that the DMOGP builds on."}],"review_version":1}