{"id":"cf994a43-e0b1-4e91-af35-5291cf435acd","arxiv_id":"2412.15554","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LC-GODE uses a graph-encoded architecture embedding inside a latent neural ODE to extrapolate learning curves, reporting lower error and better model ranking than architecture-agnostic baselines on four benchmarks.","lead":"This paper introduces LC-GODE, a model that predicts a neural network's future accuracy and loss from its first few training epochs and its architecture. It may make automated model selection faster by ranking candidate architectures after ten epochs instead of full training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central empirical claim lacks uncertainty quantification: reported gains over baselines may be within run-to-run noise.","rationale":"The reader's verdict of CONDITIONAL is reasonable, but the reader's identified weakest assumption -- the informativeness of the graph representation -- is not the most immediate threat to the paper's central claim. The paper demonstrates the architecture-aware model working on NAS-Bench-201 and MLP benchmarks, so the representation is at least sufficient for those settings; its potential failure on other architecture spaces is a generality limitation, not a refutation of the reported results. The more load-bearing issue is that the headline empirical superiority is presented without any measure of variability. Tables I and VI report single runs, and several margins are small enough that they could be reversed by seed noise. Since the paper provides public code, a multi-seed replication is feasible and would either confirm or refute the claim. I agree with the reader that if the public code reproduces the tables across seeds, the method merits acceptance; otherwise the conditional should include significance testing as a requirement. The reader did mention the lack of error bars in the rationale, so there is partial agreement, but the reader's formal weakest_assumption is a different concern.","tokens_in":19748,"tokens_out":9626,"duration_ms":89977,"concrete_test":"Using the released code, rerun every method (LC-GODE, NODE, NSDE, LSTM, VRNN, LC-PFN, LC-BNN) on car, segment, CIFAR-10, and CIFAR-100 with at least 5 independent seeds, reinitializing model weights and re-sampling the train/test split each time. Report mean and standard deviation of MAPE and RMSE at the 200-epoch prediction column for both accuracy and loss, and apply a paired significance test (e.g., Wilcoxon signed-rank or paired t-test) comparing LC-GODE against NODE and against the best non-architecture baseline. The central claim is confirmed only if LC-GODE is significantly better (p < 0.05) in the majority of dataset/metric pairs and the mean gap exceeds one standard deviation in those cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim, stated in Section IV-A, is that LC-GODE outperforms all baselines on all four datasets, reducing test-accuracy MAPE by 36% to 60% and test-loss MAPE by 20% to 65% relative to NODE. Every reported number in Tables I and VI is a single point estimate with no error bars, no number of seeds, and no significance test. This is load-bearing because some advantages are modest: for CIFAR-100 test loss at 200 epochs, LC-GODE gives MAPE 0.0630 versus 0.0723 for NSDE and 0.0709 for LSTM, and for segment test loss at 200 epochs LC-GODE (0.0925) only marginally beats LC-PFN (0.0937). Since the variational training, early stopping, and random data splits introduce substantial stochasticity, the observed ordering could plausibly change under re-initialization or different seeds. The model-ranking results in Table II and the embedding-correlation claim in Figure 2(b) also lack any uncertainty quantification. If the reported advantages are within seed-to-seed noise, the central conclusion that architecture information improves extrapolation is unsupported. This can be settled by replication, so it is a concrete and testable concern rather than a fatal flaw.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LC-GODE, a latent neural ODE model for learning-curve extrapolation in which the drift function is conditioned on a graph embedding of the neural network architecture. The architecture is encoded with GCN layers and a pooling mechanism, the observed early epochs are processed by a GRU-based variational encoder, and the future curve is decoded from the integrated latent state. Experiments are conducted per source task on two OpenML MLP tasks (car, segment) and two NAS-Bench-201 CNN tasks (CIFAR-10, CIFAR-100), with comparisons against six baselines, an ablation study, and a model-ranking analysis. The central claim is that architecture-conditioned ODE extrapolation outperforms all baselines on all four datasets and improves early model selection.","tokens_in":20054,"tokens_out":9643,"duration_ms":82766,"significance":"Architecture-aware learning-curve extrapolation is a timely and relevant topic for AutoML and NAS, and the idea of modulating the latent ODE with a graph-derived embedding is clearly presented and well motivated. The experimental protocol is generally sound: per-task training, held-out architectures, and matched sequence encoders and decoders for the NODE and NSDE ablations, with code and public benchmarks provided. If the reported gains are robust, the method offers a practical 10-epoch tool for model selection with substantial speedups. However, the significance of the empirical contribution depends on the stability of the reported margins, which is not yet established because no seed variation, error bars, or significance tests are reported anywhere in the paper. I agree with the reader's assessment that the design is not circular: architecture embeddings are computed from graphs, not from target curves, and held-out architectures are used for evaluation.","major_comments":[{"comment":"All reported results are single point estimates without error bars, number of seeds, or significance tests. This is load-bearing because the central comparative claim in Section IV-A(c) relies on several small margins: on CIFAR-10 test loss, LC-GODE's MAPE is 0.1487 versus 0.1518 for NSDE at 80 epochs and 0.1629 versus 0.1639 at 200 epochs (Table VI); on segment test loss at 200 epochs, LC-GODE (0.0925) barely beats LC-PFN (0.0937). The abstract's statement that the model quantifies uncertainty through variational parameters is also not evaluated: no predictive intervals, calibration curves, or coverage numbers are reported. Please report mean and standard deviation over at least three to five random seeds and a paired significance test such as Wilcoxon signed-rank or a bootstrap interval for Tables I, II, and VI and for Figures 3, 5, and 6, and discuss the calibration of the predictive distribution.","section":"Tables I, II, VI; Figs. 3, 5, 6"},{"comment":"The claim that LC-GODE 'reduces regret by 96% on CIFAR-10 compared to the superior baseline among NODE and NSDE for test accuracy curves' is not supported by Table II. For CIFAR-10 test accuracy, NODE's regret is 0.0025, NSDE's regret is 0.0101, and LC-GODE's regret is 0.0004; the reduction relative to NODE, which is the superior baseline, is 84%, not 96%. The 96% figure corresponds to comparison with the inferior baseline NSDE. Please correct the percentage or state the comparison baseline explicitly, and provide paired differences for the regret and ranking metrics.","section":"§IV-A(d), Table II"},{"comment":"The sentence 'This improvement is due to the incorporation of architecture information with graph embedding' is an attribution claim that is not directly established by the experiments. LC-GODE differs from NODE and NSDE not only in receiving architecture information but also in that Eq. (9) concatenates z and zG, doubling the input dimension of the ODE function, and in having an additional graph encoder with its own parameters. Without a capacity-matched control, for example a NODE or NSDE variant that receives a random or fixed architecture embedding of the same dimension, or architecture information delivered as a flat feature vector, the observed gains could in principle reflect additional model capacity rather than the graph representation. Please add such an ablation or soften the causal attribution.","section":"§IV-A(c), §III-B(c), Eq. (9)"}],"minor_comments":[{"comment":"Equation (13) defines speedup with 'Runtime for SGD over 52 Epochs' in the numerator, but the experiments train for 200 epochs according to Section IV-A and Appendix A; please reconcile this with the claimed 20x speedup in Section IV-A(d).","section":"Appendix A, Metrics"},{"comment":"The hyperparameter table states Condition Length 20% Tmax and Prediction Length 80% Tmax, while Section IV-A fixes the condition length at 10 epochs; under the mapping t_i = i Tmax/m, 20% of Tmax would correspond to 40 epochs, so the relationship between these settings needs clarification.","section":"Appendix Table V"},{"comment":"The citation placeholders '[?]' and '[NasWOT]' should be resolved to actual references before publication.","section":"Appendix A, Metrics"},{"comment":"The sentence 'We discard runs The parameters randomly sampled include ...' is incomplete; the intended discarding rule is missing.","section":"Appendix A, Datasets"},{"comment":"For directed CNN graphs, the GCN update uses the same degree matrix on both sides of the adjacency matrix; please clarify whether the adjacency matrix is symmetrized before normalization and how the integer edge weights for operation types enter the message passing.","section":"§III-B(b), Eq. (7)"},{"comment":"Minor typographical issues should be corrected: 'fomulated' in Section III-B(d), 'comuted' in Appendix A(b), and 'the the wall-clock time' in Appendix A(e).","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the main risk is not the validity of the method but the strength of the empirical claims relative to the reported precision. The authors should be given the opportunity to add multi-seed results, a capacity-matched ablation, and corrected model-selection numbers. I do not see grounds for rejection, and the problem and approach are appropriate for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a solid empirical paper. The new idea is to condition a latent neural ODE on a graph embedding of the neural network architecture, rather than treating each learning curve in isolation. The combination is not in the cited prior work, and the experiments are designed fairly: per-task training, held-out architectures, and ablations against NODE and NSDE that share the same encoder/decoder. The reported gains on test-accuracy MAPE (36–60% relative improvement over NODE) are large enough that I do not think they are pure noise, and the ranking experiment gives a practical angle. The ablation study and the embedding-correlation analysis are good supporting evidence.\n\nThe soft spot is statistical. Every number in Tables I and VI is a single point estimate with no error bars, no number of seeds, and no significance test. That matters because some of the wins are modest: on CIFAR-100 test loss at 200 epochs, LC-GODE gets 0.0630 versus 0.0723 for NSDE, and on segment test loss at 200 epochs it is 0.0925 versus 0.0937 for LC-PFN. Those margins could plausibly flip under re-initialization. The stress-test note is right that this is load-bearing for the smaller gains, but the accuracy-curve improvements are big enough to survive reasonable seed variance.\n\nThere are also a few internal inconsistencies that need cleanup: the main text sets the condition length to 10 epochs, while the appendix says 20% Tmax; the speedup formula in Eq. (13) mentions \"52 Epochs\" instead of the stated 200; and the appendix has an unfinished sentence. None of these are fatal, but they make the paper look less careful than it otherwise is.\n\nOverall, the central claim—architecture information helps extrapolation—is plausible and mostly supported. The paper is for people working on AutoML, learning curve prediction, and architecture performance prediction. It deserves a serious referee. I would send it out with a request for multi-seed results, error bars, and a reproducibility check of the tables before acceptance. If the public code reproduces the reported numbers, this is a conditional accept.","headline":"A credible, workmanlike paper that pulls architecture information into latent neural ODEs for learning curve extrapolation; the main gains look real, but missing error bars and a few sloppy details keep it short of a clean accept.","tokens_in":20528,"tokens_out":1937,"would_cite":true,"duration_ms":19837,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By encoding a network's architecture as a graph and feeding its embedding into a latent ODE, LC-GODE extrapolates learning curves from ten epochs and cuts error by 20–65% relative to architecture-blind models.","keywords":["learning curve extrapolation","neural architecture search","neural ordinary differential equations","graph neural networks","variational autoencoder","model selection","AutoML"],"falsifier":"Shuffle the graph embeddings: train the model with $z_G$ permuted across architectures (or replaced by a random vector of the same dimension) while keeping all other components identical; if extrapolation error does not rise materially on the held-out set, the claimed architecture signal is not what drives the improvement. A complementary check is cross-space transfer: train on NAS-Bench-201 cell graphs and extrapolate curves for a different cell-based search space; if the 20–65% error reductions vanish, the cell graph is benchmark-specific rather than a general architecture code.","tokens_in":19579,"feed_emoji":"📈","tokens_out":8727,"duration_ms":70618,"temperature":0.7,"pith_summary":"This paper asks whether a neural network's architecture carries useful signal for predicting its own learning curve, and answers yes: if the architecture is turned into a graph (neurons or feature maps as nodes, normalized in/out degrees as node features, operation type as edge weight), a graph embedding can be fed into the vector field of a latent ordinary differential equation that extrapolates the curve from the first 10 epochs. The reason this matters is practical: curve extrapolation drives early stopping, hyperparameter tuning, and neural architecture search, and existing methods model curves in isolation, so any architecture-aware signal is a potential free predictor. On four benchmarks spanning tabular MLPs and image CNNs, the architecture-conditioned model reduces extrapolation error by 20 to 65 percent compared to the same latent-ODE model without architecture information, and it improves the ranking of configurations in model selection. The paper's claim is that the loss landscape differs by architecture, and a low-dimensional graph embedding of that architecture is enough to modulate the ODE that traces the curve.","feed_headline":"Architecture graphs cut learning-curve forecast error by up to 65%","feed_subtitle":"Using only the first 10 epochs, the model forecasts 200-epoch curves and ranks configurations 20x faster than full training.","key_machinery":"LC-GODE (Learning Curve Graph ODE) is an architecture-conditioned latent neural ODE. Its central object is the equation $\\dot z = f_{\\theta_2}([z \\| z_G])$, where $z_G$ is a graph-level embedding of the network architecture and $z$ is a low-dimensional latent state encoding the learning curve's current position. The architecture encoder builds $z_G$ from a cell-based graph representation (nodes are neurons or feature maps, node features are normalized in/out degrees, edge weights encode operation types) using GCN message passing followed by a pooling layer, and the ODE is integrated numerically with a Runge-Kutta solver so the model can query any future time step. The graph embedding modulates the vector field at every integration step, which is how architectural structure influences the predicted trajectory.","core_discovery":"The paper's central claim is that the evolution of a learning curve can be modeled as an autonomous latent ODE whose right-hand side depends on the architecture: $\\dot z = f_{\\theta_2}([z \\| z_G])$. Here $z_G$ is produced by an architecture encoder that runs GCN message passing on the cell graph and pools to one vector, while the latent state $z$ is initialized from a variational posterior $q_{\\phi}(z_{n+1} | \\{y_i,t_i\\}_{i=1}^n)$ computed by a GRU over observed epochs. The decoder maps each integrated latent state to $\\hat y_i$, and the whole system is trained by maximizing the ELBO. Empirically the paper reports that this joint model beats six baselines, including Bayesian methods and latent NODE/NSDE without architecture, on test accuracy and test loss curves, and that including architecture raises the correlation between initial latent distance and final performance difference from 0.77 to 0.83 on CIFAR-10.","pith_inferences":["The paper varies hyperparameters such as learning rate, batch size, and weight decay when generating MLP trials but does not feed them into the ODE; conditioning the same dynamics on a configuration embedding alongside $z_G$ is a direct way to test how much of the remaining error is optimization-settings-driven rather than topology-driven.","Because the CNN representation is a repeated four-node cell, a strong test of generality is whether the learned $z_G$ transfers to a different cell-based search space; if it does not, the improvement may be a benchmark-specific encoding rather than a general architecture signal.","The reported 20x model-selection speedup is computed from ten observed epochs and near-zero inference latency; for settings requiring longer observations or where curves are noisy, the speedup and ranking gains would need to be recomputed rather than assumed."],"forward_implications":["On the paper's four benchmarks, conditioning the latent ODE on the architecture embedding reduces test-accuracy extrapolation error by 36.13%, 30.72%, 34.97%, and 59.63% and test-loss error by 65.5%, 44.61%, 20.1%, and 23.45% relative to the same model without architecture information.","All the model needs is the first 10 epochs and the architecture graph; it can then forecast the remaining 190 epochs, which makes it usable for early stopping and for ranking configurations before full training.","Because the ODE evolves a low-dimensional latent state, the runtime of the forward pass is $O(D^2 T)$ and independent of the network's total size, so prediction cost does not grow with the architecture.","Including architecture information raises the correlation between initial latent-state distance and final performance difference from 0.77 to 0.83 on CIFAR-10, and the predicted best configuration ranks 3–8 positions better on segment and CIFAR-10 with regret reduced by up to 96%."],"supporting_citations":[{"why":"Supplies the Bayesian basis-function extrapolation baseline (LC-BNN) that LC-GODE is compared against and extends.","marker":"Domhan et al., 2015"},{"why":"Provides the gradient-descent-as-ODE derivation that justifies viewing learning curve evolution as a continuous autonomous dynamical system.","marker":"Su et al., 2016"},{"why":"Introduces neural ODEs and adjoint backpropagation, the backbone of the latent ODE solver used here.","marker":"Chen et al., 2018"},{"why":"Supplies the seq2seq variational-autoencoder treatment of latent ODEs from irregularly sampled observations, including the ELBO objective.","marker":"Rubanova et al., 2019"},{"why":"Defines the GCN message-passing layer used to build node representations from the architecture graph.","marker":"Kipf and Welling, 2016"},{"why":"Provides differentiable pooling, one of the graph-level pooling mechanisms used to form the global architecture embedding $z_G$.","marker":"Ying et al., 2018"},{"why":"Contributes the NAS-Bench-201 dataset of CNN cell architectures with 200-epoch learning curves, the CNN benchmark the method is tested on.","marker":"Dong and Yang, 2020"},{"why":"Supplies the OpenML tabular tasks (car, segment) used to generate MLP learning curves.","marker":"Vanschoren et al., 2014"}],"fun_headline_variants":["Architecture graphs sharpen learning-curve forecasts","Graph ODEs predict training curves with architecture clues","Neural nets forecast their own curves via graph ODEs","Architecture-aware ODEs beat existing curve extrapolation","Learning curves forecasted faster with architecture graphs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a coarse graph of the architecture—four nodes in a repeated cell, normalized in/out degrees, and operation-type edges—preserves enough of the architecture's effect on the loss landscape to steer the latent ODE correctly for unseen architectures.","fun_headline_variants_meta":{"raw":{"variants":["Architecture graphs sharpen learning-curve forecasts","Graph ODEs predict training curves with architecture clues","Neural nets forecast their own curves via graph ODEs","Architecture-aware ODEs beat existing curve extrapolation","Learning curves forecasted faster with architecture graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1277,"prompt_tokens":921,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":537,"tokens_out":356,"duration_ms":3827,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:19:00.379760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the graph embeddings: train the model with $z_G$ permuted across architectures (or replaced by a random vector of the same dimension) while keeping all other components identical; if extrapolation error does not rise materially on the held-out set, the claimed architecture signal is not what drives the improvement. A complementary check is cross-space transfer: train on NAS-Bench-201 cell graphs and extrapolate curves for a different cell-based search space; if the 20–65% error reductions vanish, the cell graph is benchmark-specific rather than a general architecture code.","supporting_citations":[],"review_version":1}