{"id":"01f93fb1-5697-4f98-8035-34e4d433649e","arxiv_id":"2411.16975","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ExpTest auto-selects and tunes the learning rate by testing whether the training loss decays exponentially, without needing an initial learning rate choice.","lead":"ExpTest is a new algorithm that automatically finds a good starting learning rate for neural network training and adjusts it during training, using statistical tests on the loss curve. It removes the need to manually search for an initial learning rate, which could save time and resources for non-expert users.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ExpTest's F-test does not require the fitted exponential to decay: an exponentially growing loss can pass the convergence test, undermining the initial LR search; the non-nested nonlinear F-test is also uncalibrated.","rationale":"The reader's weakest assumption concerns the single-exponential model and the window-size derivation; I agree that the window heuristic is fragile, but the more load-bearing flaw is in the F-test itself. Even if the loss curve were a perfect single exponential, the test does not verify that the exponent is negative. A diverging loss curve, which is precisely the failure mode a learning-rate search must catch, can be misclassified as convergent. This flaw is upstream of the window-size concern, because it affects the correctness of the decision conditional on the assumed model. The non-nested F-test is also a statistical validity issue that the paper does not acknowledge. My recommendation remains a conditional acceptance: the central idea is plausible, but the paper should either fix the exponential test to require B > 0 and a decreasing loss, or provide a calibrated and validated decision rule. The empirical demonstrations on three small tasks are not sufficient to support the claims of state-of-the-art performance and architecture independence. Therefore I do not change the reader's CONDITIONAL verdict, but I sharpen the specific technical condition that must be resolved.","tokens_in":18037,"tokens_out":9993,"duration_ms":103248,"concrete_test":"Use the authors' provided code to run ExpTest's fitting and F-test procedure on two synthetic linear-regression loss curves: (1) a known stable exponential decay from Section II.A, and (2) an identical problem with learning rate set 10% above the stability bound so the loss grows approximately exponentially over the first w iterations. Record whether the F-test yields p < α and whether ExpTest continues at the high learning rate or reinitializes with βη. Additionally, generate 10,000 independent windows of iid Gaussian noise with no trend and compute the fraction of windows where the exponential-vs-linear F-test reports p < 0.05; if the empirical rate is not close to 5%, the p-value is uncalibrated and the α=0.05 threshold is meaningless.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central decision in Algorithm 1 is the F-test comparing an exponential fit f1 = A e^{-Bt} + C to a linear fit f2 = Dt + E over a window of w iterations (Eq. 52). The test asks whether the exponential model reduces the sum of squared errors enough to be statistically significant. Two flaws make this decision unreliable. First, the exponential fit is unconstrained in the sign of B. If the learning rate is too high and the loss diverges, the loss curve can be approximately exponential growth (B < 0), which will often fit the data far better than a line. The F-test then returns p < α, the algorithm disables exponential testing, and it continues training with a diverging learning rate instead of restarting with a smaller η. The subsequent linear-slope test may eventually catch the increase, but only after the loss has grown substantially and training progress has been wasted. Second, the linear model is not a special case of the exponential model (the exponential's B=0 limit is constant, not linear), so the models are non-nested; the F-distribution used for the test does not apply. Moreover, the residuals are successive mini-batch losses, which are correlated and heteroscedastic, so the nominal α=0.05 is not the actual false-positive rate. The paper provides no calibration or validation of this statistical test. Because the F-test is the sole mechanism for accepting or rejecting the initial learning rate, these flaws directly threaten the claim that ExpTest 'automatically finds a working initial learning rate' and is 'robust to hyperparameter choice.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ExpTest, an SGD variant that aims to remove the need to preselect an initial learning rate or a learning-rate schedule. It uses a linearized-network/NTK argument to motivate that a convergent training loss decays approximately exponentially, computes an upper-bound learning rate from the input covariance, and then runs an F-test comparing an exponential fit with a linear fit on a window of the early loss curve. If the exponential fit is not significantly better, the model is reinitialized at a reduced learning rate; once exponential decay is detected, later plateaus trigger further reductions based on a t-test of the linear slope. Experiments cover MNIST logistic regression, California Housing fully connected regression, and CIFAR-10 with a modified VGG-16, comparing against SGD, momentum SGD, Adam, RMSprop, and Adadelta, with ablations for alpha, beta, and batch size. The paper claims minimal overhead, robustness to hyperparameter choice, and state-of-the-art performance without initial learning rate selection or scheduling.","tokens_in":18387,"tokens_out":10690,"duration_ms":104989,"significance":"If the core claim holds, ExpTest would be a practically useful contribution: it turns a manual learning-rate search into an automated early-loss-curve test while remaining a simple wrapper around SGD. The manuscript's strengths include publicly released code and data, explicit robustness checks for alpha, beta, and mini-batch size, and a clear mathematical motivation from linearized training dynamics. However, the central statistical decision mechanism is not currently validated: the F-test is used outside its assumptions, and the test can accept exponentially growing losses. The empirical results are suggestive but do not yet establish the advertised 'automatic initial learning rate search' claim.","major_comments":[{"comment":"The F-test does not constrain the sign of B in the exponential fit f1 = A e^{-Bt} + C. When the learning rate is too high and the loss diverges, an exponential-growth model (B < 0) will often fit the window better than a line; the test then returns p < alpha, disables exponential testing, and training continues at a diverging learning rate instead of restarting. The subsequent t-test on the linear slope may eventually detect the rise, but only after wasted iterations. Please constrain B >= 0, or explicitly test whether the fitted exponential is decaying, and show on synthetic divergent loss curves that the first-window decision rejects them.","section":"Algorithm 1, Step 'Fit and Evaluate' (Section III)"},{"comment":"The F-test is statistically uncalibrated. The linear model f2 = Dt + E is not a special case of the exponential model (the B -> 0 limit is constant, not linear), so the models are non-nested and the F-distribution used for the test does not apply. In addition, successive mini-batch losses are serially correlated and heteroscedastic, so the nominal alpha = 0.05 is not the actual false-positive rate. Because this test is the sole decision rule for accepting the initial learning rate, the paper needs a proper null model, a calibration study against synthetic exponential, linear, and divergent loss curves, and a precise statement of the F-statistic and its degrees of freedom.","section":"Algorithm 1, Step 'Fit and Evaluate' (Section III)"},{"comment":"The derivation of the window size assumes that the early loss is well approximated by a single exponential with amplitude bounded by L0. The approximation of a sum of exponentials by one exponential is uncontrolled and can be poor when decay rates are disparate, and the window formula t_max = sqrt(2 C_exp)/e is derived for an idealized single-exponential curve. For finite-width networks with multiple decay modes, or when NTK linearization is inaccurate, the maximum-curvature time may fall outside the computed window, so the first F-test may miss a convergent learning rate and trigger a wasteful restart. The paper should provide a bound relating the true early loss to the single-exponential model, or empirically validate the window choice on the three tasks.","section":"Section III, Eq. (52); Section II.A, Eqs. (23)-(31); Appendix D"},{"comment":"The t-test logic in the pseudocode is inconsistent with the prose. The text says the one-tailed t-test checks whether the slope is significantly less than zero and decreases the learning rate if not, but the pseudocode 'One-tailed t-test if A > 0: t(A, 0); if p < alpha then Update start time; else Reinitialize' would continue training when the slope is significantly positive and restart when it is not, which is the opposite of the intended plateau detection. Please fix the pseudocode and state the null and alternative hypotheses explicitly.","section":"Algorithm 1, Step 5 (t-test branch)"},{"comment":"The architecture for the California Housing experiment is selected on the test set: Appendix E reports that several width-depth models were evaluated on the test set and the model with (width, depth) = (2, 32) was chosen because it 'consistently displays the lowest test loss.' This is a form of test-set leakage that makes the reported test MSE for ExpTest and the baselines optimistically biased. The architecture should be selected using the validation set only, or with nested cross-validation, before reporting test performance.","section":"Appendix E and Section IV.B"},{"comment":"The derivation of the CE-loss learning-rate bound uses a second-order approximation and replaces the Hessian by its maximum eigenvalue (lambda_max,H). The paper states lambda_max,H > 1 and therefore 2/(lambda_max,H lambda_max,Sigma) < 2/lambda_max,Sigma, but it does not show that this bound applies to the true CE gradient-flow dynamics, nor that the bound is usefully tight. Because eta_max is the algorithm's starting point and also enters the window formula, this derivation should be made rigorous or replaced by an empirically validated bound with a sensitivity analysis.","section":"Appendix C"}],"minor_comments":[{"comment":"The word 'comonly' should be 'commonly' in the first paragraph.","section":"Introduction"},{"comment":"The nearest-integer rounding convention should be defined explicitly; the expression floor(2 sqrt(2 L0) / (eta e) + 1/2) is not standard notation for rounding to nearest and should be clarified.","section":"Eq. (52) and Algorithm 1"},{"comment":"The ExpTest row is identical in every learning-rate column; the caption should state that ExpTest was not rerun at each eta but rather chooses its own trajectory, to avoid implying duplicated experiments.","section":"Table 3"},{"comment":"The symbol lambda_max is used both for the largest eigenvalue of the covariance matrix and for the eigenvalue that maximizes the curvature time; please rename one of them to remove ambiguity.","section":"Appendix D"},{"comment":"The experiments do not compare ExpTest against standard learning-rate schedulers such as cosine annealing, step decay, or ReduceLROnPlateau; given the abstract's 'without learning rate scheduling' claim, this comparison should be added or the claim should be softened.","section":"Experimental comparison"},{"comment":"The window correction c_w = S_mag / ||S_vec|| is undefined when the sum of gradient vectors vanishes; please add a safeguard for this edge case.","section":"Eq. (53)"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript addresses a relevant problem and is generally readable. The main reservation is the F-test, which is statistically invalid as presented and can accept divergent runs; these issues are fixable with a constrained exponential fit and a calibration study. I also note the architecture selection on the test set in Appendix E, which should be corrected. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, ExpTest is a genuinely new combination: it uses a linearized-network learning-rate upper bound to set the initial LR, then uses F- and t-tests on the training loss curve to decide when to anneal or restart. That is a fresh, practical idea. Second, the statistical core is not as solid as the presentation suggests, and the experiments do not back the phrase \"state-of-the-art.\"\n\nWhat is good: the linear-network derivation is standard but clean, and the authors correctly extend it to SGD via expectations. They ship code and data, and they show robustness to their two hyperparameters (alpha, beta) in the tested ranges. On MNIST logistic regression and CIFAR-10 VGG-16, ExpTest matches well-tuned Adam and SGD-with-momentum. That is a useful capability if it holds up.\n\nNow the soft spots. The F-test compares a linear fit to an exponential fit, but those models are not nested, so the F-distribution is not valid. The residuals are successive mini-batch losses: correlated and heteroscedastic, so the nominal alpha is not the real false-positive rate. The paper does not calibrate this test. More troublingly, the exponential fit is unconstrained in the sign of B. A diverging loss often grows roughly exponentially, so the F-test can return p