{"id":"687b0d71-5115-4699-97b4-df62d468d355","arxiv_id":"2505.24005","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Calibrated learning-rate-free optimizers Prodigy and Mechanic still trail a tuned NadamW baseline on the AlgoPerf benchmark.","lead":"A group of optimizers designed to remove the learning rate tuning step still needed careful tuning of other settings, and even then scored slightly below a well-tuned standard optimizer on an eight-workload benchmark. The result suggests current 'learning-rate-free' methods do not yet save real tuning effort.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that learning-rate-free methods show no cross-workload generalization is tested on the same AlgoPerf workloads used for calibration, and the reported 'slightly worse' gap has no seed-level uncertainty.","rationale":"The central claim in the reader's strongest_claim has two parts: (i) LRF methods don't save tuning time and (ii) they don't provide significant cross-workload generalization. Part (ii) cannot be established by the current design. Section 3.2 acknowledges no held-out workloads, and all final scores in Table 6 are computed on the calibration set. The 'calibrated default' is therefore selected and evaluated on the same data, which is the textbook setup for overfitting a default. The asymmetry in tunable dimensions makes this non-trivial: AdamW/NadamW can use the base-learning-rate dimension to fit the aggregate behavior of the 8 workloads, while Prodigy/Mechanic must rely on their internal adaptivity. On unseen workloads, the balance could shift. The paper's self-characterization in Section 5.2 as 'not demonstrably better' is appropriately cautious, but Section 6 and the reader's strongest_claim go further by saying LRF 'don't provide significant cross-workload generalization,' which the experiment cannot support. A leave-one-workload-out recalibration is the natural check: it keeps the paper's calibration procedure intact while forcing the generalization claim to be tested out-of-sample. The step-budget issue raised by the reader is less threatening: the paper says LRF methods take about 10% more time per step, so capping them at AdamW's step count gives them more wall-clock time than a fair time budget would. If anything, a wall-clock-corrected comparison would make the LRF lag larger, not overturn it. Thus verdict remains CONDITIONAL but for a different reason than the reader's weakest_assumption.","tokens_in":23063,"tokens_out":11702,"duration_ms":125207,"concrete_test":"Leave-one-workload-out recalibration: for each of the 8 AlgoPerf workloads, run the full Section 3.2 calibration (200-point quasi-random search, top-3 selection, 5-seed scoring) on the remaining 7 workloads, then evaluate the selected configuration on the held-out workload with 5 seeds. Aggregate the held-out time-to-target fractions into performance profiles for Prodigy, Mechanic, AdamW, and NadamW. If LRF methods do not beat or tie the baselines on held-out workloads, the 'no cross-workload generalization' claim is supported; if they do, the conclusion must be revised. Report per-seed scores so the small Table 6 gap can also be checked for significance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states explicitly: 'Our setup doesn't have any held-out workloads; we used the whole set of AlgoPerf base workloads to find our cross-workload hyperparameter settings.' The final comparison in Table 6 then scores those same AlgoPerf-calibrated configurations on those same workloads. The strongest conclusion in Section 6 — that learning-rate-free methods 'don't provide significant cross-workload generalization' — therefore rests entirely on in-sample behavior. This matters because AdamW/NadamW have an extra tunable dimension (base learning rate, Table 3); with the same 200 search points, the baselines have more freedom to fit the 8 calibration workloads, so their in-sample advantage may not transfer. The observed gap is also small (best AdamW score 0.498 vs best Prodigy 0.477 in Table 6) and no per-seed error bars are reported, so 'slightly worse' cannot be distinguished from noise. The reader's step-budget concern is real but points in the paper's favor: since LRF methods take about 10% more time per step (Section 3.1), granting them the same step budget gives them extra wall-clock time; correcting this would make LRF look worse, not better. The load-bearing gap is the absence of a held-out evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically evaluates learning-rate-free optimizers (Prodigy, Mechanic, DoG, MoMo, COCOB, D-Adapt, DoWG) as building blocks for hyperparameter-free training algorithms. Using the eight workloads of the AlgoPerf self-tuning track, the authors first test the algorithms with literature-default hyperparameters and find they perform poorly. They then perform a quasi-random search over the remaining non-learning-rate hyperparameters (weight decay, momentum, dropout, label smoothing, warmup, training horizon), calibrated jointly across all eight workloads, and compare the resulting 'AlgoPerf-calibrated' configurations against AdamW and NadamW baselines calibrated under the same procedure. The best learning-rate-free methods (Prodigy and Mechanic, wrapped around Adam) improve substantially but still achieve slightly lower AlgoPerf benchmark scores than the best baselines (0.477 vs. 0.498). The authors conclude that current learning-rate-free methods do not yet save tuning time and do not provide significant cross-workload generalization over conventional methods, and they identify the training-horizon/schedule choice as an important remaining tuning requirement.","tokens_in":23340,"tokens_out":5072,"duration_ms":49813,"significance":"The study is a useful, carefully executed benchmark of learning-rate-free optimizers in the practically important setting of workload-agnostic hyperparameter selection. Its strengths include a clearly specified search space (Table 3), the same calibration budget and scoring metric for all methods, a diverse set of eight workloads, and an explicit acknowledgment of several limitations (no held-out workloads, step-budget conversion). The paper also provides a good-faith approximate comparison to the competition-winning Schedule-Free Adam (Appendix F). If the results hold, they constitute a solid negative result that should inform future work on hyperparameter reduction. The main weakness is that the headline generalization claim is not supported by the experimental design, which uses the same workloads for calibration and evaluation; this is acknowledged in Section 3.2 but the Section 6 conclusion overstates what the evidence shows.","major_comments":[{"comment":"The claim that learning-rate-free methods 'don't provide significant cross-workload generalization' is not supported by the experimental design. Section 3.2 explicitly states: 'Our setup doesn't have any held-out workloads; we used the whole set of AlgoPerf base workloads to find our cross-workload hyperparameter settings.' Because the same eight workloads are used for both calibration and final evaluation, the results are in-sample. To support a claim about generalization to unseen workloads, the authors should either add a held-out evaluation (e.g., leave-one-workload-out or additional test workloads) or temper the conclusion to say that the methods underperform on the calibration workloads themselves.","section":"Section 6 (Discussion), with Section 3.2"},{"comment":"The reported AlgoPerf benchmark scores in Table 6 have no uncertainty quantification. The final selection used five independent random seeds, but only a single score per configuration is presented. The observed gap between the best AdamW (0.498) and best Prodigy (0.477) is small, and without per-seed scores, confidence intervals, or standard errors, the claim that learning-rate-free methods perform 'slightly worse' cannot be distinguished from noise. Please report seed-level results or error bars for the key comparisons.","section":"Table 6, with Section 3.2"},{"comment":"The paper claims to produce 'AlgoPerf-calibrated' default hyperparameters for the learning-rate-free methods, but the actual calibrated configurations for Prodigy and Mechanic are not disclosed anywhere. Table 7 reports only the baseline candidate configurations, and Appendix E reports scores and runtime fractions without hyperparameter values. Without these values, the study is not reproducible and practitioners cannot use the proposed evidence-based defaults. Please include the final calibrated hyperparameter settings for all methods.","section":"Table 7 and Appendix E"}],"minor_comments":[{"comment":"The phrase 'learning-free-learning algorithms' appears to be a typo; it should read 'learning-rate-free learning algorithms'.","section":"Section 1"},{"comment":"The performance profile definition uses 'time taken by the training algorithm on workload w,' but the experimental setup (Section 3.1) converts wall-clock budgets to step counts based on AdamW. Please clarify whether the reported time-to-target ratios are computed from step counts or from wall-clock time, since learning-rate-free methods take about 10% longer per step.","section":"Section 3.3"},{"comment":"The sentence 'Schedule-Free Adamwas intr' is an incomplete fragment and should be completed or removed.","section":"Appendix E"},{"comment":"The decision to omit D-Adapt and DoWG is justified by their similarity to Prodigy and DoG, but no experimental evidence is provided; consider adding a brief description of the initial experiments that motivated this exclusion.","section":"Section 5.1"},{"comment":"The reference 'Anonymous (2024)' is unusual for a journal submission; if this is an under-review work, please provide a full citation or explain the blinding convention.","section":"References"},{"comment":"The caption states 'Up arrow indicates metric is maximized, down arrow minimized,' but the table as rendered does not visibly include arrows in the metric column headers; please ensure the final formatting matches the caption.","section":"Table 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper originates from the group that created the AlgoPerf benchmark, and the scoring methodology is the authors' own prior work (Dahl et al., 2023). This is not a conflict per se, but the editorial process should weigh whether the benchmark design is treated too uncritically and whether the absence of held-out workloads is adequately reflected in the paper's conclusions. The paper's fit with the journal is reasonable for an empirical machine learning venue, though the lack of uncertainty quantification and undisclosed hyperparameters are substantive reproducibility issues that should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the learning-rate-free benchmark paper. The useful result is a clean negative: after a systematic search over non-learning-rate hyperparameters, the best LRF optimizers (Prodigy, Mechanic) still trail a similarly tuned NadamW/AdamW baseline on the AlgoPerf suite. That is worth knowing, because literature defaults for these optimizers are often used as-is, and the paper shows those defaults perform poorly on a diverse workload set. The experimental design is fair: both baselines and LRF methods get the same number of search points, and the baselines have the extra base-learning-rate dimension, which if anything should help them. The calibration procedure and the performance-profile scoring are standard and appropriate.\n\nThe main soft spot is the one the authors acknowledge: there are no held-out workloads. Section 3.2 states this explicitly, and the final comparison in Table 6 scores the same workloads used for calibration. So the phrase in the discussion about \"no significant cross-workload generalization\" is not actually supported by the experimental design. What the data show is that, even on the calibration workloads, the LRF methods don't win. That's a weaker but still useful claim. The gap between the best AdamW (0.498) and best Prodigy (0.477) is small, and the paper gives no per-seed error bars, so \"slightly worse\" might not survive more seeds. I'd also like the calibrated hyperparameter values disclosed; right now the configuration details are missing, which makes it hard to reproduce or reason about.\n\nThe step-budget conversion (from wall-clock to steps via AdamW's throughput) is a reasonable approximation, and the 10% per-step overhead of LRF methods makes the same-step budget generous to them, so that concern points in the paper's favor, not against it.\n\nBottom line: this is a solid empirical paper with an honest, modest conclusion. The load-bearing absence of held-out evaluation should be fixed or the claims softened. I'd send it to review; a good referee will ask for the missing artifacts and a more careful interpretation of the in-sample comparison.","headline":"A clean, useful negative result—calibrated learning-rate-free optimizers still trail tuned AdamW/NadamW on AlgoPerf—but the 'no cross-workload generalization' claim is not supported by the in-sample design.","tokens_in":23885,"tokens_out":1971,"would_cite":true,"duration_ms":18795,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper asks whether learning-rate-free optimizers can serve as the basis for truly hyperparameter-free training algorithms, and answers empirically: on the eight-workload AlgoPerf self-tuning benchmark, with defaults from the…","keywords":["learning-rate-free optimizers","hyperparameter-free training","AlgoPerf benchmark","cross-workload generalization","hyperparameter tuning","adaptive optimizers","learning rate schedules","neural network training"],"falsifier":"Re-run the same calibrated configurations on the original AlgoPerf competition hardware under the original wall-clock budgets rather than step budgets; if Prodigy or Mechanic then achieves a higher AlgoPerf benchmark score than the equally calibrated NadamW baseline, the paper's conclusion that learning-rate-free methods lag would be overturned.","tokens_in":1811,"feed_emoji":"⚖️","tokens_out":2597,"duration_ms":84765,"temperature":0.7,"pith_summary":"The paper evaluates the potential of learning-rate-free optimizers as components of truly hyperparameter-free training algorithms, using the AlgoPerf self-tuning benchmark across eight diverse workloads. It first shows that literature-supplied default hyperparameters for seven such optimizers (DoG, DoWG, D-Adapt, Prodigy, CoCoB, Mechanic, MoMo) perform poorly, with most failing to reach any validation target. The authors then perform an evidence-based calibration search over the remaining non-learning-rate hyperparameters, fixing one configuration per method across all workloads. The best calibrated learning-rate-free methods, Prodigy and Mechanic, become competitive but still score slightly below equally calibrated AdamW/NadamW baselines, leading to the conclusion that removing the base learning rate alone does not yet yield practical tuning advantages.","feed_headline":"Learning-rate-free optimizers lose to tuned AdamW on AlgoPerf","feed_subtitle":"Calibration closes the gap but doesn't flip it: removing the base learning rate saves no tuning time yet","key_machinery":"The machinery that carries the argument is the AlgoPerf self-tuning benchmark scoring chain: AlgoPerf wall-clock time limits are converted into per-workload maximum step counts using AdamW's observed step throughput, each training run is scored by time-to-validation-target, the times are aggregated into performance profiles, and the area under the profile becomes the AlgoPerf benchmark score. Around this sits the evidence-based calibration procedure, a quasi-random search over the non-learning-rate hyperparameters (weight decay, momentum, warmup fraction, label smoothing, dropout) at fixed training-horizon fractions of 33%, 50%, and 66% of the maximum steps, with the geometric-mean time-to-target used for ranking and the AlgoPerf score used for final selection. This design is what lets the paper compare methods at identical search budgets while holding all workloads common to discovery and testing.","core_discovery":"The central claim is that the current generation of learning-rate-free optimizers does not yet deliver on the promise of hyperparameter-free training. With the same number of searched configurations and the same AlgoPerf-calibrated selection procedure, the best representatives of the learning-rate-free families—Prodigy and Mechanic—reach AlgoPerf benchmark scores of 0.477 and 0.469 respectively, compared to 0.498 for the best AdamW and 0.480 for the best NadamW baseline. Naive, out-of-the-box defaults fail almost completely, and only after tuning the regularization and momentum hyperparameters, together with the fraction of the training budget used as a cosine-decay horizon, do the learning-rate-free methods become competitive. Even then, the advantage of not having a base learning rate in the configuration did not make the search for good cross-workload settings easier, and none of the tested methods removed the need for a workload-dependent training horizon. A best-effort extrapolation in the appendix suggests that Schedule-Free Adam, which also removes the learning-rate schedule, scores far higher (0.860), hinting that eliminating the schedule rather than the base rate is the decisive step.","pith_inferences":["If the step-budget conversion is the main disadvantage for learning-rate-free methods, then methods that reduce per-step overhead could reverse the ranking on wall-clock benchmarks even at the same step counts.","The benchmark scores were computed on the same eight workloads used to calibrate the configurations, so the calibrated defaults may be optimistic for truly held-out workloads; a held-out evaluation could widen or narrow the gap.","A direct, equal-budget comparison that includes Schedule-Free Adam in the same calibration protocol would test whether schedule removal is the decisive ingredient; the appendix's cross-platform approximation suggests it is, but this is not a same-hardware measurement.","The calibration protocol itself could be reused as a standard recipe for evaluating any proposed self-tuning optimizer, making cross-paper comparisons meaningful."],"forward_implications":["Publishing calibrated, evidence-based default hyperparameters for new optimizers is a measurable improvement over literature defaults: it converts algorithms that fail on almost every workload into ones that reach most validation targets.","Researchers proposing hyperparameter-reduction methods should benchmark against a strong workload-agnostic baseline with identical search budget, not against per-workload tuned baselines or library defaults.","The training horizon (the schedule that decays the learning rate to zero) is a workload-dependent hyperparameter that current learning-rate-free methods do not eliminate.","Removing the base learning rate from the tuning problem did not, at equal search budget, make it easier to find a configuration that generalizes across workloads.","Methods that remove the learning-rate schedule itself appear to be the more promising direction; Schedule-Free Adam's extrapolated AlgoPerf score (0.860) far exceeds anything found by this paper's search."],"supporting_citations":[{"why":"Defines the AlgoPerf benchmark, its self-tuning track, workloads, validation targets, and wall-clock time limits that structure the whole evaluation.","marker":"Dahl et al., 2023"},{"why":"Defines Adam, the base algorithm that the learning-rate-free methods wrap and that the baselines derive from.","marker":"Kingma and Ba, 2014"},{"why":"Defines AdamW with decoupled weight decay, one of the two baseline optimizers.","marker":"Loshchilov and Hutter, 2017"},{"why":"Defines NadamW (Adam with Nesterov momentum), the strongest baseline in the final comparison.","marker":"Dozat, 2016"},{"why":"Defines Prodigy, one of the two learning-rate-free methods that survives calibration and reaches competitive scores.","marker":"Mishchenko and Defazio, 2023"},{"why":"Defines Mechanic, the other learning-rate-free method that survives calibration, as a wrapper tuner around a base optimizer.","marker":"Cutkosky et al., 2023"},{"why":"Supplies the performance-profile methodology and area-under-curve score that the paper uses to aggregate time-to-target across workloads.","marker":"Dolan and Moré, 2002"},{"why":"Proposes Schedule-Free Adam, the method whose competition results the appendix uses to argue that removing the schedule matters.","marker":"Defazio et al., 2024"},{"why":"Reports the AlgoPerf competition results from which the paper extracts Schedule-Free Adam's runtime fractions.","marker":"Kasimbeg et al., 2025"}],"fun_headline_variants":["Calibrated learning-rate-free optimizers still trail tuned AdamW","LR-free optimizers fall short on AlgoPerf even after tuning","Schedule-free beats rate-free: tune the schedule, not the rate","Removing learning rate doesn't cut tuning effort yet"],"cache_read_input_tokens":25984,"weakest_assumption_plain":"The load-bearing premise is that a step, as counted from AdamW's throughput on the competition hardware, is a fair common currency for comparing time-to-target across algorithms, even though the learning-rate-free methods take about 10% more wall-clock time per step.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated learning-rate-free optimizers still trail tuned AdamW","LR-free optimizers fall short on AlgoPerf even after tuning","Schedule-free beats rate-free: tune the schedule, not the rate","Removing learning rate doesn't cut tuning effort yet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":4092,"prompt_tokens":1059,"completion_tokens":3033,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":2960}},"tokens_in":675,"tokens_out":3033,"duration_ms":18354,"temperature":1.0,"reasoning_tokens":2960,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:37:44.775679+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same calibrated configurations on the original AlgoPerf competition hardware under the original wall-clock budgets rather than step budgets; if Prodigy or Mechanic then achieves a higher AlgoPerf benchmark score than the equally calibrated NadamW baseline, the paper's conclusion that learning-rate-free methods lag would be overturned.","supporting_citations":[],"review_version":1}