How far away are truly hyperparameter-free learning algorithms?

Atish Agarwala; Fabian Pedregosa; George E. Dahl; Naman Agarwal; Priya Kasimbeg; Sourabh Medapati; Vincent Roulet

REVIEW 3 major objections 6 minor 2 references

How far away are truly hyperparameter-free learning algorithms?

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper asks whether learning-rate-free optimizers can serve as the basis for truly hyperparameter-free training algorithms, and answers empirically: on the eight-workload AlgoPerf self-tuning benchmark, with defaults from the…

desk verdict A clean, useful negative result—calibrated learning-rate-free optimizers still trail tuned AdamW/NadamW on AlgoPerf—but the 'no cross-workload generalization' claim is not supported by the in-sample design. read the letter →

arxiv 2505.24005 v1 pith:V6HM3WDG submitted 2025-05-29 cs.LG

Priya Kasimbeg , Vincent Roulet , Naman Agarwal , Sourabh Medapati , Fabian Pedregosa , Atish Agarwala , George E. Dahl This is my paper · ORCID

classification cs.LG

keywords learning-rate-freeoptimizershyperparameter-freetrainingAlgoPerfbenchmarkcross-workloadgeneralizationhyperparametertuningadaptivelearningrateschedulesneuralnetwork

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper evaluates the potential of learning-rate-free optimizers as components of truly hyperparameter-free training algorithms, using the AlgoPerf self-tuning benchmark across eight diverse workloads. It first shows that literature-supplied default hyperparameters for seven such optimizers (DoG, DoWG, D-Adapt, Prodigy, CoCoB, Mechanic, MoMo) perform poorly, with most failing to reach any validation target. The authors then perform an evidence-based calibration search over the remaining non-learning-rate hyperparameters, fixing one configuration per method across all workloads. The best calibrated learning-rate-free methods, Prodigy and Mechanic, become competitive but still score slightly below equally calibrated AdamW/NadamW baselines, leading to the conclusion that removing the base learning rate alone does not yet yield practical tuning advantages.

What carries the argument

The machinery that carries the argument is the AlgoPerf self-tuning benchmark scoring chain: AlgoPerf wall-clock time limits are converted into per-workload maximum step counts using AdamW's observed step throughput, each training run is scored by time-to-validation-target, the times are aggregated into performance profiles, and the area under the profile becomes the AlgoPerf benchmark score. Around this sits the evidence-based calibration procedure, a quasi-random search over the non-learning-rate hyperparameters (weight decay, momentum, warmup fraction, label smoothing, dropout) at fixed training-horizon fractions of 33%, 50%, and 66% of the maximum steps, with the geometric-mean time-to-target used for ranking and the AlgoPerf score used for final selection. This design is what lets the paper compare methods at identical search budgets while holding all workloads common to discovery and testing.

What would settle it

Re-run the same calibrated configurations on the original AlgoPerf competition hardware under the original wall-clock budgets rather than step budgets; if Prodigy or Mechanic then achieves a higher AlgoPerf benchmark score than the equally calibrated NadamW baseline, the paper's conclusion that learning-rate-free methods lag would be overturned.

Watch

Extended reading notes

Core claim

The central claim is that the current generation of learning-rate-free optimizers does not yet deliver on the promise of hyperparameter-free training. With the same number of searched configurations and the same AlgoPerf-calibrated selection procedure, the best representatives of the learning-rate-free families—Prodigy and Mechanic—reach AlgoPerf benchmark scores of 0.477 and 0.469 respectively, compared to 0.498 for the best AdamW and 0.480 for the best NadamW baseline. Naive, out-of-the-box defaults fail almost completely, and only after tuning the regularization and momentum hyperparameters, together with the fraction of the training budget used as a cosine-decay horizon, do the learning-rate-free methods become competitive. Even then, the advantage of not having a base learning rate in the configuration did not make the search for good cross-workload settings easier, and none of the tested methods removed the need for a workload-dependent training horizon. A best-effort extrapolation in the appendix suggests that Schedule-Free Adam, which also removes the learning-rate schedule, scores far higher (0.860), hinting that eliminating the schedule rather than the base rate is the decisive step.

Load-bearing premise

The load-bearing premise is that a step, as counted from AdamW's throughput on the competition hardware, is a fair common currency for comparing time-to-target across algorithms, even though the learning-rate-free methods take about 10% more wall-clock time per step.

Editorial extensions

If this is right

Publishing calibrated, evidence-based default hyperparameters for new optimizers is a measurable improvement over literature defaults: it converts algorithms that fail on almost every workload into ones that reach most validation targets.
Researchers proposing hyperparameter-reduction methods should benchmark against a strong workload-agnostic baseline with identical search budget, not against per-workload tuned baselines or library defaults.
The training horizon (the schedule that decays the learning rate to zero) is a workload-dependent hyperparameter that current learning-rate-free methods do not eliminate.
Removing the base learning rate from the tuning problem did not, at equal search budget, make it easier to find a configuration that generalizes across workloads.
Methods that remove the learning-rate schedule itself appear to be the more promising direction; Schedule-Free Adam's extrapolated AlgoPerf score (0.860) far exceeds anything found by this paper's search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the step-budget conversion is the main disadvantage for learning-rate-free methods, then methods that reduce per-step overhead could reverse the ranking on wall-clock benchmarks even at the same step counts.
The benchmark scores were computed on the same eight workloads used to calibrate the configurations, so the calibrated defaults may be optimistic for truly held-out workloads; a held-out evaluation could widen or narrow the gap.
A direct, equal-budget comparison that includes Schedule-Free Adam in the same calibration protocol would test whether schedule removal is the decisive ingredient; the appendix's cross-platform approximation suggests it is, but this is not a same-hardware measurement.
The calibration protocol itself could be reused as a standard recipe for evaluating any proposed self-tuning optimizer, making cross-paper comparisons meaningful.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the learning-rate-free benchmark paper. The useful result is a clean negative: after a systematic search over non-learning-rate hyperparameters, the best LRF optimizers (Prodigy, Mechanic) still trail a similarly tuned NadamW/AdamW baseline on the AlgoPerf suite. That is worth knowing, because literature defaults for these optimizers are often used as-is, and the paper shows those defaults perform poorly on a diverse workload set. The experimental design is fair: both baselines and LRF methods get the same number of search points, and the baselines have the extra base-learning-rate dimension, which if anything should help them. The calibration procedure and the performance-profile scoring are standard and appropriate.

The main soft spot is the one the authors acknowledge: there are no held-out workloads. Section 3.2 states this explicitly, and the final comparison in Table 6 scores the same workloads used for calibration. So the phrase in the discussion about "no significant cross-workload generalization" is not actually supported by the experimental design. What the data show is that, even on the calibration workloads, the LRF methods don't win. That's a weaker but still useful claim. The gap between the best AdamW (0.498) and best Prodigy (0.477) is small, and the paper gives no per-seed error bars, so "slightly worse" might not survive more seeds. I'd also like the calibrated hyperparameter values disclosed; right now the configuration details are missing, which makes it hard to reproduce or reason about.

The step-budget conversion (from wall-clock to steps via AdamW's throughput) is a reasonable approximation, and the 10% per-step overhead of LRF methods makes the same-step budget generous to them, so that concern points in the paper's favor, not against it.

Bottom line: this is a solid empirical paper with an honest, modest conclusion. The load-bearing absence of held-out evaluation should be fixed or the claims softened. I'd send it to review; a good referee will ask for the missing artifacts and a more careful interpretation of the in-sample comparison.

Referee Report

3 major / 6 minor

Summary. The paper empirically evaluates learning-rate-free optimizers (Prodigy, Mechanic, DoG, MoMo, COCOB, D-Adapt, DoWG) as building blocks for hyperparameter-free training algorithms. Using the eight workloads of the AlgoPerf self-tuning track, the authors first test the algorithms with literature-default hyperparameters and find they perform poorly. They then perform a quasi-random search over the remaining non-learning-rate hyperparameters (weight decay, momentum, dropout, label smoothing, warmup, training horizon), calibrated jointly across all eight workloads, and compare the resulting 'AlgoPerf-calibrated' configurations against AdamW and NadamW baselines calibrated under the same procedure. The best learning-rate-free methods (Prodigy and Mechanic, wrapped around Adam) improve substantially but still achieve slightly lower AlgoPerf benchmark scores than the best baselines (0.477 vs. 0.498). The authors conclude that current learning-rate-free methods do not yet save tuning time and do not provide significant cross-workload generalization over conventional methods, and they identify the training-horizon/schedule choice as an important remaining tuning requirement.

Significance. The study is a useful, carefully executed benchmark of learning-rate-free optimizers in the practically important setting of workload-agnostic hyperparameter selection. Its strengths include a clearly specified search space (Table 3), the same calibration budget and scoring metric for all methods, a diverse set of eight workloads, and an explicit acknowledgment of several limitations (no held-out workloads, step-budget conversion). The paper also provides a good-faith approximate comparison to the competition-winning Schedule-Free Adam (Appendix F). If the results hold, they constitute a solid negative result that should inform future work on hyperparameter reduction. The main weakness is that the headline generalization claim is not supported by the experimental design, which uses the same workloads for calibration and evaluation; this is acknowledged in Section 3.2 but the Section 6 conclusion overstates what the evidence shows.

major comments (3)

[Section 6 (Discussion), with Section 3.2] The claim that learning-rate-free methods 'don't provide significant cross-workload generalization' is not supported by the experimental design. Section 3.2 explicitly states: 'Our setup doesn't have any held-out workloads; we used the whole set of AlgoPerf base workloads to find our cross-workload hyperparameter settings.' Because the same eight workloads are used for both calibration and final evaluation, the results are in-sample. To support a claim about generalization to unseen workloads, the authors should either add a held-out evaluation (e.g., leave-one-workload-out or additional test workloads) or temper the conclusion to say that the methods underperform on the calibration workloads themselves.
[Table 6, with Section 3.2] The reported AlgoPerf benchmark scores in Table 6 have no uncertainty quantification. The final selection used five independent random seeds, but only a single score per configuration is presented. The observed gap between the best AdamW (0.498) and best Prodigy (0.477) is small, and without per-seed scores, confidence intervals, or standard errors, the claim that learning-rate-free methods perform 'slightly worse' cannot be distinguished from noise. Please report seed-level results or error bars for the key comparisons.
[Table 7 and Appendix E] The paper claims to produce 'AlgoPerf-calibrated' default hyperparameters for the learning-rate-free methods, but the actual calibrated configurations for Prodigy and Mechanic are not disclosed anywhere. Table 7 reports only the baseline candidate configurations, and Appendix E reports scores and runtime fractions without hyperparameter values. Without these values, the study is not reproducible and practitioners cannot use the proposed evidence-based defaults. Please include the final calibrated hyperparameter settings for all methods.

minor comments (6)

[Section 1] The phrase 'learning-free-learning algorithms' appears to be a typo; it should read 'learning-rate-free learning algorithms'.
[Section 3.3] The performance profile definition uses 'time taken by the training algorithm on workload w,' but the experimental setup (Section 3.1) converts wall-clock budgets to step counts based on AdamW. Please clarify whether the reported time-to-target ratios are computed from step counts or from wall-clock time, since learning-rate-free methods take about 10% longer per step.
[Appendix E] The sentence 'Schedule-Free Adamwas intr' is an incomplete fragment and should be completed or removed.
[Section 5.1] The decision to omit D-Adapt and DoWG is justified by their similarity to Prodigy and DoG, but no experimental evidence is provided; consider adding a brief description of the initial experiments that motivated this exclusion.
[References] The reference 'Anonymous (2024)' is unusual for a journal submission; if this is an under-review work, please provide a full citation or explain the blinding convention.
[Table 4 caption] The caption states 'Up arrow indicates metric is maximized, down arrow minimized,' but the table as rendered does not visibly include arrows in the metric column headers; please ensure the final formatting matches the caption.

Circularity Check

0 steps flagged · score 2.0 of 10

No formal circularity: the paper reports measured comparisons against an external benchmark, and the minor self-citation and in-sample calibration do not force the conclusions.

full rationale

The paper makes no derivation-based claim; it performs a fixed-workload empirical comparison. The AlgoPerf benchmark is cited from the authors' prior work (Dahl et al., 2023), but it is a public benchmark with fixed workloads, targets, and scoring rules, and its use does not smuggle in the paper's conclusion. The main potential concern is that hyperparameters are calibrated on the same eight AlgoPerf workloads on which final scores are computed: Section 3.2 states 'Our setup doesn't have any held-out workloads; we used the whole set of AlgoPerf base workloads to find our cross-workload hyperparameter settings.' This makes the reported AlgoPerf-calibrated gains in-sample and limits how strongly the Section 6 sentence about 'cross-workload generalization' can be read as evidence about unseen workloads. However, the paper explicitly discloses this design, does not rename a fitted quantity as a held-out prediction, and the central comparison (learning-rate-free methods vs. AdamW/NadamW) is a measured outcome rather than a quantity forced by construction. The step-budget conversion (Section 3.1) is also not a circular reduction: it is a fixed conversion from wall-clock to steps, and the authors note it is generous to learning-rate-free methods because they take about 10% more time per step. No equation in the paper reduces a predicted quantity to a fitted input, and no load-bearing argument depends on a self-citation chain. I therefore find no formal circular step; the minor self-citation and in-sample calibration warrant a score of 2 rather than 0.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the AlgoPerf benchmark as a measurement instrument, the representativeness of its workloads, and the step-budget conversion. The tuned hyperparameters are the objects of study, not hidden assumptions, but their final values are not disclosed, which weakens the audit trail.

free parameters (8)

weight_decay = not reported
Tuned via quasi-random search per algorithm; final values omitted.
1 - beta1 = not reported
Adam momentum factor tuned in search.
1 - beta2 = not reported
Adam second moment factor tuned in search.
dropout = not reported
Tied dropout rates tuned in search.
label_smoothing = not reported
Tuned in search for applicable workloads.
warmup_fraction = not reported
Tuned in search.
training_horizon_alpha = 0.33, 0.5, 0.66
Set to three fixed fractions of max steps; best per algorithm selected.
base_learning_rate = not reported
Tuned only for AdamW/NadamW baselines; final values omitted.

assumptions (5)

domain assumption The AlgoPerf benchmark's time-to-target targets and AlgoPerf score are valid measures of training algorithm quality.
The paper uses these to rank algorithms; if targets favor certain optimizers, the conclusion could change.
domain assumption The 8 AlgoPerf workloads are sufficiently diverse to represent real-world workloads.
The paper's claims about hyperparameter-free performance generalize only if these workloads are representative.
domain assumption The maximum step counts derived from AdamW runtime on the authors' hardware are valid training budgets for all algorithms.
Section 3.1: step budgets are converted from wall-clock limits using AdamW step time; LRF methods take 10% longer per step.
domain assumption Relative schedules (warmup plus cosine decay) can be applied on top of learning-rate-free methods without undermining their adaptive properties.
Section 2.3: the paper applies schedules to methods that were proposed as schedule-free; this may alter their behavior.
domain assumption The self-tuning track's rule of fixing all hyperparameters across workloads (except horizon fraction) is a faithful operationalization of 'hyperparameter-free'.
The paper defines hyperparameter-free this way; alternative definitions could change the evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How far away are truly hyperparameter-free learning algorithms?." pith.science (2026). https://pith.science/paper/V6HM3WDG

@misc{pith2026250524005,
  author       = {Pith},
  title        = {Pith review of: How far away are truly hyperparameter-free learning algorithms?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6HM3WDG}},
  note         = {Machine review of arXiv:2505.24005}
}

read the original abstract

Despite major advances in methodology, hyperparameter tuning remains a crucial (and expensive) part of the development of machine learning systems. Even ignoring architectural choices, deep neural networks have a large number of optimization and regularization hyperparameters that need to be tuned carefully per workload in order to obtain the best results. In a perfect world, training algorithms would not require workload-specific hyperparameter tuning, but would instead have default settings that performed well across many workloads. Recently, there has been a growing literature on optimization methods which attempt to reduce the number of hyperparameters -- particularly the learning rate and its accompanying schedule. Given these developments, how far away is the dream of neural network training algorithms that completely obviate the need for painful tuning? In this paper, we evaluate the potential of learning-rate-free methods as components of hyperparameter-free methods. We freeze their (non-learning rate) hyperparameters to default values, and score their performance using the recently-proposed AlgoPerf: Training Algorithms benchmark. We found that literature-supplied default settings performed poorly on the benchmark, so we performed a search for hyperparameter configurations that performed well across all workloads simultaneously. The best AlgoPerf-calibrated learning-rate-free methods had much improved performance but still lagged slightly behind a similarly calibrated NadamW baseline in overall benchmark score. Our results suggest that there is still much room for improvement for learning-rate-free methods, and that testing against a strong, workload-agnostic baseline is important to improve hyperparameter reduction techniques.

Figures

Figures reproduced from arXiv: 2505.24005 by the authors.

**Figure 2.** Validation CE loss vs training step for ‘out-of-the-box’ learning-rate-free methods and baselines on [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

**Figure 3.** Validation SSIM vs training step for ‘out-of-the-box’ learning-rate-free-methods and baselines on [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Validation Error Rate vs train step for ‘out-of-the-box’ learning-rate-free methods and baselines on [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

**Figure 5.** Figure 5: Validation Error Rate vs train step for ‘out-of-the-box’ learning-rate-free methods and baselines on [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

**Figure 6.** Figure 6: Validation WER vs train step for ‘out-of-the-box’ learning-rate-free methods and baselines on [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

**Figure 7.** Figure 7: Validation WER vs train step for ‘out-of-the-box’ learning-rate-free methods and baselines on [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

**Figure 8.** Figure 8: Validation Average Precision vs train step for ‘out-of-the-box’ learning-rate-free methods and [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

**Figure 9.** Figure 9: Validation BLEU score vs train step for ‘out-of-the-box’ learning-rate-free methods and baselines [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

For the regularization hyperparameters, since there are no well-defined values, we choose the default values supplied, which effectively, turns these parameters off (i.e

No tuning:We run the presented algorithms without any modification from their default values as found in the Optax implementations. For the regularization hyperparameters, since there are no well-defined values, we choose the default values supplied, which effectively, turns these parameters off (i.e. dropout rates, label smoothing strength, and weight de...

work page
[2]

Tuning other hyperparametersIn this section we use the above algorithms with their default optimization hyperparameters and tune the following regularization and LR schedule parameters on top of it •Weight Decay - [1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7] •Dropout (tied) - [0.0, 0.1] •Label Smoothing - [0.0, 0.1, 0.2] •Training Horizon - [0.33, 0.49, 0.6...

work page 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

For the regularization hyperparameters, since there are no well-defined values, we choose the default values supplied, which effectively, turns these parameters off (i.e

No tuning:We run the presented algorithms without any modification from their default values as found in the Optax implementations. For the regularization hyperparameters, since there are no well-defined values, we choose the default values supplied, which effectively, turns these parameters off (i.e. dropout rates, label smoothing strength, and weight de...

work page

[2] [2]

Tuning other hyperparametersIn this section we use the above algorithms with their default optimization hyperparameters and tune the following regularization and LR schedule parameters on top of it •Weight Decay - [1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7] •Dropout (tied) - [0.0, 0.1] •Label Smoothing - [0.0, 0.1, 0.2] •Training Horizon - [0.33, 0.49, 0.6...

work page 2024