{"id":"d972c1cc-4736-4719-93c9-18c4a4067960","arxiv_id":"2506.15025","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"As vocabulary size grows, the optimal embedding-to-hidden learning rate ratio in LLMs scales as Θ(√width), a regime the authors call the Large Vocabulary parameterization.","lead":"When training large language models, the best learning rate for the word-embedding layer compared to the rest of the model grows roughly as the square root of model width, not linearly as earlier theory suggested. This paper derives that rule from vocabulary size effects and shows it improves perplexity in a 1B model pretraining run.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 2 treats EW−Z as exogenous i.i.d. noise although it is a function of E and W; Theorems 3–4 then condition on objects that are not independent of the matrix being signed, so the √d ratio rests on an unvalidated independence assumption.","rationale":"The reader and this stress-test identify the same load-bearing weakness: Assumption 2's independence/postulated Gaussianity of the residual matrix is central to the proof of Theorem 1, and it is violated by the model's own product structure. The paper does not validate this assumption empirically or analytically, and the proof of Theorem 4 explicitly conditions on M as though M were independent of E, which is false when M = D_α(EW−Z). This makes the theoretical derivation of the √d rule insecure. However, the empirical case is not negligible: the small-scale sweeps across (d,m) show a slope near −1/2 for the optimal embedding LR, and the 1B pretraining run shows a consistent improvement for η_E/η_W ≈ √d over the equal-ratio baseline. These results provide independent support for the practical rule even if the theory is not yet rigorous. The verdict should remain CONDITIONAL rather than REJECT: the central claim is plausible and empirically suggestive, but the theoretical justification needs to be hardened by either proving the same scaling without Assumption 2 or validating the assumption directly. I do not see an internal inconsistency that would warrant outright rejection, and I do not regard the paper's heuristic extensions as disqualifying given the explicit limitations section.","tokens_in":16456,"tokens_out":8749,"duration_ms":95677,"concrete_test":"Run a Monte Carlo check with the paper's Gaussian initialization and Zipf frequencies for (d,m) = (2^k, 2^{k+3}), k = 8..12: compute (1/m)E||δ_i^E||^2 and (1/m)E||δ_i^W||^2 using the actual E and W, without imposing Assumption 2, and compare the d,m scalings with the Theorem 1 formulas. If the measured moments deviate from η_E^2 σ_W^2 (d + 2d(d−1)/(πm)) and the δ_W analogue by more than a constant factor, Assumption 2 is doing real work and the theorem requires revision; if the scalings match, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The derivation of the Θ(√d) ratio hinges on Assumption 2 in Appendix B, which states that the residual matrix (EW−Z) at initialization has i.i.d. N(0,1) entries. This assumption is used to make the gradient-covariance computation tractable, but it replaces a quantity that is a deterministic function of the trainable matrices E and W with exogenous noise that is independent of W. For δ_i^E, B.3.1 introduces a vector v independent of W, whereas in the model v = E_i W − z_i is itself a function of W (and E_i), so sign(⟨v,W_j⟩) and W_j are dependent. For δ_i^W, Theorem 4 conditions on M and treats E_i as independent Gaussian, yet M = D_α(EW−Z) includes E_i; conditioning on M fixes linear combinations E_i W_k, so E_i is not independent of M and the Stein-lemma correlation formula ρ_k = M_ik / sqrt(Σ_j M_jk^2) is not justified. The leading Θ behavior might survive through concentration, but the paper gives no argument for that. The one-step SignSGD simplification and the residual-stream analogy for transformers are additional gaps, but Assumption 2 is the specific step where the proof's premises fail against the model's product structure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how the optimal ratio between the embedding learning rate and the hidden-layer learning rate scales with model width d when the vocabulary size m is also large. Starting from a linear embedding–projection model trained with SignSGD, the authors derive asymptotic formulas for the per-token feature-learning magnitudes δ^E and δ^W under a Zipf–Mandelbrot token-frequency assumption. For fixed m these formulas reproduce the μP scaling rules; as m grows, the formulas are said to interpolate between the μP regime and a 'Large Vocabulary' regime in which the optimal ratio satisfies LR_emb/LR_hidden = Θ(√d). The paper proposes the 'Large Vocabulary Parametrization' (LVP), with SP-style initialization and embedding LR η d^{-1/2} while hidden/output LRs are η d^{-1}, and reports small-scale sweeps plus a 1B-parameter pretraining experiment showing improved perplexity over the equal-ratio baseline.","tokens_in":16753,"tokens_out":7266,"duration_ms":77125,"significance":"If the central claim holds, the paper addresses a real gap in μP theory for LLMs, where vocabulary size is large and often grows with model scale. The concrete, falsifiable prediction that LR_emb/LR_hidden ≈ Θ(√d) is valuable, and the 1B-token-scale experiment provides practical evidence that this rule helps. The paper is also refreshingly explicit about the limitations of its toy model and about the uncertainty of the optimality criterion. The main strength is a transparent set of covariance computations giving an explicit mechanism by which large vocabulary size changes the embedding-update magnitude. The main weakness is that the proofs rest on an independence assumption that is not justified by the actual product structure of the model, and the bridge from one-step SignSGD on a linear model to Adam-trained transformers is only heuristic.","major_comments":[{"comment":"As stated, the proof of Theorem 1 does not follow from the model because Assumption 2 is incompatible with the product structure it replaces. In the linear model, the vector v = E_i W − z_i used in B.3.1 is the i-th row of EW − Z, hence a function of W and E_i; it is not an exogenous N(0,1) vector independent of W. Theorem 3 requires that independence to compute the conditional covariance, and the covariance formula (3) relies on it. Similarly, in Theorem 4 the matrix M = D_α(EW − Z) contains E_i, so when the proof conditions on M, E_i is no longer independent of M, and the Stein-lemma correlation ρ_k = M_ik / sqrt(Σ_j M_jk^2) is not justified. If Assumption 2 is intended as an approximation valid when EW is small relative to Z (or in some other limit), that regime must be stated and proved; otherwise the Θ(√d) ratio is not derived from the declared model. This is load-bearing because Theorems 1 and 2 are the only theoretical support for the paper's central claim.","section":"Appendix B, Assumption 2 and Theorems 3–4"},{"comment":"The theoretical result is derived for a single SignSGD step on a model consisting only of an embedding layer and a projection layer, but the central claim and the large-scale experiments concern Adam-trained deep transformers with residual blocks. The sentence in Section 5 that the authors 'expect the results to hold' for multiple steps and for Adam 'since the main ingredient in our analysis is the normalization process' is an expectation, not a proof, and the residual-stream analogy in 'Extension to Transformer Architectures' is qualitative. Because the abstract advertises the Θ(√d) rule for LLM pretraining, the manuscript needs either a formal or a quantitatively justified extension, or a clearly weaker claim about what has been established theoretically versus what is supported only by experiments.","section":"Section 3 and Section 5"},{"comment":"The Θ(√d) ratio is obtained by imposing the feature-learning criterion that both δ_i^E and δ_i^W are Θ(1) while taking initialization variances σ_E = σ_W = d^{-1/2}. This is a design criterion from the μP literature, not a theorem about minimizing the training loss, and Section 5 itself acknowledges that it is 'unclear whether the limit with this scaling is optimal.' The experiments give empirical support, but the theoretical statement that this is the 'optimal' ratio is stronger than the model analysis establishes. Please state explicitly in which sense the ratio is optimal and separate the optimality theorem from the empirical finding.","section":"Section 3.1, definition of optimality"}],"minor_comments":[{"comment":"The sentence asserting 'LRemb/LRhidden = Θ(d^{-1/2})' appears to be inverted; Table 2 and the abstract imply LRemb/LRhidden = Θ(d^{1/2}).","section":"Section 2, page 4"},{"comment":"The sign function is defined inconsistently: the footnote says S(z_i) = 1 if z_i ≥ 0 and 0 otherwise, while the proof in Appendix B uses S_j ∈ {±1} and the text elsewhere treats it as sign. Please use one definition throughout.","section":"Section 3.1, footnote 11"},{"comment":"The figure captions and axis labels contain corrupted tokens such as 'Step/uni00A01e2' and 'Best/uni00A0(10%)'; these need to be cleaned.","section":"Figures 1 and 3"},{"comment":"There are several typographical errors, including 'subopotimal' in Section 2 and 'the the embedding layer' in Section 3, that should be corrected in a revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be influential if the central prediction survives scrutiny, and the 1B-scale experiment is a strong point in its favor. The main risk for the journal is the gap between the stated model and the proof under Assumption 2; I would recommend sending the proof of Theorems 3 and 4 to a careful probabilist before acceptance. The authors' own limitation paragraph shows awareness of the issue, but the manuscript as written does not yet close it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper for the practical rule, not the proof. The claim that the embedding LR should scale like η d^{-1/2} (i.e., a √d ratio to the hidden LR) has real empirical support here: the small-scale sweeps reproduce the d^{-1/2} slope, and the 1B pretraining run shows a clear win for the √d ratio over the standard ratio 1. That alone is worth your time if you train LLMs. The d^{-1/2} scaling itself was already in Blake et al. (u-µP), which they cite. What's new is the mechanism: vocabulary size changing the feature-learning balance through token frequencies, and the LVP parametrization that combines SP initialization with the √d embedding LR.\n\nThe soft spot is in Appendix B. Assumption 2 replaces EW−Z with i.i.d. N(0,1) noise, but in the model EW−Z is a product of independent Gaussian matrices, so the entries are not independent and the matrix is rank-limited. More concretely, in Theorem 3 the vector v is taken independent of W, but in the actual model v = E_i W − z_i depends on W, so sign(<v,W_j>) and W_j are dependent and the Stein-lemma step doesn't apply. Theorem 4 has the same issue: the matrix M includes E_i, so conditioning on M breaks the independence that justifies the correlation formula. The leading Θ(√d) behavior might survive through a more careful concentration argument, but the paper doesn't give one. The one-step SignSGD simplification and the jump to Adam are another layer of assumption, though the paper is upfront about that.\n\nI don't think this sinks the paper. The empirical case stands on its own, and the theory is presented as a controlled toy that gives intuition. But the proof of the headline ratio is not complete as written. If it goes to review, the authors should either repair the independence argument or explicitly label the theorem a heuristic. The 1B result is a single run for the two long curves—the extra ratios only ran 65B tokens—so some caution on the 'near-optimal' phrasing is warranted.\n\nRecommendation: send it to review. It's a useful, citable empirical result with a plausible but not rigorous theory. A serious referee can push on the assumption without rejecting the work.","headline":"A useful practical rule with solid empirical support, but the √d derivation rests on an independence assumption the model violates; treat the theory as a heuristic and send to review.","tokens_in":17296,"tokens_out":3486,"would_cite":true,"duration_ms":34977,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","60B20"],"pacs":[],"model":"deepseek-v4-flash","headline":"As vocabulary grows with width, the optimal embedding-to-hidden learning-rate ratio shifts from Θ(d) to roughly √d, and the paper's Large Vocabulary Parametrization built on this rule improves 1B-scale pretraining.","keywords":["embedding layer learning rate","maximal update parametrization","vocabulary size scaling","large vocabulary regime","hyperparameter transfer","feature learning","SignSGD","transformer pretraining"],"falsifier":"On the paper's two-layer model with Gaussian-initialized $E$ and $W$, take one sign-based update and measure the mean squared norm of a frequent token's embedding-row update; the theory predicts it grows as $d + 2d(d-1)/(\\pi m)$. Vary $m$ over $\\{d/4, d, 4d, 16d\\}$: if the fitted coefficient of $d(d-1)/m$ departs from $2/\\pi$, the covariance identity behind the LV regime is wrong. A complementary check on a full transformer holds the width fixed and quadruples the vocabulary, in which case the optimal embedding-to-hidden LR ratio should move measurably toward $\\sqrt{d}$.","tokens_in":16203,"feed_emoji":"📐","tokens_out":27654,"duration_ms":239144,"temperature":0.7,"pith_summary":"This paper argues that the optimal learning-rate parametrization of a large language model changes with vocabulary size, and that modern LLMs sit in a regime where the standard theory mis-scales the embedding layer. Analyzing a minimal embedding-plus-projection linear model with a single SignSGD step, the authors compute how the magnitudes of embedding and projection updates depend on both width $d$ and vocabulary size $m$, under a Zipf-Mandelbrot token-frequency law. They find the dynamics interpolate between the $\\mu$P regime (vocabulary fixed) and a Large Vocabulary regime in which the optimal ratio of embedding to hidden learning rate is $\\Theta_d(\\sqrt{d})$, not the $\\Theta_d(d)$ ratio that $\\mu$P predicts. If correct, the Large Vocabulary Parametrization (embedding LR $\\eta d^{-1/2}$, hidden and output LR $\\eta d^{-1}$, all initialization variances $d^{-1}$) should give better loss and perplexity than both standard parametrization and $\\mu$P, and the paper reports such gains in small width-vocabulary sweeps and in a 1B-parameter pretraining run.","feed_headline":"Set the embedding learning rate √width times higher than hidden layers","feed_subtitle":"Validated on a 1B-parameter pretraining run, where it outperforms both baseline scaling recipes","key_machinery":"The load-bearing object is the one-step feature-learning magnitude of a single token: after one sign-based (SignSGD) update of the embedding–projection network, the output change splits into an embedding contribution $\\delta_i^E$ (updating $E$ with $W$ fixed) and a projection contribution $\\delta_i^W$ (updating $W$ with $E$ fixed), and the optimality criterion is that both be $\\Theta_d(1)$, so neither parameter silently stops learning as width grows. The workhorse computation is the covariance of a sign–Gaussian product, $X = \\sum_{j=1}^d \\mathrm{sign}(\\langle v, W_j\\rangle) W_j$ with $v$ a standard Gaussian vector, which evaluates to $\\mathrm{Cov}(X) = d I_m + \\frac{2}{\\pi m} d(d-1) I_m$ through the Stein-type identity $\\mathbb{E}[\\mathrm{sign}(Z)G] = \\sqrt{2/\\pi}\\,\\rho$ for correlated standard Gaussians. The $d I_m$ term reproduces the $\\mu$P scalings when $m$ is fixed; the $d(d-1)/m$ term, amplified by the Zipf-Mandelbrot law's $\\Theta(m^{-1})$ average squared frequency, produces the $\\sqrt{d}$ behavior when $m$ is large.","core_discovery":"The central discovery is that vocabulary size changes the width-scaling of feature learning in the embedding layer, and with it the optimal learning-rate ratio. For the two-layer model, the one-step update magnitudes follow $\\bar\\delta^E_i = \\Theta_{m,d}(\\eta_E \\sigma_W \\sqrt{d + 2d(d-1)/(\\pi m)})$ and $\\bar\\delta^W_i = \\Theta_{m,d}(\\eta_W \\sigma_E \\sqrt{d + (\\alpha_i^2/\\bar\\alpha^2)\\, 2d(d-1)/(\\pi m)})$; with $m$ fixed these reduce to the $\\mu$P scalings, but when $m$ is large the first collapses to $\\Theta(\\eta_E \\sigma_W \\sqrt{d})$ while the second stays $\\Theta(\\eta_W \\sigma_E d)$ for the most frequent tokens. Equalizing the two feature-learning components forces $\\mathrm{LR}_{\\mathrm{emb}}/\\mathrm{LR}_{\\mathrm{hidden}} = \\Theta_d(\\sqrt{d})$ under standard $d^{-1/2}$ initialization, a rule the authors call the $\\sqrt{d}$-rule and package as the Large Vocabulary Parametrization (LVP). They validate it with coupled width-vocabulary sweeps and a 1B-parameter model pretrained on 1.75T tokens, where the $\\sqrt{d}$ ratio improves training perplexity and Wikitext test perplexity relative to both the equal-ratio baseline and the $\\Theta(d)$ ratio prescribed by $\\mu$P.","pith_inferences":["A direct extension the paper does not state: the optimal ratio exponent should interpolate continuously between $1$ (small $m/d$) and $1/2$ (large $m/d$), so sweeping $m$ at fixed width should trace a curve rather than two separated regimes; the authors' small-model setup can test this immediately.","Because the projection-update term in Theorem 1 carries a token-frequency factor $\\alpha_i^2/\\bar\\alpha^2$, the theory implicitly suggests token-frequency-aware embedding learning rates (larger updates for rare tokens) could improve on a single ratio; the paper leaves this untested.","The residual-stream coupling that carries the effect is specific to decoder-style transformers, so encoder-decoder or non-residual architectures with separate embeddings and projections may not show the $\\sqrt{d}$ regime; training the same model with and without the residual pathway would clarify how general the rule is."],"forward_implications":["Under the Large Vocabulary Parametrization, setting $\\mathrm{LR}_{\\mathrm{emb}}/\\mathrm{LR}_{\\mathrm{hidden}} \\approx \\sqrt{d}$ should give lower pretraining loss and perplexity than both the equal-ratio standard recipe and the $\\mu$P $\\Theta(d)$ ratio whenever $m \\gg d$, as in modern LLMs.","The optimal embedding learning rate is not width-constant as $\\mu$P predicts; in the LV regime it scales roughly as $d^{-1/2}$, so hyperparameter transfer for the embedding layer requires re-tuning the embedding LR constant at each width.","As models adopt larger vocabularies relative to width, the gap between $\\mu$P's predictions and the optimal embedding LR should widen, making the $\\sqrt{d}$-rule more consequential at scale.","The hidden- and output-layer rules of $\\mu$P (LR $\\propto d^{-1}$, initialization variance $d^{-1}$) survive unchanged in the LV regime, so adopting LVP changes only the embedding LR and initialization, not the rest of the model.","The 1B-parameter experiments indicate the $\\sqrt{d}$ rule converges faster as well as to a better final loss, which translates to token savings in large pretraining runs."],"supporting_citations":[{"why":"Supplies the $\\mu$P scaling rules (embedding LR $\\Theta(1)$, hidden/output LR $\\Theta(d^{-1})$, initialization variances $1, d^{-2}, d^{-1}$) that the paper's theory corrects for large vocabulary size.","marker":"[24]"},{"why":"Reports the empirical finding that optimal embedding LR decreases roughly as $d^{-1/2}$ with width under unit-scaled $\\mu$P; this is the observation the LV-regime theory is built to explain.","marker":"[1]"},{"why":"Provides evidence that standard parametrization with adjusted LR exponents can match or beat $\\mu$P, motivating the authors' choice of $d^{-1/2}$ initialization in LVP.","marker":"[5]"},{"why":"Defines the Adam optimizer whose elementwise normalization motivates replacing it with SignSGD in the tractable one-step analysis.","marker":"[12]"},{"why":"Supplies the Zipf-Mandelbrot token-frequency law (Assumption 1), from which the $\\Theta(m^{-1})$ average squared frequency follows.","marker":"[27]"},{"why":"Documents that larger models are paired with larger vocabularies, justifying joint growth of $m$ and $d$ that defines the Large Vocabulary regime.","marker":"[19]"},{"why":"Provides the 1.75T-token training corpus on which the 1B-parameter pretraining comparison of the $\\sqrt{d}$ rule against the baseline is run.","marker":"[22]"}],"fun_headline_variants":["Embedding LR: √width rule beats μP for large vocab","Large vocab flips optimal LR ratio to √width","Why embedding LR scales as √width, not width","New scaling rule: embedding LR = √width × hidden LR","Vocabulary size dictates optimal embedding LR ratio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the initial differences between model outputs and targets behave as independent random noise in every entry, even though the two random matrices producing them make the rows correlated, and that one step of a simplified sign-only optimizer faithfully represents Adam's multi-step behavior.","fun_headline_variants_meta":{"raw":{"variants":["Embedding LR: √width rule beats μP for large vocab","Large vocab flips optimal LR ratio to √width","Why embedding LR scales as √width, not width","New scaling rule: embedding LR = √width × hidden LR","Vocabulary size dictates optimal embedding LR ratio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1716,"prompt_tokens":1176,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":792,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":792,"tokens_out":540,"duration_ms":4789,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:45:57.289365+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the paper's two-layer model with Gaussian-initialized $E$ and $W$, take one sign-based update and measure the mean squared norm of a frequent token's embedding-row update; the theory predicts it grows as $d + 2d(d-1)/(\\pi m)$. Vary $m$ over $\\{d/4, d, 4d, 16d\\}$: if the fitted coefficient of $d(d-1)/m$ departs from $2/\\pi$, the covariance identity behind the LV regime is wrong. A complementary check on a full transformer holds the width fixed and quadruples the vocabulary, in which case the optimal embedding-to-hidden LR ratio should move measurably toward $\\sqrt{d}$.","supporting_citations":[{"cited_title":"Alemi, Roman Novak, Peter J","cited_arxiv_id":null,"evidence_quote":"Provides evidence that standard parametrization with adjusted LR exponents can match or beat $\\mu$P, motivating the authors' choice of $d^{-1/2}$ initialization in LVP."},{"cited_title":"Selected Studies of the Principle of Relative Frequency in Language","cited_arxiv_id":null,"evidence_quote":"Supplies the Zipf-Mandelbrot token-frequency law (Assumption 1), from which the $\\Theta(m^{-1})$ average squared frequency follows."}],"review_version":1}