{"id":"758820af-1b0f-42a9-8866-5a0e4a98da4a","arxiv_id":"2608.03494","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"For Hindi vocabulary extension of a 30B LLM, the best embedding initialization is uniform subword averaging with Hindi norm calibration on the input and character-length-weighted averaging on the output, cutting continued pre-training steps roughly 6-fold.","lead":"This paper tests more than 20 ways to initialize newly added Hindi token embeddings in a 30-billion-parameter LLM. It finds a simple asymmetric recipe, uniform subword averaging for the input with Hindi norm calibration and character-length averaging for the output head, that reaches baseline quality in about one sixth of the continued pre-training steps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lack of repeated seeds/confidence intervals makes the 50-step Val50 ranking that selects the best configuration potentially seed noise; top gaps are 0.002–0.031, which could alter the speedup claim.","rationale":"After reading the paper and the reader's verdict, I find the central family-level conclusion (subword composition > vocabulary averaging > external/learned init) credible: the Val50 differences between families are large (2.72 vs 4.74 vs 5.53). The exact best configuration and the quantitative speedup, however, depend on the small Val50 margins among top configurations. Since these are single-run measurements with no repeated seeds, the ranking could be driven by noise. This is the weakest point because the paper promotes a 50-step probe as a reliable selection protocol, yet provides no statistical evidence that the Val50 ranking is stable across seeds or that it matches longer-horizon rankings for more than the two configurations shown in Figure 3. The paper honestly lists this as a limitation, which supports a conditional acceptance pending validation. My proposed test directly asks for the missing variance estimates and a stability check at longer horizons. If the test shows the ranking is stable, the conditional concerns are resolved; if not, the protocol and speedup claims need revision. Therefore I recommend keeping the CONDITIONAL verdict. I agree with the reader's weakest assumption.","tokens_in":13629,"tokens_out":7744,"duration_ms":81524,"concrete_test":"Run at least 5 independent seeds of the 50-step CPT probe for the top five configurations in Table 1 Section C: Uniform+Hindi Norm, Char-len Out, MuRIL Out, Gemma Out, and MuRIL In+Char-len Out (same data and hyperparameters). Compute mean and standard deviation of Val50 per configuration, and a pairwise significance test (e.g., a paired bootstrap over validation batches). Additionally, for the top two configurations (Char-len Out vs Uniform+Norm), run 3 seeds each for 200 and 500 steps to check whether the 50-step ranking persists. If the gap between 2.722 and 2.724 (or 2.722 and 2.753) is within one standard deviation, or if longer runs reorder the top configurations, then the 'best' selection and the 6x speedup claim should be reported with uncertainty bounds, and the 50-step protocol should be revised (e.g., use 3 seeds per condition or a paired test).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central speedup and the recommended 50-step selection protocol rest on the Val50 ranking in Table 1. In Section C, the best configuration (Uniform In + Char-len Out + Hindi norm) reports Val50=2.722, with the MuRIL In + Char-len Out runner-up at 2.724, and the best symmetric/other variants at 2.752–2.753. These differences are 0.002 to 0.031. No repeated seeds, confidence intervals, or variance estimates are provided for any Val50 number. For a 30B model trained at batch size 1024 with a single validation set, such small gaps can easily be run-to-run noise. If the true Val50 for the top configurations overlaps, the identity of the 'best' configuration and the '6x CPT-step reduction' (Section 5.4) lose precision; the selection protocol could pick a different strategy on a new language. The paper's own Limitations state that differences among top configurations diminish with training and that the 50-step probe should be validated across more settings, which is exactly where the load-bearing assumption sits. The large gap between subword composition (2.72) and the Mean-all baseline (5.53) is credible, so the family-level conclusion is likely robust, but the asymmetric-winner and the exact speedup factor are not statistically supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies token embedding initialization for Hindi vocabulary extension in Nemotron-3-Nano-30B-A3B. It compares more than 20 strategies across five design axes—composition scope, subword weighting, norm calibration, input/output asymmetry, and external mapping—using both step-zero metrics (Init Loss, Init BPB) and a 50-step continued pre-training probe (Val50). The authors report that subword composition methods outperform vocabulary-averaging and external/learned initialization approaches; that the best observed configuration is asymmetric (uniform subword input with Hindi norm calibration, character-length-weighted output); and that this configuration reaches comparable validation loss with a roughly 6x reduction in CPT steps and exceeds the Mean-all baseline's 3,500-step MILU-Hindi accuracy after only 500 steps. They also argue that initialization-time metrics are unreliable predictors and that a lightweight 50-step probe is a reliable selection signal.","tokens_in":14081,"tokens_out":4872,"duration_ms":59660,"significance":"If the results hold, the paper is a practically useful empirical study: it systematically maps a large initialization design space on a 30B model, includes up-to-date baselines (FOCUS, WECHSEL-style retrieval, residual MLP), and examines long-run retention in addition to early loss. The decomposition of gains into subword composition, norm calibration, and input-output asymmetry is a valuable framing. The main weakness is that the quantitative headline claims—the identity of the best configuration, the 6x/7x speedups, and the reliability of the 50-step probe—rest on single-run comparisons with very small differences among top configurations. The family-level conclusion that subword composition is far better than Mean-all is likely robust, but the asymmetric-winner and the exact speedup factors are not statistically supported as written.","major_comments":[{"comment":"The selection of the 'best observed' configuration rests on Val50 differences of 0.002–0.031 among the top entries (2.722 vs 2.724 vs 2.752–2.753). No seeds, confidence intervals, or standard errors are reported for any Val50 value. Since the 6x CPT-step reduction (Section 5.4) and the 7x MILU step reduction (Section 5.5) are computed for this single chosen configuration, the headline speedups are not statistically grounded. Please provide repeated-seed runs (at least 3–5) for the top configurations and the Mean-all baseline, reporting means and intervals, or explicitly reframe the result as a single-run case study with the speedup range left open.","section":"Table 1, Sections 5.3 and 5.4"},{"comment":"A core contribution is the lightweight 50-step CPT probe as a reliable strategy-selection signal. The evidence is one model (Nemotron-3-Nano-30B), one language pair, one data mixture, and a single run per configuration. The paper's own Limitations states that the probe 'should be validated across additional training mixtures and downstream benchmarks,' which is precisely the missing support for the protocol. At minimum, show repeated 50-step probes on this setup to quantify run-to-run variance, and test one additional language or mixture to show that the ranking and the stabilization claim transfer. Absent that, this claim should be presented as an observation rather than a protocol.","section":"Section 4.2 and Section 5.4"},{"comment":"The paper claims initialization loss and Init BPB are 'unreliable predictors' because rankings invert by step 50, but these inversions are demonstrated on single trajectories with no uncertainty. This negative claim could be supported by a small table of rank correlations (e.g., Spearman) between Init Loss, Val50, and final validation/MILU, with bootstrap intervals. As written, the conclusion is qualitative and the strength of the wording is not matched by the evidence.","section":"Section 5.2 and Figure 1"}],"minor_comments":[{"comment":"Typographical issue: the abstract begins 'V ocabulary' and the introduction repeatedly uses 'V ocabulary' with an extra space. Please proofread the extracted text.","section":"Abstract and Section 1"},{"comment":"The Residual MLP loss is stated as L = L_MSE + lambda_cos * L_cos + lambda_norm * L_norm, but the weight for L_MSE is not specified. State that L_MSE has weight 1.0, or provide the full weighting.","section":"Section 3"},{"comment":"FOCUS (ft-hi) has no Val50 entry, and FOCUS (anchor-hi) and Top-k Gemma report Val50 only in some rows. Please add a footnote explaining missing entries (e.g., not measured or failed early training).","section":"Table 1, Section A"},{"comment":"The run is described as 'roughly 50 billion tokens' over 7,000 global steps. At batch size 1024 and sequence length 8192, this is approximately 58.7B tokens. Please reconcile the token count or the step count.","section":"Section 5.6"},{"comment":"Semantic softmax temperature (tau=0.1), FOCUS sharpening (tau=0.05), top-k=5, and the residual MLP loss weights are chosen without sensitivity analysis. A brief ablation of the temperature and top-k values would strengthen the generality of the conclusions.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical case study, but the revision ask is real: the central speedup and the 50-step selection protocol need variance estimates or a clear downgrade in scope. If repeated seeds for all 20+ configurations are too expensive, the authors should run seeds for the top handful and the baseline, and explicitly reframe the remaining claims as single-run observations. The family-level comparison (subword composition vs Mean-all) is likely robust; the asymmetric-winner and exact speedup are the fragile parts. An editor may also want to check whether the paper's novelty relative to prior vocabulary-adaptation work is sufficiently articulated, since AdaptiVocab already discusses input/output asymmetry."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper is worth taking seriously: it's the first systematic comparison I've seen of over 20 embedding initialization strategies on a real 30B model for Hindi vocabulary extension, and it makes a practical case that subword composition dominates both averaging and external/learned methods. The family-level finding is credible: the gap between subword composition and Mean-all on validation loss after 50 steps is 2.72 vs 5.53, and the downstream MILU-Hindi gains are large and consistent. The paper also does something useful methodologically—it shows that step-zero metrics like Init Loss and BPB rank strategies differently than short CPT, so the field shouldn't rely on them. The 50-step probe is a pragmatic suggestion.\n\nThe soft spot is statistical. Every configuration is a single run; there are no seeds, confidence intervals, or standard errors. Among the top asymmetric variants the Val50 spread is 2.722 to 2.752, which is small enough that run-to-run noise could reshuffle the order. The 'best' configuration and the exact 6x/7x speedup factors should be read as observed in this experiment, not as a guaranteed ranking. The paper's own Limitations acknowledge that differences diminish with training and that the probe needs validation across more settings—that's exactly the load-bearing assumption. It's not fatal, but it means the headline numbers are more fragile than they look.\n\nWhat the paper does well beyond the main results: the long-run retention evaluation (English and multilingual benchmarks staying within noise) is reassuring, and the fertility/eval-runtime numbers are a nice context. No code or data release is mentioned, which would be nice for reproducibility, but the methods are described in enough detail to reimplement.\n\nBottom line: for anyone working on vocab extension or low-resource adaptation, this is a worthwhile read and citable reference. The specific winner is a single-run observation; the family-level conclusion is solid. I'd send it to peer review—the empirical comparison is valuable—and ask the authors to add repeated seeds or at least report variance on the Val50 values, and to soften the claim about the 'best' configuration accordingly.","headline":"A useful, largely credible empirical survey of embedding initialization strategies for Hindi vocabulary extension on a 30B model, but the specific 'best' configuration and the 50-step selection probe rest on single-run evidence with no variance estimates.","tokens_in":14521,"tokens_out":1889,"would_cite":true,"duration_ms":20972,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Subword-based asymmetric initialization lets a Hindi-extended LLM match baseline quality in about one-sixth the continued-pretraining steps.","keywords":["vocabulary extension","embedding initialization","continued pre-training","Hindi NLP","subword composition","asymmetric initialization","norm calibration","large language models"],"falsifier":"Run the top three configurations (Uniform+Char-len+Hindi Norm, MuRIL In+Char-len Out, and Mean-all) on a second low-resource script with several random seeds; if the Val50 ordering does not survive seed variation, or if a 500-step run overturns the 50-step ranking, the paper's selection protocol and 6x speedup claim would fail to reproduce. A cheaper check: compare the 2.722 vs 2.724 vs 2.752 Val50 gaps against seed standard deviation.","tokens_in":1995,"feed_emoji":"🔤","tokens_out":3020,"duration_ms":103536,"temperature":0.7,"pith_summary":"This paper asks how to initialize the embeddings of newly added Hindi tokens when extending a pretrained English-centric LLM's vocabulary, and shows that the choice matters a lot for how quickly continued pre-training (CPT) converges. It compares more than 20 strategies—averaging baselines, external cross-lingual mappings such as FOCUS and top-k retrieval, a learned residual mapping, and a generalized subword-composition family—under a lightweight 50-step CPT probe. The central finding is that simple subword composition, with the input and output embedding matrices initialized asymmetrically, outperforms all external and learned methods. The best observed configuration uses uniform subword averaging plus Hindi-specific norm calibration on the input side and character-length-weighted averaging on the output head. If true, this gives practitioners a cheap way to select initializations without committing to full training runs, and it reduces the compute needed to adapt a model to a new language by a large factor.","feed_headline":"Better Hindi token embeddings cut language-adaptation training by 6x","feed_subtitle":"Uniform input plus length-weighted output reaches baseline loss 6x faster and beats Hindi accuracy at 500 steps.","key_machinery":"Weighted subword composition: each new token's embedding is initialized as e(τ) = Σ_i w_i e(s_i), where s_i are the token's constituent subwords and w_i come from uniform, character-length, max-character, or auxiliary semantic-similarity weighting. Norm calibration rescales the composed vector to the median L2 norm of native Hindi tokens, fixing the norm shrinkage caused by averaging. Asymmetry decouples the input lookup matrix from the output language-modeling head, allowing different weighting schemes for each. The decisive mechanism is that uniform input weights preserve broad coverage and avoid output-logit distortion, while character-length output weights give longer subwords more respo","core_discovery":"The paper's central claim is that the fastest way to start continued pre-training after vocabulary extension is not to borrow representations from external models but to compose the new token from its own subword pieces inside the original vocabulary—and to treat the input embedding matrix and the output language-modeling head as two different problems. Experimentally, the best configuration initializes the input matrix by uniformly averaging constituent subword embeddings and rescaling to the median norm of native Hindi tokens, and initializes the output head by averaging with weights proportional to each subword's character length. This combination records the lowest observed 50-step valid","pith_inferences":["If the mechanism is norm shrinkage plus the semantic breadth of uniform subword averaging, a natural extension is to test per-token adaptive weights—derived from tokenizer frequency or merger rank—in place of character length; the paper only evaluates fixed heuristics.","The same protocol (50-step probe, asymmetric subword composition, input-only norm calibration) is plausibly transferable to Arabic, CJK, or other scripts whose new tokens decompose into existing subwords, but the paper only demonstrates Hindi, so that transfer is an extrapolation.","A high-value stress test is to run the top three configurations with several random seeds on a different language and data blend; if the Val50 ranking reproduces under seed variation and matches a 500-step ranking, the selection protocol becomes a general standard, and if not, the practical takeaway is to choose within the top family rather than a single configuration."],"forward_implications":["Adopting the proposed pipeline as a default for Hindi vocabulary extension would cut cold-start compute: comparable validation loss in roughly 50 steps where the Mean-all baseline needs about 321, and MILU-Hindi accuracy at 500 steps already exceeds the baseline at 3,500 steps.","A 50-step CPT probe (about 0.42B tokens) can replace initialization loss and bits-per-byte as the selection signal, because relative rankings stabilize by step 50 in these runs.","Initialization-time metrics alone are misleading: methods with the best step-zero loss, such as FOCUS with anchor-Hindi, are overtaken after early CPT, and the sparse Max-char input mapping has the best cold-start footprint but the weakest 50-step validation loss in its cohort.","Input and output embeddings should be initialized with different rules; applying norm calibration symmetrically to the output head sharply worsens initialization loss and remains non-competitive after 50 steps.","The best initialization remains stable in a long 50B-token CPT run: Hindi accuracy improves slightly over the unextended model while English and multilingual benchmarks stay within evaluation noise, and Hindi token fertility drops from 1.95 to 1.25."],"supporting_citations":[{"why":"supplies the Mean-all default baseline (Gaussian around the mean embedding) that the paper must beat and uses as its primary reference.","marker":"(Hewitt, 2021)"},{"why":"introduced subword-averaging initialization, which the paper generalizes into the weighted composition family.","marker":"(Gee et al., 2022; Cui et al., 2023a)"},{"why":"provides FOCUS, the external sparsemax fastText alignment baseline that subword composition outperforms.","marker":"(Dobler and De Melo, 2023)"},{"why":"provides the WECHSEL retrieval-and-transfer paradigm on which the Top-k Gemma method is built.","marker":"(Minixhofer et al., 2022)"},{"why":"motivates the asymmetric input/output treatment by arguing that uniform mean initialization ignores the structural role a token plays in the output distribution.","marker":"(Nakash et al., 2025)"},{"why":"offers prior evidence that strategy rankings can shift after a short training phase, motivating the 50-step probe protocol.","marker":"(Yamaguchi et al., 2024)"},{"why":"supplies the Nanda tokenizer whose 25,600 Hindi tokens are added in the vocabulary-extension setup.","marker":"(Choudhury et al., 2025)"},{"why":"supplies the Sangraha Hindi corpus used to measure token fertility before and after extension.","marker":"(Khan et al., 2024)"},{"why":"defines the MILU-Hindi benchmark that provides the downstream accuracy checkpoints.","marker":"(Verma et al., 2025)"}],"fun_headline_variants":["Asymmetric subword init: 6x faster Hindi LLM adaptation","Uniform input, length-weighted output: best token init","Subword composition beats external embeddings for token init","Init loss unreliable: subword composition wins for Hindi"],"cache_read_input_tokens":16256,"weakest_assumption_plain":"The paper assumes that validation loss after just 50 continued-pretraining steps on one Hindi/English blend is a stable, low-noise oracle for ranking initialization strategies in much longer runs, and that the tiny gaps between the top configurations are real rather than seed-to-seed noise.","fun_headline_variants_meta":{"raw":{"variants":["Asymmetric subword init: 6x faster Hindi LLM adaptation","Uniform input, length-weighted output: best token init","Subword composition beats external embeddings for token init","Init loss unreliable: subword composition wins for Hindi"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":2905,"prompt_tokens":779,"completion_tokens":2126,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2069}},"tokens_in":523,"tokens_out":2126,"duration_ms":18035,"temperature":1.0,"reasoning_tokens":2069,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:58:45.327756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the top three configurations (Uniform+Char-len+Hindi Norm, MuRIL In+Char-len Out, and Mean-all) on a second low-resource script with several random seeds; if the Val50 ordering does not survive seed variation, or if a 500-step run overturns the 50-step ranking, the paper's selection protocol and 6x speedup claim would fail to reproduce. A cheaper check: compare the 2.722 vs 2.724 vs 2.752 Val50 gaps against seed standard deviation.","supporting_citations":[],"review_version":1}