{"id":"52087359-ae3b-432c-85b9-32304cd88ede","arxiv_id":"2608.08800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A multilingual, multi-seed study finds that LLM pre-pretraining gains are highly dependent on setup and random seed, with consistent benefits confined to small models with the Llama tokenizer on 128-Dyck.","lead":"Pre-pretraining LLMs on artificial bracket languages such as Dyck was previously reported to cut training tokens by up to 33%. Testing six languages, two tokenizers, and three model sizes across three seeds each, this paper finds the gains are highly unstable and only reliably appear for small Llama-tokenized models pretrained on 128-Dyck.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's negative token-efficiency gain is asymmetric and can manufacture 'losses'; the instability conclusion relies on these values.","rationale":"The reader's weakest assumption focused on asymptotic ordering of undertrained models. My concern is more immediate: the quantitative token-efficiency gain itself is computed by an asymmetric algorithm whose negative branch does not measure actual token inefficiency. This threatens the central empirical claim that pre-pretraining often hurts, because many of the 'loss' labels and the mixed/negative counts in Tables 3 and 6 arise from that branch. The paper does provide raw loss curves for a few setups, but not for all runs, so the summary statistics cannot be independently verified without code. The recommended verdict remains conditional, but the condition should be a re-analysis with a symmetric and robust token-efficiency metric (or extended training), not merely adding error bars or publishing code. If the re-analysis still shows seed-dependent sign flips, the instability conclusion would stand; if not, the headline claim would need to be weakened. I partially agree with the reader because the endpoint-versus-asymptotic issue is related but distinct: even with a correct metric, undertrained models might reorder with longer training, but the metric flaw is a more direct and actionable problem.","tokens_in":36429,"tokens_out":9960,"duration_ms":86986,"concrete_test":"For every run in Table 6 with a negative gain, continue training the pretrained model for an additional 10,000 steps (or until it first reaches and stays below the baseline's final validation loss), using the same optimizer and data order, and record the actual additional tokens required. Then compare this measured token cost with the magnitude implied by the Algorithm 1 negative gain. If most negative runs reach the baseline's final loss within a few thousand steps, the reported losses are artifacts of the asymmetric metric. As an analytical complement, construct two synthetic loss curves (e.g., baseline final loss 3.00, pretrained final loss 3.01 reached within 1 step) and show that Algorithm 1 returns a negative gain near -100%, whereas the true additional tokens for the pretrained model to reach 3.00 is finite and small.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that pre-pretraining 'doesn't always help' rests on token-efficiency gains computed by Algorithm 1 (Appendix A.3). The negative branch is asymmetric: if the pretrained model never reaches the baseline's final validation loss, the algorithm recursively computes the gain of the baseline relative to the pretrained model's final loss and negates it. This measures how quickly the baseline reaches the pretrained model's higher final loss, not how many additional tokens the pretrained model would need to reach the baseline's final loss. A small final-loss gap can therefore yield a large negative 'gain' even if the pretrained model would catch up within a short extension. For example, if the pretrained final loss is only 0.001 above the baseline final loss but the baseline crossed that higher value early in training, the reported gain approaches -100%. Table 6 contains many negative values (down to -32.98%) computed this way. Since the paper's evidence of instability (e.g., Figure 3, Table 3) uses these gains to label setups as losses or mixed, the conclusion that pre-pretraining often hurts may be an artifact of the metric rather than a property of the training runs. The authors assert 'little to no chance' of reordering with more training (Section 4.1), but this assertion does not address the asymmetry in the metric. The positive branch is also sensitive to the noisy final validation loss of the baseline, which is used as a single-point target.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper revisits Hu et al. (2025)'s claim that pre-pretraining on Dyck languages improves token efficiency for LLMs. It replicates the setup across six European languages, two tokenizers (Llama and Gemma), three model sizes (154M, 308M, 481M effective parameters), four formal languages (64/128-Dyck and 64/128-Shuffle Dyck), and three seeds per configuration, reporting per-run token-efficiency gains. The central finding is that gains are highly sensitive to the exact setup and seed; consistent positive gains appear only for the smallest Llama-tokenized model pretrained on 128-Dyck for most languages. The paper also correlates gains with linguistic features and finds little stable relationship, leading to the methodological recommendation that multiple training runs be performed before drawing conclusions.","tokens_in":36684,"tokens_out":7458,"duration_ms":78163,"significance":"If the conclusions hold, the paper provides a valuable cautionary counterpoint to the enthusiasm around pre-pretraining: the reported 33% token-efficiency gain does not generalize broadly, and the sign of the effect can flip with the random seed. The paper's strengths include explicit per-run results (Table 6), a multi-seed protocol, and a thorough ablation over tokenizer and model size. The paper also makes a fair methodological point about the danger of single-run evaluations. However, the main claim is currently weakened by a potentially biased token-efficiency metric: the negative branch of Algorithm 1 can manufacture large negative gains that are not a direct measure of the pretrained model's disadvantage. Because the instability conclusion relies on these values, the paper requires substantial revision before the central claim can be accepted.","major_comments":[{"comment":"The negative branch of Algorithm 1 is asymmetric and can manufacture large negative 'losses'. When the pretrained model never reaches the baseline's final validation loss, the algorithm recursively computes the gain of the baseline relative to the pretrained model's final loss and negates it. This measures how quickly the baseline reaches the pretrained model's higher final loss, not how many tokens the pretrained model would need to reach the baseline's final loss. A small final-loss gap can therefore yield a large negative gain even if the pretrained model would catch up with only a short extension of training. Many of the negative values in Table 6 (e.g., -32.98%) and the '−' classifications in Table 3 are computed by this branch. Since the paper's central claim that pre-pretraining 'doesn't always help' depends on these values, the metric must be corrected (e.g., by interpolating the number of tokens at which the pretrained curve crosses the baseline's final loss, or by comparing area under the loss curves), or the authors must demonstrate that the negative branch does not affect the sign of the conclusions.","section":"Appendix A.3, Algorithm 1; Table 6; Table 3"},{"comment":"The models are trained for only 655M tokens with token-to-parameter ratios as low as 0.74, far below the Chinchilla-optimal range of roughly 20–25. The assertion 'we see little to no chance that the setups (baseline vs. pretrained) would change order with more training' is not supported: the loss curves are shown to be stable only over the last 3,000 steps, and a small final-loss gap could plausibly reverse under continued training, especially given the documented seed variance. Since the token-efficiency gain is computed at the endpoint, an undetected reordering would change the sign of the gain and could undermine the instability conclusion. Please provide direct evidence (e.g., training a subset of setups for substantially longer, or an extrapolation analysis) or soften the conclusion.","section":"Section 4.1 and Table 2"}],"minor_comments":[{"comment":"The average token efficiency gain is shown without any measure of spread. Because the paper's main argument is that seed dependence is large, the figure should display per-run values or error bars; otherwise the reader cannot see the instability that the text emphasizes.","section":"Figure 1"},{"comment":"The comparison with Hu et al. (2025)'s token-to-parameter ratios is not exact: our ratios range from 0.74 to 2.6, while Hu et al. report 4.0 (160M) and 1.63 (1B). Please clarify that the models are generally more undertrained than the reference and discuss the implications for the comparability of the results.","section":"Section 4.1, Table 2"},{"comment":"The condition 'ifi≥T argetLoss∀i∈P' is ambiguous; it should read 'if for all i, P[i] ≥ TargetLoss'.","section":"Appendix A.3, Algorithm 1"},{"comment":"The linguistic correlation analysis is limited to 154M models and six languages; the significant correlations for 154M+GT should be presented as exploratory, not confirmatory, and the abstract/conclusion should avoid overstating the relation. The Limitations section already acknowledges some of this, but the wording in Section 5 could be more cautious.","section":"Section 5 and Limitations"},{"comment":"A brief note explaining that the gain is defined as a percentage of token savings relative to the baseline (and that a negative value under the current metric indicates the baseline reaches the pretrained model's final loss earlier) would help readers interpret the entries.","section":"Table 6"},{"comment":"There is a typo in 'P reP retrainedT okens' (the repeated 'P re' and missing space); the pseudocode would also benefit from a comment explaining why the negative branch is chosen and what it measures.","section":"Appendix A.3, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript addresses an important and timely question, and the multi-seed, per-run data are a valuable contribution. The major concern is the correctness of the token-efficiency metric in Algorithm 1; the paper's central negative result is currently supported by a potentially biased measure. I believe the issue is fixable and that a thoroughly revised version could be publishable. I also recommend asking the authors to release code and model checkpoints to facilitate verification of the per-run results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a referee's time, but read the metric before you cite it. The authors extend pre-pretraining tests to six languages, two tokenizers, three model sizes, and three seeds per configuration, and they show that the prior 33% token-efficiency claim from Hu et al. does not reproduce reliably outside a narrow setup (154M + Llama tokenizer + 128-Dyck). That is a real contribution: they make a strong case that single-run results in this area are fragile, and their call for multi-seed validation is sensible and overdue.\n\nWhat they do well: the per-run results are reported in full (Table 6), the loss curves are shown, and the limitations section is honest about the models being undertrained and the language set being small. The linguistic-correlation analysis is a reasonable first pass, and the null result for crossing dependencies is worth reporting even if it's post-hoc. The citation pattern is solid; the prior work is clearly framed and the differences are explicit.\n\nThe soft spots, in proportion. First, the token-efficiency metric in Algorithm 1 is asymmetric. The positive branch measures how many tokens the pretrained model saves to reach the baseline's final loss; the negative branch, when the pretrained never reaches that loss, measures how quickly the baseline reaches the pretrained's final loss, then negates it. Those are not the same quantity. A slightly worse pretrained model can generate a misleadingly large negative number if the baseline crossed that higher value early. Looking at the actual data, the large negative gains (-20% to -30%) do correspond to runs where the pretrained model is clearly worse, so I don't think the instability conclusion is a pure artifact. But the asymmetry inflates the magnitude of losses and makes sign-based classifications (Table 3) less clean than they look. The authors should acknowledge this and ideally report a symmetric metric, such as area under the loss curve or extrapolated convergence.\n\nSecond, the models are undertrained (0.74 to 2.6 tokens per parameter). The authors argue the ordering won't change with more training, but they don't prove it. The loss curves in the last 3k steps are stable, but that's not the same as showing the pre-pretraining advantage or disadvantage persists asymptotically. This is a moderate concern, not a fatal one.\n\nThird, Figure 1 averages over runs without error bars; the per-run figures show the spread, but the headline figure hides it. Easy fix.\n\nFor the right reader—someone working on pre-pretraining or multi-seed training protocols—this paper is useful. I'd send it to review; the core message (single-run gains are unreliable) is important, and the data quality is good enough to stand. Just expect the reviewers to push on the metric.","headline":"A useful cautionary replication study on pre-pretraining, but the token-efficiency metric's asymmetric negative branch should temper how you read the 'loss' numbers.","tokens_in":37191,"tokens_out":6665,"would_cite":true,"duration_ms":67814,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-pretraining on artificial languages does not reliably improve token efficiency across languages: benefits hinge on tokenizer, model size, and random seed, with stable gains only for small Llama-tokenizer models pretrained on 128-Dyck.","keywords":["pre-pretraining","artificial languages","token efficiency","k-Dyck languages","transfer learning","replication study","random seed variance","multilingual LLM training"],"falsifier":"Take an unstable configuration, such as the 154M model with the Gemma tokenizer pretrained on 64-Dyck and then trained on Czech, and extend training to 20,000 or 40,000 steps across at least five seeds; compare how many tokens each model needs to reach the baseline's loss at the later checkpoint. If the pretrained model consistently reaches that loss sooner, the paper's instability conclusion would be falsified for that regime; if the ordering still flips across seeds, the conclusion stands.","tokens_in":36239,"feed_emoji":"🎲","tokens_out":9240,"duration_ms":87059,"temperature":0.7,"pith_summary":"The paper tests whether pretraining an LLM on artificial bracket languages (k-Dyck and k-Shuffle Dyck) before training on natural text saves training tokens, as a previous English-only study reported with a 33% efficiency gain. Repeating the recipe on six languages, two tokenizers, three model sizes, and three random seeds each, the authors find that most configurations produce gains in some runs and losses in others, so the headline number does not survive replication. The only consistently positive configuration is a small 154M model with the Llama tokenizer pretrained on 128-Dyck, which improves token efficiency for most of the languages examined. The paper concludes that single-run evaluations can make unstable methods look reliable and argues for multi-seed validation in at least a subset of experiments.","feed_headline":"Pre-pretraining gains don't generalize across languages or seeds","feed_subtitle":"Multi-seed runs show gains flip sign; only small Llama-tokenizer models with 128-Dyck improve reliably.","key_machinery":"The central object is the token efficiency gain metric: the percentage of training tokens saved when the pretrained model reaches the baseline model's final validation loss, with the pretraining tokens counted and negative values assigned when it never catches up. The transfer recipe consists of a fixed 16,000-sequence, 2,048-token pretraining corpus of a formal bracket language (k-Dyck or k-Shuffle Dyck, with bracket types encoded as integers), followed by 10,000 steps on natural-language text. The argument is carried by comparing the loss curves of baseline and pretrained models and seeing whether the pretrained model's curve crosses the baseline's final loss before the training budget ends; the instability conclusion rests on how often that crossing point moves from before the end to after the end solely because the seed changed.","core_discovery":"The paper's central discovery is that pre-pretraining on artificial languages is not a generally beneficial transfer technique; its effect on token efficiency is controlled by configuration details and by random seed. The 33% token-efficiency gain reported in prior work is reproduced only in a narrow corner of the experiment space: small models with the Llama tokenizer and 128-Dyck pretraining, where gains are stable across runs for most languages, and English also benefits with most artificial languages at small scale. Outside that corner, the same setup can yield a gain for one seed and a loss for another, and larger models frequently lose tokens outright. The authors also find no stable, statistically significant relationship between gains and twenty-plus linguistic properties including crossing dependencies, contrary to the structural-match hypothesis.","pith_inferences":["If the seed variance is as large as reported, the common practice of reporting a single run overstates confidence in transfer results; one implication not drawn by the paper is that paired seed analysis, using the same seed for baseline and pretrained models, would isolate the pretraining effect from initialization noise.","The tokenizer dependence hints that the gain may come from how integer symbols are segmented, since the Llama tokenizer keeps multidigit numbers intact while the Gemma tokenizer splits them into digits, rather than from bracket syntax itself; a direct test would vary the symbol vocabulary while keeping the formal language fixed.","The paper's own token-to-parameter ratios are far below Chinchilla-optimal, and if longer training reverses the ordering of baseline and pretrained models, the instability could be an artifact of undertrained models, a possibility the authors flag but do not settle.","A cheap extension would be to test non-default pretraining budgets across languages: the paper's single Czech experiment suggests 1,000 pretraining steps may be better than 500 or 2,000, so the pretraining amount itself could be a tunable knob that escapes the unstable window."],"forward_implications":["For practitioners, pre-pretraining should be treated as a hyperparameter rather than a guaranteed saving; a configuration that helps in one seed can hurt in another, so budgets should include multiple seeds before adopting the method.","The stable recipe (small model, Llama tokenizer, 128-Dyck) is a concrete candidate for low-resource-language settings and for further scaling studies, but its limits outside this configuration remain unknown.","Because larger models in this study show repeated losses, the benefit of pre-pretraining appears to shrink as the model's token-to-parameter ratio decreases, which matters for compute-optimal training planning.","The near-absence of correlations with linguistic structure suggests that the mechanism of transfer is not the abstract syntax of the artificial language matching the natural language, so future work should look elsewhere, such as at tokenization interactions or optimization dynamics."],"supporting_citations":[{"why":"supplies the 33% token-efficiency claim and the pre-pretraining recipe (integer-encoded Dyck sequences, 500 pretraining steps) that this paper replicates.","marker":"Hu et al. (2025)"},{"why":"established that training on artificial languages transfers to natural-language modeling and introduced the nesting-versus-flat dependency comparison.","marker":"Ri and Tsuruoka (2022)"},{"why":"extended structural pretraining to decoder-only language models and reported perplexity reductions that motivate the transfer hypothesis.","marker":"Papadimitriou and Jurafsky (2023)"},{"why":"showed that larger pretraining corpora yield larger transfer gains, which this paper controls for by fixing the pretraining corpus size.","marker":"Kocmi and Bojar (2018)"},{"why":"provides the Chinchilla token-to-parameter benchmark used to classify this study's models as undertrained.","marker":"Hoffmann et al. (2022)"},{"why":"provides the FineWeb English corpus used in the natural-language training mixture.","marker":"Penedo et al. (2024)"},{"why":"provides FineWeb2, the multilingual corpus used for the five non-English languages.","marker":"Penedo et al. (2025)"},{"why":"supplies OpenSubtitles, one of the three corpora mixed into each language's training set.","marker":"Lison and Tiedemann (2016)"},{"why":"supplies the EUBookshop corpus used alongside FineWeb and OpenSubtitles.","marker":"Tiedemann (2012)"},{"why":"provides the parallel Bible corpus used to estimate morphological richness across languages.","marker":"Christodouloupoulos and Steedman (2015)"}],"fun_headline_variants":["Pre-pretraining gains flip sign with seed and scale","Pre-pretraining's 33% gain is a lucky-seed artifact","Stable gains only with 128-Dyck, small models, Llama tokenizer","Pre-pretraining token savings are seed-dependent, not universal","Multi-seed runs reveal pre-pretraining gains are a fluke"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a model's relative standing at step 10,000, roughly 655M tokens, predicts its relative standing with more training, so the token-efficiency gain computed at that endpoint is the true gain; if longer training would reorder the baseline and pretrained models, the reported sign of the gain could reverse.","fun_headline_variants_meta":{"raw":{"variants":["Pre-pretraining gains flip sign with seed and scale","Pre-pretraining's 33% gain is a lucky-seed artifact","Stable gains only with 128-Dyck, small models, Llama tokenizer","Pre-pretraining token savings are seed-dependent, not universal","Multi-seed runs reveal pre-pretraining gains are a fluke"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000535,"raw_usage":{"total_tokens":2540,"prompt_tokens":885,"completion_tokens":1655,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1559}},"tokens_in":501,"tokens_out":1655,"duration_ms":11701,"temperature":1.0,"reasoning_tokens":1559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:23:24.594287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an unstable configuration, such as the 154M model with the Gemma tokenizer pretrained on 64-Dyck and then trained on Czech, and extend training to 20,000 or 40,000 steps across at least five seeds; compare how many tokens each model needs to reach the baseline's loss at the later checkpoint. If the pretrained model consistently reaches that loss sooner, the paper's instability conclusion would be falsified for that regime; if the ordering still flips across seeds, the conclusion stands.","supporting_citations":[{"cited_title":"2015 , publisher=","cited_arxiv_id":null,"evidence_quote":"provides the parallel Bible corpus used to estimate morphological richness across languages."}],"review_version":1}