Pith. sign in

REVIEW 2 major objections 6 minor 20 references

Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-pretraining on artificial languages does not reliably improve token efficiency across languages: benefits hinge on tokenizer, model size, and random seed, with stable gains only for small Llama-tokenizer models pretrained on 128-Dyck.

desk verdict A useful cautionary replication study on pre-pretraining, but the token-efficiency metric's asymmetric negative branch should temper how you read the 'loss' numbers. read the letter →

arxiv 2608.08800 v1 pith:4HI7PQU2 submitted 2026-08-09 cs.CL

Sofiia Riazhskykh , Nam Luu , Ondřej Bojar This is my paper · ORCID
classification cs.CL
keywords pre-pretrainingartificiallanguagestokenefficiencyk-DycktransferlearningreplicationstudyrandomseedvariancemultilingualLLMtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether pretraining an LLM on artificial bracket languages (k-Dyck and k-Shuffle Dyck) before training on natural text saves training tokens, as a previous English-only study reported with a 33% efficiency gain. Repeating the recipe on six languages, two tokenizers, three model sizes, and three random seeds each, the authors find that most configurations produce gains in some runs and losses in others, so the headline number does not survive replication. The only consistently positive configuration is a small 154M model with the Llama tokenizer pretrained on 128-Dyck, which improves token efficiency for most of the languages examined. The paper concludes that single-run evaluations can make unstable methods look reliable and argues for multi-seed validation in at least a subset of experiments.

What carries the argument

The central object is the token efficiency gain metric: the percentage of training tokens saved when the pretrained model reaches the baseline model's final validation loss, with the pretraining tokens counted and negative values assigned when it never catches up. The transfer recipe consists of a fixed 16,000-sequence, 2,048-token pretraining corpus of a formal bracket language (k-Dyck or k-Shuffle Dyck, with bracket types encoded as integers), followed by 10,000 steps on natural-language text. The argument is carried by comparing the loss curves of baseline and pretrained models and seeing whether the pretrained model's curve crosses the baseline's final loss before the training budget ends; the instability conclusion rests on how often that crossing point moves from before the end to after the end solely because the seed changed.

What would settle it

Take an unstable configuration, such as the 154M model with the Gemma tokenizer pretrained on 64-Dyck and then trained on Czech, and extend training to 20,000 or 40,000 steps across at least five seeds; compare how many tokens each model needs to reach the baseline's loss at the later checkpoint. If the pretrained model consistently reaches that loss sooner, the paper's instability conclusion would be falsified for that regime; if the ordering still flips across seeds, the conclusion stands.

Watch

Extended reading notes

Core claim

The paper's central discovery is that pre-pretraining on artificial languages is not a generally beneficial transfer technique; its effect on token efficiency is controlled by configuration details and by random seed. The 33% token-efficiency gain reported in prior work is reproduced only in a narrow corner of the experiment space: small models with the Llama tokenizer and 128-Dyck pretraining, where gains are stable across runs for most languages, and English also benefits with most artificial languages at small scale. Outside that corner, the same setup can yield a gain for one seed and a loss for another, and larger models frequently lose tokens outright. The authors also find no stable, statistically significant relationship between gains and twenty-plus linguistic properties including crossing dependencies, contrary to the structural-match hypothesis.

Load-bearing premise

The paper assumes that a model's relative standing at step 10,000, roughly 655M tokens, predicts its relative standing with more training, so the token-efficiency gain computed at that endpoint is the true gain; if longer training would reorder the baseline and pretrained models, the reported sign of the gain could reverse.

Editorial extensions

If this is right

  • For practitioners, pre-pretraining should be treated as a hyperparameter rather than a guaranteed saving; a configuration that helps in one seed can hurt in another, so budgets should include multiple seeds before adopting the method.
  • The stable recipe (small model, Llama tokenizer, 128-Dyck) is a concrete candidate for low-resource-language settings and for further scaling studies, but its limits outside this configuration remain unknown.
  • Because larger models in this study show repeated losses, the benefit of pre-pretraining appears to shrink as the model's token-to-parameter ratio decreases, which matters for compute-optimal training planning.
  • The near-absence of correlations with linguistic structure suggests that the mechanism of transfer is not the abstract syntax of the artificial language matching the natural language, so future work should look elsewhere, such as at tokenization interactions or optimization dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the seed variance is as large as reported, the common practice of reporting a single run overstates confidence in transfer results; one implication not drawn by the paper is that paired seed analysis, using the same seed for baseline and pretrained models, would isolate the pretraining effect from initialization noise.
  • The tokenizer dependence hints that the gain may come from how integer symbols are segmented, since the Llama tokenizer keeps multidigit numbers intact while the Gemma tokenizer splits them into digits, rather than from bracket syntax itself; a direct test would vary the symbol vocabulary while keeping the formal language fixed.
  • The paper's own token-to-parameter ratios are far below Chinchilla-optimal, and if longer training reverses the ordering of baseline and pretrained models, the instability could be an artifact of undertrained models, a possibility the authors flag but do not settle.
  • A cheap extension would be to test non-default pretraining budgets across languages: the paper's single Czech experiment suggests 1,000 pretraining steps may be better than 500 or 2,000, so the pretraining amount itself could be a tunable knob that escapes the unstable window.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper revisits Hu et al. (2025)'s claim that pre-pretraining on Dyck languages improves token efficiency for LLMs. It replicates the setup across six European languages, two tokenizers (Llama and Gemma), three model sizes (154M, 308M, 481M effective parameters), four formal languages (64/128-Dyck and 64/128-Shuffle Dyck), and three seeds per configuration, reporting per-run token-efficiency gains. The central finding is that gains are highly sensitive to the exact setup and seed; consistent positive gains appear only for the smallest Llama-tokenized model pretrained on 128-Dyck for most languages. The paper also correlates gains with linguistic features and finds little stable relationship, leading to the methodological recommendation that multiple training runs be performed before drawing conclusions.

Significance. If the conclusions hold, the paper provides a valuable cautionary counterpoint to the enthusiasm around pre-pretraining: the reported 33% token-efficiency gain does not generalize broadly, and the sign of the effect can flip with the random seed. The paper's strengths include explicit per-run results (Table 6), a multi-seed protocol, and a thorough ablation over tokenizer and model size. The paper also makes a fair methodological point about the danger of single-run evaluations. However, the main claim is currently weakened by a potentially biased token-efficiency metric: the negative branch of Algorithm 1 can manufacture large negative gains that are not a direct measure of the pretrained model's disadvantage. Because the instability conclusion relies on these values, the paper requires substantial revision before the central claim can be accepted.

major comments (2)
  1. [Appendix A.3, Algorithm 1; Table 6; Table 3] The negative branch of Algorithm 1 is asymmetric and can manufacture large negative 'losses'. When the pretrained model never reaches the baseline's final validation loss, the algorithm recursively computes the gain of the baseline relative to the pretrained model's final loss and negates it. This measures how quickly the baseline reaches the pretrained model's higher final loss, not how many tokens the pretrained model would need to reach the baseline's final loss. A small final-loss gap can therefore yield a large negative gain even if the pretrained model would catch up with only a short extension of training. Many of the negative values in Table 6 (e.g., -32.98%) and the '−' classifications in Table 3 are computed by this branch. Since the paper's central claim that pre-pretraining 'doesn't always help' depends on these values, the metric must be corrected (e.g., by interpolating the number of tokens at which the pretrained curve crosses the baseline's final loss, or by comparing area under the loss curves), or the authors must demonstrate that the negative branch does not affect the sign of the conclusions.
  2. [Section 4.1 and Table 2] The models are trained for only 655M tokens with token-to-parameter ratios as low as 0.74, far below the Chinchilla-optimal range of roughly 20–25. The assertion 'we see little to no chance that the setups (baseline vs. pretrained) would change order with more training' is not supported: the loss curves are shown to be stable only over the last 3,000 steps, and a small final-loss gap could plausibly reverse under continued training, especially given the documented seed variance. Since the token-efficiency gain is computed at the endpoint, an undetected reordering would change the sign of the gain and could undermine the instability conclusion. Please provide direct evidence (e.g., training a subset of setups for substantially longer, or an extrapolation analysis) or soften the conclusion.
minor comments (6)
  1. [Figure 1] The average token efficiency gain is shown without any measure of spread. Because the paper's main argument is that seed dependence is large, the figure should display per-run values or error bars; otherwise the reader cannot see the instability that the text emphasizes.
  2. [Section 4.1, Table 2] The comparison with Hu et al. (2025)'s token-to-parameter ratios is not exact: our ratios range from 0.74 to 2.6, while Hu et al. report 4.0 (160M) and 1.63 (1B). Please clarify that the models are generally more undertrained than the reference and discuss the implications for the comparability of the results.
  3. [Appendix A.3, Algorithm 1] The condition 'ifi≥T argetLoss∀i∈P' is ambiguous; it should read 'if for all i, P[i] ≥ TargetLoss'.
  4. [Section 5 and Limitations] The linguistic correlation analysis is limited to 154M models and six languages; the significant correlations for 154M+GT should be presented as exploratory, not confirmatory, and the abstract/conclusion should avoid overstating the relation. The Limitations section already acknowledges some of this, but the wording in Section 5 could be more cautious.
  5. [Table 6] A brief note explaining that the gain is defined as a percentage of token savings relative to the baseline (and that a negative value under the current metric indicates the baseline reaches the pretrained model's final loss earlier) would help readers interpret the entries.
  6. [Appendix A.3, Algorithm 1] There is a typo in 'P reP retrainedT okens' (the repeated 'P re' and missing space); the pseudocode would also benefit from a comment explaining why the negative branch is chosen and what it measures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical multi-run replication whose gains are measured against an external baseline; Algorithm 1's negative branch is a metric-validity concern, not a circular step.

full rationale

The paper is an empirical replication study, not a derivation chain. Its central quantity, token efficiency gain, is computed by Algorithm 1 against the baseline model's final validation loss, which is an external reference, and no parameter is fitted to the instability conclusion. The reported variance across seeds, model sizes, tokenizers, and pretraining languages is a direct empirical measurement, and the few self-citations are not load-bearing: Kocmi and Bojar (2018) is cited only as background motivation for transfer learning, and Bojar (2015) only supplies the morphological-richness metric. The dependency on Hu et al. (2025) for Dyck sequence generation and for the general gain protocol is an external benchmark, and the paper re-implements the setup and reports its own three-run results, so there is no imported uniqueness theorem or forced ansatz. The skeptic concern about Algorithm 1's negative branch is legitimate as a measurement-validity issue: the branch recursively computes the baseline's gain relative to the pretrained model's final loss and negates it, so a negative value may not represent the number of tokens the pretrained model would need to reach the baseline's final loss. However, that is a critique of the metric's semantics, not circularity: the paper does not derive a prediction from that metric by construction, and the instability conclusion is not equivalent to the metric's definition. No step reduces an output to an input by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its claims rest on standard empirical choices in LLM training, such as loss as the performance metric, and on the assumption that the achieved training budget is sufficient for the comparisons.

assumptions (3)
  • domain assumption Validation loss on mC4 is an adequate metric for language modeling performance and token efficiency.
    All efficiency gains are computed from final validation loss values (Section 4.1 and Algorithm 1). If loss is not a good proxy for the intended downstream quality, the gains are not meaningful.
  • domain assumption Training and evaluation loss curves are stable in the last 3,000 steps and will not reorder with more training.
    The authors claim little chance of reordering (Section 4.1) despite undertrained models. This is an assumption about convergence that is not formally verified.
  • domain assumption The generated k-Dyck and Shuffle-Dyck sequences correctly implement the intended formal languages.
    The authors refer to Hu et al. (2025) for the exact implementation and provide no independent verification of the generated sequences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages." pith.science (2026). https://pith.science/paper/4HI7PQU2

@misc{pith2026260808800,
  author       = {Pith},
  title        = {Pith review of: Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4HI7PQU2}},
  note         = {Machine review of arXiv:2608.08800}
}
read the original abstract

Pretraining LLMs on artificial languages ("pre-pretraining") is a technique that could reportedly increase token efficiency by 33%, i.e., save up to 33% of training tokens needed to reach a certain performance. We validate this prior result for English on a larger set of natural languages across four language families, using two different tokenizers and varying model sizes. We also relate the observed gains (or losses) in token efficiency to quantified linguistic properties of the languages, such as sentence length, morphological richness, and features of dependency syntactic trees (tree depth, number of children, number of crossing dependencies). Our empirical results indicate that the reported gains depend heavily on the experiment setup and the choice of random seed, although we can confirm the trend of stable gains with 128-Dyck pretraining of small models with the Llama tokenizer for most of the examined languages. On a general note, we argue that multiple training runs should be carried out at least for a subset of experiments to avoid the community adopting unstable approaches.

Figures

Figures reproduced from arXiv: 2608.08800 by the authors.

Figure 1
Figure 1. Average token efficiency gains of all mod [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Evaluation loss of the 481M+GT model for Danish, in the last 3,000 training steps. Lower is better. Model Size Tokens Ratio Hu et al. (2025) Pythia 160M 655M ≈ 4 : 1 Pythia 1B 1.63B 1.63 : 1 Ours 154M+LT 253M 655M ≈ 2.6 : 1 (Llama-3-based 154M+GT 355M 655M ≈ 1.8 : 1 models) 308M+LT 440M 655M ≈ 1.5 : 1 308M+GT 577M 655M ≈ 1.1 : 1 481M+LT 678M 655M ≈ 0.97 : 1 481M+GT 884M 655M ≈ 0.74 : 1 [PITH_FULL_IMAGE:figures/full… view at source ↗
Figure 4
Figure 4. Average token efficiency gains by language [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (17 more)
Figure 5
Figure 5. Figure 5: Average token efficiency gains of 154M+GT pretrained on 64-Dyck for Czech, with three different pretraining checkpoints. Right is better. 5 Language Specific Analysis Might there be any correlation between the gain and the linguistic properties? Due to observed in￾cons…
Figure 7
Figure 7. Figure 7: Loss curves of the 154M+LT model for Czech. The right-side figure is the zoomed-in version of the left one, from step 7,000 to 10,000. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Loss curves of the 154M+GT model for Czech. The right-side figure is the zoomed-in version of the left one, from step 7,000 to 10,000. 0 1k 2k 3k 4k 5k 6k 7k 8k 9k 10k 2 4 6 8 10 12 Loss Full curves Base 64-Dyck 128-Dyck 64-Shuff Dyck 128-Shuff Dyck 7k 8k 9k 10k 1.80 1…
Figure 9
Figure 9. Figure 9: Loss curves of the 481M+LT model for Albanian. The right-side figure is the zoomed-in version of the left one, from step 7,000 to 10,000. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Loss curves of the 481M+GT model for Albanian. The right-side figure is the zoomed-in version of the left one, from step 7,000 to 10,000. 154M +LT 154M +GT 308M +LT 308M +GT 481M +LT 481M +GT 40 20 0 20 40 60 Token Efficiency Gain (%) Pretraining language 64-Dyck 128-…
Figure 11
Figure 11. Figure 11: Efficiency gain across all models, pretraining, and natural languages. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Efficiency gain across all models, pretraining, and natural languages (with language-specific details). [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Average correlation between linguistic features of evaluation data and token efficiency gain of 154M [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Average correlation between linguistic features of training data and token efficiency gain of 154M models [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Average correlation between linguistic features of evaluation data and token efficiency gain of 154M [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Average correlation between linguistic features of training data and token efficiency gain of 154M models [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Average correlation between linguistic features of evaluation data and token efficiency gain of 154M [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Average correlation between linguistic features of training data and token efficiency gain of 154M models [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 19
Figure 19. Figure 19: Average correlation between linguistic features of evaluation data and token efficiency gain of 154M [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]
Figure 20
Figure 20. Figure 20: Average correlation between linguistic features of training data and token efficiency gain of 154M models [PITH_FULL_IMAGE:figures/full_fig_p019_20.png]
Figure 21
Figure 21. Figure 21: Correlation between linguistic features and token efficiency gain (evaluation data) [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: Correlation between linguistic features and token efficiency gain (training data) [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 4 canonical work pages

  1. [1]

    Choosing Transfer Languages for Cross-Lingual Learning

    Lin, Yu-Hsiang and Chen, Chian-Yu and Lee, Jean and Li, Zirui and Zhang, Yuyan and Xia, Mengzhou and Rijhwani, Shruti and He, Junxian and Zhang, Zhisong and Ma, Xuezhe and Anastasopoulos, Antonios and Littell, Patrick and Neubig, Graham. Choosing Transfer Languages for Cross-Lingual Learning. Proceedings of the 57th Annual Meeting of the Association for C...

    doi:10.18653/v1/p19-1301 2019

  2. [2]

    Trivial Transfer Learning for Low-Resource Neural Machine Translation

    Kocmi, Tom and Bojar, Ond r ej. Trivial Transfer Learning for Low-Resource Neural Machine Translation. Proceedings of the Third Conference on Machine Translation: Research Papers. 2018. doi:10.18653/v1/W18-6325

    doi:10.18653/v1/w18-6325 2018

  3. [3]

    and Petty, Jackson and Shi, Chuan and Merrill, William and Linzen, Tal

    Hu, Michael Y. and Petty, Jackson and Shi, Chuan and Merrill, William and Linzen, Tal. Between Circuits and C homsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.478

    doi:10.18653/v1/2025.acl-long.478 2025

  4. [4]

    Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

    Ri, Ryokan and Tsuruoka, Yoshimasa. Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. doi:10.18653/v1/2022.acl-long.504

    doi:10.18653/v1/2022.acl-long.504 2022

  5. [5]

    Pretraining Language Models with L o RA and Artificial Languages

    Kumar, Nalin and Lango, Mateusz and Dusek, Ondrej. Pretraining Language Models with L o RA and Artificial Languages. Proceedings of the First BabyLM Workshop. 2025. doi:10.18653/v1/2025.babylm-main.37

    work page doi:10.18653/v1/2025.babylm-main.37 2025

  6. [6]

    Transfer of Structural Knowledge from Synthetic Languages

    Budnikov, Mikhail and Yamshchikov, Ivan. Transfer of Structural Knowledge from Synthetic Languages. Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025). 2025. doi:10.18653/v1/2025.xllm-1.20

    doi:10.18653/v1/2025.xllm-1.20 2025

  7. [7]

    2506.20920 , archivePrefix=

    Guilherme Penedo and Hynek Kydlíček and Vinko Sabolčec and Bettina Messmer and Negar Foroutan and Amir Hossein Kargaran and Colin Raffel and Martin Jaggi and Leandro Von Werra and Thomas Wolf , year=. 2506.20920 , archivePrefix=

    arXiv

  8. [8]

    The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    Guilherme Penedo and Hynek Kydl. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    work page

Show all 20 references
  1. [9]

    O pen S ubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

    Lison, Pierre and Tiedemann, J. O pen S ubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC '16). 2016

    2016

  2. [10]

    Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12) , year =

    Jörg Tiedemann , title =. Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12) , year =

  3. [11]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. CoRR , volume =. 2019 , url =. 1910.10683 , timestamp =

    2019 arXiv

  4. [12]

    2015 , publisher=

    Christodouloupoulos, Christos and Steedman, Mark , journal=. 2015 , publisher=

    2015

  5. [13]

    The Oxford Handbook of Inflection , publisher =

    Bojar, Ondřej , isbn =. The Oxford Handbook of Inflection , publisher =. 2015 , month =. doi:10.1093/oxfordhb/9780199591428.013.13 , url =

    2015

  6. [14]

    Injecting structural hints: Using language models to study inductive biases in language learning

    Papadimitriou, Isabel and Jurafsky, Dan. Injecting structural hints: Using language models to study inductive biases in language learning. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings-emnlp.563

    2023 doi

  7. [15]

    Aaron Grattafiori and Abhimanyu Dubey and Abhinav Jauhri and Abhinav Pandey and Abhishek Kadian and Ahmad Al-Dahle and Aiesha Letman and Akhil Mathur and Alan Schelten and Alex Vaughan and Amy Yang and Angela Fan and Anirudh Goyal and Anthony Hartshorn and Aobo Yang and Archi ...

    arXiv

  8. [16]

    Gemma Team and Aishwarya Kamath and Johan Ferret and Shreya Pathak and Nino Vieillard and Ramona Merhej and Sarah Perrin and Tatiana Matejovicova and Alexandre Ramé and Morgane Rivière and Louis Rouillard and Thomas Mesnard and Geoffrey Cideron and Jean-bastien Grill and Sabel...

    arXiv

  9. [17]

    doi:10.18653/v1/2024.acl-long.44 , booktitle=

    Bandarkar, Lucas and Liang, Davis and Muller, Benjamin and Artetxe, Mikel and Shukla, Satya Narayan and Husa, Donald and Goyal, Naman and Krishnan, Abhinandan and Zettlemoyer, Luke and Khabsa, Madian , year=. doi:10.18653/v1/2024.acl-long.44 , booktitle=

    doi

  10. [18]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and...

    doi

  11. [19]

    2023 , organization=

    Biderman, Stella and Schoelkopf, Hailey and Anthony, Quentin Gregory and Bradley, Herbie and O’Brien, Kyle and Hallahan, Eric and Khan, Mohammad Aflah and Purohit, Shivanshu and Prashanth, USVSN Sai and Raff, Edward and others , booktitle=. 2023 , organization=

    2023

  12. [20]

    Rae and Oriol Vinyals and Laurent Sifre , year=

    Jordan Hoffmann and Sebastian Borgeaud and Arthur Mensch and Elena Buchatskaya and Trevor Cai and Eliza Rutherford and Diego de Las Casas and Lisa Anne Hendricks and Johannes Welbl and Aidan Clark and Tom Hennigan and Eric Noland and Katie Millican and George van den Driessche...

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.