Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Transformers can perform in-context prediction on Permuted Congruential Generator sequences, with required context growing as the square root of the modulus.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:03 UTC pith:VWPP6L7N

load-bearing objection Solid capability result — transformers can in-context learn PCGs and generalize to unseen parameters — but the sqrt(m) scaling law is softer than the abstract implies: the largest modulus is measured under a different training protocol and not at the predicted context length. the 3 major comments →

arxiv 2510.26792 v2 pith:VWPP6L7N submitted 2025-10-30 cs.LG cond-mat.dis-nncs.CR

Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability

Tao Tao , Maissam Barkeshli This is my paper
classification cs.LG cond-mat.dis-nncs.CR
keywords pseudo-random number generatorsPermuted Congruential GeneratorsTransformersin-context learningscaling lawcurriculum learninginterpretabilityembedding analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether Transformers can learn to predict outputs of Permuted Congruential Generators (PCGs), a widely used family of pseudo-random number generators. It claims they can: a single model trained on sequences from several PCG variants predicts unseen sequences in-context with over 90% accuracy, even when the output is truncated to a single bit. The paper also establishes a scaling law: the number of past outputs needed for near-perfect prediction grows as the square root of the modulus, a steeper requirement than for simpler linear congruential generators. Finally, it reports that training on large moduli (m >= 2^20) stalls without curriculum learning, and that the learned embeddings spontaneously organize tokens by rotation-invariant binary zero-run patterns, which supports transfer from small to large moduli.

Core claim

The central discovery is that Transformers can perform in-context prediction on PCG sequences without knowing the generator's parameters, generalizing to unseen multipliers and increments. For moduli m up to 2^22, the context length required to exceed 90% test accuracy follows 0.5 sqrt(m). For LCGs the analogous law is m^0.25, so PCG's output permutations make the task substantially harder. Training on large moduli requires a curriculum: initializing from a smaller-modulus model and slowly decaying the fraction of smaller-modulus data; without it, optimization stagnates at high loss for at least 75k steps. The paper further finds that the embedding layer encodes rotation-invariant zero-run s

What carries the argument

The central object is the PCG recurrence s_i = (a s_{i-1} + c) mod m with an output permutation f (shift, XOR, rotations, truncations) controlled by state bits. The key quantitative identity is the empirical scaling law: required context C(m) ~ 0.5 sqrt(m) for 90% accuracy, derived from accuracy-position curves. The curriculum mechanism—pretrained initialization plus exponential decay of smaller-modulus data mixing—carries the argument for scaling to m >= 2^20. The interpretability result uses PCA of the embedding matrix, where PC1 and PC2 correlate with total zero count and number of zero runs, respectively.

Load-bearing premise

The scaling law and the necessity of curriculum training are measured under a single training configuration up to m=2^22 with no error bars, so the claims assume that these behaviors persist at larger moduli and under longer training budgets.

What would settle it

Train on XSLRR with m=2^24 or m=2^26 using the same 75k-step budget and curriculum; if the context length required for 90% accuracy grows faster than sqrt(m), or if direct training without curriculum converges when given, say, 300k steps, the scaling law and the 'critical necessity' of curriculum would be refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • PCG outputs, including single-bit truncations, are predictable by Transformers from sequence context alone, without knowing the multiplier or increment.
  • The required context grows as sqrt(m), steeper than the m^0.25 law for LCGs, so the permutation operations measurably increase the difficulty.
  • Curriculum learning with a pretrained small-modulus model is necessary to train on moduli 2^20 and above within a fixed step budget; without it optimization stagnates.
  • The embedding layer's rotation-invariant zero-run clusters transfer across moduli, explaining how pretrained initialization accelerates training.
  • In combined training, the model separates generator types in its middle layers, allowing a single model to handle multiple PRNG variants.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the sqrt(m) scaling extrapolates to m=2^64, a model would need roughly 2^32 in-context elements, so current Transformers would not threaten practical generators directly; the value is in studying the mechanism, not in a practical break.
  • The rotation-invariant clustering suggests the model learns a representation of binary strings modulo rotation; one could test whether this is specific to PCGs with random rotations by training on LCGs with similar rotation output and checking whether the same clusters emerge.
  • The curriculum finding may generalize to other algorithmic tasks where the target distribution is unimodal in difficulty; the paper's decay schedule comparison suggests exponential decay is a robust choice.
  • A testable extension: measure whether the required context for other PCG variants (XSHRR, XSHRS) also follows sqrt(m), or whether the exponent depends on the permutation structure; the paper only reports the scaling for XSLRR.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies whether Transformer models can learn to predict outputs of Permuted Congruential Generators (PCGs) in-context, given only a sequence of observed outputs and no explicit knowledge of the generator's parameters. The authors train GPT-style models on several PCG variants (TLCG, XSLRR, XSHRR, XSHRS) across moduli from 2^14 to 2^22, using up to 52M parameters and 5.4B tokens. They report high test accuracy on unseen (a,c) parameters, robust prediction even under single-bit truncation, successful joint learning of multiple generator types, a scaling law in which the context length required for 90% accuracy grows as 0.5*sqrt(m), the 'critical necessity' of curriculum learning/pretrained initialization for m>=2^20, and an interpretability result showing that token embeddings cluster by rotation-invariant zero-run patterns. The paper claims this goes beyond published classical PCG attacks, which assume the multiplier and modulus are known.

Significance. If the core results hold, this is a useful contribution to the empirical study of Transformers on structured algorithmic tasks: it shows that a non-linear, truncated state-update family can be learned in-context, and the embedding analysis is a valuable interpretability datapoint. The authors provide code and a detailed appendix with additional ablations, which supports reproducibility. The main advertised quantitative claim—the sqrt(m) scaling law—and the 'critical necessity' of curriculum learning are less secure and currently exceed the evidence. The capability result (reliable prediction across PCG variants) is convincing from Figures 2 and 3(left), but the scaling and curriculum claims need substantial qualification or additional experiments before the paper should be accepted.

major comments (3)
  1. [§4.2, Figure 3, Appendix B] The headline scaling law 'required context grows as 0.5*sqrt(m)' is not directly supported at the largest modulus. For m=2^22, Appendix B reports only the accuracy at position 1279 (0.9257), not at the predicted 0.5*sqrt(m)=1024. No position-1024 accuracy is shown, so the plotted point for m=2^22 does not correspond to the claimed threshold. Moreover, m=2^20 and m=2^22 use pretrained initialization and curriculum mixing from smaller moduli, while m=2^14–2^18 use direct training, so the fit conflates task difficulty with optimization budget and initialization. Please either report the actual first-crossing position under a consistent protocol, show that the conclusion is robust to these protocol differences, or explicitly reframe the claim as a property of the specific training configurations used.
  2. [§5, Figure 6, Abstract] The abstract's statement that curriculum learning is a 'critical necessity' for m>=2^20 is stronger than the evidence. Figure 6(a,b) shows that random initialization without curriculum does not converge within a fixed budget of 75k steps at m=2^20. This demonstrates a within-budget failure, not a necessity in general. To support 'requires', the authors should show that extended training still fails, or soften the claim to 'under our fixed compute budget, curriculum learning was necessary to reach strong performance.' As written, this is a load-bearing overstatement because it is presented as a main finding.
  3. [§4.2, §C.1, Figure 14] The scaling-law exponent is threshold-dependent and lacks uncertainty quantification. Using a fixed 90% threshold yields beta≈0.5, but changing to an additive epsilon-over-random threshold gives beta in [0.4,0.5], and a multiplicative gamma-over-random threshold gives beta in [0.33,0.34] (Section C.1). With a single run per modulus and no error bars, the abstract's 'grows as sqrt(m)' is not a robust law. Please report seeds or confidence intervals, and if the exponent varies with the accuracy threshold, qualify the claim accordingly.
minor comments (4)
  1. [Abstract] The phrase 'the number of in-context sequence elements required for near-perfect prediction grows as sqrt(m)' would be more accurate if qualified as 'in the configurations we tested, the context lengths used were approximately 0.5*sqrt(m)'.
  2. [Figure 3 caption] The caption states that accuracy improves 'once the context length reaches exactly 0.5*sqrt(m)'. The word 'exactly' is not supported by the data, since no accuracy at that exact position is reported for m=2^22, and for m=2^20 the reported evaluation is at position 640, not 512.
  3. [§5.2] The detail that only the overlapping portion of the embedding matrix is transferred (with new tokens randomly initialized) is important for interpreting the scaling results. Consider making it explicit in the main-text discussion around Figure 6, not only in the appendix.
  4. [General] The paper does not state whether results are single-seed or averaged. Since the scaling law is a central quantitative claim, a sentence describing the number of runs and seed handling would help.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical measurement and comparison study, not a derivation that reduces to its inputs.

full rationale

This paper does not derive predictions from assumptions whose definitions already contain the target result. The headline claims—in-context prediction of PCG variants, the sqrt(m) scaling of required context, the necessity of curriculum training at large moduli, and rotational clustering in embeddings—are supported by direct measurements with specified training/evaluation protocols. The scaling law is a fitted description of observed accuracy-versus-position curves, not a parameter fitted to the target quantity and then reported as a prediction; the same data are used to fit the exponent, which is standard empirical curve fitting rather than circularity. The curriculum conclusion is an empirical comparison under a fixed 75k/100k-step budget, and the embedding analysis is a post-hoc PCA description. Citations to Tao et al. (2025) and Bouillaguet et al. (2020) serve as comparative baselines, and the paper's PCG-specific findings do not depend on those citations for their validity. Concerns raised about the largest-modulus point (m=2^22 reported at position 1279 rather than 0.5*sqrt(m)=1024) and about different training protocols across moduli are robustness/correctness concerns about an extrapolated empirical law, not evidence of a circular derivation. No self-definitional step, fitted-parameter-renamed-as-prediction step, or load-bearing self-citation chain appears in the manuscript.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims rest on empirical measurements; the only free parameters are the fitted scaling-law constants and the chosen accuracy threshold. No new physical or mathematical entities are introduced.

free parameters (4)
  • scaling_law_prefactor = ~0.5 (claimed 1/2 sqrt(m))
    Empirical constant fit to context length at 90% accuracy threshold in Fig. 3.
  • scaling_law_exponent = 0.5 for 90% threshold; 0.33-0.5 depending on threshold
    Fitted exponent beta from Fig. 14; varies with accuracy criterion, so quoted sqrt(m) is threshold-dependent.
  • accuracy_threshold = 90%
    Choice of threshold defines the context length used for the scaling law; different thresholds yield different exponents.
  • curriculum_mixing_ratio = alpha=0.01 initial, exponential decay
    Hyperparameter chosen for curriculum experiments; not derived from first principles.
axioms (4)
  • standard math Hull-Dobell theorem conditions guarantee full period for LCG with modulus m.
    Used to select valid multipliers and increments for all generators (Appendix A.1).
  • standard math The k-th lowest bit of a power-of-two LCG cycles with period 2^k.
    Used to motivate PCG permutations and to explain stepwise accuracy improvements (Section 2.1, A.2).
  • domain assumption Transformers trained with cross-entropy can learn a deterministic recurrence from finite in-context examples.
    Basis of the entire empirical study; not proven but supported by experiments.
  • standard math PCA of the embedding matrix reveals interpretable structure.
    Used for the interpretability analysis; assumes Euclidean projections capture meaningful grouping.

pith-pipeline@v1.3.0-alltime-deepseek · 18074 in / 10052 out tokens · 85514 ms · 2026-08-04T07:03:18.415490+00:00 · methodology

0 comments
read the original abstract

We study the ability of Transformer models to learn sequences generated by Permuted Congruential Generators (PCGs), a widely used family of pseudo-random number generators (PRNGs). PCGs introduce substantial additional difficulty over linear congruential generators (LCGs) by applying a series of bit-wise shifts, XORs, rotations and truncations to the hidden state. We show that Transformers can nevertheless successfully perform in-context prediction on unseen sequences from diverse PCG variants, in tasks that are beyond published classical attacks. In our experiments we scale moduli up to $2^{22}$ using up to $50$ million model parameters and datasets with up to $5$ billion tokens. Surprisingly, we find even when the output is truncated to a single bit, it can be reliably predicted by the model. When multiple distinct PRNGs are presented together during training, the model can jointly learn them, identifying structures from different permutations. We demonstrate a scaling law with modulus $m$: the number of in-context sequence elements required for near-perfect prediction grows as $\sqrt{m}$. For larger moduli, optimization enters extended stagnation phases; in our experiments, learning moduli $m \geq 2^{20}$ requires incorporating training data from smaller moduli, demonstrating a critical necessity for curriculum learning. Finally, we analyze embedding layers and uncover a novel clustering phenomenon: the top principal components spontaneously group the integer inputs into bitwise rotationally-invariant clusters, revealing how representations can transfer from smaller to larger moduli.

Figures

Figures reproduced from arXiv: 2510.26792 by Maissam Barkeshli, Tao Tao.

Figure 1
Figure 1. Figure 1: Depiction of PCG protocols at m = 216 with 8-bit output. Left: XSLRR-16/8. (a) State si . The top 3 bits are control bits. (b) si is right-shifted by 8 bits. (c) The shifted state is XORed with si . (d) The lower 8 bits are retained and rotated right by the value of the control bits to produce the output. Middle: XSHRR-16/8. (e) State si , with the top 3 bits as control; the lowest few bits are unused. (f)… view at source ↗
Figure 2
Figure 2. Figure 2: (a) Test accuracy at the 512th token during training on combined datasets of diverse PRNG variants. (b) Accuracy during training on XSLRR-16/8 dataset. “512th” refers to the model’s prediction accuracy at the 512-th token. “Avg” denotes accuracy averaged across all token positions. (c) Final test accuracy by position index for combined training. (d) Final test accuracy when trained separately on each gener… view at source ↗
Figure 3
Figure 3. Figure 3: Left: Prediction accuracy at the 64th, 128th, and 256th sequence positions as a function of bits kept (k) in truncated LCGs with m = 216. Accuracy improves with larger k and longer context, remaining far above the random baseline 1/2 k even under severe truncation. Middle: For XSLRR, accuracy improves stepwise as more context is observed, with reliable predictions emerging once the context length reaches e… view at source ↗
Figure 4
Figure 4. Figure 4: Scaling studies of dataset size and model capacity. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Effect of mixing smaller-modulus data on training stability and final accuracy. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Impact of pretrained initialization and curriculum training on scaling to larger moduli. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: PCA of token–embedding matrix for XSLRR-16/8 (left) and cluster summary (right). To [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cosine similarity of representations across different PRNG variants for a 4-layer Trans [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Test loss curves for each generator type when the model is trained on the combined dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Curriculum training with random initialization from m=216 to m=218. (a) Training and test loss over steps. (b) Evolution of mixing ratio α. (c) Last-token accuracy over time for both moduli. (d) Learning rate schedule during training [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Curriculum training with pre-trained initialization from m=218 to m=220. (a) Training and test loss over steps. (b) Evolution of mixing ratio α. (c) Last-token accuracy over time for both moduli. (d) Learning rate schedule during training. Truncated LCG ( [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Curriculum training with random initialization from m=218 to m=220 . (a) Training and test loss over steps. (b) Evolution of mixing ratio α. (c) Accuracy at the final prediction position. (d) Learning rate schedule [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Curriculum training with pre-trained initialization from m=220 to m=222. (a) Training and test loss over steps. (b) Evolution of mixing ratio α. (c) Last-token accuracy over time for both moduli. (d) Learning rate schedule during training. 0.1, the model achieved a test accuracy on m=222 of 0.9257 at position 1279 [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Scaling of required context length with m under alternative accuracy thresholds. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: FLOPs scaling with modulus. Inference compute per sequence for models trained on PCGs with moduli m = 214–2 22 (context lengths required reach 90% accuracy listed in legend), compared to the brute-force baseline (dashed line). Inference cost grows far more slowly than the brute-force bound, showing the compute efficiency of Transformer-based prediction relative to direct state-space search [PITH_FULL_IMA… view at source ↗
Figure 16
Figure 16. Figure 16: Training (left) and test (right) loss as a function of [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Training (left) and test (right) loss for XSLRR-18/9 with [PITH_FULL_IMAGE:figures/full_fig_p018_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: 1-layer, 8-head Transformer trained on XSLRR 14/7. [PITH_FULL_IMAGE:figures/full_fig_p018_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Effect of model depth/width and size on test accuracy. Left and Middle: Token accuracy heatmaps at the 64th and 512th positions. The y-axis (Depth d) indicates the number of Transformer layers nlayer, and the x-axis shows model size in terms of attention heads and embedding dimension (nheads, dmodel). Right: Test loss versus model size (number of parameters) (a) Test loss v.s Training steps (b) Test accur… view at source ↗
Figure 20
Figure 20. Figure 20: Learning curves and final test performance of a 2-layer model trained on the combined [PITH_FULL_IMAGE:figures/full_fig_p019_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Learning curves and final test performance of a 3-layer model trained on the combined [PITH_FULL_IMAGE:figures/full_fig_p020_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Comparison of curriculum schedules. Top row: probability [PITH_FULL_IMAGE:figures/full_fig_p020_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: PCA of token embeddings at the start (purple) and after fine-tuning (yellow) for a model [PITH_FULL_IMAGE:figures/full_fig_p021_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: PCA analysis of token embeddings. The same rotation-invariant grouping structure [PITH_FULL_IMAGE:figures/full_fig_p022_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: When trained on combined datasets, the model develops a more general grouping of to [PITH_FULL_IMAGE:figures/full_fig_p022_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Token-distance distribution of the top-8 attended keys at each Transformer layer for a [PITH_FULL_IMAGE:figures/full_fig_p023_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Single-head ablation. Accuracy drop at the 512th token when zeroing the V-slice of each attention head. Darker colors indicate a larger decrease in accuracy relative to the baseline. The last two or three layers exhibit stronger head specialization, with some heads (e.g., Head 6) affecting truncated LCG but not XSLRR, and others (e.g., Heads 3 and 5) affecting XSLRR but not truncated LCG. H.3 HEAD SPECIAL… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Pseudorandom Streams within Diffusion Models Act as Learnable Inputs That Affect Generation Quality

    cs.LG 2026-08 conditional novelty 6.0

    A diffusion model's training loss and output quality depend measurably on which pseudorandom orbit supplies its randomness, even after marginal-statistics control.

Reference graph

Works this paper leans on

29 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    Curriculum learning

    Yoshua Bengio, J\' e r\^ o me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pp.\ 41–48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi:10.1145/1553374.1553380. URL https://doi.org/10.1145/1553374.1553380

    arXiv 2009

  2. [2]

    Practical seed-recovery for the pcg pseudo-random number generator

    Charles Bouillaguet, Florette Martinez, and Julia Sauvage. Practical seed-recovery for the pcg pseudo-random number generator. IACR Transactions on Symmetric Cryptology, 2020 0 (3): 0 175–196, Sep. 2020. doi:10.13154/tosc.v2020.i3.175-196. URL https://tosc.iacr.org/index.php/ToSC/article/view/8700

    work page doi:10.13154/tosc.v2020.i3.175-196 2020

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    Pith/arXiv arXiv 2020

  4. [4]

    Emergent properties with repeated examples, 2024

    François Charton and Julia Kempe. Emergent properties with repeated examples, 2024. URL https://arxiv.org/abs/2410.07041

    Pith/arXiv arXiv 2024

  5. [5]

    To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets, 2024

    Darshil Doshi, Aritra Das, Tianyu He, and Andrey Gromov. To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets, 2024. URL https://arxiv.org/abs/2310.13061

    Pith/arXiv arXiv 2024

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929

    Pith/arXiv arXiv 2021

  7. [7]

    What can transformers learn in-context? a case study of simple function classes, 2023

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes, 2023. URL https://arxiv.org/abs/2208.01066

    Pith/arXiv arXiv 2023

  8. [8]

    Grokking modular arithmetic, 2023

    Andrey Gromov. Grokking modular arithmetic, 2023. URL https://arxiv.org/abs/2301.02679

    Pith/arXiv arXiv 2023

  9. [9]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    Pith/arXiv arXiv 2024

  10. [10]

    Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks

    Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 13244--13273. Curran Associates, Inc.,...

    2024

  11. [11]

    T. E. Hull and A. R. Dobell. Random number generators. SIAM Review, 4 0 (3): 0 230--254, 1962. doi:10.1137/1004061. URL https://doi.org/10.1137/1004061

    work page doi:10.1137/1004061 1962

  12. [12]

    Donald E. Knuth. The art of computer programming, volume 2 (3rd ed.): seminumerical algorithms. Addison-Wesley Longman Publishing Co., Inc., USA, 1997. ISBN 0201896842

    1997

  13. [13]

    Testu01: A c library for empirical testing of random number generators

    Pierre L'Ecuyer and Richard Simard. Testu01: A c library for empirical testing of random number generators. ACM Trans. Math. Softw., 33 0 (4), August 2007. ISSN 0098-3500. doi:10.1145/1268776.1268777. URL https://doi.org/10.1145/1268776.1268777

    arXiv 2007

  14. [14]

    Michaud, Max Tegmark, and Mike Williams

    Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning, 2022. URL https://arxiv.org/abs/2205.10343

    Pith/arXiv arXiv 2022

  15. [15]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  16. [16]

    Progress measures for grokking via mechanistic interpretability, 2023

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023. URL https://arxiv.org/abs/2301.05217

    Pith/arXiv arXiv 2023

  17. [17]

    In-context learning and induction heads, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, a...

    Pith/arXiv arXiv 2022

  18. [18]

    Melissa E. O'Neill. Pcg: A family of simple fast space-efficient statistically good algorithms for random number generation. Technical Report HMC-CS-2014-0905, Harvey Mudd College, Claremont, CA, September 2014

    2014

  19. [19]

    Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022. URL https://arxiv.org/abs/2201.02177

    Pith/arXiv arXiv 2022

  20. [20]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf. Accessed: 2024-11-15

    2019

  21. [21]

    Cryptography and machine learning

    Ronald L Rivest. Cryptography and machine learning. In International Conference on the Theory and Application of Cryptology, pp.\ 427--439. Springer, 1991

    1991

  22. [22]

    Making hard problems easier with custom data distributions and loss regularization: A case study in modular arithmetic

    Eshika Saxena, Alberto Alfarano, Fran c ois Charton, Zeyuan Allen-Zhu, Emily Wenger, and Kristin Lauter. Making hard problems easier with custom data distributions and loss regularization: A case study in modular arithmetic. arXiv preprint arXiv:2410.03569, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    Pith/arXiv arXiv 2023

  24. [24]

    (how) can transformers predict pseudo-random numbers?, 2025

    Tao Tao, Darshil Doshi, Dayal Singh Kalra, Tianyu He, and Maissam Barkeshli. (how) can transformers predict pseudo-random numbers?, 2025. URL https://arxiv.org/abs/2502.10390

    Pith/arXiv arXiv 2025

  25. [25]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  26. [26]

    Salsa: Attacking lattice cryptography with transformers

    Emily Wenger, Mingjie Chen, Francois Charton, and Kristin E Lauter. Salsa: Attacking lattice cryptography with transformers. Advances in Neural Information Processing Systems, 35: 0 34981--34994, 2022

    2022

  27. [27]

    When do curricula work?, 2021

    Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work?, 2021. URL https://arxiv.org/abs/2012.03107

    Pith/arXiv arXiv 2021

  28. [28]

    The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023. URL https://arxiv.org/abs/2306.17844

    Pith/arXiv arXiv 2023

  29. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...