Pith. sign in

REVIEW 4 major objections 4 minor 15 references

Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the boundary between converging and diverging training of a decoder-only transformer, plotted over learning rates, is fractal-like rather than a smooth threshold: the same intricate textures appear from 10^-5 down to…

desk verdict Plausible but not yet established: the fractal boundary claim rests on hand-tuned thresholds and unchecked box-counting dimensions. read the letter →

arxiv 2501.04286 v2 pith:U4U2DPVS submitted 2025-01-08 cs.LG cs.AI

Bahman Torkamandi This is my paper · ORCID
classification cs.LGcs.AI
keywords fractaltrainabilitydecoder-onlytransformerlearning-ratelandscapeAdamoptimizerbox-countingdimensionconvergencemeasurechaotictrainingdynamicshyperparametersensitivity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the dividing line between learning-rate settings that train a decoder-only transformer successfully and settings that make it diverge is a smooth curve or something far more irregular. It trains a transformer with 95,973 trainable parameters on next-character prediction, scans learning rates for attention layers and for fully connected layers on increasingly fine grids, and labels each run with a normalized convergence measure. The reported result is that the trainability frontier is not a simple threshold: identical-looking textures and near-identical histograms of the convergence measure appear at scales down to $10^{-11}$, and the boundary's box-counting dimension stays non-integer, around 1.5 to 1.98. A sympathetic reader would take this as evidence that in transformer training, hyperparameter space is full of fine-grained chaotic structure rather than one stable region with a clean edge.

What carries the argument

The central object is the convergence measure $\mu$, a scalar in $[-1,1]$ built from a training run's loss curve. A run is classified as converged when the mean of the last 5% of losses is below 0.4, at least 0.1 below the mean of the first 5%, and has variance below 0.01; $\mu$ is then defined relative to reference curves, with 1 for the most convergent shape and -1 for the most divergent. This measure turns the two learning rates into a color map, and the fractal analysis operates on the binarized converged-versus-diverged image: Sobel edge detection finds the boundary and box counting estimates its dimension.

What would settle it

Recompute the learning-rate maps with convergence thresholds varied (for example cutoff 0.3 or 0.5, variance threshold 0.001 or 0.1) and re-estimate box-counting dimensions at the reported scales; if the non-integer dimensions or the scale-invariant histograms vanish or change discontinuously, the fractal claim collapses. A complementary check is to magnify the boundary beyond $10^{-11}$ spacing and see whether the self-similar texture persists or resolves into smooth regions.

Watch

Extended reading notes

Core claim

The paper claims that in a decoder-only transformer trained with Adam for character-level language modeling, the boundary between convergent and divergent training is self-similar and statistically reproducible across scales. Using a convergence measure normalized to lie between -1 and 1, the author renders two-dimensional maps of attention-layer learning rate versus fully-connected-layer learning rate, then zooms into boundary regions at granularities from $10^{-5}$ down to $10^{-11}$. At each finer scale the edge between blue converged zones and red diverged zones shows the same texture, and estimates from box counting give non-integer fractal dimensions, for example 1.9772 at $10^{-5}$, 1.9715 at $10^{-8}$, 1.9649 at $10^{-10}$, 1.9783 at $10^{-11}$, and lower values around 1.5 to 1.8 in other boundary regions.

Load-bearing premise

The central claim rests on the specific convergence thresholds (recent loss below 0.4, at least 0.1 below early loss, variance below 0.01); if these thresholds are arbitrary or data-specific, the binary map and the measured fractal dimensions could be an artifact of how runs are split into converged and diverged rather than a property of the training dynamics.

Editorial extensions

If this is right

  • Learning-rate selection for transformer layers cannot rely on a smooth 'safe zone'; near the boundary, tiny changes in either learning rate can flip a run from convergence to divergence.
  • The fractal character found in very small networks persists at medium transformer scale, suggesting it may be an intrinsic feature of iterative training dynamics rather than a quirk of tiny models.
  • The chosen convergence thresholds shape the boundary that gets measured, so different criteria may move or reshape the frontier while still leaving an irregular, high-dimensional edge.
  • Box-counting dimensions near 1.98 imply that at the finest resolved scale, converged and diverged regions alternate almost everywhere along the border, so the set of usable learning rates is riddled with holes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the frontier is truly fractal, coarse grid searches systematically misclassify points near it; mapping trainability reliably would require multiscale or adaptive sampling rather than a uniform grid.
  • A testable extension is to apply the same box-counting analysis to slices of other hyperparameter planes (batch size, momentum, weight decay, initialization seed) to see whether the self-similar boundary is a general property of the optimizer, not just of learning rates.
  • Because the convergence criterion uses hand-set thresholds (0.4, 0.1, 0.01), varying those thresholds and re-measuring dimensions is the direct way to separate an intrinsic fractal property from an artifact of the converged-versus-diverged split.
  • A near-2D box-counting dimension at $10^{-11}$ spacing suggests the 'converged' set may be sponge-like at every scale; if confirmed, the very idea of a single optimal learning rate becomes ill-defined for transformer training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper trains a 95,973-parameter decoder-only transformer on a character-level Shakespeare dataset and sweeps the learning rates of the attention layers and of the feedforward/remaining layers. A run is classified as converged using a threshold-based rule on recent loss, loss improvement, and loss variance, and the resulting binary map of converged versus diverged hyperparameters is examined at multiple resolutions. The authors report box-counting dimensions between roughly 1.54 and 1.98 for the boundaries, present convergence-measure heatmaps and histograms at several granularities, and conclude that the trainability frontier is fractal-like with self-similar statistical structure. An appendix verifies the box-counting and edge-detection code on Sierpinski and Mandelbrot images.

Significance. If the central claim holds, the paper extends Sohl-Dickstein's fractal-boundary observation from very small networks to a medium-sized decoder-only transformer language model, which would be a useful step toward understanding hyperparameter sensitivity in modern architectures. The manuscript has real strengths: the architecture and data are described concretely, the code is publicly available, the convergence measure is explicitly defined and calibrated with qualitative generation samples, and the image-processing pipeline is validated on known fractals. These positive elements are, however, not yet sufficient to support the quantitative fractal claims, because the binary partition on which all dimension estimates rest is produced by hand-tuned thresholds with no sensitivity analysis, and the reported dimension values are not compared with any null model or accompanied by fitting diagnostics.

major comments (4)
  1. [Section 4] The binary converged/diverged partition that feeds every box-counting estimate is determined by three hand-set thresholds (recent mean loss below 0.4, recent mean loss at least 0.1 below early mean loss, and recent loss variance below 0.01), which the text describes only as 'determined based on the data and generation quality.' This is a load-bearing choice: small changes to any threshold can add or remove tendrils and isolated speckles in the binary maps, and the reported box-counting dimensions would change accordingly. Please provide a sensitivity analysis over plausible threshold values and show that the non-integer dimension estimates and the qualitative conclusion of self-similarity are robust; alternatively, report the range of dimension values obtained across the threshold variations.
  2. [Section 5, Figures 5–9, 14–15, 18–19] The eight box-counting dimensions reported (1.9783, 1.9772, 1.9715, 1.9649, 1.8218, 1.8118, 1.5810, 1.5413) are presented without per-image error bars, without diagnostics for the linear fit in the log-log box-counting plot, and without any null baseline. Since a random binary-speckle map has dimension close to 2 and the main-region values cluster near 1.96–1.98, the current numbers do not by themselves distinguish fractal boundaries from threshold noise or near-space-filling random patterns. Please add scaling-range fits, confidence intervals, and a comparison to null models such as randomly permuted or spatially autocorrelated binary maps matched for converged fraction and edge density.
  3. [Figures 10–13 and 16–21] The claim that the convergence-measure histograms are 'nearly identical' or 'statistically consistent' across scales is made by visual inspection only. No quantitative histogram distance such as Jensen–Shannon divergence or a Kolmogorov–Smirnov statistic is reported, and the large spread of box-counting dimensions across boundary regions undercuts the impression of scale-invariant statistics. Please report a formal comparison of the histograms across resolutions and address the variability of the dimension estimates across the different sampled regions.
  4. [Appendix, Figures 22–25] The box-counting and Sobel tests on the Sierpinski triangle and Mandelbrot set verify the image-processing pipeline, but they do not validate the mapping from training runs to the binary image that is fed into that pipeline. The fractal claim depends as much on the threshold-based classification of Section 4 as on the box-counting implementation, so the Appendix cannot by itself establish that the measured dimensions reflect genuine structure in the training dynamics. Please separate the two validation steps and, in particular, test the sensitivity of the full pipeline end-to-end from loss curves to binary edge map to dimension estimate.
minor comments (4)
  1. [Throughout] There are several typos and inconsistent notations, including 'porpuses' in the Appendix, 'hyperprameter' in Section 5, and the use of both η and μ for learning rates; please copyedit and unify the notation.
  2. [Section 4] The definition introduces M as 'Max' but the formula uses M in a way that is not fully specified; please define exactly how M is computed for each training run and state its role in the normalization.
  3. [Figures 4–6] Figure 4 is described as a binary heatmap with converged regions in black, while Figure 6 shows a colored convergence measure; please clarify the relationship between the two visualizations and the color scale used for the intermediate values.
  4. [Section 5 and code repository] Please report the number of training runs per grid, the total compute budget, and the initialization and dataset details needed to reproduce the maps; the public code repository is a helpful start, but the paper should state the run count and random-seed policy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical measurement whose fractal claims are computed from, but not presupposed by, its convergence criterion.

full rationale

This paper is an empirical study rather than a derivation chain, and none of its central claims reduce by construction to its inputs. The load-bearing operational choice is the convergence criterion in Section 4: a run is converged when "the mean of recent losses is below 0.4 (determined based on the data and generation quality)," when "the mean of recent losses is at least 0.1 lower than the mean of early losses," and when "the variance of recent losses is below a threshold of 0.01." These thresholds do calibrate the binary converged/diverged map that feeds the boundary images, histograms, and box-counting estimates, but the fractal dimension and self-similarity statistics are not assumed by the criterion; they are measured outputs of the training runs. The paper does not fit a parameter to a subset of data and then relabel that fit as a prediction. The convergence measure is defined from the loss curves, and the claimed fractal structure is an observed property of the resulting two-dimensional map. The appendix validation on the Sierpinski triangle and Mandelbrot set checks the image-processing and box-counting pipeline independently of the transformer experiments. The reliance on Sohl-Dickstein [1] is external motivation, not a self-citation, and it is not used as a proof that forces the transformer result. Thus the central claim does not reduce to a self-citation chain or to the convergence thresholds by definition. Whether the hand-tuned thresholds make the fractal conclusion robust to perturbations is a legitimate correctness and sensitivity concern, but it is not circularity in the sense required here.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a custom convergence measure with hand-selected thresholds, a maximum-loss normalization that is not fully specified, and the assumption that box-counting on binarized finite-resolution images estimates the true fractal dimension. No new physical entities are introduced.

free parameters (4)
  • recent mean loss threshold = 0.4
    Used in convergence criterion and cutoff C; 'determined based on the data and generation quality' (Section 4).
  • early-to-recent loss improvement threshold = 0.1
    Second condition in convergence criterion (Section 4).
  • recent loss variance threshold = 0.01
    Third condition in convergence criterion (Section 4).
  • Max normalization M = not specified
    Bounds the divergence side of convergence measure mu; existence is assumed but its computation is not stated (Section 4).
assumptions (4)
  • ad hoc to paper The five hypothetical loss curves f1..f5 define degrees of convergence, and mu computed against the f3 baseline ranks runs monotonically with true trainability.
    Section 4 introduces f1..f5 and assigns mu=1 to f1, -1 to f2, and 0 to f3; no external validation beyond generated text samples.
  • domain assumption The boundary in a binarized finite-resolution learning-rate grid has a well-defined fractal dimension that box-counting on Sobel edges estimates.
    Appendix applies box-counting and Sobel to Sierpinski and Mandelbrot as sanity checks, but no scaling-range analysis or error bars are given for the transformer maps.
  • domain assumption Training dynamics are deterministic functions of the two learning rates given fixed initialization, data, and batch order, so the 2D grid represents the trainability landscape.
    Section 3 states fixed initialization and data; no stochasticity analysis is reported.
  • ad hoc to paper Histogram similarity across zoom levels is evidence of statistical self-similarity.
    Section 5 asserts histograms are nearly identical without a quantitative distance or overlap measure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models." pith.science (2026). https://pith.science/paper/U4U2DPVS

@misc{pith2026250104286,
  author       = {Pith},
  title        = {Pith review of: Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4U2DPVS}},
  note         = {Machine review of arXiv:2501.04286}
}
read the original abstract

In the realm of fractal geometry, intricate structures emerge from simple iterative processes that partition parameter spaces into regions of stability and instability. Likewise, training large language models involves iteratively applying update functions, such as Adam, where even slight hyperparameter adjustments can shift the training process from convergence to divergence. Recent evidence from miniature neural networks suggests that the boundary separating these outcomes displays fractal characteristics. Building on these insights, this study extends them to medium-sized, decoder-only transformer architectures by employing a more consistent convergence measure and examining the learning rate hyperparameter landscape for attention and fully connected layers. The results show that the trainability frontier is not a simple threshold; rather, it forms a self-similar yet seemingly random structure at multiple scales, with statistically consistent and repeating patterns. Within this landscape, a region of stable convergence is surrounded by a complex chaotic border, illustrating the sensitive nature of the underlying training dynamics.

Figures

Figures reproduced from arXiv: 2501.04286 by the authors.

Figure 1
Figure 1. Fractal-like Dust Clusters From Monte Carlo Random Placements [4]. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of The Language Model. The model is trained using Adam, fixed initialization and data, softmax cross-entropy loss, a context size of 64 tokens/characters and a batch size of 256. The key-query dimension in the attention layers is set to 64, aligning with the model’s embedding dimension. A non-trainable sinusoidal positional encoder is integrated to maintain the sequential order of input characters, faci… view at source ↗
Figure 3
Figure 3. Loss Functions Illustrating Various Convergence Behaviors. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (19 more)
Figure 5
Figure 5. Figure 5: Boundaries Between Convergence and Divergence [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 16
Figure 16. Figure 16: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 17
Figure 17. Figure 17: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p011_17.png]
Figure 18
Figure 18. Figure 18: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]
Figure 19
Figure 19. Figure 19: Convergence Measure Heatmap – Boundaries at Granularity [PITH_FULL_IMAGE:figures/full_fig_p012_19.png]
Figure 20
Figure 20. Figure 20: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p012_20.png]
Figure 21
Figure 21. Figure 21: Histogram of Convergence Measure, Granularity [PITH_FULL_IMAGE:figures/full_fig_p012_21.png]
Figure 22
Figure 22. Figure 22: Sierpinski Fractal with Theoretical Dimen [PITH_FULL_IMAGE:figures/full_fig_p015_22.png]
Figure 25
Figure 25. Figure 25: Edges Detected by Sobel [PITH_FULL_IMAGE:figures/full_fig_p015_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages

  1. [1]

    The boundary of neural network trainability is fractal

    Jascha Sohl-Dickstein. The boundary of neural network trainability is fractal. arXiv preprint arXiv:2402.06184, 2024

    arXiv 2024

  2. [2]

    Fractals everywhere

    Michael F Barnsley. Fractals everywhere. Academic press, 2014

    work page 2014

  3. [3]

    Fractal geometry: mathematical foundations and applications

    Kenneth Falconer. Fractal geometry: mathematical foundations and applications . John Wiley & Sons, 2013

    work page 2013

  4. [4]

    Brian H. Kaye. A Random Walk Through Fractal Dimensions. VCH Verlagsgesellschaft / VCH Publishers, 2nd edition, 1994

    work page 1994

  5. [5]

    Chaos and fractals: new frontiers of science, volume 106

    Heinz-Otto Peitgen, Hartmut Jürgens, Dietmar Saupe, and Mitchell J Feigenbaum. Chaos and fractals: new frontiers of science, volume 106. Springer, 2004

    work page 2004

  6. [6]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    arXiv 2014

  7. [7]

    Jascha Sohl-Dickstein. fractal. https://github.com/Sohl-Dickstein/fractal, 2024. Accessed: 2024-12- 02

    work page 2024

  8. [8]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 2017

    work page 2017

Show all 15 references
  1. [9]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by gen- erative pre-training. https://cdn.openai.com/research-covers/language-unsupervised/language_ understanding_paper.pdf, 2018. Accessed: 2024-04-27

    2018

  2. [10]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. https://cdn.openai.com/better-language-models/language_ models_are_unsupervised_multitask_learners.pdf, 2019. Accessed: 2024-04-27

    2019

  3. [11]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler,...

    2020

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2010 arXiv

  5. [13]

    Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition

    Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 5884–5888. IEEE, 2018

    2018

  6. [14]

    Reducing bert pre-training time from 3 days to 76 minutes

    Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing bert pre-training time from 3 days to 76 minutes. arXiv preprint arXiv:1904.00962, 12:2, 2019

    1904 arXiv

  7. [15]

    The Complete Works of William Shakespeare

    William Shakespeare. The Complete Works of William Shakespeare . Project Gutenberg, 1994. Accessed: 2024-12-02. 14 A PREPRINT - F EBRUARY 18, 2025 7 Appendix I use box-counting method to estimate fractal dimensions and Sobel operator to detect edges. For verification porpuses,...

    1994

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.