{"id":"b5a433b0-f5c7-42d8-85ee-f793ad8f9ff5","arxiv_id":"2501.04286","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A custom convergence measure applied to a 96k-parameter decoder-only transformer shows learning-rate boundaries between convergent and divergent training are fractal-like across zoom levels.","lead":"This paper trains a small decoder-only transformer across a grid of attention and feed-forward learning rates and reports that the boundary between stable and unstable training is fractal-like, repeating as you zoom in. It matters because it suggests transformer hyperparameter sensitivity is more complex than a simple threshold, which complicates grid search and hyperparameter optimization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fractal claim may be an artifact of the hand-tuned convergence thresholds; the paper never checks robustness of the converged/diverged partition under plausible variations of the criterion in Section 4.","rationale":"The reader's weakest_assumption points at threshold sensitivity of the convergence measure, and I independently reached the same conclusion: the entire fractal analysis flows through a binary converged/diverged partition that is defined by three arbitrary thresholds. The paper gives no evidence that the reported box-counting dimensions or the near-identical histograms survive plausible changes to those thresholds. The Appendix's Sierpinski and Mandelbrot checks are genuine independent support for the box-counting and edge-detection code, for which the paper deserves credit, but they do not validate the mapping from loss curves to the binary image. I also considered whether the strongest concern might instead be that only one initialization, one dataset, one model size, and one optimizer are used, which limits generalizability but does not undermine the internal claim; or that the box-counting dimensions lack error bars and null comparisons, which is a real reporting weakness but secondary to the threshold issue because even perfect error bars would not settle whether the boundary itself is an artifact. The proposed concrete test directly settles the load-bearing concern: perturb the thresholds and re-measure dimensions and map texture. Because the concern is addressable and the conclusion is conditional, the verdict stays CONDITIONAL rather than moving to REJECT; if the robustness test fails, the claim would be unverified rather than disproven, since a different reasonable criterion might still yield fractal structure.","tokens_in":7368,"tokens_out":1719,"duration_ms":15284,"concrete_test":"Recompute the full-results pipeline at Granularity 10^-5 (or one 10^-8 zoom) under a systematically varied convergence criterion: independently perturb each Section 4 threshold (mean-loss cutoff 0.4 -> 0.3/0.5, required drop 0.1 -> 0.05/0.2, variance 0.01 -> 0.005/0.02), and also test a criterion based on final-loss only and on loss-slope. If the box-counting dimension of the resulting boundary varies by more than roughly ±0.05 across these perturbations, or if the texture of the maps changes qualitatively, then the fractal conclusion is an artifact of threshold choice and the paper should be CONDITIONAL on reporting robustness. If the dimension and visual self-similarity are stable across all perturbations, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the converged/diverged boundary in (eta_att, eta_fc) space is fractal-like, with self-similar statistics and box-counting dimensions near 1.98. Every map, histogram, and dimension estimate depends on the binary partition induced by the Section 4 convergence rule: recent mean loss < 0.4, recent mean loss at least 0.1 below early mean loss, and recent loss variance < 0.01. These thresholds are asserted as 'determined based on the data and generation quality' with no sensitivity analysis. A small change in one threshold can add or remove thin tendrils and speckles from the binary map, which directly changes the box-counting dimension and the apparent self-similarity. The issue is not merely cosmetic: the reported dimensions (1.98, 1.97, 1.96, 1.81, 1.58...) span a wide range across boundary regions, yet no null model or error bar shows that these values are distinguishable from the dimension of a smooth curve with threshold noise. Without a robustness check, the strongest claim—statistically consistent repeating patterns at multiple scales—is not established. The independent-support elements in the paper are real but insufficient: the Appendix verifies box-counting on Sierpinski (1.585) and edge detection on Mandelbrot, but that only validates the image-processing pipeline, not the mapping from training runs to the binary image that feeds that pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains a 95,973-parameter decoder-only transformer on a character-level Shakespeare dataset and sweeps the learning rates of the attention layers and of the feedforward/remaining layers. A run is classified as converged using a threshold-based rule on recent loss, loss improvement, and loss variance, and the resulting binary map of converged versus diverged hyperparameters is examined at multiple resolutions. The authors report box-counting dimensions between roughly 1.54 and 1.98 for the boundaries, present convergence-measure heatmaps and histograms at several granularities, and conclude that the trainability frontier is fractal-like with self-similar statistical structure. An appendix verifies the box-counting and edge-detection code on Sierpinski and Mandelbrot images.","tokens_in":7713,"tokens_out":4620,"duration_ms":40982,"significance":"If the central claim holds, the paper extends Sohl-Dickstein's fractal-boundary observation from very small networks to a medium-sized decoder-only transformer language model, which would be a useful step toward understanding hyperparameter sensitivity in modern architectures. The manuscript has real strengths: the architecture and data are described concretely, the code is publicly available, the convergence measure is explicitly defined and calibrated with qualitative generation samples, and the image-processing pipeline is validated on known fractals. These positive elements are, however, not yet sufficient to support the quantitative fractal claims, because the binary partition on which all dimension estimates rest is produced by hand-tuned thresholds with no sensitivity analysis, and the reported dimension values are not compared with any null model or accompanied by fitting diagnostics.","major_comments":[{"comment":"The binary converged/diverged partition that feeds every box-counting estimate is determined by three hand-set thresholds (recent mean loss below 0.4, recent mean loss at least 0.1 below early mean loss, and recent loss variance below 0.01), which the text describes only as 'determined based on the data and generation quality.' This is a load-bearing choice: small changes to any threshold can add or remove tendrils and isolated speckles in the binary maps, and the reported box-counting dimensions would change accordingly. Please provide a sensitivity analysis over plausible threshold values and show that the non-integer dimension estimates and the qualitative conclusion of self-similarity are robust; alternatively, report the range of dimension values obtained across the threshold variations.","section":"Section 4"},{"comment":"The eight box-counting dimensions reported (1.9783, 1.9772, 1.9715, 1.9649, 1.8218, 1.8118, 1.5810, 1.5413) are presented without per-image error bars, without diagnostics for the linear fit in the log-log box-counting plot, and without any null baseline. Since a random binary-speckle map has dimension close to 2 and the main-region values cluster near 1.96–1.98, the current numbers do not by themselves distinguish fractal boundaries from threshold noise or near-space-filling random patterns. Please add scaling-range fits, confidence intervals, and a comparison to null models such as randomly permuted or spatially autocorrelated binary maps matched for converged fraction and edge density.","section":"Section 5, Figures 5–9, 14–15, 18–19"},{"comment":"The claim that the convergence-measure histograms are 'nearly identical' or 'statistically consistent' across scales is made by visual inspection only. No quantitative histogram distance such as Jensen–Shannon divergence or a Kolmogorov–Smirnov statistic is reported, and the large spread of box-counting dimensions across boundary regions undercuts the impression of scale-invariant statistics. Please report a formal comparison of the histograms across resolutions and address the variability of the dimension estimates across the different sampled regions.","section":"Figures 10–13 and 16–21"},{"comment":"The box-counting and Sobel tests on the Sierpinski triangle and Mandelbrot set verify the image-processing pipeline, but they do not validate the mapping from training runs to the binary image that is fed into that pipeline. The fractal claim depends as much on the threshold-based classification of Section 4 as on the box-counting implementation, so the Appendix cannot by itself establish that the measured dimensions reflect genuine structure in the training dynamics. Please separate the two validation steps and, in particular, test the sensitivity of the full pipeline end-to-end from loss curves to binary edge map to dimension estimate.","section":"Appendix, Figures 22–25"}],"minor_comments":[{"comment":"There are several typos and inconsistent notations, including 'porpuses' in the Appendix, 'hyperprameter' in Section 5, and the use of both η and μ for learning rates; please copyedit and unify the notation.","section":"Throughout"},{"comment":"The definition introduces M as 'Max' but the formula uses M in a way that is not fully specified; please define exactly how M is computed for each training run and state its role in the normalization.","section":"Section 4"},{"comment":"Figure 4 is described as a binary heatmap with converged regions in black, while Figure 6 shows a colored convergence measure; please clarify the relationship between the two visualizations and the color scale used for the intermediate values.","section":"Figures 4–6"},{"comment":"Please report the number of training runs per grid, the total compute budget, and the initialization and dataset details needed to reproduce the maps; the public code repository is a helpful start, but the paper should state the run count and random-seed policy.","section":"Section 5 and code repository"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a potentially interesting empirical observation, but the quantitative support is currently incomplete. The robustness analysis described in the major comments is essential; if the authors can supply it and the results hold, the paper could be suitable for publication in a machine-learning venue. I would not reject on the basis of disagreement with the fractal interpretation, but I would not yet accept the quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll give you the short version. The paper is a direct extension of Sohl-Dickstein's fractal trainability result to a decoder-only transformer, with separate learning rates for attention and FFN layers, and it ships code. That's a reasonable thing to check, and the author does some things right: the convergence measure is explicitly defined, the box-counting pipeline is validated on a Sierpinski triangle and edge detection on the Mandelbrot set, and the text is upfront about compute constraints. The generated-text samples that calibrate the measure are a nice touch.\n\nThe load-bearing weak spot is the binary converged/diverged map. Section 4 defines convergence with three thresholds (recent loss < 0.4, recent loss at least 0.1 below early loss, variance < 0.01) and says they were determined from the data. No sensitivity analysis is shown. Since every heatmap, histogram, and box-count dimension comes from that binary partition, the whole fractal claim rests on those three numbers. A small threshold shift can add or remove thin tendrils and speckles, which directly changes the dimension. The main maps give dimensions around 1.96-1.98, which is also what you'd get from a noisy binary image with a smooth underlying boundary. Without a null baseline (e.g., a smooth separatrix plus threshold noise, or a random binary map) and without error bars or scaling ranges on the box-count fits, the reported dimensions don't yet discriminate fractal structure from threshold artifacts.\n\nThe multi-scale histograms are presented as evidence of statistical self-similarity, but histograms of a continuous convergence measure can look similar across resolutions even if the spatial structure is not self-similar. That evidence is weaker than it appears. The lower dimensions in the additional regions (1.58, 1.54, 1.81, 1.82) show a wide spread that the paper does not explain, which further muddies the story.\n\nNone of this makes the paper worthless. The finding is plausible, and the right experiments are obvious: vary the thresholds, show that the boundaries and dimensions are stable, and compare against null models. If the fractal structure survives, it's a solid empirical contribution; if not, it's a useful cautionary tale about thresholding artifacts. I'd send it to review with major revisions requested. I wouldn't cite it in its current form.","headline":"Plausible but not yet established: the fractal boundary claim rests on hand-tuned thresholds and unchecked box-counting dimensions.","tokens_in":8158,"tokens_out":4124,"would_cite":false,"duration_ms":40316,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the boundary between converging and diverging training of a decoder-only transformer, plotted over learning rates, is fractal-like rather than a smooth threshold: the same intricate textures appear from 10^-5 down to…","keywords":["fractal trainability","decoder-only transformer","learning-rate landscape","Adam optimizer","box-counting dimension","convergence measure","chaotic training dynamics","hyperparameter sensitivity"],"falsifier":"Recompute the learning-rate maps with convergence thresholds varied (for example cutoff 0.3 or 0.5, variance threshold 0.001 or 0.1) and re-estimate box-counting dimensions at the reported scales; if the non-integer dimensions or the scale-invariant histograms vanish or change discontinuously, the fractal claim collapses. A complementary check is to magnify the boundary beyond $10^{-11}$ spacing and see whether the self-similar texture persists or resolves into smooth regions.","tokens_in":7169,"feed_emoji":"🌀","tokens_out":7318,"duration_ms":62210,"temperature":0.7,"pith_summary":"This paper asks whether the dividing line between learning-rate settings that train a decoder-only transformer successfully and settings that make it diverge is a smooth curve or something far more irregular. It trains a transformer with 95,973 trainable parameters on next-character prediction, scans learning rates for attention layers and for fully connected layers on increasingly fine grids, and labels each run with a normalized convergence measure. The reported result is that the trainability frontier is not a simple threshold: identical-looking textures and near-identical histograms of the convergence measure appear at scales down to $10^{-11}$, and the boundary's box-counting dimension stays non-integer, around 1.5 to 1.98. A sympathetic reader would take this as evidence that in transformer training, hyperparameter space is full of fine-grained chaotic structure rather than one stable region with a clean edge.","feed_headline":"Transformer training boundary is fractal, not smooth","feed_subtitle":"At learning-rate scales down to 1e-11, converged and divergent runs repeat the same chaotic patterns.","key_machinery":"The central object is the convergence measure $\\mu$, a scalar in $[-1,1]$ built from a training run's loss curve. A run is classified as converged when the mean of the last 5% of losses is below 0.4, at least 0.1 below the mean of the first 5%, and has variance below 0.01; $\\mu$ is then defined relative to reference curves, with 1 for the most convergent shape and -1 for the most divergent. This measure turns the two learning rates into a color map, and the fractal analysis operates on the binarized converged-versus-diverged image: Sobel edge detection finds the boundary and box counting estimates its dimension.","core_discovery":"The paper claims that in a decoder-only transformer trained with Adam for character-level language modeling, the boundary between convergent and divergent training is self-similar and statistically reproducible across scales. Using a convergence measure normalized to lie between -1 and 1, the author renders two-dimensional maps of attention-layer learning rate versus fully-connected-layer learning rate, then zooms into boundary regions at granularities from $10^{-5}$ down to $10^{-11}$. At each finer scale the edge between blue converged zones and red diverged zones shows the same texture, and estimates from box counting give non-integer fractal dimensions, for example 1.9772 at $10^{-5}$, 1.9715 at $10^{-8}$, 1.9649 at $10^{-10}$, 1.9783 at $10^{-11}$, and lower values around 1.5 to 1.8 in other boundary regions.","pith_inferences":["If the frontier is truly fractal, coarse grid searches systematically misclassify points near it; mapping trainability reliably would require multiscale or adaptive sampling rather than a uniform grid.","A testable extension is to apply the same box-counting analysis to slices of other hyperparameter planes (batch size, momentum, weight decay, initialization seed) to see whether the self-similar boundary is a general property of the optimizer, not just of learning rates.","Because the convergence criterion uses hand-set thresholds (0.4, 0.1, 0.01), varying those thresholds and re-measuring dimensions is the direct way to separate an intrinsic fractal property from an artifact of the converged-versus-diverged split.","A near-2D box-counting dimension at $10^{-11}$ spacing suggests the 'converged' set may be sponge-like at every scale; if confirmed, the very idea of a single optimal learning rate becomes ill-defined for transformer training."],"forward_implications":["Learning-rate selection for transformer layers cannot rely on a smooth 'safe zone'; near the boundary, tiny changes in either learning rate can flip a run from convergence to divergence.","The fractal character found in very small networks persists at medium transformer scale, suggesting it may be an intrinsic feature of iterative training dynamics rather than a quirk of tiny models.","The chosen convergence thresholds shape the boundary that gets measured, so different criteria may move or reshape the frontier while still leaving an irregular, high-dimensional edge.","Box-counting dimensions near 1.98 imply that at the finest resolved scale, converged and diverged regions alternate almost everywhere along the border, so the set of usable learning rates is riddled with holes."],"supporting_citations":[{"why":"Prior demonstration that the hyperparameter boundary in very small neural networks is fractal; this paper extends that result to decoder-only transformers.","marker":"[1]"},{"why":"Defines the Adam update rule, the iterative dynamics whose convergence and divergence are being mapped.","marker":"[6]"},{"why":"Introduces the transformer architecture, whose decoder-only variant is the model under study.","marker":"[8]"},{"why":"Supplies the public-domain Shakespeare character dataset used for all training runs in the landscape.","marker":"[15]"},{"why":"Provides the fractal-geometry definitions and the box-counting dimension method used to quantify the boundary.","marker":"[3]"},{"why":"The prior implementation and convergence-measure code that this paper's criterion modifies and replaces.","marker":"[7]"}],"fun_headline_variants":["Transformer training boundary is fractal, not smooth","Fractal chaos marks transformer convergence edge","Self-similar frontier: transformer trainability is fractal","Zoom to 1e-11: transformer training edge repeats itself","Fractal geometry governs transformer convergence boundary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the specific convergence thresholds (recent loss below 0.4, at least 0.1 below early loss, variance below 0.01); if these thresholds are arbitrary or data-specific, the binary map and the measured fractal dimensions could be an artifact of how runs are split into converged and diverged rather than a property of the training dynamics.","fun_headline_variants_meta":{"raw":{"variants":["Transformer training boundary is fractal, not smooth","Fractal chaos marks transformer convergence edge","Self-similar frontier: transformer trainability is fractal","Zoom to 1e-11: transformer training edge repeats itself","Fractal geometry governs transformer convergence boundary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1591,"prompt_tokens":881,"completion_tokens":710,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":637}},"tokens_in":497,"tokens_out":710,"duration_ms":6570,"temperature":1.0,"reasoning_tokens":637,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:36:19.225720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the learning-rate maps with convergence thresholds varied (for example cutoff 0.3 or 0.5, variance threshold 0.001 or 0.1) and re-estimate box-counting dimensions at the reported scales; if the non-integer dimensions or the scale-invariant histograms vanish or change discontinuously, the fractal claim collapses. A complementary check is to magnify the boundary beyond $10^{-11}$ spacing and see whether the self-similar texture persists or resolves into smooth regions.","supporting_citations":[{"cited_title":"Gomez, Lukasz Kaiser, and Illia Polosukhin","cited_arxiv_id":null,"evidence_quote":"Introduces the transformer architecture, whose decoder-only variant is the model under study."},{"cited_title":"The Complete Works of William Shakespeare","cited_arxiv_id":null,"evidence_quote":"Supplies the public-domain Shakespeare character dataset used for all training runs in the landscape."},{"cited_title":"Fractal geometry: mathematical foundations and applications","cited_arxiv_id":null,"evidence_quote":"Provides the fractal-geometry definitions and the box-counting dimension method used to quantify the boundary."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior implementation and convergence-measure code that this paper's criterion modifies and replaces."}],"review_version":1}