{"id":"76a07551-2ca4-4f45-8351-604203f6ae87","arxiv_id":"2512.01870","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A GPT-2 trained on the first 10^11 integers encoded as rooted-tree Dyck words reaches ~0.4 next-word accuracy, but the body does not contain the claimed controls or far-range test blocks.","lead":"This paper trains a GPT-2 transformer on the sequence of tree-shaped encodings of integers and asks whether it predicts the next tree better than a Markov baseline. It reports about 40% word accuracy, but the advertised tests at 10^13–10^15 and the controls that would show learning beyond frequencies are absent from the manuscript.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claims 10^13–10^15 extrapolation and control-separation results that appear nowhere in Section 5; the central 'beyond frequency' claim rests on these absent experiments.","rationale":"The reader's weakest assumption correctly identifies the load-bearing gap: the abstract promises control experiments and far extrapolation that do not exist in the body. My stress-test confirms this by reading Section 5 closely: the dataset is 10^11, the test set is the validation split of the 10th chunk, and no control sequences or 10^13–10^15 blocks are mentioned. The reported results—training curves, accuracy vs temperature, precision/recall—are internally consistent and might support a modest claim about GPT-2 outperforming a Markov baseline on this arithmetic text. However, the paper's headline claim is stronger: it asserts that the model's performance remains stable far beyond training and that likelihood separation from frequency-matched controls widens with context. These are the only pieces of evidence that would distinguish 'capturing grammar' from 'matching frequency profiles'. Their absence means the central claim is unsupported, not merely weakly supported. My verdict is UNCHANGED because the reader already reached REJECT, and my analysis does not alter that conclusion; it reinforces it.","tokens_in":15267,"tokens_out":2299,"duration_ms":22339,"concrete_test":"Run the frequency-matched control experiment described in the abstract: construct a shuffled sequence with identical unigram word frequencies but no sequential order, evaluate the trained GPT-2's per-token likelihood on it and on the true arithmetic text at context lengths 64, 256, and 1024. If the likelihood gap does not increase with context, the 'beyond frequency profile' claim fails; if no such experiment is present in the paper or code, the abstract is overclaiming.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the transformer captures regularities beyond frequency statistics, evidenced by stable accuracy on test blocks at 10^13–10^15 and by likelihood separation from two controls that widens with context—is not reported in the experimental section. Section 5 describes one dataset NT_n with n=10^11, split into 10 chunks; the 10th chunk's validation portion is used as test. No experiments at 10^13–10^15 are described, and no controls (frequency-matched shuffled sequences or sequences with four consecutive square-free numbers) appear anywhere in the manuscript. Figures 2–8 show loss curves, accuracy versus temperature, precision/recall, confusion matrices, and masked-token accuracy; none involve the controls or far extrapolation. The conclusion that 'the distinction between statistical correlation and structural dependence emerges clearly' is argued from this absent evidence. Because the abstract's strongest claim is precisely the one that would distinguish 'learning grammar' from 'matching frequency profiles', and because that evidence is missing, the central claim is unsupported as written. This is not an internal inconsistency in the reported results; it is a gap between assertion and evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains a GPT-2-scale transformer on the sequence of Dyck-word encodings of rooted trees associated with the prime tower factorization of the first 10^11 integers, and evaluates it on next-word prediction and masked language modelling against a Markov-chain baseline. The body reports loss curves, word accuracy and KL divergence as functions of temperature, precision/recall/F1 for frequent Dyck words, confusion matrices, prime-position error analysis, and a temperature/mask-probability sweep for MLM. The abstract and conclusion, however, advertise two further results that are not described anywhere in the experimental section: stable accuracy on test blocks at 10^13--10^15, and likelihood separation from two control texts (frequency-matched shuffled sequences and sequences violating the 'no four consecutive square-free numbers' rule). The same mismatch applies to the advertised 'scaling analysis over context window, dataset size, vocabulary size and model size', which is not present in the body beyond a vocabulary-size comparison.","tokens_in":15458,"tokens_out":3555,"duration_ms":35533,"significance":"If the absent experiments were actually performed and reported, the paper would make a meaningful empirical contribution: demonstrating that a transformer can partially extrapolate a deterministic arithmetic text beyond its training interval and distinguish structural regularities from frequency-matched controls would be a useful probe of learnability in arithmetic sequences. The reported results that are present -- especially the accuracy above the Markov baseline in Figures 3--6 -- represent a modest but real empirical finding for the Dyck-word representation. The paper would also gain from the clear construction of the arithmetic-text dataset and the use of BPE tokenization. As it stands, however, the strongest advertised conclusions rest on experiments that are not described, so the significance of the work cannot be assessed on the evidence provided.","major_comments":[{"comment":"The abstract states that 'performance remains stable on test blocks located at 10^13--10^15, far beyond the training interval.' The experimental section describes only one dataset, NT_n with n=10^11, split into 10 chunks; the test set is the last 25% of the 10th chunk (NT_n^(10,vt)), which lies within n=10^11. No experiment with n=10^13, 10^14, or 10^15 is described, and no figure or table reports such test blocks. This is the load-bearing evidence for the claimed extrapolation capability, and it is missing.","section":"Section 5, Data paragraph"},{"comment":"The abstract claims that 'the likelihood assigned by the model separates the arithmetic text from two controls: synthetic sequences reproducing its word frequencies exactly but carrying no sequential organization... and sequences containing more than three consecutive square-free integers.' No control experiment appears anywhere in Section 5 or in Figures 2--8. Consequently, the Section 6 statement that 'the distinction between statistical correlation and structural dependence emerges clearly' is unsupported by the reported results. This distinction is the central point that would separate 'learning grammar' from 'matching frequency profiles', so the experiment must be present or the claim must be withdrawn.","section":"Abstract and Section 5/6"},{"comment":"The abstract promises 'a scaling analysis over context window, dataset size, vocabulary size and model size.' The body only varies the tokenizer vocabulary size D (64, 256, 1024 for NWP in Figure 2, and 64, 256 for MLM in Figure 7). There is no context-window sweep, no dataset-size sweep, and no model-size sweep (the architecture is fixed to a 12-layer GPT-2 with 8.7e7 parameters). This advertised scaling analysis is not reported in the experimental section.","section":"Abstract and Section 5"},{"comment":"The conclusion interprets the model's errors as revealing 'the limit of what can be inferred statistically from local dependencies' and claims that 'the model predicts as far as its structure allows.' This boundary claim is not operationalized: no experiment varies the context length or the model capacity while holding data fixed, so the manuscript does not measure a context or capacity horizon. The conclusion goes beyond what the reported results can support.","section":"Section 6, Conclusions"}],"minor_comments":[{"comment":"Typo: 'model tained' should be 'model trained'.","section":"Section 5.2, first sentence"},{"comment":"The dimensions of Q and K are written as R^{R x d}; the usual convention is d x R (or d x d_k). Please clarify the notation.","section":"Equation (3)"},{"comment":"The sum in the definition of KL_w runs over all distinct words in w(t), but the first factor f_w(hat w) is defined on words in the true sentence. The support mismatch makes the definition ambiguous; it should be over the union of the two supports or otherwise specified.","section":"KL_w definition, Section 5.2"},{"comment":"The text says 'red dotted line' for the Markov baseline, while the figure caption describes a 'green dotted line.' Please make the color reference consistent.","section":"Figure 3"},{"comment":"The caption says the x-axis is 'ordered by words’ frequencies,' but no frequency scale is shown. Please provide the ordering or a labeled axis.","section":"Figure 4 caption"},{"comment":"The notation uses |ND_{9·10^10}| before defining the training-set index range; please reconcile this with the definition of ND_n in Section 2.","section":"Section 5, Tokenization"}],"recommendation":"reject","confidential_remarks":"The abstract and the body of the manuscript report substantially different experiments. The headline claims about 10^13--10^15 extrapolation, the two control sequences, and the scaling analysis are absent from the experimental section. This is not a minor wording issue; the manuscript would need new experiments and new analysis to support its stated conclusions. If the authors have run those experiments, a revised submission could be reconsidered, but the current text is not suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is a new testbed: training GPT-2 on the Dyck-word encoding of integers via prime tower factorization. The paper shows, with some plausibility, that a transformer beats a first-order Markov baseline on next-token and masked prediction for this sequence. That modest result is supported by the figures: the loss curves, the accuracy-vs-temperature plots, and the precision/recall analysis for primes are internally consistent and worth a look.\n\nBut the abstract goes further. It claims stable performance on test blocks at 10^13–10^15, likelihood separation from two controls, and a scaling analysis over context window, dataset size, vocabulary size, and model size. None of that is in the experimental section. Section 5 describes a single dataset up to n=10^11, split into ten chunks, with the last chunk's validation portion used as test. There are no controls, no far-range tests, and no context-window or model-size sweeps. The only scaling that appears is vocabulary size and mask probability. So the paper's central claim—that the model captures structure beyond frequency statistics—rests on evidence that is not in the manuscript. That is a critical gap, not a minor omission.\n\nAlso worth noting: the headline accuracies have no error bars, and no code or data is shipped. The conclusion talks about 'context-sensitive reasoning' and a 'distinction between statistical correlation and structural dependence' emerging clearly, but those conclusions are argued from the absent control experiments.\n\nWhat the paper does well is set up a clean, well-motivated benchmark. The Dyck-word encoding is a natural way to expose multiplicative structure, and the Markov baseline is an appropriate minimal comparator. The self-citations to earlier statistical work on the tree encoding are motivational and not circular—that part is fine.\n\nMy take: the paper deserves a serious referee, not a desk reject, because the core idea is worth testing and the missing experiments are easy to specify. But as it stands, it overclaims. The authors need to add the control experiments (frequency-matched shuffled sequences and the square-free-block constraint), the far-range test evaluations, the scaling analyses, error bars, and reproducibility artifacts. That would turn a plausible modest finding into a solid one. Until then, treat the abstract's stronger claims as unverified.","headline":"Interesting benchmark idea with a solid modest result, but the abstract promises control and extrapolation experiments that aren't in the body.","tokens_in":16057,"tokens_out":1781,"would_cite":false,"duration_ms":17760,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A GPT-2 transformer, trained on the first 10^11 integers encoded as Dyck-word trees, partially learns the grammar of the arithmetic sequence, reaching word accuracy around 0.4 and stable performance on test blocks at 10^13–10^15.","keywords":["arithmetic text","rooted trees","prime factorization","Dyck words","transformers","next-word prediction","masked language modeling","learnability"],"falsifier":"A concrete way to settle the claim is to evaluate the trained model on integers between 10^13 and 10^15 and compare its word accuracy to the Markov chain baseline; if the accuracy drops to the baseline level, the claimed stable extrapolation is falsified. Similarly, if the reported separation from frequency-matched shuffled controls cannot be reproduced when the experiment is run as described in the abstract, the claim that learning goes beyond frequency statistics collapses.","tokens_in":15050,"feed_emoji":"🌳","tokens_out":3441,"duration_ms":33642,"temperature":0.7,"pith_summary":"The paper attempts to show that a transformer language model can learn the deterministic sequence of rooted trees encoding the iterated prime factorization of the integers. It trains a GPT-2 model on the first 10^11 elements of this arithmetic text and evaluates next-word and masked-word prediction against a Markov baseline. The central claim is that the model reaches about 0.4 word accuracy, well above the baseline, and that its performance remains stable on blocks located at 10^13–10^15, far beyond the training interval. The abstract further claims that the likelihood assigned by the model separates the arithmetic text from control sequences that match its word frequencies but lack its sequential organization. If true, this suggests that a transformer can infer structure beyond frequency statistics, offering a controlled probe of how neural sequence models extract latent rules.","feed_headline":"GPT-2 learns the prime-factor tree sequence at 0.4 accuracy","feed_subtitle":"Trained on the first 10^11 integers, it keeps performing on blocks at 10^13–10^15, separating syntax from frequency.","key_machinery":"The central object is the rooted planar tree representation of each integer via iterated prime factorization, mapped to a Dyck word—a balanced binary string—so that the sequence of integers becomes an arithmetic text NT. The learning machinery is a GPT-2-style transformer decoder with 12 layers, 12 attention heads, and an embedding dimension of 768, trained on tokenized Dyck words using Byte-Pair Encoding. The two self-supervised tasks, next-word prediction and masked language modeling, with a temperature-controlled softmax, are used to measure how much of the text's grammar the model captures. A Markov chain transition matrix serves as the minimal baseline.","core_discovery":"On its own terms, the paper claims that a transformer can partially infer the internal syntax of the arithmetic text NT, the sequence of Dyck words representing the rooted-tree structure of each integer's prime factorization. In next-word prediction, the model's word accuracy peaks around 0.4 at low sampling temperatures, outperforming a Markov chain baseline; precision, recall, and F1 for primes are each around 0.3, with higher values for square-free words. The model's errors are systematic, confusing primes with square-free numbers. In masked-language modeling, token accuracy exceeds 0.4 in the low-temperature regime and degrades as masking probability and temperature increase. The abstrac","pith_inferences":["The experimental section as printed describes only the dataset up to n=10^11 with ten chunks and no control experiments, no context-length sweep, and no model-size sweep; the abstract's claims about 10^13–10^15 stability and control separation rest on experiments that are not reported in the body of this version.","If those control experiments were actually run, a natural extension would be to verify whether the separation from frequency-matched shuffled sequences grows with context length, as the abstract claims, and whether it holds for other architecture sizes.","The finding that the model confuses primes with square-free numbers suggests a testable hypothesis: the transformer may be learning a local rule based on the structure of the Dyck word (e.g., the absence of nested parentheses) rather than a genuine global primality criterion.","A direct extension would be to train on a shorter prefix (e.g., 10^9 or 10^10) and measure how accuracy at a fixed distant block scales with training data, which would sharpen the distinction between memorization and genuine rule induction."],"forward_implications":["If the central claim holds, transformers can partially predict prime factorizations from the preceding sequence, even though they never see the integers themselves.","The arithmetic text becomes a controlled testbed for studying how sequence models generalize from a known generative rule, since the data-generating law is exactly known.","The reported separation from frequency-matched controls would indicate that the model captures sequential, syntax-like structure rather than mere token statistics.","The stable performance on blocks far beyond the training interval would suggest that the learned regularities are not local artifacts but reflect properties of the underlying arithmetic.","This opens a path toward probing whether a model's latent representations encode algebraic relationships among numbers, as proposed in the paper's future-work section."],"fun_headline_variants":["GPT-2 learns prime-factor tree sequence at 0.4 word accuracy","Transformer generalizes on arithmetic tree sequence from 10^11","Model beats Markov baseline on next-word prediction of tree text","GPT-2 captures syntax in arithmetic sequence, not just frequency","Transformer learns number-theoretic tree order with 0.4 accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the control experiments and the 10^13–10^15 test blocks described in the abstract were actually performed and evaluated as claimed, because Section 5 only reports experiments on data up to 10^11 and does not include those controls or distant test blocks.","fun_headline_variants_meta":{"raw":{"variants":["GPT-2 learns prime-factor tree sequence at 0.4 word accuracy","Transformer generalizes on arithmetic tree sequence from 10^11","Model beats Markov baseline on next-word prediction of tree text","GPT-2 captures syntax in arithmetic sequence, not just frequency","Transformer learns number-theoretic tree order with 0.4 accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000419,"raw_usage":{"total_tokens":1995,"prompt_tokens":744,"completion_tokens":1251,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":1162}},"tokens_in":488,"tokens_out":1251,"duration_ms":9544,"temperature":1.0,"reasoning_tokens":1162,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:39:46.636520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete way to settle the claim is to evaluate the trained model on integers between 10^13 and 10^15 and compare its word accuracy to the Markov chain baseline; if the accuracy drops to the baseline level, the claimed stable extrapolation is falsified. Similarly, if the reported separation from frequency-matched shuffled controls cannot be reproduced when the experiment is run as described in the abstract, the claim that learning goes beyond frequency statistics collapses.","supporting_citations":[],"review_version":1}