{"id":"c046df11-dd16-45a3-b1c3-9e62c138bcd3","arxiv_id":"2608.03930","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Logic pre-pretraining, training a small LM on next-step formal derivations before natural language, accelerates skill acquisition and improves pruning robustness at a 100B-token scale.","lead":"Researchers trained a 254M-parameter language model first on formal logic derivations, then on about 100 billion tokens of ordinary text. The logic-first model reached 80% accuracy on linguistic skill tests with roughly 36 billion fewer tokens and survived pruning better, though each main run was done once.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-hoc selection of the 17-task evaluation subset may inflate the headline 80%-accuracy and 36B-token savings; full-suite means should be the primary numbers.","rationale":"The paper is a well-controlled study with a plausible mechanism, and the appendix's full-task trajectories are a genuine partial safeguard. The most load-bearing issue is not the geometric or pruning analysis per se, but the dependence of the headline behavioral improvement on a post-hoc task-selection rule. The 80% threshold is used both to select the evaluation tasks and to define the token-saving metric, so the main quantitative claim can be optimistic even if every run is reproducible. A cheap and decisive check is to recompute the same summary statistics on the full 130-task suite; the data appear to already exist in the appendix. The single-seed limitation is also real, but it is secondary: a 7.1-point final gap and a 36B-token crossover are large enough that seed noise alone is less likely to explain them, whereas subset selection can mechanically create or inflate such gaps. The reader's weakest-assumption statement already flagged both issues; I agree with the selection concern and would emphasize it as the primary reason the current abstract's precision is not yet warranted. The appropriate verdict remains conditional acceptance: the stated claims should be reported on the full suite and, ideally, with seed-averaged behavioral curves before the numbers are taken at face value.","tokens_in":25920,"tokens_out":3864,"duration_ms":43320,"concrete_test":"Recompute the headline behavioral curves (Figure 1A and the mean panel of Figure 3) using all 130 elemental tasks instead of the post-hoc 17-task subset, from the checkpoints already saved. Compute (i) final mean accuracy per condition, (ii) number of training tokens to reach 0.8 mean accuracy per condition, and (iii) the Logic-PPT vs PT margin. If Logic-PPT retains roughly the same final margin and a ~36B token savings on the full suite, the selection concern is resolved. If the margin shrinks materially or PT does not reach 0.8 on the full suite, the full-suite numbers must be reported as primary and the abstract claims requalified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—80% accuracy with 36B fewer tokens and a 7.1-point final advantage over the strongest baseline—is computed on a 17-task subset of the 130 elemental tasks selected, per Section 3, as tasks 'for which at least one PPT configuration achieves 80% accuracy during language pretraining within the 100B token budget.' Because the selection rule depends on the same model outcomes the headline summarizes, the evaluation set is not independent of the treatment. The criterion can mechanically favor Logic-PPT: tasks where Logic-PPT crosses 80% early while PT never reaches 80% are included, and tasks where all conditions are near ceiling or Logic-PPT is weak are excluded. The appendix does report full 130-task trajectories, which partially mitigates this, but the main text and abstract average only the selected subset, and no full-suite summary mean or token-to-threshold is reported. The single-seed issue compounds this: the 100B runs are one seed per condition, and the three-seed check covers only 10B tokens and only CKA geometry, not behavioral accuracy or pruning. Of the two concerns, the task-selection dependence is more load-bearing because it bears directly on whether the reported 36B-token saving and 7.1-point margin are estimates of a real treatment effect or artifacts of subset choice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Logic-PPT, a pre-pretraining stage in which a 220M-parameter Transformer is trained on 2.5B tokens of synthetic formal-logic derivation next-step prediction before 100B tokens of FineWeb-Edu language pretraining. Compared with random initialization and with Dyck/Shuffle-Dyck and algorithmic (Set/Sort/Union) pre-pretraining, the authors report faster skill acquisition on a subset of 17 elemental tasks (80% accuracy reached with 36B fewer tokens), persistent lower-rank/spectrally concentrated representations, and improved Wanda pruning robustness. The paper includes detailed mechanistic analyses (CKA, RankMe, spectral decay, stable rank) and appendix trajectories for the full 130-task elemental suite.","tokens_in":26206,"tokens_out":6733,"duration_ms":65995,"significance":"If the results hold, the paper makes a useful empirical contribution: it scales symbolic pre-pretraining evaluation to 100B tokens, introduces a new symbolic data family (formal derivations with 247 schemata from propositional, term, and first-order logic), and provides dense checkpoint-level behavioral and representational trajectories. The controlled comparison across four training conditions, the use of public datasets, and the full-suite trajectory plots in the appendix are strengths. The pruning result, if confirmed, is practically interesting. However, the headline behavioral numbers are computed on an outcome-dependent task subset and from single-seed runs; the mechanistic 'enables' claim is correlational; and there is an internal inconsistency in the pruning numbers. These issues are fixable by reanalysis and reframing, but they currently overstate the certainty of the central claims.","major_comments":[{"comment":"The main behavioral headline - 80% accuracy with 36B fewer tokens and the 7.1-point final gap over Formal-PPT - is computed on the 17 elemental tasks selected by the rule 'tasks for which at least one PPT configuration achieves 80% accuracy during language pretraining within the 100B token budget' (Section 3). Because this selection uses the same model outcomes that the headline summarizes, the evaluation set is not independent of the treatment. Tasks in which Logic-PPT crosses 80% while PT never does are included; tasks in which Logic-PPT is weak or all conditions saturate are excluded. This can mechanically create or inflate the reported advantage. The appendix's full 130-task trajectories (Figures 11-15) are helpful but do not provide full-suite mean accuracies or token-to-threshold values; without these, the abstract's 36B-token saving is not a reliable estimate. Please report full-s","section":"Section 3 and Section 4.2"},{"comment":"The text states that at 40% sparsity Logic-PPT mean accuracy decreases from 0.87 to 0.745, a 14.4% relative drop. However, Figure 9's Logic-PPT Mean column at 40% sparsity shows 0.72, which corresponds to a 17.2% relative drop. One of these numbers is wrong. Since the 14.4% figure and the 'matches dense baseline at ≈33% sparsity' claim are central to the compression section, this inconsistency must be reconciled, and the interpolation to ≈33% should be made explicit (e.g., whether it is read from a checkpoint or linearly interpolated).","section":"Section 6 and Figure 9"},{"comment":"All 100B-token behavioral, representational, and pruning results are single-seed point estimates. The three-seed stability check covers only 10B tokens and only self-layer CKA; it does not cover the accuracy trajectories, the 36B token-to-threshold comparison, or the pruning numbers. Consequently the headline margins (e.g., 7.1 points vs Formal-PPT, 14.4% vs 26.8% pruning degradation) have no quantified uncertainty. The authors should either add seed variability for at least the main behavioral endpoints (even at lower token budgets) or explicitly present all headline numbers as descriptive single-run results, avoiding language that implies population-level effect sizes.","section":"Limitations and Appendix F"},{"comment":"The claim that 'this internal geometry enables improved model compressibility via pruning' goes beyond the evidence. The paper demonstrates that Logic-PPT's models have lower RankMe, higher spectral decay, clearer CKA block structure, and better pruning robustness than the other conditions, but all measures are obtained from the same training conditions; no analysis rules out other differences (final accuracy levels, different effective training trajectories, or different weight spectra) as the driver of pruning robustness. To support 'enables,' please provide an intervention or mediation analysis (e.g., pruning models matched on final accuracy, or perturbing the geometry) or soften the claim to a correlational association.","section":"Abstract and Section 6"},{"comment":"The appendix states: 'the 17 tasks in Figure 10 are the subset without a clean emergence point, used for the emergence-order analysis.' This directly contradicts Section 3's description of the 17-task subset as tasks for which at least one PPT configuration reaches 80% accuracy. This internal inconsistency affects how the reader interprets the main evaluation set and must be corrected.","section":"Appendix K"}],"minor_comments":[{"comment":"Typographical issues: Montague (2002) title has 'Quntification'; Kornblith et al. (2019) has 'PMlR'; Table 3 has 'V ocabulary'; Table 7 uses 'Algo-PPT' while the rest of the paper uses 'Alg-PPT'.","section":"References / Table 3"},{"comment":"The method for reading the 36B token saving from the trajectories is not specified. State whether this is from the first checkpoint crossing 0.8 mean accuracy, from linear interpolation between checkpoints, or from smoothed curves.","section":"Section 4.2 / Figure 1"},{"comment":"Table 8 lists 17 tasks, but the counts per category are not summarized. A sentence confirming the category counts would help readers verify the composition of the 17-task subset.","section":"Table 8 / Figure 10"},{"comment":"The sentence 'At 40% sparsity, Logic-PPT incurs only a 14.4% relative drop compared to an average 26.8% degradation across alternative symbolic PPT methods' should list the individual drops for Formal-PPT and Alg-PPT in the same sentence for reproducibility; currently the reader must infer them from Figure 9.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a competent large-scale empirical study with a serious evaluation-subset problem. I would not reject on the basis of scale or the correlational mechanistic analysis alone, but the authors must address the task-selection dependence and the Section 6/Figure 9 inconsistency in revision. If full-suite means overturn the headline, the paper may still be publishable as a more modest claim about Logic-PPT's effects."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is the first paper I've seen that pre-trains LMs on formal logical derivations before language pretraining, and it scales the comparison to 100B tokens. That alone is a meaningful extension of the pre-pretraining paradigm, and the pruning-robustness result is genuinely new. The controlled setup—matched architecture, roughly matched symbolic token budgets, comparisons against Dyck/Shuffle-Dyck and algorithmic baselines—is done carefully. The appendix also reports full 130-task trajectories, which is more than most papers in this subfield give you.\n\nWhat it does well: the data generation is adapted from LogicTree but applied to a new purpose; the 247-schema library spanning propositional, term, and first-order logic is a solid resource. The mechanistic tracing with CKA, RankMe, and spectral decay is thoughtful, and the weight-space stable-rank analysis during symbolic pretraining is a nice addition. The paper is also honest: the limitations section openly acknowledges the single-seed main runs and the narrow evaluation scope.\n\nThe soft spots are real but not fatal. The headline \"80% accuracy with 36B fewer tokens\" and the 7.1-point final advantage are computed on 17 of 130 elemental tasks, selected as those where at least one PPT configuration reached 80%. That selection rule depends on the same model outcomes being summarized, so it can inflate Logic-PPT's apparent advantage. The appendix does show full-suite trajectories, which mitigates cherry-picking concerns, but the main text never reports full-suite means, and the abstract states the 36B saving without qualification. That needs to change. The single-seed issue compounds this: the 100B runs are one seed per condition, and the three-seed check covers only 10B tokens and only CKA geometry, not behavioral accuracy or pruning. Finally, \"geometry enables compressibility\" is correlation, not causation; the authors phrase it carefully, but the title's \"Fosters\" oversells a bit.\n\nWho this is for: anyone working on symbolic pre-pretraining, training efficiency, or pruning robustness will want to read it and cite it. The core idea is plausible and the experimental design, apart from the subset selection, is controlled. It deserves a serious referee. I'd send it to peer review, but I'd require the authors to ship code and data, report full-suite means as primary numbers, and either add seed-averaged behavioral curves or downscope the precision of the headline claims.","headline":"Solid new result on logic pre-pretraining, but the headline numbers rest on a post-hoc selected evaluation subset and single-seed runs; worth refereeing with those issues addressed.","tokens_in":26722,"tokens_out":1112,"would_cite":true,"duration_ms":13368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a 254M language model on 2.5B tokens of formal proof steps before language pretraining lets it hit 80% accuracy on elemental linguistic tasks with 36B fewer tokens, and its internal low-rank organization tolerates about one-third w","keywords":["logic pre-pretraining","formal derivations","inductive bias","skill acquisition","representational geometry","model pruning","backward chaining","language model pretraining"],"falsifier":"Re-run each training condition with multiple seeds at the full 100B-token budget and compare mean elemental-task accuracy trajectories: the central claim fails if the roughly 36B-token advantage and roughly 33%-sparsity parity do not reproduce across seeds. A cheaper discriminating check is to score the existing final checkpoints on the full 130-task suite instead of the 17 tasks selected by the 80% criterion; if Logic-PPT's advantage shrinks to those tasks, the accelerated-skill-acquisition claim is selection-driven.","tokens_in":25763,"feed_emoji":"🧮","tokens_out":6485,"duration_ms":59511,"temperature":0.7,"pith_summary":"This paper argues that the symbolic data used to warm up a language model should be as structurally rich as language itself, and proposes formal logical derivations as that warm-up material. The authors train a 254M-parameter transformer on 2.5B tokens of next-step formal proof prediction, using derivations built from 247 inference schemata spanning propositional, term, and first-order logic, and then continue with ordinary language pretraining on 100B tokens. Their central claim is that this logic warm-up accelerates skill acquisition: the model reaches 80% mean accuracy on a suite of 17 elemental linguistic tasks using roughly 36B fewer natural-language tokens than a randomly initialized baseline, and beats other symbolic warm-ups (Dyck brackets, sorting/set algorithms) by 7.1 accuracy points at the final checkpoint. They further claim that the logic-trained model reorganizes its internal representations into a lower-rank, spectrally concentrated geometry that persists through training, and that this geometry makes the model measurably more compressible: at about 33% weight pruning it matches the dense baseline, and at 40% it degrades much less than the alternatives. If true, the paper shows that a cheap, fully synthetic warm-up stage can buy both data efficiency and compressibility in the same model.","feed_headline":"2.5B logic tokens skip 36B language tokens","feed_subtitle":"Proof-step warm-up makes language models learn skills faster and survive pruning, all for 2.5B synthetic tokens.","key_machinery":"The central object is the formal derivation, serialized as a next-step prediction task: given premises, a partial derivation prefix, and a goal, the model must predict the next valid proof step. Training examples are generated by backward chaining over a library of 247 inference-rule schemata (propositional, term, and first-order logic), with structure-preserving symbol renaming for diversity and a post-order traversal that turns each backward-built proof tree into a valid forward derivation sequence. This object carries the argument because it is what distinguishes Logic-PPT from earlier pre-pretraining data: it forces simultaneous handling of variable binding, quantifier scope, and predica","core_discovery":"The paper's central claim is that pre-pretraining on formal derivations, rather than on simpler symbolic data such as balanced-bracket strings or algorithmic operations, transfers a genuinely useful inductive bias to natural-language learning. A formal derivation requires the model to bind variables, track quantifier and predicate dependencies, and compose premises into conclusions across a long context, which the authors treat as the abstract mechanisms that natural language also relies on. After 2.5B tokens of this next-step derivation training, the model's Transformer backbone retains a persistent structural imprint through 100B tokens of language pretraining: representations stay lower-r","pith_inferences":["The paper's 17-task evaluation set was selected by the condition that at least one configuration reached 80% accuracy; whether Logic-PPT's advantage holds across the full 130-task suite, or concentrates on rule-like tasks, is untested and would sharpen the claim.","The low-rank geometry result hints that other compression schemes that exploit spectral structure, such as quantization, low-rank factorization, or layer dropping, may also benefit, but the paper only tests Wanda pruning; a direct test of spectrum-aware compression is a natural next step.","If variable-binding and quantifier structure is the active ingredient, logic warm-up might transfer to non-English or code-like token distributions that share those mechanisms; the paper's English-only scope leaves that open.","The persistent mid-layer reorganization suggests the effect is an initialization prior rather than a transient warm-up; testing transfer through supervised fine-tuning or instruction tuning would show whether the benefit survives post-training."],"forward_implications":["A 2.5B-token synthetic logic corpus can replace roughly 36B tokens of natural-language training to reach the same skill threshold, cutting pretraining cost materially.","Richer symbolic structure matters: formal derivations beat both formal-language and algorithmic warm-ups at the final checkpoint, so previous narrow-primitive pre-pretraining left gains on the table.","Representational geometry is a lever on compressibility: the lower-rank, spectrally concentrated space induced by logic training persists to the end of pretraining and shows up as pruning tolerance, suggesting geometry-based criteria could guide which initializations are worth keeping.","The pruning advantage is specific to Logic-PPT, not to pre-pretraining in general: algorithmic warm-up degrades worst under pruning despite also being symbolic."],"supporting_citations":[{"why":"Introduced pre-pretraining on formal languages (Dyck/Shuffle-Dyck); supplies the formal-language baseline and the narrow-primitive limitation Logic-PPT addresses.","marker":"Hu et al., 2025"},{"why":"Introduced procedural algorithmic pre-pretraining (Set/Sort/Union); supplies the algorithmic baseline and the small-token-budget limitation Logic-PPT scales beyond.","marker":"Jiang et al., 2026"},{"why":"LogicTree; supplies the backward-chaining derivation-tree generation method used to build the formal-derivation dataset.","marker":"Wang et al., 2025"},{"why":"Supplies the elemental-task suite, the emergence/implicit-curriculum framing, and the 80%-accuracy task-selection criterion used for evaluation.","marker":"Liu et al., 2026"},{"why":"Supplies the entropy-seeking/compression-seeking pretraining-dynamics framing and the RankMe/spectral-decay metrics used in the mechanistic analysis.","marker":"Li et al., 2025"},{"why":"Wanda; the pruning method used to test compressibility at sparsity levels up to 40%.","marker":"Sun et al., 2023"},{"why":"Linear CKA; the representation-similarity metric used for layer-wise and cross-model comparisons.","marker":"Kornblith et al., 2019"},{"why":"BLiMP; the linguistic minimal-pairs benchmark used to track acceptability-skill acquisition.","marker":"Warstadt et al., 2020"},{"why":"FineWeb-Edu; the natural-language pretraining corpus used for the 100B-token runs.","marker":"Lozhkov et al., 2024"}],"fun_headline_variants":["Logic pretraining: 80% accuracy, 36B fewer tokens","Formal derivations pretrain: faster skills, prunable to 33%","2.5B logic tokens: skip 36B language tokens, keep pruning","Proof-step warm-up: better LMs, compress to one-third","Logic before language: less data, more compressible"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The behavioral and pruning numbers come from a single 100B-token run per condition; the only multi-seed check ran 10B tokens and measured representation geometry, not accuracy or pruning, and the headline 80% threshold is computed on 17 of 130 tasks chosen because at least one condition reached that threshold.","fun_headline_variants_meta":{"raw":{"variants":["Logic pretraining: 80% accuracy, 36B fewer tokens","Formal derivations pretrain: faster skills, prunable to 33%","2.5B logic tokens: skip 36B language tokens, keep pruning","Proof-step warm-up: better LMs, compress to one-third","Logic before language: less data, more compressible"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2539,"prompt_tokens":749,"completion_tokens":1790,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":1694}},"tokens_in":493,"tokens_out":1790,"duration_ms":16924,"temperature":1.0,"reasoning_tokens":1694,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:29:23.505169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run each training condition with multiple seeds at the full 100B-token budget and compare mean elemental-task accuracy trajectories: the central claim fails if the roughly 36B-token advantage and roughly 33%-sparsity parity do not reproduce across seeds. A cheaper discriminating check is to score the existing final checkpoints on the full 130-task suite instead of the 17 tasks selected by the 80% criterion; if Logic-PPT's advantage shrinks to those tasks, the accelerated-skill-acquisition claim is selection-driven.","supporting_citations":[],"review_version":1}