Pith. sign in

REVIEW 5 major objections 4 minor 87 references

Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a 254M language model on 2.5B tokens of formal proof steps before language pretraining lets it hit 80% accuracy on elemental linguistic tasks with 36B fewer tokens, and its internal low-rank organization tolerates about one-third w

desk verdict Solid new result on logic pre-pretraining, but the headline numbers rest on a post-hoc selected evaluation subset and single-seed runs; worth refereeing with those issues addressed. read the letter →

arxiv 2608.03930 v1 pith:QG7C3XJ4 submitted 2026-08-04 cs.CL cs.AIcs.LG

Jo-Ku Cheng , Nikolaos Aletras , Marco Valentino This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords logicpre-pretrainingformalderivationsinductivebiasskillacquisitionrepresentationalgeometrymodelpruningbackwardchaininglanguagepretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the symbolic data used to warm up a language model should be as structurally rich as language itself, and proposes formal logical derivations as that warm-up material. The authors train a 254M-parameter transformer on 2.5B tokens of next-step formal proof prediction, using derivations built from 247 inference schemata spanning propositional, term, and first-order logic, and then continue with ordinary language pretraining on 100B tokens. Their central claim is that this logic warm-up accelerates skill acquisition: the model reaches 80% mean accuracy on a suite of 17 elemental linguistic tasks using roughly 36B fewer natural-language tokens than a randomly initialized baseline, and beats other symbolic warm-ups (Dyck brackets, sorting/set algorithms) by 7.1 accuracy points at the final checkpoint. They further claim that the logic-trained model reorganizes its internal representations into a lower-rank, spectrally concentrated geometry that persists through training, and that this geometry makes the model measurably more compressible: at about 33% weight pruning it matches the dense baseline, and at 40% it degrades much less than the alternatives. If true, the paper shows that a cheap, fully synthetic warm-up stage can buy both data efficiency and compressibility in the same model.

What carries the argument

The central object is the formal derivation, serialized as a next-step prediction task: given premises, a partial derivation prefix, and a goal, the model must predict the next valid proof step. Training examples are generated by backward chaining over a library of 247 inference-rule schemata (propositional, term, and first-order logic), with structure-preserving symbol renaming for diversity and a post-order traversal that turns each backward-built proof tree into a valid forward derivation sequence. This object carries the argument because it is what distinguishes Logic-PPT from earlier pre-pretraining data: it forces simultaneous handling of variable binding, quantifier scope, and predica

What would settle it

Re-run each training condition with multiple seeds at the full 100B-token budget and compare mean elemental-task accuracy trajectories: the central claim fails if the roughly 36B-token advantage and roughly 33%-sparsity parity do not reproduce across seeds. A cheaper discriminating check is to score the existing final checkpoints on the full 130-task suite instead of the 17 tasks selected by the 80% criterion; if Logic-PPT's advantage shrinks to those tasks, the accelerated-skill-acquisition claim is selection-driven.

Watch

Extended reading notes

Core claim

The paper's central claim is that pre-pretraining on formal derivations, rather than on simpler symbolic data such as balanced-bracket strings or algorithmic operations, transfers a genuinely useful inductive bias to natural-language learning. A formal derivation requires the model to bind variables, track quantifier and predicate dependencies, and compose premises into conclusions across a long context, which the authors treat as the abstract mechanisms that natural language also relies on. After 2.5B tokens of this next-step derivation training, the model's Transformer backbone retains a persistent structural imprint through 100B tokens of language pretraining: representations stay lower-r

Load-bearing premise

The behavioral and pruning numbers come from a single 100B-token run per condition; the only multi-seed check ran 10B tokens and measured representation geometry, not accuracy or pruning, and the headline 80% threshold is computed on 17 of 130 tasks chosen because at least one condition reached that threshold.

Editorial extensions

If this is right

  • A 2.5B-token synthetic logic corpus can replace roughly 36B tokens of natural-language training to reach the same skill threshold, cutting pretraining cost materially.
  • Richer symbolic structure matters: formal derivations beat both formal-language and algorithmic warm-ups at the final checkpoint, so previous narrow-primitive pre-pretraining left gains on the table.
  • Representational geometry is a lever on compressibility: the lower-rank, spectrally concentrated space induced by logic training persists to the end of pretraining and shows up as pruning tolerance, suggesting geometry-based criteria could guide which initializations are worth keeping.
  • The pruning advantage is specific to Logic-PPT, not to pre-pretraining in general: algorithmic warm-up degrades worst under pruning despite also being symbolic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 17-task evaluation set was selected by the condition that at least one configuration reached 80% accuracy; whether Logic-PPT's advantage holds across the full 130-task suite, or concentrates on rule-like tasks, is untested and would sharpen the claim.
  • The low-rank geometry result hints that other compression schemes that exploit spectral structure, such as quantization, low-rank factorization, or layer dropping, may also benefit, but the paper only tests Wanda pruning; a direct test of spectrum-aware compression is a natural next step.
  • If variable-binding and quantifier structure is the active ingredient, logic warm-up might transfer to non-English or code-like token distributions that share those mechanisms; the paper's English-only scope leaves that open.
  • The persistent mid-layer reorganization suggests the effect is an initialization prior rather than a transient warm-up; testing transfer through supervised fine-tuning or instruction tuning would show whether the benefit survives post-training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper proposes Logic-PPT, a pre-pretraining stage in which a 220M-parameter Transformer is trained on 2.5B tokens of synthetic formal-logic derivation next-step prediction before 100B tokens of FineWeb-Edu language pretraining. Compared with random initialization and with Dyck/Shuffle-Dyck and algorithmic (Set/Sort/Union) pre-pretraining, the authors report faster skill acquisition on a subset of 17 elemental tasks (80% accuracy reached with 36B fewer tokens), persistent lower-rank/spectrally concentrated representations, and improved Wanda pruning robustness. The paper includes detailed mechanistic analyses (CKA, RankMe, spectral decay, stable rank) and appendix trajectories for the full 130-task elemental suite.

Significance. If the results hold, the paper makes a useful empirical contribution: it scales symbolic pre-pretraining evaluation to 100B tokens, introduces a new symbolic data family (formal derivations with 247 schemata from propositional, term, and first-order logic), and provides dense checkpoint-level behavioral and representational trajectories. The controlled comparison across four training conditions, the use of public datasets, and the full-suite trajectory plots in the appendix are strengths. The pruning result, if confirmed, is practically interesting. However, the headline behavioral numbers are computed on an outcome-dependent task subset and from single-seed runs; the mechanistic 'enables' claim is correlational; and there is an internal inconsistency in the pruning numbers. These issues are fixable by reanalysis and reframing, but they currently overstate the certainty of the central claims.

major comments (5)
  1. [Section 3 and Section 4.2] The main behavioral headline - 80% accuracy with 36B fewer tokens and the 7.1-point final gap over Formal-PPT - is computed on the 17 elemental tasks selected by the rule 'tasks for which at least one PPT configuration achieves 80% accuracy during language pretraining within the 100B token budget' (Section 3). Because this selection uses the same model outcomes that the headline summarizes, the evaluation set is not independent of the treatment. Tasks in which Logic-PPT crosses 80% while PT never does are included; tasks in which Logic-PPT is weak or all conditions saturate are excluded. This can mechanically create or inflate the reported advantage. The appendix's full 130-task trajectories (Figures 11-15) are helpful but do not provide full-suite mean accuracies or token-to-threshold values; without these, the abstract's 36B-token saving is not a reliable estimate. Please report full-s
  2. [Section 6 and Figure 9] The text states that at 40% sparsity Logic-PPT mean accuracy decreases from 0.87 to 0.745, a 14.4% relative drop. However, Figure 9's Logic-PPT Mean column at 40% sparsity shows 0.72, which corresponds to a 17.2% relative drop. One of these numbers is wrong. Since the 14.4% figure and the 'matches dense baseline at ≈33% sparsity' claim are central to the compression section, this inconsistency must be reconciled, and the interpolation to ≈33% should be made explicit (e.g., whether it is read from a checkpoint or linearly interpolated).
  3. [Limitations and Appendix F] All 100B-token behavioral, representational, and pruning results are single-seed point estimates. The three-seed stability check covers only 10B tokens and only self-layer CKA; it does not cover the accuracy trajectories, the 36B token-to-threshold comparison, or the pruning numbers. Consequently the headline margins (e.g., 7.1 points vs Formal-PPT, 14.4% vs 26.8% pruning degradation) have no quantified uncertainty. The authors should either add seed variability for at least the main behavioral endpoints (even at lower token budgets) or explicitly present all headline numbers as descriptive single-run results, avoiding language that implies population-level effect sizes.
  4. [Abstract and Section 6] The claim that 'this internal geometry enables improved model compressibility via pruning' goes beyond the evidence. The paper demonstrates that Logic-PPT's models have lower RankMe, higher spectral decay, clearer CKA block structure, and better pruning robustness than the other conditions, but all measures are obtained from the same training conditions; no analysis rules out other differences (final accuracy levels, different effective training trajectories, or different weight spectra) as the driver of pruning robustness. To support 'enables,' please provide an intervention or mediation analysis (e.g., pruning models matched on final accuracy, or perturbing the geometry) or soften the claim to a correlational association.
  5. [Appendix K] The appendix states: 'the 17 tasks in Figure 10 are the subset without a clean emergence point, used for the emergence-order analysis.' This directly contradicts Section 3's description of the 17-task subset as tasks for which at least one PPT configuration reaches 80% accuracy. This internal inconsistency affects how the reader interprets the main evaluation set and must be corrected.
minor comments (4)
  1. [References / Table 3] Typographical issues: Montague (2002) title has 'Quntification'; Kornblith et al. (2019) has 'PMlR'; Table 3 has 'V ocabulary'; Table 7 uses 'Algo-PPT' while the rest of the paper uses 'Alg-PPT'.
  2. [Section 4.2 / Figure 1] The method for reading the 36B token saving from the trajectories is not specified. State whether this is from the first checkpoint crossing 0.8 mean accuracy, from linear interpolation between checkpoints, or from smoothed curves.
  3. [Table 8 / Figure 10] Table 8 lists 17 tasks, but the counts per category are not summarized. A sentence confirming the category counts would help readers verify the composition of the 17-task subset.
  4. [Section 6] The sentence 'At 40% sparsity, Logic-PPT incurs only a 14.4% relative drop compared to an average 26.8% degradation across alternative symbolic PPT methods' should list the individual drops for Formal-PPT and Alg-PPT in the same sentence for reproducibility; currently the reader must infer them from Figure 9.

Circularity Check

2 steps flagged · score 5.0 of 10

Headline 80%/36B-token claim is computed on a 17-task subset selected by the same 80% performance criterion; selection-on-outcome makes the reported mean partially circular.

  1. fitted input called prediction [Section 3, 'Evaluating Language and Skill Acquisition'; used in Abstract and §4.2]
    "Following Liu et al. (2026), we retain the elemental tasks for which at least one PPT configuration achieves 80% accuracy during language pretraining within the 100B token budget. This leads to a total of 17 elemental tasks spanning emergent linguistic capabilities such as string operations, morphology, reading comprehension, and compositional tasks (Table 8 in the Appendix)."

    The headline claim — 'achieving 80% accuracy on linguistic tasks with 36B fewer tokens' — uses the same 80% threshold that defines the evaluation set. The 17 tasks are retained precisely because at least one PPT configuration reaches 80% on them, so the target is attainable by construction for at least one condition. The abstract and §4.2 then average only over this outcome-selected subset, and the reported token-to-80% and 7.1-point margin are conditional on a filter built from the same runs being summarized. Tasks on which no configuration reaches 80% are excluded, and tasks on which Logic-PPT is strong are preferentially retained whenever Logic-PPT is the configuration that crosses 80%; this mechanically inflates the reported mean gain and the apparent token saving.

  2. other [Appendix K, 'Task Performance']
    "To check that this curated subset is representative rather than cherry-picked, Figures 11–15 report the same trajectories for the full 130-task elemental task suite, split across five panels for legibility; the 17 tasks in Figure 10 are the subset without a clean emergence point, used for the emergence-order analysis."

    This appendix gives a second, incompatible definition of the 17-task subset: not Section 3's 'at least one PPT configuration achieves 80% accuracy' rule, but 'the subset without a clean emergence point.' Both definitions are outcome-dependent, and they identify different sets unless the two criteria happen to coincide, which the paper does not show. Because the abstract's central 80%-accuracy/36B-token result is reported for 'the 17 elemental tasks' without a stable, pre-specified selection rule, the headline aggregate is attached to a moving target. This compounds the selection-on-outcome circularity: the main quantitative claims are not estimates from a fixed evaluation set independent of the models being compared.

full rationale

The core comparison is run against external benchmarks (BLiMP and the ElementalTask suite), and the logic pre-pretraining data are generated from an independent rule library (247 schemata) before any language evaluation; the pipeline therefore has substantial independent content. The mechanistic and pruning claims are also empirical measurements (CKA, RankMe, spectral decay, Wanda pruning), not quantities fitted to the conclusion. There are self-citations (e.g., Yamaguchi et al. 2026 for architecture setup), but these are not load-bearing for the central behavioral claim. The circularity that does exist is the outcome-dependent selection of the 17-task subset: the 80% threshold used to retain tasks is the same 80% threshold reported as the milestone, and Appendix K offers yet another outcome-based definition of the same subset. The paper partially mitigates this by plotting all 130-task trajectories, but it never reports a full-suite summary mean or token-to-threshold in the main text, so the headline numbers remain conditional on the selection rule. The single-seed limitation is a statistical concern, not a circular one. Overall, the central '80% with 36B fewer tokens' result is partially reduced by the selection criterion, while the underlying effect and the compression results still have independent evidence; hence a moderate score of 5.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No parameters are fitted to the target evaluation, but the post hoc 80% task-inclusion threshold is a selection choice that shapes the headline. The transfer of backbone weights, the validity of generated derivations, and the causal interpretation of geometry-to-pruning are the key assumptions.

free parameters (4)
  • Task inclusion threshold for main evaluation subset = 0.80
    Main-text mean accuracy is computed on 17/130 elemental tasks kept because at least one PPT configuration reached 80% during pretraining (Section 4.2). This post hoc selection affects the reported 36B-token and final-accuracy deltas.
  • Maximum backward-expansion depth L = 10
    Appendix D; manually chosen cap on derivation-tree depth, shapes the difficulty and length of generated logic data.
  • Number of symbol-renaming variants per tree = 5
    Appendix D; set to increase data diversity, not fitted to downstream performance.
  • Symbolic pre-pretraining token budget = 2.6B tokens for Logic-PPT
    Section 3; approximately matched across conditions, but not optimized against the target skill-acquisition metric.
assumptions (5)
  • standard math Backward chaining from a goal using sound schemata yields a valid forward derivation via post-order traversal.
    Section 2.4 and Appendix D; the next-step prediction targets are guaranteed derivable by construction.
  • domain assumption Structural inductive biases learned on character-level logic transfer through Transformer backbone weights after embedding and head reinitialization.
    Section 2.3; this is the core transfer premise of Logic-PPT, asserted rather than proven.
  • domain assumption FineWeb-Edu, BLiMP, and the elemental task suite measure natural language acquisition relevant to the claim.
    Sections 3 and 4; benchmarks assumed to operationalize skill acquisition.
  • ad hoc to paper Lower RankMe, higher spectral decay, and clearer CKA blocks cause pruning robustness rather than merely correlating with it.
    Section 6; the causal framing is not isolated by an intervention that changes geometry while holding other factors fixed.
  • domain assumption Wanda importance scores correctly identify removable weights in this regime.
    Appendix B; prior work is cited, and the metric is assumed valid for these models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility." pith.science (2026). https://pith.science/paper/QG7C3XJ4

@misc{pith2026260803930,
  author       = {Pith},
  title        = {Pith review of: Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QG7C3XJ4}},
  note         = {Machine review of arXiv:2608.03930}
}
abstract

Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural language. Moreover, prior studies remain restricted to relatively small token budgets, offering limited insight into skill emergence and representational dynamics. To address these limitations, we propose logic pre-pretraining (Logic-PPT) as a principled initialization strategy, leveraging formal derivations to impart richer structural and linguistic biases. Formal derivations require abstract mechanisms that are central to natural language, simultaneously binding variables, connecting quantifiers and relational dependencies, and composing predicate-argument structures over long contexts. Scaling our evaluation to a 100B-token regime, logic pre-pretraining substantially accelerates skill acquisition in LMs, achieving 80\% accuracy on linguistic tasks with 36B fewer tokens than standard initialization, and outperforming alternative pre-pretraining baselines. Mechanistically, formal derivations induce persistent structural reorganization, distinctively characterized by a lower-rank, spectrally concentrated representation space. Crucially, we show that this internal geometry enables improved model compressibility via pruning, matching the dense baseline performance even at $\approx$33\% sparsity.

Figures

Figures reproduced from arXiv: 2608.03930 by the authors.

Figure 1
Figure 1. (A) Faster learning: Logic-PPT reaches 0.8 mean accuracy using 36B fewer natural language train￾ing tokens than the PT baseline in the 100B-token run. (B) Lower-rank representations: Logic-PPT promotes a more structured and compact internal geometry, orga￾nizing representations around a smaller set of directions. (C) Greater compression robustness: At ≈33% spar￾sity, the Logic-PPT model retains accuracy comparable t… view at source ↗
Figure 2
Figure 2. (Left) We introduce Logic Pre-pretraining (Logic-PPT), a new framework for initializing LMs on symbolic data before exposure to natural language. Compared to previous approaches relying on formal languages and procedural algorithms, Logic-PPT adopts more expressive and structurally richer formal derivations to impart deeper linguistic inductive biases. (Right) Overview of the methodology to generate synthetic formal… view at source ↗
Figure 3
Figure 3. Behavioral trajectories over the 100B-token natural language pretraining run. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Layer-to-layer CKA between baseline PT and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 4
Figure 4. Figure 4: Self layer-to-layer CKA (each model com [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 7
Figure 7. Figure 7: Stable rank trajectories of attention weight [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: Layer-group trajectories of RankMe and the [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Mean accuracy under Wanda pruning across [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Accuracy trajectories for the 17 elemental tasks across language training tokens. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Accuracy trajectories for the first subset of tasks in the full 130-task ElementalTask suite. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Accuracy trajectories for the second subset of tasks in the full 130-task ElementalTask suite. [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Accuracy trajectories for the third subset of tasks in the full 130-task ElementalTask suite. [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Accuracy trajectories for the fourth subset of tasks in the full 130-task ElementalTask suite. [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Accuracy trajectories for the fifth subset of tasks in the full 130-task ElementalTask suite. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: BLiMP accuracy trajectories by linguistic field. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 40 canonical work pages

  1. [1]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. https://arxiv.org/abs/2412.08905 Phi-4 technic...

    arXiv 2024

  2. [2]

    Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/70596d70542c51c8d9b4e423f4bf2736-Paper-Conference.pdf alpha-req : Assessing representation quality in self-supervised learning by measuring eigenspectrum decay . In Advances in Neural Information Processing Systems, volume ...

    2022

  3. [3]

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International C...

    2023

  4. [6]

    Zheng Chen, Chuan Zhou, Fengxiang Cheng, Yip Tin Po, Fenrong Liu, Yisen Wang, Jiajun Chai, Xiaohan Wang, Guojun Yin, Wei Lin, Bo Li, Haoxuan Li, and Zhouchen Lin. 2026. https://openreview.net/forum?id=ULEHJkolxB Logiconbench: Benchmarking logical consistencies of LLM s . In The Fourteenth International Conference on Learning Representations

    2026

  5. [8]

    Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. 2023. https://openreview.net/forum?id=uGEBxC8dnEh Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank

    2023

  6. [9]

    Xu Guo, Runyu Peng, Jian Tong, Yunhua Zhou, Haijun Lv, Zhihui Lu, and Qipeng Guo. 2026. https://arxiv.org/abs/2605.10129 Synthetic pre-pre-training improves language model robustness to noisy pre-training data . Preprint, arXiv:2605.10129

    work page Pith review arXiv 2026

  7. [10]

    Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. 2026. https://arxiv.org/abs/2607.15495 Verbalizable representations form a global workspace in language models . Pre...

    work page Pith review arXiv 2026

  8. [11]

    Gary M Hardegree. 1994. Symbolic logic: A first course. McGraw-Hill

    1994

Show all 87 references
  1. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  2. [15]

    Liangze Jiang, Zachary Shinnick, Anton van den Hengel, Hemanth Saratchandran, and Damien Teney. 2026. https://openreview.net/forum?id=XFTTezxLdU Procedural pretraining: Warming up language models with abstract data . In Forty-third International Conference on Machine Learning

    2026

  3. [17]

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519--3529. PMlR

    2019

  4. [18]

    Atharva Kulkarni, Jacob Mitchell Springer, Arjun Subramonian, and Swabha Swayamdipta. 2026. https://openreview.net/forum?id=kNb6NNhKgE Disentangling geometry, performance, and training in language models . In Forty-third International Conference on Machine Learning

    2026

  5. [19]

    Valentin Lacombe, Valentin Quesnel, and Damien Sileo. 2026. https://arxiv.org/abs/2603.02208 Reasoning core: A scalable procedural data generation suite for symbolic pre-training and post-training . Preprint, arXiv:2603.02208

    2026

  6. [20]

    Dan Lee, Seungwook Han, Akarsh Kumar, and Pulkit Agrawal. 2026 a . https://arxiv.org/abs/2603.10055 Training language models via neural cellular automata . Preprint, arXiv:2603.10055

    2026

  7. [22]

    Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Adam Santoro, Guillaume Lajoie, and Blake Aaron Richards. 2025. https://openreview.net/forum?id=FDruZlKWUb Tracing the representation geometry of language models from pretraining to post-training . In The T...

    2025

  8. [23]

    Emmy Liu, Kaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja, Jen tse Huang, and Graham Neubig. 2026. https://arxiv.org/abs/2604.08510 What do language models learn and when? the implicit curriculum hypothesis . Preprint, arXiv:2604.08510

    2026 arXiv

  9. [24]

    Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, Mozhi Zhang, Pengyu Zhao, and Junxian He. 2025. https://openreview.net/forum?id=XtNiw8OQsy Synlogic: Synthesizing verifiable reasoning data at scale for learn...

    2025

  10. [28]

    Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. 2024. https://openreview.net/forum?id=mljDUaQpln Enhancing reasoning capabilities of LLM s via principled synthetic logic corpus . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  11. [29]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, an...

    2024 arXiv

  12. [31]

    Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Damien Teney, and Anton van den Hengel. 2026. Can you learn to see without images? procedural warm-up for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p...

    2026

  13. [32]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    2023 arXiv

  14. [34]

    Qwen Team. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    2025 arXiv

  15. [35]

    Zehao Wang, Lin Yang, Jie Wang, Kehan Wang, Hanzhu Chen, Bin Wang, Jianye HAO, Defu Lian, Bin Li, and Enhong Chen. 2025. https://openreview.net/forum?id=z4AMrCOetn Logictree: Improving complex reasoning of LLM s via instantiated multi-step synthetic logical data . In The Thirt...

    2025

  16. [36]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. https://doi.org/10.1162/tacl\_a\_00321 Blimp: The benchmark of linguistic minimal pairs for english . Transactions of the Association for Computational Linguistics,...

    2020 doi

  17. [38]

    Yuhuai Wu, Felix Li, and Percy S Liang. 2022. Insights into pre-training via simpler synthetic tasks. Advances in Neural Information Processing Systems, 35:21844--21857

    2022

  18. [39]

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. 2025. https://arxiv.org/abs/2502.14768 Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning . Preprint, arXiv:2502.14768

    2025 arXiv

  19. [41]

    Junjie Yu, Yue Wang, Zihan Deng, Yan Zhu, Wenxiao Ma, and Quanying Liu. 2026. https://arxiv.org/abs/2605.07302 Pretraining induces a reusable spectral basis for downstream task adaptation . Preprint, arXiv:2605.07302

    2026 arXiv

  20. [42]

    David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, and Matthew R. Walter. 2024. https://arxiv.org/abs/2408.11804 Approaching deep learning through the spectral dynamics of weights . Preprint, arXiv:2408.11804

    2024 arXiv

  21. [43]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  22. [44]

    Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models

    Cao, Mingzi and Tan, Xingwei and Akhter, Mahmud Elahi and Valentino, Marco and Liakata, Maria and Wang, Xi and Aletras, Nikolaos. Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models. Findings of the A ssociation for C omputational L inguistic...

    2026 doi

  23. [45]

    FOL -Traces: Verified First-Order Logic Reasoning Traces at Scale

    Lee, Isabelle and Liaw, Sarah and Yogatama, Dani. FOL -Traces: Verified First-Order Logic Reasoning Traces at Scale. Findings of the A ssociation for C omputational L inguistics: EACL 2026. 2026. doi:10.18653/v1/2026.findings-eacl.115

    2026 doi

  24. [46]

    Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference

    Kim, Geonhee and Valentino, Marco and Freitas, Andre. Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.525

    2025 doi

  25. [47]

    Enhancing Logical Reasoning in Language Models via Symbolically-Guided M onte C arlo Process Supervision

    Tan, Xingwei and Valentino, Marco and Akhter, Mahmud Elahi and Liakata, Maria and Aletras, Nikolaos. Enhancing Logical Reasoning in Language Models via Symbolically-Guided M onte C arlo Process Supervision. Proceedings of the 2025 Conference on Empirical Methods in Natural Lan...

    2025 doi

  26. [48]

    Advances in Neural Information Processing Systems , volume=

    Insights into pre-training via simpler synthetic tasks , author=. Advances in Neural Information Processing Systems , volume=

  27. [49]

    Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers

    Phang, Jason and Liu, Haokun and Bowman, Samuel R. Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers. Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. 2021. doi:10.18653/v1/2021.blackboxnlp-1.42

    2021 doi

  28. [50]

    A Structural Probe for Finding Syntax in Word Representations

    Hewitt, John and Manning, Christopher D. A Structural Probe for Finding Syntax in Word Representations. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pap...

    2019 doi

  29. [51]

    The Proper Treatment of Quntification in Ordinary English , booktitle =

    Montague, Richard , publisher =. The Proper Treatment of Quntification in Ordinary English , booktitle =. doi:https://doi.org/10.1002/9780470758335.ch1 , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470758335.ch1 , year =

    doi

  30. [52]

    What do RNN Language Models Learn about Filler -- Gap Dependencies?

    Wilcox, Ethan and Levy, Roger and Morita, Takashi and Futrell, Richard. What do RNN Language Models Learn about Filler -- Gap Dependencies?. Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing and Interpreting Neural Networks for NLP. 2018. doi:10.18653/v1/W18-5423

    2018 doi

  31. [53]

    and McGuinness, Brian

    Wittgenstein, Ludwig and Pears, David Francis. and McGuinness, Brian. , address =. Tractatus logico philosophicus , year =

  32. [54]

    2001 , publisher=

    A mathematical introduction to logic , author=. 2001 , publisher=

    2001

  33. [55]

    1994 , publisher=

    Symbolic logic: A first course , author=. 1994 , publisher=

    1994

  34. [56]

    Logical Syntax and Semantics: Their Linguistic Relevance , volume =

    Noam Chomsky , doi =. Logical Syntax and Semantics: Their Linguistic Relevance , volume =. Journal of Symbolic Logic , number =

  35. [57]

    Theoria , volume =

    Montague, Richard , title =. Theoria , volume =. doi:https://doi.org/10.1111/j.1755-2567.1970.tb00434.x , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1755-2567.1970.tb00434.x , year =

    1970

  36. [58]

    , title =

    Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei and Wang, Sheng-Fu and Bowman, Samuel R. , title =. Transactions of the Association for Computational Linguistics , volume =. 2020 , doi =. https://doi.org/10.1162/tacl_a_00321 , abstract =

    2020 doi

  37. [59]

    2025 , eprint=

    Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking , author=. 2025 , eprint=

    2025

  38. [60]

    P roof W riter: Generating Implications, Proofs, and Abductive Statements over Natural Language

    Tafjord, Oyvind and Dalvi, Bhavana and Clark, Peter. P roof W riter: Generating Implications, Proofs, and Abductive Statements over Natural Language. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021. doi:10.18653/v1/2021.findings-acl.317

    2021 doi

  39. [61]

    Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

    Blevins, Terra and Gonen, Hila and Zettlemoyer, Luke. Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.234

    2022 doi

  40. [62]

    2026 , eprint=

    Verbalizable Representations Form a Global Workspace in Language Models , author=. 2026 , eprint=

    2026

  41. [63]

    Understanding the Inner-workings of Language Models Through Representation Dissimilarity

    Brown, Davis and Godfrey, Charles and Konz, Nicholas and Tu, Jonathan and Kvinge, Henry. Understanding the Inner-workings of Language Models Through Representation Dissimilarity. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:...

    2023 doi

  42. [64]

    alpha-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay , url =

    Agrawal, Kumar K and Mondal, Arnab Kumar and Ghosh, Arna and Richards, Blake , booktitle =. alpha-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay , url =

  43. [65]

    2023 , url=

    RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank , author=. 2023 , url=

    2023

  44. [66]

    International conference on machine learning , pages=

    Similarity of neural network representations revisited , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  45. [67]

    2023 , eprint=

    Mistral 7B , author=. 2023 , eprint=

    2023

  46. [68]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=. 2024 , url=

    2024

  47. [69]

    Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

    Yamaguchi, Atsuki and Mi, Maggie and Aletras, Nikolaos. Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 2: Short Papers). 20...

    2026 doi

  48. [70]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  49. [71]

    doi:10.57967/hf/2497 , publisher =

    Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas , title =. doi:10.57967/hf/2497 , publisher =

    doi

  50. [72]

    Language acquisition: do children and language models follow similar learning stages?

    Evanson, Linnea and Lakretz, Yair and King, Jean R \'e mi. Language acquisition: do children and language models follow similar learning stages?. Findings of the Association for Computational Linguistics: ACL 2023. 2023. doi:10.18653/v1/2023.findings-acl.773

    2023 doi

  51. [73]

    Enhancing Reasoning Capabilities of

    Terufumi Morishita and Gaku Morio and Atsuki Yamaguchi and Yasuhiro Sogawa , booktitle=. Enhancing Reasoning Capabilities of. 2024 , url=

    2024

  52. [74]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  53. [75]

    2025 , eprint=

    Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning , author=. 2025 , eprint=

    2025

  54. [76]

    2024 , eprint=

    Phi-4 Technical Report , author=. 2024 , eprint=

    2024

  55. [77]

    2026 , eprint=

    Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Training , author=. 2026 , eprint=

    2026

  56. [78]

    2026 , eprint=

    Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data , author=. 2026 , eprint=

    2026

  57. [79]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Shinnick, Zachary and Jiang, Liangze and Saratchandran, Hemanth and Teney, Damien and van den Hengel, Anton , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2026 , pages =

    2026

  58. [80]

    2026 , eprint=

    Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation , author=. 2026 , eprint=

    2026

  59. [81]

    2024 , eprint=

    2 OLMo 2 Furious , author=. 2024 , eprint=

    2024

  60. [82]

    2024 , eprint=

    Approaching Deep Learning through the Spectral Dynamics of Weights , author=. 2024 , eprint=

    2024

  61. [83]

    2026 , eprint=

    What do Language Models Learn and When? The Implicit Curriculum Hypothesis , author=. 2026 , eprint=

    2026

  62. [84]

    Forty-third International Conference on Machine Learning , year=

    Disentangling Geometry, Performance, and Training in Language Models , author=. Forty-third International Conference on Machine Learning , year=

  63. [85]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Tracing the Representation Geometry of Language Models from Pretraining to Post-training , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  64. [86]

    2023 , journal=

    A Simple and Effective Pruning Approach for Large Language Models , author=. 2023 , journal=

    2023

  65. [87]

    LogiConBench: Benchmarking Logical Consistencies of

    Zheng Chen and Chuan Zhou and Fengxiang Cheng and Yip Tin Po and Fenrong Liu and Yisen Wang and Jiajun Chai and Xiaohan Wang and Guojun Yin and Wei Lin and Bo Li and Haoxuan Li and Zhouchen Lin , booktitle=. LogiConBench: Benchmarking Logical Consistencies of. 2026 , url=

    2026

  66. [88]

    LogicTree: Improving Complex Reasoning of

    Zehao Wang and Lin Yang and Jie Wang and Kehan Wang and Hanzhu Chen and Bin Wang and Jianye HAO and Defu Lian and Bin Li and Enhong Chen , booktitle=. LogicTree: Improving Complex Reasoning of. 2025 , url=

    2025

  67. [89]

    Proceedings of the 40th International Conference on Machine Learning , articleno =

    Biderman, Stella and Schoelkopf, Hailey and Anthony, Quentin and Bradley, Herbie and O'Brien, Kyle and Hallahan, Eric and Khan, Mohammad Aflah and Purohit, Shivanshu and Prashanth, USVSN Sai and Raff, Edward and Skowron, Aviya and Sutawika, Lintang and Van Der Wal, Oskar , tit...

    2023

  68. [90]

    2026 , eprint=

    Training Language Models via Neural Cellular Automata , author=. 2026 , eprint=

    2026

  69. [91]

    Language Acquisition Device in Large Language Models

    Mita, Masato and Someya, Taiga and Yoshida, Ryo and Oseki, Yohei. Language Acquisition Device in Large Language Models. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). 2026. doi:10.18653/v1/2026.acl-long.895

    2026 doi

  70. [92]

    and Petty, Jackson and Shi, Chuan and Merrill, William and Linzen, Tal

    Hu, Michael Y. and Petty, Jackson and Shi, Chuan and Merrill, William and Linzen, Tal. Between Circuits and C homsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume ...

    2025 doi

  71. [93]

    Forty-third International Conference on Machine Learning , year=

    Procedural Pretraining: Warming Up Language Models with Abstract Data , author=. Forty-third International Conference on Machine Learning , year=

  72. [94]

    The Fourteenth International Conference on Learning Representations , year=

    Conditioned Initialization for Attention , author=. The Fourteenth International Conference on Learning Representations , year=

  73. [95]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  74. [96]

    Publications Manual , year = "1983", publisher =

    1983

  75. [97]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  76. [98]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  77. [99]

    Dan Gusfield , title =. 1997

    1997

  78. [100]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  79. [101]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.