pith. machine review for the scientific record. sign in

arxiv: 2605.09949 · v1 · submitted 2026-05-11 · 💻 cs.LG

Recognition: 2 theorem links

· Lean Theorem

From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models

Hiroyuki Kusuhara, Shumpei Nemoto, Tadahaya Mizuno, Yasuhiro Yoshikai, Zehao Li

Pith reviewed 2026-05-12 03:58 UTC · model grok-4.3

classification 💻 cs.LG
keywords chiralitySMILES translationchemical language modelssemantic emergencetransformer modelslatent space analysisattention headsencoder-decoder
0
0 comments X

The pith

Chemical translation models show a sudden jump in chiral accuracy after a long plateau, driven by encoder-side reorganization of representations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how transformer models that translate SMILES strings acquire the ability to distinguish molecular chirality, a property that determines how molecules interact with biological systems yet often eludes current chemical language models. By examining model checkpoints at fine time intervals during training, the authors document that accuracy on tokens marking chiral centers stays low for many steps and then rises sharply, and this pattern holds across model variants of different sizes. The transition aligns with a temporary drop and recovery in the norm of internal vectors in the encoder along with a shift in how chiral molecules are arranged in the model's latent space. These observations lead the authors to conclude that the difficulty in learning chirality stems from the structure of the constraints themselves rather than from limits in model scale or data volume.

Core claim

In autoregressive encoder-decoder models for SMILES translation, chiral-token accuracy exhibits a reproducible abrupt increase after an extended plateau phase. This jump coincides with a V-shaped pattern of decreasing then recovering vector norms and directional stability in the residual stream, accompanied by a clear reorganization of chiral molecular representations in latent space. Encoder-decoder cross evaluations and ablation of specific attention heads confirm that the transition is localized to the encoder and involves a small subset of heads that become selectively sensitive to chiral features.

What carries the argument

The encoder-centered mechanism of chiral emergence, marked by transient destabilization and reconstruction of residual-stream vectors together with reorganization of chiral representations in latent space.

If this is right

  • Chiral learning plateaus arise from the intrinsic complexity of chiral constraints rather than insufficient model capacity.
  • A small number of attention heads can be isolated whose removal selectively lowers chiral-token accuracy even after full training.
  • SMILES translation supplies a controlled experimental system for studying how semantic features emerge in chemical language models.
  • High-temporal-resolution checkpoint tracking exposes dynamic representation changes that endpoint evaluations alone would miss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Vector-norm monitoring during training could flag the moment when complex semantic features such as chirality are being internalized.
  • The encoder-focused transition pattern may appear for other stereochemical or conformational properties beyond chirality.
  • Architecture choices that strengthen encoder capacity or stabilize residual streams might shorten the plateau phase for difficult chemical properties.

Load-bearing premise

The abrupt accuracy jump, V-shaped norm changes, and latent-space reorganization reflect genuine acquisition of chiral semantics rather than training artifacts, data imbalances, or choices in which checkpoints and heads are examined.

What would settle it

Training the same models on SMILES strings where all chiral center labels have been randomly reassigned or removed entirely would produce the same jump and norm trajectory if the observed dynamics are unrelated to actual chirality.

Figures

Figures reproduced from arXiv: 2605.09949 by Hiroyuki Kusuhara, Shumpei Nemoto, Tadahaya Mizuno, Yasuhiro Yoshikai, Zehao Li.

Figure 1
Figure 1. Figure 1: Overview of the Pan-CORE architecture. (a) Overall encoder-decoder structure. The encoder processes input [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training progress (a) pancore-baseline, (b) pancore-addonce, (c) pancore-xattn-adalnzero. The top row shows the per-token cross-entropy loss for both training and evaluation data across each sequence-length bucket, while the bottom row shows the accuracy for each bucket. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Token accuracy progress of (a) ZINC20 (∼100) and (b) PubChem (∼100) for pancore-addonce. Highlighted lines indicate stereochemistry tokens; chirality tokens (@ and @@), geometric isomerism tokens (/ and \). 21 [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Trajectories of logits-based metrics for [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Trajectories of the chiral tokens’ attention weight mass across heads from L0H0 (layer 0, head 0) to L7H7, [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Trajectories of residual stream–related metrics from L0 (layer 0) to L7, for [PITH_FULL_IMAGE:figures/full_fig_p024_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Trajectories of latent vector–related metrics for [PITH_FULL_IMAGE:figures/full_fig_p025_7.png] view at source ↗
read the original abstract

Understanding how chemical language models (CLMs) learn chemical meaning from molecular string representations, rather than only surface-level string patterns, is an important question in chemical representation learning and machine learning for chemistry. Chirality provides a demanding test case: enantiomers can differ greatly in pharmacological activity and toxicity, yet CLMs often struggle to distinguish chiral configurations reliably. Here we present Pan-CORE (Pan-Chemical Omniscale Representation Engine), a family of autoregressive Transformer-based encoder-decoder models for SMILES translation, and use high-temporal-resolution checkpoint analysis to investigate how chiral information is learned during training. Across all tested Pan-CORE variants, we observe a reproducible jump-up in which chiral-token accuracy rises abruptly after a long plateau, suggesting that chiral learning stagnation is not explained by model capacity alone and instead reflects the complexity of chiral constraints. Analyses of attention dynamics, residual-stream trajectories, and latent-space geometry support an encoder-centered mechanism in which chiral-token representations undergo transient destabilization and reconstruction, seen as a V-shaped drop and recovery in vector norm and directional stability, together with a clear reorganization of chiral molecular representations in the latent space. Encoder-decoder cross-evaluation further supports the encoder-centered nature of the transition, and targeted attention-head ablation identifies a small set of chiral-sensitive heads whose removal selectively reduces chiral-token accuracy even in the fully trained model. These findings show that SMILES translation can serve as a useful experimental system for mechanistic analysis of semantic emergence in CLMs, with implications for interpretable chemical representation learning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces the Pan-CORE family of autoregressive Transformer encoder-decoder models for SMILES translation and uses high-temporal-resolution checkpoint analysis to demonstrate that chiral-token accuracy exhibits a reproducible abrupt jump after a long plateau across model variants. It interprets this as evidence that chiral learning stagnation reflects the complexity of chiral constraints rather than insufficient model capacity, supported by analyses of attention dynamics, V-shaped residual norm trajectories, latent-space reorganization, encoder-decoder cross-evaluations, and targeted attention-head ablations that identify a small set of chiral-sensitive heads.

Significance. If the transition and supporting dynamics are shown to be specific to chirality, the work would provide a useful mechanistic case study for semantic emergence in chemical language models, with the high-resolution checkpointing, residual-stream analysis, and head-ablation results offering a concrete template for interpretability research in CLMs. The reproducibility across Pan-CORE variants and the identification of encoder-centered mechanisms are particular strengths that could inform more reliable handling of stereochemistry in downstream applications.

major comments (3)
  1. [Results on training dynamics and ablation studies] The central claim that the abrupt jump in chiral-token accuracy demonstrates chirality-specific constraint complexity (rather than a generic training artifact) is load-bearing for the interpretation of semantic emergence. However, the training-dynamics and ablation sections do not report accuracy trajectories for non-chiral tokens or include control experiments on datasets with randomized chirality labels, so the plateau-jump pattern, V-shaped norm drop, and latent reorganization cannot yet be distinguished from general autoregressive SMILES phase transitions.
  2. [Analyses of residual-stream trajectories and latent-space geometry] The encoder-centered mechanism is supported by cross-evaluation and head-ablation results, but the manuscript provides no quantitative baseline (e.g., norm dynamics or latent geometry) for non-chiral molecular representations or shuffled-chirality controls. This omission leaves the V-shaped vector-norm behavior and latent-space reorganization open to alternative explanations as post-hoc observations of general training phenomena.
  3. [Methods and experimental setup] The experimental protocol lacks sufficient detail on data splits, exact definition and computation of chiral-token accuracy, checkpoint sampling intervals, and statistical controls for the reported reproducibility of the jump. These omissions directly affect the ability to assess whether the observed transition is robust or sensitive to unexamined confounds.
minor comments (2)
  1. [Abstract and §1] The abstract and introduction would benefit from an explicit definition of 'chiral-token accuracy' and 'chiral-sensitive heads' before the results are presented.
  2. [Figures] Figure captions should specify the exact Pan-CORE variants, number of runs, and checkpoint resolution used for each panel to improve reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their positive assessment of the work's potential significance and for the constructive major comments, which help clarify the evidence needed to support chirality-specific emergence. We address each point below and have prepared revisions that add the requested controls, baselines, and methodological details.

read point-by-point responses

  1. Referee: [Results on training dynamics and ablation studies] The central claim that the abrupt jump in chiral-token accuracy demonstrates chirality-specific constraint complexity (rather than a generic training artifact) is load-bearing for the interpretation of semantic emergence. However, the training-dynamics and ablation sections do not report accuracy trajectories for non-chiral tokens or include control experiments on datasets with randomized chirality labels, so the plateau-jump pattern, V-shaped norm drop, and latent reorganization cannot yet be distinguished from general autoregressive SMILES phase transitions.

    Authors: We agree that direct comparisons to non-chiral tokens and randomized-chirality controls are necessary to strengthen the claim of specificity. In the revised manuscript we have added accuracy trajectories for non-chiral tokens (e.g., atom and bond tokens without stereodescriptors), which improve monotonically without an abrupt jump. We have also included results from shuffled-chirality control datasets in a new supplementary figure; these show the absence of both the plateau-jump and the V-shaped residual-norm signature, supporting that the observed transition reflects the complexity of chiral constraints rather than a generic autoregressive phase change. revision: yes

  2. Referee: [Analyses of residual-stream trajectories and latent-space geometry] The encoder-centered mechanism is supported by cross-evaluation and head-ablation results, but the manuscript provides no quantitative baseline (e.g., norm dynamics or latent geometry) for non-chiral molecular representations or shuffled-chirality controls. This omission leaves the V-shaped vector-norm behavior and latent-space reorganization open to alternative explanations as post-hoc observations of general training phenomena.

    Authors: We acknowledge the value of explicit quantitative baselines. The revised manuscript now reports residual-norm trajectories and latent-space geometry metrics for non-chiral molecular representations, which lack the V-shaped drop and directional reorganization. Parallel analyses on shuffled-chirality controls are included and demonstrate that the encoder-centered reorganization is absent when chirality labels are randomized. These additions, placed alongside the original cross-evaluation and ablation results, reduce the plausibility of purely generic training explanations. revision: yes

  3. Referee: [Methods and experimental setup] The experimental protocol lacks sufficient detail on data splits, exact definition and computation of chiral-token accuracy, checkpoint sampling intervals, and statistical controls for the reported reproducibility of the jump. These omissions directly affect the ability to assess whether the observed transition is robust or sensitive to unexamined confounds.

    Authors: We have expanded the Methods section with the requested information. Data splits are now described as an 80/10/10 train/validation/test partition on a curated set of 500k SMILES strings containing explicit stereochemistry, generated with RDKit canonicalization. Chiral-token accuracy is defined as the fraction of correctly predicted '@' and '@@' tokens across all chiral centers in the decoded SMILES, computed token-wise on the test set. Checkpoints were sampled every 500 steps during the plateau and every 50 steps in the transition window; reproducibility was assessed across five independent runs with distinct random seeds, confirming the jump occurs within a consistent 200-step window. These details have been added to the main text and supplementary material. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical training dynamics analysis

full rationale

The paper presents no derivation chain, equations, or theoretical claims that reduce to inputs by construction. All load-bearing results (chiral-token accuracy jumps, V-shaped norm dynamics, latent reorganization, head ablations) are direct measurements from model training runs and checkpoint analyses on Pan-CORE variants. These are falsifiable experimental observations rather than self-definitional, fitted-input predictions, or self-citation-dependent uniqueness arguments. The study is self-contained against external benchmarks via ablation controls and cross-evaluations.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work relies on standard transformer training assumptions and observational analysis; no free parameters, ad-hoc axioms, or new invented entities are introduced.

axioms (1)
  • standard math Standard properties of autoregressive transformer attention and residual streams hold during training
    Analyses of attention dynamics and vector norms presuppose conventional neural network behavior.

pith-pipeline@v0.9.0 · 5593 in / 1141 out tokens · 45630 ms · 2026-05-12T03:58:15.548347+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • IndisputableMonolith/Foundation/RealityFromDistinction reality_from_one_distinction unclear

    Across all tested Pan-CORE variants, we observe a reproducible jump-up in which chiral-token accuracy rises abruptly after a long plateau... Analyses of attention dynamics, residual-stream trajectories, and latent-space geometry support an encoder-centered mechanism in which chiral-token representations undergo transient destabilization and reconstruction, seen as a V-shaped drop and recovery in vector norm

  • IndisputableMonolith/Cost/FunctionalEquation washburn_uniqueness_aczel unclear

    We identify the jump-up as the abrupt learning of SMILES-level chiral constraints, characterized by a sudden rise in chiral-token accuracy and a sharp improvement in model confidence over R/S configurations.

Reference graph

Works this paper leans on

58 extracted references · 58 canonical work pages · 12 internal anchors

  1. [1]

    A comprehensive overview of large language models.ACM Trans

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models.ACM Trans. Intell. Syst. Technol., 16(5):1–72, 2025. ISSN 2157-6904,2157-6912. doi:10.1145/3744746

    work page doi:10.1145/3744746 2025

  2. [2]

    A systematic review of deep learning chemical language models in recent era.J

    Hector Flores-Hernandez and Emmanuel Martinez-Ledesma. A systematic review of deep learning chemical language models in recent era.J. Cheminform., 16(1):129, 2024. ISSN 1758-2946,1758-2946. doi:10.1186/s13321- 024-00916-y

    work page doi:10.1186/s13321- 2024

  3. [3]

    Protein large language models: A comprehensive survey

    Yijia Xiao, Wanjia Zhao, Junkai Zhang, Yiqiao Jin, Han Zhang, Zhicheng Ren, Renliang Sun, Haixin Wang, Guancheng Wan, Pan Lu, Xiao Luo, Yu Zhang, James Zou, Yizhou Sun, and Wei Wang. Protein large language models: A comprehensive survey. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 23080–23103, Stroudsburg, PA, USA, 2025....

    work page doi:10.18653/v1/2025.findings-emnlp.1255 2025

  4. [4]

    A comprehensive survey of genome language models in bioinformatics.Brief

    Liyuan Shu, Jiao Tang, Xiaoyu Guan, and Daoqiang Zhang. A comprehensive survey of genome language models in bioinformatics.Brief. Bioinform., 27(1):bbaf724, 2026. ISSN 1467-5463,1477-4054. doi:10.1093/bib/bbaf724

    work page doi:10.1093/bib/bbaf724 2026

  5. [5]

    Smiles, a chemical language and information system

    David Weininger. SMILES, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules.J. Chem. Inf. Comput. Sci., 28(1):31–36, 1988. ISSN 0095-2338,1520-5142. doi:10.1021/ci00057a005

    work page doi:10.1021/ci00057a005 1988

  6. [6]

    Automatic chemical design using a data-driven continuous representation of molecules.ACS Cent

    Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez- Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru- Guzik. Automatic chemical design using a data-driven continuous representation of molecules.ACS Cent. Sci., 4 (2):268–276, 2018. ISSN 2374-7943,23...

    work page doi:10.1021/acscentsci.7b00572 2018

  7. [7]

    Beyond performance: how design choices shape chemical language models.J

    Inken Fender, Jannik Adrian Gut, and Thomas Lemmin. Beyond performance: how design choices shape chemical language models.J. Cheminform., 17(1):173, 2025. ISSN 1758-2946,1758-2946. doi:10.1186/s13321-025-01099- w

    work page doi:10.1186/s13321-025-01099- 2025

  8. [8]

    Measuring chemical LLM robustness to molecular representations: a SMILES variation-based framework.J

    Veronika Ganeeva, Kuzma Khrabrov, Artur Kadurin, and Elena Tutubalina. Measuring chemical LLM robustness to molecular representations: a SMILES variation-based framework.J. Cheminform., 17(1):164, 2025. ISSN 1758-2946,1758-2946. doi:10.1186/s13321-025-01079-0

    work page doi:10.1186/s13321-025-01079-0 2025

  9. [9]

    Graph neural networks in particle physics

    Mario Krenn, Florian Häse, Akshatkumar Nigam, Pascal Friederich, and Alán Aspuru-Guzik. Self-referencing em- bedded strings (SELFIES): A 100% robust molecular string representation.arXiv [cs.LG], 2019. doi:10.1088/2632- 2153/aba947

    work page doi:10.1088/2632- 2019

  10. [10]

    Attention Is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.arXiv [cs.CL], 2017. doi:10.48550/arXiv.1706.03762

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1706.03762 2017

  11. [11]

    Chemberta: Large-scale self- supervised pretraining for molecular property prediction, 2020

    Seyone Chithrananda, Gabe Grand, and Bharath Ramsundar. ChemBERTa: Large-scale self-supervised pretraining for molecular property prediction.arXiv [cs.LG], 2020. doi:10.48550/arXiv.2010.09885. 10 From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models

    work page doi:10.48550/arxiv.2010.09885 2020

  12. [12]

    Large-scale chemical language representations capture molecular structure and properties.Nat

    Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties.Nat. Mach. Intell., 4(12):1256–1264,

    work page

  13. [13]

    doi:10.1038/s42256-022-00580-7

    ISSN 2522-5839,2522-5839. doi:10.1038/s42256-022-00580-7

    work page doi:10.1038/s42256-022-00580-7

  14. [14]

    An open-source family of large encoder-decoder foundation models for chemistry.Commun

    Eduardo Soares, Emilio Vital Brazil, Victor Shirasuna, Dmitry Zubarev, Renato Cerqueira, and Kristin Schmidt. An open-source family of large encoder-decoder foundation models for chemistry.Commun. Chem., 8(1):193,

    work page

  15. [15]

    doi:10.1038/s42004-025-01585-0

    ISSN 2399-3669,2399-3669. doi:10.1038/s42004-025-01585-0

    work page doi:10.1038/s42004-025-01585-0

  16. [16]

    MolFM: A multimodal molecular foundation model.arXiv [q-bio.BM], 2023

    Yizhen Luo, Kai Yang, Massimo Hong, Xing Yi Liu, and Zaiqing Nie. MolFM: A multimodal molecular foundation model.arXiv [q-bio.BM], 2023. doi:10.48550/arXiv.2307.09484

    work page doi:10.48550/arxiv.2307.09484 2023

  17. [17]

    ChemBERTa-3: An open source training framework for chemical foundation models.ChemRxiv, 2025

    Riya Singh, Aryan Amit Barsainyan, Rida Irfan, Connor Joseph Amorin, Stewart He, Tony Davis, Arun Thia- garajan, Shiva Sankaran, Seyone Chithrananda, Walid Ahmad, Derek Jones, Kevin McLoughlin, Hyojin Kim, Anoushka Bhutani, Shreyas Vinaya Sathyanarayana, Venkat Viswanathan, Jonathan E Allen, and Bharath Ram- sundar. ChemBERTa-3: An open source training fr...

    work page doi:10.26434/chemrxiv-2025-4glrl-v2 2025

  18. [18]

    Multi-view mixture-of-experts for predicting molecular properties using SMILES, SELFIES, and graph-based representations

    Eduardo Soares, Victor Yukio Shirasuna, Emilio Vital Brazil, Indra Priyadarsini, and Seiji Takeda. Multi-view mixture-of-experts for predicting molecular properties using SMILES, SELFIES, and graph-based representations. Mach. Learn. Sci. Technol., 6(2):025070, 2025. ISSN 2632-2153. doi:10.1088/2632-2153/ade4ef

    work page doi:10.1088/2632-2153/ade4ef 2025

  19. [19]

    What does bert look at? an analysis of bert’s attention

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does BERT look at? an analysis of BERT’s attention.arXiv [cs.CL], 2019. doi:10.48550/arXiv.1906.04341

    work page doi:10.48550/arxiv.1906.04341 2019

  20. [20]

    A primer in BERTology: What we know about how BERT works.arXiv [cs.CL], 2020

    Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works.arXiv [cs.CL], 2020. doi:10.48550/arXiv.2002.12327

    work page doi:10.48550/arxiv.2002.12327 2020

  21. [21]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah. A mat...

    work page 2021

  22. [22]

    How do transformers learn to associate tokens: Gradient leading terms bring mechanistic interpretability

    Shawn Im, Changdae Oh, Zhen Fang, and Sharon Li. How do transformers learn to associate tokens: Gradient leading terms bring mechanistic interpretability. InThe F ourteenth International Conference on Learning Representations, 2025

    work page 2025

  23. [23]

    Evolution of concepts in language model pre-training.arXiv [cs.CL], 2026

    Xuyang Ge, Wentao Shu, Jiaxing Wu, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Evolution of concepts in language model pre-training.arXiv [cs.CL], 2026. doi:10.48550/arXiv.2509.17196

    work page doi:10.48550/arxiv.2509.17196 2026

  24. [24]

    Investigation of chemical structure recognition by encoder-decoder models in learning progress.J

    Shumpei Nemoto, Tadahaya Mizuno, and Hiroyuki Kusuhara. Investigation of chemical structure recognition by encoder-decoder models in learning progress.J. Cheminform., 15(1):45, 2023. ISSN 1758-2946,1758-2946. doi:10.1186/s13321-023-00713-z

    work page doi:10.1186/s13321-023-00713-z 2023

  25. [25]

    Circuits, features, and heuristics in molecular transformers.arXiv [cs.LG], 2025

    Kristof Varadi, Mark Marosi, and Peter Antal. Circuits, features, and heuristics in molecular transformers.arXiv [cs.LG], 2025. doi:10.48550/arXiv.2512.09757

    work page doi:10.48550/arxiv.2512.09757 2025

  26. [26]

    A short history of thalidomide embryopathy.Teratology, 38(3):203–215, 1988

    W Lenz. A short history of thalidomide embryopathy.Teratology, 38(3):203–215, 1988. ISSN 1096-9926,0040-

    work page 1988

  27. [27]

    doi:10.1002/tera.1420380303

    work page doi:10.1002/tera.1420380303

  28. [28]

    Chromatographic separation of racemic thalidomide and teratogenic activity of its enantiomers (author’s transl).Arzneimittelforschung, 29(10):1640–1642, 1979

    G Blaschke, H P Kraft, K Fickentscher, and F Köhler. Chromatographic separation of racemic thalidomide and teratogenic activity of its enantiomers (author’s transl).Arzneimittelforschung, 29(10):1640–1642, 1979. ISSN 0004-4172,1616-7066

    work page 1979

  29. [29]

    MolAI: A deep learning framework for data-driven molecular descriptor generation and advanced drug discovery applications.J

    Sayyed Jalil Mahdizadeh and Leif A Eriksson. MolAI: A deep learning framework for data-driven molecular descriptor generation and advanced drug discovery applications.J. Chem. Inf. Model., 2025. ISSN 1549- 9596,1549-960X. doi:10.1021/acs.jcim.5c00491

    work page doi:10.1021/acs.jcim.5c00491 2025

  30. [30]

    Difficulty in chirality recognition for transformer architectures learning chemical structures from string representations.Nat

    Yasuhiro Yoshikai, Tadahaya Mizuno, Shumpei Nemoto, and Hiroyuki Kusuhara. Difficulty in chirality recognition for transformer architectures learning chemical structures from string representations.Nat. Commun., 15(1):1197,

    work page

  31. [31]

    doi:10.1038/s41467-024-45102-8

    ISSN 2041-1723,2041-1723. doi:10.1038/s41467-024-45102-8

    work page doi:10.1038/s41467-024-45102-8 2041

  32. [32]

    ZINC20—A Free Ultralarge-Scale Chemical Database for Ligand Discovery

    John J Irwin, Khanh G Tang, Jennifer Young, Chinzorig Dandarchuluun, Benjamin R Wong, Munkhzul Khurelbaatar, Yurii S Moroz, John Mayfield, and Roger A Sayle. ZINC20-a free ultralarge-scale chemical database for ligand discovery.J. Chem. Inf. Model., 60(12):6065–6073, 2020. ISSN 1549-9596,1549-960X. doi:10.1021/acs.jcim.0c00675

    work page doi:10.1021/acs.jcim.0c00675 2020

  33. [33]

    PubChem 2025 update.Nucleic Acids Res., 53(D1):D1516–D1525, 2025

    Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E Bolton. PubChem 2025 update.Nucleic Acids Res., 53(D1):D1516–D1525, 2025. ISSN 0305-1048,1362-4962. doi:10.1093/nar/gkae1059. 11 From Syntax to Semantics: Unveiling the Emergence of C...

    work page doi:10.1093/nar/gkae1059 2025

  34. [34]

    Neural Machine Translation of Rare Words with Subword Units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv [cs.CL], 2015. doi:10.48550/arXiv.1508.07909

    work page internal anchor Pith review doi:10.48550/arxiv.1508.07909 2015

  35. [35]

    Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Ri...

    work page internal anchor Pith review doi:10.48550/arxiv.1609.08144 2016

  36. [36]

    S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.arXiv [cs.CL], 2018. doi:10.48550/arXiv.1808.06226

    work page internal anchor Pith review doi:10.48550/arxiv.1808.06226 2018

  37. [37]

    RDKit: Open-source cheminformatics

    work page

  38. [38]

    Smiles enumeration as data augmentation for neural network modeling of molecules.ArXiv, abs/1703.07076, 2017

    Esben Jannik Bjerrum. SMILES enumeration as data augmentation for neural network modeling of molecules. arXiv [cs.LG], 2017. doi:10.48550/arXiv.1703.07076

    work page doi:10.48550/arxiv.1703.07076 2017

  39. [39]

    Randomized SMILES strings improve the quality of molecular generative models.J

    Josep Arús-Pous, Simon Viet Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum, Christian Tyrchan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist. Randomized SMILES strings improve the quality of molecular generative models.J. Cheminform., 11(1):71, 2019. ISSN 1758-2946,1758-2946. doi:10.1186/s13321-019-0393-0

    work page doi:10.1186/s13321-019-0393-0 2019

  40. [40]

    A novel molecule generative model of V AE combined with transformer.arXiv [q-bio.BM], 2024

    Yasuhiro Yoshikai, Tadahaya Mizuno, Shumpei Nemoto, and Hiroyuki Kusuhara. A novel molecule generative model of V AE combined with transformer.arXiv [q-bio.BM], 2024. doi:10.48550/arXiv.2402.11950

    work page doi:10.48550/arxiv.2402.11950 2024

  41. [41]

    RoFormer: Enhanced Transformer with Rotary Position Embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.arXiv [cs.CL], 2021. doi:10.48550/arXiv.2104.09864

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2104.09864 2021

  42. [42]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  43. [43]

    gpt-oss-120b & gpt-oss-20b Model Card

    OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vl...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.10925 2025

  44. [44]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  45. [45]

    Gemma: Open Models Based on Gemini Research and Technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Anto...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2403.08295 2024

  46. [46]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019

    work page 2019

  47. [47]

    GLU Variants Improve Transformer

    Noam Shazeer. GLU variants improve transformer.arXiv [cs.LG], 2020. doi:10.48550/arXiv.2002.05202

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2002.05202 2020

  48. [48]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding.arXiv [cs.CL], 2018. doi:10.48550/arXiv.1810.04805

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1810.04805 2018

  49. [49]

    Scalable Diffusion Models with Transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers.arXiv [cs.CV], 2022. doi:10.48550/arXiv.2212.09748

    work page internal anchor Pith review doi:10.48550/arxiv.2212.09748 2022

  50. [50]

    Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations.Chem

    Robin Winter, Floriane Montanari, Frank Noé, and Djork-Arné Clevert. Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations.Chem. Sci., 10(6):1692–1701, 2019. ISSN 2041-6520,2041-6539. doi:10.1039/c8sc04175j

    work page doi:10.1039/c8sc04175j 2019

  51. [51]

    In Proceedings of the 26th Annual International Conference on Machine Learning (Montreal, Quebec, Canada) (ICML ’09)

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th Annual International Conference on Machine Learning, New York, NY , USA, 2009. ACM. ISBN 9781605585161. doi:10.1145/1553374.1553380

    work page doi:10.1145/1553374.1553380 2009

  52. [52]

    The road less scheduled.arXiv [cs.LG], 2024

    Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled.arXiv [cs.LG], 2024. doi:10.48550/arXiv.2405.15682

    work page doi:10.48550/arxiv.2405.15682 2024

  53. [53]

    Zclip: Adaptive spike mitigation for llm pre-training,

    Abhay Kumar, Louis Owen, Nilabhra Roy Chowdhury, and Fabian Güra. ZClip: Adaptive spike mitigation for LLM pre-training.arXiv [cs.LG], 2025. doi:10.48550/arXiv.2504.02507

    work page doi:10.48550/arxiv.2504.02507 2025

  54. [54]

    Similarity of Neural Network Representations Revisited, in: Proceedings of the 36th International Conference on Machine Learning (ICML), pp

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited.arXiv [cs.LG], 2019. doi:10.48550/arXiv.1905.00414

    work page doi:10.48550/arxiv.1905.00414 2019

  55. [55]

    What happens during the loss plateau? understanding abrupt learning in transformers

    Pulkit Gopalani and Wei Hu. What happens during the loss plateau? understanding abrupt learning in transformers. arXiv [cs.LG], 2025. doi:10.48550/arXiv.2506.13688

    work page doi:10.48550/arxiv.2506.13688 2025

  56. [56]

    The geometric anatomy of capability acquisition in transformers.arXiv [cs.LG], 2026

    Jayadev Billa. The geometric anatomy of capability acquisition in transformers.arXiv [cs.LG], 2026. doi:10.48550/arXiv.2602.15997

    work page doi:10.48550/arxiv.2602.15997 2026

  57. [57]

    arXiv preprint arXiv:2309.07311 (2023)

    Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs.arXiv [cs.CL], 2023. doi:10.48550/arXiv.2309.07311. 14 From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models Tables and Figures Table 1: V o...

    work page doi:10.48550/arxiv.2309.07311 2023

  58. [58]

    (a, b) Encoder; (c, d) Decoder

    to L7H7, forpancore-addonceon ZINC20 ( ∼100)). (a, b) Encoder; (c, d) Decoder. (a, c) The region bounded by the two vertical dashed lines indicates the jump-up interval defined by perplexity. (b, d) Changes in the three heads with the largest entropy decrease within the jump-up interval. The red line and the light red shaded region indicate the perplexity...

    work page