Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SMI-Editor, a SMILES language model pre-trained with edit operations over molecular fragments, outperforms token-masked baselines on seven MoleculeNet classification tasks, reaching an average ROC-AUC of 77.8.

desk verdict Genuinely new edit-based SMILES pretraining objective and a useful MLM saturation analysis, but the SOTA claim needs a contamination check before the numbers are trustworthy. read the letter →

arxiv 2412.05569 v2 pith:V4XWZZW2 submitted 2024-12-07 cs.LG q-bio.BM

Kangjie Zheng , Siyue Liang , Junwei Yang , Bin Feng , Zequn Liu , Wei Ju , Zhiping Xiao , Ming Zhang This is my paper · ORCID
classification cs.LGq-bio.BM
keywords SMILESlanguagemodelmolecularpropertypredictionedit-basedpretrainingfragment-levelsupervisionLevenshteintransformermaskedmodelingMoleculeNetsubstructuresemantics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that masked-token pretraining is the wrong supervision for SMILES language models: molecules are understood through substructures, and token-level masking saturates quickly and fails to capture functional-group semantics. The authors propose SMI-Editor, which deletes random molecular fragments from valid SMILES strings and trains the model to restore the original SMILES through deletion and insertion edits. On seven MoleculeNet classification tasks, SMI-Editor reaches an average ROC-AUC of 77.8, ahead of 73.2 for a matched MLM baseline and ahead of several 3D molecular representation models. If the result holds, it means the choice of pretraining objective, not just scale or architecture, is what unlocks molecular semantics from SMILES.

What carries the argument

The machinery is the Levenshtein-style editing head on top of a Transformer encoder, combined with fragment-level supervision generated by a modified BRICS/RMCF fragmentation. Given a SMILES string with selected fragments dropped, the model predicts three operations: which tokens to delete, how many placeholders to insert at each position, and which vocabulary token fills each placeholder. The dual deletion loss, which supervises deleting erroneous tokens directly on the corrupted input rather than only on model-generated insertions, is what makes deletion learning non-trivial. Together these operations convert pretraining from a single-token prediction task into a structure reconstruction task over chemically meaningful fragments.

What would settle it

Compute the exact overlap between the SMILES in the pretraining corpus and the molecules in each MoleculeNet test set after canonicalization and salt stripping; if any non-trivial fraction of test molecules appears in pretraining, retrain SMI-MLM and SMI-Editor on the deduplicated corpus and compare the gap. A large drop in SMI-Editor's advantage would indicate the edit objective's gains were partly memorization; a sustained gap would confirm transferable fragment learning.

Watch

Extended reading notes

Core claim

SMI-Editor's central claim is that an edit-based objective with fragment-level supervision lets a SMILES-only transformer learn molecular substructure semantics that masked language modeling misses. During pretraining, the model is given a valid SMILES string with randomly deleted substructures and must reconstruct the original string by predicting token deletions, placeholder insertions, and tokens for placeholders, with a dual deletion loss that supervises removal of corrupted tokens. The paper shows this objective avoids the rapid saturation that makes MLM pretraining trivial, and that removing any of the editing signals degrades downstream performance. The result is presented as a new state of the art on MoleculeNet property classification, including gains over 3D models.

Load-bearing premise

The whole comparison rests on the assumption that the 19-million-molecule pretraining set does not contain the molecules used in the MoleculeNet test splits, so the reported ROC-AUC gains reflect transferred fragment knowledge rather than memorization of benchmark molecules.

Editorial extensions

If this is right

  • Fragment-level edit supervision can be added to any SMILES encoder without changing the downstream fine-tuning protocol, so the reported gains should transfer to other encoder-based molecular property models.
  • The same pretraining objective, adapted to an encoder-decoder, reaches top results on USPTO-50K retrosynthesis in the paper's appendix, suggesting the editing task learns generative chemical knowledge, not just classification features.
  • Because the edit objective avoids rapid saturation, bigger models keep improving with more capacity and training steps, whereas the paper's matched MLM baseline plateaus at small scale.
  • The paper's ablation indicates placeholder insertion, the operation that localizes missing fragments, is the largest single contributor, meaning positional reasoning about where substructures belong is a key learned skill.
  • Under an equal training budget, SMI-Editor still outperforms an MLM trained three times longer, suggesting the benefit comes from the objective rather than from extra computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper never reports whether the 19M pretraining molecules overlap the MoleculeNet test sets, the cleanest extension of this work is to rerun the comparison with a deduplicated corpus; the ROC-AUC gap would shrink if memorization, not fragment knowledge, drove part of the gain.
  • The edit-based objective resembles a denoising autoencoder over chemical structures; connecting SMI-Editor's fragment perturbations to reaction templates or retrosynthetic disconnections could make the reconstruction task chemically meaningful beyond BRICS fragments.
  • The paper's analysis of ESOL shows molecular-weight confounds can mimic substructure sensitivity; future evaluations of 'substructure understanding' should control for size before attributing prediction changes to functional-group semantics.
  • An encoder-only model cannot generate, but the appendix's SMI-Editor-Gen extends the same fragment-reconstruction idea to sequence-to-sequence retrosynthesis; combining the edit objective with contrastive or 3D alignment signals is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SMI-EDITOR, a SMILES language model pretrained with an edit-based objective: random molecular fragments are dropped, and the model learns to restore the original SMILES via token deletion, placeholder insertion, and token prediction, using Levenshtein-distance expert actions (LevT) and a dual deletion loss. The authors argue that MLM pretraining on SMILES suffers from rapid saturation and fails to capture substructure semantics, and they provide analyses supporting that SMI-EDITOR mitigates these issues. Experiments on 7 MoleculeNet classification tasks report a mean ROC-AUC of 77.8 for SMI-EDITOR, exceeding the matched MLM baseline SMI-MLM (73.2) and several 2D/3D baselines. Ablations confirm the contributions of fragment-level dropping and the three editing operations.

Significance. If the results hold, SMI-EDITOR is a conceptually valuable contribution: it is, to my knowledge, the first edit-based pretraining framework for SMILES, and it demonstrates a clear and consistent improvement over a matched MLM baseline. The paper includes open-source code, ablation studies, and novel analytical experiments on substructure semantics, which are useful for the community. However, the central empirical claim requires stronger validation due to potential data contamination and missing statistical rigor.

major comments (4)
  1. [Section 4.1, Table 1, Appendix J] The central SOTA claim depends on an unchecked assumption that the 19M-molecule pretraining set from Zhou et al. (2023) does not overlap the MoleculeNet test splits. The paper never reports an overlap analysis. The appendix's DeepChem-split result for SMI-MLM (BBBP 89.4, Table 8) is far above published baselines (e.g., ChemBerta 64.3, MolCLR 73.6), which is the expected fingerprint of leakage. Because SMI-EDITOR is trained to reconstruct whole molecules from dropped fragments, contamination would inflate its scores more than SMI-MLM's token-level masking. Please report the exact overlap (e.g., canonical SMILES or InChI matching) between the pretraining corpus and every test split, and rerun the main comparison on a decontaminated subset or after removing overlapping molecules.
  2. [Table 1] The headline results are reported as single-run ROC-AUC values without error bars or significance tests. The mean gap over SMI-MLM (77.8 vs 73.2) is large, but per-task differences are more modest (e.g., SIDER 63.0 vs 59.6, ToxCast 67.4 vs 69.2 for MolCLR), and run-to-run variance in fine-tuning can exceed one point. Please report means and standard deviations over at least five fine-tuning seeds for SMI-EDITOR and SMI-MLM on all seven tasks, and indicate whether the differences are statistically significant.
  3. [Section 4.2, Table 1] Uni-Mol (Zhou et al., 2023) is the source of the pretraining dataset and the fine-tuning split used in this paper, yet it is omitted from Table 1. Without Uni-Mol, the claim of 'outperforming several 3D molecular representation models' omits the most directly relevant 3D baseline. Please add Uni-Mol's results for the same splits, or state explicitly if its published numbers were obtained under a different protocol and are therefore not directly comparable.
  4. [Appendix J vs Section 4.1] The main text states that pretraining uses the 19M-molecule dataset from Zhou et al. (2023), but Appendix J says the DeepChem-split experiments use SMI-EDITOR 'pre-trained on datasets provided by Ross et al. (2022).' This inconsistency makes Table 8 difficult to interpret. Please clarify which pretraining checkpoint is used for each experiment, and confirm whether SMI-MLM in Table 8 was trained on exactly the same pretraining data and protocol as SMI-EDITOR.
minor comments (6)
  1. [Abstract] The phrase 'and even outperforming several 3D molecular representation models' should be 'and even outperforms several 3D molecular representation models'.
  2. [Section 2.1] The sentence 'These findings highlight the limited scalability of SMILE MLMs' contains a typo: 'SMILE' should be 'SMILES'.
  3. [Section 4.3.1] The text refers to the 'unrealistic special symbol [MASA]' but the intended symbol is '[MASK]'.
  4. [Table 1] The text says SMI-EDITOR achieves SOTA performance on 4 out of 7 tasks, but the table shows it is best on 5 tasks (BBBP, Tox21, SIDER, MUV, ClinTox); please correct this count.
  5. [Section 3.2, Eq. (4)] The display equation for the dual deletion loss is typeset in a confusing way (Σ_{y_i ∈ M} d*_i ∈ d* ...); please introduce the notation for d*_i and the summation index more clearly.
  6. [Table 4] The column header 'Max Tokens' likely refers to maximum tokens per batch rather than a model dimension; please rename it to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

The derivation is self-contained: the edit-based pre-training objective is defined independently of downstream labels, and no reported result reduces to its own inputs.

full rationale

No load-bearing circular step is present. SMI-EDITOR's pre-training objective (Section 3.2) is to reconstruct the original SMILES from a version with randomly dropped BRICS/RMCF fragments, using deletion and insertion heads supervised by Levenshtein-distance expert actions; this objective is defined without reference to any downstream property label. Downstream ROC-AUC values (Table 1) come from separate fine-tuning on MoleculeNet splits, not from the pre-training loss itself, and the matched SMI-MLM baseline shares the same data, architecture, and hyperparameters, so the reported gap is an empirical result rather than an equality by construction. Ablations in Tables 2 and 3 directly test the fragment-level supervision and each editing operation. The dual deletion loss is related to Zheng et al. (2023), prior work by the first author, but the loss is fully specified in the paper and its contribution is isolated by the w/o DualDel ablation, so the citation is not load-bearing. No uniqueness theorem, ansatz, or renamed known result is imported from the authors' prior work to force the design. The concern about possible overlap between the 19M-molecule pre-training corpus and MoleculeNet test sets is a benchmark-contamination/external-validity risk, not a circularity of the derivation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on one unverified data-hygiene assumption (no overlap between the 19M pre-training corpus and the benchmark test sets), one design premise (fragment dropping teaches useful substructure semantics), and the borrowed optimality of Levenshtein-expert edits. The model itself adds no new physical entities and has one hand-tuned corruption ratio (0.15) that ablation confirms is the best of the tested values.

free parameters (3)
  • fragment drop ratio = 0.15
    Chosen by hand; ablation in Table 11 shows 0.15 gives best mean (73.0 vs 71.7 at 0.30 and 71.5 at 0.45), so the reported main results are tied to this hand-picked value.
  • max insertion tokens (K) = 255
    Copied from Levenshtein Transformer, bounds the placeholder insertion classification to 256 classes; not tuned for SMILES.
  • mask ratio for MLM comparison = 0.15
    Standard MLM hyperparameter used for SMI-MLM baseline; not part of SMI-EDITOR itself but affects the fairness of the MLM comparison.
assumptions (4)
  • domain assumption The 19M-molecule pretraining corpus (Zhou et al., 2023) does not contain the downstream MoleculeNet test molecules.
    Section 4.1 adopts the corpus without decontamination; if false, the reported test AUC improvements are partly memorization.
  • domain assumption Randomly dropping chemical fragments and reconstructing the original SMILES teaches fragment-level semantics that transfer to property prediction.
    Core pretraining premise; supported only indirectly by downstream accuracy and the perturbation analyses.
  • ad hoc to paper The Levenshtein-distance expert actions provide optimal edit supervision for SMILES reconstruction.
    Taken from LevT (Gu et al., 2019) and Zheng et al. (2023); no validation that Levenshtein distance aligns with chemical validity.
  • ad hoc to paper Linear-probe prediction deltas on ESOL/FreeSolv measure whether the model encodes substructure semantics.
    Section 2.2 interprets the deltas as semantic sensitivity, but the ESOL result is confounded by molecular weight (Appendix R).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision." pith.science (2026). https://pith.science/paper/V4XWZZW2

@misc{pith2026241205569,
  author       = {Pith},
  title        = {Pith review of: SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4XWZZW2}},
  note         = {Machine review of arXiv:2412.05569}
}
read the original abstract

SMILES, a crucial textual representation of molecular structures, has garnered significant attention as a foundation for pre-trained language models (LMs). However, most existing pre-trained SMILES LMs focus solely on the single-token level supervision during pre-training, failing to fully leverage the substructural information of molecules. This limitation makes the pre-training task overly simplistic, preventing the models from capturing richer molecular semantic information. Moreover, during pre-training, these SMILES LMs only process corrupted SMILES inputs, never encountering any valid SMILES, which leads to a train-inference mismatch. To address these challenges, we propose SMI-Editor, a novel edit-based pre-trained SMILES LM. SMI-Editor disrupts substructures within a molecule at random and feeds the resulting SMILES back into the model, which then attempts to restore the original SMILES through an editing process. This approach not only introduces fragment-level training signals, but also enables the use of valid SMILES as inputs, allowing the model to learn how to reconstruct complete molecules from these incomplete structures. As a result, the model demonstrates improved scalability and an enhanced ability to capture fragment-level molecular information. Experimental results show that SMI-Editor achieves state-of-the-art performance across multiple downstream molecular tasks, and even outperforming several 3D molecular representation models.

Figures

Figures reproduced from arXiv: 2412.05569 by the authors.

Figure 1
Figure 1. The framework of MLM for SMILES. The Masked Language Model (MLM) is a widely used approach for textual information modeling and has been extensively applied to SMILES representation learning (Wang et al., 2019a; Chithrananda et al., 2020; Ross et al., 2022). During the training process, tokens in a SMILES sequence – representing single atoms, chemical bonds, or special symbols – are randomly masked with a fixed mask… view at source ↗
Figure 2
Figure 2. Rapid Saturation Problem. We train SMILES MLMs of various sizes and masking ratios using the dataset from Zhou et al. (2023). Figure (a) displays the training loss and masking prediction accuracy of different-sized models, showing a rapid decrease in loss and an increase in accuracy at the start of the training. Figure (b) presents similar trends for the validation set. Figure (c) illustrates the training loss and a… view at source ↗
Figure 3
Figure 3. Substructure Semantics Modeling. We compared two molecular perturbation meth￾ods—removing hydrophilic groups and randomly deleting atoms—and their effects on the model’s predictions of hydrophilicity and related properties. Figure (a) presents the impact of these perturba￾tions on model predictions in the ESOL dataset, including the distribution of prediction changes. The average prediction change is similar for bot… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Substructure Semantics Modeling from SMI-EDITOR. We compared the effects of two molecular perturbation methods on the SMI-EDITOR’s predictions of hydrophilicity and related properties. Figure (a) and Figure (b) show that the impact of deleting hydrophilic groups (HG De…
Figure 5
Figure 5. Figure 5: Overall Framework of SMI-EDITOR. The framework includes a data processing module, a SMILES encoder, and an edit-based pre-training process. In data processing, some fragments of the input molecule are randomly removed, and the resulting SMILES is fed into the model. Th…
Figure 6
Figure 6. Figure 6: The training and validation loss curves of different-sized [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Training Loss Curves of Editing Operations. We train SMI-EDITOR models of varying sizes and compare their loss curves during training for three different editing operations. As shown in the results, the loss for the token prediction process represented in Figure (c) is…
Figure 8
Figure 8. Figure 8: Validation Loss Curves of Editing Operations. Similar to the training loss curves, the validation loss for the token prediction process shown in Figure (c) consistently remains the highest among the three types, while the loss for token deletion is the lowest. Addition…
Figure 9
Figure 9. Figure 9: The performance of MLMs of different sizes and training steps on BBBP task. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The performance of SMI-EDITOR of different sizes and training steps on BBBP task. G HYPER-PARAMETER CONFIGURATION FOR PRE-TRAINING We implement SMI-EDITOR using 12 stacked Transformer layers, each with 12 attention heads. The model dimension and feedforward dimension …
Figure 11
Figure 11. Figure 11: The training loss and perplexity (PPL) curves of the SMI-GPT model. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: The results show that as the drop ratio increases, both training and validation losses rise significantly. Compared to Figure 2c of the paper, the loss increase for SMI-EDITOR is more pronounced than for MLM, confirming that SMI-EDITOR’s task is inherently more challe…
Figure 12
Figure 12. Figure 12: The training loss and valid loss of the SMI-E [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: An Example Workflow of Molecule Fragmentation and Assemble with Paracetamol [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: The relationship between molecular weight and solubility in the ESOL training set [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Substructure Semantics Modeling on ESOL Dataset. We compared the effects of two molecular perturbation methods on the SMI-EDITOR’s and SMI-MLM’s predictions of hydrophilicity. Figure 15a show that the impact of replacing hydrophilic groups (HG Rep) and randomly replac…
Figure 16
Figure 16. Figure 16: Substructure Semantics Modeling on ESOL Dataset of Auto-regressive LM. [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Corrupted, ambiguous context semantics, not the presence of [MASK] symbols, drive MLM accuracy loss; expanding each [MASK] into multiple modeled states mitigates this.

Reference graph

Works this paper leans on

35 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Molgpt: molecular generation using a transformer-decoder model.Journal of Chemical Information and Modeling, 62(9): 2064–2076,

    Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. Molgpt: molecular generation using a transformer-decoder model.Journal of Chemical Information and Modeling, 62(9): 2064–2076,

    work page

  2. [4]

    Table 4: Hyper-parameters for pre-train models with different scales

    and training datasets, remain consistent to ensure a fair comparison of results. Table 4: Hyper-parameters for pre-train models with different scales. Model Max Tokens Layers Attn Heads Embed Dim FFN Dim Dropout Num of Paras Big 64K 9 12 768 2048 0.1 50.5M Base 64K 6 8 512 2048 0.1 19.4M Small 64K 3 8 512 1024 0 6.8M E PERFORMANCE OFMLMS ONDOWNSTREAMTASK ...

    work page

  3. [5]

    Table 5: SMI-EDITORhyper-parameters for pre-training. Hyper-parameters Value Learning rate 5e-4 LR scheduler polynomial decay Warmup updates 10K Max updates 120K Max tokens 64k FFN dropout 0.1 Attention dropout 0.1 Activation dropout 0 Num of layers 12 Num of attention heads 12 Encoder embedding dim 768 Encoder FFN dim 3072 Adam (β1, β2) (0.9,0.98) Fragme...

    work page 2025

  4. [7]

    Table 7: Summary information of the MoleculeNet benchmark datasets

    This table offers information about the dataset sizes, task types, and compositions, providing readers with essential background information to better understand the experimental setup and subsequent analysis. Table 7: Summary information of the MoleculeNet benchmark datasets. Dataset Tasks Task type Molecules (train/valid/test) Describe ESOL 1 Regression...

    work page 2022

  5. [8]

    A group symmetric stochastic differential equation model for molecule multi-modal pretraining

    Shengchao Liu, Weitao Du, Zhi-Ming Ma, Hongyu Guo, and Jian Tang. A group symmetric stochastic differential equation model for molecule multi-modal pretraining. InInternational Conference on Machine Learning, pp. 21497–21526. PMLR, 2023a. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and ...

    arXiv 1907

  6. [9]

    Encode, tag, realize: High-precision text editing.arXiv preprint arXiv:1909.01187,

    Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. Encode, tag, realize: High-precision text editing.arXiv preprint arXiv:1909.01187,

    arXiv 1909

  7. [10]

    Compared to the performance of the MLM (Figure 9), the larger SMI-EDITORmodel (i.e., Big Model) consistently outperforms the smaller models (i.e., Small Model)

    Again, we evaluate each checkpoint on the downstream tasks five times and take the mean result to ensure accuracy. Compared to the performance of the MLM (Figure 9), the larger SMI-EDITORmodel (i.e., Big Model) consistently outperforms the smaller models (i.e., Small Model). As the number of training steps increases, the performance gap between large and ...

    work page 2025

  8. [11]

    This indicates that SMI-EDITOR represents a more challenging training task

    From Table 11, it can be observed that as the mask ratio increases, the average performance of the SMI-MLMmodel shows no significant change, while the performance of the SMI-EDITORmodel declines as the drop ratio increases. This indicates that SMI-EDITOR represents a more challenging training task. Here is a more detailed explanation: • SMI-EDITORdiscards...

    work page 2025

Show all 35 references
  1. [12]

    Lexically constrained neural machine translation with levenshtein transformer.arXiv preprint arXiv:2004.12681,

    Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. Lexically constrained neural machine translation with levenshtein transformer.arXiv preprint arXiv:2004.12681,

    2004 arXiv

  2. [13]

    These results are shown in Table

    24 Published as a conference paper at ICLR 2025 P THE SCALABILITY OFSMI-EDITOR We added results showing the performance of SMI-EDITORand SMI-MLM models of varying sizes on downstream tasks, which further demonstrate SMI-EDITOR’s strong scalability. These results are shown in Table

    2025

  3. [14]

    Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande

    URLhttp://arxiv.org/abs/1703.00564. Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513–530,

    arXiv

  4. [15]

    Poisoning medical knowledge using large language models

    Junwei Yang, Hanwen Xu, Srbuhi Mirzoyan, Tong Chen, Zixuan Liu, Zequn Liu, Wei Ju, Luchen Liu, Zhiping Xiao, Ming Zhang, et al. Poisoning medical knowledge using large language models. Nature Machine Intelligence, 6(10):1156–1168, 2024a. Junwei Yang, Kangjie Zheng, Siyu Long, ...

    2024

  5. [16]

    Towards a unified training for levenshtein transformer

    Kangjie Zheng, Longyue Wang, Zhihao Wang, Binqi Chen, Ming Zhang, and Zhaopeng Tu. Towards a unified training for levenshtein transformer. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  6. [17]

    A large range of Pre-trained SMILES LMs (Wang et al., 2019a; Chithrananda et al., 2020; Ross et al.,

    13 Published as a conference paper at ICLR 2025 A RELATEDWORKS Simplified Molecular Input Line Entry System (SMILES) has emerged as a cornerstone sequential representation for molecular data, making it a focal point in molecular representation learning. A large range of Pre-tr...

    2025

  7. [18]

    These models typically leverage techniques like MLM and autoregressive pre-training to capture the complex syntax and semantics embedded in SMILES sequences

    have been developed to address challenges in SMILES-based molecular modeling, demonstrating their effectiveness across diverse downstream tasks (Bagal et al., 2021; Tong et al., 2021; Feng et al., 2024; Yang et al., 2024a). These models typically leverage techniques like MLM a...

    2019

  8. [19]

    (2021) applied generative models to drug design tasks

    employs an autoregressive mechanism, while Tong et al. (2021) applied generative models to drug design tasks. More recently, Liu et al. (2023b) introduced MolXPT, unifying SMILES and textual data using a generative pre-training strategy. Overall, these works highlight the grow...

    2021

  9. [20]

    Recognizing that edit operations from an input sequence to a target output can be diverse and difficult to compute directly, Gu et al

    extended this idea by integrating a pointer-based mechanism with an MLM backbone to handle insertion and deletion tasks more efficiently. Recognizing that edit operations from an input sequence to a target output can be diverse and difficult to compute directly, Gu et al. (201...

    2019

  10. [26]

    The findings indicate that SMI-EDITORcan perform better than SMI-GPT. Below is an analysis of these results: K.1 IMPLEMENTATION DETAILS FORSMI-GPT(NT)ANDSMI-GPT(EMB) SMI-GPT(NT): This approach uses next-token prediction for downstream classification tasks by appending a specia...

    2025

  11. [29]

    Table 13: Performance of SMI-EDITORand SMI-MLMwith different scales on downstream tasks

    It is evident that while increasing model size has minimal impact on MLMs, larger SMI-EDITORmodels show more consistent performance gains.This confirms the claim that SMI-EDITORhas better scalability compared to MLMs. Table 13: Performance of SMI-EDITORand SMI-MLMwith differen...

    2025

  12. [30]

    A clear negative correlation with a coefficient of R=−0.69 is observed

    showing the relationship between molecular weight and solubility in the ESOL training set. A clear negative correlation with a coefficient of R=−0.69 is observed. Consequently, when functional groups or atoms are removed from a molecule, its molecular weight decreases, leading...

    2025

  13. [31]

    This further highlights the superiority of the SMI-EDITORin modeling the semantics of molecular substructures

    The results show that these models perform significantly worse than the SMI-EDITORin distinguishing between random replacements and hydrophilic group replacements. This further highlights the superiority of the SMI-EDITORin modeling the semantics of molecular substructures. S ...

    2017

  14. [32]

    , the features extracted by the encoder are passed to the decoder through 27 Published as a conference paper at ICLR 2025 encoder-decoder attention (Vaswani et al.,

    2025

  15. [33]

    Compared to SMI-EDITOR, the most significant difference is that the encoder-decoder architecture enables SMI-EDITOR-Gen to perform sequence- to-sequence generative tasks, allowing us to explore the model’s capabilities in such tasks. SMI-EDITOR-Gen Exhibits Strong Performance ...

    2024

  16. [34]

    This validates the effectiveness of SMI-EDITOR’s training approach. U MASKEDSPANLM DOESNOTIMPROVESMILES LM PERFORMANCE To highlight the differences between SMI-EDITORand Masked Span LMs (MSLMs), we trained a SMILES model using MSLM, which randomly masks continuous sequences in...

    2022

  17. [35]

    explicitly leverage atom-level tokenization to enhance the granularity of molecular representations, facilitating downstream tasks such as molecule generation and property prediction. Atom-level tokenization has the advantage of maintaining a straightforward correspondence bet...

    2020

  18. [255]

    Thus, this step essentially becomes a 256-class classification task at each token, predicting the number of words (0–255) to insert after each token. This process can be represented as follows: Lins θ =− X yi ∈Y0 p∗ i ∈p∗ logπ ins θ (p∗ i |i, Y0), where p∗ i is the optimal pla...

    2019

  19. [2008]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    arXiv

  20. [2017]

    Regularized molecular conformation fields.Advances in Neural Information Processing Systems, 35:18929– 18941, 2022a

    Lihao Wang, Yi Zhou, Yiqun Wang, Xiaoqing Zheng, Xuanjing Huang, and Hao Zhou. Regularized molecular conformation fields.Advances in Neural Information Processing Systems, 35:18929– 18941, 2022a. Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert: ...

    2025

  21. [2018]

    Translation between molecules and natural language.arXiv preprint arXiv:2204.11817,

    Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language.arXiv preprint arXiv:2204.11817,

    arXiv

  22. [2019]

    Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations.arXiv preprint arXiv:2310.07276,

    Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations.arXiv preprint arXiv:2310.07276,

    arXiv

  23. [2020]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    arXiv

  24. [2021]

    Chemberta: large-scale self- supervised pretraining for molecular property prediction.arXiv preprint arXiv:2010.09885,

    Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale self- supervised pretraining for molecular property prediction.arXiv preprint arXiv:2010.09885,

    2010 arXiv

  25. [2022]

    Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization.arXiv preprint arXiv:1908.01000,

    Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization.arXiv preprint arXiv:1908.01000,

    1908 arXiv

  26. [2023]

    N-gram graph: Simple unsupervised repre- sentation for graphs, with applications to molecules.Advances in neural information processing systems, 32, 2019a

    Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised repre- sentation for graphs, with applications to molecules.Advances in neural information processing systems, 32, 2019a. Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, an...

    arXiv

  27. [2024]

    Contextual representation learning beyond masked language modeling.arXiv preprint arXiv:2204.04163,

    Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, and Lei Li. Contextual representation learning beyond masked language modeling.arXiv preprint arXiv:2204.04163,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.