Pith. sign in

REVIEW 3 major objections 3 minor 4 cited by

ChemMLLM: Chemical Multimodal Large Language Model

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ChemMLLM is a unified chemical multimodal model that understands and generates molecules in text, SMILES, and image, and reports state-of-the-art performance on five cross-modal tasks.

desk verdict Solid new chemical MLLM and dataset suite, but the headline SOTA numbers are built on non-equivalent baseline comparisons and a misidentified best baseline. read the letter →

arxiv 2505.16326 v2 pith:WF5A5HOQ submitted 2025-05-22 cs.LG

Qian Tan , Dongzhan Zhou , Peng Xia , Wanhao Liu , Wanli Ouyang , Lei Bai , Yuqiang Li , Tianfan Fu This is my paper · ORCID
classification cs.LG
keywords ChemMLLMchemicalmultimodallargelanguagemodelmoleculeimagegenerationunderstandingSMILESVQGANtokenizationmolecularpropertypredictionoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes ChemMLLM, a single multimodal language model that both understands and generates molecules across three modalities: text, SMILES strings, and 2D molecule images. The authors define five tasks — molecule image captioning, property prediction, image-to-SMILES conversion, property-conditioned molecule image design, and molecule image optimization — and build training and evaluation datasets for each. They report that ChemMLLM outperforms the strongest baseline on all five tasks; for example, in molecule image optimization it raises LogP by 4.27 versus 1.97 for the best general multimodal baseline, a 116.75% relative improvement. The claim matters because it suggests chemists could interact with molecules visually rather than through SMILES strings alone, using one model for perception, reasoning, and generation. If correct, it positions domain-specific visual tokenization as the key to making general multimodal models useful in chemistry.

What carries the argument

The load-bearing mechanism is Mol-VQGAN, a vector-quantized generative adversarial image tokenizer that compresses a 256×256 molecule image into discrete codebook indices and later reconstructs an image from those indices. Because molecule images are sparse line drawings rather than natural photographs, the paper fine-tunes the VQGAN on a large corpus of molecule images so that the discrete tokens preserve atoms, bonds, and geometry. The LLM then sees a single token sequence mixing text, SMILES, and image tokens, and is supervised with next-token prediction plus a z-loss regularizer in a two-stage training scheme: first train the tokenizer, then fine-tune the language model on the five tasks.

What would settle it

Take the image-generation test set and evaluate every model in the same output modality: have the general multimodal baseline also output an image, or have human chemists parse ChemMLLM's generated images into SMILES with an independent tool and recompute LogP gains; if the 4.27 versus 1.97 margin shrinks to near zero under this matched comparison, the central superiority claim for image optimization would be refuted.

Watch

Extended reading notes

Core claim

The central discovery claimed is that an LLM can be made chemically multimodal by discretizing molecule images into token sequences with a molecule-tuned VQGAN and interleaving those image tokens with text and SMILES tokens in one autoregressive transformer. On this foundation, ChemMLLM handles both comprehension tasks (captioning, property prediction, image-to-SMILES) and generation tasks (property-to-image, image-to-image) within a single fine-tuned model. The paper's headline number is a 116.75% relative improvement over the strongest general baseline in LogP increase for molecule image optimization (4.27 vs 1.97), with consistent wins across all evaluated metrics on every task.

Load-bearing premise

The headline improvement in the two image-generation tasks assumes it is fair to score ChemMLLM's images after converting them to SMILES with an external recognition tool while scoring comparison models on SMILES they output directly; any systematic recognition bias in that tool inflates or deflates the apparent gap.

Editorial extensions

If this is right

  • A single ChemMLLM-style model can replace separate optical-recognition, property-prediction, and molecule-generation pipelines for image-based chemical workflows.
  • Molecule optimization can be performed end-to-end in image space, so the output is directly interpretable to chemists instead of requiring a SMILES renderer.
  • The five constructed datasets give the community a common benchmark for evaluating cross-modal molecule understanding and generation.
  • The ablation results imply that both molecule-specific image tokenizer training and image augmentation are necessary; removing either degrades property correlations substantially.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported evaluation is unbiased, the large margin over general multimodal baselines suggests that domain-specific visual tokenization, rather than model scale alone, drives chemical image competence; a general model scaled up may not close the gap.
  • The property-to-image and image-to-image evaluations depend on an external SMILES recognizer to score generated images, so the true end-to-end quality may differ under human or alternative optical-recognition assessment; a direct comparison in which every model outputs the same modality would be a stronger test.
  • The same discrete-token recipe could extend to 3D structures, spectra, or reaction schemes, since the model only needs a tokenizer that maps the new modality into the LLM's vocabulary.
  • A practical next step would be to measure whether chemists can use the generated images as working outputs (e.g., in patents or electronic lab notebooks) without correcting them, which the current property-based metrics do not capture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper presents ChemMLLM, a chemical multimodal large language model that uses a Mol-VQGAN image tokenizer/de-tokenizer and a Chameleon-based LLM to understand and generate molecules across text, SMILES, and 256x256 molecule images. The authors define five tasks (image captioning, property prediction, image-to-SMILES, property-to-image, and image-to-image optimization), curate datasets from PubChem, ChEBI-20, Mol-Instructions, and TDC, and report benchmarks against general MLLMs, chemical LLMs, and task-specific models. The central claim is that ChemMLLM achieves "superior performance across all evaluated tasks," including a 116.75% improvement over GPT-4o on molecule image optimization.

Significance. If the performance claims were supported, ChemMLLM would be a valuable first step toward a unified chemical multimodal assistant that can both parse and render molecular images, and the released datasets would be a useful resource. The architecture is reasonable, and the ablations in Appendix I indicate that Mol-VQGAN fine-tuning, SFT, and data augmentation each contribute. However, the headline empirical claims are not supported by the evidence as presented, for the reasons in the major comments; the system and dataset contributions remain credible even though the reported evaluations need substantial revision.

major comments (3)
  1. [Section 5.2, Table 6] The claim that ChemMLLM outperforms the best baseline in img2img by 116.75% is based on comparing against GPT-4o (Increased LogP 1.97), but Table 6 shows that Chemformer achieves Increased LogP 3.03, which is higher than GPT-4o. Against the actual strongest baseline in the table, ChemMLLM-7B's 4.27 is about a 41% improvement, not 116.75%. The same misidentification of the best baseline appears in the abstract and Section 1, and the sentence in Section 5.2 should be corrected.
  2. [Section 5.2 (property2img and img2img paragraphs) and Appendix F] The evaluation protocol for the two image-generation tasks is asymmetric and does not support a like-for-like comparison. In property2img, ChemMLLM's image outputs are converted to SMILES via MolScribe before properties are computed, while the MLLM baselines output SMILES directly. In img2img, GPT-4o is given the input molecule as SMILES text instead of the image, because the paper states the API cannot produce optimized SMILES directly from images, while ChemMLLM receives the image and outputs an image. These are different tasks measured through different pipelines, and MolScribe recognition errors are included in ChemMLLM's scores but not in the text-output baselines' scores. The central performance claims for these two tasks therefore need either a common measurement pipeline or a claim explicitly restricted to models that generate images directly.
  3. [Section 5.2 Overview, Table 5, Tables 14-15] The statement that ChemMLLM "outperforms the strongest baseline in all five tasks" is contradicted by the paper's own tables. In img2smiles (Table 5), MolScribe and Decimer achieve higher Tanimoto similarity and accuracy than ChemMLLM-7B and ChemMLLM-34B. In img2property (Tables 14-15), Chemprop achieves higher Pearson correlation and lower MSE/MAE for all seven properties. The text acknowledges these gaps in the img2smiles and img2property paragraphs, yet the overview claim in Section 5.2 and Figure 3 remains unqualified. The claim should be narrowed to comparisons against general MLLMs and chemical LLMs, or the caveat that task-specific models are excluded should be stated explicitly wherever the ``all five tasks'' claim appears.
minor comments (3)
  1. [Appendix F] The evaluation section states that for property2img and img2img, "Each model is run five times, and the best result is reported." Reporting the best of five runs rather than the mean is an optimistic protocol, especially for stochastic generation; please report mean and standard deviation for these tasks as done elsewhere in the paper, or explicitly justify the best-of-five choice.
  2. [Abstract and Section 5.1] The abstract gives an anonymous code URL while Section 5.1 gives a public GitHub URL; these should be unified, and the abstract should match the final public repository link.
  3. [Figure 3 and Section 5.2] The phrase "best baseline" is used without defining which baselines are included per task; please define the baseline set explicitly for each panel of Figure 3 and in the overview paragraph, since the composition varies across tasks.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ChemMLLM is an empirical training-and-benchmark paper; its claimed improvements are measured on held-out test sets and its self-citations are not load-bearing.

full rationale

This paper does not contain a derivation chain in which a predicted quantity is defined in terms of the fitted input. The core components are standard and externally sourced: Mol-VQGAN uses the VQGAN objective of Eq. (3), the language model is trained with the next-token-prediction loss of Eq. (4), and the architecture follows Chameleon and Lumina-mGPT. The five task evaluations are held-out benchmarks, and the property2img and img2img scores are computed by decoding generated images with MolScribe, an external tool, not by a parameter fitted to the ChemMLLM outputs. The self-citations to ChemVLM and ChemLLM appear in data curation templates and as baselines, but the central performance claims do not reduce to those references; the reported gains are measured against external and task-specific models rather than forced by construction. The asymmetric evaluation protocols in property2img and img2img — where ChemMLLM outputs images decoded by MolScribe while baselines emit SMILES text — are a legitimate fairness and correctness concern, but they are not circularity: the evaluation metric is not defined as the fitted quantity, and no equation-level equivalence between input and output is present. No circular step can be quoted, so the appropriate score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on the training data, the evaluation pipeline, and standard MLLM machinery. No new physical entities are introduced. The free parameters are training and data choices made by the authors; the most fragile assumptions are the fidelity of RDKit-rendered images and the MolScribe-based evaluation chain.

free parameters (7)
  • z-loss weight lambda = 1e-5
    Chosen by hand following Lumina-mGPT; analysis in Section J.1 shows only marginal impact.
  • SFT learning rate = 2e-5
    Set for all ChemMLLM variants; AdamW optimizer.
  • VQGAN perceptual loss weight lambda_1 = not reported numerically
    Used in VQGAN objective; balance with reconstruction loss; exact value not given.
  • GAN loss adaptive weight lambda_2 = adaptive (computed dynamically)
    Follows VQGAN to stabilize training.
  • Data augmentation rotations = 90, 180, 270 degrees
    Applied to property2img data only; ablation shows it is critical for performance.
  • Mol-VQGAN training sample size = 1,000,000 molecule images
    Sampled from PubChem by the authors; affects tokenizer quality.
  • Number of training epochs = 2 (VQGAN), 3 (SFT)
    Training schedule choices.
assumptions (4)
  • domain assumption RDKit's Draw.MolToImage and property calculators produce correct molecular images and ground-truth property values.
    All data is synthesized with RDKit; errors here propagate to every task.
  • domain assumption MolScribe reliably converts generated molecule images back to SMILES for evaluation.
    Appendix F states generated images are converted to SMILES via MolScribe; if MolScribe misreads images, measured properties are wrong.
  • domain assumption Template-based prompts represent a fair and diverse set of natural language instructions.
    Data synthesis follows ChemVLM templates; limited template diversity could overstate generalization.
  • standard math Next-token prediction with discretized VQ tokens is an adequate training objective for molecule image generation.
    Standard MLLM objective inherited from Chameleon and Lumina-mGPT.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChemMLLM: Chemical Multimodal Large Language Model." pith.science (2026). https://pith.science/paper/WF5A5HOQ

@misc{pith2026250516326,
  author       = {Pith},
  title        = {Pith review of: ChemMLLM: Chemical Multimodal Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WF5A5HOQ}},
  note         = {Machine review of arXiv:2505.16326}
}
read the original abstract

Multimodal large language models (MLLMs) have made impressive progress in many applications in recent years. However, chemical MLLMs that can handle cross-modal understanding and generation remain underexplored. To fill this gap, we propose ChemMLLM, a unified chemical multimodal large language model for molecule understanding and generation. Also, we design five multimodal tasks across text, molecular SMILES strings, and image, and curate the datasets. We benchmark ChemMLLM against a range of general leading MLLMs and Chemical LLMs on these tasks. Experimental results show that ChemMLLM achieves superior performance across all evaluated tasks. For example, in molecule image optimization task, ChemMLLM outperforms the best baseline (GPT-4o) by 116.75\% (4.27 vs 1.97 property improvement). The code is publicly available at https://github.com/bbsbz/ChemMLLM.git.

Figures

Figures reproduced from arXiv: 2505.16326 by the authors.

Figure 1
Figure 1. Motivation. Comparison between task-specific [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall architecture of ChemMLLM. (a) Image&SMILES tokenizer and de-tokenizer. The image tokenizer employs [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Performance of ChemMLLM-7B, ChemMLLM￾34B and the best baseline on five tasks. Mean Pearson means the mean value of Pearson correlation of seven properties; Avg Sim means Tanimoto similarities; Normalized ∆LogP means normalized (i.e., divided by the maximum value) In￾creased LogP. 4.1 Data Curation We employ RDKit (Landrum et al. 2006) to convert the original SMILES strings into molecular images across all five tasks… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: An example on img2caption task, comparison be [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: A comparison of answers on img2property task on [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 8
Figure 8. Figure 8: Examples of ChemMLLM on img2img task. Model Increased LogP (↑) Diversity (↑) Novelty (↑) valid%(↑) Seq2Seq 1.95 ±0.12 0.79±0.007 1.0±0.0 80.5% Chemformer 3.03 ±0.08 0.85 ±0.002 0.93 ±0.01 100% Qwen-VL-Chat (7B) 1.50 ±0.93 0.95±0.01 1.0±0.0 4.0% InternVL-Chat-v1.5 (20B)…
Figure 7
Figure 7. Figure 7: Examples on property2img task on our ChemM [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: Metric bar for different runs of img2caption task. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Metric bar for different runs of img2property [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 12
Figure 12. Figure 12: Metric bar for different runs of img2smiles task. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Metric bar for different runs of property2img [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: More examples for property2img task. σy = PN i=1(yi − 1 N PN j=1 yj ) 2 and σyb = PN i=1(ybi − 1 N PN j=1 ybj ) 2 are the standard deviations of ground truth and prediction, respectively. The value ranges from -1 to 1. A higher Pearson correlation value indicates bett…
Figure 15
Figure 15. Figure 15: More examples for img2img task [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Training curve from start to the best checkpoint, [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Examples of origin VQGAN and Mol-VQGAN. The origin images are shown in the first row. As shown in the second [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Results on img2caption task with different [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?

    cs.AI 2025-06 conditional novelty 7.0 of 10

    A new benchmark called ToxiMol evaluates how well 43 multimodal LLMs can edit toxic molecules into structurally similar, non-toxic, drug-like candidates; the best model succeeds on 43.3% of tasks.

  2. MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    MolSight integrates a Molecular Topology Module and Molecular Grounding Module into VLMs to enhance molecular image understanding and claims to outperform prior models on chemical visual tasks.

  3. LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning

    physics.chem-ph 2026-02 conditional novelty 6.0 of 10

    LatentChem reasons in continuous latent space for chemistry, achieving a 59.88% non-tie win rate over explicit CoT on ChemCoTBench with a 10.84x average reduction in reasoning overhead.

  4. Reasoning-Driven Retrosynthesis Prediction with Large Language Models via Reinforcement Learning

    cs.CE 2025-07 conditional novelty 6.0 of 10

    RetroDFM-R, a ChemDFM-based LLM trained with reasoning distillation and reinforcement learning, reaches 65.0% top-1 retrosynthesis accuracy on USPTO-50K.

Reference graph

Works this paper leans on

58 extracted references · 19 canonical work pages · cited by 4 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198

    arXiv 2022

  4. [4]

    D.; and Weininger, D

    Anderson, E.; Veith, G. D.; and Weininger, D. 1987. SMILES , a line notation and computerized interpreter for chemical structures . US Environmental Protection Agency, Environmental Research Laboratory

    work page 1987

  5. [5]

    Awadalla, A.; Gao, I.; Gardner, J.; Hessel, J.; Hanafy, Y.; Zhu, W.; Marathe, K.; Bitton, Y.; Gadre, S.; Sagawa, S.; et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390

    arXiv 2023

  6. [6]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, Z.; Wang, P.; Chen, J.; Zhou, J.; and Zhou, J. 2023. Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  7. [7]

    O'Reilly Media, Inc

    Bird, S.; Klein, E.; and Loper, E. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc."

    work page 2009

  8. [8]

    Cai, Z.; Cao, M.; Chen, H.; Chen, K.; Chen, K.; Chen, X.; Chen, X.; Chen, Z.; Chen, Z.; Chu, P.; et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

    arXiv 2024

Show all 58 references
  1. [9]

    Cao, H.; Liu, Z.; Lu, X.; Yao, Y.; and Li, Y. 2023. Instructmol: Multi-modal integration for building a versatile and reliable molecular assistant in drug discovery. arXiv preprint arXiv:2311.16208

    2023 arXiv

  2. [10]

    Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; Li, B.; Luo, P.; Lu, T.; Qiao, Y.; and Dai, J. 2023. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. arXiv preprint arXiv:2312.14238

    2023 arXiv

  3. [11]

    Cho, K.; Van Merri \"e nboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

    2014 arXiv

  4. [12]

    W.; Sutton, C.; Gehrmann, S.; et al

    Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1--113

    2023

  5. [13]

    Du, Y.; Fu, T.; Sun, J.; and Liu, S. 2022. MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design. arXiv preprint arXiv:2203.14500

    2022 arXiv

  6. [14]

    Edwards, C.; Lai, T.; Ros, K.; Honke, G.; Cho, K.; and Ji, H. 2022. Translation between molecules and natural language. arXiv preprint arXiv:2204.11817

    2022 arXiv

  7. [15]

    Esser, P.; Rombach, R.; and Ommer, B. 2021. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12873--12883

    2021

  8. [16]

    Fang, Y.; Liang, X.; Zhang, N.; Liu, K.; Huang, R.; Chen, Z.; Fan, X.; and Chen, H. 2023. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.08018

    2023 arXiv

  9. [17]

    Fu, T.; Xiao, C.; and Sun, J. 2020. CORE : Automatic Molecule Optimization using Copy and Refine Strategy. AAAI

    2020

  10. [18]

    B.; Siegel, C.; Vishnu, A.; Hodas, N

    Goh, G. B.; Siegel, C.; Vishnu, A.; Hodas, N. O.; and Baker, N. 2017. Chemception: a deep neural network with minimal chemistry knowledge matches the performance of expert-developed QSAR/QSPR models. arXiv preprint arXiv:1706.06689

    2017 arXiv

  11. [19]

    P.; Chung, Y.; Li, S.-C.; Graff, D

    Heid, E.; Greenman, K. P.; Chung, Y.; Li, S.-C.; Graff, D. E.; Vermeire, F. H.; Wu, H.; Green, W. H.; and McGill, C. J. 2023. Chemprop: a machine learning package for chemical property prediction. Journal of Chemical Information and Modeling, 64(1): 9--17

    2023

  12. [20]

    W.; Xiao, C.; Sun, J.; and Zitnik, M

    Huang, K.; Fu, T.; Gao, W.; Zhao, Y.; Roohani, Y.; Leskovec, J.; Coley, C. W.; Xiao, C.; Sun, J.; and Zitnik, M. 2021. Therapeutics data Commons: machine learning datasets and tasks for therapeutics. NeurIPS Track Datasets and Benchmarks

    2021

  13. [21]

    Irwin, R.; Dimitriadis, S.; He, J.; and Bjerrum, E. J. 2022. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1): 015022

    2022

  14. [22]

    Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1125--1134

    2017

  15. [23]

    Jin, W.; Yang, K.; Barzilay, R.; and Jaakkola, T. 2018. Learning multimodal graph-to-graph translation for molecular optimization. arXiv preprint arXiv:1812.01070

    2018 arXiv

  16. [24]

    T.; Jammalamadaka, N.; Huang, J.; Yuen, H.; et al

    Kalamkar, D.; Mudigere, D.; Mellempudi, N.; Das, D.; Banerjee, K.; Avancha, S.; Vooturi, D. T.; Jammalamadaka, N.; Huang, J.; Yuen, H.; et al. 2019. A study of BFLOAT16 for deep learning training. arXiv preprint arXiv:1905.12322

    2019 arXiv

  17. [25]

    A.; Thiessen, P

    Kim, S.; Chen, J.; Cheng, T.; Gindulyte, A.; He, J.; He, S.; Li, Q.; Shoemaker, B. A.; Thiessen, P. A.; Yu, B.; et al. 2021. PubChem in 2021: new data content and improved web interfaces. Nucleic acids research, 49(D1): D1388--D1395

    2021

  18. [26]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  19. [27]

    Kosenkov, Y.; and Kosenkov, D. 2021. Computer vision in chemistry: Automatic titration

    2021

  20. [28]

    Landrum, G.; et al. 2006. RDKit: Open-source cheminformatics

    2006

  21. [29]

    Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461

    2019 arXiv

  22. [30]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

    2023

  23. [31]

    Li, J.; Zhang, D.; Wang, X.; Hao, Z.; Lei, J.; Tan, Q.; Zhou, C.; Liu, W.; Yang, Y.; Xiong, X.; et al. 2025. Chemvlm: Exploring the power of multimodal large language models in chemistry area. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 415--423

    2025

  24. [32]

    Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74--81

    2004

  25. [33]

    Liu, D.; Zhao, S.; Zhuo, L.; Lin, W.; Qiao, Y.; Li, H.; and Gao, P. 2024. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.02657

    2024 arXiv

  26. [34]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems, 36: 34892--34916

    2023

  27. [35]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  28. [36]

    J.; Chen, L.; Saylor, G.; Van Eyk, J

    Lu, Y.; Wu, C.-T.; Parker, S. J.; Chen, L.; Saylor, G.; Van Eyk, J. E.; Herrington, D. M.; and Wang, Y. 2021. COT : an efficient Python tool for detecting marker genes among many subtypes. bioRxiv, 2021--01

    2021

  29. [37]

    Y.; Nie, Z.; Zhou, H.; and Nie, Z

    Luo, Y.; Yang, K.; Hong, M.; Liu, X. Y.; Nie, Z.; Zhou, H.; and Nie, Z. 2024. Learning multi-view molecular representations with structured and unstructured knowledge. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2082--2093

    2024

  30. [38]

    OpenAI. 2023. GPT-4V(ision) System Card. Accessed: 2024-07-20

    2023

  31. [39]

    OpenAI. 2024. GPT-4o: Our most advanced AI model. Accessed: 2024-07-20

    2024

  32. [40]

    W.; and Barzilay, R

    Qian, Y.; Guo, J.; Tu, Z.; Li, Z.; Coley, C. W.; and Barzilay, R. 2023. MolScribe : Robust Molecular Structure Recognition with Image-to-Graph Generation. Journal of Chemical Information and Modeling

    2023

  33. [41]

    Rajan, K.; Zielesny, A.; and Steinbeck, C. 2020. DECIMER: towards deep learning for chemical image recognition. Journal of Cheminformatics, 12(1): 65

    2020

  34. [42]

    Sennrich, R.; Haddow, B.; and Birch, A. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909

    2015 arXiv

  35. [43]

    Software, O. S. 2023. OpenEye Toolkits Documentation

    2023

  36. [44]

    Sun, P.; Jiang, Y.; Chen, S.; Zhang, S.; Peng, B.; Luo, P.; and Yuan, Z. 2024. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525

    2024 arXiv

  37. [45]

    Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2818--2826

    2016

  38. [46]

    Team, C. 2024. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818

    2024 arXiv

  39. [47]

    M.; Hauth, A.; Millican, K.; et al

    Team, G.; Anil, R.; Borgeaud, S.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; Millican, K.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  40. [48]

    Van Den Oord, A.; Vinyals, O.; et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30

    2017

  41. [49]

    Wang, X.; Zhang, X.; Luo, Z.; Sun, Q.; Cui, Y.; Wang, J.; Zhang, F.; Wang, Y.; Li, Z.; Yu, Q.; et al. 2024. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869

    2024 arXiv

  42. [50]

    Wang, Z.; Chen, Y.; Ma, P.; Yu, Z.; Wang, J.; Liu, Y.; Ye, X.; Sakurai, T.; and Zeng, X. 2025. Image-based generation for molecule design with SketchMol. Nature Machine Intelligence, 1--12

    2025

  43. [51]

    Weininger, D. 1988. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1): 31--36

    1988

  44. [52]

    J.; Wang, W.; Lin, K

    Xie, J.; Mao, W.; Bai, Z.; Zhang, D. J.; Wang, W.; Lin, K. Q.; Gu, Y.; Chen, Z.; Yang, Z.; and Shou, M. Z. 2024. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528

    2024 arXiv

  45. [53]

    Zhang, D.; Liu, W.; Tan, Q.; Chen, J.; Yan, H.; Yan, Y.; Li, J.; Huang, W.; Yue, X.; Ouyang, W.; et al. 2024 a . Chemllm: A chemical large language model. arXiv preprint arXiv:2402.06852

    2024 arXiv

  46. [54]

    Zhang, J.; Bian, Y.; Chen, Y.; and Yao, Q. 2024 b . Unimot: Unified molecule-text language model with discrete token representation. arXiv preprint arXiv:2408.00863

    2024 arXiv

  47. [55]

    A.; Shechtman, E.; and Wang, O

    Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586--595

    2018

  48. [56]

    Zhao, Y.; Gu, A.; Varma, R.; Luo, L.; Huang, C.-C.; Xu, M.; Wright, L.; Shojanazeri, H.; Ott, M.; Shleifer, S.; et al. 2023. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277

    2023 arXiv

  49. [57]

    Zhou, C.; Yu, L.; Babu, A.; Tirumala, K.; Yasunaga, M.; Shamis, L.; Kahn, J.; Ma, X.; Zettlemoyer, L.; and Levy, O. 2024. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039

    2024 arXiv

  50. [58]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.