Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces Optical Chemical Structure Understanding (OCSU) and presents evidence that a single vision-language model, Mol-VL-7B, captions molecular structure diagrams at multiple levels, beating recognition-based cascades on…

desk verdict A useful benchmark and a fair two-paradigm comparison, with one overstated practical-transfer claim. read the letter →

arxiv 2501.15415 v2 pith:F3KA37ZJ submitted 2025-01-26 cs.CV

Siqi Fan , Yuguang Xie , Bowen Cai , Ailin Xie , Gaochao Liu , Mu Qiao , Jie Xing , Zaiqing Nie This is my paper · ORCID
classification cs.CV
keywords opticalchemicalstructureunderstandingOCSUvision-languagemodelrecognitionmoleculeimagecaptioningfunctionalgroupIUPACnamingVis-CheBI20
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Optical Chemical Structure Understanding (OCSU), an image-caption task that reads a molecular structure diagram and outputs strings at three levels: functional groups, plain-language molecule description, IUPAC name, and SMILES. To support it, the authors construct Vis-CheBI20, a dataset of 29.7K RDKit-generated diagrams with 117.7K image-text pairs spanning these four subtasks. They test two technical routes: an OCSR-based cascade that first converts the image to SMILES via an improved recognizer called DoubleCheck and then uses SMILES-based molecule understanding, and an OCSR-free vision-language model called Mol-VL fine-tuned end-to-end on all tasks. The central result is that Mol-VL-7B, the end-to-end model, matches or beats the cascade on most tasks, most clearly on functional-group captioning where it leads the strongest baseline by 7.7 F1 points, despite lagging on IUPAC-naming recall. A sympathetic reading is that end-to-end visual understanding of chemical diagrams is viable without an explicit recognition step, which matters for any downstream application that starts from diagrams in papers and patents.

What carries the argument

The two load-bearing mechanisms are DoubleCheck and Mol-VL. DoubleCheck is an OCSR module built on MolScribe's image-to-graph generation: after an autoregressive atom predictor emits candidate atoms, atoms with low prediction confidence get a second look — a 2D Gaussian mask crops a local region, a Swin-B encoder produces a local feature, and a learned MLP gate fuses that aligned local feature into the global graph feature via $F_e = F_g + \mathrm{MLP}(F_g \oplus \hat{F}_l) * \hat{F}_l$ to improve atom and bond prediction. Mol-VL is a Qwen2-VL vision-language model fine-tuned on Vis-CheBI20 with a multitask objective over the four caption tasks plus an auxiliary functional-group-recognition task, so the same model learns to attend from motifs to whole-molecule context to abstract naming.

What would settle it

Run Mol-VL-7B on the paper's own real-image OCSR benchmarks (USPTO patents and ACS journal diagrams) and measure functional-group captioning and molecule-description quality: if its scores fall below the DoubleCheck cascade on those images, the claim that end-to-end understanding generalizes beyond synthetic diagrams is falsified.

Watch

Extended reading notes

Core claim

The paper's claim is that OCSU — translating a chemical structure diagram into chemist- and machine-readable strings at multiple levels — can be solved both by a two-stage cascade built around a stronger OCSR module and by an end-to-end vision-language model, and that the end-to-end model is generally the stronger of the two. Concretely, Mol-VL-7B, a Qwen2-VL-based model fine-tuned on the Vis-CheBI20 training set, achieves the best F1 of 97.32% on functional-group captioning, 3.69 points above the DoubleCheck-plus-RDKit cascade, and the best vision-input scores on molecule description (BLEU-4 46.14, BERTScore-F1 61.51) and IUPAC BLEU-based precision, while trailing the cascade on IUPAC recall (ROUGE scores). The improved recognizer DoubleCheck, which re-examines locally ambiguous atoms with an attention-weighted fusion of local and global features, lifts OCSR accuracy by about 2.3 points over its MolScribe backbone on average and also improves the cascade's downstream caption scores. The paper reads this as evidence that both paradigms are useful, with the OCSR-free one holding greater long-term potential.

Load-bearing premise

The entire Vis-CheBI20 benchmark is generated by RDKit, and the paper assumes these synthetic-looking diagrams are representative enough of real journal and patent diagrams that performance on them transfers to practical use.

Editorial extensions

If this is right

  • A single fine-tuned vision-language model can handle all four OCSU subtasks at once, so one model replaces a cascade of recognition plus task-specific SMILES-based tools.
  • Improving the OCSR submodule improves downstream caption quality: DoubleCheck lifts the cascade's molecule-description scores by about 0.81% and IUPAC scores by about 2.21% over MolScribe.
  • Vis-CheBI20's public release provides a standard benchmark for training and comparing future OCSU methods, covering structural motif, molecule-level, and abstract naming levels.
  • End-to-end multi-task training lets the model leverage structural context (functional groups) better than atom-focused OCSR, which shows in the 3.69-point F1 advantage on functional-group captioning.
  • The paper's published error analysis (Appendix D) shows that long-tail functional groups drive most of Mol-VL-7B's remaining errors, so error rates track training frequency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Vis-CheBI20 images are purely RDKit-generated, the reported Mol-VL gains are formally about synthetic diagrams; the paper validates transfer to real patent and journal images only for DoubleCheck, so a charitable reader should treat Mol-VL's real-world superiority as an open question until tested on USPTO and ACS images.
  • The IUPAC result — OCSR-free leads on BLEU precision while OCSR-based leads on ROUGE recall — suggests that a hybrid that generates names from the VLM and also retrieves from PubChem using its own predicted SMILES could outperform both, a direction the paper itself hints at.
  • The long-tail error pattern suggests that targeted augmentation or retrieval over a functional-group ontology could extend the gains more efficiently than simply scaling the dataset, since the paper's own analysis ties errors to low training frequency.
  • The same end-to-end formulation could plausibly extend to Markush structures and structure-based reasoning tasks that the paper names only as future work, because the VLM's attention is not constrained by atom-level token boundaries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces the Optical Chemical Structure Understanding (OCSU) task, which extends traditional optical chemical structure recognition (OCSR) to multiple caption subtasks: functional group caption, molecule description, IUPAC naming, and SMILES naming. The authors construct Vis-CheBI20, a large-scale dataset of about 29.7K molecular diagrams with 117.7K image-text pairs, generated by RDKit from SMILES strings and using ChEBI-20 descriptions, PubChem IUPAC names, and an expert-defined functional group ontology. They propose two approaches: DoubleCheck, an OCSR-based method that augments MolScribe with an attentive local-global feature enhancement module for ambiguous atoms, and Mol-VL, an end-to-end vision-language model based on Qwen2-VL. The paper reports that Mol-VL-7B achieves state-of-the-art results on Vis-CheBI20 for functional group caption, molecule description, and IUPAC naming (except recall on IUPAC), while DoubleCheck improves over MolScribe on OCSR benchmarks including USPTO and ACS.

Significance. The paper makes three valuable contributions: a new task formulation that expands OCSR to multi-level understanding; a large, open-sourced dataset that will likely become a useful benchmark; and a systematic comparison of two technical paradigms (cascaded recognition-and-retrieval versus end-to-end VLM). The authors ship code, models, and data, which supports reproducibility. If the reported results are robust, the study provides a solid foundation for future work on chemical structure understanding. However, the two main technical claims—the effectiveness of the DoubleCheck enhancement and the practical transferability of the Mol-VL results—are weakened by experimental design issues and unsupported extrapolations, respectively.

major comments (3)
  1. [Section 5.1 and 5.5] The comparison between DoubleCheck and MolScribe in Fig. 3 and Table 7 is confounded by training budget. DoubleCheck is trained for 30 epochs in stage 1 plus 10 additional epochs in stage 2 (Sec 5.1), while the MolScribe baseline is not retrained for the same total number of steps. The reported gains (e.g., +2.27% average on Vis-CheBI20) could therefore be due to the extra 10 epochs of optimization rather than the proposed attentive feature enhancement. The authors should include a control where MolScribe is trained for 40 epochs without the augmentation, or equivalently train DoubleCheck for 30 epochs total, to isolate the effect of the mechanism.
  2. [Section 3.2 and 5.5] The statement that Vis-CheBI20 'ensures that performance on Vis-CheBI20 reflects capabilities in practical applications' (Sec 3.2) is too strong. The dataset consists exclusively of RDKit-generated images, and the only evidence of transfer to real-world diagrams is provided for DoubleCheck (USPTO, ACS, Staker_p in Sec 5.5 and Appendix B). Mol-VL, which is the source of the headline gains (e.g., +7.7 F1 on functional group caption), is never evaluated on a real-world image. The authors should either evaluate Mol-VL on a sample of real patent/journal diagrams or explicitly state that the practical transferability of the OCSR-free paradigm remains unverified.
  3. [Section 5.3 and 3.2] The functional group caption evaluation does not specify how the free-form text outputs of Mol-VL are converted into a set of predicted functional groups for computing precision, recall, and F1. In contrast, the OCSR-based method uses RDKit substructure matching, which is well-defined. Without a description of the parsing procedure, the reported F1 scores for Mol-VL are not reproducibly computable, and comparisons across models may be inconsistent. The authors should detail this step in the main text or appendix.
minor comments (5)
  1. [Section 4.2] The word 'auto-agressive' should be 'autoregressive'.
  2. [Section 5.1] The phrase 'the first10% steps' is missing a space; it should read 'the first 10% of steps'.
  3. [Section 5.5] The heading 'Abation study on DoubleCheck' should be 'Ablation study on DoubleCheck'.
  4. [Section 3.2] The word 'metircs' should be 'metrics'.
  5. [Section 5.5] The sentence 'consistent with that on other realistic dataset' should use the plural 'datasets'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is an empirical benchmark comparison with external data sources; the RDKit-only generalizability concern is an extrapolation, not a circular loop.

full rationale

No circularity found. The derivation chain is empirical rather than definitional: Vis-CheBI20 is constructed from ChEBI-20 descriptions, PubChem IUPAC names, and RDKit functional group extractions, all external to the proposed models. DoubleCheck is a MolScribe backbone augmented with a local re-check module, and its evaluation against MolScribe on USPTO, ACS, and Staker_p is an external benchmark comparison; no parameter is fitted to those test sets. Mol-VL is a finetuned Qwen2-VL evaluated on held-out molecules under standard splits; training and test images come from the same RDKit generation pipeline, but the ground-truth texts and labels are not produced by the proposed models. The claimed improvements, e.g., the +7.7 F1 gain on functional group caption, are measured on a held-out test set and are not baked in by construction. The only closely related self-citations, MolFM and BioMedGPT, appear as weak baselines and are not load-bearing for the headline results; the decisive comparisons are against Qwen2-VL and MolScribe/BioT5+, which are external systems. The assertion in Sec. 3.2 that RDKit-generated images "ensures that performance on Vis-CheBI20 reflects capabilities in practical applications" is an unsupported extrapolation, and the paper itself partially tests this only for DoubleCheck on USPTO and ACS; Mol-VL's real-world transfer is untested. That is a generalizability/correctness concern, not a circularity, because Vis-CheBI20 is not defined in terms of Mol-VL's outputs and the evaluation does not reduce to the model's own training objective.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. OCSU is a task definition; DoubleCheck and Mol-VL are systems assembled from existing components (MolScribe, Swin-B, Qwen2-VL). The hand-chosen functional group ontology and the extra DoubleCheck training epochs are the clearest design choices that affect the reported numbers.

free parameters (2)
  • Functional group ontology scope = 59 general + 106 cyclic groups
    Expert-curated label space for the functional group caption task; it defines the achievable F1 and is a design choice, not fitted to test data.
  • DoubleCheck second-stage training epochs = 10 epochs (after 30 first-stage epochs)
    Chosen by the authors; because the MolScribe baseline is only trained 30 epochs, this extra training confounds the DoubleCheck-vs-MolScribe comparison.
assumptions (6)
  • domain assumption RDKit-generated diagrams are representative of real-world chemical structure images.
    Used to build Vis-CheBI20; Sec 3.2 asserts this ensures practical applicability, but real-image validation is only provided for DoubleCheck, not for Mol-VL.
  • domain assumption SMILES is an information-lossless enough representation that the factorization P(T|I)=P(S|I)P(T|S) preserves caption quality.
    Sec 3.1 defines the OCSR-based paradigm with this factorization; any structure lost in image-to-SMILES conversion cannot be recovered in the text stage.
  • domain assumption PubChem IUPAC names retrieved by exact SMILES matching are valid ground truth for IUPAC naming.
    Used both to build the dataset and as the OCSR-based inference lookup (Sec 5.4), making the IUPAC task partly a database retrieval problem.
  • domain assumption ChEBI-20 description pairs are trustworthy molecule-level captions.
    Adopted directly as the Vis-CheBI20 molecule description source (Sec 3.2) without independent verification.
  • domain assumption Pretrained Qwen2-VL provides a suitable initialization for chemical diagram understanding.
    Mol-VL relies on this; no ablation with alternative base VLMs is provided.
  • domain assumption The expert-defined functional group scope (59 general, 106 cyclic) is an adequate ontology for functional group captioning.
    Defines the label space for the main headline result; a different scope would change F1 scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery." pith.science (2026). https://pith.science/paper/F3KA37ZJ

@misc{pith2026250115415,
  author       = {Pith},
  title        = {Pith review of: OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3KA37ZJ}},
  note         = {Machine review of arXiv:2501.15415}
}
read the original abstract

Understanding the chemical structure from a graphical representation of a molecule is a challenging image caption task that would greatly benefit molecule-centric scientific discovery. Variations in molecular images and caption subtasks pose a significant challenge in both image representation learning and task modeling. Yet, existing methods only focus on a specific caption task that translates a molecular image into its graph structure, i.e., OCSR. In this paper, we propose the Optical Chemical Structure Understanding (OCSU) task, which extends low-level recognition to multilevel understanding and aims to translate chemical structure diagrams into readable strings for both machine and chemist. To facilitate the development of OCSU technology, we explore both OCSR-based and OCSR-free paradigms. We propose DoubleCheck to enhance OCSR performance via attentive feature enhancement for local ambiguous atoms. It can be cascaded with existing SMILES-based molecule understanding methods to achieve OCSU. Meanwhile, Mol-VL is a vision-language model end-to-end optimized for OCSU. We also construct Vis-CheBI20, the first large-scale OCSU dataset. Through comprehensive experiments, we demonstrate the proposed approaches excel at providing chemist-readable caption for chemical structure diagrams, which provide solid baselines for further research. Our code, model, and data are open-sourced at https://github.com/PharMolix/OCSU.

Figures

Figures reproduced from arXiv: 2501.15415 by the authors.

Figure 1
Figure 1. Introduction of OCSU. (a) Example of OCSU task. Optical chemical structure under￾standing is a special image caption task that describes the molecular diagrams from multiple levels, including four typical subtasks, i.e., functional group caption, molecule description, chemist-readable IUPAC naming, and machine-readable SMILES naming. (b) Two technical paradigms for OCSU. OCSR-based paradigm can fully leverage the po… view at source ↗
Figure 2
Figure 2. Exploration on OCSR-based and OCSR-free paradigms for OCSU. (a) Architecture of DoubleCheck. An attentive feature enhancement module is introduced for local ambiguous atoms. (b) Architecture of Mol-VL. A vision-language model is end-to-end optimized via multi-task learning. 4 Method In this section, we explore both OCSR-based and OCSR-free paradigms. Specifically, (1) we introduce attentive feature enhancement mecha… view at source ↗
Figure 3
Figure 3. Performance evaluation on OCSR. (1) Performance on Vis-CheBI20. The performance advantage of DoubleCheck demonstrate the effectiveness of the proposed feature enhancement mechanism. (2) Performance on USPTO. DoubleCheck outperforms MolScribe on real-world patent scenario. (3) Performance on ACS. DoubleCheck surpasses MolScribe on real-world journal scenario. Abation study on DoubleCheck. We propose DoubleCheck to im… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: A qualitative example of Mol-VL-7B and an application example in practical scenario. Qualitative analysis of Mol-VL. We provide a qualitative example of Mol-VL-7B and present a practical scenario of OCSU, as shown in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Statistical analysis on functional groups. We have separately counted the distribution of functional groups in the training set and the test set. The most common functional groups in the training set and the test set are shown in (a) and (b), respectively. For the “oth…
Figure 6
Figure 6. Figure 6: The results reveal that: (1) the majority of descriptions have word counts ranging from 20 to [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 6
Figure 6. Figure 6: Statistical analysis on text length of molecule description. (a) Statistical analysis of training set. (b) Statistical analysis of test set [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Statistical analysis on atom number of molecule. (a) Statistical analysis of training set. (b) Statistical analysis of test set. 30, with the vast majority containing fewer than 100 words; and (2) the text length distribution in the training set and the test set is ess…
Figure 8
Figure 8. Figure 8: More case studies of optical chemical structure understanding on Vis-CheBI20. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Error analysis of Mol-VL-7B on functional group caption. Mol-VL-7B achieves an F1 score of 97.32% on the functional group caption task. To gain deeper insights, we conduct a detailed statistical analysis of the error cases, as presented in [PITH_FULL_IMAGE:figures/ful…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MinerU.Chem: A High-Precision System for Optical Chemical Structure and Reaction Recognition

    cs.CV 2026-08 conditional novelty 5.0 of 10

    MinerU.Chem reports 93.02% SMILES exact-match accuracy on a subset of MolRecBench-Wild and 79.66% graph accuracy on the full set.

Reference graph

Works this paper leans on

36 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  2. [2]

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  3. [3]

    Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    arXiv 2024

  4. [4]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024

    work page 2024

  5. [5]

    Clevert, T

    D.-A. Clevert, T. Le, R. Winter, and F. Montanari. Img2Mol: accurate smiles recognition from molecular graphical depictions. Chemical Science, 12(42):14174–14181, 2021

    work page 2021

  6. [6]

    Patch n’ Pack: Navit, a vision transformer for any aspect ratio and resolution

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, An- dreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Lucic, and Neil Houlsby. Patch n’ Pack: Navit, a vision transformer for any aspect ratio and resolution. In Advances in Neura...

    work page 2023

  7. [7]

    Edwards, C

    C. Edwards, C. Zhai, and H. Ji. Text2Mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 595–607, 2021

    work page 2021

  8. [8]

    Y . Fang, X. Liang, N. Zhang, K. Liu, R. Huang, Z. Chen, X. Fan, and H. Chen. Mol-Instructions: A large-scale biomolecular instruction dataset for large language models. In ICLR, 2024

    work page 2024

Show all 36 references
  1. [9]

    H. Feng, Q. Liu, H. Liu, J. Tang, W. Zhou, H. Li, and C. Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. Science China Information Sciences, 67(12):1–14, 2024

    2024

  2. [10]

    I. V . Filippov and M. C. Nicklaus. Optical structure recognition software to recover chemical information: Osra, an open source solution, 2009

    2009

  3. [11]

    A. Hu, H. Xu, J. Ye, M. Yan, L. Zhang, B. Zhang, C. Li, J. Zhang, Q. Jin, F. Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895, 2024

    2024 arXiv

  4. [12]

    Landrum, P

    G. Landrum, P. Tosco, B. Kelley, R. Rodriguez, D. Cosgrove, R. Vianello, sriniker, P. Gedeck, G. Jones, NadineSchneider, E. Kawashima, D. Nealschneider, A. Dalke, M. Swain, B. Cole, S. Turk, A. Savelev, A. Vaucher, M. Wójcikowski, I. Take, tadhurst cdd, V . F. Scalfani, R. Wal...

    2024

  5. [13]

    K. Lee, M. Joshi, I. R. Turc, H. Hu, F. Liu, J. M. Eisenschlos, U. Khandelwal, P. Shaw, M.-W. Chang, and K. Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In ICML, pages 18893–18912. PMLR, 2023

    2023

  6. [14]

    S. Li, Z. Liu, Y . Luo, X. Wang, X. He, K. Kawaguchi, T.-S. Chua, and Q. Tian. Towards 3d molecule-text interpretation in language models. In ICLR, 2024

    2024

  7. [15]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  8. [16]

    P. Liu, Y . Ren, J. Tao, and Z. Ren. Git-mol: A multi-modal large language model for molecular science with graph, image, and text. Computers in biology and medicine, 171:108073, 2024

    2024

  9. [17]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021

    2021

  10. [18]

    Z. Liu, S. Li, Y . Luo, H. Fei, Y . Cao, K. Kawaguchi, X. Wang, and T.-S. Chua. MolCA: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15623–156...

    2023

  11. [19]

    Y . Luo, K. Yang, M. Hong, X. Y . Liu, and Z. Nie. MolFM: A multimodal molecular foundation model. arXiv preprint arXiv:2307.09484, 2023

    2023 arXiv

  12. [20]

    Y . Luo, J. Zhang, S. Fan, K. Yang, M. Hong, Y . Wu, M. Qiao, and Z. Nie. BioMedGPT: An open multimodal large language model for biomedicine. IEEE Journal of Biomedical and Health Informatics, 2024

    2024

  13. [21]

    Morin, M

    L. Morin, M. Danelljan, M. I. Agea, A. Nassar, V . Weber, I. Meijer, P. Staar, and F. Yu. MolGrapher: Graph-based visual recognition of chemical structures. In ICCV, pages 19552–19561, 2023

    2023

  14. [22]

    Pubchem, 2024

    National Institutes of Health (NIH). Pubchem, 2024

    2024

  15. [23]

    Oldenhof, E

    M. Oldenhof, E. De Brouwer, A. Arany, and Y . Moreau. Atom-level optical chemical structure recognition with limited supervision. In CVPR, pages 17669–17678, 2024

    2024

  16. [24]

    Q. Pei, W. Zhang, J. Zhu, K. Wu, K. Gao, L. Wu, Y . Xia, and R. Yan. BioT5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  17. [25]

    Q. Pei, L. Wu, K. Gao, X. Liang, Y . Fang, J. Zhu, S. Xie, T. Qin, and R. Yan. BioT5+: Towards generalized biological understanding with iupac integration and multi-task tuning. arXiv preprint arXiv:2402.17810, 2024

    2024 arXiv

  18. [26]

    Peryea, D

    T. Peryea, D. Katzel, T. Zhao, N. Southall, and D.-T. Nguyen. Molvec v0.9.8, 2022

    2022

  19. [27]

    Y . Qian, J. Guo, Z. Tu, Z. Li, C. W. Coley, and R. Barzilay. MolScribe: robust molecular structure recognition with image-to-graph generation. Journal of Chemical Information and Modeling , 63(7): 1925–1934, 2023

    1925

  20. [28]

    Rajan, H

    K. Rajan, H. O. Brinkhaus, A. Zielesny, and C. Steinbeck. A review of optical chemical structure recognition tools. Journal of Cheminformatics, 12:1–13, 2020

    2020

  21. [29]

    Rajan, A

    K. Rajan, A. Zielesny, and C. Steinbeck. DECIMER: towards deep learning for chemical image recognition. Journal of Cheminformatics, 12(1):65, 2020

    2020

  22. [30]

    Smolov, F

    V . Smolov, F. Zentsev, and M. Rybalkin. Imago: Open-source toolkit for 2d chemical structure image recognition. In TREC, 2011

    2011

  23. [31]

    B. Su, D. Du, Z. Yang, Y . Zhou, J. Li, A. Rao, H. Sun, Z. Lu, and J.-R. Wen. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.05481, 2022

    2022 arXiv

  24. [32]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-VL: Enhanc- ing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  25. [33]

    Weininger

    D. Weininger. SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988

    1988

  26. [34]

    J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, G. Xu, C. Li, J. Tian, Q. Qian, J. Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023

    arXiv

  27. [35]

    S. Yoo, O. Kwon, and H. Lee. Image-to-graph transformers for chemical structure recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3393–3397. IEEE, 2022. A Details of Vis-CheBI20 A.1 Data Examples We present data examples f...

    2022

  28. [36]

    ACS [27] is the only OCSR benchmark with 331 molecular images collected from American Chemical Society publications, which exhibit greater diversity in drawing styles

    is a widely adopted patent benchmark comprising 5,719 molecular images. ACS [27] is the only OCSR benchmark with 331 molecular images collected from American Chemical Society publications, which exhibit greater diversity in drawing styles. Following [5], we also evaluate robus...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.