Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

CAST: Cross Attention based multimodal fusion of Structure and Text for materials property prediction

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that CAST, a cross-attention model fusing atom-level graph features with text tokens and pretrained with masked node prediction, outperforms unimodal and prior multimodal baselines on four material properties, with…

desk verdict A fair, carefully evaluated multimodal method for materials property prediction, with a real reproducibility gap on text truncation and an abstract that overstates per-property wins. read the letter →

arxiv 2502.06836 v2 pith:SRSMBXMN submitted 2025-02-06 cs.LG cond-mat.mtrl-scics.AI

Jaewan Lee , Changyoung Park , Hongjun Yang , Sungbin Lim , Woohyung Lim , Sehui Han This is my paper · ORCID
classification cs.LGcond-mat.mtrl-scics.AI MSC 68T07
keywords cross-attentionmultimodalfusionmaterialspropertypredictiongraphneuralnetworksmaskednoderobocrystallographerMatSciBERTcrystalstructure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CAST is proposed as a way to give graph neural networks the global crystal information they miss. The paper claims that fusing atom-level graph embeddings with token-level embeddings of automatically generated text summaries of the same crystals, through cross-attention rather than whole-vector concatenation, produces consistently better predictions of formation energy, band gap, bulk modulus, and shear modulus. A masked node prediction pretraining step, which hides half the atoms and forces the model to recover them from text and graph context, is what makes the attention patterns informative. On the four tasks, the reported average relative MAE improvements over six baselines range from 10.2% to 35.7%. A careful reader would care because this offers a recipe for injecting global structural context into structure-based models without hand-engineering descriptors.

What carries the argument

The central mechanism is cross-attention: four transformer attention layers with eight heads, where each atom's node embedding is the query and each text token's embedding is the key and value, so each atom can attend to different words rather than a single pooled sentence vector. The supporting pretraining task is masked node prediction, a task that hides 50% of graph nodes and asks the model to recover their element types from surrounding graph and text context. This pretraining is what changes attention from uniform stripe patterns, where all atoms look at the same tokens, to diverse node-specific distributions, which the paper connects to lower MAE.

What would settle it

Use the same pretraining and finetuning pipeline but replace the 512-token text encoder with one that accepts the full Robocrystallographer description; if longer text does not improve MAE over the truncated version, or if the cross-attention gain over descriptor vectors disappears, the claim that text fusion drives the improvement is undermined.

Watch

Extended reading notes

Core claim

CAST integrates a crystal graph encoder and a materials-science language model through cross-attention, where atom embeddings act as queries and text token embeddings act as keys and values. Before regression finetuning, the model is pretrained by masking 50% of nodes and predicting their element types, which the paper shows changes attention from uniform, stripe-like patterns to diverse node-specific patterns. On test sets from Materials Project, CAST reaches MAE of 0.256 for total energy, 0.354 for band gap, 0.069 for log shear modulus, and 0.039 for log bulk modulus, beating the best baseline on three tasks and trailing by only 0.001 on the fourth. The paper also reports that replacing text embeddings with descriptor vectors loses an average of 13.6% MAE improvement, supporting the claim that language-model embeddings carry information numeric descriptors do not.

Load-bearing premise

The central claim depends on the assumption that compressing the automatically written Robocrystallographer text summaries, which average about 1,800 tokens, down to the 512-token limit of the chosen text encoder still preserves the global structural information that the text modality is supposed to add.

Editorial extensions

If this is right

  • Any structure-based materials model can in principle gain a global-context channel by adding a text encoder and cross-attention, without changing the underlying GNN.
  • Masked node prediction is a transferable pretraining objective: masking a fraction of atoms and using text to recover them can align any pair of graph and text encoders.
  • The attention-diversity analysis implies that the benefit of multimodal pretraining is visible before downstream evaluation, so attention maps can serve as a diagnostic.
  • The descriptor-vs-text ablation suggests that future descriptor engineering should be weighed against simply using a pretrained language model, since text embeddings outperformed descriptors by 13.6%.
  • For bandgap, the property where the unimodal GNN was strongest, pretraining produced a 7.1% improvement, indicating the multimodal approach specifically helps where local graph structure alone is insufficient.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: run CAST with a long-context text encoder on the full Robocrystallographer descriptions; the paper's 512-token truncation is never described, so full-length text could either strengthen or weaken the reported gains.
  • The attention-diversity metric could be turned into a general early-stop or debugging signal for multimodal pretraining, though the paper only uses it for post-hoc analysis.
  • Because the paper uses Robocrystallographer as the sole text source, an open question is whether the fusion benefit survives with other description generators or with text whose style is less structured.
  • If the gains replicate on larger and more diverse datasets, the aligned embeddings could be reused as a foundation representation across many properties rather than trained per property.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CAST, a cross-attention model that fuses graph node embeddings from the coGN structure encoder with text token embeddings from MatSciBERT, and introduces a masked node prediction (MNP) pretraining step intended to align the two modalities. The model is evaluated on four Materials Project regression tasks: total energy, bandgap, and logarithmic shear and bulk moduli. The authors compare CAST against unimodal baselines (coGN, MatSciBERT), multimodal baselines (CrysMMNet, CrysMMNet-LoRA, MultiMat), and an ablation without pretraining (CAST-base), reporting mean MAE over three random seeds. They report average relative MAE improvements of 10.2% to 35.7% over the baselines, along with attention-map analyses suggesting that pretraining increases the diversity of node-to-token attention patterns.

Significance. If the reported results hold, the paper makes a useful contribution to multimodal materials property prediction by demonstrating that token-level cross-attention fusion plus masked node pretraining can outperform concatenation- and contrastive-based fusion on a standard benchmark. The manuscript has several strengths: results are reported with means and standard deviations over three seeds; ablation studies isolate the contributions of pretraining, masking ratio, and text encoding versus descriptor vectors; and the attention analyses provide qualitative evidence for the pretraining mechanism. The main caveats are that Robocrystallographer text is generated from the same crystal structures, so the text modality is an alternative encoding rather than an independent knowledge source, and that the paper's central claims are weakened by an unreported text-truncation procedure and by an abstract that overstates the per-property superiority of the method.

major comments (3)
  1. [Section 4.1.2 and Table 2] The manuscript never specifies how Robocrystallographer descriptions are truncated, padded, or windowed before being passed to MatSciBERT, whose maximum input length is 512 tokens. Table 2 reports mean token counts of 1,796 for Etot and 2,326 for bandgap, with standard deviations above 3,500, so a large fraction of the texts cannot be fed in full. Because the paper's central motivation is that text supplies global structural information that GNNs lose, and because the largest reported gains over CrysMMNet-LoRA occur for Etot and bandgap, the missing preprocessing detail leaves the interpretation of the text-modality benefit underdetermined and the experiment unreproducible. Please state the exact truncation strategy and, if truncation is used, provide an analysis of whether the omitted portions contain the global descriptors (e.g., space group, connectivity) that motivate the method.
  2. [Abstract and Section 2.2] The abstract claims that CAST "outperforms existing baseline models across four key material properties," but Table 1 shows that on log(Kvrh), CrysMMNet-LoRA achieves MAE 0.038 while CAST achieves 0.039. The paper itself concedes this in Section 2.2 by noting that CAST trails the best baseline by 0.001 MAE on log(Kvrh). The 10.2%-35.7% figures are average relative improvements per baseline, not per-property improvements. Please revise the abstract and conclusion to state average improvements explicitly and to acknowledge the per-property exception, or the central claim will be read as overstating the results.
  3. [Section 2.2, Table 1] The comparison does not include several recent multimodal baselines cited in the paper's own references, such as UniMat (ref. [14]) and Lattice Lingo (ref. [12]). Since the abstract and conclusion phrase the contribution as outperforming "existing baseline models" without qualification, the claim is broader than the evaluation. Either add at least one of these recent methods to the comparison or explicitly scope the claim to the methods actually evaluated.
minor comments (4)
  1. [Data availability] The Data availability section only provides access to the Materials Project; no code, model weights, or training configuration files are released, which limits reproducibility beyond the reported statistics. Please consider releasing the code and the exact data-processing scripts.
  2. [Methods, Section 4.3] The manuscript does not report several training hyperparameters, including the number of epochs, learning rate, weight decay, and early-stopping criteria. Reporting these would improve reproducibility and make the comparison with baselines easier to assess.
  3. [Figure 4 caption and Section 2.4] The caption states that CrysMMNet outperforms coGN-desc by 13.6%, but the main text reports reductions relative to coGN of 5.7% for coGN-desc and 17.5% for CrysMMNet; the arithmetic behind the 13.6% figure is not transparent and should be clarified, ideally by reporting per-property values.
  4. [Author affiliations and text formatting] There are formatting errors, including "Koreal" in the author affiliation and broken LaTeX macros in Section 4.1.1 such as "GV oigt" and "KReuss"; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CAST's reported gains are supervised regression results on held-out splits, and the cited self-works are not load-bearing.

full rationale

The paper's central claim is an empirical comparison of supervised regression models on a fixed train/validation/test split (8:1:1), with all methods evaluated by MAE on the same held-out targets. No equation reduces a fitted quantity to the predicted target, and no pretraining or fusion step uses the target property values to construct the input features. The text descriptions are generated from the same crystal structures via Robocrystallographer, so the text modality is not independent of the graph modality; however, this is a data-construction and interpretability concern, not a circular derivation, because the prediction target is not used to generate the text or to define the model. The paper's self-citations, [12] and [15], appear only in a survey-style enumeration of existing multimodal materials works and are not used to justify the central architectural claim or to forbid alternatives. The acknowledged 512-token limitation of MatSciBERT raises reproducibility and interpretation questions about truncation, but it does not make any prediction equivalent to an input by construction. The masked node prediction pretraining predicts element types, not the target properties, and the fine-tuning is a standard supervised regression. Therefore, no circular step meeting the evidentiary bar is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper contributes an empirical architecture, not a derivation. Its central claim depends on pretrained encoders, Robocrystallographer text quality, dataset filtering choices, and an unspecified text truncation strategy. No new physical entities are introduced.

free parameters (4)
  • masking_ratio = 50%
    MNP pretraining masks 50% of nodes; the ablation in Supplementary Information 5.1 shows higher masking improves downstream MAE by 3.9%, so this hand-chosen value affects the central claim.
  • cross_attention_layers = 4 layers, 8 heads, 128 dim
    Fusion module capacity is chosen by hand with no reported search; it directly controls how much node-token interaction the model can express.
  • batch_size = 64 (360 for contrastive pretraining)
    Batch sizes are set following the MultiMat paper for contrastive learning and chosen as 64 elsewhere; they affect optimization and pretraining dynamics.
  • data_filter_thresholds = 150 meV/atom above hull; -10 eV/atom floor; 1000 GPa cap; noble-gas exclusion
    The custom dataset filters from Section 4.1.1 define the benchmark and may change relative model performance; they are hand-chosen additions to MatBench criteria.
assumptions (4)
  • standard math Transformer cross-attention and GNN message passing are valid and implemented correctly.
    The architecture relies on standard attention and graph convolution; no proof is given, which is normal for an applied ML paper.
  • domain assumption coGN and MatSciBERT pretrained weights provide useful inductive biases for crystal property regression.
    The paper adopts these encoders without retraining from scratch; their quality is taken from prior work in Sections 4.2.1 and 4.2.2.
  • domain assumption Robocrystallographer text descriptions encode global structural information that complements the graph representation.
    The entire motivation for the text modality rests on this; Section 4.1.2 generates text with default settings and Section 2.4 uses it as evidence.
  • domain assumption Long descriptions can be truncated or compressed to MatSciBERT's 512-token limit without losing property-relevant information.
    Table 2 reports mean tokens around 1,796, while the conclusion in Section 3 acknowledges the 512-token limit; the paper does not state the handling strategy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAST: Cross Attention based multimodal fusion of Structure and Text for materials property prediction." pith.science (2026). https://pith.science/paper/SRSMBXMN

@misc{pith2026250206836,
  author       = {Pith},
  title        = {Pith review of: CAST: Cross Attention based multimodal fusion of Structure and Text for materials property prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRSMBXMN}},
  note         = {Machine review of arXiv:2502.06836}
}
read the original abstract

Recent advancements in graph neural networks (GNNs) have significantly enhanced the prediction of material properties by modeling crystal structures as graphs. However, GNNs often struggle to capture global structural characteristics, such as crystal systems, limiting their predictive performance. To overcome this issue, we propose CAST, a cross-attention-based multimodal model that integrates graph representations with textual descriptions of materials, effectively preserving critical structural and compositional information. Unlike previous approaches, such as CrysMMNet and MultiMat, which rely on aggregated material-level embeddings, CAST leverages cross-attention mechanisms to combine fine-grained graph node-level and text token-level features. Additionally, we introduce a masked node prediction pretraining strategy that further enhances the alignment between node and text embeddings. Our experimental results demonstrate that CAST outperforms existing baseline models across four key material properties-formation energy, band gap, bulk modulus, and shear modulus-with average relative MAE improvements ranging from 10.2% to 35.7%. Analysis of attention maps confirms the importance of pretraining in effectively aligning multimodal representations. This study underscores the potential of multimodal learning frameworks for developing more accurate and globally informed predictive models in materials science.

Figures

Figures reproduced from arXiv: 2502.06836 by the authors.

Figure 1
Figure 1. Overview of the CAST framework. (a)Pretraining: Mask 50% of graph nodes and predict their element types via cross attention between node embeddings and text embeddings. (b)Finetuning: Replace classification head with a regression head to predict material properties using the aligned multimodal embeddings. ments compared to coGN(28.8%), MatSciBERT(24.7%), CrysMMNet(24.6%), CrysMMNet-LoRA(10.2%), MultiMat (35.7%), and… view at source ↗
Figure 2
Figure 2. Cross attention maps for test set crystal mp-1963. Top: CAST-base(w/o pretraining) showed uniform attention(stripe patterns) Bottom: CAST(w/ pretraining) exhibited more diverse attention across tokens, indicating stronger cross modal alignments. More examples can be seen in supplementary information Figure S5. (Supplementary Information 5.1) showed that a higher masking ratio (50%) improved prediction performance by… view at source ↗
Figure 3
Figure 3. Distributions of cosine similarities between node pair attention vectors across layers. An example illustrated the computation of cosine similarity between pairs of node attention vectors. Top row showed the distribution of cosine similarity values for CAST-base(w/o pretraining), with most values concentrated near 1. Bottom row showed the distribution for CAST(w/ pretraining), exhibiting a broader range that reflect… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Descriptor vs text embeddings. (a) Model architecture for comparison of vector and text coGN-desc concatenated descriptor vectors with GNN embeddings. CrysMMNet concatenates text embeddings. (b) Prediction performance comparison coGN-desc achieved an average test MAE r…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Structure-Informed Learning of Flat Band 2D Materials

    cond-mat.mtrl-sci 2025-06 conditional novelty 7.0 of 10

    A structure-only machine-learning model, trained on a physics-based flatness score, predicts and validates new 2D materials with topological flat bands, including Nb3TeI7 and Cu3AsO4.

Reference graph

Works this paper leans on

32 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [14]

    Unimat: Unifying materials embeddings through multi-modal learning

    Janghoon Ock, Joseph Montoya, Daniel Schweigert, Linda Hung, Santosh K Suram, and Weike Ye. Unimat: Unifying materials embeddings through multi-modal learning. arXiv preprint arXiv:2411.08664, 2024

    arXiv 2024

  2. [12]

    Lattice Lingo: Effect of Textual Detail on Multimodal Learning for Property Prediction of Crystals

    Mrigi Munjal, Jaewan Lee, Changyoung Park, and Sehui Han. Lattice lingo: Effect of textual detail on multimodal learning for property prediction of crystals. arXiv preprint arXiv:2412.04670, 2024. 10 A PREPRINT - AUGUST 11, 2025

    work page Pith review arXiv 2024

  3. [1]

    Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties

    Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14):145301, 2018

    work page 2018

  4. [2]

    Directional message passing for molecular graphs

    Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint arXiv:2003.03123, 2020

    arXiv 2003

  5. [3]

    Gemnet: Universal directional graph neural networks for molecules

    Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790–6802, 2021

    work page 2021

  6. [4]

    Atomistic line graph neural network for improved materials property predictions

    Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1):185, 2021

    2021

  7. [5]

    Connectivity optimized nested line graph networks for crystal structures

    Robin Ruff, Patrick Reiser, Jan Stühmer, and Pascal Friederich. Connectivity optimized nested line graph networks for crystal structures. Digital Discovery, 3(3):594–601, 2024

    work page 2024

  8. [6]

    Scaling deep learning for materials discovery

    Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80–85, 2023

    work page 2023

Show all 32 references
  1. [7]

    Examining graph neural networks for crystal structures: limitations and opportunities for capturing periodicity

    Sheng Gong, Keqiang Yan, Tian Xie, Yang Shao-Horn, Rafael Gomez-Bombarelli, Shuiwang Ji, and Jeffrey C Grossman. Examining graph neural networks for crystal structures: limitations and opportunities for capturing periodicity. Science Advances, 9(45):eadi3245, 2023

    2023

  2. [8]

    Large-scale multi-modal pre-trained models: A comprehensive survey

    Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, 20(4):447–482, 2023

    2023

  3. [9]

    Multimodal learning with transformers: A survey

    Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113–12132, 2023

    2023

  4. [10]

    Graph-text multi-modal pre-training for medical representation learning

    Sungjin Park, Seongsu Bae, Jiho Kim, Tackeun Kim, and Edward Choi. Graph-text multi-modal pre-training for medical representation learning. In Conference on Health, Inference, and Learning, pages 261–281. PMLR, 2022

    2022

  5. [11]

    Crysmmnet: multimodal representation for crystal property prediction

    Kishalay Das, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. Crysmmnet: multimodal representation for crystal property prediction. In Uncertainty in Artificial Intelligence, pages 507–517. PMLR, 2023

    2023

  6. [13]

    Graph-text contrastive learning of inorganic crystal structure toward a foundation model of inorganic materials

    Keisuke Ozawa, Teppei Suzuki, Shunsuke Tonogai, and Tomoya Itakura. Graph-text contrastive learning of inorganic crystal structure toward a foundation model of inorganic materials. Science and Technology of Advanced Materials: Methods, (just-accepted):2406219, 2024

    2024

  7. [15]

    Clcs: Contrastive learning between compositions and structures for practical li-ion battery electrodes design

    Jaewan Lee, Changyoung Park, Hongjun Yang, Sehui Han, and Woohyung Lim. Clcs: Contrastive learning between compositions and structures for practical li-ion battery electrodes design. In AI for Accelerated Materials Design-NeurIPS 2023 Workshop

    2023

  8. [16]

    Robocrystallographer: automated crystal structure text descriptions and analysis

    Alex M Ganose and Anubhav Jain. Robocrystallographer: automated crystal structure text descriptions and analysis. MRS Communications, 9(3):874–881, 2019

    2019

  9. [17]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  10. [18]

    Multimodal foundation models for material property prediction and discovery

    Viggo Moro, Charlotte Loh, Rumen Dangovski, Ali Ghorashi, Andrew Ma, Zhuo Chen, Samuel Kim, Peter Y Lu, Thomas Christensen, and Marin Soljaˇci´c. Multimodal foundation models for material property prediction and discovery. Newton, 1(1), 2025

    2025

  11. [19]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  12. [20]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  13. [21]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota, 2019

    2019

  14. [22]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019

    1907 arXiv

  15. [23]

    Matscibert: A materials domain language model for text mining and information extraction

    Tanishq Gupta, Mohd Zaki, NM Anoop Krishnan, and Mausam. Matscibert: A materials domain language model for text mining and information extraction. npj Computational Materials, 8(1):102, 2022

    2022

  16. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  17. [25]

    Transtab: Learning transferable tabular transformers across tables

    Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35:2902–2915, 2022

    2022

  18. [26]

    Commentary: The materials project: A materials genome approach to accelerating materials innovation

    Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materi...

    2013

  19. [27]

    Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm

    Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials, 6(1):138, 2020

    2020

  20. [28]

    Scibert: Pretrained language model for scientific text

    Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: Pretrained language model for scientific text. In EMNLP, 2019

    2019

  21. [29]

    Efficient approximations of complete interatomic potentials for crystal property prediction

    Yuchao Lin, Keqiang Yan, Youzhi Luo, Yi Liu, Xiaoning Qian, and Shuiwang Ji. Efficient approximations of complete interatomic potentials for crystal property prediction. In International Conference on Machine Learning, pages 21260–21287. PMLR, 2023

    2023

  22. [30]

    The impact of domain-specific pre-training on named entity recognition tasks in materials science

    Nicholas Walker, Amalie Trewartha, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin Persson, Gerbrand Ceder, and Anubhav Jain. The impact of domain-specific pre-training on named entity recognition tasks in materials science. Available at SSRN 3950...

    2021

  23. [31]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learni...

    2011

  24. [32]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 11

    2016 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.